Haiku Depot Server - Publishing Data
Following on from “where to next” in this article, a project has been rolled out in HaikuDepotServer (HDS) over the first half of 2026 related to how data is marshaled and delivered to the desktop application.
HDS was producing bulk data to send to the desktop application in real time. Producing data in this way worked very well early on and was simple to implement, but as the quantity of data has grown over the last decade, the technique has again needed refinement.
In reality, HDS was actually caching the data and delivering from cache, but if the data was stale or the cache was cold then it would have to revert to an on-the-fly approach. Users of the desktop application will have sometimes experienced delays in the past as the server produced the requested data and, in some cases, the first request from a desktop application may have timed out because it took too long.
A data publishing architecture works differently. Instead of being reactive to incoming requests, the system will instead produce + store data proactively and inbound requests for data will be satisfied from the published data storage.
Publishing the data is made somewhat complex because the payloads returned to the desktop application are locale-specific. As an example, a user requesting data for en-AU (Australian English) locale are going to receive different data to a user requesting data for pt-BR (Brazilian Portuguese).
The publishing system runs a scheduled task every 5 minutes to refresh the data. Some popular languages are always considered alongside any languages requested in the last 48 hours. For each language, the logic will assess if the underlying data was published in the last 15 minutes. If there was data less than 15 minutes old then it will skip producing new data to avoid publishing too often. Otherwise, various data payloads are produced and stored;
- Repositories (
repository_all_<locale>.json.gz) - Reference-data (
reference_all_<locale>.json.gz) - Packages’ icons (
pkgicon-all.tar.gz) - Package data for each repository of interest (
pkg-all-<repositorysourcecode>-<locale>.json.gz)
You can see these files at ${HOME}/config/cache/HaikuDepot in your Haiku system that have been downloaded after you use Haiku Depot.
What happens when a request arrives for a language that has no published data? The server will generate data for the requested language on-demand but as discussed above, this can take some time. Recent desktop applications (version >= 0.0.10) receive an HTTP 503 Unavailable response, back-off and try again after a delay. For older desktop clients that don’t have this capability, the server will block until the data is ready. Eventually, this latter approach won’t work because the HTTP request will timeout, but hopefully most users will be migrated to a newer version before this is a concrete problem.
On the server side, the data is persisted into a structure within the HDS Postgres database. Storing data of this form into a relational database isn’t ideal, but given the relatively low volumes, taking this approach will keep the deployment and administration overhead low for the time being; interfaces are in place to allow a shift to a dedicated object storage system such as AWS S3 or Wasabi later when it is necessary.
A downside of the new publishing system is that HDS may be serving data which is not current with the data in its database but the delay should not be too large and, it probably won’t have a huge impact on users. Most likely the only situation where users will notice is if they update a rating on a package and, it takes some minutes to be reflected on others’ systems.
With this change in place, in the vast majority of cases, data will be delivered to the desktop application without delays or timeouts.