Haiku Depot Server - Architecture Modernization

Early deployment strategy

HaikuDepotServer (HDS) was first deployed in 2014. At that time, it was typical to deploy a small service such as HDS together with its resources on a server computer and hand-manage the server.

Although it had its downsides, this arrangement was easy to administer and worked well for some years. The Java-based application server would store working files on the local file-system, access the on-host Postgres database and could run the necessary graphics binaries on the local operating system too.

HDS at this time was packaged as RPM for deployment.

Introduction of Kubernetes and an Architectural Mismatch

Around 2018, the Haiku admin team moved away from deploying on static servers with RPM packages to using more modern container image + Kubernetes technology to run the support systems for the Haiku project. Kubernetes represents a more sophisticated deployment architecture where running applications are treated as ephemeral and can be started and recreated easily. HDS moved onto Kubernetes without substantial modifications; essentially operating as if it were still running on a curated static server. Ideally under Kubernetes there would be at least two HDS instances running concurrently but this wasn’t super easy when the software was expecting to store state on a local disk; an example of one of a number of constraints.

Moving to Redundant Deployment and Ephemeral Compute

During 2022 and 2023 the HDS system saw some modernization work to get the system from Spring 5 to SpringBoot 3. This year, the work slowly got underway to ensure that multiple instances of HDS can run at the same time on different Kubernetes nodes.

Factoring out Graphics Services

Early in 2025, some work was undertaken to separate out the graphics operations into a specific microservice.

Storage System

The next step was to refactor the storage system away from a local disk solution. The data being stored is “object storage” in nature such as reports, pre-prepared data to supply to HaikuDepot etc… The source of truth for the stored objects is the main relational database. This means the stored objects can be treated as disposable and don’t need to be backed up but they still need to be stored somewhere for a period.

An off-the-shelf cloud object storage solution would have been ideal, but given…

• the low volume of material involved
• a desire for a low-cost solution
• the practical need for minimal admin overhead

…the best strategy that functions well with multiple instances is to use the existing Postgres database in a separate schema. The stored objects are broken into fragments across multiple table rows which allows for chunked streaming on both input and output.

Jobs System

The jobs sub-system of HDS is designed to perform asynchronous background tasks such as;

• handle import of Haiku package repository files (HPKR)
• perform maintenance tasks
• import localization data
• produce any reports
• produce payloads for the HaikuDepot application

The jobs system was working in-memory and so was only able to operate on a single HDS instance. This has now been refactored with the job system’s state stored in the main Postgres database with the instances of HDS collaborating around the same store. HDS instances jostle to take work from the job queue stored in the database using skip-locks to prevent contention.

The Postgres database’s LISTEN / NOTIFY system has been employed to relay a message from one HDS instance to the others so that all instances know about the availability of work in the job queue. Traditionally, specific off-the-shelf systems such as AWS SNS or RabbitMQ would be employed for a job like this, but again, in the interests of keeping the solution simple at current scale, the capabilities of the LISTEN / NOTIFY mechanism are quite sufficient.

ORM Query Caches

The Apache Cayenne object-relational mapping (ORM) system is used widely in HDS to interact with the database. HDS uses Cayenne with an in-memory query cache. The query caches are dropped in a given instance when database changes are made through the ORM in same instance. This caching system poses a problem with multiple HDS instances running at once because the state of the instances’ in-memory query caches will become inconsistent with the state of the data in the database. This happens because database changes initiated in an instance would only cause query cache drops in the same instance and not the others.

The Postgres LISTEN / NOTIFY system has also been used to solve this problem. If a database change is made from one instance, the instance will inform the other instances which query caches should be consequently dropped by sending out messages. This mechanism isn’t bullet-proof, but it’s absolutely fine for the nature of the data in the HDS project.

Sundry

There are a number of other smaller accommodations required but, in the end, nothing too major. The rework of the jobs system was the largest and most complex component involved.

The job and data services are accessed through Java interface-s put in place early in the project. These interfaces abstract the actual implementation. Leveraging the interface-s, it was possible to quasi config-flag each tranche of changes across into running system over the last 9 months or so. When the solution of using the Postgres database for object storage and inter-process communications is no longer viable, these same Java interface-s or, descendents of them, should likewise provide another upgrade path.

Speaking of Java, the upgrade to Java 25 was also carried out in the same time-frame.

Conclusion

The end result is that HDS now has the capability of running on ephemeral compute; if the compute instance or instances fail then new instances can be started and nothing is lost. It’s also possible to operate multiple instances of HDS across multiple Kubernetes nodes such that if load increases, additional compute capacity can be stood up.

Using Spring’s dependency injection system and Java interface-s, it has been a relatively smooth job to transition the underlying sub-systems to the more modern architecture.

For now, it’s been possible to keep the deployment simple by avoiding introduction of additional infrastructure; the existing Postgres server has been able to provide the necessary capabilities instead. A low-maintenance philosophy is a key factor in keeping the project viable given the limited person-hours available.

Where to next

Ideally HDS would be running with two instances all the time, but there are presently not enough funds to operate additional server capacity to do this. Hopefully running multiple instances will become possible some time in 2026 but, it’s not a big problem if not.

The next piece of work around HDS is related to catering for the growth in data. When a current data download is not already cached for a HaikuDepot client, HDS produces it in real time using some database query magic. This process was instantaneous in 2014, acceptable in 2020, but alas, in 2025 it’s starting to get too slow. This arrangement will change to one where HDS publishes the downloaded data periodically. The downside is that data may be a little stale, but it will be available without delay to the client. Hopefully this improvement, leveraging the changes above from 2025, will be completed sometime during 2026.

Thanks to

Thanks for the Haiku admin team for supporting the deployment and infra required to run HDS. There have been a lot of little deployments to get this through.