Subsystem-Runner design rework

[Ichthyostega]

The Subsystem Runner in the application core is responsible to start and stop several self-contained units of operation within the Lumiera application, and to maintain their lifecycle and dependencies. It relies on the Subsystem Descriptor provided by each part, which in turn is an interface and contract to be fulfilled by each subsystem. This overall architecture worked out quite well and seems adequate for the time being — however, the implementation of the subsystem runner itself can be characterised as naive. At that time, in Dec.2008, we had a contentious discussion about the way of modularisation, and this subsystem concept arouse from a design sketch on which the project settled down eventually.

After a round of bugfixes and more in-depth test coverage, this „one-shot“ implementation turned out to be surprisingly robust over all those years, but the interplay of actions is hard to understand and relies way too much on some fine points and minute details of the implementation. We should review the design and rebuild that implementation from scratch, while mostly retaining the interface and the protocol. The architecture based on a small fixed number of coarse grained subsystems in itself is not questioned however.

Critique

Like most „I-want-everything-nailed-in-one-shot“ implementations, this piece of code wants to achieve way too much by overloading several concerns into a single structure.

• we need to distinguish between signalling, state protocol and the management work
• the reliability of state detection and protocol compliance should not be treated as afterthought
• we acknowledge that the process is highly complex and chaotic in its very nature
• the subsystem runner should operate within a safe-guarded (static) area with minimal dependencies
• no attempt to „keep it simple and stupid“ please. Matters are not that simple

Design Proposal

Each Subsystem Descriptor should be a self contained thread safe state machine. The subsystem itself should only ever have to talk to its descriptor, and never interact with the Subsystem Runner. The descriptor ABC should provide a complete set of hooks to be implemented by the subsystem, explicitly covering all phases of the lifecycle. We should avoid the common temptation to „cut some corners“ here; each lifecycle phase should be segregated and finished with unambiguous and reliable signalling. There must not be any contention regarding this signalling and the established lifecycle state. Moreover, there should be well established timeouts, and timestamps on each step in the lifecycle. The descriptor exposes a dedicated API for the runner, just to query the lifecycle state unambiguously and without contention.

As an alternative proposal, instead of a set of state machines, the Subsystem Runner could possibly be based on a task queue, maybe even a priority queue, since there is dependency ordering. We should acknowledge the fact that planned tasks can be superseded by changed circumstances, like a prerequisite subsystem not reaching operative state, causing an emergency tear-down of the whole system. Like in a DB with two phase commit, final operative state should entered only after all prerequisites are guaranteed. And in a similar vein, on the shutdown side, each task must be constructed such as to guarantee it will progress eventually, to avoid deadlocks.

[Ichthyostega]

As part of the »Playback Vertical Slice« effort in 2023, the implementation technique for thread handling was reassessed, leading to a switch from a POSIX implementation base to the concurrency framework available in the standard library since C++14. During this rework, the interplay of threads in subsystems with the Subsystem Runner was reinvestigated in detail, and while the general reasoning seems sound, there are indeed some brittle corner cases, especially when a subsystem fails to complete its initialisation. Understanding the behaviour of this piece of code turns out especially challenging because the current lifecycle phase of each subsystem is treated in a purely contextual way, and not materialised into a state flag.

The Subsystem Runner is one of the very few users of the object monitor's conditional waiting capabilities — and the mentioned refactoring incurred some API changes pertaining that part. This led to a sharpened definition of »emergency« in the shutdown phase: it is now understood as a heightened danger for deadlocks. Under normal circumstances, the assumption is that each subsystem is able to catch all major problems and maintain a sufficient level of control — especially regarding the ability to react reliably to a shutdown request. Yet if the top-level of a subsystem exits exceptionally, other parts of the application might also be already in partially corrupted state, increasing the probability of a regular shutdown request to become stuck. In such a case it seems adequate to set a timeout on waiting for all parts of the application to wrap up — even if cutting this wait leads to follow-up errors and hard termination in the post-main() static clean-up phase. From a user's perspective, given that all work is always properly recorded by the event logging system anyway, it is desirable for a corrupted system to unwind quickly and to signal an error exit to the operating system.

[Undercover Agent]

Migration MasterTickets ⟼ Subtickets-plugin