Event-Log Undo and Replay

[Ichthyostega]

As Lumiera Architect,
I want analysis, decision and specification regarding Events and capturing of changes,
as consistent foundation for data manipulation, change propagation and persistence.

A pivotal decision

A goal defined right from the beginning of the Lumiera project was to provide unlimited »UNDO« capabilities; the initial understanding was to provide such a feature by persistently recording all commands issued by the user — each combined with a functor able to reverse its effects. However, as requirement analysis and design proceeded, increasing doubts emerged pertaining the feasibility of such an absolute command log.
To start with, numerous actions will only change the interaction state, e.g. moving the playhead around or zooming the timeline, while even some commands could be identified which both produce a difficult to reproduce interaction state change, as well as a change in the model. The prominent example would be starting a render for the first time, which might start an animated display of render progress, possibly reconnect a preview viewer, but will also create a new render configuration, which must be persisted, possibly even creates a mark of a rendered range, but also most definitively alters the external environment (creates a file) in a way that can never be reproduced (without overwriting the previous copy, which in turn might have been opened and integrated into the editing composite meanwhile).
Another source of concern is the possibility to generate inconsistent model state by repeated un-doing and re-applying of commands, and these inconsistencies may either be transient only or even persistent. Here, a transient discrepancy means that the present state of the application and even the results of rendering from this state will not be reproducible after re-loading from persistence, while a persistent discrepancy could lead to the even worse consequences of corrupting the model state, thereby tainting an extended amount of work.

Over time, the design of the application increasingly shifted towards asynchronous message based processing and loosely coupled subsystems, putting the reliance on a central model data structure as such into question. As it turns out, similar large-scale development efforts, especially in industrial context, were facing the same issues and concerns, and the ensuing professional discourse led to an architectural model known as Command and Query Responsibility Separation (CQRS). Building this kind of architecture is known to be challenging, and requires experienced and disciplined developers, but could be shown successfully to address the aforementioned issues, which are hard to cope with otherwise.

So, as of this writing (9/2023), the intention is to further stir the Lumiera project into this direction. Unfortunately, another deeply entrenched antinomy arises from the goal to open Lumiera towards rules-based configuration, which likewise is a cornerstone of the project. Rules — consider especially the codification of environmental givens like a studio setup — can be changed and re-evaluated, leading to non-localised and possibly disrupting changes. As such, a declarative specification is assumed to be the only source of truth, which is in direct contradiction to an Event-Log of happenings.

The Goal

In the context of a media building application, it seems this inner contradiction can be mitigated, by basing all decisions on the following principle: Whatever changes the tangible results must be captured in its entirety and recorded reproducibly as an Event. Care has to be taken when capturing consequences.

So the goal for this task is to show the viability of this approach. This can be achieved by developing a reasonably complete concept for Event-Sourcing in Lumiera. This concept need not be detailed, nor must it be implemented beyond a draft; it suffices to show that the fundamental problems can be addressed.

The Task

The following questions should be investigated and the results shall be documented:

• What is a tangible result? How can this be delineated from presentation state and interaction state?
• Can a formal language be developed to describe completely all causes for tangible changes, and do so precisely once, at a well defined boundary? Each »Event« would then be a sentence in this formal language, represented as symbolic term.
• To what extent can (and should) redundant consequences be captured alongside, and how can doing so be arranged structurally? Consider especially issues of numeric stability (floating point arithmetics), the use of random values (seed chaining) and the impact of timing (which is essentially random).
• How can the event language be adapted, extended and evolved over time? At which level to introduce versioning? How to handle breaking changes, especially in data representation? Consider both backward- and forward-compatibility.

[Undercover Agent]

Migration MasterTickets ⟼ Subtickets-plugin