Iterator-function to piggyback a lazy evaluation

[Ichthyostega]

As a Lumiera developer,
I want to pass an entirely self-contained iterator over interfaces,
to represent a pre-arranged lazy evaluation without tight coupling to the implementation.

represent a preconfigured computation as a value

Iteration can be seen as a form of processing data within a container — or it can be conceptualised as a lazy evaluattion. The design of Lumiera is leaning towards the second option, as iswitnessed by the concept of the Lumiera Forward Iterator.

However, sometimes a possible lazy evaluation must be passed over an interface, and the optimal arrangement then depends heavily on the circumstances:

• if retrieving an evaluation is directly a capability expressed on that interface, so that it can be initiated at will by the client, then a classical iteration interface is preferrable (⟶ see the IterSource interface)
• yet when the possible future evaluation arises rather as part of an ongoing process — which implies that the evaluation is »pushed« towards the client — then it can be beneficially to model this future evaluation more like a value, and less like a service.

In the latter case it can be desirable to shape the ability to engage into the evaluation more like a function, abstracting an otherwise opaque value handle.

Such an Iterator-function should not define a new scheme for iteration though, but is expected to fit into the framework for iterative processing, that has evolved within the Lumiera code base since a long time. This requirement implies that the actual activities should map down into the scheme of a »State Core«, while being encoded such as to be represented as a single function signature.

Create an opaque holder, allowing to pass an iterator "piggyback" thorugh C interfaces.

This is an obvious extension, and while we don't need it right now, we'll need it quite sure when the Session interface matures. Because usually iterators are implemented as lightweight value objects (typically with 2 pointers), it isn't difficult to "shock freeze" them into a small array (of implementation dependant size) and revive them on the other side basically with zero overhead.

[Ichthyostega]

hint: the PolymorphicValue template implements the same idea in a way more general frame, allowing to pass subclasses within an opaque package. When stripping most of the involved metaprogramming magic, we should be almost there...

Yet I still think we should have a dedicated small template for piggybacking iterators, instead of (ab)using PolymorphicValue for this purpose. Still the more since iterators in Lumiera (and in the STL as well) aren't derived from a base class.

[Ichthyostega]

our iteration pipeline builder (currently named TreeExplorer) has to deal with a releated problem: here we have to pick up a kind-of "implementation defined" concrete iterator / pipeline type and package it into an opaque container which exposes a virtual iteration interface (we use our IterSource which by itself has some design flaws, see #1125)

[Undercover Agent]

Migration MasterTickets ⟼ Subtickets-plugin

[Ichthyostega]

Add subticket #1421 (define Instance-Manager interface).

[Ichthyostega]

This is a very old ticket, and while the original requirement was driven by a push towards using plain-C interfaces within a plugin-centric architecture (a tendency, that was found to be detrimental and abandoned subsequently), the need for an abstracted yet self-contained iterator is a recurring theme, and often driven by the need to avoid tight coupling of unrelated components, as could arise through the use of dedicated callback functions. Prompted by current needs, the search for such a solution was revived recently.

Design Sketch

It should be acknowledged, that the protocol related to a »State Core« comprises three operations — and not a single one. Exposing these through a single function seems more like a distinct mapping step, then a quest for yet another iteration scheme; this implies that the three basic operations might be encoded as a function indicator ENUM. Another relevant question is how to represent the »exhausted« state, which is closely linked to the type of the result, as exposed by the iterator.

• by default, a Lumiera Forward Iterator yields a reference to the result. In that case, the result of the Iterator-function can be a pointer
• yet the possibility for Iterators that deliver result values is acknowledged and supported explicitly, albeit labelled as a slightly problematic corner case. If so, the result of the Iterator-function can be an optional.

These assumptions can be used to specify an iterator function. The intended operation is marked as CHECKPOINT, YIELD and ITERATE. Each of these variants will have to expose the current result, even while an adapter layered above such an Iterator-function will sometimes ignore this invocation result, or use it only to derive the state of the iteration (active, exhausted). This leads to defining the signature as

IteratorFunction

StateCoreAction ⟼ Result