preliminary BufferProvider for rendering

[Ichthyostega]

As a Lumiera developer,
I need »just some reasonably braindead« BufferProvider implementation,
so that I can use the Render Node network for actual media computations.

Acceptance Criteria

Given

• a suitably defined and connected setup of Render Nodes
• a render-job to trigger invocation of this network's exit point

When

the preliminary BufferProvider is configured through the EngineCtx

Then

the job can perform real media data computations

• with arbitrary (but reasonable) buffer sizes
• at »base level« performance (enough to compute a simple animation for a small window in real time)
• without leaking memory.

[Undercover Agent]

Migration MasterTickets ⟼ Subtickets-plugin

[Ichthyostega]

Add subticket #1410 (clean-up BufferProvider base implementation).

[Ichthyostega]

Attach to parent #1411 (Aliasing of Buffer TypeHandler instances).

[Ichthyostega]

Add subtickets to define a path towards a BufferProvider setup capable to operate in a massive multithreaded environment:

• #1425 Buffer handling slices
• #1426 Worker-local BufferProvider implementation

[Ichthyostega]

Identified and postponed a possible performance-penalty related to thread_local storage: see #1427 (performance-impact of thread_local).

Generally speaking, I prefer to build a clearly structured implementation of the engine first, placing more weight on separation of concerns (even at the cost of some additional indirection), than immediately engaging into a performance-centric solution; the rationale is that the latter is known to be a one-way route.

[Ichthyostega]

Add subticket #836 (frontend for plain simple allocations).

[Ichthyostega]

Attach to parent #1429 (control effort for global buffer allocator).

[Ichthyostega]

During the last two weeks, I have effectively laid the groundworks for a production-grade buffer manager for the render engine. What is there now goes far beyond »preliminary« — yet I skipped any elaboration regarding the Buffer Allocator.

Both »pillars« of the BufferProvider implementation are now configured as LocalSlice, so that each worker thread will see its own thread-local copy of the relevant low-level service — which implies there is a distinct BufferMetadata instance and a LocalMemPool in each thread. Furthermore, an asynchronous communication scheme has been established to connect those local services to the central EngineMetadata and EngineBufferManager

The latter can receive and submit allocation requests — yet will pass-through any actual allocation or deallocation directly to the regular heap allocator. However, the plan is to build a set of buffer pools here — so in this respect the current implementation is »preliminary« (and presumably good enough for quite some time to come)

Conducting some load tests currently, but the new implementation seems to work correct thus far...

[Ichthyostega <prg@…>]

In 9461443/Lumiera:

BufferProvider: successfully complete the load test (see #1396)

With this third test case, a fairly continuous processing
can be demonstrated, while also coordinating allocations
in a decentralised way.

The implementation behaves correct, and seemingly with
satisfactory performance -- while there is certainly
some room for improvement: with the processing pattern
in this test, a nominal computation load of 4 seconds
can be processed with 4 cores in ~ 1.3 seconds

[Ichthyostega <prg@…>]

In ab22f37/Lumiera:

MERGE dev/play into integration

Integrate the work related to buffer management in the render engine.

• in-depth analysis and rework of the BufferProvider framework (see #1395)
• build a prototype for production-grade buffer allocation handling (closes: #1415)(see #1396)
• as an aside: Application code and namespace reorganisation

[Ichthyostega <prg@…>]

In ab22f37/Lumiera:

MERGE dev/play into integration

Integrate the work related to buffer management in the render engine.

• in-depth analysis and rework of the BufferProvider framework (see #1395)
• build a prototype for production-grade buffer allocation handling (closes: #1415)(see #1396)
• as an aside: Application code and namespace reorganisation

[Ichthyostega <prg@…>]

In ab22f37/Lumiera:

MERGE dev/play into integration

Integrate the work related to buffer management in the render engine.

• in-depth analysis and rework of the BufferProvider framework (see #1395)
• build a prototype for production-grade buffer allocation handling (closes: #1415)(see #1396)
• as an aside: Application code and namespace reorganisation