out-of-tree build with SCons variant-dir

[Ichthyostega]

As Lumiera developer,
I want the ability to build variants of the code,
to allow for experimentation and research and to prepare release builds.

Limitations of SCons

The SCons buildsystem has a well-known legacy with problems related to the use of several build directories for out-of-tree builds. In the end, these problems are rooted in the fundamental way how SCons organises the build process: it defines a project model as a graph of relationships — instead of executing instructions or generating makefiles (as most other buildsystems do). Over time, these problems were largely mitigated by the SCons developers — yet support for out-of-tree builds remains to be a challenge for any scripting built on top of the SCons framework and project model. And, by embedding the project model definition DSL into the Python programming language, SCons outright encourages such customised scripting.

[Ichthyostega]

Assessment

• SCons v4 was improved to the degree that most difficulties with the so called variant dir are resolved
• yet various aspects of our own project-specific scripting become brittle when building out-of-tree
• all these secondary issues can be resolved — yet this resolution incurs some tricky considerations and remains difficult to understand and to maintain
• most importantly: there is an inner contradiction to our choice to establish a target directory and to attach installation targets automatically to the build results.

workaround

My own practical experience shows that the lack of a build-directory feature causes almost no practical problems. Two observations contribute to this findings:

• the use of build variants is always tied to very special cases or long-term commitments (like building with Clang). The very nature of these tasks implies that they are carried out irregularly and so require a full build most of the time.
• the findings from such additional research tasks and commitments are typically cross-cutting and need to be kept apart from ongoing development work.

Taken together, the use of a git clone and a separate tree is a better fit, since frequent and quick switches between different built configurations are rarely necessary.

Conclusion

We will not support that feature for the time being and continue to build in-tree.

However, the possible solution should be documented for later

[Ichthyostega]

Build directory (SCons variant_dir) -- investigation 10/2026

Knowledge-Technology(Claude Opus 5) was tasked with an assessment through implementing a prototype to determine the necessary adaptations. After review, the decision was made not to pursue this feature further. Claude was tasked to generate the findings report with implementation notes, included below.

---

Stashed findings from an exploratory investigation

Lumiera builds in-tree: object files land beside their sources (src/vault/*.os),
while all linked artefacts go to the relocatable target/ tree. This is widely considered
a quirk, so the question was whether SCons' variant_dir support could relocate the
intermediates, and at what cost.

---

Why one might want it

Two arguments, of very unequal weight.

• Conformity. In-tree builds are frowned upon. Weak in our case: *.o/*.os are gitignored, nothing is generated into the source tree (icons go to target/), and IDE clutter is a filter setting. This buys appearances, not capability.
• Side-by-side build configurations. Today, flipping DEBUG or OPTIMIZE in the sticky optcache invalidates the whole tree; with target/build/<config>/ one could keep debug and release simultaneously, and likewise try CXX=clang++ cheaply. This is a real capability -- but see the ramification below, and note that separate checkouts are the project's preferred answer to that need, since such experiments tend to breed changes one would rather not intermingle with development work.

What was measured

SCons 4.8.1, variant_dir=..., duplicate=0, in an isolated test project.

Works out of the box, contrary to folklore:

• Objects mirror the source tree below the variant root; the source tree stays clean.
• env.Append(CPPPATH='.') expands automatically to both sides (-Ibuild/x -Ix), so the tests/SConscript idiom survives untouched.
• #src and $TARGDIR are root-anchored and therefore unaffected -- the wrapped ELF builders and the relocatable target tree need no change at all.
• str(node) on a source node resolves to the real source file (verified: os.path.exists() is True). So custom builders reading files by path string keep working -- IconSvgRenderer and the Doxygen scanner among them.
• The CompilationDatabase builder produces correct source paths under variant_dir, in both the abspath and the relative mode.

The trap, and it is a nasty one:

SCons chdirs into the variant directory while reading an SConscript if that directory
already exists on disk, and into the source directory otherwise:

### build dir ABSENT:  CWD: .../vd/src         scanned: ['./a.c', './sub/b.c']
### build dir EXISTS:  CWD: .../vd/build/src   scanned: []

Consequence: Buildhelper.scanSubtree(), which drives the entire build through os.walk
and glob.glob, would find every source on the first clean build and silently find
nothing on every build thereafter. No error, just an empty target.

What would need adapting

1. Buildhelper: re-anchor the source scanning at env.Dir('.').srcnode().abspath (or switch to env.Glob). Affects scanSubtree, globRootdirs, findSrcTrees. This is the prerequisite, not an optional tidy-up. ~20 lines.
2. tests/SConscript: env.File("test.sh").abspath and env.File("test.conf").abspath -- .abspath yields the variant path, which does not exist under duplicate=0. Needs .srcnode().abspath. Likewise env.File(glob('*.tests')).
3. research/SConscript: envR.Append(CPPPATH='research') is root-relative rather than SConscript-relative; would need #research.
4. SConstruct: the SConscript(dirs=[...]) one-liner has to be unrolled into per-directory calls carrying variant_dir=. Only four directories actually compile anything -- src, src/tool, tests, research. data/ and doc/ produce no objects and should stay as they are, which incidentally keeps BuilderDoxygen out of it (it uses node.path in exists(), which would break).

The deeper ramification

variant_dir relocates object files only. The linked artefacts go to $TARGDIR,
which is root-anchored and thus unaffected -- so two configurations would still collide
at target/lumiera and target/modules/*.so. Obtaining genuinely side-by-side configs
therefore means making $TARGDIR config-dependent as well, and that reaches into the
relocatable target tree, setup.ini, the $ORIGIN RUNPATH convention, the install paths
and the DEB packaging. That is a considerably larger blast radius than the variant dir
itself, and it is the part that would actually have to be designed.

Verdict (10/2026)

Not pursued. The strong argument (side-by-side configurations) is answered in this
project by separate checkouts, which is the preferred practice for alternate toolchains
and alternate configurations anyway. What remains is conformity with what other developers
expect, which does not buy a capability.

The delegation-readiness move is therefore the cheap half rather than the expensive one:
state the in-tree build as a decision in the build documentation, rather than
restructuring. A newcomer does not need the directory layout changed; they need to know
it was a choice.

Should the question be reopened, item 1 above is the gate: everything else is mechanical.

---

Incidental finding while surveying: tests/SConscript has
env.Clean('check', [',testlog', ...]) with bare relative names, which resolve against
tests/ -- but test.sh runs with chdir=$TARGDIR, so those scratch files are actually
produced in target/. Confirmed: none of them exist in tests/. Harmless (SCons ignores
absent clean targets), but scons -c check never removes them.