Text templating engine

[Ichthyostega]

As a Lumiera developer,
I want to perform placeholder substitution in a text template,
to formalise and document the process of code generation for external tool integration.

Research: a very common task without obvious solution

There must be something readily available for such an ubiquitous task — but surprisingly enough, other than for Python, not even the least common denominator is provided in a standardised way within the C++ ecosystem. Already the very foundation, which is the rendering into a formatted string, is riddled with complexities, and available solutions are lacking in scope. For tasks like debugging messages and error reporting, Lumiera happens to maintain its own front-end as util::_Fmt, backed by the implementation of boost::format, with added failure safeguards and integrating the rendering of custom data types — which is only viable within the confines of a dedicated development project and can not be part of a general purpose library.

There seem to be several reasons leading to this surprisingly challenging situation. For one, by deliberate choice, the C++ language rejected the idea of a reflection framework back in the 90ies. While, at that time, the refusal to address one of the user's most common demands (together with perceived complexity and instability) contributed much to the language's falling out of favour, after an initial hype cycle. Notably the Java language, appearing at the same time, took the opposite stance, both on reflection and garbage collection, which contributed to its tremendous success, but also shaped the notorious and unsurmountable problematic aspects haunting the Java ecosystem ever since. While, in turn, the deterministic memory management and the ability to project an object- and type-centric abstraction onto even the tiniest bits of data with zero runtime overhead, became unique traits of C++, not shared by any other programming language in widespread use.

It seems prudent thus to stick to the specific traits and strengths of the language. Unfortunately, people's expectations are shaped by the abilities of dynamic data discovery, which is especially viable with dynamic languages. This mismatch of expectations might explain why every library or framework I have reviewed employs its own dynamic object or property-tree framework. A secondary problem, which seems related, is that of scope; most libraries remaining active until today offer a wide range of features, often amounting to an embedded secondary document manipulation language of sorts. Moreover, the primary use case seems to be dynamic content generation for the web, where a text templating language together with a clever data binding is employed to provide features akin to a »CMS light«.

Notable library solutions for C++ in active use

• ​TEng
  • a variant for C++ and for Python is provided
  • compiles the text template into some form of byte code
  • this compiled template is invoked with a dictionary to generate stream output
  • provides an elaborate caching system to memorise template instantiations,
    even with specific dictionary instances, to speed-up individual page impressions
• ​Jinja
  • is a text templating engine and language, originally developed for python
    and modelled after the templating mechanisms of Django
  • typically highly configurable, offering conditionals, looping constructs and processing functions
  • ​Inja is »jinja for C++«
    • inja::render("Hello {{ name }}!", data);
    • the placeholder / escape bracket syntax is configurable
    • can be extended by function / λ-callback from the templates
    • templates can specialise other templates as building-block
    • header-only C++ lib
      • ⚠ requires ​nlohmann/json.hpp, which in itself is a very notable piece of software, offering complete JSON support through a single pure C++ header. JSON-objects can be constructed using the familiar nested initialiser syntax of C++
      • Inja requires all data to be provided in the form of such JSON-objects.
  • ​jinja2cpp
    • is a mature library developed actively since 2018
    • full library (not header-only) with CMake build
    • has further build dependencies like Boost, and the Nonstd-libs
    • supports nlohmann/json.hpp (see above), but supports also RapidJSON
• ​»Mustache«: a »Standard for logic-less Templates« in many languages
  • instead of embedding a manipulation language, several kinds of tags work in concert with the structure of the actual data provided
  • these tags offer rather tricky semantics, especially when picking up further mustache templates embedded into nested data scopes
  • is specifically oriented towards HTML generation (e.g. built-in HTML / XML escapes)
  • ​mstch is »Mustache for C++11«
    • a library implementation, 8 years old, stale
    • has its own JSON implementation based on Boost::Variant
    • allows to bind C++ objects as view model
  • ​kainjow/Mustache »for modern C++«
    • a single-header implementation (provided under Boost-Software-License)
    • stable and in maintenance mode (author accepts pull requests, last 3 years ago)
    • based on a custom data type with set-like behaviour and nested scopes

      mustache tmpl{"{{#employees}}{{name}}, {{/employees}}"};
      data employees{data::type::list};
      employees << data{"name", "Steve"} << data{"name", "Bill"};
      tmpl.render({"employees", employees}, std::cout);
      // generates: "Steve, Bill, "
      

    • few documentation, but very extensive unit-tests
• ​NLTemplate
  • rather simple and well focused text templating engine
  • has a C++98 and a modern-C++ branch, last pull requests merged 2022
  • provides in-memory backend and a load-from-file backend
  • data has to be supplied by direct setter invocations
  • supports named placeholders, repetitive blocks and include of nested templates
  • a single translation unit, standard C++, no further dependencies
  • only minimal unit-test coverage

All of the above are more-or-less maintained up to recent past and are mentioned at various places in the net. A lot of similar solutions used to exist but have gone stale during the last 10 years.

the essential functionality

It is worth mentioning that the essential functionality of text template instantiation can be implemented directly, based on standard C++ constructs. A canonical example can be found on ​Stackoverflow

[Ichthyostega]

Based on the above assessment of the situation, I decide to build a minimalistic engine myself.

• the feature set shall be confined to the bare minimum, without magic tags.
• only the abstract assumption of associative data is allowed, but no concrete data type
• and kind of binding has to be provided through extension points
• all processing and rendering functionality must be kept apart
• reasonable performance is sufficient

To restate the intent, I am about to generate a Gnuplot script, and want that integration to be extensible, readable and self-documenting to the degree possible

[Ichthyostega <prg@…>]

In 5881b01/Lumiera:

Library: work out a treatment for text template substitution (see: #1359)

• establish the feature set to provide
• choose scheme for runtime representation
• break down analysis to individual parsing and execution steps
• conclude which actions to conduct and the necessary data
• derive the abstract binding API required

[Ichthyostega <prg@…>]

In 918f96b/Lumiera:

Library: complete ETD data-source binding and test (closes #1359)

A minimalist TextTemplate engine is available for in-project use.

• supports only the bare minimum of features (no programming language)
  • substitution of ${placeholder} by key-name data access
  • conditional section ${if key}...${end if}
  • iteration over a data sequence
• other then most solutions available as library, this implementation does not require a specific data type, nor does it invent a dynamic object system or JSON backend; rather, a generic Data Source Adapter is used, which can be specialised to access any kind of structured data
• the following DataSource specialisations are provided
  • std::map<string,string>
  • Lumiera »External Tree Description« (based on GenNode)
  • a string-based spec for testing

[Undercover Agent]

Migration MasterTickets ⟼ Subtickets-plugin