DataLayerGenerator

📖 Full documentation on GitHub Pages

A header-only C++20/23 framework with a Python/Jinja2 code generator for building type-safe, zero-allocation data layers — targeting embedded systems (MCU / bare-metal) as well as host applications.

You describe your data model in JSON or YAML files. The generator produces a single datalayer.h header that contains fully type-safe constinit datapoint globals, with access control enforced at compile time, optional range checking, robust file persistence, change callbacks, optional fmt formatters, optional Python bindings, and dependency-free MQTT/CAN adapter boundaries.

• CMake ≥ 3.25
• C++23 compiler (GCC 13+, Clang 16+, MSVC 2022 17.5+)
• Python ≥ 3.9 and uv for virtual environments and Python dependencies
• Conan 2.x

# Install dependencies via Conan and configure
conan install . --build=missing
cmake --preset gcc-14-release   # see CMakePresets.json for available presets

# Build
cmake --build --preset gcc-14-release

# Run tests
ctest --preset gcc-14-release

The optional Google Benchmark target measures direct datapoint access and runtime dispatcher lookup for early, late, and missing IDs. The generated dispatcher uses a constexpr two-level perfect hash for current and legacy IDs, giving deterministic $O(1)$ runtime lookup without heap allocation or runtime initialization. Build it separately in Release mode:

cmake -S . -B build/benchmark -DCMAKE_BUILD_TYPE=Release -DBUILD_EXAMPLES=OFF -DENABLE_TESTING=OFF -DENABLE_BENCHMARKS=ON
cmake --build build/benchmark --config Release --target datapoints_benchmark
.\build\benchmark\test\benchmark\Release\datapoints_benchmark.exe

See the benchmark documentation for each case and guidance on interpreting results.

cd docker
./build-dev-image.sh   # developer image with all tools pre-installed
./build-ci-image.sh    # CI image

src/
├── include/        ← Header-only C++ framework (ship this + generated header)
├── generator/      ← Python code generator (generator.py, validators, templates)
├── template/       ← Jinja2 templates (datalayer.h, formatter.h, pythonBinding.cpp)
├── model/          ← Example model files (JSON)
├── main.cpp        ← Host demo executable
└── embedded.cpp    ← Bare-metal demo executable

test/
├── catch2/         ← C++ unit tests (Catch2)
└── python/         ← Python unit tests for the generator

docs/               ← Sphinx documentation source
cmake/              ← CMake helpers, toolchains, find modules

Place your model files in any directory and pass it via --model_dir. Five file types are recognised (JSON or YAML):

{
  "Groups": [
    { "name": "DefaultGroup", "persistence": "None",   "baseId": "0x4000", "version": "1.0.1" },
    { "name": "CyclicGroup",  "persistence": "Cyclic", "baseId": "0x5000", "version": "2.0.0" }
  ]
}

{
  "Datapoints": [
    {
      "name": "test", "group": "DefaultGroup", "id": 1,
      "type": "int32_t", "default": 4211,
      "access": "READ_WRITE", "namespace": "Testify", "version": "1.0.1",
      "renamedFrom": [17]
    }
  ]
}

See the Model File Reference for the full specification.

python src/generator/generator.py \
  --model_dir   src/model \
  --out_dir     build/ \
  --template_dir src/template \
  --schema_dir   src/generator \
  --module_name  my_module      # optional pybind11 module name

Validate a model in CI without writing generated files:

python src/generator/generator.py \
  --model_dir src/model --out_dir build --template_dir src/template \
  --schema_dir src/generator --check

For a local browser UI to author canonical JSON models, install the generator requirements and run:

python src/generator/model_editor.py --model-dir src/model

Open http://127.0.0.1:5000. The editor supports the canonical groups.json, datapoints.json, structs.json, enums.json, and types.json layout; use the generator directly for YAML or custom split-file models.

Generated output under build/generated/:

See the SEGGER RTT monitoring guide for the target frame protocol and PyLink usage.

Build the generated extension as a wheel and install it:

uv venv
uv pip install --python .venv\Scripts\python.exe build
Push-Location datalayer_example
uv build
uv pip install --python ..\.venv\Scripts\python.exe --force-reinstall .\dist\datalayer_example-0.0.1-*.whl
Pop-Location

The module exposes generated struct classes plus static get/set methods for each datapoint. With the sample model:

import datalayer_example as dl

dl.Test.set(42)
print(dl.Test.get())

temperature = dl.Temperature(2350, 23.5)
temperature.value = 24.0
dl.Test4.set(temperature)

dl.Arraytest2.set([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
print(dl.Arraytest2.get())

The generated class names follow the datapoint names in the model; use dir(dl) to inspect an extension generated from a custom model. Set -DPYBIND11_MODULE_NAME=my_datalayer when configuring CMake to choose a different import name.

#include <include/datalayer.h>    // generated
#include <include/formatter.h>   // generated (requires ENABLE_FMT)
#include <fmt/format.h>

int main()
{
    // Print group structure
    DefaultGroup.printDatapoints();

    // Read / write a typed datapoint
    std::ignore = Testify::test.set(42);     // returns RangeCheck::ok
    int32_t val = Testify::test.get();       // val == 42

    // Range-checked alias type (min=100, max=200)
    auto rc = TestAlias.set(AliasTypeUint32t{42});
    // rc == DataLayer::Detail::RangeCheck::underflow

    // Struct datapoint
    constexpr Temperature a{ .raw = 1234, .value = 42.2f };
    std::ignore = test4.set(a);

    // Runtime-ID dispatch across all groups
    auto result = Dispatcher.setDatapoint(0x4001, 99);
    // result.success == true, result.check == RangeCheck::ok

#ifdef USE_FILE_PERSISTENCE
    // Persist and restore CyclicGroup
    CyclicGroup.serializeGroup("sample.bin");
    CyclicGroup.deserializeGroup("sample.bin");
#endif
}

Datapoints can notify application-owned code after a successful write. The callback receives the new typed value and the supplied context pointer.

void onTemperature(const Temperature& value, void* context) noexcept;
test4.setChangeCallback(onTemperature, applicationContext);

mqttAdapter.h and canAdapter.h deliberately do not include networking or driver libraries. Implement DataLayer::Mqtt::Client or DataLayer::Can::Bus, then use the typed adapter to publish values and apply validated incoming writes. CAN defaults to an 8-byte payload; instantiate DataLayer::Can::Adapter<decltype(Dispatcher), 64> for CAN FD.

conan create . --build=missing   # build & install into local cache

In your project's conanfile.txt:

[requires]
datalayer-generator/0.0.1

[options]
datalayer-generator/*:with_fmt=True

[generators]
CMakeDeps
CMakeToolchain

In your CMakeLists.txt:

find_package(DataLayerGenerator REQUIRED)

add_executable(my_app main.cpp)
target_link_libraries(my_app PRIVATE DataLayerGenerator::DataLayerGenerator)

generate_datalayer(my_app
    MODEL_DIR ${CMAKE_CURRENT_SOURCE_DIR}/model
    OUT_DIR   ${CMAKE_CURRENT_BINARY_DIR}
)

The generate_datalayer() helper is automatically available after find_package — it wires up the Python generator as a stamp-file-based custom command so the generator only reruns when model files change.

Full documentation — including API reference, model file specification, and integration guide — is published automatically to GitHub Pages on every push to main:

➡ https://StephanKa.github.io/DataLayerGenerator/

To build the docs locally:

uv venv
uv pip install --python .venv/bin/python sphinx sphinx-rtd-theme breathe sphinx-copybutton
mkdir -p build-docs/doxygen
sed \
  -e "s|@DOXYGEN_INPUT_DIR@|$(pwd)/src/include|g" \
  -e "s|@DOXYGEN_OUTPUT_DIR@|$(pwd)/build-docs/doxygen|g" \
  docs/Doxyfile.in > build-docs/Doxyfile
doxygen build-docs/Doxyfile
sphinx-build -b html \
  -Dbreathe_projects.DataLayerGenerator="$(pwd)/build-docs/doxygen/xml" \
  docs/ build-docs/sphinx
# Output: build-docs/sphinx/index.html

This project is released into the public domain under the Unlicense.