Alexandru Maxiniuc

Smarty - Matter on ESP32 with Zephyr - Part 2

2026-01-25T00:00:00+00:00

Disclaimer

I am an engineer who knows his sh*t and prioritizes learning and innovation to getting certifications for tinkering with hardware. To follow along with the hardware portions of this guide, you should either:

Be an engineer who also knows their sh*t.
Accept that you are responsible for your own components and safety.
Or just sit back and enjoy the read.

Reading is still learning, and you are more than welcome to just follow the code without touching a single wire. However, if you do decide to dive into the hardware and things go sideways, whether it’s a fried ESP32, a tripped breaker, or a “spicy” encounter with mains power, I am not to be held accountable. I take no responsibility for your hardware, your home, or yourself. You are the captain of your own (hopefully well-insulated) ship.

In the previous part we built the Hello World Zephyr sample project and analyzed the build environment, dependencies and generated artifacts.

Now I will setup a yanga project to build the same example. The goal is to configure in yanga:

the two external dependencies zephyr and hal_espressif
the toolchain riscv64-zephyr-elf
the platform esp32h2_devkitm
a hello_world component

One shall be able to clone the repository and build it using yanga on both Ubuntu and Windows by running a single command. No extra steps to install toolchains or other dependencies.

PART 2: Hello World on ESP32-H2 with Yanga

I followed the yanga getting started guide to setup my project. This project has two variants, English and German, which greet the user in their language (e.g. “Hello World” or “Hallo Welt”).

I pushed the code to github cuinixam/smarty.

Next steps are to add a new platform for ESP32-H2 and Zephyr.

The new platform esp32h2-zephyr will:

use the riscv64-zephyr-elf toolchain
define two dependencies: zephyr and hal_espressif

To be continued…

Smarty - Matter on ESP32 with Zephyr - Part 1

2026-01-25T00:00:00+00:00

Disclaimer

Be an engineer who also knows their sh*t.
Accept that you are responsible for your own components and safety.
Or just sit back and enjoy the read.

This is the first in a series of articles about upgrading my DIY smart home devices software stack to use Zephyr and Matter.

My existing smart home devices are based on esp32-h2 development boards and use the ESP-IDF framework with the esp-matter stack. I had Zephyr on my todo list for a while now and I finally decided to give it a try.

The plan is to migrate my existing smart home devices to use Zephyr and Matter.

Start with the Hello World Zephyr example and try to understand the build system
Build the same example using yanga build system generator. This will allow me to separate the Zephyr build tools from the Zephyr source code, and to use my own build system generator with the proper abstraction layer for defining software products.
Make the onboard LED a “smart light” Matter device that I can integrate with Home Assistant and Apple HomeKit.

PART 1: Hello World on ESP32-H2

I started by following the Zephyr Getting Started Guide. I followed the Ubuntu instructions because I am using an Ubuntu VM on my Mac using Parallels.

These are basically the commands I ran:

# Create a virtual environment and activate it
python3 -m venv ~/zephyrproject/.venv
source ~/zephyrproject/.venv/bin/activate

# Install west and initialize the zephyr project
pip install west
west init ~/zephyrproject\
cd ~/zephyrproject\
west update

# Export the zephyr project
west zephyr-export

# Install west packages
west packages pip --install
cd ~/zephyrproject/zephyr

# Install Zephyr SDK
west sdk install --help
west sdk install --toolchains riscv64-zephyr-elf

# Fetch the hal_espressif zephyr module
west blobs fetch hal_espressif

# Check available boards
west boards | grep esp32h2

# Build the hello world example for the esp32-h2
west build -b esp32h2_devkitm samples/hello_world

I have an ESP32-H2 devkit board from Waveshare:

I connected it to my computer, answered yes to a dozen questions to confirm that I want to make the USB port available to my VM and then I wanted to flash the hello world example to it.

west flash

Error

This failed because it could not access the serial port.

Serial port /dev/ttyACM0:
/dev/ttyACM0 failed to connect: Could not open /dev/ttyACM0: Permission denied

I had to add my user to the dialout group (sudo usermod -aG dialout $USER) and reboot to make it work. I skipped installing the official espressif udev tools which ensures the system recognizes the ESP32-H2 correctly and assigns the right permissions every time you plug it in.

I now flashed the hello world example to the esp32-h2 and connected to it using the espressif monitor.

west flash
west espressif monitor --port /dev/ttyACM0

It worked! 🤓

*** Booting Zephyr OS build v4.3.0-5066-gf28522e03666 ***
Hello World! esp32h2_devkitm/esp32h2

Tip

If it still doesn’t work, your board might be in the download mode. You can exit it by pressing and holding the BOOT button shortly.

Analyze the build environment and generated artifacts

First, let’s analyze what did we installed:

~/zephyrproject - 9GB:
- modules - 6.7GB: it contains the hardware abstraction layers (HAL) modules, including hal_espressif
- zephyr - 1.4GB: this is the main Zephyr source code (core, drivers, cmake scripts, west scripts, etc.)
- .venv - 800MB: this is the virtual environment used to run west and other Zephyr tools
~/zephyr-sdk-0.17.4 - 2.4GB: this is the Zephyr SDK having the compiler suite for riscv64-zephyr-elf and the host tools

It seems we need around 12GB of disk space to build the hello world example. 🫣 The modules directory has not only the espressif HAL, but also many other HAL modules. Of course, we downloaded more than we needed, but it is still a lot of space.

Let’s us now have a look at the binary file to understand which components are included and the dependencies.

I implemented a python app clanguru to help me analyze binary files and C code in general.

# Install clanguru
pipx install clanguru
clanguru --version
# Generate the objects dependencies graph
clanguru analyze --help
clanguru analyze --compilation-database build/compile_commands.json --output-file obj_deps.html

I linked below the interactive objects dependencies graph. This is a interactive view where you can select which directories shall be visible, search for particular files or click on nodes to highlight their direct dependencies. Check the help button for an overview of all features.

Objects dependencies

/objects_deps_hello_world_zephyr/index.html

We can see that there are two main parts:

zephyr: for the Zephyr kernel, drivers and espressif soc support
hal_espressif: for the espressif hardware abstraction layer including the bootloader support

Next steps

We identified the main parts and the toolchain required to build the hello world example. Next I will setup a yanga project to build the same example. The goal is to configure in yanga:

the two external dependencies zephyr and hal_espressif
the toolchain riscv64-zephyr-elf
the platform esp32h2_devkitm
a hello_world component

One shall be able to clone the repository and build it using yanga on both Ubuntu and Windows by running a single command. No extra steps to install toolchains or other dependencies.

I think this will be fun and I hope I’ll get to it by the end of the week.

Hello Diátaxis

2026-01-25T00:00:00+00:00

As a software engineer, I tend to treat user documentation as something to be written late in the development when I think that someone might use my software. I could improve this klingonian approach, but not today.

Anyway, this weekend I decided to spend 10 minutes and request my apprentice (aka AI Agent) to refactor the documentation I wrote for one of my python application. I already had some documentation, but this was … well, let’s say not the best ;)

So I need to tell my apprentice what I want, but writing prompts takes also time, so I wrote a prompt to write me a prompt.

This was my request:

I have a python application and want to write the documentation
and publish it to readthedocs.
Please create a prompt for an ai agent to parse the code,
existing documentation and generate the documentation.
Look for best practices for writing documentation and include them in the prompt.
Remember that less is more.

In seconds I got this prompt:

The AI Agent Prompt

Role: You are an expert Technical Writer specializing in Python
and the Diátaxis framework.

Objective: Parse the provided codebase and existing notes to generate
a comprehensive documentation suite optimized for Read the Docs (MkDocs or Sphinx).

Guidelines:

- Clarity over Volume: Use the "less is more" principle. Avoid fluff; prioritize "Time to First Hello World."
- Standards: Ensure all docstrings follow PEP 257 (Google or NumPy style).
- Structure (Diátaxis): Organize content into four distinct categories:
  - Tutorials: Learning-oriented (Step-by-step for beginners).
  - How-to Guides: Problem-oriented (Solving specific tasks).
  - Reference: Information-oriented (API technical descriptions).
  - Explanation: Understanding-oriented (Concepts and architecture).

Task Requirements:

- README.md: Create a high-level overview including installation, a 5-line "Quick Start" code block, and links to the full docs.
- API Reference: Automatically generate documentation for all public classes and functions based on code logic and docstrings.
- Requirement File: Ensure requirements.txt or pyproject.toml is correctly reflected in the setup instructions.
- Configuration: Generate a basic .readthedocs.yaml and mkdocs.yml (or conf.py) file to ensure seamless deployment.

Output Format: Provide the documentation in Markdown (for MkDocs) or reStructuredText (for Sphinx).

Like a good master I am ;), I actually checked the prompt my apprentice wrote and immediately stopped at the Diátaxis framework.

At this point I was like, why does my apprentice invent words and frameworks again? :)

In the end, what should have been a 10 minutes task to let my apprentice generate the documentation and then me to review and approve it, took me over one hour to understand what Diátaxis is and why didn’t I know about it before. :)

I had to adapt the prompt slightly for my project setup (Sphinx, Myst Parser, etc.). The Sphinx theme and layout remained untouched but the chapters changed completely. Here is the refactored documentation: pypeline-runner.readthedocs.io.

I think I will use this “framework” for my future documentation projects and of course create an agentic SKILL to do this for me. ;)

Have fun and happy documenting!

SPL Bootstrap - Handle Python Version

2025-11-24T00:00:00+00:00

Introduction

The pipeline implementation for our Software Product Line (SPL) repositories uses the pypeline Python application. In order to start the pypeline application, we need to install Python (with a specified version) and create the Python virtual environment. This step we call bootstrap and it is implemented here.

Note

Currently bootstrap supports only Windows for installing Python.

Workflow

The bootstrap process is managed by two main scripts: a PowerShell script (bootstrap.ps1) for environment setup and a Python script (bootstrap.py) for dependency management. The diagram below shows the high-level workflow.

        flowchart TD
    A[bootstrap.ps1] --> B{Python installed?};
    B -- No --> C[Install Python w/ Scoop 1️⃣];
    B -- Yes --> D;
    C --> D[Execute bootstrap.py];

    D --> E{Dependencies changed?};
    E -- No --> F[End];
    E -- Yes --> G[Create/Update .venv 2️⃣];
    G --> H[Install Dependencies];
    H --> F;

This step will first install the scoop windows package manager and then call scoop to install the configured python version.
The python virtual environment is updated in multiple steps:
- create an empty virtual environment with python venv.create
- install the package manager (e.g., poetry, uv, etc.) and pip-system-certs for using the system SSL certificates
- run the package manager to install the python dependencies

bootstrap.ps1

Scoop configuration and installation
Python version detection and installation
Delegates to bootstrap.py for virtual environment setup

bootstrap.py

OS-agnostic virtual environment creation (Windows/Unix)
Smart caching with hash-based dependency tracking. Only runs if dependencies have been updated.
Package manager support: Poetry, UV, Pipenv
Automatic pip configuration for custom PyPI sources

bootstrap.json Configuration

{
    "python_version": "3.11",
    "python_package_manager": "poetry>=2.1.0",
    "scoop_ignore_scoopfile": true
}

Things to Improve

Python Version Management

Currently the bootstrap process installs a fixed Python version as specified in the bootstrap.json configuration file. When checking if python is installed, it only verifies that the major and minor version match (e.g., 3.11), but does not check for patch versions (e.g., 3.11.4 vs 3.11.5). This can lead to situations that on different machines, slightly different patch versions of Python are installed.

Note

This might not be a big issue in most cases, but it can lead to inconsistencies in our particular case, because we use the python venv module to create the initial virtual environment, which will include the standard library of the installed Python version.

The solution could be:

to find all installed python versions in path and check if the exact version is already installed
it might be that python versions installed with scoop are not available in path, so we need to query scoop for installed versions
if the exact version is not installed, install it with scoop
the bootstrap.py shall fail if the python interpreter version does not match exactly the configured version. This can happen if the user runs the pypeline including the CreateVEnv step which calls directly the bootstrap.py script.

Virtual Environment Management

As mention above, the virtual environment is created in three steps:

create an empty virtual environment with python venv.create
install the package manager (e.g., poetry, uv, etc.) and pip-system-certs for using the system SSL certificates
run the package manager to install the python dependencies

There is a problem with the second step: the package manager and pip-system certs are installed with pip. These packages have their own dependencies, which are not tracked by pip because pip has no support for .lock files.

When step three is executed, the package manager might need to upgrade/downgrade some packages which were implicitly installed in step two and this can cause crashes due to dependency conflicts.

Solution 1: User-defined Initial Packages

One way to solve this is to let the user define the list of package to install in step two in the bootstrap.json configuration file. For example:

{
    "python_version": "3.11.4",
    "venv_initial_packages": [
        "poetry==2.1.0",
        "pip-system-certs==2.2.1",
        "wrapt==1.14.0"
    ],
}

In case there is a conflict between the initial packages and the packages defined in the package manager lock file, the user can adjust the versions in the configuration file accordingly.

Solution 2: Install Package Manager with Scoop

Another way could be to install both python and the package manager with scoop in the bootstrap.ps1 script, so that pip is not involved at all in step two. This means that package manager will install exactly the version specified in the .lock file and therefore always have a consistent set of dependencies.

The problem with this approach is that pip-system-certs is not available as a scoop package and package installation will fail when ran against an on-premise PyPI server with self-signed certificates.

Package manager command

Currently the command to install dependencies with the package manager is hardcoded in the bootstrap.py script with extra arguments being supported via the bootstrap.json configuration file.

{
    "python_package_manager_args": "--clean"
}

We added the feature to support extra arguments because we needed for some projects to pass the --clean to pipenv to remove unused packages already existing in the virtual environment.

A better approach would be to let the user define the full command to install dependencies with the package manager. For example:

{
    "venv_install_command": "poetry install --no-dev --clean"
}

This would give the user full control over how dependencies are installed.

Conclusion

If all these improvements are implemented, the bootstrap process will be more reliable and flexible, ensuring consistent Python environments.

The configuration will allow users to tailor the bootstrap process to their specific needs.

{
    "python_version": "3.11.4",
    "venv_initial_packages": [
        "poetry==2.1.0",
        "pip-system-certs==2.2.1"
    ],
    "venv_install_command": "poetry install --no-dev --clean"
}

This configuration will ensure that:

Python 3.11.4 is installed
The package manager and pip-system-certs are installed with the specified versions
Dependencies are installed with the specified command

Have fun bootstrapping! ✌️

SPL components in YANGA

2025-10-15T00:00:00+00:00

When building software products, you often need different variants for different purposes (like a basic vs. a pro version). These variants consist of components that implement the features required for that particular variant. Each variant might be built for different platforms, different hardware environments with their own constraints and capabilities.

When thinking about components, there are two main categories:

Components which implement your product’s features
Components which provide the functionality for other components to run on a certain hardware platform

Sometimes you might hear these referred to as the application layer and drivers or low-level layer.

It seems natural to have two separate places to configure these components:

One place for components that define what your product does
Another place for components that every product needs when targeting a specific platform

Indeed in YANGA, one can configure variant components and platform components.

Example for a variant configuration:

variants:
- name: Spa
  description: The LED brightness pulsates and cycles through different colors
  components:
  - light_controller           # Light control algorithms
  - power_button               # Power button handling
  # ... and other core features

Example for a platform configuration:

platforms:
  - name: arduino_uno
    components:
      - arduino_core
      - digital_pins
    toolchain_file: arduino.cmake

I happen to find a third use case that doesn’t fit neatly into either of these two categories: components that only some product variants need, and only on specific platforms.

I had a component that it only worked for Arduino, but I used it only for one of the four variants I was building. It didn’t make sense to put it in the platform configuration, because not every product variant needed it. But it also didn’t make sense to put it in the variant configuration, because it was platform specific.

Three Places for Components

YANGA gives you three places to define components:

1. In Your Product Variant

For components that define what your product does, regardless of platform.

variants:
  - name: WeatherStation
    components:
      - sensor_reader
      - data_logger
      - weather_predictor

2. In Your Platform Configuration

For components that every product needs when targeting this platform.

platforms:
  - name: arduino_uno
    components:
      - arduino_core
      - digital_pins
    toolchain_file: arduino.cmake

3. In Your Variant’s Platform-Specific Section

For components that only some products need, and only on specific platforms.

variants:
  - name: WeatherStation
    components:
      - sensor_reader
      - data_logger
    platforms:
      arduino_uno:
        components:
          - hc_sr04_ultrasonic  # Arduino-specific sensor
      raspberry_pi:
        components:
          - camera_module       # RPi-specific camera

Another use case for this third place is when you want to write integration tests. One can create components that combine together multiple components and only define them in the variant-platform section to be used only in that context.

variants:
- name: Disco
  description: The LED blinks and one can change the blinking frequency
  components:
  - spled
  - power_signal_processing
  - light_controller
  features_selection_file: variants/Disco/config.txt
  platforms:
    gtest:
      components: [test_integrations_spled]

Conclusion

Three places to define components might seem like overkill, but they help keep a clear structure in your project configuration. Each place has its purpose, and using them correctly can make your project easier to manage as it grows in complexity.

If you want to learn more about YANGA please check the documentation.

✌️

SPLED meets Arduino Uno

2025-10-03T00:00:00+00:00

For our Software Product Line Engineering (SPLE) training we use the SPLED repository, which is a simple software product line for controlling an LED with different variants:

Disco: the LED blinks and one can change the blinking frequency
Sleep: the LED has a constant color and one can change the brightness
Spa: the LED brightness pulsates and cycles through different colors

The current implementation creates an executable which can be run in the terminal and uses ANSI escape codes to change the color of the text.

What I want to do is to port this example to an Arduino Uno, so that we can control an RGB LED with it.

I will not use the CMake build environment based on the spl-core CMake modules, but instead use YANGA which is a software product line build environment written in Python. YANGA uses .yaml configuration files and generates the CMake files for the actual build.

Although YANGA is platform independent, SPLED tools dependencies are only defined as scoop packages for Windows. I am using a Windows 11 machine for this hacking session.

Get the code

I forked SPLED and created the YANGA configuration files to build the different variants as with the original spl-core environment.

Clone the repository and setup the virtual environment to be able to run yanga:

git clone --branch arduino_platform --single-branch https://github.com/cuinixam/SPLed C:\temp\spled
cd C:\temp\spled
.\build.ps1 -install

We can check the SPL configuration with:

.\yanga.ps1 run --print

This should print the variants, components and supported platforms:

PS C:\temp\spled> .\yanga.ps1 run --print
START    | Starting run
INFO     | -----------------------------------------
INFO     | Project directory: C:\temp\spled
INFO     | Parsed 6 configuration file(s).
INFO     | Found 20 component(s).
INFO     | Found 5 variant(s):
INFO     |   - BlinkLed
INFO     |   - Base/Dev
INFO     |   - Disco
INFO     |   - Sleep
INFO     |   - Spa
INFO     | Found 4 platforms(s):
INFO     |   - win_exe
INFO     |   - gtest
INFO     |   - arduino_nano
INFO     |   - arduino_uno_r3
INFO     | Found pipeline config:
INFO     |     Group: install
INFO     |         CreateVEnv
INFO     |         ScoopInstall
INFO     |         WestInstall
INFO     |         GenerateEnvSetupScript
INFO     |     Group: gen
INFO     |         KConfigGen
INFO     |     Group: build
INFO     |         GenerateBuildSystemFiles
INFO     |         ExecuteBuild
INFO     | -----------------------------------------
STOP     | Finished run in 0.05s
PS C:\temp\spled>

Our focus will be to build the Spa variant for the arduino_uno_r3 platform.

Build the Spa variant for Arduino Uno R3

In order to build the Spa variant for the Arduino platform we have to make sure that the Windows executable (win_exe platform) specific components are not included when building for Arduino. YANGA supports variant and platform specific components, so we can define that some components are only included for the win_exe platform.

- name: Spa
  description: The LED brightness pulsates and can cycle through different colors
  components:
  - rte
  - spled
  - power_signal_processing
  - light_controller
  - power_button
  - main_control_knob
  - brightness_controller
  features_selection_file: variants/Spa/config.txt
  platforms:
    win_exe:
      components:
      # Windows main
      - main
      # Simulates a period task using `usleep`
      - os
      # Prints to console using ANSI escape codes
      - console_interface
      # Reads key presses to detect `power on/off` and `up/down`
      - keyboard_interface

Note

The variant platform specific components should be used to defined components which are relevant for this specific variant when built for the given platform. main and os components are needed actually by all variants when built for win_exe, so placing them in the variant platform specific section is not ideal.

For this, YANGA supports platform specific components at the platform level, which would be a better place to define these components.

To be continued …

What you did not want to know about your code

2025-09-26T00:00:00+00:00

As any other engineer, I also like to hack around in the weekends at six o’clock in the morning when the family is still asleep. This time I wanted to play around with the new Matter smart home connectivity standard and add some temperature and humidity sensors to my smart home setup.

I bought some esp32-h2 development boards (they support WiFi, Thread and BLE), some SHT31-D sensors and started the old fashioned read the docs to get started. I must say the espressif docs are well written and the community is very active.

I am using a WSL Ubuntu on Windows 11 and getting the build environment to work was not as straightforward as I hoped:

the idf.py build system comes in a separate repository and needs to be set up to work
the esp-matter repo is huge and takes a while to clone
getting the esp32 device visible as /dev/ttyUSB0 was tricky

After a while I got the first example compiled and flashed to the device.

What I have noticed though was that the resulting binary was quite large (over 1.5MB) and I wondered what is actually in there.

Instead of just looking at the final binary and the object files, I thought it would be more interesting to see it as a dependency graph between the object files. For this I added a new command in a Python app I use to parse and analyze C/C++ sources, called clanguru.

The new analyze command does the following:

parse the compile_commands.json file to find all the object files
run nm on each object file to get the global symbols and determine their type:
- extern for undefined globals - these are the required symbols
- local for local globals - these are the provided symbols
create an html report using cytoscape.js to visualize the dependencies between the object files. I also added an option to group the object files in named containers based on their path.

The result is a nice interactive graph where you can zoom in and out, drag the nodes around and enjoy the complexity of the code base.

For my project it looks like this:

You can find the actual report here.

Use clanguru to analyze your C/C++ code

Install clanguru using pipx for isolated installation:

pipx install clanguru

Check the help for the analyze command:

clanguru analyze --help

You need to build your project and provide the compilation database (compile_commands.json). Also, ensure that the nm command-line tool is available in your user PATH.

To generate an html dynamic dependency report, run the following command:

clanguru analyze --compilation-database compile_commands.json --output-file dependencies.html --use-parent-deps

The --use-parent-deps option will group the object files based on their parent directory.

If the output file extension is .xlsx, an Excel report will be generated instead of an HTML one.

If you have any questions or feedback, feel free to create an issue in the clanguru GitHub repository.

Have fun exploring your code dependencies! 😎

Why unit tests crashed on windows but worked on the microcontroller

2025-09-24T00:00:00+00:00

When writing cross-platform code, it’s easy to forget that pointers don’t always have the same size. This caused a segmentation fault in one of our unit tests.

What happened

On the Aurix MCU, pointers are 32-bit. In the code a pointer (memory address) was stored in a uint32_t variable. That worked fine on the target.

But, when running the unit tests on the PC (Windows with GCC), pointers are 64-bit. Storing a 64-bit pointer into a 32-bit integer truncated the address. The compiler only issued a warning and still compiled.

Later, when the truncated value was cast back to a pointer and dereferenced, it pointed to invalid memory and crashed with a segmentation fault.

How to fix it and make the code portable

🚫 Never store a pointer in a fixed-width integer type (uint32_t, uint64_t, …).
✅ Always use uintptr_t from <stdint.h> because it is guaranteed to be wide enough for a pointer on the current platform.

Wrong vs Right

#include <stdint.h>
#include <inttypes.h>
#include <stdio.h>

int main(void) {
    int value = 42;
    int *ptr = &value;

    // ❌ Wrong: pointer stored in fixed 32-bit type
    uint32_t wrong = (uint32_t)ptr;   // Works on 32-bit MCU, truncates on 64-bit host
    int *bad = (int *)wrong;          // may segfault
    printf("wrong addr = 0x%08" PRIx32 "\n", wrong);

    // ✅ Right: use uintptr_t (pointer-sized integer type)
    uintptr_t right = (uintptr_t)ptr;
    int *good = (int *)right;
    printf("right addr = 0x%" PRIxPTR ", *good = %d\n", right, *good);

    return 0;
}

Both PRIx32 and PRIxPTR come from <inttypes.h>. They expand to the correct printf format string for the given integer type:

PRIx32 - prints a 32-bit integer in hex.
PRIxPTR - prints a uintptr_t (pointer-sized integer) in hex.

This ensures portable and correct logging across platforms.

Final thought

Using uintptr_t instead of uint32_t to store pointers makes the code portable when running unit tests on a 64-bit host.

Coding Dojo - Detect CI Context

2025-03-08T00:00:00+00:00

Introduction

Coding Dojos are a great way to practice and enhance your skills in Test-Driven Development (TDD), software craftsmanship, and incremental software design. In this blog post, we’ll explore how to implement the Detect CI Context coding example using Python.

You’ll learn how to:

Clearly define APIs and data structures.
Translate the requirements to tests.
Implement the solution incrementally using TDD.
Refactor code for clarity, maintainability, and scalability.

Understand the problem and define the API

The problem is to detect whether a program runs on Jenkins or locally. The inputs are the specific environment variables that Jenkins sets when running a build.

The expected outputs are:

The CI system name (e.g. Jenkins or Unknown).
Is the build trigger a Pull Request.
Name of the target branch (branch to merge into).
Name of the current branch being built or tested.

Using our module should be as straightforward as calling a method that returns the required data.

In Python, our method signature could look like this:

def detect_ci_context() -> CIContext:
    pass

We can use a data class to represent the CI context:

@dataclass
class CIContext:
    #: CI system where the build is running
    name: str
    #: Whether the build is for a pull request
    is_pull_request: bool
    #: The branch to merge into
    target_branch: Optional[str]
    #: Branch being built
    current_branch: Optional[str]

Implement the Solution Incrementally

The way to determine the CI context for Jenkins can be summarized as follows:

Presence of JENKINS_HOME indicates Jenkins.
If running a pull request (CHANGE_ID), CHANGE_TARGET is the target branch and CHANGE_BRANCH is the current branch.
If not a pull request, BRANCH_NAME is the current branch.

First requirement - Detect Jenkins

Let’s start by writing a test that checks if the CI system is Jenkins.

Our test should:

Set the JENKINS_HOME environment variable to a non-empty value.
Call the detect_ci_context method.
Check if the name is JENKINS.

The first issue we encounter is to be able to manipulate the environment variables.

In Python, we can mock the os.getenv function using the unittest.mock module:

@pytest.fixture
def mock_on_getenv():
    with patch("os.getenv") as mock_os_getenv:
        yield mock_os_getenv

What this fixture does is to replace the os.getenv function with a mock object that we can control.

Now we can write the test:

def test_jenkins_branch_push(mock_on_getenv: MagicMock) -> None:
    # Setup
    mock_on_getenv.side_effect = lambda var, default=None: {
        "JENKINS_HOME": "/jenkins/home"
    }.get(var, default)
    # Run
    ci_context = detect_ci_context()
    # Check
    assert ci_context.name == "JENKINS"

We override the os.getenv function with a lambda that returns the value of the JENKINS_HOME environment variable. It has the same signature as the original function, taking two arguments: the variable name and a default value.

To fix the test, we need to implement the detect_ci_context method:

def detect_ci_context() -> CIContext:
    return CIContext(
        name="JENKINS",
        is_pull_request=False,
        target_branch=None,
        current_branch=None,
    )

Note

You might be surprised that we hardcoded the name to JENKINS. This is one idea behind TDD: write the simplest code that makes the test pass.

Second requirement - Detect Unknown

Let’s write a test that checks if the CI system is unknown:

def test_unknown_ci_system(mock_on_getenv: MagicMock) -> None:
    # Setup
    mock_on_getenv.side_effect = lambda var, default=None: {}
    # Run
    ci_context = detect_ci_context()
    # Check
    assert ci_context.name == "Unknown"

To fix the test and do not break the previous one, we can’t just return a constant value but need to check the environment variables:

def detect_ci_context() -> CIContext:
    return CIContext(
        name="JENKINS" if os.getenv("JENKINS_HOME", None) else "Unknown",
        is_pull_request=False,
        target_branch=None,
        current_branch=None,
    )

Implement the rest of the requirements

We can continue implementing the rest of the requirements in the same way, writing tests and fixing them by implementing the detect_ci_context method.

def detect_ci_context() -> CIContext:
    ci_name = "UNKNOWN"
    is_pull_request = False
    target_branch = None
    current_branch = None
    if os.getenv("JENKINS_HOME", None) is not None:
        ci_name = "JENKINS"
        is_pull_request = os.getenv("CHANGE_ID", None) is not None
        if is_pull_request:
            target_branch = os.getenv("CHANGE_TARGET")
            current_branch = os.getenv("CHANGE_BRANCH")
        else:
            target_branch = os.getenv("BRANCH_NAME")
            current_branch = target_branch
    return CIContext(
        name=ci_name,
        is_pull_request=is_pull_request,
        target_branch=target_branch,
        current_branch=current_branch,
    )

Refactor

First thing that looks odd is the CIContext.name variable type. Strings are too generic and can lead to errors. Imagine a typo in the string or different capitalization, JENKINS vs Jenkins.

We can use an enumeration to represent the CI system:

class CISystem(Enum):
    JENKINS = auto()
    UNKNOWN = auto()

and update the CIContext class:

@dataclass
class CIContext:
    #: CI system where the build is running
    ci_system: CISystem
    #: Whether the build is for a pull request
    is_pull_request: bool
    #: The branch to merge into
    target_branch: Optional[str]
    #: Branch being built
    current_branch: Optional[str]

    @property
    def name(self) -> str:
        return self.ci_system.name.upper()

Note

This refactoring caused no tests to fail nor did it change the API.

New Feature Request

There is a new requirement to detect GitHub Actions.

The way to determine the CI context for GitHub Actions can be summarized as follows:

GITHUB_ACTIONS indicates GitHub Actions.
If running a pull request (GITHUB_EVENT_NAME is pull_request), GITHUB_BASE_REF is the target branch and GITHUB_HEAD_REF is the current branch.
If not a pull request, GITHUB_REF_NAME is the current branch.

First requirement - Detect Github Actions

We can write a test for this new requirement:

def test_github_actions_pull_request(mock_on_getenv: MagicMock) -> None:
    # Setup
    mock_on_getenv.side_effect = lambda var, default=None: {
        "GITHUB_ACTIONS": "true",
    }.get(var, default)
    # Run
    ci_context = detect_ci_context()
    # Check
    assert ci_context.name == "GITHUB_ACTIONS"

We can implement the new feature by extending the detect_ci_context method:

def detect_ci_context() -> CIContext:
    ci_system = CISystem.UNKNOWN
    is_pull_request = False
    target_branch = None
    current_branch = None
    if os.getenv("JENKINS_HOME", None) is not None:
        ci_system = CISystem.JENKINS
        is_pull_request = os.getenv("CHANGE_ID", None) is not None
        if is_pull_request:
            target_branch = os.getenv("CHANGE_TARGET")
            current_branch = os.getenv("CHANGE_BRANCH")
        else:
            target_branch = os.getenv("BRANCH_NAME")
            current_branch = target_branch
    elif os.getenv("GITHUB_ACTIONS", None) is not None:
        ci_system = CISystem.GITHUB_ACTIONS
    return CIContext(
        ci_system=cis_system,
        is_pull_request=is_pull_request,
        target_branch=target_branch,
        current_branch=current_branch,
    )

Implement the rest of the requirements

We can continue implementing the rest of the requirements in the same way, writing tests and fixing them by implementing the detect_ci_context method.

def detect_ci_context() -> CIContext:
    ci_system = CISystem.UNKNOWN
    is_pull_request = False
    target_branch = None
    current_branch = None
    if os.getenv("JENKINS_HOME", None) is not None:
        ci_system = CISystem.JENKINS
        is_pull_request = os.getenv("CHANGE_ID", None) is not None
        if is_pull_request:
            target_branch = os.getenv("CHANGE_TARGET")
            current_branch = os.getenv("CHANGE_BRANCH")
        else:
            target_branch = os.getenv("BRANCH_NAME")
            current_branch = target_branch
    elif os.getenv("GITHUB_ACTIONS", None) is not None:
        ci_system = CISystem.GITHUB_ACTIONS
        is_pull_request = os.getenv("GITHUB_EVENT_NAME", None) == "pull_request"
        if is_pull_request:
            target_branch = os.getenv("GITHUB_BASE_REF")
            current_branch = os.getenv("GITHUB_HEAD_REF")
        else:
            current_branch = os.getenv("GITHUB_REF_NAME")
            target_branch = current_branch
    return CIContext(
        ci_system=ci_system,
        is_pull_request=is_pull_request,
        target_branch=target_branch,
        current_branch=current_branch,
    )

Refactor

Is obvious that the detect_ci_context code smells. It does too many things and is not extensible. Every time we add a new CI system, we need to modify this method. This is a violation of the Open/Closed Principle.

This translates to a new non-functional requirement: Support more CI systems without modifying the detect_ci_context method.

To do this, “abstraction is the key”. We can create a new class that will be responsible for detecting the CI system.

class CIDetector(ABC):
    @abstractmethod
    def detect(self) -> Optional[CIContext]:
        pass

We can now create a detector for every CI system and the detect_ci_context method will only iterate over the detectors and return the first result.

def detect_ci_context() -> CIContext:
    detectors = [
        JenkinsDetector(),
        GitHubActionsDetector(),
    ]
    for detector in detectors:
        ci_context = detector.detect()
        if ci_context is not None:
            return ci_context
    return CIContext(
        ci_system=CISystem.UNKNOWN,
        is_pull_request=False,
        target_branch=None,
        current_branch=None,
    )

Hmm, we still have to modify the detect_ci_context method every time we add a new CI system detector 😧 and the new value in the CISystem enumeration.

One solution could be to link the CISystem enumeration with the CIDetector class.

class CISystem(Enum):
    UNKNOWN = (auto(), None)  # Special case for unknown
    JENKINS = (auto(), JenkinsDetector)
    # Add new CI systems here:  MY_CI = (auto(), MyCIDetector)

    def __init__(self, _: Any, detector_class: Optional[Type[CIDetector]]):
        self._value_ = _  # Use auto() value, but ignore it in __init__
        self.detector_class = detector_class

    def get_detector(self) -> Optional[CIDetector]:
        return self.detector_class() if self.detector_class else None

Some of you might be surprised to see that one can add extra arguments to an enumeration member.

What we’ve done is to add a detector_class attribute to each enumeration member to link it with the detector class. This way, we can create a detector for each CI system and the detect_ci_context method will only iterate over the detectors and return the first result.

def detect_ci_context() -> CIContext:
    ci_context: Optional[CIContext] = None
    for ci_system in CISystem:
        detector = ci_system.get_detector()
        if detector:
            ci_context = detector.detect()
            if ci_context:
                break  # Stop at the first detected CI
    # If no CI system was detected, return unknown CIContext
    else:
        ci_context = CIContext(
            ci_system=CISystem.UNKNOWN,
            is_pull_request=False,
            target_branch=None,
            current_branch=None,
        )
    return ci_context

Now, to add a new CI system, we only need to create the new detector class and add the enumeration member. 😎

Conclusion

In this blog post, we explored how to implement the Detect CI Context coding example using Python.

We’ve learned how to:

incrementally implement the solution using TDD, refactoring, and design patterns.
learned how to use the Unittest patch method to manipulate the environment variables.
learned how to add extra arguments to an enumeration.

I hope you enjoyed this coding dojo and learned something new.

Happy coding! ⌨️

References

Making Unit Tests Visible in Visual Studio Code with CTest and GoogleTest

2025-02-05T00:00:00+00:00

Introduction

Many C/C++ projects rely on CMake, GoogleTest (GTest), and GoogleMock (GMock) for unit testing. Sometimes developers want to see and run/debug these tests directly in their IDE.

In this blog post, I will show how to make the unit tests visible in Visual Studio Code (VS Code) using CTest and GoogleTest. I will start by explaining the standard way to build and run the tests with CMake. Then we will see how CTest can help us discover the tests and how to display them in the VS Code interface.

Building and Running Tests with CMake and GoogleTest

In a typical setup one defines executables for each “component” with its own sources and unit tests files. There are CMake custom targets for every component to link the component sources and the test sources and execute the tests.

An instructive example of a CMakeLists.txt file is shown below:

cmake_minimum_required(VERSION 3.20)
project(MyProject)

set(CMAKE_CXX_STANDARD 14)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(gtest_force_shared_crt ON CACHE BOOL "" FORCE)

# Add GoogleTest components
add_subdirectory(${CMAKE_SOURCE_DIR}/build/external/gtest ${CMAKE_BUILD_DIR}/.gtest)

# Configure directories to search for headers
include_directories(
    ${CMAKE_SOURCE_DIR}
    ${CMAKE_SOURCE_DIR}/build/external/gtest/googletest/include
    ${CMAKE_SOURCE_DIR}/build/external/gtest/googlemock/include
)

# Create the test executable
add_executable(greeter greeter.c greeter_test.cc)

target_link_libraries(greeter GTest::gtest_main GTest::gmock_main pthread)
target_compile_options(greeter PRIVATE -ggdb --coverage)
target_link_options(greeter PRIVATE --coverage)

# Run the test executable, generate JUnit report and return success independent of the test result
add_custom_target(greeter_test ALL
    BYPRODUCTS ${CMAKE_BUILD_DIR}/greeter_junit.xml
    DEPENDS greeter
    COMMAND greeter --gtest_output="xml:greeter_junit.xml" || ${CMAKE_COMMAND} -E true
)

What this basically does is:

create a test executable greeter that links the sources greeter.c and greeter_test.cc with the GoogleTest libraries
create a custom target greeter_test that runs the test executable and generates a JUnit report

To build and run the tests, one would typically execute the following commands:

cmake -B build -S .
cmake --build build

This will build all targets and generate the test executable greeter and the JUnit report greeter_junit.xml.

greeter_junit.xml

<?xml version="1.0" encoding="UTF-8"?>
<testsuites tests="2" failures="1" disabled="0" errors="0" time="0." name="AllTests">
  <testsuite name="GreeterTest" tests="2" failures="1" disabled="0" skipped="0" errors="0" time="0." >
    <testcase name="Greeting1" file="greeter_test.cc" line="14" status="run" result="completed" time="0." classname="GreeterTest" />
  </testsuite>
</testsuites>

Discovering Tests

Let us first shortly discuss on how one would discover the tests in the project.

Having the tests source files, one could try to parse them and extract the test cases. However, this is not a trivial task and it is error-prone. Imagine that a test is only enabled if a certain preprocessor definition is set. One should properly parse the (preprocessed) test source files to determine the list of relevant tests. I am not even touching on how creative some developers might be on creating their on macros to define tests.

Another approach is to “ask” the test executable to list the tests. This has the disadvantage that the test executable must be built and run. On the good side, the test executable knows exactly which tests are available and can provide additional information about them.

Indeed, the GTTest framework provides a way to list the tests. Running the test executable with the --gtest_list_tests flag will list all the tests and their test cases.

greeter.exe --gtest_list_tests

Note

Calling the test executable with the --help option will print all the available options.

We will go with the second approach and determine the tests by running the test executable. CMake has a built-in tool called CTest that can help us with this.

Defining and running tests with CTest

CTest is a testing tool that is part of the CMake suite. CMake facilitates testing through special commands and the CTest executable.

The integration is quite simple. One needs to “enable testing” and define tests using the add_test command:

include(CTest) # automatically calls enable_testing()
add_test(NAME greeter_test COMMAND greeter)

The add_test command has a simple syntax:

NAME - the name of the test, can be any string
COMMAND - the command to run the test, in our case the test executable greeter. This can be any command that is run in the shell and returns a status code (non-zero for failure).

To run the tests, one would execute the following command:

cd build
ctest

CTest has many options to control the test execution, like running only a specific test, running tests in parallel, or generating a JUnit report. For more information check the help or the documentation.

Details on how CMake/CTest work

When testing is enabled, CMake generates a CTestTestfile.cmake file in the build directory. This file will include all the tests defined with the add_test command. When running ctest, CTest will read this file and execute the tests.

# CMake generated Testfile
add_test([=[greeter_test]=] "C:/project/build/greeter.exe")
set_tests_properties([=[greeter_test]=] PROPERTIES  _BACKTRACE_TRIPLES "C:/project/CMakeLists.txt;34;add_test;C:/project/CMakeLists.txt;0;")
subdirs("/.gtest")

Important

This file is generated during CMake configure and does not require the test executable to be built. It is only used by CTest to run the tests, like having a target to run the test executable.

Discovering tests with GoogleTest CTest integration

CMake provides a helpful module GoogleTest which simplifies test discovery and registration when using GTest:

include(CTest)
include(GoogleTest)

# Automatically discover all GoogleTest test cases in the greeter binary
gtest_discover_tests(greeter
    # Optional: specify a WORKING_DIRECTORY or other properties
)

This approach removes the need to manually call add_test(NAME <some_name> COMMAND <test_executable>) for each test. Instead, gtest_discover_tests takes care of enumerating them by parsing the output of the actual test binary.

Details on how GoogleTest CTest integration works

When using gtest_discover_tests, CMake generates a CTestTestfile.cmake file that includes the tests discovered by GoogleTest.

After CMake configure (no tests executable are yet built nor executed) there is a CTestTestfile.cmake file that includes the tests discovered by GoogleTest. There is a <component>_include.cmake file generated for every component which is just defines a dummy test.

greeeter[1]_include.cmake

if(EXISTS "C:/build/greeter[1]_tests.cmake")
  include("C:/build/greeter[1]_tests.cmake")
else()
  add_test(greeter_NOT_BUILT greeter_NOT_BUILT)
endif()

Executing ctest will fail because the command greeter_NOT_BUILT does not exist.

Important

CMake will add a custom command (see build.ninja build file) to generate the greeter_tests.cmake file. This command will run the test executable and parse the output to generate the test cases.

After building the test executable, the greeeter[1]_tests.cmake file is generated and this contains all tests discovered by GoogleTest.

greeeter[1]_tests.cmake

add_test([=[GreeterTest.Greeting1]=]  C:/project/build/greeter.exe [==[--gtest_filter=GreeterTest.Greeting1]==] --gtest_also_run_disabled_tests)
set_tests_properties([=[GreeterTest.Greeting1]=]  PROPERTIES WORKING_DIRECTORY C:/project/build SKIP_REGULAR_EXPRESSION [==[\[  SKIPPED \]]==])
add_test([=[GreeterTest.Greeting2]=]  C:/project/build/greeter.exe [==[--gtest_filter=GreeterTest.Greeting2]==] --gtest_also_run_disabled_tests)
set_tests_properties([=[GreeterTest.Greeting2]=]  PROPERTIES WORKING_DIRECTORY C:/project/build SKIP_REGULAR_EXPRESSION [==[\[  SKIPPED \]]==])
set(  greeter_TESTS GreeterTest.Greeting1 GreeterTest.Greeting2)

Executing ctest will now run the tests and generate the JUnit report.

VS Code integration

The CMake Tools extension for VS Code supports since version 1.14 a Test Explorer for CTest. The extension will automatically detect the CTest tests from the generated CTest files and display them in the Test Explorer.

This means that one can run, debug, and see the test results directly in the VS Code interface.

Of course, this approach inherits all the “limitations” of discovering the tests with CTest. After CMake configure, one will see only “_NOT_BUILT” tests in the Test Explorer.

Only after building the test executable, the tests will be visible.

Important

When executing the tests from the Test Explorer, before running the tests, the CMake Tools extension will start the current select CMake build target. This might be confusing if one switches between build targets and suddenly when executing a test, a target is build which has nothing to do with the test.

I think their idea is to ensure that the test executable is built before running the tests, to update the test cases.

When executing single tests, CTest is called with the --tests-regex option to run only the selected tests. The problem is that when a test fails there is no way to go from the Test Explorer to the failing test. This is a general issue with CTest that there is no automatic traceability between the test name and the test source file and line. There is another VS Code extension CMake Test Explorer which attempts to address this issue by using the CTest properties feature. I did not test this extension. Maybe in a future blog post.

Conclusion

Integrating the unit tests in the VS Code interface is a nice feature to have. CTest and GoogleTest provide a good way to discover and run the tests. While not perfect, the integration with the CMake Tools extension for VS Code is a good start. One can run, debug, and see the test results directly in the VS Code interface.

References

Allocate Code and Data to Specific Memory Sections

2025-01-07T00:00:00+00:00

Introduction

There was a question in our support channel about the memory sections from an external library. The colleague was trying to understand how the different sections were defined for the library and how they were placed in memory. When checking the linker configuration file (*.lsl) the observation was that the memory sections for:

all the objects for our code, the section pattern was .<type>.<filename>.<symbol>. For example, select .text.component.main or select .data.component.my_var.
the library, the section pattern was .<type>.<some_name>.<symbol. For example, .rodata.adjustable_parameters.my_param.

The question was: how can we define the sections for the library in the linker script file and if they can use the same pattern as our code?

Understanding the sections

Why do we need to define different memory sections?

When integrating code with different safety levels or from external libraries, it is important to separate the memory sections to restrict the access to certain areas of the memory only to specific code. This safety measure is important to avoid that a bug in one part of the code can corrupt the data of another part of the code.

How are the sections defined?

The SW architecture describes the logical allocation of the components based on different abstraction levels and safety requirements. Based on this and the memory map of the micro-controller, the sections are defined in the linker script file.

Here are some section types that are commonly used:

.text - the code section
.data - the initialized data section
.bss - the uninitialized data section
.rodata - the read-only data section

How does my C code ends up in the sections?

By default, the compiler generates the sections based on the file name and the symbol name. Let us consider the following code:

int my_var = 1;
void my_func(void) {
    my_var++;
}

The compiler will generate the following sections:

.text.my_file.my_func - the code of the function my_func
.data.my_file.my_var - the initialized data of the variable my_var

In the linker script file, we can select the sections to be placed in the memory map.

In the memory group where this component code shall be placed, we can refer to it as:

select .text.my_file.*

This means that all the sections that start with .text.my_file. will be placed in this memory group.

Renaming the sections

Let’s say that one has some adjustable parameters which shall be allocated to a specific memory area.

const parameters_t my_parameters = {
    .param1 = 1,
    .param2 = 2,
};

By default the compiler will generate a section named .rodata.my_file.my_parameters for the read-only data of the variable my_parameters.

If one wants to give a specific name to this section, one can instruct the compiler to use a specific section name. Depending on the compiler, there are different ways to do this.

When using the TASKING compiler, there is a special ´#pragma´ to define the section name.

#pragma section all "sw_constants" // define the section name
const parameters_t my_parameters = {
    .param1 = 1,
    .param2 = 2,
};
#pragma section restore all // restore the default section name

For more information see the TASKING User Guide, section 1.12.1 Rename Sections in the References.

When using the GCC compiler, there is a __attribute__ that can be used to define the section name.

const parameters_t my_parameters __attribute__((section("sw_constants"))) = {
    .param1 = 1,
    .param2 = 2,
};

See more about the __attribute__ in the GCC documentation.

Answering the question(s)

The linker configuration file (*.lsl) is used to define the memory layout, collect and group memory sections from objects and libraries. The compiler generates the sections based on the file name and the symbol name and type. The user can rename the sections using compiler specific directives.

The memory sections defined for the objects inside the library have a different pattern because there were renamed using the compiler specific directives (pragma or __attribute__ section). One can not change the section names after the compilation and therefore must rely on naming conventions to know which sections are present in the library.

References

Homelab - setup main machine

2024-11-17T00:00:00+00:00

Introduction

When starting my homelab journey, I needed a main machine to manage all devices and services in my network. Many homelab enthusiasts typically choose Linux for this purpose. However, my primary computer runs Windows 11, so I decided to use WSL2 with an Ubuntu distribution to fulfill this role.

Installing WSL2

I followed the instructions provided in the official documentation to install WSL2 and had Ubuntu up and running quickly.

Network configuration

By default, WSL2 uses NAT networking mode, assigning dynamic IP addresses, which prevents access to other devices in the local network, like routers and Raspberry Pis. To solve this, I switched WSL2 to mirrored network mode via the WSL Settings. After shutting down (wsl --shutdown) and restarting WSL2 (wsl), I could successfully ping other devices in my network:

ping router.local
ping rpi3.local

Using VS Code Remote - WSL

I found the easiest way to interact with files inside WSL2 was to use the VS Code Remote - WSL extension. Installing this extension allowed me to seamlessly edit files inside the Ubuntu distribution directly through VS Code’s interface.

Installing Ansible

To automate the configuration of remote devices, I chose Ansible. Following the instructions from Ansible’s installation page, I created a script called bootstrap_main_machine.sh to install both pipx and ansible:

# Install pipx - see https://pipx.pypa.io/stable/
sudo apt update
sudo apt install pipx
pipx ensurepath

# Install ansible - see https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html
pipx install --include-deps ansible
ansible --version

Creating the Ansible inventory

I created an Ansible inventory file named inventory.yml within a new directory ~/ansible. It lists the devices I want to manage:

all:
  hosts:
    rpi3test.local:
      ansible_user: pi

I initially tested my Ansible setup using the ping module:

ansible -i ~/ansible/inventory.yml rpi3test.local -m ping

This attempt failed, showing the following SSH error:

rpi3test.local | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: pi@rpi3test.local: Permission denied (publickey,password).",
    "unreachable": true
}

Since I wasn’t using SSH keys yet, I tried providing the password explicitly with the --ask-pass option:

ansible -i ~/ansible/inventory.yml rpi3test.local -m ping --ask-pass

However, this still didn’t work due to a missing package:

rpi3test.local | FAILED! => {
    "msg": "to use the 'ssh' connection type with passwords or pkcs11_provider, you must install the sshpass program"
}

To resolve this, I installed the required package:

sudo apt install sshpass

After installing sshpass, running the ping command with --ask-pass succeeded:

rpi3test.local | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3.11"
    },
    "changed": false,
    "ping": "pong"
}

Important

It is not recommended to authenticate using passwords because this opens the possibility for brute force attacks. The best practice is to use SSH keys for authentication. We will cover this in the next section.

Creating SSH keys

To enhance security, I generated SSH keys specifically for Ansible connections using the ED25519 algorithm:

ssh-keygen -t ed25519 -f ~/.ssh/ansible

Next, I wrote an Ansible playbook to automate the process of copying SSH keys to the devices and disabling password authentication:

---
- name: Setup authentication
  hosts: all
  become: yes
  tasks:
    - name: Create .ssh directory if it doesn't exist
      file:
        path: /home/{{ ansible_user }}/.ssh
        state: directory
        mode: "0700"
        owner: "{{ ansible_user }}"
        group: "{{ ansible_user }}"

    - name: Copy public key from local .ssh directory
      copy:
        src: "~/.ssh/ansible.pub"
        dest: /home/{{ ansible_user }}/.ssh/authorized_keys
        mode: "0600"
        owner: "{{ ansible_user }}"
        group: "{{ ansible_user }}"

    - name: Ensure password authentication is disabled in SSHD
      lineinfile:
        path: /etc/ssh/sshd_config
        regexp: "^PasswordAuthentication"
        line: "PasswordAuthentication no"
        state: present
      notify:
        - Restart SSHD

    - name: Lock password for {{ ansible_user }}
      user:
        name: "{{ ansible_user }}"
        password_lock: yes

  handlers:
    - name: Restart SSHD
      service:
        name: sshd
        state: restarted

Optimizing Tool Installation in SPL Build Environments

2024-10-26T00:00:00+00:00

I assume that the reader is familiar with the concept of software product lines (SPLs) and build environments. If not, as this is a blog post about optimizing tool installation in SPL build environments, I recommend you read up on these topics first.

Introduction

In software product line (SPL) engineering, efficiently managing tool dependencies is essential for optimizing build environments and enhancing developer productivity. This blog post explores the thought process in addressing the challenge of optimizing tool installations in an SPL. We’ll walk through the problem statement, initial ideas, the evolution of those ideas, and the final approach.

The Problem: Inefficient Tool Installation

In our SPL build environments, all tools are being installed regardless of the specific needs of a developer or build target. This was not noticeable initially, but as the number of variants and use cases (e.g. different packaging formats, on target debugging, etc.) increased, the inefficiency became apparent.

Basically, there is an install step at the beginning of the build pipeline that installs all the tools required by all variants and all possible build targets.

Some use cases where this inefficiency is particularly noticeable include:

A developer working on a specific variant needs only a subset of tools.
A CI pipeline for running static analysis (e.g. Mathworks Polyspace) does not need to build or package the software.

The problem is more noticeable for developers spinning up a virtual machine (VM) to quickly test a change or for a developer setting up their environment for the first time.

On the other hand, when releasing a product, it is essential to ensure that all tools are available because we have to make sure that all variants are passing the defined quality gates.

Brainstorming

💡 Categorizing Tools into Generic, Variant, and Build Target Sets

Let us categorize tools into three distinct sets:

Generic Tools: Tools required by all variants and build targets (e.g., CMake, Python).
Variant-Specific Tools: Tools required only for certain variants (e.g., specific compilers for microcontrollers).
Build Target-Specific Tools: Tools required for specific build targets (e.g., static analysis tools like Polyspace).

Implementation Concept

Separate Tool Dependency Files:
- Generic Tool Dependencies File: Contains tools common to all variants.
- Variant-Specific Tool Dependencies Files: Contains tools specific to a variant or overrides for generic tools.
Merging Process:
- When building a variant, merge the generic and variant-specific tool files, with the variant-specific tools overriding any conflicts.
Handling Build Target-Specific Tools:
- Dynamically add tools based on the selected build target.

This means that the install step in the pipeline will install only the tools required for the specific variant and build target.

Considerations

While this idea provides a structured approach to managing tool dependencies, there are some challenges/limitations to consider:

Complexity in Linking Tools to Build Targets: Tightly coupling tools to build targets might limit flexibility for other activities that require specific tools, such as generating reports or on-target debugging.
Aggregation of Tools for CI: Collecting all tools from generic, variant, and build target files for CI builds might introduce complexity in managing tool versions and resolving conflicts.

💡 Introducing “Scopes” as an Abstraction Layer

One issue with the previous idea is the tight coupling of tools to build targets. The problem is that, in contrast to variant, a build target is not a domain term in SPL engineering. It is a technical term from the build system. This tight coupling might lead to confusion and make the system less flexible.

To address this limitation, we proposed introducing scopes as a higher-level abstraction.

Scopes

Scopes encompass a specific activity or domain within the development process, representing a set of tools, configurations and steps required for that activity.

Examples of scopes may include: Build, StaticAnalysis, Debugging, Reporting, and Deployment.

Implementation Concept

Scope-Specific Configuration Files:
- Each scope has its own configuration that lists the tools required for that scope.
- It might also need to specify also what has to be done, which steps have to be executed.
Merging Tool Dependencies:
- Hierarchy: Generic tools < Variant Build Tools < Scope-specific tools.
- When building, the script merges tools from these files, with later ones overriding earlier ones in case of conflicts.
- There shall no be no variant specific tools anymore other than the toolchain for building the software.

Considerations

Complexity in Managing Scopes: Managing a growing number of scopes might become complex.
User Awareness: Developers need to understand which scopes are relevant for their tasks.
- How are scopes related to build targets?

💡 Keep current solution and speed up installation of all tools

Another idea was to keep the current solution but speed up the installation process. This could be achieved by:

CI jobs to install all tools on all agents
Disable automatic installation of tools on developer machines. The developer can then install the required tools manually.

This idea is the least intrusive and requires the least amount of changes. However, it does not address the root cause of the problem. It is more of a workaround. 🙈

Final Approach: Introducing Scopes

Introducing “scopes” as an abstraction layer offers a more flexible and domain-aligned solution for optimizing tool installation in an SPL build environment.

Scopes align with the development process, makes it easier for developers and platform engineers to understand and manage not only tool dependencies but also the steps required for a specific activity.

The final approach involves:

Defining Scopes: Identify and define scopes that represent activities or domains within the development process.
Scope-Specific Configuration Files: Create configuration files for each scope that list the tools required for that scope.
Merging Tool Dependencies: Merge tools from generic, variant build, and scope-specific files, with later ones overriding earlier ones in case of conflicts.
Implementing Scope-Specific Steps: Define and implement steps for each scope that specify what has to be done, which tools have to be installed, and which steps have to be executed.

This approach ensures that only the tools required for a specific activity are installed, optimizing the build environment and enhancing developer productivity.

There is one important aspect to consider: in the end one must be able insure a certain quality level for the software product as a whole. This means one needs to:

define quality gates for the whole product and breaking them down to the individual variants
map scopes to quality gates for each variant. Different variants might have different quality gates based on their maturity (e.g., prototype, beta, street ready).

The topics of quality gates and maturity levels, although related, deserve a separate blog post and I will not go into detail here.

Conclusion

I hope this blog post has provided you with insights into the challenges of optimizing tool installation in SPL build environments and the thought process in addressing this challenge.

There will for sure be more questions to answer when actually implementing the solution but until then, keep fingers crossed and stay tuned for more updates on this topic.

Efficiently Embedding Git Information in C Projects

2024-10-01T00:00:00+00:00

Introduction

In software development, embedding Git metadata (like commit IDs, branches, and tags) into your binaries is useful for debugging and traceability. However, how you integrate this information can significantly impact your build times and workflow efficiency. This post explores the challenges of embedding Git information in C projects using CMake and proposes an optimized solution.

Current Approaches

Project One: Command-Line Defines

In the first project, Git information is passed directly as compiler command-line arguments using add_compile_options in CMake:

add_compile_options(
    -DGIT_CUR_COMMIT=\"${GIT_CUR_COMMIT}\"
    -DGIT_CUR_USER=\"${GIT_CUR_USER}\"
)

These defines are then used in the codebase to embed Git metadata:

const uint8 git_commit[] = GIT_CUR_COMMIT;

Advantages

Simplicity: Easy to implement and understand.
Direct Integration: Git information is directly available in the code via macros.

Disadvantages

Inefficient Builds: Any change in the command-line arguments (e.g., a new commit ID) invalidates the build cache, forcing a full recompilation.
Poor Incremental Build Support: Full builds slow down development, especially in large product with multiple variants.
Inconsistent Updates: The file generation relies on the CMake configure step, which may not run if no CMake files have changed.

Project Two: Generated Source File

The second project generates a C source file during the CMake configuration step:

set(GIT_INFO_TEMPLATE ${CMAKE_SOURCE_DIR}/src/git_info.c.in)
set(GIT_INFO_FILE_OUT ${CMAKE_BINARY_DIR}/Src/git_info.c)

add_custom_target(version_info COMMAND ${CMAKE_COMMAND}
    -DUPDATE_VERSION_INFORMATION_REQUESTED=1
    -DVERSION_INFORMATION_FILE=${GIT_INFO_TEMPLATE}
    -DVERSION_INFORMATION_FILE_OUT=${GIT_INFO_FILE_OUT}
    -DGIT_VARIANT=${VARIANT}
    -P ${CMAKE_SOURCE_DIR}/src/gitinfo.cmake
    BYPRODUCTS ${GIT_INFO_FILE_OUT}
)

spl_add_source(${GIT_INFO_FILE_OUT})
spl_create_component()

Advantages

Selective Compilation: Only the generated file re-compiles when Git information changes.
Improved Build Times: Reduces the need for full recompilations.

Disadvantages

Inefficient Builds: The generated file is recompiled every time, even if the Git information hasn’t changed.
Complexity: Adds extra steps and dependencies in the build process.

Proposed Solution: Embedding Git Information in the Binary

To overcome the problems of current methods, we propose embedding Git information directly into the binary (using Intel-HEX format) after the build process. This approach eliminates unnecessary recompilation and simplifies the build process by embedding Git metadata in a dedicated memory section.

Create the git information source files

We need to have:

global constants for the Git information
compiler directives to place the constants in a specific memory section

git_info.c

#include "git_info.h"

#define GITINFO_START_SEC_CONST
#include "git_info_mem_map.h"

const unsigned char git_commit[GIT_COMMIT_LENGTH] = "0123456701234567";

#define GITINFO_STOP_SEC_CONST
#include "git_info_mem_map.h"

Important

The git_info.c file contains dummy Git information and is used only for a placeholder. The actual Git information will be updated after the build process.

git_info_mem_map.h

#if defined( GITINFO_START_SEC_CONST )
#pragma protect
#pragma section nearrom "GitInfoSection"
#pragma section farrom "GitInfoSection"
# undef GITINFO_START_SEC_CONST
# define START_SEC_CODE
#endif

#if defined( GITINFO_STOP_SEC_CONST )
#pragma endprotect
#pragma section nearrom restore
#pragma section farrom restore
# undef GITINFO_STOP_SEC_CONST
# define STOP_SEC_CODE
#endif

Note

The GitInfoSection memory section shall be defined in the linker script.

For other modules to access the Git information, we need to define the Git information in a header file:

git_info.h

#ifndef GIT_INFO_H
#define GIT_INFO_H

#define GIT_COMMIT_LENGTH 16
extern const unsigned char git_commit[GIT_COMMIT_LENGTH];

#endif

Linker Script to Define the Git Information Memory Section

/* Start address to store the git information */
#define GITINFO_ADDRESS                         (0xABCD)

section_layout :vtc:linear
{
	group PFLASH0(fill = 0x00)
	{
		group GitInfoSectionGroup (ordered, run_addr=GITINFO_ADDRESS)
		{
			section "GitInfoSectionGroup_SEC" (fill, blocksize = 2, attributes = rx)
			{
					select "[.]rodata.GitInfoSection";
			}
		}
		"_GitInfoSectionGroup_START" = "_lc_gb_GitInfoSectionGroup";
		"_GitInfoSectionGroup_END" = ("_lc_ge_GitInfoSectionGroup" == 0) ? 0 : "_lc_ge_GitInfoSectionGroup" - 1;
		"_GitInfoSectionGroup_LIMIT" = "_lc_ge_GitInfoSectionGroup";
	}
}

This linker script defines a memory section GitInfoSection at the specified address GITINFO_ADDRESS to store the Git information. The constants defined in git_info.c (rodata) will be placed in this memory section.

CMake script to update the Git information

We need to define custom commands to create a git_info.hex file containing the Git information.

Requirements

the git_info.hex file shall only be generated if the git commit has changed
the command for checking the git commit shall always run, to make sure the git_info.hex file is up-to-date

Important

As you might have noticed, we need to always run the command for checking the git commit but only generate the git_info.hex file if the git commit has changed. This is a bit tricky to achieve with CMake, but it is possible.

Always Generate Git Commit Temporary File

Purpose: Ensures the Git commit ID is updated every build.
Mechanism: Uses a fictive output git_commit_force_update to force the command to run every time.

add_custom_command(
    OUTPUT __git_commit_force_update__
    BYPRODUCTS ${GIT_COMMIT_TMP_FILE}
    COMMAND git describe --always --dirty --exclude '*' --abbrev=8 > ${GIT_COMMIT_TMP_FILE}
    COMMENT "Generate the git commit tmp file"
    VERBATIM
)

Update Git Commit File if Changed

Purpose: Copies the temporary commit ID file to the final file only if it has changed.
Mechanism: Uses copy_if_different to avoid unnecessary updates.

add_custom_command(
    OUTPUT ${GIT_COMMIT_FILE}
    COMMAND ${CMAKE_COMMAND} -E copy_if_different ${GIT_COMMIT_TMP_FILE} ${GIT_COMMIT_FILE}
    COMMENT "Checking and updating git commit ID"
    DEPENDS __git_commit_force_update__ ${GIT_COMMIT_TMP_FILE}
    VERBATIM
)

Create Git Info Hex File

Purpose: Converts the Git commit ID into a hex file at the specified address.
Mechanism: Uses hextool to generate the hex file.

add_custom_command(
    OUTPUT ${GIT_INFO_HEX_FILE}
    COMMAND hextool create --input-binary ${GIT_COMMIT_FILE} --offset ${GITINFO_ADDRESS} --output ${GIT_INFO_HEX_FILE}
    DEPENDS ${GIT_COMMIT_FILE}
    COMMENT "Creating git info hex file"
    VERBATIM
)

Note

Please notice the --input-binary hextool option to read the git information as binary data directly from the file.

Merge Git Info Hex File with the Binary

Purpose: Combines the main output hex file with the Git info hex file.
Output: Produces link_out_with_git_info.hex containing the embedded Git commit ID.

add_custom_command(
    OUTPUT ${CMAKE_BINARY_DIR}/link_out_with_git_info.hex
    COMMAND hextool merge --file ${CMAKE_BINARY_DIR}/link_out.hex --file ${GIT_INFO_HEX_FILE} --output ${CMAKE_BINARY_DIR}/link_out_with_git_info.hex
    COMMENT "Merging git info to the output hex file"
    DEPENDS ${GIT_INFO_HEX_FILE} ${CMAKE_BINARY_DIR}/link_out.hex
    VERBATIM
)

Process Flow Diagram

        flowchart TD
    A[Start Build] --> B[Generate git_commit_tmp.txt]
    B --> C{Has Commit ID Changed?}
    C -- Yes --> D[Update git_commit.txt]
    C -- No --> E[Skip Update]
    D & E --> F[Create git_info.hex]
    F --> G[Generate link_out.hex]
    G --> H[Merge git_info.hex with link_out.hex]
    H --> I[Produce link_out_with_git_info.hex]

Alternative approach as POST_BUILD command

Purpose: Merges git_info.hex directly into link_out.hex as a POST_BUILD step, overwriting the original file.
Mechanism:
- Post-Build Command: Executes after the link target finishes building.
- Force Relinking: Adds git_info.hex as a dependency to ensure that the linker runs again if the Git info changes.
Output: The link_out.hex file now contains the embedded Git commit ID without creating a new file.

# Add post-build command to merge the git info to the link_out.hex.
add_custom_command(
    TARGET link
    POST_BUILD
    COMMAND hextool merge --file ${CMAKE_BINARY_DIR}/link_out.hex --file ${GIT_INFO_HEX_FILE} --output ${CMAKE_BINARY_DIR}/link_out.hex
    COMMENT "Merging git info to the output hex file"
    # (!) Adding DEPENDS has no effect on POST_BUILD commands. So next line is useless.
    DEPENDS ${GIT_INFO_HEX_FILE}
    VERBATIM
)

# (!) Force linking again if the git info hex has changed.
# This hack adds the git info hex file as a dependency to the src_git_info target. When the git info hex file changes, the src_git_info target will be considered out of date and will be rebuilt.
set_property(TARGET src_git_info PROPERTY INTERFACE_LINK_DEPENDS ${GIT_INFO_HEX_FILE})

        flowchart TD
    A[Start Build] --> B[Generate git_commit_tmp.txt]
    B --> C{Has Commit ID Changed?}
    C -- Yes --> D[Update git_commit.txt]
    C -- No --> Z[End Build]
    D --> F[Create git_info.hex]
    F --> G{Has git_info.hex Changed?}
    G -- Yes --> H[Link link_out.hex]
    G -- No --> Z
    H --> K[Merging git_info.hex into link_out.hex - Post-Build]
    K --> L[Updated link_out.hex with Git info]
    L --> Z

Advantages

Single Output File: Avoids creating an extra hex file; simplifies deployment.
Always Updated: Ensures link_out.hex always contains the latest Git info.

Disadvantages

Forced Relinking: Changes in git_info.hex cause the linker to run again, potentially increasing build times. Note: Only the linking step is rerun; source files are not recompiled.
Overwriting Output: Original link_out.hex is modified, which may not be desirable in all workflows.

Conclusion

We have explored the challenges of embedding Git information in C projects and proposed an optimized solution using CMake and Intel-HEX format. This approach ensures that Git metadata is efficiently embedded in the firmware binary without unnecessary recompilations by:

Extracting the current Git commit ID during each build.
Updating the commit ID file only when changes occur to avoid unnecessary rebuilds.
Embedding the commit ID into the hex file at a specific memory address.
Merging the Git info hex file with the main output using one of the two approaches.

I hope this post is helpful in optimizing your build process.