SPL Bootstrap - Handle Python Version#

Introduction#

The pipeline implementation for our Software Product Line (SPL) repositories uses the pypeline Python application. In order to start the pypeline application, we need to install Python (with a specified version) and create the Python virtual environment. This step we call bootstrap and it is implemented here.

Note

Currently bootstrap supports only Windows for installing Python.

Workflow#

The bootstrap process is managed by two main scripts: a PowerShell script (bootstrap.ps1) for environment setup and a Python script (bootstrap.py) for dependency management. The diagram below shows the high-level workflow.

        flowchart TD
    A[bootstrap.ps1] --> B{Python installed?};
    B -- No --> C[Install Python w/ Scoop 1️⃣];
    B -- Yes --> D;
    C --> D[Execute bootstrap.py];

    D --> E{Dependencies changed?};
    E -- No --> F[End];
    E -- Yes --> G[Create/Update .venv 2️⃣];
    G --> H[Install Dependencies];
    H --> F;

  1. This step will first install the scoop windows package manager and then call scoop to install the configured python version.

  2. The python virtual environment is updated in multiple steps:

    • create an empty virtual environment with python venv.create

    • install the package manager (e.g., poetry, uv, etc.) and pip-system-certs for using the system SSL certificates

    • run the package manager to install the python dependencies

bootstrap.ps1#

  • Scoop configuration and installation

  • Python version detection and installation

  • Delegates to bootstrap.py for virtual environment setup

bootstrap.py#

  • OS-agnostic virtual environment creation (Windows/Unix)

  • Smart caching with hash-based dependency tracking. Only runs if dependencies have been updated.

  • Package manager support: Poetry, UV, Pipenv

  • Automatic pip configuration for custom PyPI sources

bootstrap.json Configuration#

{
    "python_version": "3.11",
    "python_package_manager": "poetry>=2.1.0",
    "scoop_ignore_scoopfile": true
}

Things to Improve#

Python Version Management#

Currently the bootstrap process installs a fixed Python version as specified in the bootstrap.json configuration file. When checking if python is installed, it only verifies that the major and minor version match (e.g., 3.11), but does not check for patch versions (e.g., 3.11.4 vs 3.11.5). This can lead to situations that on different machines, slightly different patch versions of Python are installed.

Note

This might not be a big issue in most cases, but it can lead to inconsistencies in our particular case, because we use the python venv module to create the initial virtual environment, which will include the standard library of the installed Python version.

The solution could be:

  • to find all installed python versions in path and check if the exact version is already installed

  • it might be that python versions installed with scoop are not available in path, so we need to query scoop for installed versions

  • if the exact version is not installed, install it with scoop

  • the bootstrap.py shall fail if the python interpreter version does not match exactly the configured version. This can happen if the user runs the pypeline including the CreateVEnv step which calls directly the bootstrap.py script.

Virtual Environment Management#

As mention above, the virtual environment is created in three steps:

  1. create an empty virtual environment with python venv.create

  2. install the package manager (e.g., poetry, uv, etc.) and pip-system-certs for using the system SSL certificates

  3. run the package manager to install the python dependencies

There is a problem with the second step: the package manager and pip-system certs are installed with pip. These packages have their own dependencies, which are not tracked by pip because pip has no support for .lock files.

When step three is executed, the package manager might need to upgrade/downgrade some packages which were implicitly installed in step two and this can cause crashes due to dependency conflicts.

Solution 1: User-defined Initial Packages#

One way to solve this is to let the user define the list of package to install in step two in the bootstrap.json configuration file. For example:

{
    "python_version": "3.11.4",
    "venv_initial_packages": [
        "poetry==2.1.0",
        "pip-system-certs==2.2.1",
        "wrapt==1.14.0"
    ],
}

In case there is a conflict between the initial packages and the packages defined in the package manager lock file, the user can adjust the versions in the configuration file accordingly.

Solution 2: Install Package Manager with Scoop#

Another way could be to install both python and the package manager with scoop in the bootstrap.ps1 script, so that pip is not involved at all in step two. This means that package manager will install exactly the version specified in the .lock file and therefore always have a consistent set of dependencies.

The problem with this approach is that pip-system-certs is not available as a scoop package and package installation will fail when ran against an on-premise PyPI server with self-signed certificates.

Package manager command#

Currently the command to install dependencies with the package manager is hardcoded in the bootstrap.py script with extra arguments being supported via the bootstrap.json configuration file.

{
    "python_package_manager_args": "--clean"
}

We added the feature to support extra arguments because we needed for some projects to pass the --clean to pipenv to remove unused packages already existing in the virtual environment.

A better approach would be to let the user define the full command to install dependencies with the package manager. For example:

{
    "venv_install_command": "poetry install --no-dev --clean"
}

This would give the user full control over how dependencies are installed.

Conclusion#

If all these improvements are implemented, the bootstrap process will be more reliable and flexible, ensuring consistent Python environments.

The configuration will allow users to tailor the bootstrap process to their specific needs.

{
    "python_version": "3.11.4",
    "venv_initial_packages": [
        "poetry==2.1.0",
        "pip-system-certs==2.2.1"
    ],
    "venv_install_command": "poetry install --no-dev --clean"
}

This configuration will ensure that:

  • Python 3.11.4 is installed

  • The package manager and pip-system-certs are installed with the specified versions

  • Dependencies are installed with the specified command

Have fun bootstrapping! ✌️

SPL components in YANGA