Local Pipeline

Overview

A local pipeline combines several project commands into one reproducible development workflow.

It does not introduce a second command implementation. Instead, it orchestrates the same ci_commands that can also be executed individually or from a CI provider.

The command architecture itself is described in CI Commands.

This guide focuses only on combining those commands into a larger workflow.

Current cikit Workflow

The cikit repository provides a project-specific convenience command:

uv run cikit all

Its current sequence is:

init
 |
ruff
 |
pylint
 |
unit tests
 |
system tests
 |
coverage
 |
sphinx

This sequence belongs to the cikit project.

Projects using cikit may define a different workflow or omit an all command entirely.

Orchestrating Existing Commands

The local all command imports existing project command modules and calls their command() functions.

Conceptually:

steps = [
    ("Init", init.command, ...),
    ("Ruff", ruff.command, ...),
    ("Pylint", pylint.command, ...),
    ("Unit tests", test.command, ...),
    ("System tests", test.command, ...),
    ("Coverage", coverage.command, ...),
    ("Sphinx", sphinx.command, ...),
]

for title, func, step_args in steps:
    result = func(step_args)
    if result != 0:
        return result

Because the normal command entry points are reused, every step keeps its normal cikit lifecycle, run data, artifacts, provider handling, and error behavior.

The pipeline command is responsible only for:

  • step order

  • child-command arguments

  • logical run names

  • shared intermediate paths

  • stop/continue policy

Fail-Fast Behavior

The current cikit all command stops after the first child command that returns a non-zero exit code.

step returns 0
    -> continue

step returns non-zero
    -> stop
    -> return that exit code

This decision happens between commands.

Inside one command, recoverable errors may still be handled by CiSection so that reports and artifacts can be completed before that command returns its final non-zero status.

The distinction is therefore:

inside one command
    -> recoverable failure may continue to reporting

between commands
    -> non-zero command result stops all

Run Names

Each child command has its own logical run name.

The cikit local workflow uses runs such as:

init
ruff
pylint
unittests
systemtests
coverage
sphinx

The same test command is executed twice with different run names:

test command
    |
    +-- unittests
    |
    `-- systemtests

This keeps test data, timelines, reports, and artifact metadata separated.

Logical run names are persistent identifiers and should remain stable when historical cikit data is used.

Local Pipeline Identity

Separate child commands can still belong to one local pipeline execution.

LocalGitProvider generates a local pipeline id on first access:

auto-<timestamp>-<random-id>

and stores it in the current process environment as:

CIKIT_PIPELINE_ID

Later provider instances created by subsequent child commands reuse the same value.

Conceptually:

uv run cikit all
    |
    +-- init          run=init
    +-- ruff          run=ruff
    +-- pylint        run=pylint
    +-- unit tests    run=unittests
    +-- system tests  run=systemtests
    +-- coverage      run=coverage
    `-- sphinx        run=sphinx
          |
          `-- same local pipeline_id

This mirrors the separation between pipeline identity and logical run names used by remote CI providers.

Pipeline Initialization

The cikit workflow begins with the init command.

Among other project-specific setup steps, it:

  • prepares the output directories

  • displays the resolved environment

  • generates static badges

  • initializes cikit data for the current pipeline

The reusable:

data_prepare(env)

macro creates the pipeline entry in the configured cikit data backend.

It is intended to run once near the beginning of a pipeline sequence.

Shared Intermediate Data

Logical commands normally exchange persistent information through cikit data and registered artifacts.

Some tools additionally require temporary intermediate files.

Python coverage is an example.

Coverage Workflow

The cikit local workflow uses one shared directory for coverage fragments:

build/coverage

Both test runs write into it:

unit tests
    |
    +-- coverage fragment
    |
system tests
    |
    +-- coverage fragment
    |
    v
coverage
    |
    +-- combine fragments
    +-- generate reports
    +-- generate badge
    `-- generate timeline

Before starting, the all command removes stale coverage data:

if args.coverage_dir.exists():
    shutil.rmtree(args.coverage_dir)

This prevents files from a previous local execution from contaminating the new result.

The default can be changed with:

uv run cikit all --coverage-dir <directory>

Argument Construction

When one project command invokes another command function directly, it must provide the arguments that child command normally receives from argparse.

The cikit all command uses a small helper:

def ns(args: argparse.Namespace, name: str = None, **kwargs) -> argparse.Namespace:
    values = {"verbose": args.verbose, **kwargs}

    if name is not None:
        values["name"] = name

    return argparse.Namespace(**values)

For example, the unit-test step can be represented as:

ns(args, "unittests", test_path=Path("tests/unittests"), coverage_dir=args.coverage_dir)

This supplies:

  • the global verbosity setting

  • the logical run name

  • the test path

  • the shared coverage directory

Nested command arguments are not parsed again. The orchestration command must propagate or construct the required values explicitly.

Running Steps Individually

The local pipeline is only a convenience layer.

Every child command can still be executed on its own.

For example:

uv run cikit init

uv run cikit ruff src tests ci_commands

uv run cikit pylint src tests ci_commands

uv run cikit --name unittests test tests/unittests --coverage-dir build/coverage

uv run cikit --name systemtests test tests/systemtests --coverage-dir build/coverage

uv run cikit coverage build/coverage

uv run cikit sphinx docs

This is useful when only one part of the workflow is being developed or debugged.

Producer and Consumer Ordering

Some commands consume data or artifacts produced by previous commands.

The local pipeline must preserve that dependency order.

Coverage is one example:

tests
   |
   v
coverage

Sphinx is another:

lint / tests / coverage / build
   |
   +-- RST reports
   +-- badges
   +-- timelines
   +-- logs
   +-- generated artifacts
   |
   v
sphinx.prepare()
   |
   v
sphinx build

Running Sphinx before its producer commands would result in a documentation build without those newly generated inputs.

Build workflows follow the same principle:

configure
   |
build
   |
register outputs
   |
bundle
   |
build report
   |
sphinx

See Builds and Sphinx Documentation Build.

Local and CI Orchestration

A local pipeline and a CI pipeline do not need to have identical scheduling.

The important goal is that they reuse the same project commands.

Locally, a sequential workflow is convenient:

init
  -> lint
  -> tests
  -> coverage
  -> sphinx

A CI platform may schedule independent jobs in parallel:

           +-- ruff
           |
init ------+-- pylint
           |
           +-- tests
                 |
                 v
              coverage
                 |
                 v
              sphinx

The individual commands can remain unchanged.

Provider-specific concerns such as:

  • GitLab stages

  • Azure DevOps jobs

  • runners or agents

  • artifact retention

  • deployment rules

belong in the CI configuration, not in the local orchestration command.

Project-Specific Workflows

Projects are not expected to copy cikit’s own all sequence.

A Python project may use:

init
 |
lint
 |
tests
 |
coverage
 |
sphinx

A CMake project may use:

init
 |
lint
 |
configure
 |
build
 |
ctest
 |
package
 |
sphinx

A PlatformIO project may use:

init
 |
build
 |
tests
 |
package
 |
sphinx

The project decides which commands exist and how they depend on one another.

Implementing an all Command

A local orchestration command can remain small.

For example:

import argparse

from ci_commands import init, lint, test


def run_step(title: str, func, args: argparse.Namespace) -> int:
    print()
    print("=" * 80)
    print(f"CI STEP: {title}")
    print("=" * 80)

    return func(args)


def command(args: argparse.Namespace) -> int:
    steps = [("Init", init.command, argparse.Namespace(verbose=args.verbose)),
             ("Lint", lint.command, argparse.Namespace(verbose=args.verbose,
                                                       paths=["src", "tests"])),
             ("Tests", test.command, argparse.Namespace(verbose=args.verbose, name="tests")),
    ]

    for title, func, step_args in steps:
        result = run_step(title, func, step_args)
        if result != 0:
            return result

    return 0


def register(subparsers: argparse._SubParsersAction):
    parser = subparsers.add_parser("all", help="Run the complete local workflow.")
    parser.set_defaults(func=command)

The exact steps and arguments are project policy.

The implementation should not contain reusable tool integration; that remains inside the individual commands and cikit macros.

Artifacts and Run Data

Child commands persist their own run data and artifact metadata.

The local orchestration command does not need to maintain a second shared artifact model.

For example:

ruff
   -> run data
   -> BADGE
   -> RST
   -> TIMELINE

unittests
   -> run data
   -> XUNIT
   -> RST
   -> TIMELINE

coverage
   -> run data
   -> FOLDER
   -> BADGE
   -> RST
   -> TIMELINE

Later commands can consume these persisted outputs through the normal cikit data and artifact APIs.

See Cikit Data and Artifacts.

Design Guidelines

Keep the local pipeline focused on orchestration.

It should primarily define:

  • step ordering

  • child-command arguments

  • logical run names

  • shared intermediate directories

  • dependency ordering

  • fail-fast behavior

Clean Shared Temporary State

If several commands write into one temporary working directory, remove stale state before the pipeline begins.

Coverage fragments are the current cikit example.

Keep Run Names Stable

Run names participate in cikit data, artifacts, reports, and timeline history.

Use explicit names when one command implementation represents several logical runs.

Preserve Dependency Order

Consumer commands must run after their producers.

Typical dependencies are:

tests -> coverage
build -> build report
reports -> sphinx

Keep CI Scheduling Outside

Do not encode GitLab or Azure DevOps scheduling concepts into the local all command.

The common layer is the project command itself. Local and remote orchestration can differ.

See Also