mpyl

Modular Pypeline Library

coverage

What is MPyL?

MPyL stands for Modular Pipeline Library (in Python).

This library is loosely based on the principles described in https://www.jenkins.io/blog/2019/01/08/mpl-modular-pipeline-library/ but completely independent of Jenkins or any other CI/CD platform.

🔖 Documentation

Detailed, complete, searchable documentation can be found at https://vandebron.github.io/mpyl

📚 Principles

MPyL is built with the following principles in mind.

Not a platform It is not tied to any CI/CD platform. MPyL is a library, not a framework. It is not a runner, not a scheduler, and it doesn't have a GUI. It can be plugged into any CI/CD platform. Execution flows for Jenkins or Dagster are included as an example.
Minimal footprint: It is self-contained and has very few external dependencies (e.g. Git, Docker). It can be run anywhere, also on your local machine.
Accessible and maintainable
- Written in Python, the most widely adopted scripting language with an extensive amount of client libraries relevant to CI/CD.
- Strongly typed: MyPy type hints and schemas for all YAML files. Clearly defined interfaces for inputs and outputs of steps.
- Focus on short feedback loop. Unit testable and everything can be run locally.
Self documented by docstrings in the code, descriptions in the schemas and explanations in the CLI.
Python as YAML templating engine Attempts to mix markup and logic tend to have a worst of both worlds outcome. MPyL uses YAML strictly for configuration. Where YAML needs to produced dynamically, like for CRDs via HELM charts, it is generated by Python code. The produced YAML is validated against the corresponding schemas.
Extensible Support for the most common use cases (e.g. Docker, Helm, Kubernetes, AWS, etc.) is built-in. But it is easy to extend with your own mpyl.steps.

💻 Technologies

Requirements

The following technologies are expected to be present on the local OS:

Python >= 3.9
Pip >= 23.0.1
Docker > 20
Docker compose installed as plugin (docker compose version) >= v2.2.3
Git SCM

Bundled

MPyL is extensible and has a minimal footprint. Having said that, batteries for the following technologies are included.

CI/CD

Build

Docker mpyl.steps.build.dockerbuild
Scala (SBT) mpyl.steps.build.sbt
Jenkins mpyl.cli.commands.build.jenkins and mpyl.utilities.jenkins.runner.JenkinsRunner

Testing

Junit mpyl.steps.models.ArtifactType.JUNIT_TESTS

Deployment

Reporting

🖐️ Usage

..MPyL CLI

Suggested first time use

1. Install MPyL

pip install mpyl
mpyl --help

2. Health check

⭐ It is recommended to run this before running any other commands.

mpyl health

Will validate the configuration and check if all required tools are installed.

3. Run a local build via the CLI

Find out which projects need to be built.

mpyl build status

Run a build.

mpyl build run

4. Run a CI build on your Pull Request

Create a pull request.

gh pr create --draft

If you use MPyL in a github action, a build will be triggered automatically and the results will be reported there. If you use jenkins as your CI tool, you can trigger a build on your pull request:

mpyl build jenkins

Command structure

Usage: mpyl [OPTIONS] COMMAND [ARGS]...

  Command Line Interface for MPyL

Options:
  --help  Show this message and exit.

Commands:
  build     Pipeline build commands
  health    Health check
  projects  Commands related to MPyL project configurations (project.yml)
  version   Version information

Top level commands options are passed on to sub commands and need to be specified before the sub command. In mpyl projects --filter <name> list, the --filter option applies to all project commands, like list or lint.

Projects

Usage: mpyl projects [OPTIONS] COMMAND [ARGS]...

  Commands related to MPyL project configurations (project.yml)

Options:
  -c, --config PATH  Path to the config.yml. Can be set via `MPYL_CONFIG_PATH`
                     env var.   [required]
  -v, --verbose
  -f, --filter TEXT  Filter based on filepath
  --help             Show this message and exit.

Commands:
  lint     Validate the yaml of changed projects against their schema
  list     List found projects
  names    List found project names
  show     Show details of a project
  upgrade  Upgrade projects to conform with the latest schema

Repo

Usage: mpyl repo [OPTIONS] COMMAND [ARGS]...

  Manage CVS (git) repositories

Options:
  -c, --config PATH      Path to the config.yml. Can be set via
                         `MPYL_CONFIG_PATH` env var.
  -p, --properties PATH  Path to run properties  [default: run_properties.yml]
  -v, --verbose
  --help                 Show this message and exit.

Commands:
  init    Initialize the repository for a build run
  status  The status of the current local repository

Build

Usage: mpyl build [OPTIONS] COMMAND [ARGS]...

  Pipeline build commands

Options:
  -c, --config PATH      Path to the config.yml. Can be set via
                         `MPYL_CONFIG_PATH` env var.   [required]
  -p, --properties PATH  Path to run properties  [default: run_properties.yml]
  -v, --verbose          Verbose output
  --help                 Show this message and exit.

Commands:
  artifacts  Commands related to artifacts like build cache and k8s manifests
  clean      Clean all MPyL metadata in `.mpyl` folders
  run        Run an MPyL build
  status     The status of the current local branch from MPyL's perspective

MPyL configuration

MPyL can be configured through a file that adheres to the mpyl_config.yml schema.
Which configuration fields need to be set depends on your usecase. The error messages that you encounter while using the cli may guide you through the process. Note that the included mpyl_config.example.yml is just an example.

Secrets can be injected through environment variable substitution via the pyaml-env library. Note that values for which the ENV variable is not set, will be absent in the resulting configuration dictionary.

Example config

vcs:
  changedFilesPath: !ENV ${CHANGED_FILES_PATH}
  git:
    mainBranch: 'origin/main'
    ignorePatterns: ['*.md', '*.svg']
    projectSubFolder: "deployment"
    projectFile: "project.yml"
    remote:
      url: 'https://github.com/acme/repo.git'
      userName: !ENV ${GIT_CREDENTIALS_USR}
      password: !ENV ${GIT_CREDENTIALS_PSW}
      email: !ENV ${GIT_EMAIL_ADDRESS:somebody@somewhere}
  github:
    repository: 'acme/repo'
    token: !ENV ${GITHUB_TOKEN}
    app:
      privateKeyBase64Encoded: !ENV ${MPYL_GITHUB_APP_PRIVATE_KEY}
      appId: '295700'
  cachingRepository:
    mainBranch: 'main'
    remote:
      url: 'https://github.com/acme/artifact-repo.git'
      userName: !ENV ${GIT_CREDENTIALS_USR}
      password: !ENV ${GIT_CREDENTIALS_PSW}
      email: "employee@acme.com"
  argoRepository:
    mainBranch: 'main'
    remote:
      url: 'https://github.com/acme/argocd.git'
      userName: !ENV ${GIT_CREDENTIALS_USR}
      password: !ENV ${GIT_CREDENTIALS_PSW}
      email: !ENV ${GIT_EMAIL_ADDRESS}
  argoGithub:
    repository: 'acme/argocd'
    token: !ENV ${GITHUB_TOKEN}
slack:
  botToken: !ENV ${SLACK_TOKEN}
  icons:
    building: '60fps_parrot'
    success: 'thug-parrot'
    failure: 'sadparrot'
jira:
  site: 'https://acme.atlassian.net'
  userName: !ENV ${MPYL_JIRA_TOKEN_USR:jira_user}
  password: !ENV ${MPYL_JIRA_TOKEN_PSW:jira_password}
  ticketPattern: '[A-Za-z]{2,}-\d+'
docker:
  defaultRegistry: 'acme.docker.com'
  registries:
    - hostName: 'aws.amazonaws.com'
      userName: !ENV ${AWS_ACCESS_KEY_ID:user}
      password: !ENV ${AWS_SECRET_ACCESS_KEY:password}
      region: 'us-east-1'
      provider: 'aws'
    - hostName: 'acme.docker.com'
      userName: !ENV ${DOCKER_REGISTRY_USR:docker_user}
      password: !ENV ${DOCKER_REGISTRY_PSW:docker_password}
      cache:
        cacheFromRegistry: false
        custom:
          to: 'type=gha,mode=max'
          from: 'type=gha'
  build:
    rootFolder: '.'
    buildTarget: 'builder'
    testTarget: 'tester'
    dockerFileName: 'Dockerfile-mpl'
  compose:
    periodSeconds: 2
    failureThreshold: 20
sbt:
  command: 'sbt'
  clientCommand: 'sbtn'
  testWithCoverage: !ENV ${SBT_RUN_WITH_COVERAGE:false}
  verbose: false
  javaOpts: '-Xmx4G -Xms4G -XX:+UseG1GC -Xss2M'
  sbtOpts: 'user.timezone=GMT jline.terminal=jline.UnixTerminal'
  clientMode:
    build: false
    test: false
whiteLists:
  default: [ "VPN" ]
  addresses:
    - name: "VPN"
      all: [ "10.0.0.1" ]
    - name: 'Outside-World'
      all: [ '0.0.0.0/0' ]
    - name: 'K8s-Test'
      all: [ '1.2.3.0', '1.2.3.1' ]
    - name: 'TargetSpecificWhitelist'
      pr: ['1.2.3.4']
      test: ['1.2.3.4']
      acceptance: ['2.3.4.5']
      production: ['3.4.5.6']
kubernetes:
  deploymentStrategy:
    rollingUpdate:
      maxUnavailable: "25%"
      maxSurge: "25%"
    type: "RollingUpdate"
  deployAction: HelmDeploy
  defaultCluster:
    pr: 'K8S-Test'
    test: 'K8S-Test'
    acceptance: 'K8S-Acceptance'
    production: 'K8S-Production'
  clusters:
    - name: K8S-Test
      clusterId: c-acme
      projectId: p-acme
      clusterEnv: test
      context: kind-chart-testing
    - name: K8S-Acceptance
      clusterId: c-acme
      projectId: p-acme
      clusterEnv: acce
      context: acme-k8s-acce
    - name: K8S-Production
      clusterId: c-acme
      projectId: p-lb52t
      clusterEnv: prod
      context: acme-k8s-prod
project: # default values
  allowedMaintainers: [ 'Team1', 'Team2', 'MPyL' ]
  deployment:
    kubernetes:
      rancher:
        projectId:
          all: invalidId
      imagePullSecrets:
        - name: 'acme-registry'
      job:
        ttlSecondsAfterFinished:
          all: 3600
      resources:
        instances:
          pr: 1
          test: 1
          acceptance: 1
          production: 3
        limit:
          cpus:
            pr: 0.5
            test: 0.5
            acceptance: 0.5
            production: 1.0
          mem:
            pr: 1024
            test: 1024
            acceptance: 1024
            production: 2048
      startupProbe:
        path:
          all: '/health'
        initialDelaySeconds: 4  # 0 - We expect service to rarely be up within 4 secs.
        periodSeconds: 2  # 10 - We want the service to become available as soon as possible
        timeoutSeconds: 3  # 1 - If the app is very busy during the startup stage, 1 second might be too fast
        successThreshold: 1  # 1 - We want the service to become available as soon as possible
        failureThreshold: 60  # 3 - 4 + 60 * 2 = more than 2 minutes
      livenessProbe:
        path:
          all: '/health'
        periodSeconds: 30  # 10
        timeoutSeconds: 20  # 1 - Busy apps may momentarily have long timeouts
        successThreshold: 1  # 1
        failureThreshold: 3  # 3
      metrics:
        path: '/metrics'
        enabled: true
    traefik:
      enabled: true
      hosts:
        - host:
            pr: "Host(`{SERVICE-NAME}-{PR-NUMBER}.{CLUSTER-ENV}-backend.nl`)"
            test: "Host(`{namespace}-{SERVICE-NAME}.{CLUSTER-ENV}-backend.nl`)"
            acceptance: "Host(`{namespace}-{SERVICE-NAME}.{CLUSTER-ENV}-backend.nl`)"
            production: "Host(`{namespace}-{SERVICE-NAME}.{CLUSTER-ENV}-backend.nl`)"
          tls:
            all: "le-custom-prod-wildcard-cert"
          insecure: false
          whitelists:
            all: [ "VPN" ]
    additionalTraefikRoutes:
      - name: "ingress-intracloud-https"
        clusterEnv:
          pr: "test-other"
          test: "test-other"
          acceptance: "acce-other"
          production: "acce-other"
        middlewares:
          - "intracloud-middleware@kubernetescrd"
        entrypoints:
          - "intracloud"
    traefikDefaults:
      httpMiddleware: "traefik-https-redirect@kubernetescrd"
      tls: "le-custom-prod-wildcard-cert"

Check the schema for run_properties.yml, which contains detailed documentation and can be used to enable on-the-fly validation and auto-completion in your IDE.

Stage configuration

MPyL can be configured to use an arbitrary set of build stages. Typical CI/CD stages are build, test or deploy. See mpyl.steps for the steps that come bundled and how to define and register your own.

Example stage configuration

$schema: http://json-schema.org/draft-07/schema#
$id: mpyl_stages.schema.yml
definitions:
  stageNames:
    enum:
      - "build"
      - "test"
      - "deploy"
      - "postdeploy"
  dependencies:
    type: object
    properties:
      build:
        type: array
        minItems: 1
      test:
        type: array
        minItems: 1
      deploy:
        type: array
        minItems: 1
      postdeploy:
        type: array
        minItems: 1
    additionalProperties: false

Auto completion

Usability of the CLI is greatly enhanced by autocompletion. To enable autocompletion, depending on your terminal, do the following:

Bash

Add this to ~/.bashrc:

eval "$(_MPYL_COMPLETE=bash_source mpyl)"

Zsh

Add this to ~/.zshrc:

eval "$(_MPYL_COMPLETE=zsh_source mpyl)"

Fish

Add this to ~/.config/fish/completions/foo-bar.fish:

eval (env _MPYL_COMPLETE=fish_source mpyl)

YAML auto completion

Schema based autocompletion

Intellij IDEA or PyCharm

Go to: Preferences | Languages & Frameworks | Schemas and DTDs | JSON Schema Mappings

Add new schema
Add matching schema file from latest release:
- */deployment/project.yml -> https://vandebron.github.io/mpyl/schema/project.schema.yml
- mpyl_config.example.yml -> https://vandebron.github.io/mpyl/schema/mpyl_config.schema.yml
- run_properties.yml -> https://vandebron.github.io/mpyl/schema/run_properties.schema.yml
Select version: JSON Schema Version 7
Add YAML files corresponding to the schema or add the file pattern. (For instance, adding the file pattern project.yml to the project.schema.yml will take care of autocompletion in any project.yml.)

..defining projects

File structure

All CI/CD related files reside in a ./deployment sub folder, relative to the project source code folder. A typical deployment folder may contain the following files

├── Dockerfile-mpl
├── project.yml
└── docker-compose-test.yml

project.yml

The project.yml defines which steps needs to be executed during the CI/CD process.

name: batterypackApi
stages:
  build: Sbt Build
  test: Sbt Test
  deploy: Kubernetes Deploy

name is a required parameter
stages are optional parameters. Stages that are undefined will be skipped. Depending on the type of project you want to build, you need to specify an appropriate action to be performed in each stage. For example: Sbt Build can be used for scala projects, and Docker Build can be used for front-end projects.
kubernetes is a required parameter if deploy stage is set to Kubernetes Deploy.

The schema for project.yml contains detailed documentation and can be used to enable on-the-fly validation and auto-completion in your IDE.

..setting up a CI-CD flow

MPyL is not a taskrunner nor is it a tool to define and run CI-CD flows. It does however provide a building blocks that can easily be plugged into any existing CI-CD platform.

Github actions

Github actions are a natural fit for MPyL. To build a pull request, you can use the following workflow:

name: Build pull request
on:
  push:
    branches-ignore: [ 'main' ]

jobs:
  Build_PR:
    name: Build and deploy the pull request
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-python@v4
        with:
          python-version: '3.9'

      - name: Install MPyL
        run: pip install 'mpyl==<latest_version>'

      - name: Initialize repo
        run: mpyl repo init --branch ${{ github.ref }} --url https://${{ env.GITHUB_TOKEN }}@github.com/${{ github.repository }}.git --pristine

      - name: Print execution plan
        run: mpyl build status

      - name: Build run
        run: mpyl build run

The --pristine flag in the mpyl repo init command will clone the repository into the current empty workspace, using

git clone --shallow-exclude main --single-branch --branch <branch_name> https://github.com/<org>/<repo>.git

This will result in a shallow clone of the repository, containing only the files that are relevant for the current pull request.

Dagster

Although dagster's primary focus is data processing and lineage, it can be used as a runner for MPyL. It provides a nice UI to inspect the flow and logs. It supports concurrent execution of steps in a natural way. These features make it a convenient runner for local development and debugging.

Dagster flow runner

from pathlib import Path

from dagster import (
    config_from_files,
    op,
    DynamicOut,
    DynamicOutput,
    get_dagster_logger,
    Output,
    Failure,
    job,
)

from mpyl.project import load_project, Project
from mpyl.stages.discovery import find_projects_to_execute
from mpyl.steps import build, test, deploy
from mpyl.steps.collection import StepsCollection
from mpyl.steps.run_properties import construct_run_properties
from mpyl.steps.steps import Steps, StepResult
from mpyl.utilities.pyaml_env import parse_config
from mpyl.utilities.repo import Repository, RepoConfig

ROOT_PATH = "./"


def execute_step(proj: Project, stage: str, dry_run: bool = True) -> StepResult:
    config = parse_config(Path(f"{ROOT_PATH}mpyl_config.yml"))
    run_properties = construct_run_properties(
        config=config, properties={}, run_plan=RunPlan.empty()
    )
    dagster_logger = get_dagster_logger()
    executor = Steps(dagster_logger, run_properties)
    step_result = executor.execute(stage, proj, dry_run)
    if not step_result.output.success:
        raise Failure(description=step_result.output.message)
    return step_result


@op(description="Build stage. Build steps produce a docker image")
def build_project(context, project: Project) -> Output:
    return Output(execute_step(project, build.STAGE_NAME))


@op(description="Test stage. Test steps produce junit compatible test results")
def test_project(context, project) -> Output:
    return Output(execute_step(project, test.STAGE_NAME))


@op(
    description="Deploy a project to the target specified in the step",
    config_schema={"dry_run": bool},
)
def deploy_project(context, project: Project) -> Output:
    dry_run: bool = context.op_config["dry_run"]
    return Output(execute_step(project, deploy.STAGE_NAME, dry_run))


@op(
    description="Deploy all artifacts produced over all runs of the pipeline",
    config_schema={"simulate_deploy": bool},
)
def deploy_projects(
    context, projects: list[Project], outputs: list[StepResult]
) -> Output[list[StepResult]]:
    simulate_deploy: bool = context.op_config["simulate_deploy"]
    res = []
    if simulate_deploy:
        for proj in projects:
            res.append(execute_step(proj, deploy.STAGE_NAME))
    else:
        get_dagster_logger().info(f"Not deploying {projects}")
    return Output(res)


def find_projects(stage: str) -> list[DynamicOutput[Project]]:
    yaml_values = parse_config(Path(f"{ROOT_PATH}mpyl_config.yml"))
    with Repository(RepoConfig.from_config(yaml_values)) as repo:
        changes_in_branch = repo.changes_in_branch_including_local()
        project_paths = repo.find_projects()
    all_projects = set(
        map(lambda p: load_project(Path("."), Path(p), strict=False), project_paths)
    )
    dagster_logger = get_dagster_logger()
    steps = StepsCollection(logger=dagster_logger)
    project_executions = find_projects_to_execute(
        logger=dagster_logger,
        all_projects=all_projects,
        stage=stage,
        changeset=changes_in_branch,
        steps=steps,
    )
    return list(
        map(
            lambda project: DynamicOutput(
                project, mapping_key=project.name.replace("-", "_")
            ),
            project_executions,
        )
    )


@op(out=DynamicOut(), description="Find artifacts that need to be built")
def find_build_projects() -> list[DynamicOutput[Project]]:
    return find_projects(build.STAGE_NAME)


@op(out=DynamicOut(), description="Find artifacts that need to be tested")
def find_test_projects(_projects) -> list[DynamicOutput[Project]]:
    return find_projects(test.STAGE_NAME)


@op(out=DynamicOut(), description="Find artifacts that need to be deployed")
def find_deploy_projects(_projects) -> list[DynamicOutput[Project]]:
    return find_projects(deploy.STAGE_NAME)


@job(config=config_from_files(["mpyl-dagster-example.yml"]))
def run_build():
    build_projects = find_build_projects()
    build_results = build_projects.map(build_project)

    test_projects = find_test_projects(build_results.collect())
    test_results = test_projects.map(test_project)

    projects_to_deploy = find_deploy_projects(test_projects.collect())

    deploy_projects(
        projects=projects_to_deploy.collect(), outputs=test_results.collect()
    )

It can be started from the command line with dagit --workspace workspace.yml.

Dagster flow Dagster run

..caching build artifacts

Docker images

Docker image layers built in previous runs can be used as a cache for subsequent runs. An external cache source can be configured in mpyl_config.yml as follows:

docker:
  registry:
    cache:
      cacheFromRegistry: true
      custom:
        to: 'type=gha,mode=max'
        from: 'type=gha'

The to and from fields map to --cache-to and --cache-from buildx arguments.

The docker cache can be used in both the mpyl.steps.build.dockerbuild and mpyl.steps.test.dockertest steps.

Artifacts

MPyL's artifact metadata is stored in the hidden .mpyl folders next to project.yml. These folders are used to cache information about (intermediate) build results. A typical .mpyl folder has a file for each executed stage. The BUILD.yml file contains the metadata for the build step. For example:

message: Pushed ghcr.io/samtheisens/nodeservice:pr-6
produced_artifact: !Artifact
  artifact_type: !ArtifactType DOCKER_IMAGE-1
  revision: b6c87b70c3c16174bdacac6c7dd4ef71b4bb0047
  producing_step: After Docker Build
  spec: !DockerImageSpec
    image: ghcr.io/samtheisens/nodeservice:pr-6

These files speed up subsequent runs by preventing steps from being executed when their inputs have not changed.

🧹 These .mpyl folders can be safely deleted to force a full rebuild via

mpyl build clean

Remote caching

To preserve intermediate build results between runs, you can use the

mpyl build artifacts push

command at the end of a run. This will push the .mpyl folder to the remote repository configured in mpyl_config.yml

vcs:
  cachingRepository:
      mainBranch: 'main'
      remote:
        url: 'https://github.com/acme/artifact-repo.git'
        userName: !ENV ${GIT_CREDENTIALS_USR}
        password: !ENV ${GIT_CREDENTIALS_PSW}
        email: "employee@acme.com"

To pull the previously pushed artifacts, use

mpyl build artifacts pull

at the beginning of your run.

..report the outcome of a pipeline run

MPyL comes with built-in reporters for Github, Jira and Slack. See mpyl.reporting.targets how to configure them and for instructions on how to create your own reporter.

..create a custom step

See mpyl.steps.

..create a build step

Building a docker image

If the output of your build step is a docker image, you can use the mpyl.steps.build.docker_after_build step to make sure the resulting image is tagged, pushed to the registry and made available as an artifact for later (deploy) steps.

..create a test step

Junit test results

MPyL can parse Junit test results for reporting purposes. Your test step needs to produce a mpyl.steps.models.ArtifactType.JUNIT_TESTS artifact. See mpyl.steps.test.echo for an example of how such an artifact can be created.

Integration tests

If your project includes "integration tests" that require a docker container to run during the test stage, you can define these containers in a file named docker-compose-test.yml. For example, to test your database schema upgrades, with a real postgres database:

Example docker-compose-test.yml

version: '2.1'
services:
  postgres:
    image: postgres:13.4
    environment:
      POSTGRES_DB: test_db
      POSTGRES_PASSWORD: test_password
    ports:
      - "5432:5432"
    healthcheck:
      test: [ "CMD-SHELL", "pg_isready -U postgres" ]
      interval: 2s
      timeout: 5s
      retries: 10

Note: make sure to define a reliable healthcheck to prevent your tests from being run before the database is fully up and running.

Developer instructions

..install mpyl for development

Clone the mpyl repo
```
gh repo clone Vandebron/mpyl
```
Install dependencies
```
pipenv install -d
```

..run tests and checks

To run linting (pylint), type checking (mypy) and testing (pytest) in one go, run:

pipenv run validate

..create a pull request build

After working on a branch in MPyL repo, you can open a PR. After every push, if all validations pass, a test release is pushed to https://test.pypi.org/project/mpyl/. The naming of the version follows a <pr_number>.<build_number> pattern.

A pull request build can be used in Pipfile via

pipenv install --index https://test.pypi.org/simple/ mpyl==<PR_NUMBER>.*

Resulting in:

[[source]]
url = "https://test.pypi.org/simple"
verify_ssl = false
name = "test"

[packages]
mpyl = { version = "==28.403", index = "test" }

..code style

We use the black formatter in our codebase. Check the instructions on how to set it up for your IDE here.

..create a new release

Create a new release notes file in releases/notes/ and name it <version>.md Noteworthy changes should be added to this file. Think: new cli commands, new features, breaking changes, upgrade instructions, etc. Ideally create this file already when starting to work on a new version. Each PR that is to be included in that release, can add their notes to this file.
Check out main and pull the latest changes
Choose what release type you want to create. We use semantic versioning. The most important distinction is between regular releases and release candidates.
1. A release candidate does not require release notes and will be published to test pypi.
2. A regular release requires does require and will be published to pypi.
Run pipenv run release create
Merge the PR created by this command
Run pipenv run release publish on main to publish the release

..troubleshoot Python setup

Check if you're in the correct venv To check this, run first:
```
pipenv shell
```
Then check if the correct virtual environment (named pympl) is launched.
Check your bashrc (or zshrc) if you have any overrides of environmental variables like PIPENV_PIPFILE. If so, remove those, source your bash config and try Step 1. again
To see if everything is running as intended, execute
```
pipenv run test
```
which should now succeed.

..installing a test release of MPyL

Test versions of MPyL are published for every pull request to test pypi. To install a test release, run:

pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple mpyl==<version>

..running the mpyl sourcecode against another repository

For a shorter feedback loop, you can run the mpyl sourcecode against another repository. To test the mpyl sourcecode against the peculiarities of your own repository, you can run the following command:

PIPENV_PIPFILE=/<absolute_path_to_mpyl_repo>/Pipfile pipenv run cli-ext build status

Assign PIPENV_PIPFILE to the absolute path of your Pipfile and run the command. ⚠️Note that an .env file needs to be present in the root if this repository, containing the following variable:

PYTHONPATH=/<absolute_path_to_mpyl_repo>/src

Release notes

MPyL 1.8.7

Ensure correct artifact is used as required input

By taking the most recent one

Details on Github

MPyL 1.8.6

Improve logging output

A single line per execution

Details on Github

MPyL 1.8.5

Upgrade dependencies

kubernetes = "==32.0.1" python-on-whales = "==0.76.1" click = "==8.1.8" boto3 = "==1.37.23"

Details on Github

MPyL 1.8.4

Remove validations on config load

To allow for other schemas to be used

kubernetes = "==32.0.1" junitparser = "==2.8.0" pyjwt = "==2.5.0" # forced downgrade due to https://stackoverflow.com/questions/33198428/jwt-module-object-has-no-attribute-encode cryptography = "==39.0.2" python-on-whales = "==0.76.1" pygithub = "==2.4.0" slack-sdk = "==3.33.3" atlassian-python-api = "==3.39.0" click = "==8.1.8" rich = "==13.9.4" pyaml-env = "==1.2.2" python-dotenv = "==1.0.0" questionary = "==2.0.1" toml = "==0.10.2" boto3 = "==1.37.23"

Details on Github

MPyL 1.8.3

Rename Steps to Executor

And remove late validation of mypl_config.yaml to allow for different config schemas

Details on Github

MPyL 1.8.2

Pass changed files down to steps

So that steps can decide to check changed files only

Details on Github

MPyL 1.8.1

Upgrade to python 3.12

And remove dagster and all its transient dependencies

Details on Github

MPyL 1.8.0

Increased flexibility

Allow project definitions to be described in files named different from project.yaml

Upgrade mypy

To latest version

Details on Github

MPyL 1.7.6

Use pathlib to determine path of project during discovery

Using pathlib instead of string manipulation to determine the build plan of project avoids name overshadowing like project_A and project_A_B.

Upgrading `rich` and `slack` dependencies

slack-sdk from 3.21.3 to 3.33.3
rich from 13.8.0 to 13.9.4

Details on Github

MPyL 1.7.5

Demonstrate MPyL plugin mechanism.

MPyL allows steps to be implemented outside and registered via a plugin mechanism

Creating steps outside the MPyL library gives a much quicker roundtrip, because you can build and test steps locally in the repo in which you're using MPyL.

The mechanism is demonstrated in plugin-run.py which imports two Gradle build steps defined in plugins/gradle.py.

Details on Github

MPyL 1.7.4

Add an ever-changing label to Dagster Helm deployments

use ChartBuilder.to_labels() to attach an ever changing label to dagster user code to trigger reloads of projects with every deploy
filter out app-labels from dagster user-code values since dagster attaches its own app-labels to the helm chart

Force recreations of K8s jobs when using ArgoCD

Adds a label to K8s jobs to force recreations when using ArgoCD

Make project folder configurable

Allow the subfolder (default deployment) to be configurable
Allows for better or different naming/structure

Details on Github

MPyL 1.7.3

Fix Dagster Kubernetes deployments

Fixed an issue where Dagster Kubernetes deployments were using the wrong mpyl_config value for their Kubernetes context
- Old scenario: used cluster_env
- New scenario: uses context
- This change makes sure the correct Kubernetes context is now chosen while deploying

Revert fix on PrometheusRules alerts description

A fix was added to allow for variable interpolation in PrometheusRules alerts description.
As this fix was only added for Helm deployments, and we are now using ArgoCD, this fix is no longer needed and has been reverted.

Various dependencies upgraded

Various dependencies have been upgraded

Details on Github

MPyL 1.7.2

Docker test fix

Ensure the docker test step uses build args

Details on Github

MPyL 1.7.1

BPM Diagram deploy remove Docker dependency

Remove the usage of Docker when deploying BPM diagrams
Instead, rely on the tool being pre-installed in the cluster (similar to Scala etc.)

Details on Github

MPyL 1.7.0

Project.yml changes

Add a pipeline field to select which pipeline to run (e.g. which Github Actions workflow) to use for a specific project

Run plan output

changed the format of the run plan JSON file to make parsing easier, reduce the amount of redundant values and consequently the size of the file

Details on Github

MPyL 1.6.11

Project.yml changes

Make projectId field mandatory to prevent rbac issues in rancher
Add optional field for project type

Camunda

Some small changes and bugfixes to the camunda step logic

Details on Github

MPyL 1.6.10

Deploy existing jobs

Fix the helm list cmd function to include the kube-context flag
Fixes the issue where existing jobs wouldn't be found and thus not deleted

Details on Github

MPyL 1.6.9

Cypress postdeploy

Remove the cypress postdeploy step
Slightly change the postdeploy config to align with other stages

Details on Github

MPyL 1.6.8

Configurable deployment cluster

Be able to specify a list of target clusters to deploy to
Be able to specify default targets per environment
Be able to override the default targets in the project.yaml

deployment:
  cluster:
    pr: 'leaf-test'
    test: 'test'
    acceptance: 'acce'
    production: 'prod'

Additional Traefik routes

Be able to define additional, generic Traefik routes in the project.yaml that are referenced from the mpyl_config.yaml

project.yaml

traefik:
    hosts:
      - host:
          pr: "Host(`payments-{PR-NUMBER}.{CLUSTER-ENV}.nl`)"
          test: "Host(`payments.test.nl`)"
          acceptance: "Host(`payments.acceptance1.nl`)"
          production: "Host(`payments.nl`)"
        tls:
          all: "le-custom-prod-wildcard-cert"
        insecure: true
        whitelists:
          test:
            - "Test"
        additionalRoute: "ingress-intracloud-https"

mpyl_config.yaml

deployment:
    additionalTraefikRoutes:
      - name: "ingress-intracloud-https"
        clusterEnv:
          pr: "test-other"
          test: "test-other"
          acceptance: "acce-other"
          production: "acce-other"
        middlewares:
          - "intracloud-middleware@kubernetescrd"
        entrypoints:
          - "intracloud"

Additional Traefik configuration

It is also possible to define the additional traefik configuration in the mpyl_config.yml:

deployment:
    traefikDefaults:
      httpMiddleware: "traefik-https-redirect@kubernetescrd"
      tls: "le-custom-prod-wildcard-cert"

Details on Github

MPyL 1.6.7

Improvements

Improve dependency linting: projects cannot depend on themselves
General logging improvements
Improved BPM Diagram Deploy step
Cypress code improvements

Details on Github

MPyL 1.6.6

Pass changed files

Allow to pass a .json of changed files to determine the run plan
The file format has to be a list/dict of {"path/to/file": "change_type"}, where change_type is one of ["A", "C", "D", "M", "R"]

Run plan

Add the maintainers field to the run plan json

Details on Github

MPyL 1.6.5

Tests

Execute a single sbt test command instead of test:compile and test:test sequentially (removes the need for the experimental thin client)

Dependency management

Always re-execute a stage for a project when one or more of their dependencies are modified
Produce a hash of the modified files even when the stage is cached (so a follow-up commit can be cached)

Details on Github

MPyL 1.6.4

Run results

Account for parallel runs when reading and writing the run results
Test results are now also added to the output artifact instead of just to a file

Run plan

The run plan file is now written to .mpyl/run_plan.pickle and .mpyl/run_plan.json (replaces the old confusing build_plan name)

Other changes

The root .mpyl folder is now also cleaned up as part of mpyl build clean
Do not fail the build when trying to create a Kubernetes namespace that already exists

Details on Github

MPyL 1.6.3

Build status

Write a simple build plan in json format to .mpyl/build_plan.json when using the build status command.

CloudFront Kubernetes Deploy

Remove support for CloudFront Kubernetes Deploy step.

Details on Github

MPyL 1.6.2

Memory limits

Doubled the general memory limit for dagster repos to 1024, as agreed within the dagster guild to prevent OOM.

Chart improvements

Be able to specify alerts in the project yaml when doing Kubernetes Job Deploy step
Create charts for these prometheus rules

Details on Github

MPyL 1.6.1

Bugfixes

Do not try to hash deleted or renamed files when building a cache key

Details on Github

MPyL 1.6.0

Improvements

Implement a different way to determine the build plan
Change the layout of the build plan print to improve transparency
Allow passing --stage to the build status command

Build set caching

Store the build set in disk and read it back when --sequential is passed as a parameter, preventing us to rebuild the plan on subsequent stages. Which means there is no need to diff with the main branch, thus no need to fetch the entire main branch history before running mpyl. This is a significant performance improvement as you only need to do a clone with full history for the first stage, and run all others using a shallow clone (much faster to check out on repositories with many commits).

Details on Github

MPyL 1.5.1

Bugfixes

Always add changes to the build plan for the deploy stage

Details on Github

MPyL 1.5.0

Argocd

Add support for an argocd repository and workflow:

Can be toggled on by setting kubernetes.deployAction to 'KubectlManifest' in the mpyl_config.yaml file.
An argoRepository and argoGithub object have been added to the vcs section in the mpyl_config.yaml file to configure the argocd repository and github repository.
The manifest.yaml file will be created during the deploy step.
The command to push the k8s manifests to the argocd repository is mpyl build artifacts push --artifact-type argo.
An additional deployment.yaml file will be created in the push command, it contains some extra metadata for argocd.
A pull request will be created in the configured argocd repository with the created k8s manifest files.
The folder in the argocd repo structure has the following pattern: k8s-manifests/{project_name}/{target}/{namespace}/*.yaml.

Manual selection

Include project overrides in list of projects.

Multiple deploy targets

Allow multiple deploy targets when using the jenkins cli command.

Stage configuration

Example stage configuration

Details on Github

MPyL 1.4.20

Traefik priority rules per environment

You can now configure Traefik route priorities per environment:

deployment:
  traefik:
    hosts:
      - host:
          all: "Host(`host1.example.com`)"
          servicePort: 1234
          priority:
            all: 10
      - host:
          all: "Host(`host2.example.com`)"
          servicePort: 1235
          priority:
            pr: 20
            test: 30
            acceptance: 40
            production: 50

Details on Github

MPyL 1.4.19

ECR repo

New ECR repositories now allow for mutable image tags

Details on Github

MPyL 1.4.18

Project id

Project id is now a required property for non override project.yml's that have a deploy stage defined. It can/will be used for rbac purposes in rancher.

Deployment strategy

Make deployment strategy configurable in project.yml and mpyl_config.yml:

kubernetes:
  deploymentStrategy:
    rollingUpdate:
      maxUnavailable: "25%"
      maxSurge: "25%"
    type: "RollingUpdate"

Cache repo

Fix bug when pushing artifacts

ECR repo

Create ECR repository if it doesn't exist

Details on Github

MPyL 1.4.17

Bugfixes

Fix updating of pr body's that include "----"
Don't fail the Github pr body update function based on the mpyl run result
Fix loading of custom ExecutionException with pickle
Add a retry on the artifact caching push to avoid issues on parallel runs
Fix the cypress docker image

Details on Github

MPyL 1.4.16

Bugfixes

Fix non-cron jobs deployments

Details on Github

MPyL 1.4.15

Add AWS ECR

MPyL can now push/pull images to/from AWS ECR. Check mpyl_config file to set this.

Enhancements

Adds the optional hasSwagger field to enhance PR descriptions. This update ensures accurate URL display for services that do not use Swagger

Manual build improvements

Do not get changes from the remote repository when doing manual build

Cron job improvements

Allow cron jobs to be configured per environment

Details on Github

MPyL 1.4.14

Sealed Secrets and Dagster Deploy

The DagsterDeploy step now supports sealed secrets.
The Dagster UserCode helm chart is deployed with an extra manifest that contains the sealed secrets that are manually sealed and documented in the project.yml

Details on Github

MPyL 1.4.13

Ship typing information

MPyL can now be type checked via mypy.

Details on Github

MPyL 1.4.12

Enhancements

Several small code improvements
Improve sbt test step

Bug Fixes

Fix mpyl build clean command for override projects
Fix test results collection on compile errors for sbt test
Make sure sbt test coverage always gets turned off again

Details on Github

MPyL 1.4.11

Enhancements

Set resources for Dagster user code servers (hardcoded)

Cronjob deployment

Add timeZone field to job specification

CLI improvements

Add --dryrun flag

Details on Github

MPyL 1.4.10

Bug fixes

Fix tag builds by taking the tag env variable into account in the new logic of v1.4.9

Details on Github

MPyL 1.4.9

Bug fixes

Fix manual deployment bug where the build plan for the deploy stage wasn't taking the manual selection into account.

Details on Github

MPyL 1.4.8

Discovery

Add debug logging to build plan discovery methods This provides more explanation on _why_ certain projects are selected for each stage Can be invoked by setting the --verbose for the build subcommand, e.g. mpyl build --verbose status

Bug fixes

Add command and args fields to Kubernetes jobs
Fixes a bug when a non-changed project is selected whose base path includes fully a changed file's path. I.e. when the changed file is projects/project-name/src/main.py and a project's base path is projects/project-name-other, this other project was wrongly selected as changed.

Details on Github

MPyL 1.4.7

Spark Deployment

Configure replica count for spark jobs

Dagster deployment

Set DOCKER_IMAGE build argument to the full docker image path

Cloudfront Deploy

Use local docker image when running with --dry-run

Details on Github

MPyL 1.4.6

Enhancement

Shorten helm release names for dagster user deployment helm charts by using fullnameOverride
Cover more special cases for dagster user deployment helm charts by unittests

Linting

Enable extended checks by default
Fail command if one of the extended checks fail

Details on Github

MPyL 1.4.5

Override option for dagster user deployments' helm charts

Enable users to override the serviceAccount in the user deployment helm chart of dagster with a global serviceAccount

Bugfixes

Fix Spark Deploy Step: inject sealed secrets, set job argument correctly

Details on Github

MPyL 1.4.4

Bugfixes

Filter out override projects from the projects cli commands
Sort the projects names results alphabetically
Fix the fact that whitelists are being cached between charts/projects, thus subsequent charts/projects contain the whitelists from previous ones
Add default build_args for DockerBuild and DockerTest stages

Details on Github

MPyL 1.4.3

Manual project selection

Allows for passing a comma separated string of projects to be passed to the run cli, using the -p or --projects flags. This will override the default change detection behavior and the -all flag.

Traefik configuration

Create HTTP ingress routes that redirect to the HTTPS one
Add priority for routes
Add insecure option

Kubernetes configuration

Set both maintainer and maintainers fields in the metadata
Use “service” as the default name of containers

Bugfixes

Use the full image path when pulling images in CloudFront Kubernetes Deploy and Deploy From Docker Container

Details on Github

MPyL 1.4.2

Bugfixes

Fix on how the selected DockerRegistry is being opened when writing the values of a dagster user code helm chart
Fix ruamel.yaml.YAML() name overshadowing with the yaml package in the k8s module

Details on Github

MPyL 1.4.1

Bugfixes

Fix project overlays (merging children projects with their parent)
Github Check output only shows test summary, not the full test output
Get target branch in cache repo from the run properties
Only transition non-Epic Jira tickets to 'To Do'

Details on Github

MPyL 1.4.0

Customizable stages

Stages are now customizable. You can add the stages to the run_properties according to the defined schema, for example:

stages:
  - name: 'build'
    icon: '🏗️'
  - name: 'test'
    icon: '📋'

Support single stage runs

It is now possible to run a single stage. For example, to run only the build stage:

mpyl build run --stage build

If you want the results / report of the previous stage run to be combined with your current stage run, use the --sequential flag. Without this flag, the previous results will be overwritten. The results are stored in a local file in .mpyl using pickle, see mpyl-reporter.py for an example on how to use them.

Remote artifact caching

Remote caching can be used now to significantly speed up builds. The mechanisms are described in the documentation

Artifact caching

Is done by bookending your build commands with mpyl build artifacts push and mpyl build artifacts pop.

mpyl build artifacts pull
mpyl build run
mpyl build artifacts push --artifact-type cache

Docker image caching

Allows you to cache from docker images in the registry. This is particularly useful in scenarios where the local filesystem cannot be relied upon to persist between builds, such as in CI environments.

Implicit dependencies

If dependencies are defined for the build stage they now implicitly also apply for the test and deploy stages.

Support for project overlaying

The MPyL recognizes the project-override-*.yml files and merges them to the parent yml(project.yml) file in the same directory. Using this functionality, you can define different deployments for the same project. For example, you can deploy the same project with different settings to different environments.

Bugfixes

Fix TLS creation in ingress routes

Details on Github

MPyL 1.3.2

Bug Fixes

Fix the cypress kubectl config merging and passing to docker for linux environments
Fix jira ticket state switching to only switch from 'to do' to 'in progress'

Details on Github

MPyL 1.3.1

Target specific whitelists

Now have the possibility to specify target specific whitelisting rules. This means that for the same rule, we can apply different lists of IPs, depending on the target environment: Change in the mpyl_config.yaml file:

whiteLists:
  default: [ "VPN" ]
  addresses:
    - name: "VPN"
      all: [ "10.0.0.1" ]
    - name: 'Outside-World'
      all: [ '0.0.0.0/0' ]
    - name: 'K8s-Test'
      all: [ '1.2.3.0', '1.2.3.1' ]
    - name: 'TargetSpecificWhitelist'
      pr: ['1.2.3.4']
      test: ['1.2.3.4']
      acceptance: ['2.3.4.5']
      production: ['3.4.5.6']

Add support for various kubernetes resources

Add support for Role and RoleBinding resources
Be able to specify command and args in Container resources

Fix bug in the Cypress tests

Account for multiple config files being passed in the KUBECONFIG env var

Details on Github

MPyL 1.3.0

Support for multiple docker registries

You can now specify multiple docker registries in the config file. The docker field in the mpyl_config.yml now takes a list of registries:

docker:
  defaultRegistry: 'acme.docker.com'
  registries:
    - hostName: 'acme.docker.com'
      userName: !ENV ${DOCKER_REGISTRY_USR:docker_user}
      password: !ENV ${DOCKER_REGISTRY_PSW:docker_password}

which can be referenced in the project.yaml by the hostName field

docker:
  hostName: 'acme.docker.com'

Automatic config updates

Running mpyl health will now automatically update your config file with the latest version of the config file from the repo. This will allow you to get the latest changes to the config file without having to manually update it.

Details on Github

MPyL 1.2.1

Details on Github

MPyL 1.2.0

Automated project.yml upgrades

A new cli command mpyl projects upgrade allows you to upgrade your project.yml to the latest version. It will automatically add new sections and fields when necessary.

Future upgrade scripts should be added to mpyl.projects.versioning.

Kubernetes deploy actions

In mpyl_config.yaml the deploy action now needs to be explicitly set.

kubernetes:
  deployAction: HelmDeploy

There are four deploy actions available:

HelmDeploy - deploys the helm chart
HelmDryRun - runs a helm dry run against the cluster
HelmTemplate - renders a helm chart on the file system to the folder specified in the helmTemplateOutputPath property
KubectlManifest - renders the deployment as manifest file specified in the kubectlManifestOutputPath property. This manifest can be deployed with a kubectl -f <path> command.

Image pull secrets

Default image pull secrets now need to be configured globally in mpyl_config.yml

  deployment:
    kubernetes:
      imagePullSecrets:
        - name: 'acme-registry'

Details on Github

MPyL 1.1.0

Hotfix for mapping multiple ports to the same service

Due to a bug in the mapping of multiple ports to the same service, the following configuration:

deployment:
  kubernetes:
  portMappings:
    8081: 8081
  traefik:
    hosts:
      ...
      - host:
          all: "Host(`some.other.host.com`)"
        servicePort: 4091
        priority: 1000

resulted in 8081 being used as servicePort in the treafik rule instead of 4091.

Release notes

The release notes (as you are reading them now) are generated from the releases/notes directory in the project repository. Whenever a release has changes that require your attention like: new cli commands, new features, breaking changes, upgrade instructions, etc. they will be included here.

Create startup probes by default

When a project is using livenesProbes, but has no startupProbe defined, we resort to creating a startup probe from the default values defined in the mpyl_config.yml file. This is done to prevent the project from being marked as unhealthy.

Fix namespace interpolation in the Traefik default hosts

The default hosts for Traefik are now interpolated with the namespace of the project in test/acceptance/production.

Details on Github

MPyL 1.0.11

Retraction note

Due to a bug in the release process, the 1.0.11 had to be retracted. It is not available in the registry anymore and all changes have been subsumed into 1.0.12.

Details on Github

MPyL 1.0.10

Support for service monitor

The prometheus ServiceMonitor CRD and a corresponding PrometheusRule are deployed whenever the metrics field is defined in project.yml

Details on Github

MPyL 1.0.9

Support for reference environment variables

Support for reference environment variables. All standard types: secretkeyref, fieldRef and resourceFieldRef are support. This allows one to reference secrets, fields and resource fields from other resources in the same namespace.

Repo command

mpyl repo is new command group with the following subcommands:

status shows the status of the repository in terms of branch checked out, revision and revisions since branching off from base (main/master).
init allows you to initialize the local repository to prepare it for use with MPyL PR builds.

Details on Github

MPyL 1.0.8

Parallel execution of cypress tests is now supported, increasing performance on longer suites more than 2x.

Details on Github

MPyL 1.0.7

Step executors are discovered by a plugin mechanism. This allows for custom step executors to be added to the system without having to modify the core codebase. See the steps documentation

Details on Github

MPyL 1.0.6

Details on Github

MPyL 1.0.5

Details on Github

MPyL 1.0.4

Upload assets to S3 deploy step

Details on Github

MPyL 1.0.3

Details on Github

MPyL 1.0.2

Display build and ticket info in Github PR comment.

Details on Github

MPyL 1.0.1

mpyl build jenkins uses --follow by default, as it it's more instructive for first time use
mpyl build jenkins has --silent mode, in which we block until the Jenkins build is finished but filter out the logs
rename hidden .mpl folder to .mpyl
introduce possibility to filter documentation changes from invalidation logic

Details on Github

MPyL 1.0.0

First stable release

This release supports both PR and release/tag builds. MPyL now pulls in the main branch (to determine revision deltas) independently when necessary.

Details on Github

View Source

 1"""
 2.. include:: ../../README.md
 3
 4.. include:: ../../README-usage.md
 5
 6.. include:: ../../README-dev.md
 7
 8.. include:: ../../releases/README.md
 9"""
10import logging
11
12import click
13
14from .cli.build import build
15from .cli.health import health
16from .cli.meta_info import get_version
17from .cli.meta_info import version
18from .cli.projects import projects
19from .utilities.pyaml_env import parse_config
20from .utilities.repo import RepoConfig, Repository
21
22
23def _disable_package_loggers(offending_loggers: list[str]):
24    for name, _ in logging.root.manager.loggerDict.items():  # pylint: disable=no-member
25        for offending_logger in offending_loggers:
26            if name.startswith(offending_logger):
27                logging.getLogger(name).setLevel(logging.WARNING)
28
29
30@click.group(name="mpyl", help="Command Line Interface for MPyL")
31def main_group():
32    """Command Line Interface for MPyL"""
33
34
35def add_commands():
36    main_group.add_command(projects)
37    main_group.add_command(build)
38    main_group.add_command(version)
39    main_group.add_command(health)
40
41
42def main():
43    _disable_package_loggers(["markdown", "asyncio"])
44    add_commands()
45    main_group()  # pylint: disable = no-value-for-parameter

main_group = <Group mpyl>

Command Line Interface for MPyL

def add_commands(): View Source

36def add_commands():
37    main_group.add_command(projects)
38    main_group.add_command(build)
39    main_group.add_command(version)
40    main_group.add_command(health)

def main(): View Source

43def main():
44    _disable_package_loggers(["markdown", "asyncio"])
45    add_commands()
46    main_group()  # pylint: disable = no-value-for-parameter

mpyl

Modular Pypeline Library

What is MPyL?

🔖 Documentation

📚 Principles

💻 Technologies

Requirements

Bundled

CI/CD

Build

Testing

Deployment

Reporting

🖐️ Usage

..MPyL CLI

Suggested first time use

1. Install MPyL

2. Health check

3. Run a local build via the CLI

4. Run a CI build on your Pull Request

Command structure

MPyL configuration

Stage configuration

Auto completion

Bash

Zsh

Fish

YAML auto completion

Intellij IDEA or PyCharm

..defining projects

File structure

project.yml

..setting up a CI-CD flow

Github actions

Dagster

..caching build artifacts

Docker images

Artifacts

Remote caching

..report the outcome of a pipeline run

..create a custom step

..create a build step

Building a docker image

..create a test step

Junit test results

Integration tests

Developer instructions

..install mpyl for development

..run tests and checks

..create a pull request build

..code style

..create a new release

..troubleshoot Python setup

..installing a test release of MPyL

..running the mpyl sourcecode against another repository

Release notes

MPyL 1.8.7

Ensure correct artifact is used as required input

MPyL 1.8.6

Improve logging output

MPyL 1.8.5

Upgrade dependencies

MPyL 1.8.4

Remove validations on config load

MPyL 1.8.3

Rename Steps to Executor

MPyL 1.8.2

Pass changed files down to steps

MPyL 1.8.1

Upgrade to python 3.12

MPyL 1.8.0

Increased flexibility

Upgrade mypy

MPyL 1.7.6

Use pathlib to determine path of project during discovery

Upgrading rich and slack dependencies

MPyL 1.7.5

Demonstrate MPyL plugin mechanism.

MPyL 1.7.4

Add an ever-changing label to Dagster Helm deployments

Upgrading `rich` and `slack` dependencies