mpyl.steps

  1"""
  2## What is a Step?
  3A `Step` is a unit of execution within a pipeline run. Each step relates to one particular `mpyl.project.Stage`.
  4For example, there are some built in steps in the `Build` stage like `mpyl.steps.build.dockerbuild.BuildDocker` or
  5`mpyl.steps.build.sbt.BuildSbt`. Both of these steps compile and assemble source code into a bundle that is embedded
  6in a docker image.
  7
  8
  9## How do I create my own custom step?
 10
 11### Before you start
 12A step can either be *built in* to `MPyL` or defined within the context of a particular pipeline. The latter is easier
 13to do and is the best option if the likelihood of the logic being of use in other contexts is small.
 14
 15.. note:: The benefits of standard artifact types
 16   It is recommended to have your steps produce any of the standard artifact types. This allows you to benefit from
 17   the tooling embedded in MPyL to process them. For example: if a step in the `Test` stage produces an artifact of type
 18   `ArtifactType.JUNIT_TESTS`, the test results can be reported in a unified way by any of the `mpyl.reporting`
 19   reporters regardless of whether the tests were run by a `jest` or a `scala` test.
 20
 21
 22### Implementation
 23
 24Let's assume you want to create a new step for the `Build` stage that creates a builds a `Java` maven project and embeds
 25it a docker image.
 26
 27Implement the `Step` interface as `BuildJava(Step)`, or however you want to call it. For an example you could have
 28 a look at the `mpyl.steps.build.echo.BuildEcho`.
 29
 30##### Important constructor properties
 31 - `Meta.name` is how you can refer to this step from a `project.yml`. The `mpyl.steps.executor.Steps` executor will
 32 pick up the `Step` implementation that has the name described in `stages.build` to execute the build step.
 33```yaml
 34name: 'javaService'
 35stages:
 36    build: 'Java Build'
 37description: 'A simple Java service'
 38```
 39 -  `Meta.stage` describes the `mpyl.project.Stage to which the step relates. It can only be executed in this context.
 40 -  `Step.produced_artifact` defines the `mpyl.steps.models.ArtifactType` that this step produces. In our example case
 41 this would be `mpyl.steps.models.ArtifactType.DOCKER_IMAGE`.
 42 - The `Step.after` is a postprocessing step, which we can set to `mpyl.steps.build.docker_after_build.AfterBuildDocker`
 43 in this case. It will push the image produced by this step to a registry.
 44
 45##### Return type
 46
 47Your step needs to return an `mpyl.steps.models.Output` object with fields that are hopefully self-explanatory.
 48The `produced_artifact` can be constructed with `mpyl.steps.models.input_to_artifact`. The `spec` parameter is for
 49specifying `ArtifactType` specific metadata, that can be picked up by another step that has `ArtifactType.DOCKER_IMAGE`
 50as `required_artifact`, like `mpyl.steps.deploy.kubernetes.DeployKubernetes` for example.
 51```python
 52input_to_artifact(ArtifactType.DOCKER_IMAGE, step_input, spec=DockerImageSpec(image=image_tag)
 53```
 54
 55##### Step input
 56The step receives an `mpyl.steps.models.Input` for `execute`. If your step needs configuration settings, like for
 57example `mpyl.utilities.docker.DockerConfig`, this can be constructed from the `mpyl.steps.models.RunProperties.config`
 58dictionary on `mpyl.steps.models.Input.run_properties`.
 59Make sure to update the schema under `src/mpyl/schema/mpyl_config.schema.yml` accordingly, so that the configuration
 60remains type safe and mistakes are found as early as possible.
 61
 62##### Registration with the executor
 63Importing the module in which your step is defined is enough to register it.
 64Steps are automatically registered with the `mpyl.steps.executor.Steps` executor via the `IPluginRegistry` metaclass.
 65
 66Example:
 67```python
 68.. include:: ../../../plugin-run.py
 69```
 70
 71"""
 72from __future__ import annotations
 73
 74from dataclasses import dataclass
 75from logging import Logger
 76from typing import Optional, List
 77
 78from .models import ArtifactType, Input, Output
 79from ..project import Stage
 80
 81
 82class IPluginRegistry(type):
 83    plugins: List[type] = []
 84
 85    def __init__(cls, name, _bases, _attrs):
 86        super().__init__(cls)
 87        if name != "Step":
 88            IPluginRegistry.plugins.append(cls)
 89
 90
 91@dataclass(frozen=True)
 92class Meta:
 93    name: str
 94    """External, unique identifier. The step can be referred to by this name from `project.yml`"""
 95    description: str
 96    stage: str
 97    version: str = "0.0.1"
 98    """The stage that this step relates to"""
 99
100    def __str__(self) -> str:
101        return f"{self.name}: {self.version}"
102
103
104class Step(metaclass=IPluginRegistry):
105    """Abstract base class for execution steps. Any execution step (e.g. build, test, deploy) will need to implement
106    this interface.
107    """
108
109    meta: Meta
110    """Information _about_ the specific instance of `Step`. For example its name, description, version or the stage
111    to which it applies.
112    """
113    produced_artifact: ArtifactType
114    """The type of the artifact produced by this step """
115    required_artifact: ArtifactType
116    """Is set to something other than `ArtifactType.NONE` if this step depends on an artifact produced by the execution
117    of an earlier step. For example: a step in the `Deploy` stage, may need to deploy a docker image that was produced
118    in the `Build` stage."""
119    before: Optional[Step]
120    after: Optional[Step]
121    """Will be executed after completion of this step. Can be used for shared post processing steps, like pushing the
122    produced docker image to a registry or filing test results."""
123
124    def __init__(
125        self,
126        logger: Logger,
127        meta: Meta,
128        produced_artifact: ArtifactType,
129        required_artifact: ArtifactType,
130        before: Optional[Step] = None,
131        after: Optional[Step] = None,
132    ) -> None:
133        self._logger = logger
134        self.meta = meta
135        self.produced_artifact = produced_artifact
136        self.required_artifact = required_artifact
137        self.before = before
138        self.after = after
139
140    def execute(self, step_input: Input) -> Output:
141        """Execute an individual step for a specific `project` at a specific `stage` of the pipeline.
142        :param step_input: The input of the project along with its build properties and required artifact (if any).
143        :return Output: The result of the execution. `success` will be `False` if any exception was thrown during
144        execution.
145        """
146        return Output(
147            success=False,
148            message=f"Not implemented for {step_input.project_execution.name}",
149            produced_artifact=None,
150        )