Define a simple CD pipeline with Knative

In this blog post I will describe how to setup a simple CD pipeline using Knative pipeline. The pipeline takes a helm chart from a git repository and performs the following steps:

  • it builds three docker images using Kaniko
  • it pushes the built images to a private container registry
  • it deploys the chart against a kubernetes cluster

I used IBM Cloud, both the container service IKS as well as the IBM Container Registry, to host Knative as well as the deployed Health app. I used the same IKS kubernetes cluster to run the Knative service as well as the deployed application. They are separated using dedicated namespaces, service accounts and roles. For details about setting up Knative on IBM Cloud checkout my previous blog post. The whole code is available on GitHub.

Knative Pipelines

Pipelines are the newest addition to the Knative project, which already included three components: serving, eventing and build. Quoting from the official README, “The Pipeline CRD provides k8s-style resources for declaring CI/CD-style pipelines”. The pipeline CRD is meant as a replacement for the build CRD. The build-pipeline project introduces a few new custom resource definitions (CRDs) to extend the Kubernetes API:

  • tasks
  • tasksrun
  • pipeline
  • pipelinerun
  • pipelineresource

A pipeline is made of tasks; tasks can have input and output pipelineresources. The output of a task can be the input of another one. A taskrun is used run a single task. It binds the task inputs and outputs to specific pipelineresources. Similarly, a pipelinerun is used to run a pipeline. It binds the pipeline inputs and outputs to specific pipelineresources. For more details on Knative pipeline CRDs, see the official project documentation.

The “Health” application

The application deployed by the helm chart is OpenStack Health, or simply “Health”, a dashboard to visualize CI test results. The chart deploys three components:

  • An SQL backend that uses the postgres:alpine docker image, plus a custom database image that runs database migrations through an init container
  • A python based API server, which exposes the data in the SQL database via HTTP based API
  • A javascript frontend, which interacts on client side with the HTTP API
OpenStack Health
The OpenStack Health Dashboard

The Continuous Delivery Pipeline

The pipeline for “Health” uses two different tasks and several type of pipelineresources: git, image and cluster. In the diagram below circles are resources, boxes are tasks:

cd pipeline
CD Pipeline

At the moment of writing the execution of tasks is strictly sequential; the order of execution is that specified in the pipeline. The plan is to use inputs and outputs to define task dependencies and thus a graph of execution, which would allow for independent tasks to run in parallel.

Source to Image (Task)

The docker files and the helm chart for “Health” are hosted in the same git repository. The git resource is thus input to the both the source-to-image task, for the docker images, as well as the helm-deploy one, for the helm char.

When a resource of type git is used as an input to a task, the knative controller clones the git repository and prepares it at the specified revision, ready for the task to use it. The source-to-image task uses Kaniko to build the specified Dockerfile and to pushes the resulting container image to the container registry.

The destination URL of the image is defined in the image output resource, and then pulled into the args passed to the Kaniko container. At the moment of writing the image output resource is only used to hold the target URL. In future Knative will enforce that every task that defines image as an output actually produces the container images and pushes it to the expected location, it will also enrich the resource metadata with the digest of the pushed image, for consuming tasks to use. The health chart includes three docker images; identically three image pipeline resources are required. They are very similar to each other in their definition, only the target URL changes. One of the three:

Helm Deploy (Task)

The three source-to-image tasks build and push three images to the container registry. They don’t report the digest of the image, however, they associate the latest tag to that digest, which allows the helm-deploy task to pull the right images, assuming only one CD pipeline runs at the time. The helm-deploy has five inputs: one git resource to get the helm chart, three image resources that correspond to three docker files and finally a cluster resource, that gives the task access to a kubernetes cluster where to deploy the helm chart to:

The resource name highlighted on line 4 must match the cluster name. The username is that of a service account that has enough rights to deploy helm to the cluster. For isolation, I defined a namespace in the target cluster called health. Both helm itself, as well as the health chart, are deployed in that namespace only. Helm runs with a service account health-admin that can only access the health namespace. The manifests required to set up the namespace, the service accounts, the roles and role bindings are available on GitHub. The health-admin service account token and the cluser CA certificate should not be stored in git. They are defined in a secret, which can be setup using the following template:

The template can be filled in with a simple bash script. The script works after a successful login was performed via ibmcloud login and ibmcloud target.

Once a cluster resource is used as input to a task, it generates a kubeconfig file that can be used in the task to perform actions against the cluster. The helm-deploy task takes the image URLs from the image input resources and passes them to the helm command as set values; it also overrides the image tags to latest. The image pull policy is set to always since the tag doesn’t change, but the image does. The upgrade --install command is required so that both the first as well as following deploys may work.

The location of the generated kubeconfig file is passed to the alpine/helm container via the KUBECONFIG environment variable passed on the task container. The path where the file is generated is /workspace/[name of cluster resource/kubeconfig], so it ultimately depends on the name of the target cluster.

The Pipeline

The pipeline defines the sequence of tasks that will be executed, with their inputs and outputs. The from syntax can be used to express dependencies between tasks, i.e. input of a tasks is taken from the output of another one.

Tasks can accept parameters, which are specified as part of the pipeline definition. One example in the pipeline above is the ingress domain of the target kubernetes cluster, which is used by the helm chart to set up of ingress of the deployed application. Pipelines can also accept paramters, which are specified as part of the pipelinerun definition. Parameters make it possible to keep environment and run specific values confined to pipelinerun and pipelineresource. You may notice that the pipeline includes an extra task helm-init which is invoked before helm-deploy. As the name suggests, the task initializes helm in the target cluster/namespace. Tasks can have multiple steps, so that could be implemented as a first step within the helm-deploy task. However, I wanted to keep it separated so that helm-init runs using a service account with an admin role in the target namespace, while helm-deploy runs with an unprivileged account.

Running the Pipeline

To run a pipeline, a pipelinerun is needed. The pipelinerun binds the tasks inputs and outputs to specific pipelineresources and defines the trigger for the run. At the momement of writing only manual trigger is supported. Since there is no native mechanism available to create a pipelinerun from a template, running a pipeline again requires changing the pipelinerun YAML manifest and applying it to the cluster again. When executed, the pipelinerun controller automatically creates a taskrun when a task is executed. Any kubernetes resources created during the run, such as taskruns, pods and pvcs, stays once the pipelinerun execution is complete; they are cleaned up only when the pipelinerun is deleted.

To execute the pipeline, all static resources must be created first, and then the pipeline can be executed. Using the code from the GitHub repo:

A successful execution of the pipeline will create the following resources in the health namespace:

All Knative pipeline resources are visible in the default namepace:

To check if the “Health” application is running, you can hit the API using curl:

You can point your browser to ${HEALTH_URL}/health-health/# to see the frontend.

Conclusions

Even if the project only started towards the end of 2018, it is already possible to run a full CD pipeline for a relatively complex helm chart. It took me only a few small PRs to get everything working fine. There are several limitations to be aware of, however, the team is well aware of them and working quickly towards their resolution. Some examples:

  • Tasks can only be executed sequentially
  • Images as output resources are not really implemented yet
  • Triggers are only manual

There’s plenty of space for contributions, in the form of bugs, documentation, code or even simply sharing your experience with knative pipelines with the community. A good place to start are the GitHub issues marked as help wanted. All the code I wrote for this is available on GitHub under Apache 2.0, feel free to re-use any of it, feedback and PRs are welcome.

Caveats

The build-pipeline project is rather new, at the moment of writing the community is working towards its first release. Part of the API may still be subject to backward incompatible changes and examples in this blog post may stop working eventually. See the API compatibility policy for more details.

One thought on “Define a simple CD pipeline with Knative

Leave a Reply