A beautiful GitOps day VII - Create a CI+CD workflow

part-08

Use GitOps workflow for building a production grade on-premise Kubernetes cluster on cheap VPS provider, with complete CI/CD 🎉

This is the Part VII of more global topic tutorial. Back to guide summary for intro.

Workflow

It’s now time to step back and think about how we’ll use our CI. Our goal is to build our above dotnet Web API with Concourse CI as a container image, ready to deploy to our cluster through Flux. So we finish the complete CI/CD pipeline. To resume the scenario:

Concourse CI check the Gitea repo periodically (pull model) for any new code and trigger a build if applicable
When container image build passed, Concourse CI push the new image to our private registry, which is already included into Gitea
Image Automation, which is a component as part of Flux, check the registry periodically (pull model), if new image tag detected, it will write the last tag into Flux repository
Flux check the flux GitHub registry periodically (pull model), if any new or updated manifest detected, it will deploy it automatically to our cluster

Note

Although it’s the most secured way and configuration less, instead of default pull model, which is generally a check every minute, it’s possible to use WebHook instead in order to reduce time between code push and deployment.

The flow pipeline is pretty straightforward:

CI Architecture

CI part

The credentials

We need to:

Give read/write access to our Gitea repo and container registry for Concourse. Note as we need write access in code repository for concourse because we need to store the new image tag. We’ll using semver resource for that.
Give read registry credentials to Flux for regular image tag checking as well as Kubernetes in order to allow image pulling from the private registry.

Let’s create 2 new user concourse with admin acces and container as standard user on Gitea. Store these credentials on new variables:

1
variable "concourse_git_username" {
2
  type = string
3
}
4

5
variable "concourse_git_password" {
6
  type      = string
7
  sensitive = true
8
}
9

10
variable "container_registry_username" {
11
  type = string
12
}
13

14
variable "container_registry_password" {
15
  type      = string
16
  sensitive = true
17
}

1
concourse_git_username      = "concourse"
2
concourse_git_password      = "xxx"
3
container_registry_username = "container"
4
container_registry_password = "xxx"

Apply the credentials for Concourse:

1
resource "kubernetes_secret_v1" "concourse_registry" {
2
  metadata {
3
    name      = "registry"
4
    namespace = "concourse-main"
5
  }
6

7
  data = {
8
    name     = "gitea.${var.domain}"
9
    username = var.concourse_git_username
10
    password = var.concourse_git_password
11
  }
12

13
  depends_on = [
14
    helm_release.concourse
15
  ]
16
}
17

18
resource "kubernetes_secret_v1" "concourse_git" {
19
  metadata {
20
    name      = "git"
21
    namespace = "concourse-main"
22
  }
23

24
  data = {
25
    url            = "https://gitea.${var.domain}"
26
    username       = var.concourse_git_username
27
    password       = var.concourse_git_password
28
    git-user       = "Concourse CI <concourse@kube.rocks>"
29
    commit-message = "bump to %version% [ci skip]"
30
  }
31

32
  depends_on = [
33
    helm_release.concourse
34
  ]
35
}

Note as we use concourse-main namespace, already created by Concourse Helm installer, which is a dedicated namespace for the default team main. Because of that, we should keep depends_on to ensure the namespace is created before the secrets.

Note

Don’t forget the [ci skip] in commit message, which is the commit for version bumping, otherwise you’ll have an infinite build loop !

Then same for Flux and the namespace that will receive the app:

1
resource "kubernetes_secret_v1" "image_pull_secrets" {
2
  for_each = toset(["flux-system", "kuberocks"])
3
  metadata {
4
    name      = "dockerconfigjson"
5
    namespace = each.value
6
  }
7

8
  type = "kubernetes.io/dockerconfigjson"
9

10
  data = {
11
    ".dockerconfigjson" = jsonencode({
12
      auths = {
13
        "gitea.${var.domain}" = {
14
          auth = base64encode("${var.container_registry_username}:${var.container_registry_password}")
15
        }
16
      }
17
    })
18
  }
19
}

Note

Create the namespace kuberocks first by k create namespace kuberocks, or you’ll get an error.

The Dockerfile

Now that all required credentials are in place, we have to tell Concourse how to check our repo and build our container image. This is done through a pipeline, which is a specific Concourse YAML file.

Firstly create following files in root of your repo that we’ll use for building a production ready container image:

1
**/bin/
2
**/obj/

1
FROM mcr.microsoft.com/dotnet/aspnet:8.0
2

3
WORKDIR /publish
4
COPY /publish .
5

6
EXPOSE 80
7
ENTRYPOINT ["dotnet", "KubeRocks.WebApi.dll"]

The pipeline

Let’s reuse our flux repository and create a file pipelines/demo.yaml with following content:

1
resources:
2
  - name: version
3
    type: semver
4
    source:
5
      driver: git
6
      uri: ((git.url))/kuberocks/demo
7
      branch: main
8
      file: version
9
      username: ((git.username))
10
      password: ((git.password))
11
      git_user: ((git.git-user))
12
      commit_message: ((git.commit-message))
13
  - name: source-code
14
    type: git
15
    icon: coffee
16
    source:
17
      uri: ((git.url))/kuberocks/demo
18
      branch: main
19
      username: ((git.username))
20
      password: ((git.password))
21
  - name: docker-image
22
    type: registry-image
23
    icon: docker
24
    source:
25
      repository: ((registry.name))/kuberocks/demo
26
      tag: latest
27
      username: ((registry.username))
28
      password: ((registry.password))
29

30
jobs:
31
  - name: build
32
    plan:
33
      - get: source-code
34
        trigger: true
35

36
      - task: build-source
37
        config:
38
          platform: linux
39
          image_resource:
40
            type: registry-image
41
            source:
42
              repository: mcr.microsoft.com/dotnet/sdk
43
              tag: "8.0"
44
          inputs:
45
            - name: source-code
46
              path: .
47
          outputs:
48
            - name: binaries
49
              path: publish
50
          caches:
51
            - path: /root/.nuget/packages
52
          run:
53
            path: /bin/sh
54
            args:
55
              - -ec
56
              - |
57
                dotnet format --verify-no-changes
58
                dotnet build -c Release
59
                dotnet publish src/KubeRocks.WebApi -c Release -o publish --no-restore --no-build
60

61
      - task: build-image
62
        privileged: true
63
        config:
64
          platform: linux
65
          image_resource:
66
            type: registry-image
67
            source:
68
              repository: concourse/oci-build-task
69
          inputs:
70
            - name: source-code
71
              path: .
72
            - name: binaries
73
              path: publish
74
          outputs:
75
            - name: image
76
          run:
77
            path: build
78
      - put: version
79
        params: { bump: patch }
80

81
      - put: docker-image
82
        params:
83
          additional_tags: version/number
84
          image: image/image.tar

A bit verbose compared to other CI, but it gets the job done. The price of maximum flexibility. Now in order to apply it we may need to install fly CLI tool. Just a matter of scoop install concourse-fly on Windows. Then:

# login to your Concourse instance
fly -t kuberocks login -c https://concourse.kube.rocks

# create the pipeline and active it
fly -t kuberocks set-pipeline -p demo -c pipelines/demo.yaml
fly -t kuberocks unpause-pipeline -p demo

A build will be trigger immediately. You can follow it on Concourse UI.

Concourse pipeline

If everything is ok, check in https://gitea.kube.rocks/admin/packages, you should see a new image appear on the list ! A new file version is automatically pushed in code repo in order to keep tracking of the image tag version.

Concourse build

Automatic pipeline update

If you don’t want to use fly CLI every time for any pipeline update, you maybe interested in set_pipeline feature. Create following file:

1
resources:
2
  - name: ci
3
    type: git
4
    icon: git
5
    source:
6
      uri: https://github.com/kuberocks/demo-kube-flux
7

8
jobs:
9
  - name: configure-pipelines
10
    plan:
11
      - get: ci
12
        trigger: true
13
      - set_pipeline: demo
14
        file: ci/pipelines/demo.yaml

Then apply it:

fly -t kuberocks set-pipeline -p main -c pipelines/main.yaml

Now you can manually trigger the pipeline, or wait for the next check, and it will update the demo pipeline automatically. If you’re using a private repo for your pipelines, you may need to add a new secret for the git credentials and set username and password accordingly.

You almost no need of fly CLI anymore, except for adding new pipelines ! You can even go further with set_pipeline: self which is always an experimental feature.

CD part

The deployment

If you followed the previous parts of this tutorial, you should have clue about how to deploy your app. Let’s create deploy it with Flux:

1
apiVersion: apps/v1
2
kind: Deployment
3
metadata:
4
  name: demo
5
  namespace: kuberocks
6
spec:
7
  replicas: 1
8
  selector:
9
    matchLabels:
10
      app: demo
11
  template:
12
    metadata:
13
      labels:
14
        app: demo
15
    spec:
16
      imagePullSecrets:
17
        - name: dockerconfigjson
18
      containers:
19
        - name: api
20
          image: gitea.kube.rocks/kuberocks/demo:latest
21
          ports:
22
            - containerPort: 80
23
---
24
apiVersion: v1
25
kind: Service
26
metadata:
27
  name: demo
28
  namespace: kuberocks
29
  labels:
30
    app: demo
31
spec:
32
  selector:
33
    app: demo
34
  ports:
35
    - name: http
36
      port: 80
37
---
38
apiVersion: traefik.io/v1alpha1
39
kind: IngressRoute
40
metadata:
41
  name: demo
42
  namespace: kuberocks
43
spec:
44
  entryPoints:
45
    - websecure
46
  routes:
47
    - match: Host(`demo.kube.rocks`)
48
      kind: Rule
49
      services:
50
        - name: demo
51
          port: http

Note as we have set imagePullSecrets in order to use fetch previously created credentials for private registry access. The rest is pretty straightforward. Once pushed, after about 1 minute, you should see your app deployed in https://demo.kube.rocks. Check the API response on https://demo.kube.rocks/WeatherForecast.

However, one last thing is missing: the automatic deployment.

Image automation

If you checked the above flowchart, you’ll note that Image automation is a separate process from Flux that only scan the registry for new image tags and push any new tag to Flux repository. Then Flux will detect the new commit in Git repository, including the new tag, and automatically deploy it to K8s.

By default, if not any strategy is set, K8s will do a rolling deployment, i.e. creating new replica firstly before terminating the old one. This will prevent any downtime on the condition of you set as well readiness probe in your pod spec, which is a later topic.

Let’s define the image update automation task for main Flux repository:

1
apiVersion: image.toolkit.fluxcd.io/v1beta2
2
kind: ImageUpdateAutomation
3
metadata:
4
  name: flux-system
5
  namespace: flux-system
6
spec:
7
  interval: 1m0s
8
  sourceRef:
9
    kind: GitRepository
10
    name: flux-system
11
  git:
12
    checkout:
13
      ref:
14
        branch: main
15
    commit:
16
      author:
17
        email: fluxcdbot@kube.rocks
18
        name: fluxcdbot
19
      messageTemplate: "{{range .Updated.Images}}{{println .}}{{end}}"
20
    push:
21
      branch: main
22
  update:
23
    path: ./clusters/demo
24
    strategy: Setters

Now we need to tell Image Reflector how to scan the repository, as well as the attached policy for tag update:

1
apiVersion: image.toolkit.fluxcd.io/v1beta2
2
kind: ImageRepository
3
metadata:
4
  name: demo
5
  namespace: flux-system
6
spec:
7
  image: gitea.kube.rocks/kuberocks/demo
8
  interval: 1m0s
9
  secretRef:
10
    name: dockerconfigjson
11
---
12
apiVersion: image.toolkit.fluxcd.io/v1beta2
13
kind: ImagePolicy
14
metadata:
15
  name: demo
16
  namespace: flux-system
17
spec:
18
  imageRepositoryRef:
19
    name: demo
20
    namespace: flux-system
21
  policy:
22
    semver:
23
      range: 0.0.x

Note

As usual, don’t forget dockerconfigjson for private registry access.

And finally edit the deployment to use the policy by adding a specific marker next to the image tag:

1
# ...
2
      containers:
3
        - name: api
4
          image: gitea.kube.rocks/kuberocks/demo:latest # {"$imagepolicy": "flux-system:demo"}
5
# ...

It will tell to Image Automation where to update the tag in the Flux repository. The format is {"$imagepolicy": "<policy-namespace>:<policy-name>"}.

Push the changes and wait for about 1 minute then pull the flux repo. You should see a new commit coming and latest should be replaced by an explicit tag like so:

1
# ...
2
      containers:
3
        - name: api
4
          image: gitea.kube.rocks/kuberocks/demo:0.0.1 # {"$imagepolicy": "flux-system:demo"}
5
# ...

Check if the pod as been correctly updated with kgpo -n kuberocks. Use kd -n kuberocks deploy/demo to check if the same tag is here and no latest.

1
Pod Template:
2
  Labels:  app=demo
3
  Containers:
4
   api:
5
    Image:        gitea.kube.rocks/kuberocks/demo:0.0.1
6
    Port:         80/TCP

Retest all workflow

Damn, I think we’re done 🎉 ! It’s time retest the full process. Add new controller endpoint from our demo project and push the code:

1
//...
2
public class WeatherForecastController : ControllerBase
3
{
4
    //...
5

6
    [HttpGet("{id}", Name = "GetWeatherForecastById")]
7
    public WeatherForecast GetById(int id)
8
    {
9
        return new WeatherForecast
10
        {
11
            Date = DateOnly.FromDateTime(DateTime.Now.AddDays(id)),
12
            TemperatureC = Random.Shared.Next(-20, 55),
13
            Summary = Summaries[Random.Shared.Next(Summaries.Length)]
14
        };
15
    }
16
}

Wait the pod to be updated, then check the new endpoint https://demo.kube.rocks/WeatherForecast/1. The API should return a new unique random weather forecast with the tomorrow date.

7th check ✅

We have done for the set-up of our automated CI/CD workflow process. Go next part for going further with a real DB app that handle automatic migrations.