A beautiful GitOps day IV - CD with Flux

part-05

Use GitOps workflow for building a production grade on-premise Kubernetes cluster on cheap VPS provider, with complete CI/CD 🎉

This is the Part IV of more global topic tutorial. Back to guide summary for intro.

Flux

In GitOps world, 2 tools are leading for CD in k8s: Flux and ArgoCD. As Flux is CLI first and more lightweight, it’s my personal goto. You may wonder why don’t continue with actual k3s Terraform project ?

You already noted that by adding more and more Helm dependencies to terraform, the plan time is increasing, as well as the state file. So not very scalable.

It’s the perfect moment to draw a clear line between IaC (Infrastructure as Code) and CD (Continuous Delivery). IaC is for infrastructure, CD is for application. So to resume our GitOps stack:

IaC for Hcloud cluster initialization (the basement): Terraform
IaC for Kubernetes configuration (the walls): Helm through Terraform
CD for any application deployments (the furniture): Flux

Note

You can probably eliminate with some efforts the 2nd stack by using both Kube-Hetzner, which take care of ingress and storage, and using Flux directly for the remaining helms like database cluster. Or maybe you can also add custom helms to Kube-Hetzner.
But as it’s increase complexity and dependencies problem, I prefer personally to keep a clear separation between the middle part and the rest, as it’s more straightforward for me. Just a matter of taste 🥮

Flux bootstrap

Create a dedicated Git repository for Flux somewhere, I’m using GitHub, which with his CLI is just a matter of:

gh repo create demo-kube-flux --private --add-readme
gh repo clone demo-kube-flux

Note

Put --add-readme option to have a non-empty repo, otherwise Flux bootstrap will give you an error.

Let’s back to demo-kube-k3s terraform project and add Flux bootstrap connected to above repository:

1
terraform {
2
  //...
3

4
  required_providers {
5
    flux = {
6
      source = "fluxcd/flux"
7
    }
8
    github = {
9
      source = "integrations/github"
10
    }
11
  }
12
}
13

14
//...
15

16
variable "github_token" {
17
  sensitive = true
18
  type      = string
19
}
20

21
variable "github_org" {
22
  type = string
23
}
24

25
variable "github_repository" {
26
  type = string
27
}

1
github_org           = "mykuberocks"
2
github_repository    = "demo-kube-flux"
3
github_token         = "xxx"

Note

Create a Github token with repo permissions and add it to github_token variable.

1
provider "github" {
2
  owner = var.github_org
3
  token = var.github_token
4
}
5

6
resource "tls_private_key" "flux" {
7
  algorithm   = "ECDSA"
8
  ecdsa_curve = "P256"
9
}
10

11
resource "github_repository_deploy_key" "this" {
12
  title      = "Flux"
13
  repository = var.github_repository
14
  key        = tls_private_key.flux.public_key_openssh
15
  read_only  = false
16
}
17

18
provider "flux" {
19
  kubernetes = {
20
    config_path = "~/.kube/config"
21
  }
22
  git = {
23
    url = "ssh://git@github.com/${var.github_org}/${var.github_repository}.git"
24
    ssh = {
25
      username    = "git"
26
      private_key = tls_private_key.flux.private_key_pem
27
    }
28
  }
29
}
30

31
resource "flux_bootstrap_git" "this" {
32
  path               = "clusters/demo"
33
  embedded_manifests = true
34

35
  components_extra = [
36
    "image-reflector-controller",
37
    "image-automation-controller"
38
  ]
39

40
  depends_on = [github_repository_deploy_key.this]
41
}

Note as we’ll use components_extra to add image-reflector-controller and image-automation-controller to Flux, as it will serve us later for new image tag detection.

After applying this, use kg deploy -n flux-system to check that Flux is correctly installed and running.

Managing secrets

As always with GitOps, a secured secrets management is critical. Nobody wants to expose sensitive data in a git repository. An easy to go solution is to use Bitnami Sealed Secrets, which will deploy a dedicated controller in your cluster that will automatically decrypt sealed secrets.

Open demo-kube-flux project and create helm deployment for sealed secret.

1
---
2
apiVersion: source.toolkit.fluxcd.io/v1
3
kind: HelmRepository
4
metadata:
5
  name: sealed-secrets
6
  namespace: flux-system
7
spec:
8
  interval: 1h0m0s
9
  url: https://bitnami-labs.github.io/sealed-secrets
10
---
11
apiVersion: helm.toolkit.fluxcd.io/v2
12
kind: HelmRelease
13
metadata:
14
  name: sealed-secrets
15
  namespace: flux-system
16
spec:
17
  chart:
18
    spec:
19
      chart: sealed-secrets
20
      reconcileStrategy: ChartVersion
21
      sourceRef:
22
        kind: HelmRepository
23
        name: sealed-secrets
24
      version: ">=2.12.0"
25
  interval: 1m
26
  releaseName: sealed-secrets-controller
27
  targetNamespace: flux-system
28
  install:
29
    crds: Create
30
  upgrade:
31
    crds: CreateReplace

Note

Don’t touch manifests under flux-system folder, as it’s managed by Flux itself and overload on each flux bootstrap.

Then push it and check that sealed secret controller is correctly deployed with kg deploy sealed-secrets-controller -n flux-system.

Private key is automatically generated, so last step is to fetch the public key. Type this in project root to include it in your git repository:

kpf svc/sealed-secrets-controller -n flux-system 8080
curl http://localhost:8080/v1/cert.pem > pub-sealed-secrets.pem

Note

By the way install the client with brew install kubeseal (Mac / Linux) or scoop install kubeseal (Windows).

Install some tools

It’s now finally time to install some tools to help us in our CD journey.

pgAdmin

A 1st good example is typically pgAdmin, which is a web UI for Postgres. We’ll use it to manage our database cluster. It requires a local PVC to store its data user and settings.

1
apiVersion: apps/v1
2
kind: Deployment
3
metadata:
4
  name: pgadmin
5
  namespace: postgres
6
spec:
7
  strategy:
8
    type: Recreate
9
  selector:
10
    matchLabels:
11
      app: pgadmin
12
  template:
13
    metadata:
14
      labels:
15
        app: pgadmin
16
    spec:
17
      securityContext:
18
        runAsUser: 5050
19
        runAsGroup: 5050
20
        fsGroup: 5050
21
        fsGroupChangePolicy: "OnRootMismatch"
22
      containers:
23
        - name: pgadmin
24
          image: dpage/pgadmin4:latest
25
          ports:
26
            - containerPort: 80
27
          env:
28
            - name: PGADMIN_DEFAULT_EMAIL
29
              valueFrom:
30
                secretKeyRef:
31
                  name: pgadmin-auth
32
                  key: default-email
33
            - name: PGADMIN_DEFAULT_PASSWORD
34
              valueFrom:
35
                secretKeyRef:
36
                  name: pgadmin-auth
37
                  key: default-password
38
          volumeMounts:
39
            - name: pgadmin-data
40
              mountPath: /var/lib/pgadmin
41
      volumes:
42
        - name: pgadmin-data
43
          persistentVolumeClaim:
44
            claimName: pgadmin-data
45
---
46
apiVersion: v1
47
kind: PersistentVolumeClaim
48
metadata:
49
  name: pgadmin-data
50
  namespace: postgres
51
spec:
52
  resources:
53
    requests:
54
      storage: 128Mi
55
  volumeMode: Filesystem
56
  storageClassName: longhorn
57
  accessModes:
58
    - ReadWriteOnce
59
---
60
apiVersion: v1
61
kind: Service
62
metadata:
63
  name: pgadmin
64
  namespace: postgres
65
spec:
66
  selector:
67
    app: pgadmin
68
  ports:
69
    - port: 80
70
---
71
apiVersion: traefik.io/v1alpha1
72
kind: IngressRoute
73
metadata:
74
  name: pgadmin
75
  namespace: postgres
76
spec:
77
  entryPoints:
78
    - websecure
79
  routes:
80
    - match: Host(`pgadmin.kube.rocks`)
81
      kind: Rule
82
      middlewares:
83
        - name: middleware-ip
84
          namespace: traefik
85
      services:
86
        - name: pgadmin
87
          port: 80

Here are the secrets to adapt to your needs:

1
apiVersion: v1
2
kind: Secret
3
metadata:
4
  name: pgadmin-auth
5
  namespace: postgres
6
type: Opaque
7
data:
8
  default-email: YWRtaW5Aa3ViZS5yb2Nrcw==
9
  default-password: YWRtaW4=

Now be sure to encrypt it with kubeseal and remove original file:

cat clusters/demo/postgres/secret-pgadmin.yaml | kubeseal --format=yaml --cert=pub-sealed-secrets.pem > clusters/demo/postgres/sealed-secret-pgadmin.yaml
rm clusters/demo/postgres/secret-pgadmin.yaml

Note

Don’t forget to remove the original secret file before commit for obvious reason ! If too late, consider password leaked and regenerate a new one.
You may use VSCode extension

Push it and wait a minute, and go to pgadmin.kube.rocks and login with chosen credentials. Now try to register a new server with postgresql-primary.postgres as hostname, and the rest with your PostgreSQL credential on previous installation. It should work !

Note

If you won’t wait each time after code push, do flux reconcile kustomization flux-system --with-source (require flux-cli). It also allows easy debugging by printing any syntax error in your manifests.

You can test the read replica too by register a new server using the hostname postgresql-read.postgres. Try to do some update on primary and check that it’s replicated on read replica. Any modification on replicas should be rejected as it’s on transaction read only mode.

It’s time to use some useful apps.

n8n

Let’s try some app that require a bit more configuration and real database connection with n8n, a workflow automation tool.

1
apiVersion: v1
2
kind: Namespace
3
metadata:
4
  name: n8n
5
---
6
apiVersion: apps/v1
7
kind: Deployment
8
metadata:
9
  name: n8n
10
  namespace: n8n
11
spec:
12
  strategy:
13
    type: Recreate
14
  selector:
15
    matchLabels:
16
      app: n8n
17
  template:
18
    metadata:
19
      labels:
20
        app: n8n
21
    spec:
22
      containers:
23
        - name: n8n
24
          image: n8nio/n8n:latest
25
          ports:
26
            - containerPort: 5678
27
          env:
28
            - name: N8N_PROTOCOL
29
              value: https
30
            - name: N8N_HOST
31
              value: n8n.kube.rocks
32
            - name: N8N_PORT
33
              value: "5678"
34
            - name: NODE_ENV
35
              value: production
36
            - name: WEBHOOK_URL
37
              value: https://n8n.kube.rocks/
38
            - name: DB_TYPE
39
              value: postgresdb
40
            - name: DB_POSTGRESDB_DATABASE
41
              value: n8n
42
            - name: DB_POSTGRESDB_HOST
43
              value: postgresql-primary.postgres
44
            - name: DB_POSTGRESDB_USER
45
              value: n8n
46
            - name: DB_POSTGRESDB_PASSWORD
47
              valueFrom:
48
                secretKeyRef:
49
                  name: n8n-db
50
                  key: password
51
            - name: N8N_EMAIL_MODE
52
              value: smtp
53
            - name: N8N_SMTP_HOST
54
              value: smtp.mailgun.org
55
            - name: N8N_SMTP_PORT
56
              value: "587"
57
            - name: N8N_SMTP_USER
58
              valueFrom:
59
                secretKeyRef:
60
                  name: n8n-smtp
61
                  key: user
62
            - name: N8N_SMTP_PASS
63
              valueFrom:
64
                secretKeyRef:
65
                  name: n8n-smtp
66
                  key: password
67
            - name: N8N_SMTP_SENDER
68
              value: n8n@kube.rocks
69
          volumeMounts:
70
            - name: n8n-data
71
              mountPath: /home/node/.n8n
72
            - name: n8n-cache
73
              mountPath: /home/node/.cache
74
      volumes:
75
        - name: n8n-data
76
          persistentVolumeClaim:
77
            claimName: n8n-data
78
        - name: n8n-cache
79
          emptyDir: {}
80
---
81
apiVersion: v1
82
kind: PersistentVolumeClaim
83
metadata:
84
  name: n8n-data
85
  namespace: n8n
86
spec:
87
  resources:
88
    requests:
89
      storage: 1Gi
90
  volumeMode: Filesystem
91
  storageClassName: longhorn
92
  accessModes:
93
    - ReadWriteOnce
94
---
95
apiVersion: v1
96
kind: Service
97
metadata:
98
  name: n8n
99
  namespace: n8n
100
  labels:
101
    app: n8n
102
spec:
103
  selector:
104
    app: n8n
105
  ports:
106
    - port: 5678
107
---
108
apiVersion: traefik.io/v1alpha1
109
kind: IngressRoute
110
metadata:
111
  name: n8n
112
  namespace: n8n
113
spec:
114
  entryPoints:
115
    - websecure
116
  routes:
117
    - match: Host(`n8n.kube.rocks`)
118
      kind: Rule
119
      services:
120
        - name: n8n
121
          port: 5678

Here are the secrets to adapt to your needs:

1
apiVersion: v1
2
kind: Secret
3
metadata:
4
  name: n8n-db
5
  namespace: n8n
6
type: Opaque
7
data:
8
  password: YWRtaW4=

1
apiVersion: v1
2
kind: Secret
3
metadata:
4
  name: n8n-smtp
5
  namespace: n8n
6
type: Opaque
7
data:
8
  user: YWRtaW4=
9
  password: YWRtaW4=

While writing these secrets, create n8n DB and set n8n user with proper credentials as owner.

Then don’t forget to seal secrets and remove original files the same way as pgAdmin. Once pushed, n8n should be deploying, automatically migrate the db, and soon after n8n.kube.rocks should be available, allowing you to create your 1st account.

NocoDB

Let’s try a final candidate with NocoDB, an Airtable-like generator for Postgres. It’s very similar to n8n.

1
apiVersion: v1
2
kind: Namespace
3
metadata:
4
  name: nocodb
5
---
6
apiVersion: apps/v1
7
kind: Deployment
8
metadata:
9
  name: nocodb
10
  namespace: nocodb
11
spec:
12
  strategy:
13
    type: Recreate
14
  selector:
15
    matchLabels:
16
      app: nocodb
17
  template:
18
    metadata:
19
      labels:
20
        app: nocodb
21
    spec:
22
      containers:
23
        - name: nocodb
24
          image: nocodb/nocodb:latest
25
          ports:
26
            - containerPort: 8080
27
          env:
28
            - name: DB_PASSWORD
29
              valueFrom:
30
                secretKeyRef:
31
                  name: nocodb-db
32
                  key: password
33
            - name: DATABASE_URL
34
              value: postgresql://nocodb:$(DB_PASSWORD)@postgresql-primary.postgres/nocodb
35
            - name: NC_AUTH_JWT_SECRET
36
              valueFrom:
37
                secretKeyRef:
38
                  name: nocodb-auth
39
                  key: jwt-secret
40
            - name: NC_SMTP_HOST
41
              value: smtp.mailgun.org
42
            - name: NC_SMTP_PORT
43
              value: "587"
44
            - name: NC_SMTP_USERNAME
45
              valueFrom:
46
                secretKeyRef:
47
                  name: nocodb-smtp
48
                  key: user
49
            - name: NC_SMTP_PASSWORD
50
              valueFrom:
51
                secretKeyRef:
52
                  name: nocodb-smtp
53
                  key: password
54
            - name: NC_SMTP_FROM
55
              value: nocodb@kube.rocks
56
          volumeMounts:
57
            - name: nocodb-data
58
              mountPath: /usr/app/data
59
      volumes:
60
        - name: nocodb-data
61
          persistentVolumeClaim:
62
            claimName: nocodb-data
63
---
64
apiVersion: v1
65
kind: PersistentVolumeClaim
66
metadata:
67
  name: nocodb-data
68
  namespace: nocodb
69
spec:
70
  resources:
71
    requests:
72
      storage: 1Gi
73
  volumeMode: Filesystem
74
  storageClassName: longhorn
75
  accessModes:
76
    - ReadWriteOnce
77
---
78
apiVersion: v1
79
kind: Service
80
metadata:
81
  name: nocodb
82
  namespace: nocodb
83
spec:
84
  selector:
85
    app: nocodb
86
  ports:
87
    - port: 8080
88
---
89
apiVersion: traefik.io/v1alpha1
90
kind: IngressRoute
91
metadata:
92
  name: nocodb
93
  namespace: nocodb
94
spec:
95
  entryPoints:
96
    - websecure
97
  routes:
98
    - match: Host(`nocodb.kube.rocks`)
99
      kind: Rule
100
      services:
101
        - name: nocodb
102
          port: 8080

Here are the secrets to adapt to your needs:

1
apiVersion: v1
2
kind: Secret
3
metadata:
4
  name: nocodb-db
5
  namespace: nocodb
6
type: Opaque
7
data:
8
  password: YWRtaW4=

1
apiVersion: v1
2
kind: Secret
3
metadata:
4
  name: nocodb-auth
5
  namespace: nocodb
6
type: Opaque
7
data:
8
  jwt-secret: MDAwMDAwMDAtMDAwMC0wMDAwLTAwMDAtMDAwMDAwMDAwMDAw

1
apiVersion: v1
2
kind: Secret
3
metadata:
4
  name: nocodb-smtp
5
  namespace: nocodb
6
type: Opaque
7
data:
8
  user: YWRtaW4=
9
  password: YWRtaW4=

The final process is identical to n8n.

4th check ✅

We now have a functional continuous delivery with some nice no-code tools to play with ! The final missing stack for a production grade cluster is to install a complete monitoring stack, this is the next part.