A beautiful GitOps day X - QA with testing & code metrics

part-11

Use GitOps workflow for building a production grade on-premise Kubernetes cluster on cheap VPS provider, with complete CI/CD 🎉

This is the Part X of more global topic tutorial. Back to guide summary for intro.

Code Metrics

SonarQube is leading the code metrics industry for a long time, embracing full Open Core model, and the community edition it’s completely free of charge even for commercial use. It covers advanced code analysis, code coverage, code duplication, code smells, security vulnerabilities, etc. It ensures high quality code and help to keep it that way.

SonarQube installation

SonarQube has its dedicated Helm chart which is perfect for us. However, it’s the most resource hungry component of our development stack so far (because built with Java ? End of troll), so be sure to deploy it on almost empty free node (which should be ok with 3 workers), maybe a dedicated one. In fact, it’s the last Helm chart for this tutorial, I promise!

Create dedicated database for SonarQube same as usual, then we can use flux for deployment.

1
apiVersion: v1
2
kind: Namespace
3
metadata:
4
  name: sonarqube
5
---
6
apiVersion: source.toolkit.fluxcd.io/v1
7
kind: HelmRepository
8
metadata:
9
  name: sonarqube
10
  namespace: sonarqube
11
spec:
12
  interval: 1h0m0s
13
  url: https://SonarSource.github.io/helm-chart-sonarqube
14
---
15
apiVersion: helm.toolkit.fluxcd.io/v2
16
kind: HelmRelease
17
metadata:
18
  name: sonarqube
19
  namespace: sonarqube
20
spec:
21
  chart:
22
    spec:
23
      chart: sonarqube
24
      reconcileStrategy: ChartVersion
25
      sourceRef:
26
        kind: HelmRepository
27
        name: sonarqube
28
      version: ">=10.0.0"
29
  interval: 1m
30
  releaseName: sonarqube
31
  targetNamespace: sonarqube
32
  values:
33
    resources:
34
      limits:
35
        cpu: 1000m
36
        memory: 2Gi
37
      requests:
38
        cpu: 500m
39
        memory: 2Gi
40

41
    prometheusMonitoring:
42
      podMonitor:
43
        enabled: true
44
        namespace: sonarqube
45

46
    monitoringPasscode: null
47
    monitoringPasscodeSecretName: sonarqube-secret
48
    monitoringPasscodeSecretKey: monitoring-passcode
49

50
    jdbcOverwrite:
51
      enable: true
52
      jdbcUrl: jdbc:postgresql://postgresql-primary.postgres/sonarqube
53
      jdbcUsername: sonarqube
54
      jdbcSecretName: sonarqube-secret
55
      jdbcSecretPasswordKey: db-password
56

57
    postgresql:
58
      enabled: false
59
---
60
apiVersion: traefik.io/v1alpha1
61
kind: IngressRoute
62
metadata:
63
  name: sonarqube
64
  namespace: sonarqube
65
spec:
66
  entryPoints:
67
    - websecure
68
  routes:
69
    - match: Host(`sonarqube.kube.rocks`)
70
      kind: Rule
71
      services:
72
        - name: sonarqube-sonarqube
73
          port: http

Here are the secrets to adapt to your needs:

1
apiVersion: v1
2
kind: Secret
3
metadata:
4
  name: sonarqube-secret
5
  namespace: sonarqube
6
type: Opaque
7
data:
8
  db-password: YWRtaW4=
9
  monitoring-passcode: YWRtaW4=

As seen in part 4 of this guide, seal these secrets with kubeseal under sealed-secret-sonarqube.yaml and delete original secret file.

Inside Helm values, be sure to disable the PostgreSQL sub chart and use our self-hosted cluster with both postgresql.enabled and jdbcOverwrite.enabled. If needed, set proper tolerations and nodeSelector for deploying on a dedicated node.

The installation take many minutes, be patient. Once done, you can access SonarQube on https://sonarqube.kube.rocks and login with admin / admin.

Project configuration

Firstly create a new project through SonarQube UI and retain the project key which is his identifier. Then create a global analysis token named Concourse CI that will be used for CI integration from your user account under /account/security.

Now we need to create a Kubernetes secret which contains this token value for Concourse CI, for usage inside the pipeline. The token is the one generated above.

Add a new concourse terraform variable for the token:

1
variable "concourse_analysis_token" {
2
  type      = string
3
  sensitive = true
4
}

1
concourse_analysis_token = "xxx"

The secret:

1
resource "kubernetes_secret_v1" "concourse_sonarqube" {
2
  metadata {
3
    name      = "sonarqube"
4
    namespace = "concourse-main"
5
  }
6

7
  data = {
8
    url            = "https://sonarqube.${var.domain}"
9
    analysis-token = var.concourse_analysis_token
10
  }
11

12
  depends_on = [
13
    helm_release.concourse
14
  ]
15
}

We are ready to tackle the pipeline for integration.

SonarScanner for .NET

As we use a dotnet project, we will use the official SonarQube scanner for .net. But sadly, as it’s only a .NET CLI wrapper, it requires a java runtime to run and there is no official SonarQube docker image which contains both .NET SDK and Java runtime. But we have a CI now, so we can build our own QA image on our own private registry.

Create a new Gitea repo dedicated for any custom docker images with this one single Dockerfile:

1
FROM mcr.microsoft.com/dotnet/sdk:7.0
2

3
RUN apt-get update && apt-get install -y ca-certificates-java && apt-get install -y \
4
    openjdk-17-jre-headless \
5
    unzip \
6
    && rm -rf /var/lib/apt/lists/*
7

8
RUN dotnet tool install --global dotnet-sonarscanner
9
RUN dotnet tool install --global dotnet-coverage
10

11
ENV PATH="${PATH}:/root/.dotnet/tools"

Note as we add the dotnet-sonarscanner tool to the path, we can use it directly in the pipeline without any extra step. I’ll also add dotnet-coverage global tool for code coverage generation that we’ll use later.

Then the pipeline:

1
resources:
2
  - name: docker-images-git
3
    type: git
4
    icon: coffee
5
    source:
6
      uri: https://gitea.kube.rocks/kuberocks/docker-images
7
      branch: main
8
  - name: dotnet-qa-image
9
    type: registry-image
10
    icon: docker
11
    source:
12
      repository: ((registry.name))/kuberocks/dotnet-qa
13
      tag: "7.0"
14
      username: ((registry.username))
15
      password: ((registry.password))
16

17
jobs:
18
  - name: dotnet-qa
19
    plan:
20
      - get: docker-images-git
21
      - task: build-image
22
        privileged: true
23
        config:
24
          platform: linux
25
          image_resource:
26
            type: registry-image
27
            source:
28
              repository: concourse/oci-build-task
29
          inputs:
30
            - name: docker-images-git
31
          outputs:
32
            - name: image
33
          params:
34
            DOCKERFILE: docker-images-git/dotnet-qa.dockerfile
35
          run:
36
            path: build
37
      - put: dotnet-qa-image
38
        params:
39
          image: image/image.tar

Update the main.yaml pipeline to add the new job, then trigger it manually from Concourse UI to add the new above pipeline:

1
#...
2

3
jobs:
4
  - name: configure-pipelines
5
    plan:
6
      #...
7
      - set_pipeline: images
8
        file: ci/pipelines/images.yaml

The pipeline should now start and build the image, trigger it manually if needed on Concourse UI. Once done, you can check it on your Gitea container packages that the new image gitea.kube.rocks/kuberocks/dotnet-qa is here.

Concourse pipeline integration

It’s finally time to reuse this QA image in our Concourse demo project pipeline. Update it accordingly:

1
#...
2

3
jobs:
4
  - name: build
5
    plan:
6
      - get: source-code
7
        trigger: true
8

9
      - task: build-source
10
        config:
11
          platform: linux
12
          image_resource:
13
            type: registry-image
14
            source:
15
              repository: ((registry.name))/kuberocks/dotnet-qa
16
              tag: "7.0"
17
              username: ((registry.username))
18
              password: ((registry.password))
19
          #...
20
          run:
21
            path: /bin/sh
22
            args:
23
              - -ec
24
              - |
25
                dotnet format --verify-no-changes
26

27
                dotnet sonarscanner begin /k:"KubeRocks-Demo" /d:sonar.host.url="((sonarqube.url))"  /d:sonar.token="((sonarqube.analysis-token))"
28
                dotnet build -c Release
29
                dotnet sonarscanner end /d:sonar.token="((sonarqube.analysis-token))"
30

31
                dotnet publish src/KubeRocks.WebApi -c Release -o publish --no-restore --no-build
32

33
      #...

Note as we now use the dotnet-qa image and surround the build step by dotnet sonarscanner begin and dotnet sonarscanner end commands with appropriate credentials allowing Sonar CLI to send report to our SonarQube instance. Trigger the pipeline manually, all should pass, and the result will be pushed to SonarQube.

SonarQube

Feature testing

Let’s cover the feature testing by calling the API against a real database. This is the opportunity to tackle the code coverage as well.

xUnit

First add a dedicated database for test in the docker compose file as we won’t interfere with the development database:

1
version: "3"
2

3
services:
4
  #...
5

6
  db_test:
7
    image: postgres:15
8
    environment:
9
      POSTGRES_USER: main
10
      POSTGRES_PASSWORD: main
11
      POSTGRES_DB: main
12
    ports:
13
      - 54320:5432

Expose the startup service of minimal API:

1
//...
2

3
public partial class Program
4
{
5
    protected Program() { }
6
}

Then add a testing JSON environment file for accessing our database db_test from the docker-compose.yml:

1
{
2
  "ConnectionStrings": {
3
    "DefaultConnection": "Host=localhost:54320;Username=main;Password=main;Database=main;"
4
  }
5
}

Now the test project:

dotnet new xunit -o tests/KubeRocks.FeatureTests
dotnet sln add tests/KubeRocks.FeatureTests
dotnet add tests/KubeRocks.FeatureTests reference src/KubeRocks.WebApi
dotnet add tests/KubeRocks.FeatureTests package Microsoft.AspNetCore.Mvc.Testing
dotnet add tests/KubeRocks.FeatureTests package Respawn
dotnet add tests/KubeRocks.FeatureTests package FluentAssertions

The WebApplicationFactory that will use our testing environment:

1
using Microsoft.AspNetCore.Mvc.Testing;
2
using Microsoft.Extensions.Hosting;
3

4
namespace KubeRocks.FeatureTests;
5

6
public class KubeRocksApiFactory : WebApplicationFactory<Program>
7
{
8
    protected override IHost CreateHost(IHostBuilder builder)
9
    {
10
        builder.UseEnvironment("Testing");
11

12
        return base.CreateHost(builder);
13
    }
14
}

The base test class for all test classes that manages database cleanup thanks to Respawn:

1
using KubeRocks.Application.Contexts;
2

3
using Microsoft.EntityFrameworkCore;
4
using Microsoft.Extensions.DependencyInjection;
5

6
using Npgsql;
7

8
using Respawn;
9
using Respawn.Graph;
10

11
namespace KubeRocks.FeatureTests;
12

13
[Collection("Sequencial")]
14
public class TestBase : IClassFixture<KubeRocksApiFactory>, IAsyncLifetime
15
{
16
    protected KubeRocksApiFactory Factory { get; private set; }
17

18
    protected TestBase(KubeRocksApiFactory factory)
19
    {
20
        Factory = factory;
21
    }
22

23
    public async Task RefreshDatabase()
24
    {
25
        using var scope = Factory.Services.CreateScope();
26

27
        using var conn = new NpgsqlConnection(
28
            scope.ServiceProvider.GetRequiredService<AppDbContext>().Database.GetConnectionString()
29
        );
30

31
        await conn.OpenAsync();
32

33
        var respawner = await Respawner.CreateAsync(conn, new RespawnerOptions
34
        {
35
            TablesToIgnore = new Table[] { "__EFMigrationsHistory" },
36
            DbAdapter = DbAdapter.Postgres
37
        });
38

39
        await respawner.ResetAsync(conn);
40
    }
41

42
    public Task InitializeAsync()
43
    {
44
        return RefreshDatabase();
45
    }
46

47
    public Task DisposeAsync()
48
    {
49
        return Task.CompletedTask;
50
    }
51
}

Note the Collection attribute that will force the test classes to run sequentially, required as we will use the same database for all tests.

Finally, the tests for the 2 endpoints of our articles controller:

1
using System.Net.Http.Json;
2

3
using FluentAssertions;
4

5
using KubeRocks.Application.Contexts;
6
using KubeRocks.Application.Entities;
7
using KubeRocks.WebApi.Models;
8

9
using Microsoft.Extensions.DependencyInjection;
10

11
using static KubeRocks.WebApi.Controllers.ArticlesController;
12

13
namespace KubeRocks.FeatureTests.Articles;
14

15
public class ArticlesListTests : TestBase
16
{
17
    public ArticlesListTests(KubeRocksApiFactory factory) : base(factory) { }
18

19
    [Fact]
20
    public async Task Can_Paginate_Articles()
21
    {
22
        using (var scope = Factory.Services.CreateScope())
23
        {
24
            var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
25

26
            var user = db.Users.Add(new User
27
            {
28
                Name = "John Doe",
29
                Email = "john.doe@email.com"
30
            });
31

32
            db.Articles.AddRange(Enumerable.Range(1, 50).Select(i => new Article
33
            {
34
                Title = $"Test Title {i}",
35
                Slug = $"test-title-{i}",
36
                Description = "Test Description",
37
                Body = "Test Body",
38
                Author = user.Entity,
39
            }));
40

41
            await db.SaveChangesAsync();
42
        }
43

44
        var response = await Factory.CreateClient().GetAsync("/api/Articles?page=1&size=20");
45

46
        response.EnsureSuccessStatusCode();
47

48
        var body = (await response.Content.ReadFromJsonAsync<ArticlesResponse>())!;
49

50
        body.Articles.Count().Should().Be(20);
51
        body.ArticlesCount.Should().Be(50);
52

53
        body.Articles.First().Should().BeEquivalentTo(new
54
        {
55
            Title = "Test Title 50",
56
            Description = "Test Description",
57
            Body = "Test Body",
58
            Author = new
59
            {
60
                Name = "John Doe"
61
            },
62
        });
63
    }
64

65
    [Fact]
66
    public async Task Can_Get_Article()
67
    {
68
        using (var scope = Factory.Services.CreateScope())
69
        {
70
            var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
71

72
            db.Articles.Add(new Article
73
            {
74
                Title = $"Test Title",
75
                Slug = $"test-title",
76
                Description = "Test Description",
77
                Body = "Test Body",
78
                Author = new User
79
                {
80
                    Name = "John Doe",
81
                    Email = "john.doe@email.com"
82
                }
83
            });
84

85
            await db.SaveChangesAsync();
86
        }
87

88
        var response = await Factory.CreateClient().GetAsync($"/api/Articles/test-title");
89

90
        response.EnsureSuccessStatusCode();
91

92
        var body = (await response.Content.ReadFromJsonAsync<ArticleDto>())!;
93

94
        body.Should().BeEquivalentTo(new
95
        {
96
            Title = "Test Title",
97
            Description = "Test Description",
98
            Body = "Test Body",
99
            Author = new
100
            {
101
                Name = "John Doe"
102
            },
103
        });
104
    }
105
}

Ensure all tests passes with dotnet test.

Note

You may be interested in Testcontainers for native support of containers inside code, including parallelism.

CI tests & code coverage

Now we need to integrate the tests in our CI pipeline. As we testing with a real database, create a new demo_test database through pgAdmin with basic test / test credentials.

Note

In real world scenario, you should use a dedicated database for testing, and not the same as production.

Let’s edit the pipeline accordingly for tests:

1
#...
2

3
jobs:
4
  - name: build
5
    plan:
6
      #...
7

8
      - task: build-source
9
        config:
10
          #...
11
          params:
12
            ConnectionStrings__DefaultConnection: "Host=postgres-primary.postgres;Username=test;Password=test;Database=demo_test"
13
          run:
14
            path: /bin/sh
15
            args:
16
              - -ec
17
              - |
18
                dotnet format --verify-no-changes
19

20
                dotnet sonarscanner begin /k:"KubeRocks-Demo" /d:sonar.host.url="((sonarqube.url))"  /d:sonar.token="((sonarqube.analysis-token))" /d:sonar.cs.vscoveragexml.reportsPaths=coverage.xml
21
                dotnet build -c Release
22
                dotnet-coverage collect 'dotnet test -c Release --no-restore --no-build --verbosity=normal' -f xml -o 'coverage.xml'
23
                dotnet sonarscanner end /d:sonar.token="((sonarqube.analysis-token))"
24

25
                dotnet publish src/KubeRocks.WebApi -c Release -o publish --no-restore --no-build
26

27
#...

Note as we already include code coverage by using dotnet-coverage tool. Don’t forget to precise the path of coverage.xml to sonarscanner CLI too. It’s time to push our code with tests or trigger the pipeline manually to test our integration tests.

If all goes well, you should see the tests results on SonarQube with some coverage done:

SonarQube

Coverage detail:

SonarQube

You may exclude some files from analysis by adding some project properties:

1
<Project Sdk="Microsoft.NET.Sdk">
2
  <!-- ... -->
3

4
  <ItemGroup>
5
    <SonarQubeSetting Include="sonar.exclusions">
6
      <Value>appsettings.Testing.json</Value>
7
    </SonarQubeSetting>
8
  </ItemGroup>
9
</Project>

Same for coverage:

1
<Project Sdk="Microsoft.NET.Sdk">
2
  <!-- ... -->
3

4
  <ItemGroup>
5
    <SonarQubeSetting Include="sonar.coverage.exclusions">
6
      <Value>Migrations/**/*</Value>
7
    </SonarQubeSetting>
8
  </ItemGroup>
9
</Project>

Sonar Analyzer

You can enforce many default sonar rules by using Sonar Analyzer directly locally before any code push.

Create this file at the root of your solution for enabling Sonar Analyzer globally:

1
<Project>
2
  <PropertyGroup>
3
    <AnalysisLevel>latest-Recommended</AnalysisLevel>
4
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
5
    <CodeAnalysisTreatWarningsAsErrors>true</CodeAnalysisTreatWarningsAsErrors>
6
  </PropertyGroup>
7
  <ItemGroup>
8
    <PackageReference
9
      Include="SonarAnalyzer.CSharp"
10
      Version="9.8.0.76515"
11
      PrivateAssets="all"
12
      Condition="$(MSBuildProjectExtension) == '.csproj'"
13
    />
14
  </ItemGroup>
15
</Project>

Any rule violation is treated as error at project building, which block the CI before execution of tests. Use latest-All as AnalysisLevel for psychopath mode.

At this stage as soon this file is added, you should see some errors at building. If you use VSCode with correct C# extension, these errors will be highlighted directly in the editor. Here are some fixes:

1
#...
2

3
builder.Host.UseSerilog((ctx, cfg) => cfg
4
    .ReadFrom.Configuration(ctx.Configuration)
5
    .Enrich.WithSpan()
6
    .WriteTo.Console(
7
        outputTemplate: "[{Timestamp:HH:mm:ss} {Level:u3}] |{TraceId}| {Message:lj}{NewLine}{Exception}",
8
        // Enforce culture
9
        formatProvider: CultureInfo.InvariantCulture
10
    )
11
);
12

13
#...

Delete WeatherForecastController.cs.

1
<Project Sdk="Microsoft.NET.Sdk">
2

3
  <PropertyGroup>
4
    <!-- ... -->
5

6
    <NoWarn>CA1707</NoWarn>
7
  </PropertyGroup>
8

9
  <!-- ... -->
10
</Project>

10th check ✅

We have done for code quality process. Go to the final part with load testing, and some frontend !