Setup a Docker Swarm cluster Part VIII - Code metrics & Load testing

Build your own cheap but powerful self-hosted cluster and be free from any SaaS solutions by following this opinionated guide 🎉

This is the Part VIII of more global topic tutorial. Back to first part for intro.

Tracing with OpenTelemetry 🕰️

Now it’s time to test our Jaeger tracing for further integration to our cluster. For that we’ll use OpenTelemetry toolkit.

Install all the required libraries to above weather project :

# we use prerelease for better .NET 6 compatibility
dotnet add package OpenTelemetry.Extensions.Hosting --prerelease
dotnet add package OpenTelemetry.Instrumentation.AspNetCore --prerelease
dotnet add package OpenTelemetry.Exporter.Jaeger --prerelease

Then add automatic ASP.NET instrumentation and configure Jaeger exporter in Program.cs by adding following lines before builder.Build() :

1
using OpenTelemetry.Resources;
2
using OpenTelemetry.Trace;
3

4
//...
5

6
builder.Services.AddOpenTelemetryTracing(b => b
7
    .SetResourceBuilder(ResourceBuilder
8
        .CreateDefault()
9
        .AddService("Weather API"))
10
    .AddAspNetCoreInstrumentation()
11
    .AddJaegerExporter(o =>
12
    {
13
        o.AgentHost = builder.Configuration.GetValue<string>("Jaeger:Host");
14
        o.AgentPort = builder.Configuration.GetValue<int>("Jaeger:Port");
15
    })
16
);
17

18
//...

Push the code and ensure all CI/CD workflow passes.

Then edit the weather docker stack and configure Jaeger connection.

1
version: '3.8'
2

3
services:
4
  app:
5
    image: registry.sw.dockerswarm.rocks/adr1enbe4udou1n/my-weather-api
6
    environment:
7
      ASPNETCORE_ENVIRONMENT: Development
8
      Jaeger__Host: tasks.jaeger_agent
9
      Jaeger__Port: 6831
10
    networks:
11
      - traefik_public
12
      - jaeger_private
13
    #...
14

15
networks:
16
  traefik_public:
17
    external: true
18
  jaeger_private:
19
    external: true

Ensure the weather API is deployed and do some API calls. Finally, go back to Jaeger UI, a second service Weather API should appear, select it and click on Find Traces. You should get all API call traces detail !

Feel free to add any other traces. There are 2 types of traces :

• Automatic : like above ASP.NET instrumentation, for any standardized library integration. If you use PostgreSQL, use Npgsql OpenTelemetry package for each SQL call traces.
• Manual : simple manual traces inside your application, for even further granular view.

SonarQube 📈

Let’s get some automatic quality code metrics.

On manager-01 :

sudo mkdir -p /mnt/storage-pool/sonar/data
sudo mkdir -p /mnt/storage-pool/sonar/logs
sudo mkdir -p /mnt/storage-pool/sonar/extensions

# specific thing for embed elasticsearch of sonarqube
echo "vm.max_map_count=262144" | tee /etc/sysctl.d/local.conf
sudo service procps restart

Create a sonar PostgresSQL database, and create next stack :

1
version: '3.8'
2

3
services:
4
  server:
5
    image: sonarqube:9-community
6
    volumes:
7
      - /etc/hosts:/etc/hosts
8
      - ${ROOT_PATH}/data:/opt/sonarqube/data
9
      - ${ROOT_PATH}/logs:/opt/sonarqube/logs
10
      - ${ROOT_PATH}/extensions:/opt/sonarqube/extensions
11
    environment:
12
      SONAR_JDBC_URL: jdbc:postgresql://data-01:5432/sonar
13
      SONAR_JDBC_USERNAME: sonar
14
      SONAR_JDBC_PASSWORD:
15
    networks:
16
      - traefik_public
17
    deploy:
18
      labels:
19
        - traefik.enable=true
20
        - traefik.http.routers.sonar.entrypoints=https
21
        - traefik.http.services.sonar.loadbalancer.server.port=9000
22
      placement:
23
        constraints:
24
          - node.role == manager
25

26
networks:
27
  traefik_public:
28
    external: true

Set proper ROOT_PATH with /mnt/storage-pool/sonar and SONAR_JDBC_PASSWORD with above DB password.

Go to https://sonar.sw.dockerswarm.rocks, use admin / admin credentials and update password.

Project analysis

Create manual project “My Weather API” and go to Other CI. Generate a token, select .NET project, back to above my-weather-api project, and go through all prerequisite.

dotnet tool install --global dotnet-sonarscanner

dotnet sonarscanner begin /k:"My-Weather-API" /d:sonar.host.url="https://sonar.sw.dockerswarm.rocks"  /d:sonar.login="above-generated-token"

dotnet build

dotnet sonarscanner end /d:sonar.login="above-generated-token"

Wait few minutes and the final rapport analysis should automatically appear. Add .sonarqube to .gitignore.

CI integration

Because running scanner manually is boring, let’s integrate it in our favorite CI. Create following secrets through Drone UI :

Change the build step on .drone.yml file :

1
#...
2
  - name: build
3
    image: mcr.microsoft.com/dotnet/sdk:6.0
4
    commands:
5
      - apt-get update
6
      - apt-get install --yes openjdk-11-jre
7
      - dotnet tool install --global dotnet-sonarscanner
8
      - export PATH="$PATH:$HOME/.dotnet/tools"
9
      - dotnet sonarscanner begin /k:"My-Weather-API" /d:sonar.login="$SONAR_TOKEN" /d:"sonar.host.url=$SONAR_HOST_URL"
10
      - dotnet build
11
      - dotnet sonarscanner end /d:sonar.login="$SONAR_TOKEN"
12
      - dotnet publish -c Release -o ./publish
13
#...

And voilà ! You should have automatic code analysis on every code push.

Load testing 💣

We have now a perfect environment for a well-balanced load testing ! There are 2 popular options :

• Locust, from Python world, which offers a nice integrated chart web UI and master-workers architecture which allows distributed loading. Load scripts are written on Python.
• k6, written in Go, which is more efficient and less resource demanding for the same load. Load scripts are written on JavaScript.

Here I’ll cover usage of k6. Note that it can be integrated to a time series database for a nice graphical extraction. And guess what, Grafana is the perfect tool for that ! Let’s profit of our powerful tools we have !

TSDB with InfluxDB

Create a new influxdb stack :

1
version: '3.8'
2

3
services:
4
  db:
5
    image: influxdb:1.8
6
    volumes:
7
      - data:/var/lib/influxdb
8
    networks:
9
      - private
10
    deploy:
11
      placement:
12
        constraints:
13
          - node.labels.influxdb.data=true == true
14

15
networks:
16
  private:
17

18
volumes:
19
  data:

Add InfluxDB private network to Grafana stack :

1
version: '3.8'
2

3
services:
4
  grafana:
5
    #...
6
    networks:
7
      - influxdb_private
8
      - traefik_public
9
    #...
10

11
networks:
12
  influxdb_private:
13
    external: true
14
  traefik_public:
15
    external: true

Test loading with k6

First create a simple JS script as docker swarm Config named k6_weather_test_01 through Portainer UI :

1
import http from "k6/http";
2
import { check } from "k6";
3

4
export default function () {
5
  http.get('https://weather.sw.dockerswarm.rocks/WeatherForecast');
6
}

Then create the following stack :

1
version: '3.8'
2

3
services:
4
  load:
5
    image: grafana/k6
6
    configs:
7
      - source: k6_weather_test_01
8
        target: /scripts/k6-weather.js
9
    environment:
10
      - K6_VUS=100
11
      - K6_DURATION=1m
12
      - K6_OUT=influxdb=http://db:8086/k6weather
13
    entrypoint: ['k6', 'run', '/scripts/k6-weather.js']
14
    networks:
15
      - influxdb_private
16
    deploy:
17
      restart_policy:
18
        condition: none
19
      placement:
20
        constraints:
21
          - node.labels.environment == build
22

23
networks:
24
  influxdb_private:
25
    external: true
26

27
configs:
28
  k6_weather_test_01:
29
    external: true

Deploy the stack, and it should launch a load testing for 1 minute. In the end, the task status of docker service should indicate complete status.

With Loki as the default log driver, we get only current logs of running tasks in Portainer, so when completed, the logs are wiped. You must go through Grafana in the Logs dashboard that we built on previous part in order to check the final raw results.

Visualization through Grafana

It’s now time to go back to Grafana and try to get some charts from influxdb ! First Add a new InfluxDB Data source. Set http://influxdb_db:8086 inside URL field and k6weather in Database field, then Save & test.

Now create a new dashboard, a new panel, and keep Time series as main graph system. Select above InfluxDB data source and switch to raw query expression. Finally, put following query SELECT sum("value") FROM "http_reqs" WHERE $timeFilter GROUP BY time(1s) in fields. Some graph should appear, select the right time interval where you have done previous load testing and voilà !

We now have the current HTTP requests count for every second. Below the Chart.js result of my own loading test.

Test loading scenario

You can go even further with more progressive scenario. Create a new k6_weather_test_02 docker config script :

1
import http from "k6/http";
2
import { check } from "k6";
3

4
export const options = {
5
  stages: [
6
    { duration: '5m', target: 200 },
7
  ],
8
};
9

10
export default function () {
11
  http.get('https://weather.sw.dockerswarm.rocks/WeatherForecast');
12
}

This is a progressive 5 minutes load testing scenario from 1 user to 200 concurrent users.

Use this script on above k6 stack and be sure to comment K6_VUS and K6_DURATION environment variables. Check the logs to ensure that you have correct scenario :

Then check the raw and graph result in Grafana. Here the corresponding Chart.js result in my case for the 1st minute :

Final check 🎊🏁🎊

Congratulation if you’re getting that far !!!

I’m personally happy to finish this pretty massive tutorial…

You’ve done all the basic parts of installing, using, testing a professional grade Docker Swarm cluster.