CI & automation: Multi-architecture build of software with Jenkins and Docker

Multi-arch software building and distribution is a complex and time-consuming maintenance task in which maintainer/developer need to compile somehow their software for all supported architectures, often, on a single machine. Automation solution such as Jenkins allows to automatize this task and facilitating continuous integration and continuous delivery. Multi-arch software building on a single machine involves the use of different techniques:

• Cross-build: using a cross-toochain to build the software for each target system. This solution is simple but with the cost of polluting the host system with many toolchains which maybe incompatible one to another. Maintenance and update of these toolchains are complicated to manage.
• Virtualization solutions such as docker address these problem by using a sandboxed/containerized image with all tools necessary to build the software for each target architecture. Docker facilitate the setup, maintenance and update of different building environments using images.

This post introduces the basic steps of setting up an automation server that allows to build and distribute multi-arch software using Jenkins and docker.

This post does not cover the process of installing docker and assumes that reader has already installed docker on their system. For more information on how to install docker on a specific system, please refer to this link.

One of important steps after installing docker is to add the current user to the group docker to ensure that the user has sufficient right to access to docker daemon via the binded socket.

sudo usermod -aG docker $USER

Running Jenkins server using docker

An easy and clean way to set up a Jenkins server on the host system is to use docker. Create the following docker-compose.yml

version: '3.7'
services:
  jenkins:
    image: jenkins/jenkins:lts
    privileged: true
    user: 1000:998
    ports:
      - 8080:8080
      - 50000:50000
    container_name: jenkins
    volumes:
      - /var/jenkins_home:/var/jenkins_home
      - /var/run/docker.sock:/var/run/docker.sock
      - /usr/bin/docker:/usr/local/bin/docker

This instructs docker to set up a container name jenkins from the image jenkins/jenkins:lts. As we will run Jenkins pipeline with docker container as agent, it is important to ensure that the jenkins container has access right to the host docker daemon. For this purpose, the container is configured with:

• user: 1000:998 : run jenkins server with the id of the current user and the groupid of the docker group. This value should be changed according to the correct id of the current host user and the gid of the docker group (sudo getent group docker to find the gid). This ensures that jenkins has enough access right to the host docker socket.
• /var/run/docker.sock:/var/run/docker.sock mount the docker socket from the host system to the jenkins container
• /usr/bin/docker:/usr/local/bin/docker mount the docker client from the host system to the jenkins container
• /var/jenkins_home:/var/jenkins_home we should use the same path for the JENKINS_HOME both on the host system and the jenkins container, otherwise, there maybe problems when running jenkins pipeline with docker agent

Run the container by using the following command:

docker compose up -d

Browse to http://localhost:8080 to unlock Jenkins and perform all the necessary post-install configurations.
The credential required to unlock Jenkins from browser can be obtained by:

docker exec jenkins cat /var/jenkins_home/secrets/initialAdminPassword

More information on setup Jenkins with docker can be found here https://www.jenkins.io/doc/book/installing/docker/

Install necessary plugins

On Jenkins, from Manage Jenkins > Plugin Manager > Available, with administrator account, install the following plugins: AnsiColor, Docker Pipeline, Pipeline

Build multi-platform tool-images with docker buildkit

Idea: A custom docker image with all necessary tools needed to compile the software is built for each supported architecture. Jenkins will use these images to build multi-architecture binaries from the software.

docker buildx is a Docker CLI plugin for extended build capabilities with BuildKit. This plugin should be already installed when installing docker. It allows to facilitates the cross-platform images building. We begin with a Dockerfile that define the image and all necessary tools to build our software, for example:

FROM  ubuntu:focal AS build-env
RUN apt-get update &&  DEBIAN_FRONTEND="noninteractive" \
    apt-get --yes --no-install-recommends install \
    wget \
    build-essential \
    make \
    libsqlite3-dev \
    cmake \
    zlib1g-dev \
    libreadline-dev \
    libssl-dev \
    autotools-dev \
    autoconf \
    libtool \
    automake \
    libffi-dev

From this example, we will build a docker image based on Ubuntu focal with some additional softwares build-essential, make,cmake, auto-tools, etc. and libraries libssl,libsqlite3, etc as build environment for our software. Reader should include in this image all the necessary tools for their software.

Assume that our software will be distributed as AMD64,ARM64 and ARM binaries. We need to build 3 images, one for each architecture based on this Dockerfile. This can be easily done using docker buildx.

First we need to create and switch to a new builder instance. This should be executed only once:

docker buildx create --use

Assume that our host system is in x86_64/AMD64 architecture, to be able to build images for other architectures such as ARM64 and ARM, we need emulators for these architectures on the host system. These emulators can be registered to the host system by running

 docker run --privileged --rm tonistiigi/binfmt --install arm64,arm

Output:

installing: arm64 OK
installing: arm OK
{
  "supported": [
    "linux/amd64",
    "linux/arm64",
    "linux/riscv64",
    "linux/ppc64le",
    "linux/s390x",
    "linux/386",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
    "qemu-aarch64",
    "qemu-arm",
    "qemu-ppc64le",
    "qemu-riscv64",
    "qemu-s390x"
  ]
}

Now we can build images for the supported architectures:

for arch in arm arm64 amd64  ; do
    docker buildx build \
        --platform $arch \
        --output "type=docker,push=false,name=local/ci-tools:latest-$arch"  .
done

Note that we build an image for each architecture and register these images to the host docker so that they can lately be used locally by Jenkins without the need of pulling the images from a remote registry (e.g Docker Hub). The following three images will be built:

• local/ci-tools:latest-amd64
• local/ci-tools:latest-arm64
• local/ci-tools:latest-arm

Multi-arch build with Jenkins and the generated tool-images

Jenkins job can be configured to run the entire pipeline or each pipeline stage inside a docker container. In our case we want a pipeline with three stages in which each stage build the software inside a container from the previously generated images (AMD64, ARM64 and ARM).

From Jenkins > New Item > Pipeline create a new pipeline name build-multi-arch, for now keep every thing as default and enter the following pipeline declarative script

def do_build()
{
  sh '''
  set -e
  echo "Build software for $arch"
  echo $WORKSPACE
  uname -a
  '''
}
pipeline{
  agent { node{ label'master' }}
  options {
    // Limit build history with buildDiscarder option:
    // daysToKeepStr: history is only kept up to this many days.
    // numToKeepStr: only this many build logs are kept.
    // artifactDaysToKeepStr: artifacts are only kept up to this many days.
    // artifactNumToKeepStr: only this many builds have their artifacts kept.
    buildDiscarder(logRotator(numToKeepStr: "1"))
    // Enable timestamps in build log console
    timestamps()
    // Maximum time to run the whole pipeline before canceling it
    timeout(time: 1, unit: 'HOURS')
    // Use Jenkins ANSI Color Plugin for log console
    ansiColor('xterm')
    // Limit build concurrency to 1 per branch
    disableConcurrentBuilds()
  }
  stages
  {
    stage('Build AMD64') {
      agent {
          docker {
              image 'local/ci-tools:latest-amd64'
              // Run the container on the node specified at the
              // top-level of the Pipeline, in the same workspace,
              // rather than on a new node entirely:
              reuseNode true
          }
      }
      steps {
        script{
          env.arch = "amd64"
        }
        do_build()
      }
    }
    stage('Build ARM64') {
      agent {
          docker {
              image 'local/ci-tools:latest-arm64'
              reuseNode true
          }
      }
      steps {
        script{
          env.arch = "arm64"
        }
        do_build()
      }
    }
    stage('Build ARM') {
      agent {
          docker {
              image 'local/ci-tools:latest-arm'
              reuseNode true
          }
      }
      steps {
        script{
          env.arch = "arm"
        }
        do_build()
      }
    }
  }
}

Note that the function do_build defines all common building steps that are reused across stages. In each stage, we declare a docker agent with one of the images created earlier. Three stages use three images from different architecture AMD64, ARM64, ARM. Steps in each stage run inside the associated docker agent. Precisely, the do_build step in each stage run in different docker container.

Save the pipeline and run a job from that pipeline, the job should be success with the log as follow:

[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/build-multi-arch
[Pipeline] {
[Pipeline] timestamps
[Pipeline] {
[Pipeline] timeout
23:28:56  Timeout set to expire in 1 hr 0 min
[Pipeline] {
[Pipeline] ansiColor
[Pipeline] {
23:28:56  
[Pipeline] stage
[Pipeline] { (Build AMD64)
[Pipeline] getContext
[Pipeline] isUnix
[Pipeline] withEnv
[Pipeline] {
[Pipeline] sh
23:28:56  + docker inspect -f . local/ci-tools:latest-amd64
23:28:56  .
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] withDockerContainer
23:28:56  Jenkins does not seem to be running inside a container
23:28:56  $ docker run -t -d -u 1000:998 -w /var/jenkins_home/workspace/build-multi-arch 
-v /var/jenkins_home/workspace/build-multi-arch:/var/jenkins_home/workspace/build-multi-arch:rw,z -
v /var/jenkins_home/workspace/build-multi-arch@tmp:/var/jenkins_home/workspace/build-multi-arch@tmp:rw,z -
e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** 
-e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** 
-e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** local/ci-tools:latest-amd64 cat
23:28:57  $ docker top e478018bf84b6cffab61419b06c61c96343bcfd95681f2b901754108a7ab64f3 -eo pid,comm
[Pipeline] {
[Pipeline] script
[Pipeline] {
[Pipeline] }
[Pipeline] // script
[Pipeline] sh
23:28:57  + set -e
23:28:57  + echo Build software for amd64
23:28:57  Build software for amd64
23:28:57  + echo /var/jenkins_home/workspace/build-multi-arch
23:28:57  /var/jenkins_home/workspace/build-multi-arch
23:28:57  + uname -a
23:28:57  Linux e478018bf84b 5.10.0-16-amd64 #1 SMP Debian 5.10.127-2 (2022-07-23) x86_64 x86_64 x86_64 GNU/Linux
[Pipeline] }
23:28:57  $ docker stop --time=1 e478018bf84b6cffab61419b06c61c96343bcfd95681f2b901754108a7ab64f3
23:28:59  $ docker rm -f e478018bf84b6cffab61419b06c61c96343bcfd95681f2b901754108a7ab64f3
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Build ARM64)
[Pipeline] getContext
[Pipeline] isUnix
[Pipeline] withEnv
[Pipeline] {
[Pipeline] sh
23:28:59  + docker inspect -f . local/ci-tools:latest-arm64
23:28:59  .
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] withDockerContainer
23:28:59  Jenkins does not seem to be running inside a container
23:28:59  $ docker run -t -d -u 1000:998 -w /var/jenkins_home/workspace/build-multi-arch 
-v /var/jenkins_home/workspace/build-multi-arch:/var/jenkins_home/workspace/build-multi-arch:rw,z 
-v /var/jenkins_home/workspace/build-multi-arch@tmp:/var/jenkins_home/workspace/build-multi-arch@tmp:rw,z 
-e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** 
-e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** 
-e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** local/ci-tools:latest-arm64 cat
23:28:59  $ docker top 3d9cc1166ca26cebfe0a416742980a685d6e127c50cb66b5ba54d5f8ea16f094 -eo pid,comm
[Pipeline] {
[Pipeline] script
[Pipeline] {
[Pipeline] }
[Pipeline] // script
[Pipeline] sh
23:29:00  + set -e
23:29:00  + echo Build software for arm64
23:29:00  Build software for arm64
23:29:00  + echo /var/jenkins_home/workspace/build-multi-arch
23:29:00  /var/jenkins_home/workspace/build-multi-arch
23:29:00  + uname -a
23:29:00  Linux 3d9cc1166ca2 5.10.0-16-amd64 #1 SMP Debian 5.10.127-2 (2022-07-23) aarch64 aarch64 aarch64 GNU/Linux
[Pipeline] }
23:29:00  $ docker stop --time=1 3d9cc1166ca26cebfe0a416742980a685d6e127c50cb66b5ba54d5f8ea16f094
23:29:01  $ docker rm -f 3d9cc1166ca26cebfe0a416742980a685d6e127c50cb66b5ba54d5f8ea16f094
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Build ARM)
[Pipeline] getContext
[Pipeline] isUnix
[Pipeline] withEnv
[Pipeline] {
[Pipeline] sh
23:29:02  + docker inspect -f . local/ci-tools:latest-arm
23:29:02  .
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] withDockerContainer
23:29:02  Jenkins does not seem to be running inside a container
23:29:02  $ docker run -t -d -u 1000:998 -w /var/jenkins_home/workspace/build-multi-arch 
-v /var/jenkins_home/workspace/build-multi-arch:/var/jenkins_home/workspace/build-multi-arch:rw,z 
-v /var/jenkins_home/workspace/build-multi-arch@tmp:/var/jenkins_home/workspace/build-multi-arch@tmp:rw,z 
-e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** 
-e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** 
-e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** local/ci-tools:latest-arm cat
23:29:02  $ docker top 32187da6cba1fe8cadaf1ec8d04b51881b07454b1ef1c6324488e2b624346380 -eo pid,comm
[Pipeline] {
[Pipeline] script
[Pipeline] {
[Pipeline] }
[Pipeline] // script
[Pipeline] sh
23:29:03  + set -e
23:29:03  + echo Build software for arm
23:29:03  Build software for arm
23:29:03  + echo /var/jenkins_home/workspace/build-multi-arch
23:29:03  /var/jenkins_home/workspace/build-multi-arch
23:29:03  + uname -a
23:29:03  Linux 32187da6cba1 5.10.0-16-amd64 #1 SMP Debian 5.10.127-2 (2022-07-23) armv7l armv7l armv7l GNU/Linux
[Pipeline] }
23:29:03  $ docker stop --time=1 32187da6cba1fe8cadaf1ec8d04b51881b07454b1ef1c6324488e2b624346380
23:29:04  $ docker rm -f 32187da6cba1fe8cadaf1ec8d04b51881b07454b1ef1c6324488e2b624346380
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
23:29:04  
[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

From the log, we can clearly see the do_build step which simply run uname -a is executed in three different build environments across the three stages

23:28:57  Linux e478018bf84b 5.10.0-16-amd64 #1 SMP Debian 5.10.127-2 (2022-07-23) x86_64 x86_64 x86_64 GNU/Linux
...
23:29:00  Linux 3d9cc1166ca2 5.10.0-16-amd64 #1 SMP Debian 5.10.127-2 (2022-07-23) aarch64 aarch64 aarch64 GNU/Linux
...
23:29:03  Linux 32187da6cba1 5.10.0-16-amd64 #1 SMP Debian 5.10.127-2 (2022-07-23) armv7l armv7l armv7l GNU/Linux

The pipeline can be easily extended to perform the actual multi-arch build of the software by redefining the do_build function.

Go further...

From this basic setup, there are many options to further automatize the software building and distribution with Jenkins, such as integration with Git:

• The software is hosted on a git SCM
• The Jenkins multi-arch pipeline is defined on a Jenkinfile and hosted on the same git repo with the software
• The git SCM using webhook to trigger a Jenkins build when a push is performed on a specific branch
• Jenkins clones the software and look for the Jenkinfile, then executes the multi-arch pipeline defined in this file
• The multi-arch binaries are distributed as Jenkins artifacts