Running Docker in GitHub Action steps

Working on the packaging and publishing pipeline for the Git-Mastery app revealed many interesting problems in software packaging, particularly for Linux distributions like Debian and Arch Linux.

In my last post, I shared about the packaging and publishing process for Debian, and how it is pretty non-trivial to get going, especially if you want to support multiple architectures and automate the pipeline using GitHub Actions.

A silver lining?

One benefit of packaging and publishing for Debian through GitHub Actions is that there is a GitHub-hosted runner for Ubuntu, a Debian-based distribution. This means we can easily translate the steps we would take regularly to package and publish a .deb manually to the steps a GitHub Action job would use.

But packaging for Arch Linux does not share the same benefit. There is no GitHub-hosted runner for Arch Linux, so when performing certain actions during the packaging process (like using makepkg), we do not have access to these commands in the Ubuntu hosted runner, and this is a major limitation in the packaging process.

So, what do we do?

Luckily, we can run jobs in a container, where you can:

1. Define a Dockerfile to set up the container
2. Run all your steps within this container

In doing so, we can define a job that runs an Arch Linux base image as a Docker container

That’s it right? Simple as that.

Well, not exactly…

Wait, what’s wrong?

Depending on the container we select, we might need to perform additional set up to prepare the container to include the typical dependencies of the GitHub-hosted runners.

Common tools like python and git may not exist, and it is up to us - the designers of these pipelines - to set up these dependencies and ensure that the container is fit to use.

It might seem simple enough to fix, but some actions that you rely on may also make assumptions about their dependencies, which creates a pretty painful experience all the way down.

Instead, it may be easier (given your specific context), to use Docker containers in certain steps, so you get the full benefit of the GitHub-hosted runners while giving you the option to access a different underlying operating system for a step or two.

What the title promised

To achieve this, an implicit requirement exists: the Docker container and the runner must be able to share files with each other.

This is very easily achievable by mounting a volume in Docker to a folder in the runner.

By doing this, we can now communicate between the Docker container and the runner.

The rest is relatively straight forward:

docker run --rm \
  -v $PWD:/pkg \
  archlinux:base-devel \
  bash -c "
    pacman -Sy --noconfirm sudo git base-devel && \
    useradd -m builder && \
    echo 'builder ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers && \
    chown -R builder:builder /pkg && \
    su builder -c 'cd /pkg && makepkg --printsrcinfo > .SRCINFO'
  "

sudo chown -R $(id -u):$(id -g) .

That seems like a whole lot of nonsense, but in essence, we are:

1. --rm: telling Docker to remove the container when it exits
2. -v $PWD:/pkg: mounting the current working directory of the runner with a directory called /pkg in the Docker container
3. archlinux:base-devl: specifying the image that Docker will use
4. bash -c: runs the following string as a bash command

You don’t need to concern yourself with the specific bash commands running, these are all specific to publishing on Arch Linux (I will be making a post on this soon!).

The final step involves fixing the permissions of the files generated by the Docker container so that the runner is able to properly access and interact with them.

When the files are generated from within the Docker container, even if they are stored in a mounted volume, they will still “belong” to the user within the container. This user will differ from the user in the runner, and this can be problematic for permissions on the runner.

So sudo chown -R $(id -u):$(id -g) . sets the permissions of all files (recursively) within the current directory (aka the mounted volume) back to the runner’s user’s details.

You can extend this by treating the mounted volume as a communication protocol between the Docker container and the runner: using it to store values in shared files that can be read by the runner and used to set environment variables afterwards:

docker run --rm \
  -v $PWD:/pkg \
  archlinux:base-devel \
  bash -c "
    echo 'FOO=hi' >> /pkg/test.txt
  "

sudo chown -R $(id -u):$(id -g) .

cat test.txt >> "$GITHUB_ENV"