Writing a CI runner

At a position I spent a lot of time writing working with GitHub runners. Candidly...it's not great, but at least it's functional simply through together a few actions in a YAML things build\work. Sure you might have some serious build delays because the runners are ephemeral if you don't self-host them, but you will likely have something that works. This same tooling isn't cleanly available or comparable if you deploy say a local git repository onto your network.

That said...this is something I was able to write-up and get deployed\working in about 8 hours, so I felt it was worth writing up how I did it because I am sure someone will find it at a minimum interesting.

Disclaimer and my particular requirements explaining why I did this

You probably don't want to do this. This is something that I did due to a very specific set of circumstances and partly intellectual curiosity. Of note my local git hosting service of choice did not support dynamic CI runner scaling if you are not using their enterprise plan. I also had already written numerous scripts, Makefiles, and utility programs to automate deployments, so I was not going to need things like actions you could load into a pipeline.

A primary concern of mine was that CI needed work to with an offline internet connection. A lot of software such as utilities expects you to have an internet connection at build time so I wanted a custom container image to prevent this. I could not find a clean way to easily do this in Gitea. Not ideal, but Gitea's code is open maybe this is something I could improve or I could sideload the changes.

A second concern of mine which I have experienced from having worked on infrastructure components and distributed systems was not unnecessarily waiting on my CI\CD and my git host of choice does not support dynamic scaling on K8s unless you are an enterprise customer. So my options were to have a couple CI runners at all times knowing that if I (or my partner) kicked off a long-running workload one of them could be offline for hours (ex testing a custom kernel build). As a result I would need to do this myself.

Lastly due to having been burned from CI\CD infrastructure failing on me in the past (ex a single node in a K8s cluster that just so happened to have my workload) I wanted the CI to have a state so it can be restored to any specified state without having to rerun the whole things. This was also a requirement because I wanted my K8s repo to upgrade itself when the respective repo was updated.

For history and context...or how I got in this situation

I have worked on a number of projects in my life where source being leaked would be a massive issue. For example, I have worked on a few cybersecurity proofs of concept POCs for companies, storing in these in GitHub where they could be leaked while the company responds to my report seems negligent at best. Another use case is in a past life I used to write a lot of market analysis algorithms to assist me in research on my investments.

Sensitivity\security aside a fair number of projects I just do not see the value of them being released as open source, for example my home lab cluster provisioning code has little benefit to anyone to be uploaded to GitHub beyond just as an intellectual exercise\to show what I have running. Or I may view the code as being too slapdash\low quality (or a one-off from an AI agent that was written because I didn't have time\interest) to be comfortable releasing it as open source.

The choice of Gitea was made many years ago when my first servers in my home lab were brought online. The hardware I had the time was not powerful enough to run GitLab and all the respective services I wanted on the network. At a certain point I got used to the workflow as did my partner when we moved in together in 2024. Additionally, the process of installing GitLab would have been fairly involved as the instructions discourage the usage of NFS which we use for a lot of our K8s deployments. This meant moving away from Gitea was going to need a excellent justification because while GitLab had more features Gitea had the ones I needed and was just insanely responsive. The CI limitations of Gitea just were not enough to justify a migration.

Designing the CI orchestrator flow

Something interesting is Gitea has a full and very well documented API (link may take a second to load). So what you can do is just request the commit hash of a branch, see if it changes, and then kick off the CI workload if it does. If your workload is going into K8s you simply deploy a job while fire off as a single run then exit. And to make things clean you keep all your repos configuration for the CI in one single location.

Designing the CI repo configuration

Within the repo configuration in addition to the repo API URL we also include a raw file link to a file named ci.sh. This is the actual file that runs on any\all CI changes that will copy into our container. After this file has been copied the container will stop searching the file and continue execution then quit. This provides a very simple and very clean interface for anything we wish to build.

Also within the repo configuration is a key specify what assets we will copy into the container at deployment time. These can be things like credentials, configs, etc. The reason we are not using secrets here is the orchestrator may echo the generated YAML in the event of an error or might write to disk during debugging. It is far simpler\safer to just copy these assets at runtime.

Lastly, in the event there is a docker image to build there is a configuration body that will be loaded by the node the orchestrator is located on so that the docker file can be built (a tool bundled in the build container connects to the node). Sure you can build docker images on K8s, but this has a lot of complexity around caching, and I'm only building a handful of images a week at most. Assuming the Dockerfile is set up correctly due to layer caching the builds are likely to take seconds with the cache so there's not a huge benefit of distributing the workload for building these images.

Designing the CI job recovery\restart

Allowing the CI job to automatically restart\recover was an interesting challenge to say the least. For each repo there is the option to deploy the CI as restart OnFailure or Never. If a build is brittle running it with OnFailure is not the worst idea or if it's something I know can fail randomly (the K8s cluster upgrading itself and the upgrader pod getting deleted in the mix).

The problem here is compared to bare metal or a VM is K8s (or container runtimes in general) after a pod is deleted the entire file system is just wiped so if you want to preserve any data you need to write it to disk. So for each job we run we mount a volume on the NFS share where a workload can write to the /state directory and wipe the directory once the ci.sh finishes. Irrespective of how the CI job is configured the /state directory will always contain the ci.sh and any respective copied files at the pod start time. When the pod starts we check /state and copy any files to their respective locations if they exist.

Designing the CI runner image

Boring as it maybe the CI runner image is based off of Debian Linux and contains the compiled binary for deploying my website due it being a custom page generator. It contains a variety of common utilities for CI and server cluster interactions like curl, jq, git, rsync, openssh, bash, kubectl, and talosctl. It also contains tooling building projects in languages I actively worth with: Python, Rust, Node.js, C\C++, and Go.

The magic in the Docker image is on a single shell script that copies the assets like kubeconfigs, ssh keys, and git credentials sent over to the container to and from /state depending upon if the container is fresh (we copy to /state) or if it has been restarted (we copy from /state). This script blocks until a ci.sh is found either in /state or copied into the container from the orchestrator after which we finally run the actual CI workload that the repo has requested.

Edge cases and interesting challenges

While working on the CI I learned that there is a 0.087% chance that the Gitea instance returns a blank commit hash. I will need to follow up on this further to see exactly where this is from, but for now if I receive an empty commit hash from Gitea I just run continue on the repo loop to skip that execution. I mention this more because I find it funny, and it resulted in my website being redeployed 9 times in the span of a few days.