Infra Projects and Tooling

A lot of my infrastructure is custom and much of it is hyperspecific to the deployment I have. Here is a list of things that have been built...some of these you might find interesting from a design perspective. Much of this expands off of my last homelab rebuild, but this is more the custom tooling and platform I use itself.

Much of this tooling is written for a very small team or solo developer.

CI\CD

Of note the CI\CD on the network is fully custom. I use a custom docker image with a set of tools and a number of services on the network to build and deploy applications. This was done this way because fundamentally I do not like the idea of a yaml being the default config since often times it just ends up containing shell scripting anyways. From my perspective here given everything is local is I should for speed package all the dependencies\config I need in the image and simply load a single shell script. This also would let me run distrbuted builds across the K8s which was something that Gitea does not support outside of Enterprise deployments.

A fully custom pipeline where the entire config is a single small code block with utilities are also present in the image means that 10-15 second build is very viable and even normal (for example the binary builder for the website you are on lives in that image). This also abstracts a lot of the decisions away from the user which means it is very hard for the user to misconfigure something and it makes it very easy to spin up new projects. Because it is abstracted so much it allows a human to in a couple minutes setup a new repo or to hand it off to an AI to hook up the CI pipeline. And because builds are quick you can rapidly build\test changes.

Do I recommend you do this...no...candidly this is something really only makes sense in enterprise organizations with a very established tech stack. For me in two past positions of mine a nontrivial portion of my work involved building engineering platforms and one of the highest returns in that situation is a CI\CD. The best implementations always kept the actual runner config as clean as possible. Here I am just completely removing that yaml so a single shell script runs. This CI lacks a lot of features most people would expect today and heavily trades simplicity\speed of setup over feature set. It is also fundamentally is a different design perspective from most CI\CD systems...most emphasize the engineer loading and configuring what they need then doing so and can leverage a lot of existing actions\tools while here...this CI\CD system expects any and all tools to be in the CI image which makes the stack highly opinionated and heavily leans towards expecting custom tooling to exist. Because this network is entirely code as infrastructure and all of the network is configured\managed from one multithreaded project, effectively this is the NixOS design ideology for the entire network. It is entirely reasonable to assume that only binaries and tooling explicitly configured\packaged for the stack are reasonably expected to be used.

If you are doing this, you likely are a platform engineer and have an insanely good reason for doing so. Mine reason was speed of builds\testing so I could maximize how quickly I built things. In doing so I have knowingly sacrificed flexibility and introduced complexity. If you want a writeup on exactly how I did it the info is here.

Arch AUR CI

It's no secret I am an active Arch Linux user and this is my prefered distro of choice. In fact I have made contributions to the Arch wiki and the AUR and regularly continue to do so. Arch has for many years placed unofficial packages as PKGBUILD files through the AUR (Arch User Repository) and in general this works very well as it can be an excellent way to get uncommon software or bleeding edge software onto your system. In fact most of VR Linux as an Arch user makes use of the AUR, a bit more information here if you want it.

The downside with this it is a bit of a supply chain risk. So your options are to audit every package before you update or to use something like a prebuilt archive such as the chaotic aur...or you can build them yourself from source, but I couldn't find a solid CI builder that would let me run them in k8s and upload them to a proper Arch repo so I built my own. More information on that here.

Docker\OCI Images

Rather than deal with caching across distributed docker builds or trying to run my builds on K8s directly I use a pretty lightweight daemon\service that is running on my thin client\my main developer box. In most cases these are fairly small builds so they can be completed in under a minute. This daemon works by queueing the builds and then pushing to the respective docker repos on the local network; we make use of a queue here purely to not cause an OOM error on the builds. While I am aware of buildkit, in my situation this would involve figuring out the build cache and setup with pretty low benefit. If a build is exceptionally slow for me I can simply build the binary in the K8s cluster and then copy that back in the dockerfile.

Of course I also use custom base images to save myself time\effort when spinning up new projects.

For more information take a look here.

Orchestration and Upgrades

The orchestration for my software is largely a mix of templated NixOS systems, K8s templated deployments\resources deployed on Talos, and some legacy shell scripts\shell commands. One could use something like Ansible for this and that has a lot of value assuming: the existing modules do what you want, you do not already have the code written\tested for this functionality, and you can cleanly articulate what you want in Ansible. For me the answer to all of these was no because I wanted a very specific mechanism for templating NixOS files so would likely need a custom module to do it cleanly, I already had the code as my network grew out of shell scripts\into more complex tooling over about 6 years, and I don't think writing in yaml what is needed is as effective as a function call for a human or an AI to read (and it also sacrifices a lot in things like tooling\validation). Because of this I wrote my own system for templating these components and generally managing the network, you can read more about it here. Do I recommend you write this...no as most deployments can be cleanly articulated in Ansible, I just have so much custom tooling as is and am using less common deployment targets that something like Ansible requires enough custom code I might as well just write it myself as realistically I am the only engineer (or one of the few) who is ever going to read the code.

VM Recovery and Network Monitoring

When writing software that is highly experimental and frequently a prototype there is an insanely high posibility that it will crash. So...in order to avoid this the simplest solution is to deploy a simple watchdog\monitoring process that can heal and restart services when they freeze. This is useful for the rare case where something I write is not an high availability deployment or service. This also allows more structured failover\recovery to work around vendor issues as software is never perfect. After the patch is applied I am able to keep the workload in place until one of the following happens: I get the time to fix it properly, the vendor fixes the issue, or the system gets retired. This same set of tooling is also used to return the network status to anyone on the network as well as myself in a clean little icon on my thin client. For more information take a look here.

VM Thin client

The thin client and primary dev box I use is actually a virtual machine. This is an entire conversation, but it actually works insanely well and runs on top of one of the more beefy servers where I did not want to dedicate a full machine to what I was doing. I have been running the system for about the past 2 years with relatively few issues. This box is also the home of multiple services to orchestrate the network that were opted to not be put in high availability because the complexity was not worth it.

This isn't explicitly infra, but it is an unusual hardware deployment hence why it is mentioned here. For the full writeup you can take a look here.

Website Stats:

Website Build Version: 2026/11/20 11:46:51 AM (+00:00)

Last Website Update: 62193f6

Site Generator: Serpent Page Generator created by Lucia Smith