NixOS

Background

I'm a firm believer of using the best tool for the job rather than the one I am currently familiar with. In other words if I think that a different technology does a better job at a task than what I'm currently using then I should use it. And I believe the best way to do this is by actually applying the technology into a project that I can fully leverage rather than just reading upon a recommendation from another engineer. I need to get into the nuts and bolts...see what I can actually do...what are the sharp edges...and what does it do well.

A tangible example is this website being in Rust here, while it might appear at face value to be an insane decision there was in fact a very good reason for it. Or the decisions to deploy the home lab with Talos on the K8s nodes in spite of having integrated with the k8s API, written operators, and written kublets.

By extension to this if there isn't a technology for what I want then I should build my own version...this is how Athena and my patents ended up coming into existence. But we don't always need to write our own implementation and ideally doing so should be the last option if something works to meet the needs on the table.

I'm not even going to pretend I completely understand NixOS as it's not a distro I have years of experience in or have contributed directly to. For me that would be Arch as I have contributed to that one. Or even something in the Debian\Ubuntu family where I many years ago dabbled with packaging up Hybrid graphics in Gnome (anyone remember Prime and Bumblee...and GDM not working correctly...that chaos?). But...as someone who has torn open the internals of the Linux operating system a fair bit I felt that something that would allow me to configure the OS declaratively as I often break default configs\tweak them to match my needs.

This was further amplified by my home lab's movement away from virtualization as heavily largely due to increased pricing on memory modules. And virtualization adds up very quickly on memory consumption and storage for the OS installs. While we do still make use of virtualization on the network it's mostly for security isolation rather than package isolation (not everything can or should be packaged in a docker container ex your NTP server). Even if this were not an issue you're still looking at CPU overhead when you have to do context switching by splitting physical CPUs into virtual ones. Throw in the fact I wanted to move the having a full CI\CD pipeline (which was built agnostic of the git provider and local) having most of the machines be configurable with a commit was a huge selling point.

For the curious on why I made such heavy usage of virtualization at the time please take a look at the original home lab and the homelab rebuild. This will give more context on what I was working with.

The plan

Simplify my deployment logic. Ideally with a single file and with stable versions. The intent here was mostly to migrate long running services and a few containers to a stable distro that would be easier to configure automatically.

The problem is...my goal was to be simple and if you have written Nix configuration before you will learn very quickly it is effectively it's own language so diverging too far can very quickly make simple changes take longer than they needed to. As a result I chose to not dig into flakes and opted to leave some workloads of mine which could not be cleanly packaged on NixOS running on Ubuntu (will get into more detail on this later). While I understand this is a major part of NixOS my goal here was simplicity of deployment and I do not want to get into dealing with packaging edge cases.

What went well

NixOS has really strong documentation and there are a lot of examples on how to configure various services. Samba, NFS, mysql (mariadb), reverse proxies (nginx), ntp (ntpd), and podman (OCI runtime) were all very simple to add for example. It was lovely to have a single file change that in a few minutes would just configure the box without needing to write an ansible playbook for said functionality.

It is really hard to complain though when adding a new domain\website to the network is a small block:

virtualHosts."git.friedmicro-lab.org" = { locations."/" = { proxyPass = "http://192.168.1.19:30011"; proxyWebsockets = true; extraConfig = "client_max_body_size 512M; " + "proxy_set_header Host $host; " + "proxy_set_header X-Real-IP $remote_addr; " + "proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; " + "proxy_set_header X-Forwarded-Proto $scheme; " + "proxy_redirect http://192.168.1.19:30011 http://$host;" + "proxy_set_header Connection $http_connection; " + "proxy_set_header Upgrade $http_upgrade;"; }; };

Or for NTP:

services.ntp = { enable = true; servers = [ "0.pool.ntp.org" "1.pool.ntp.org" "2.pool.ntp.org" ]; };

I actually have really good reasons for running an NTP server. I have multiple K8S nodes which do best with an internal NTP server. I also prefer having a NTP server as I have been burned by having NTP over the internet fail (I've actually been burned by it in virtualized environments professionally as well...wondering why a service was killed due to a healthcheck because the clock jumped). I also have had issues with my VR setup due to NTP because when you need percise time to keep track of objects in a 3D space you really don't want to mess with that over the internet if you can. And let's just say I'm not a fan of the fact timekeeping is UDP so does not guarantee delivery in suboptimal environments.

You may have noticed that port range is within the NodePort range of k8s. This is because I use a reverse proxy due to the fact there are also VMs deployed on the network which have been forwarded to unusual ports due to the network isolation in place.

The single configuration file for each of the 7 VMs currently deployed (largely security isolation) that can be copied to a machine and rebuilt with a single command made building out the CI\CD for the homelab much easier. While a lot of this benefit can be gained from something like Ansible it is really hard to argue with the benefit of getting this functionality built into the distro in a way where everything is basically handled assuming you don't need something not packaged currently.

And strangely I actually quite liked using configuration.nix for storing the docker compose\containers. It's just so elegant to have the OS and containers right there. For example here's my Piwigo configuration:

virtualisation.oci-containers.containers."piwigo_lucia" = { image = "lscr.io/linuxserver/piwigo"; ports = [ "8080:80" ]; extraOptions = [ "--pull=always" ]; environment = { PUID = "1000"; PGID = "1000"; TZ = "Etc/UTC"; }; volumes = [ "/mnt/piwigo-lucia-config:/config" "/mnt/piwigo-lucia-gallery:/gallery" ]; };

What did not work well...or why this is not my desktop distro

NixOS is extremely opionated as far as distros go. You have things like Arch that have a design ethos of the user owns their system and they are fully responsible for the maintenance. They may say x is unsupported (ex AUR helpers have never been officially supported), but nothing stops you from breaking it. You're discouraged from running a custom kernel (or warned of what can happen and that you are on your own), but nothing stops you. NixOS on the otherhand...it will not let you deploy a system that is functional if you do not follow the packaging system.

There's some path anomalies as well. Because there's package prefixes for everything dropping a binary into something like /usr/local/bin is not that simple anymore. On the same note adding a systemd service into something like /etc/systemd/system is not that simple rather you have to write that into your configuration.nix file. Similar to Arch and meta distros (ex Gentoo) the number of installed packages is pretty low, effectively you get a small core of packages, but if you don't explicitly add something you may not have utilities you have grown used to. And whether or not we want to admit it the above means that random script off of Github is not as likely to work if things are not in the path.

Some packages have never been ported to nix or are only available as flakes which means either enabling an experimental feature, it is possible this may change in time, but that's not the case today. While the number of packages is likely to improve over time there are still a few especially when you get to software typically provided by companies (ex we have only tested our software on Ubuntu and RHEL family). So while you might be able to make it work the path is definitely more tedious than it would be otherwise.

Installing a package (actually installing not just opening a nix shell) takes longer. On something like Arch or Ubuntu I can just run a command like sudo apt install htop and immediately my program is available whereas with NixOS I need to add that to my configuration file and trigger a rebuild. Even if this is a small change it's still worth mentioning. And this gets worse if you want to just build something from source because now you need to actually package it rather than just cheat and symbolic link it\add a systemd config.

My biggest complaint is that not all configuration options are exposed the way you expect and you often need to do some cognitive remapping to apply a configuration change. For example I have a dns server running on dnsmasq, there are a few examples online with how to do this and an LLM might even be able to configure it correctly. Here is that same config in NixOS:

services.dnsmasq = { enable = true; settings = { domain = "friedmicro-lab.org"; listen-address = "192.168.1.14"; server = ["1.1.1.1"]; }; }; services.resolved.enable = false; networking.networkmanager.dns = "none"; environment.etc = { "resolv.conf".text = "nameserver 1.1.1.1\n"; }; networking.hosts = { "192.168.1.14" = [ "aurora.friedmicro-lab.org" ];

A wrap up

Overall NixOS was a pretty good addition to the homelab where it works well, but the moment I need to break out of the packaging it gets a lot more tricky and debugging issues. My biggest issue with NixOS has to be the learning curve because while I have been involved with Linux since 2013 and have worked across multiple distros I do not feel like NixOS is a good distro for a beginner because amount of lessons\materials is just so much smaller. Even as a more advanced user it still takes a moment to remap things from a distro like Ubuntu (ex change config x...ok where is that in NixOS...ok why is it broken...got it). And the additional time to install a program or test a code change I have made to the OS is really not a good fit for me on everything.

All that said...it is wonderful for use in a CI and it is quite nice to be able to just add some NFS shares, a new database, etc with a single commit without needing to touch an ansible playbook.