OS orchestration binaries

First, you probably do not need this and should not build this. This is fairly bespoke and esoteric to my very specific needs. And candidly the context on why this code was written and what problem\why it was solved this way is more important than the actual way it was solved.

For context my primary system is a thin client running on a box with a passed through GPU, USB PCIe card, and network card. It also sits behind a double NAT and serves as the jump point as well as the CI\build server for personal projects of mine across my network. Lastly I make heavy use of very latency sensitive software (a target window of 2-3ms for encoding\decoding\transmitting) for game streaming (Sunshine\Moonlight).

My remote box includes a 3 OS install: Windows, Linux (Arch), Proxmox. I regularly bootup whatever I need on this box and either use it directly (such as with VR) or I stream it back (games, software compilation, AI workloads).

Unfortunately the tooling required for remoting orchestrating a 3 OS system is pretty much nonexistant so if you want something like this you pretty much just have to build it yourself.

Why

The thin client is running as a VM because realistically on a system that is used for productivity applications, programming, software builds, and streaming games back to it I don't need to allocate an entire box. Also if I want to keep latency low as well as high quality on the game streaming image I need to have a reasonably powerful GPU for the decoding as the resolution I use being 1440p at 120hz is very bandwidth\computationally heavy.

The double NAT is mostly a security measure and this machine being the jump server is largely because there are very few VMs on the host node. Additionally if you want to do something like development over ssh when traveling you don't want a jump server in there as your dev tooling may not properly handle it.

The CI\CD on here is largely because I test builds on the box and make heavy use of containers so in the interest of minimizing docker image downloads (or just general OCI images) I would rather build on the same box. This saves me bandwidth and saves me time. Additionally, considering the number of people who are actively using my CI is very low I'm not likely to hit issues here and complex binaries for things like a desktop application (or a highly complex microservice) can be easily compiled within my k8s cluster. CI builds are usually under 1 minute compute on the thin client so why bother with a seperate dev box? If you want more info on the docker deployment check here, for more infra on the CI check here, and for more information on the general infra check here.

As for application\game streaming my partner and I's PCs are both located in our VR play spaces which require a large amount of space, but these spaces do not have comfortable room for desks nor would we want to cut into the space available. We have a den where our thin clients are located, but that room traps heat exceptionally well so just deploying additional high performance machines simply is not an option. By the nature of our situation we cannot run all of our software on our local systems which is why we both make heavy use of application streaming. This is also why my application launcher Athena was created.

Why 3 operating systems

The Windows bare metal is because sometimes you want to run a game which simply cannot run in a compatibility layer and requires bare metal access due to an anticheat or interaction with hardware. While Linux compatibility has gotten better it is still not 100% and I personally would like to be able to continue to use all of my software as some of it does not have a free equivalent.

The Linux (Arch) bare metal is the default OS that the system boots into and it's mostly used for remotely streaming games as well as running computationally heavy builds during development tasks. This is for me the main system I offload compute tasks to and is almost exclusively configured for that purpose. Sadly, I cannot boot a VM with hardware passthrough on it correctly\dynamically. Additionally it is also the box I use for VR and if you want more information on how that was setup take a look here which is also how I ended up with writing my AUR CI for Arch. Some consumer hardware will load release the GPU and then pass it into a VM without a system reboot, but sadly this is not one of those systems so I would still have to reboot and also dynamically blacklist the driver. Additionally setting up VFIO on this box would be a fair amount of complexity and while I can do this in Arch\have done it before. There is far better documentation and tooling doing this in something like Proxmox. So while I don't need proxmox at the end of the day it is far faster\smarter to use the existing tooling.

The proxmox install is to allow me to cleanly deploy VMs on the box. That's all the box is for. If I want to spin up a test system with a lot of GPU or run a fairly specialized workload and don't want to mix tooling I can just spin up a VM here.

Could I do fewer operating systems

Yes.

I could retire the Windows system if I didn't have software that required it or if I was willing to loose that software and I have over the years leaned less on it, but sadly some of that is audio related due to my love of Music which Linux is still not great at.

I could also spend several days finding hardware that properly resets and\or add logic to reboot\properly blacklist drivers and rework what proxmox has done for hardware passthrough. But that is a lot of time and potentially a nontrivial amount of money that I do not want to spend.

How it works

If starting the remote box a webhook request will be sent to turn on a smart plug (Wake On Lan is not supported properly on that motherboard and even if it was it wouldn't be super reliable). That PC is set to always power on when power is restored. We then attempt to ssh into the box to check /etc/os-release file. Assuming this is on Arch then we know the remote box started as expected. The next steps depend upon if we are staying in this OS or if we are booting into another. If we are staying the binary exits here.

If we are booting into Windows a script is ran on the Arch OS install that runs the efibootmgr command to set the respective OS to start and we then reboot the box. Additionally we will create a file called /tmp/windows-bare-metal so additional tooling of mine can be aware we are in Windows (currently this is a daemon to control my monitors\screen based upon what a physical KVM is connected to). And lastly we also run a webhook in homeassistant that toggles on all my base stations in the room. Exiting the Windows OS instance simply deletes the file, reboots back into Arch, powers off the OS, and then flips the smart plug.

If we are booting into Proxmox then we run a efibootmgr command to set proxmox as the next boot and then reboot the box into it. We then poll until it actually stays Debian (though Proxmox is also checked in the event this changes down the road). At this point any VMs you want started such as a Windows VM or an AI VM will be started depending on what binary was ran. To power down we spin down the VM, reboot back into Arch, and same process as if we were powering down from Windows.

If we are starting an application stream in addition to starting the Linux box we also spin down the CI (so that new work will not get run while the box is unreachable) and then we switch our network interface online from a virtual nic to a passed through nic that skips the double NAT. The close-nic-port and open-nic-port are automatically granted permissions to run on my system so this is done without interupting any applications using them.

Design decision questions:

More information

A mirror of the code is on Github.

Website Stats:

Website Build Version: 2026/11/20 11:46:51 AM (+00:00)

Last Website Update: 62193f6

Site Generator: Serpent Page Generator created by Lucia Smith