NixOS Homelab with K3S

Welcome to my homelab! This is a massive upgrade for me personally - I have been working with Linux for years and containerization for a while, so I wanted to step up my game and work with Orchestration.

Setup:

I like Pokemon, so naturally I named my three systems after the starters from my original Leaf Green run.

I have a repurposed Chromebox, an HP prodesk, and an old dell latitude. They are nicknamed "bulbasaur", "charmander", and "squirtle" respectively.

For the OS, I wanted something I could set, forget, and easily copy to new machines when I obviously expand my hardware selection. Thus, NixOS was a natural choice.

Tasks:

• Setup FluxCD to automate deployments, monitoring, and scaling
• Re-install Longhorn so Flux can manage configuration
• Replace servicelb with metallb
• Install tailscale kube operator
• Install and configure Prometheus/Grafana
• Setup CloudNativePG for DB storage
• Setup Cert-manager for automated wildcard certificate management
• Setup Bitnami Chart for Redis
• Setup Garage for S3 compatible storage
• Install Mijn Bureau production implementation of La Suite Numerique
• Cry tears of joy

Layout:

I tried to make it simple. It makes sense to me at least. Each machine is funneled into flake.nix

in /hosts, I have all my machines and hardware
in /common, I have common services, networking, and configs
in /modules, I have the specific roles for various machines defined
in /kubes, I have my fluxCD monorepo configs for all my container

• clusters is my fluxCD config folder
• infrastructure holds pods required to run services
• apps is where actual services will run

Issues:

My biggest concern currently is hardware. In the etcd guide it specifically states "Fast disks are the most critical factor for etcd deployment performance and stability"
Currently, my prodesk only has an HDD that I will not be upgrading anytime soon due to PC part pricing :,(

I rescind all previous statements about NixOS. While it is extremely complex to learn being so different from every other Linux distribution, I now manage my entire stack, from operating system to applications, entirely through Git. NixOS automatically updates itself daily from this git repo without any manual intervention.

Another issue is mentally shifting from "Deploying" to "Automating".

docker compose up -d

this deploys an app

kubectl apply -f
# OR
helm install

This also deploys an app

Now, FluxCD does that job and my role is to create the "platform" to get apps running