The lab earns its keep

Everything on this site is backed by somewhere I can break things without billing anyone for the privilege. That somewhere is the lab.

The hardware is not pretty and it is not new. It is a couple of secondhand rack servers, a stack of SAS drives that were cheap because they are slow, and enough memory to run more nodes than I strictly need. It is loud, it heats the room, and it costs me in electricity. It is also the most useful thing I own.

The lab is where I actually learn the things this company runs on. Kubernetes the way it behaves when you have to operate it, not the way it looks in a tutorial: upgrade paths, etcd recovery, draining a node that does not want to drain, swapping an ingress controller without dropping traffic. The Linux underneath it, because half of every Kubernetes problem turns out to be storage, networking or a kernel parameter. And the container platforms on top, comparing distributions and seeing where k3s stops being enough and full kubeadm starts paying for itself.

The rule is simple. If I have not done a thing in the lab, I do not do it for the first time on your cluster. I would rather find out here, at three in the morning, that the documented procedure is wrong, than find out the same way on your production at the same hour.

So when something on this site says we know how a particular failure mode behaves, it usually means I have already caused that failure on purpose, downstairs, and watched it recover. That is where the knowledge comes from. Not blog posts, not conference talks. A noisy rack and a lot of broken clusters that nobody was paying for.