I love my infra stack.
Need to work on a new machine? Checkout main, install task, run `infra:bootstrap` and everything gets pulled down and set up, including all logins, kubectl contexts.
My taskfile gets expanded as I use it. My agents prefer using tasks over writing their own scripts (they're just using a custom skill for it, no magic to it). Output saves significant amounts of tokens because tasks are short and scripts are robust and to the point.
Stack fundamentals are quite boring; nothing too fancy. Every component has a solid release record, active contributors and is easy to patch myself if needed. A YAML-formatted inventory does IP Address Management (IPAM), workload (VMs, Kubernetes, endpoints, etc) and is pruned regularly via automated merge requests from `ci:drift` actions. Proxmox is my hypervisor for VM lifecycle management and basic networking. Talos handles statelessly managed Kubernetes clusters which host most of my public-facing stuff, from relay and blossom to wallhalle as well as client workloads. (If preferred on own VMs or a custom Kubernetes cluster - hit me up if admin stuff annoys you; I take sats/Monero!) Renovate is simply irreplaceable in handling security updates, Keyverno is great at enforcing non-network related policies (what are running apps allowed to do, basically), trivy handles security/CVE scans for me. Monitoring is Prometheus + Grafana (currently run by Grafana.com) with alerting going to ntfy, with on-call rules pushing alerts through my DND for paying clients.
All of it is wired up to the tailnet. Network policies guard namespaces and ensure no internet-facing app can reach anything of value. All containers run rootless. DNS management is completely automated. Add a new service with an httproute (exposing it to the internet) and records get auto-created with LE certificates. All workloads are managed via gitops (flux), gitea actions; all changes go through automated tests (linters, bats for shell scripts, etc). Backups land end-to-end encrypted in S3 buckets at a paid 3rd party service managed by Velero. Secrets are stored with age in SOPS and get put into environment variables by tasks as well so agents never get to see any credentials. If one were to be leaked and had to be rotated, there's another task for that.
Thought I'd regret not going with terraform and Ansible but so far, my task scripts are doing a great job of handling state drift, config management. 2 fewer tools to babysit and terraform + CI never felt right; always pain with some state invalidation issues. My tasks default to output to terraform-style diffs. Great feature.
Ansible is also really clunky to use with agents because they tend to trip over its "the more shit we can encapsulate in YAML, the better" pattern. I think it paid off to trust my intuition here and go my own way.
My local agent is doing weekly git history diffs combined with some probe results to create issues whenever there's significant drift that something like Ansible wouldn't have left behind (a leftover config wasn't removed, some package got installed and not removed when no longer needed, nothing critical but it might grow to become messy); so far it happened once during initial setup of this entire stack; while my `ci:drift` checks found some duplicate aliases for a task before they ever landed in main. If my more complex agent doesn't find anything by the end of year it's just wasting resources and is going to go.
This way of working feels like what I always imagined Kubernetes to behave like at some point. No more YAML acrobatics, no fiddling with terrible Helm templating, no need for writing different querying syntaxes โ just build a monitoring dashboard for some new service before it gets deployed. New services are online faster than most SaaS I've used at previous jobs could've achieved.
With more patience I could easily offload driving (with me steering, no crazy stuff!) the stack to a local Qwen agent but it'd be too slow for me right now and I have some tokens to spare for the speed. But it feels really quite sovereign.
Perhaps I should try to distill it to the basics, optimize the out of the box experience and just launch something Omarchy-like for SRE/devops folks and homelabbers. ๐
Need to work on a new machine? Checkout main, install task, run `infra:bootstrap` and everything gets pulled down and set up, including all logins, kubectl contexts.
My taskfile gets expanded as I use it. My agents prefer using tasks over writing their own scripts (they're just using a custom skill for it, no magic to it). Output saves significant amounts of tokens because tasks are short and scripts are robust and to the point.
Stack fundamentals are quite boring; nothing too fancy. Every component has a solid release record, active contributors and is easy to patch myself if needed. A YAML-formatted inventory does IP Address Management (IPAM), workload (VMs, Kubernetes, endpoints, etc) and is pruned regularly via automated merge requests from `ci:drift` actions. Proxmox is my hypervisor for VM lifecycle management and basic networking. Talos handles statelessly managed Kubernetes clusters which host most of my public-facing stuff, from relay and blossom to wallhalle as well as client workloads. (If preferred on own VMs or a custom Kubernetes cluster - hit me up if admin stuff annoys you; I take sats/Monero!) Renovate is simply irreplaceable in handling security updates, Keyverno is great at enforcing non-network related policies (what are running apps allowed to do, basically), trivy handles security/CVE scans for me. Monitoring is Prometheus + Grafana (currently run by Grafana.com) with alerting going to ntfy, with on-call rules pushing alerts through my DND for paying clients.
All of it is wired up to the tailnet. Network policies guard namespaces and ensure no internet-facing app can reach anything of value. All containers run rootless. DNS management is completely automated. Add a new service with an httproute (exposing it to the internet) and records get auto-created with LE certificates. All workloads are managed via gitops (flux), gitea actions; all changes go through automated tests (linters, bats for shell scripts, etc). Backups land end-to-end encrypted in S3 buckets at a paid 3rd party service managed by Velero. Secrets are stored with age in SOPS and get put into environment variables by tasks as well so agents never get to see any credentials. If one were to be leaked and had to be rotated, there's another task for that.
Thought I'd regret not going with terraform and Ansible but so far, my task scripts are doing a great job of handling state drift, config management. 2 fewer tools to babysit and terraform + CI never felt right; always pain with some state invalidation issues. My tasks default to output to terraform-style diffs. Great feature.
Ansible is also really clunky to use with agents because they tend to trip over its "the more shit we can encapsulate in YAML, the better" pattern. I think it paid off to trust my intuition here and go my own way.
My local agent is doing weekly git history diffs combined with some probe results to create issues whenever there's significant drift that something like Ansible wouldn't have left behind (a leftover config wasn't removed, some package got installed and not removed when no longer needed, nothing critical but it might grow to become messy); so far it happened once during initial setup of this entire stack; while my `ci:drift` checks found some duplicate aliases for a task before they ever landed in main. If my more complex agent doesn't find anything by the end of year it's just wasting resources and is going to go.
This way of working feels like what I always imagined Kubernetes to behave like at some point. No more YAML acrobatics, no fiddling with terrible Helm templating, no need for writing different querying syntaxes โ just build a monitoring dashboard for some new service before it gets deployed. New services are online faster than most SaaS I've used at previous jobs could've achieved.
With more patience I could easily offload driving (with me steering, no crazy stuff!) the stack to a local Qwen agent but it'd be too slow for me right now and I have some tokens to spare for the speed. But it feels really quite sovereign.
Perhaps I should try to distill it to the basics, optimize the out of the box experience and just launch something Omarchy-like for SRE/devops folks and homelabbers. ๐
4