[{"content":"","date":"September 8, 2026","externalUrl":null,"permalink":"/categories/","section":"Categories","summary":"","title":"Categories","type":"categories"},{"content":"","date":"September 8, 2026","externalUrl":null,"permalink":"/categories/debugging/","section":"Categories","summary":"","title":"Debugging","type":"categories"},{"content":"","date":"September 8, 2026","externalUrl":null,"permalink":"/categories/homelab/","section":"Categories","summary":"","title":"Homelab","type":"categories"},{"content":"My cluster — a single-node K3s instance running about 60 ArgoCD-managed applications — had been slowly dying for months. Pods crash-looped with \u0026ldquo;context deadline exceeded\u0026rdquo; errors. Leader elections failed. The metrics server stopped working entirely. I assumed it was resource pressure and tuned probe timeouts. The real problem was underneath all of it: K3s\u0026rsquo;s SQLite datastore had silently grown to 40 GB.\nThe symptoms # Everything pointed at the API server. Operators that use leader election kept losing their leases:\nFailed to update lock: Put \u0026#34;https://10.43.0.1:443/.../leases/my-operator\u0026#34;: context deadline exceeded failed to renew lease: timed out waiting for the condition kube-state-metrics crash-looped because its liveness probe got HTTP 503 — it couldn\u0026rsquo;t reach the API server\u0026rsquo;s /livez endpoint. The metrics-server hadn\u0026rsquo;t served data in over a month. And the API server health endpoints themselves confirmed it:\n$ kubectl get --raw /readyz [-]etcd failed: reason withheld [-]etcd-readiness failed: reason withheld readyz check failed But there was no etcd. This is K3s.\nK3s and kine # K3s doesn\u0026rsquo;t run etcd by default. On single-node installs, it uses SQLite through a translation layer called kine that makes SQLite look like etcd to the Kubernetes API server. The health endpoint still says \u0026ldquo;etcd\u0026rdquo; because the API server doesn\u0026rsquo;t know the difference.\nThe kine schema is simple. There\u0026rsquo;s one table — kine — and every Kubernetes API write (creating a pod, updating a deployment, renewing a lease) inserts a new row. Old revisions are supposed to be compacted away automatically. On my cluster, they weren\u0026rsquo;t.\n7.2 million rows # SELECT COUNT(*) FROM kine; -- 7223033 Over 7 million rows. The top offenders were all leader election leases:\nKey Rows Operator A lease 1,285,730 Operator B lease 1,049,397 Deleted ingress controller leader 448,295 API server master lease 375,136 Node lease 339,453 cert-manager leases (x2) ~236,000 each Storage operator leases (x6) ~139,000 each Each operator renews its lease every few seconds. Each renewal is an INSERT. With no working compaction, these rows accumulated over the entire lifetime of the cluster.\nThe database file:\n-rw-r--r-- 1 root root 40G state.db -rw-r--r-- 1 root root 22G state.db-wal 62 GB total. The k3s server process had ballooned to 49 GB of RAM trying to manage it. The machine had enough RAM that it wasn\u0026rsquo;t OOM-killed — it just made everything slow.\nThe vicious cycle # This is the part that made it hard to diagnose. The crash-looping pods weren\u0026rsquo;t just victims of the slow database — they were making it worse.\nThe database gets slow enough that a lease renewal takes \u0026gt;10 seconds The operator\u0026rsquo;s leader election times out (default deadline: 10 seconds) The operator crashes and restarts On startup, it immediately tries to acquire the lease — a burst of writes Those writes make the database slower Repeat from step 1 One operator was the worst case. It runs a heavy reconciliation on startup (~4 minutes of CPU-intensive work), during which it can\u0026rsquo;t renew its lease. So it would start, begin reconciling, fail to renew, crash, restart, and reconcile again — each cycle adding dozens of rows to the database.\nThe fix # The compaction had to happen with k3s stopped. Running containers keep operating when the API server is down — they just can\u0026rsquo;t be scheduled or updated — so the impact is limited.\n# Stop k3s sudo systemctl stop k3s # Back up first sudo cp /var/lib/rancher/k3s/server/db/state.db \\ /var/lib/rancher/k3s/server/db/state.db.bak-$(date +%Y%m%d) # Compact in a single sqlite3 session: checkpoint WAL, delete old # revisions (keep only the latest per key), then VACUUM to reclaim space sudo sqlite3 /var/lib/rancher/k3s/server/db/state.db \u0026lt;\u0026lt; \u0026#39;EOF\u0026#39; PRAGMA wal_checkpoint(TRUNCATE); DELETE FROM kine WHERE id NOT IN (SELECT MAX(id) FROM kine GROUP BY name); VACUUM; EOF # Restart k3s sudo systemctl start k3s The DELETE took about 15 minutes on the 40 GB database. VACUUM took another 90 seconds. The result:\nBefore: 40 GB database, 22 GB WAL, 7.2M rows, 49 GB k3s RSS After: 271 MB database, 134K rows, ~1 GB k3s RSS A 99.3% reduction in database size.\nWhy compaction broke # K3s\u0026rsquo;s kine driver has a built-in compaction routine that should prevent this. Looking through the logs after the fix, I found zero compaction-related entries — it appears the compaction goroutine either wasn\u0026rsquo;t running or was blocked by the same database contention it was supposed to prevent.\nThis is a known class of issue. The kine compaction runs as part of the k3s process, using the same SQLite connection pool. When the database is under heavy write load (hundreds of lease renewals per minute), the compaction queries compete with the writes and can time out or get starved. Once the database is large enough that compaction queries are slow, they\u0026rsquo;ll never catch up.\nPreventing it from happening again # Beyond the database fix, I also addressed the pods that were generating the most write amplification:\nRelaxed leader election timeouts. The worst-offending operator had a default lease duration of 15 seconds with a 10-second renew deadline. On a single-node cluster where there\u0026rsquo;s no other replica to take over, aggressive leader election is pure overhead. I increased the lease duration to 120 seconds and the renew deadline to 90 seconds. Similar changes for other operators\u0026rsquo; liveness probes.\nSet resource requests. kube-state-metrics was running as BestEffort (no resource requests), making it the first eviction candidate under memory pressure. Being evicted means restarting, which means more lease churn. Adding a 64Mi request moved it to Burstable QoS.\nRemoved dead state. A deleted ingress controller\u0026rsquo;s leader lease still had 448K rows — the controller was removed months ago, but the lease key kept accumulating rows from other controllers trying to clean it up.\nMonitoring so it doesn\u0026rsquo;t happen again # I also added Prometheus alerting for the database size. The tricky part: K3s holds the SQLite database open with an exclusive lock during heavy writes, so you can\u0026rsquo;t always query the row count. But you can always stat the file.\nI enabled the textfile collector on node-exporter and wrote a small cron script that runs every 5 minutes:\n#!/bin/bash # /usr/local/bin/k3s-db-metrics.sh OUTPUT=/var/lib/node-exporter/textfile/k3s_kine.prom DB=/var/lib/rancher/k3s/server/db/state.db [ -f \u0026#34;$DB\u0026#34; ] || exit 0 DB_SIZE=$(stat -c %s \u0026#34;$DB\u0026#34; 2\u0026gt;/dev/null || echo 0) WAL_SIZE=0 for f in \u0026#34;${DB}-wal\u0026#34; \u0026#34;${DB}-journal\u0026#34;; do [ -f \u0026#34;$f\u0026#34; ] \u0026amp;\u0026amp; WAL_SIZE=$(($WAL_SIZE + $(stat -c %s \u0026#34;$f\u0026#34;))) done # Row count times out gracefully if the DB is locked ROW_COUNT=$(timeout 5 sqlite3 \u0026#34;$DB\u0026#34; \u0026#34;SELECT COUNT(*) FROM kine;\u0026#34; 2\u0026gt;/dev/null) ROW_COUNT=${ROW_COUNT:--1} cat \u0026gt; \u0026#34;${OUTPUT}.tmp\u0026#34; \u0026lt;\u0026lt; EOF # HELP k3s_kine_total_size_bytes Total size of K3s kine database including WAL/journal # TYPE k3s_kine_total_size_bytes gauge k3s_kine_total_size_bytes $(($DB_SIZE + $WAL_SIZE)) # HELP k3s_kine_row_count Number of rows in the kine table (-1 if locked) # TYPE k3s_kine_row_count gauge k3s_kine_row_count ${ROW_COUNT} EOF mv \u0026#34;${OUTPUT}.tmp\u0026#34; \u0026#34;$OUTPUT\u0026#34; The key insight is that stat never blocks on SQLite locks, so the file size metric is always available. The row count uses a 5-second timeout and reports -1 if the database is busy — Prometheus can filter those out.\nThe alert rules fire on total size (database + WAL/journal combined):\n- alert: K3sKineDBSizeLarge expr: k3s_kine_total_size_bytes \u0026gt; 2e+09 for: 1h labels: severity: warning - alert: K3sKineDBSizeCritical expr: k3s_kine_total_size_bytes \u0026gt; 5e+09 for: 30m labels: severity: critical - alert: K3sKineRowCountHigh expr: k3s_kine_row_count \u0026gt; 500000 for: 1h labels: severity: warning 2 GB is a generous threshold — my compacted database is 271 MB. If it ever hits 2 GB, something has gone wrong with compaction and I\u0026rsquo;ll get a Pushover notification before it spirals.\nIf you\u0026rsquo;re running K3s with SQLite (the default for single-node), check your database size. If it\u0026rsquo;s over a few hundred megabytes, you might be on the same path.\n","date":"September 8, 2026","externalUrl":null,"permalink":"/posts/k3s-sqlite-kine-40gb-database-compaction/","section":"Posts","summary":"My cluster — a single-node K3s instance running about 60 ArgoCD-managed applications — had been slowly dying for months. Pods crash-looped with “context deadline exceeded” errors. Leader elections failed. The metrics server stopped working entirely. I assumed it was resource pressure and tuned probe timeouts. The real problem was underneath all of it: K3s’s SQLite datastore had silently grown to 40 GB.\n","title":"How K3s Silently Grew a 40 GB SQLite Database and Took Down My Cluster","type":"posts"},{"content":"","date":"September 8, 2026","externalUrl":null,"permalink":"/tags/k3s/","section":"Tags","summary":"","title":"K3s","type":"tags"},{"content":"","date":"September 8, 2026","externalUrl":null,"permalink":"/tags/kine/","section":"Tags","summary":"","title":"Kine","type":"tags"},{"content":"","date":"September 8, 2026","externalUrl":null,"permalink":"/categories/kubernetes/","section":"Categories","summary":"","title":"Kubernetes","type":"categories"},{"content":"","date":"September 8, 2026","externalUrl":null,"permalink":"/posts/","section":"Posts","summary":"","title":"Posts","type":"posts"},{"content":"","date":"September 8, 2026","externalUrl":null,"permalink":"/","section":"Rusty Bower","summary":"","title":"Rusty Bower","type":"page"},{"content":"","date":"September 8, 2026","externalUrl":null,"permalink":"/tags/sqlite/","section":"Tags","summary":"","title":"Sqlite","type":"tags"},{"content":"","date":"September 8, 2026","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"","date":"August 25, 2026","externalUrl":null,"permalink":"/tags/fire-tablet/","section":"Tags","summary":"","title":"Fire-Tablet","type":"tags"},{"content":"I have three Amazon Fire tablets mounted around the house running Fully Kiosk Browser as Home Assistant dashboards. They\u0026rsquo;re cheap ($35-50 each during sales), they have decent screens, and Fully Kiosk turns them into locked-down kiosks that show a dashboard and nothing else.\nThe problem is configuring them. Each tablet has 60+ settings — kiosk lockdown, screensaver behavior, motion detection wake, screen brightness, crash recovery. Changing a setting means opening the Fully Kiosk admin UI on each tablet individually, navigating through menus, and tapping through options three times. When you inevitably forget one tablet, it behaves differently from the others in a way you don\u0026rsquo;t notice for weeks.\nI wanted the same thing I want for everything else in the homelab: a config file in Git that describes the desired state, applied automatically.\nThe approach # Fully Kiosk exposes a REST API on port 2323. You can read device info, push settings, trigger reloads, and control the screen. The key endpoints for config management are setStringSetting and setBooleanSetting:\nhttp://\u0026lt;tablet-ip\u0026gt;:2323/?cmd=setStringSetting\u0026amp;key=startURL\u0026amp;value=https://...\u0026amp;password=\u0026lt;pw\u0026gt; http://\u0026lt;tablet-ip\u0026gt;:2323/?cmd=setBooleanSetting\u0026amp;key=kioskMode\u0026amp;value=true\u0026amp;password=\u0026lt;pw\u0026gt; So the plan: define all settings in a config file, iterate over tablets, push each setting via curl. Run it as a Kubernetes CronJob every six hours to enforce drift correction.\nThe config file # Settings live in a plain text file with a typed format — type:key=value:\n# Kiosk Mode — lock down so users can\u0026#39;t exit bool:kioskMode=true bool:lockSafeMode=true bool:advancedKioskProtection=true bool:disableHomeButton=true bool:disablePowerButton=true bool:disableVolumeButtons=true # Screen Unlock — bypass Android lock screen bool:forceScreenUnlock=true bool:forceSwipeUnlock=true bool:launchOnBoot=true # Display — clean dashboard look bool:showNavigationBar=false bool:showStatusBar=false string:startURL=https://ha.example.com/home-yaml/overview # Screensaver — dim to black after 90s, wake on motion int:screenBrightness=160 int:timeToScreensaverV2=90000 string:screensaverType=black int:screensaverBrightness=0 bool:screenOffInDarkness=true # Motion Detection — front camera wakes screen on presence bool:motionDetection=true bool:screenOnOnMotion=true int:movementSensitivity=80 int:motionDetectionInterval=500 # Stability — auto-recover from crashes bool:restartAfterCrash=true bool:restartAfterUpdate=true bool:reloadOnConnectivityChange=true # Remote Admin — keep REST API enabled (required for this sync) bool:enableRemoteAdmin=true int:remoteAdminPort=2323 The type prefix (bool:, string:, int:) maps to the correct Fully Kiosk API command. Comments and blank lines are ignored. The file is human-readable and diffable — when I change a setting, the Git diff shows exactly what changed.\nThe sync script # A short shell script reads the config and pushes each setting to each tablet:\n#!/bin/sh push_setting() { local tablet=\u0026#34;$1\u0026#34; type=\u0026#34;$2\u0026#34; key=\u0026#34;$3\u0026#34; value=\u0026#34;$4\u0026#34; local cmd case \u0026#34;$type\u0026#34; in string|int) cmd=\u0026#34;setStringSetting\u0026#34; ;; bool) cmd=\u0026#34;setBooleanSetting\u0026#34; ;; esac local url=\u0026#34;http://${tablet}:${FK_PORT}/?cmd=${cmd}\u0026amp;key=${key}\u0026amp;value=${value}\u0026amp;password=${FK_PASSWORD}\u0026#34; http_code=$(curl -s -o /dev/null -w \u0026#34;%{http_code}\u0026#34; -m 30 \u0026#34;$url\u0026#34;) || true if [ \u0026#34;$http_code\u0026#34; = \u0026#34;200\u0026#34; ]; then echo \u0026#34; OK: ${key}=${value}\u0026#34; else echo \u0026#34; FAIL: ${key} (HTTP ${http_code})\u0026#34; fi } IFS=\u0026#39;,\u0026#39; for tablet in $TABLETS; do echo \u0026#34;=== Syncing $tablet ===\u0026#34; # Check connectivity first if ! curl -s -m 30 \u0026#34;http://${tablet}:${FK_PORT}/?cmd=deviceInfo\u0026amp;password=${FK_PASSWORD}\u0026#34; \u0026gt; /dev/null 2\u0026gt;\u0026amp;1; then echo \u0026#34; WARN: Tablet unreachable, skipping\u0026#34; continue fi # Push each setting while IFS= read -r line; do case \u0026#34;$line\u0026#34; in \\#*|\u0026#34;\u0026#34;) continue ;; esac type=$(echo \u0026#34;$line\u0026#34; | cut -d: -f1) rest=$(echo \u0026#34;$line\u0026#34; | cut -d: -f2-) key=$(echo \u0026#34;$rest\u0026#34; | cut -d= -f1) value=$(echo \u0026#34;$rest\u0026#34; | cut -d= -f2-) push_setting \u0026#34;$tablet\u0026#34; \u0026#34;$type\u0026#34; \u0026#34;$key\u0026#34; \u0026#34;$value\u0026#34; done \u0026lt; \u0026#34;$CONFIG_FILE\u0026#34; done It\u0026rsquo;s deliberately simple — curl in a loop. No Python, no dependencies beyond a shell and curl. The curlimages/curl container image is 11 MB.\nThe connectivity pre-check is important. Fire tablets go to sleep, lose Wi-Fi, or occasionally just stop responding. Without the check, curl would time out on every setting for unreachable tablets, turning a 30-second job into a 15-minute job. With it, unreachable tablets are skipped in one second.\nThe CronJob # apiVersion: batch/v1 kind: CronJob metadata: name: fullykiosk-sync spec: schedule: \u0026#34;0 */6 * * *\u0026#34; timeZone: \u0026#34;America/Chicago\u0026#34; concurrencyPolicy: Forbid jobTemplate: spec: template: spec: containers: - name: sync image: curlimages/curl:8.14.1 command: [\u0026#34;/bin/sh\u0026#34;, \u0026#34;/scripts/sync.sh\u0026#34;] env: - name: FK_PASSWORD valueFrom: secretKeyRef: name: fullykiosk-credentials key: password - name: TABLETS valueFrom: configMapKeyRef: name: fullykiosk-settings key: tablets volumeMounts: - name: scripts mountPath: /scripts - name: settings mountPath: /config volumes: - name: scripts configMap: name: fullykiosk-sync-scripts - name: settings configMap: name: fullykiosk-settings The tablet IP list is a comma-separated string in a ConfigMap literal. Adding or removing a tablet is a one-line change. The Fully Kiosk password is a Sealed Secret.\nThe environment overlay wires everything together:\n# environments/bowerhaus/apps/fullykiosk-sync/kustomization.yaml configMapGenerator: - name: fullykiosk-settings literals: - tablets=192.168.1.51,192.168.1.52,192.168.1.53 files: - settings.conf The companion: battery management # Config sync handles the software side. The hardware side — specifically, not swelling the batteries by leaving them perpetually charging — is handled by an AppDaemon automation that manages smart plugs:\nclass TabletCharger(hass.Hass): def initialize(self): self.tablets = { \u0026#34;kitchen\u0026#34;: {\u0026#34;battery\u0026#34;: \u0026#34;sensor.kitchen_tablet_battery\u0026#34;, \u0026#34;plug\u0026#34;: \u0026#34;switch.kitchen_charger\u0026#34;, \u0026#34;low\u0026#34;: 25, \u0026#34;high\u0026#34;: 80}, \u0026#34;loft\u0026#34;: {\u0026#34;battery\u0026#34;: \u0026#34;sensor.loft_tablet_battery\u0026#34;, \u0026#34;plug\u0026#34;: \u0026#34;switch.loft_charger\u0026#34;, \u0026#34;low\u0026#34;: 25, \u0026#34;high\u0026#34;: 80}, \u0026#34;charging_station\u0026#34;:{\u0026#34;battery\u0026#34;: \u0026#34;sensor.station_tablet_battery\u0026#34;, \u0026#34;plug\u0026#34;: \u0026#34;switch.station_charger\u0026#34;, \u0026#34;low\u0026#34;: 25, \u0026#34;high\u0026#34;: 80}, } for name, cfg in self.tablets.items(): self.listen_state(self.battery_check, cfg[\u0026#34;battery\u0026#34;], tablet=name) self.run_every(self.periodic_check, self.datetime(), 300) Each tablet sits on a Shelly smart plug. When the battery drops to 25%, the plug turns on. At 80%, it turns off. A 5-minute periodic sweep catches any state changes the event listener might miss. There\u0026rsquo;s a hard cutoff at 100% as a safety net.\nThe 25-80% range is a compromise. Lithium batteries last longest between 20-80%, but fire tablets\u0026rsquo; battery sensors aren\u0026rsquo;t precise enough below 20% to be safe. 25% gives enough margin that a tablet won\u0026rsquo;t die between the battery reading and the plug turning on.\nTwo systems, one fleet # The full tablet management stack is:\nLayer Tool What it manages Frequency Software config K8s CronJob + curl 40+ Fully Kiosk settings Every 6 hours Battery health AppDaemon + Shelly plugs Charge cycles (25-80%) Event-driven + 5 min sweep Dashboard content Home Assistant Lovelace dashboards Real-time The tablets themselves are stateless from a configuration perspective. If one dies, I can buy a $35 Fire tablet, install Fully Kiosk, point it at the HA URL, and the CronJob configures everything else within six hours. The settings are the same because they come from a config file in Git, not from someone tapping through menus and trying to remember what they set last time.\nThis is probably the lowest-stakes use of Kubernetes CronJobs in my cluster. But it solves a real problem — keeping three devices identically configured without manual effort — using the same patterns (Git-tracked config, sealed secrets, scheduled reconciliation) that manage everything else.\n","date":"August 25, 2026","externalUrl":null,"permalink":"/posts/fire-tablet-fleet-management-kubernetes/","section":"Posts","summary":"I have three Amazon Fire tablets mounted around the house running Fully Kiosk Browser as Home Assistant dashboards. They’re cheap ($35-50 each during sales), they have decent screens, and Fully Kiosk turns them into locked-down kiosks that show a dashboard and nothing else.\n","title":"Fleet-Managing Fire Tablets from Kubernetes","type":"posts"},{"content":"","date":"August 25, 2026","externalUrl":null,"permalink":"/tags/home-assistant/","section":"Tags","summary":"","title":"Home-Assistant","type":"tags"},{"content":"","date":"August 25, 2026","externalUrl":null,"permalink":"/categories/home-automation/","section":"Categories","summary":"","title":"Home-Automation","type":"categories"},{"content":"","date":"August 11, 2026","externalUrl":null,"permalink":"/categories/audio/","section":"Categories","summary":"","title":"Audio","type":"categories"},{"content":"","date":"August 11, 2026","externalUrl":null,"permalink":"/tags/music-assistant/","section":"Tags","summary":"","title":"Music-Assistant","type":"tags"},{"content":"","date":"August 11, 2026","externalUrl":null,"permalink":"/tags/picoreplayer/","section":"Tags","summary":"","title":"Picoreplayer","type":"tags"},{"content":"","date":"August 11, 2026","externalUrl":null,"permalink":"/tags/raspberry-pi/","section":"Tags","summary":"","title":"Raspberry-Pi","type":"tags"},{"content":"","date":"August 11, 2026","externalUrl":null,"permalink":"/tags/subsonic/","section":"Tags","summary":"","title":"Subsonic","type":"tags"},{"content":"My music setup had accreted into a mess. Across two Kubernetes clusters I was running Navidrome, Lyrion Music Server (LMS), beets, and my own AI radio station (SUB/WAVE) - plus a Raspberry Pi running piCorePlayer wired into a Dayton DAX66 matrix amp for whole-home audio. Four things that all \u0026ldquo;played music,\u0026rdquo; none of which talked to each other, and no single place to control any of it from Home Assistant.\nI wanted one hub. This is the story of consolidating all of it onto Music Assistant, and every gotcha that fought me on the way there - most of which come down to the same theme: software that assumes it can see the network the same way its clients do, running inside Kubernetes, where it can\u0026rsquo;t.\nWhat I had, and what I wanted # The pieces, and what each actually does:\nComponent Role beets Library tagging/ingest - writes clean files to shared storage Navidrome Subsonic API + web/phone player; also the library backend for my radio station Lyrion (LMS) SlimProto server for the piCorePlayer / Squeezebox client SUB/WAVE AI internet-radio station (liquidsoap → Icecast), pulls tracks from Navidrome piCorePlayer Raspberry Pi player wired into the DAX66\u0026rsquo;s line input The trap is thinking Navidrome and LMS are redundant - \u0026ldquo;two music servers reading the same library.\u0026rdquo; They\u0026rsquo;re not. They speak different protocols to different clients: Navidrome is Subsonic (web/phone apps), LMS is SlimProto (the Squeezebox hardware protocol the Pi uses). The real redundancy was that LMS\u0026rsquo;s only job was to drive the Pi, and Music Assistant can do that itself - it has a built-in SlimProto server.\nSo the target:\nNavidrome (library, Subsonic API) │ ▼ Music Assistant ──SlimProto──► piCorePlayer ──► DAX66 ──► whole-home audio (the hub: browse, queue, (the player) (the amp/matrix) transport, HA integration) Music Assistant becomes the brain - it reads Navidrome over Subsonic, drives the Pi over SlimProto, exposes every zone to Home Assistant as a media_player entity, and can throw SUB/WAVE or Spotify or a radio stream at the same speakers. Navidrome stays (SUB/WAVE needs it, and it\u0026rsquo;s MA\u0026rsquo;s library). LMS gets retired.\nSimple plan. Then I turned it on.\nGotcha #1: Authelia breaks the Subsonic API # First step: point Music Assistant\u0026rsquo;s Subsonic provider at Navidrome. Instant failure:\nAttempt to decode JSON with unexpected mimetype: text/html; charset=utf-8 url=\u0026#39;https://auth.rustybower.com/?rd=https://navidrome.rustybower.com/rest/ping.view\u0026#39; Navidrome sits behind Authelia forward-auth via a Traefik middleware. The Subsonic client hit /rest/ping.view, Authelia bounced it to an HTML login page, and the client choked trying to parse HTML as JSON.\nThis isn\u0026rsquo;t Music-Assistant-specific - any Subsonic client (phone apps included) hits the same wall, because API clients can\u0026rsquo;t complete an interactive browser login. The Subsonic API authenticates itself with its own credentials; putting forward-auth in front of it is a category error.\nThe fix is to let the API path bypass Authelia while keeping the web UI protected. Traefik applies middleware per-router, and its default router priority is by rule length, so a second Ingress matching the more-specific /rest prefix - with no auth middleware - wins for API traffic:\napiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: navidrome-subsonic # No Authelia here: the Subsonic API (/rest/*) uses its own credentials. spec: tls: - hosts: [navidrome.rustybower.com] secretName: navidrome-tls rules: - host: navidrome.rustybower.com http: paths: - path: /rest pathType: Prefix backend: service: {name: navidrome, port: {number: 4533}} ingressClassName: traefik Verify from a client\u0026rsquo;s perspective - /rest should return a Subsonic error (not an auth redirect), while / still bounces to Authelia:\n$ curl -s https://navidrome.rustybower.com/rest/ping.view \u0026lt;subsonic-response status=\u0026#34;failed\u0026#34; ...\u0026gt;\u0026lt;error code=\u0026#34;10\u0026#34; message=\u0026#34;missing parameter: \u0026#39;u\u0026#39;\u0026#34;/\u0026gt;\u0026lt;/subsonic-response\u0026gt; $ curl -sI https://navidrome.rustybower.com/ | grep location location: https://auth.rustybower.com/?rd=... The \u0026ldquo;missing parameter\u0026rdquo; error is exactly what you want: the API is reachable and now only needs Subsonic credentials. Library imported.\nGotcha #2: Music Assistant advertises its pod IP (the big one) # Now the player. I added the Squeezelite provider in MA, pointed piCorePlayer at MA\u0026rsquo;s LoadBalancer IP, and it connected. MA showed the track \u0026ldquo;playing.\u0026rdquo; No sound.\nThis one ate hours, because everything looked right. The player was connected, the control link was up, Home Assistant even flapped the player between \u0026ldquo;available\u0026rdquo; and \u0026ldquo;unavailable\u0026rdquo; every couple minutes, which sent me down a rabbit hole about SlimProto connection stability and MetalLB externalTrafficPolicy (a red herring - more below).\nThe answer was in the player\u0026rsquo;s own debug log. SlimProto is two channels: a control connection, and a separate HTTP audio stream the player fetches when told to play. When I hit play:\nprocess_strm: strm command s # START - I pressed play connect_socket: connecting to 10.42.4.197:8097 10.42.4.197 is Music Assistant\u0026rsquo;s Kubernetes pod IP. It\u0026rsquo;s handing the player an address that only exists inside the cluster\u0026rsquo;s pod network - completely unroutable from a Raspberry Pi on the LAN. The control link worked (the Pi reached MA\u0026rsquo;s LoadBalancer IP fine), but the moment it tried to pull audio, it was sent into the void.\nMA detects \u0026ldquo;its\u0026rdquo; IP from the network interface it\u0026rsquo;s running on. In a pod, that\u0026rsquo;s the pod IP. Its own docs recommend host networking for exactly this reason - but there\u0026rsquo;s a cleaner fix that keeps the LoadBalancer intact. Buried in Settings → Streams is a Published IP address field, and it was set to the pod IP. Change it to the LoadBalancer IP:\nPublished IP address: 172.30.0.5 (the MetalLB IP - what MA advertises) TCP Port: 8097 (reachable through the LoadBalancer) Bind to IP/interface: 0.0.0.0 (leave it - the pod can\u0026rsquo;t bind an IP it doesn\u0026rsquo;t own) The subtlety that trips people: published and bind are different. Published is the address MA hands to clients; it must be the reachable LoadBalancer IP. Bind is where MA actually listens inside the pod; it must stay 0.0.0.0, because 172.30.0.5 lives on MetalLB, not on the pod\u0026rsquo;s interface. Set bind to the LoadBalancer IP and the server won\u0026rsquo;t start.\nOne more trap: use the raw LoadBalancer IP, not the ingress hostname. musicassistant.bowerha.us goes through Traefik on ports 80/443, which doesn\u0026rsquo;t expose the stream port 8097. The LoadBalancer IP does.\nAfter the change, the player finally opened a stream socket to the right place:\ntcp 367537 0 10.0.10.198:34266 → 172.30.0.5:8097 ESTABLISHED 368 KB buffered and flowing. And still\u0026hellip; quiet. Because I\u0026rsquo;d been fighting a second problem the whole time.\nGotcha #3: the Raspberry Pi headphone jack\u0026rsquo;s brutal volume curve # The Pi outputs through its onboard 3.5mm jack (the bcm2835 \u0026ldquo;Headphones\u0026rdquo; device). A laptop plugged into the same DAX66 input played fine, so the amp and speakers were good. squeezelite was connected and streaming. But the mixer told the real story:\nSimple mixer control \u0026#39;PCM\u0026#39; Mono: Playback -3856 [60%] [-38.56dB] [on] [on], not muted, at 60% - so the naive read is \u0026ldquo;that\u0026rsquo;s fine.\u0026rdquo; It is not fine. The bcm2835\u0026rsquo;s volume control is logarithmic with a range that bottoms out around -102 dB, so \u0026ldquo;60%\u0026rdquo; is -38 dB - technically audible, practically silence into a line input. Crank it to 100% and it\u0026rsquo;s a healthy +4 dB:\namixer sset PCM 100% unmute A test tone straight to the device confirmed the hardware path end to end:\nspeaker-test -D plughw:CARD=Headphones -c2 -t sine -f 440 Two related pitfalls while I was in here:\nhw: vs plughw:. squeezelite was configured with -o hw:CARD=Headphones, the raw device with no format conversion - it only plays if the stream\u0026rsquo;s sample rate exactly matches what the chip accepts. plughw:CARD=Headphones inserts ALSA\u0026rsquo;s plug layer for automatic conversion. On finicky onboard audio, always use plughw. Software vs hardware volume. squeezelite (with no hardware-mixer flag) attenuates in software from the volume Music Assistant sends. For a matrix amp like the DAX66, you want the opposite: a constant line-level feed from the Pi, with per-room volume handled by the amp. So keep the MA player near 100% and control loudness at the DAX66 zone. Gotcha #4: persisting mixer state on a RAM-based player # piCorePlayer runs from RAM (it\u0026rsquo;s TinyCore Linux). I set PCM to 100%, ran the pCP backup, rebooted to verify - and it came right back at -38 dB. The usual alsactl store wasn\u0026rsquo;t even present on this build.\nThe RAM-based fix is to force the value at boot. TinyCore runs /opt/bootlocal.sh late in startup, and it\u0026rsquo;s included in the backup, so:\necho \u0026#34;amixer sset PCM 100% unmute\u0026#34; | sudo tee -a /opt/bootlocal.sh pcp bu # persist the backup Because it runs after the pCP startup that was resetting the level, it wins. Reboot again and it held at 100%. If your player is RAM-based, don\u0026rsquo;t trust a settings backup to capture live ALSA state - pin it explicitly at boot.\nBonus: ICY metadata and the HEAD-vs-GET trap # A smaller one from wiring SUB/WAVE\u0026rsquo;s now-playing into Home Assistant. I wanted to know whether the Icecast stream carried inline track titles, so I probed it:\ncurl -I -H \u0026#34;Icy-MetaData: 1\u0026#34; https://subwave.rustybower.com/stream.mp3 # station headers, but no icy-metaint I concluded it didn\u0026rsquo;t. Wrong. curl -I sends a HEAD request, and Icecast omits icy-metaint on HEAD - it only sends it on a real GET with the metadata header:\n$ curl -s -H \u0026#34;Icy-MetaData: 1\u0026#34; -D - -o /dev/null https://subwave.rustybower.com/stream.mp3 | grep icy-metaint icy-metaint: 16000 The stream carried live StreamTitle metadata all along. Lesson: to check streaming behavior, probe the way the client actually connects. HEAD lies.\nFor the richer bits ICY can\u0026rsquo;t carry (album, year, the AI DJ\u0026rsquo;s name, listener count) I poll the station\u0026rsquo;s JSON API with a RESTful sensor - the ICY title covers the stream itself, everywhere, for free.\nThe red herrings # Debugging honesty - two theories I chased that were wrong:\nexternalTrafficPolicy: Local. The player flapping \u0026ldquo;unavailable\u0026rdquo; every 2-3 minutes screamed \u0026ldquo;connection getting rerouted through the LoadBalancer.\u0026rdquo; I\u0026rsquo;d fixed exactly that for LMS once, so I added it to the MA service. It didn\u0026rsquo;t help, because the flapping was a symptom of the pod-IP stream problem (#2) - stalled stream fetches tipping the player unhealthy - not a transport issue. Fixing the published IP made the flapping stop on its own. Device contention. The bcm2835 exposes 8 subdevices but they must all share one sample rate, so an active AirPlay session (shairport-sync, holding the device) really can block squeezelite. That was a genuine issue at one point - but it wasn\u0026rsquo;t the issue, and I burned time on it before finding #2. Both were plausible, both wasted time. The lesson I keep relearning: get the primary evidence before theorizing. The squeezelite debug log (-d slimproto=debug) handed me the pod IP in one line. I should have gone there an hour earlier.\nWhere it landed # Navidrome (Subsonic, /rest bypasses Authelia) │ ▼ Music Assistant ──► piCorePlayer (plughw, PCM pinned 100%) ──► DAX66 zones (Published IP = LoadBalancer IP) │ └──► Home Assistant integration → media_player.picoreplayer + every zone One hub. Navidrome is the library, Music Assistant drives the Pi over SlimProto and surfaces every DAX66 zone to Home Assistant, SUB/WAVE is a one-tap favorite, and LMS is retired - its app directory deleted, ArgoCD pruned it.\nThe recurring theme, if there is one: nearly every failure was a networking assumption that holds on a flat LAN and breaks inside Kubernetes - a service advertising an IP its clients can\u0026rsquo;t reach, an auth proxy in front of a machine-to-machine API, a probe that doesn\u0026rsquo;t connect like a real client. None of it was exotic. All of it was invisible until I looked at the primary evidence instead of the symptom.\n","date":"August 11, 2026","externalUrl":null,"permalink":"/posts/music-assistant-kubernetes-whole-home-audio/","section":"Posts","summary":"My music setup had accreted into a mess. Across two Kubernetes clusters I was running Navidrome, Lyrion Music Server (LMS), beets, and my own AI radio station (SUB/WAVE) - plus a Raspberry Pi running piCorePlayer wired into a Dayton DAX66 matrix amp for whole-home audio. Four things that all “played music,” none of which talked to each other, and no single place to control any of it from Home Assistant.\n","title":"Whole-Home Audio with Music Assistant: Kubernetes Gotchas and a Silent Raspberry Pi","type":"posts"},{"content":"","date":"July 30, 2026","externalUrl":null,"permalink":"/tags/bgp/","section":"Tags","summary":"","title":"Bgp","type":"tags"},{"content":"","date":"July 30, 2026","externalUrl":null,"permalink":"/tags/metallb/","section":"Tags","summary":"","title":"Metallb","type":"tags"},{"content":"","date":"July 30, 2026","externalUrl":null,"permalink":"/categories/networking/","section":"Categories","summary":"","title":"Networking","type":"categories"},{"content":"I started the day asking why one of my Frigate cameras was using more CPU than the others. I finished it having moved every LoadBalancer in my Kubernetes cluster onto BGP-routed addresses, because along the way I found a service that had been silently unreachable for two years.\nThis is the write-up of that migration: MetalLB from L2 mode to BGP, peered to a UniFi Dream Machine Pro. Mostly it\u0026rsquo;s the six things that bit me, because those are the parts nobody puts in the tutorials.\nThe two-year-old bug # While poking at Frigate I tried to hit its LoadBalancer address directly:\n$ for p in 5000 1984 8554 8555; do nc -zv 172.30.0.1 $p; done all refused Every port refused. Frigate worked fine, but only through the Traefik ingress. Its LoadBalancer had never worked at all.\nThe address was the tell. 172.30.0.1 is the .1 of the node subnet: my UDM\u0026rsquo;s own interface. MetalLB had auto-assigned my gateway\u0026rsquo;s address to a Kubernetes service.\nThe confirmation was easy, and it\u0026rsquo;s a useful trick:\n$ ping 172.30.0.1 # replies $ ping 172.30.0.5 # silent, and it\u0026#39;s a working LoadBalancer A MetalLB address never answers ICMP. MetalLB doesn\u0026rsquo;t put the address on an interface; it just answers ARP for it and lets the node\u0026rsquo;s DNAT rules do the rest. So a \u0026ldquo;LoadBalancer address\u0026rdquo; that replies to ping isn\u0026rsquo;t a LoadBalancer address. Something else owns it.\nHow it happened # The pool was 172.30.0.0/16. Reasonable-looking. Then I looked at a node:\n$ ip -4 addr show eno1 inet 172.30.0.11/15 ... # /15, not /24 $ ip route | grep default default via 172.30.0.1 A /15. So 172.30.0.0–172.31.255.255 is one flat L2 segment, and it holds:\nthe gateway at .1 five DHCP-assigned nodes at .11–.15 a NAS at 172.30.1.245 both MetalLB pools MetalLB allocates the lowest free address in a pool and knows nothing about node addresses, gateways, DHCP scopes, or filers. It took .1 because .1 was first.\nAnd it was going to keep going. Allocations so far were .1 through .8 plus .10, in exact ascending order. The next two services would have taken .9 and then .11, a node\u0026rsquo;s own address. avoidBuggyIPs doesn\u0026rsquo;t help; it only skips .0 and .255.\nThe immediate fix was to narrow the pool to the addresses actually in use plus a growth range well clear of the infrastructure. But the real fix was to stop sharing a segment with the gateway at all.\nWhy BGP, and why the prefix matters # L2 mode requires the pool to be on-link with the nodes. BGP doesn\u0026rsquo;t: you advertise a prefix and the router installs a route. That means the service range can live somewhere with nothing else in it.\nI picked 10.44.0.0/16, continuing the k3s numbering: 10.42 pods, 10.43 services, 10.44 load balancers.\nThe important property isn\u0026rsquo;t the mnemonic, it\u0026rsquo;s that the prefix is off-segment. If I\u0026rsquo;d carved the BGP pool out of the existing /15, the nodes would still be on-link for it and ARP would carry traffic even if BGP were completely broken. Every test would pass and I\u0026rsquo;d have learned nothing. Off-segment means a working address proves BGP is genuinely forwarding.\nVerified it was free three ways before using it: no route from a node or the LAN, no ICMP reply, and no ARP presence probed from a node on the segment.\nThe UDM Pro side # BGP on UniFi is a config file you upload (Settings → Routing → BGP), and it\u0026rsquo;s plain FRR:\nrouter bgp 64512 bgp router-id 172.30.0.1 bgp log-neighbor-changes no bgp ebgp-requires-policy ! neighbor k8s peer-group neighbor k8s remote-as 64513 neighbor k8s timers 10 30 ! neighbor 172.30.0.11 peer-group k8s # one per node ... address-family ipv4 unicast neighbor k8s activate neighbor k8s prefix-list metallb-in in maximum-paths 5 exit-address-family exit ! ip prefix-list metallb-in seq 10 permit 10.44.0.0/16 le 32 ip prefix-list metallb-in seq 99 deny any Three things worth calling out:\nno bgp ebgp-requires-policy is not optional. Modern FRR implements RFC 8212: without an explicit policy it establishes the session and then silently discards every route. The session shows up. Nothing works. You will not enjoy debugging that on a UDM.\nThe prefix-list is the actual safety net. It confines the cluster to injecting 10.44.0.0/16 and nothing else, so a MetalLB misconfiguration can\u0026rsquo;t push arbitrary routes into the home network. UniFi requires prefix-lists after the router bgp block.\nSet DHCP reservations for your nodes first. Mine were on DHCP leases. Five neighbor statements pinned to addresses that could drift is a trap you set for your future self.\nOn the cluster side it\u0026rsquo;s three CRDs: a pool, a BGPAdvertisement, and a BGPPeer. All five speakers came up:\n\u0026#34;msg\u0026#34;:\u0026#34;BGP session established\u0026#34; x5 \u0026#34;event\u0026#34;:\u0026#34;sessionUp\u0026#34; x5 Six things that bit me # 1. Native mode rejects keepaliveTime # MetalLB v0.16 still ships the native BGP implementation alongside FRR and FRR-K8s. It works fine for a plain IPv4 eBGP session, but it\u0026rsquo;s feature-frozen, and the webhook is blunt about it:\npeer 172.30.0.1 has keepalive-time set on native bgp mode holdTime is accepted, keepaliveTime isn\u0026rsquo;t. No real loss, because MetalLB derives keepalive as holdTime/3, and BGP negotiates the lower of the two hold times anyway, so the router\u0026rsquo;s value governs. Native mode also has no BFD, which makes hold time your only failure detection.\n2. Client-side apply silently ate a UDP port # My Frigate service declared 8555 on both TCP and UDP. The live object only ever had the TCP one, while ArgoCD cheerfully reported Synced.\nService.spec.ports carries two different merge semantics:\nx-kubernetes-patch-merge-key: port ← client-side apply x-kubernetes-list-map-keys: [port, protocol] ← server-side apply Client-side apply, which is kubectl\u0026rsquo;s default and ArgoCD\u0026rsquo;s, merges the port list on port alone. Two entries with port 8555 collide on that key and the second is silently discarded. The last-applied-configuration annotation kept listing the UDP port, which made it look like it had been applied.\nI reproduced it deliberately: kubectl apply reported configured and still produced four ports; kubectl apply --server-side produced all five. Fix is argocd.argoproj.io/sync-options: ServerSideApply=true.\nAny resource needing two entries that differ only by a non-merge-key field has this problem.\n3. Two ways to pin an address, and you may not use both # MetalLB accepts spec.loadBalancerIP (deprecated) and the metallb.io/loadBalancerIPs annotation. If a service has both, it doesn\u0026rsquo;t pick one:\nWarning AllocationFailed service can not have both metallb.io/loadBalancerIPs and svc.Spec.LoadBalancerIP It allocates nothing. My Paperless FTP service came up with no address at all because the base set the deprecated field and my overlay added the annotation.\nAlso worth knowing: the legacy metallb.universe.tf/loadBalancerIPs prefix is still honored in v0.16.1. I assumed it was dead, concluded a service was unpinned, and briefly believed I\u0026rsquo;d created a hazard that didn\u0026rsquo;t exist. Tested it with a throwaway service, and it works fine. Don\u0026rsquo;t write new config with it, but don\u0026rsquo;t panic when you find it.\n4. One Service can\u0026rsquo;t hold two addresses of the same family # The obvious zero-downtime move, putting both old and new addresses on one Service, doesn\u0026rsquo;t work:\nFailed to allocate IP: IPFamilyForAddresses: same address family loadBalancerIPs is for dual-stack, not two IPv4 addresses. So I used a parallel Service: a second object selecting the same pods on the new address. Both serve at once, external references move one at a time, and then the old one goes. No flag day.\nOne trap in that pattern: for Mosquitto I nearly promoted the parallel service and deleted the original. That would have broken Home Assistant, whose broker is configured as mosquitto.mosquitto.svc.cluster.local. The cluster DNS name follows the Service name. The right move was to repoint the original and delete the parallel one.\n5. external-dns upsert-only leaves litter, and repoints non-atomically # My external-dns runs --policy=upsert-only, so it never deletes records. Over time that leaves orphans pointing at addresses that stopped existing. I found three (syncthing, glance, glance-test), all from apps deleted long ago. When two names didn\u0026rsquo;t follow a migration I assumed they were lagging; they had no Ingress at all.\nMore disruptive: it rewrites records delete-then-add. Watching a repoint of 36 records:\npoll 1: 172.31.0.2 x36 poll 2: 172.31.0.2 x15 ← 21 names briefly did not exist poll 3: 10.44.1.2 x34 For ~30 seconds those names returned NXDOMAIN. A phone that queried in that window cached the negative answer and \u0026ldquo;couldn\u0026rsquo;t resolve\u0026rdquo; for a while after everything was fine. Worth knowing before you repoint anything critical.\n6. load-balancer-cleanup zombies # Five separate Services refused to delete, some terminating since the previous year, still holding their addresses and erroring in the controller every 17 minutes. All stuck on the service.kubernetes.io/load-balancer-cleanup finalizer, several orphaned in namespaces that had been force-deleted out from under them, which makes them unpatchable:\n$ kubectl -n deepstack patch svc deepstack-service ... Error from server (NotFound): namespaces \u0026#34;deepstack\u0026#34; not found The fix is to recreate the namespace so the object becomes addressable again, clear the finalizer, then delete the namespace:\nkubectl create ns deepstack kubectl -n deepstack patch svc deepstack-service --type=merge \\ -p \u0026#39;{\u0026#34;metadata\u0026#34;:{\u0026#34;finalizers\u0026#34;:null}}\u0026#39; kubectl delete ns deepstack If a Service won\u0026rsquo;t die, this finalizer is the first thing to check.\nStatic or dynamic? # The pools split into auto-assigned and pinned, and the test I settled on is:\nDoes anything outside Kubernetes reference this address?\nBecause an auto-assigned address changes when the Service is recreated, and whatever held it breaks silently.\nApplying that honestly, almost nothing qualified as dynamic. Only three services had zero references anywhere and were reached purely by name through the ingress. Everything else needed pinning: the broker, the ingress controller, Plex, the DNS server.\nOne I got wrong at first: Music Assistant looked like a browser-only app until I read its ports. It exposes streamserver:8097, streams:8098 and lms:3483. It hands players URLs built from its own address and is discovered by Squeezebox clients. Definitively static. Read the ports before classifying.\nVerification, and the mistake I made twice # The single most useful technique: capture on the node\u0026rsquo;s interface, not in the pod. kube-proxy DNATs the destination before the pod sees it, so a pod cannot tell which address a client used. On the node, pre-DNAT, you see the original:\ntcpdump -i any -n -Q in \u0026#34;dst host 172.31.0.254 and port 53\u0026#34; That\u0026rsquo;s how I proved nothing was still using an old address before retiring it, and how I proved the remote cluster\u0026rsquo;s CoreDNS had actually picked up a config change. Resolution succeeded either way, because both addresses answered, so the only real evidence was which one the packets went to.\nAlways send a canary query first. A broken filter and a genuinely idle address produce identical output. If you haven\u0026rsquo;t proven the capture works, silence means nothing. I built that check into the script as a hard failure.\nAnd the mistake, which I made twice: I verified reachability from one subnet and generalized. First I declared a whole /16 unreachable after testing two addresses in it. Later I confirmed the new prefix worked from my workstation and called it done, then found my network has eight client VLANs, and the one my workstation sits in has an Internal → Internal Any/Any rule while the IoT VLAN is default-deny with explicit allows. My testing had proven nothing about the VLAN that mattered.\nBGP installs a route. A route is not permission. The firewall rules referenced the old addresses explicitly and all needed updating:\nAllow Pihole DNS IoT → 172.31.0.254:53 Allow MQTT IoT → 172.31.0.9:1883 Port Forward WAN → \u0026lt;media-server\u0026gt; The DNS one was the dangerous one: paired with a Block External DNS rule, an IoT device that renewed its lease would have had no resolver and no fallback.\nConfig is layered, and the top layer lies # My NAS kept querying the old DNS server after I\u0026rsquo;d fixed it. Host config was unambiguous:\n/etc/resolv.conf nameserver 10.44.1.254 midclt network.configuration nameserver1 10.44.1.254 The culprit was a Docker container running with host networking, holding a copy of /etc/resolv.conf taken at creation, three weeks earlier. Updating the host didn\u0026rsquo;t touch it. Neither did the Tailscale admin UI, because that container has accept-dns off (CorpDNS: false), so Tailscale deliberately doesn\u0026rsquo;t manage its resolver. A container restart fixed it in seconds.\n\u0026ldquo;The host config is correct\u0026rdquo; is necessary, not sufficient. Check the layer that\u0026rsquo;s actually sending the packets.\nWas it worth it? # For the stated goal of stopping MetalLB handing out infrastructure addresses, the pool narrowing alone would have done it. BGP was the bigger swing, and it bought:\nService addresses in a range with nothing else in it, so the whole class of collision is gone by construction ECMP across all five nodes (announcing from node \u0026quot;kube03\u0026quot; and \u0026quot;kube05\u0026quot; for the same service) Failure detection in ~30s instead of ~3min, via hold timers The cost is that half the config now lives on an appliance, outside Git, where a firmware update could reset it. For a homelab that\u0026rsquo;s an acceptable trade. If your router doesn\u0026rsquo;t do BGP, narrow your pools and sleep fine.\nThe thing I\u0026rsquo;d tell past-me isn\u0026rsquo;t about BGP at all:\nTest from the network that matters, not the one you\u0026rsquo;re sitting on. I learned that twice in one day.\n","date":"July 30, 2026","externalUrl":null,"permalink":"/posts/metallb-l2-to-bgp-migration-udm-pro/","section":"Posts","summary":"I started the day asking why one of my Frigate cameras was using more CPU than the others. I finished it having moved every LoadBalancer in my Kubernetes cluster onto BGP-routed addresses, because along the way I found a service that had been silently unreachable for two years.\n","title":"The LoadBalancer That Never Worked: Migrating MetalLB from L2 to BGP","type":"posts"},{"content":"","date":"July 30, 2026","externalUrl":null,"permalink":"/tags/unifi/","section":"Tags","summary":"","title":"Unifi","type":"tags"},{"content":"","date":"July 28, 2026","externalUrl":null,"permalink":"/tags/coral-tpu/","section":"Tags","summary":"","title":"Coral-Tpu","type":"tags"},{"content":"","date":"July 28, 2026","externalUrl":null,"permalink":"/tags/frigate/","section":"Tags","summary":"","title":"Frigate","type":"tags"},{"content":"","date":"July 28, 2026","externalUrl":null,"permalink":"/categories/llm/","section":"Categories","summary":"","title":"Llm","type":"categories"},{"content":"A notification that says \u0026ldquo;person detected on doorbell\u0026rdquo; is useful. A notification that says \u0026ldquo;a delivery driver in a brown UPS uniform placed a medium-sized box on the porch and is walking back to the truck\u0026rdquo; is significantly more useful — especially when you\u0026rsquo;re in a meeting and trying to decide whether to get up.\nI wanted that second kind of notification. The pieces were already in my homelab: Frigate NVR for camera management, a Coral USB TPU for real-time object detection, and a Mac Studio running llama-swap for LLM inference. The challenge was wiring them together so that every time Frigate detects a person or package, a vision model looks at the image and writes a one-sentence description.\nThe detection pipeline # The full chain has four stages, each handled by different hardware:\nCamera (RTSP) → Frigate (Coral TPU) → Vision LLM (Mac Studio) → Notification 5 cams object detect describe image HA / phone ~50ms ~3-5s Stage 1: Camera streams. Five cameras — doorbell, garage, dog run, and two library-facing cameras — push RTSP streams to Frigate through go2rtc for restreaming. The doorbell is an Amcrest; the others are Dahua. All decode via VAAPI hardware acceleration on the Intel NUC that runs Frigate.\nStage 2: Object detection on the Coral TPU. Frigate pulls frames at 4 FPS and runs them through a Frigate+ custom model on the Coral Edge TPU. The model detects people and packages. A USB Coral on a dedicated node (a dedicated node) handles this — it\u0026rsquo;s fast enough that detection latency is under 50ms per frame.\nStage 3: Vision LLM description. When Frigate creates a review alert — meaning a tracked object persisted long enough to be interesting — it sends the snapshot to a vision language model. Frigate\u0026rsquo;s built-in GenAI support handles this natively. I\u0026rsquo;m using qwen3.5-vl:9b, a 9-billion parameter vision-language model, served by llama-swap on the Mac Studio.\nStage 4: Notification. The description gets attached to the Frigate event. Home Assistant picks it up and pushes it to my phone.\nGetting the Coral into Kubernetes # The Coral TPU is a USB device physically plugged into one specific node. Making it available to a Kubernetes pod requires a few layers:\nFirst, the node is labeled and tainted so only Frigate can schedule there:\n# Node label + taint on a dedicated node coral-node: \u0026#34;true\u0026#34; coral-node=true:NoSchedule Then generic-device-plugin runs as a DaemonSet to register USB devices as Kubernetes resources. It watches for the Coral\u0026rsquo;s USB vendor/product IDs (Google and Global Unichip both make Coral variants):\nargs: - --device - | name: coral groups: - usb: - vendor: \u0026#34;18d1\u0026#34; product: \u0026#34;9302\u0026#34; - vendor: \u0026#34;1a6e\u0026#34; product: \u0026#34;089a\u0026#34; Frigate\u0026rsquo;s deployment patch adds the corresponding nodeSelector and toleration:\nnodeSelector: coral-node: \u0026#34;true\u0026#34; tolerations: - key: coral-node value: \u0026#34;true\u0026#34; effect: NoSchedule In practice, the Frigate container runs privileged and accesses the Coral directly through /dev. The device plugin is mostly there for scheduling correctness — making sure nothing else lands on the Coral node and that Frigate only schedules where the hardware exists.\nThe vision LLM integration # Frigate 0.14+ has native GenAI support. You configure it in config.yaml and Frigate handles the API calls, prompt construction, and response parsing:\ngenai: provider: openai api_key: \u0026#34;sk-no-key-required\u0026#34; base_url: http://ollama.ollama.svc.cluster.local:11434/v1 model: qwen3.5-vl:9b review: alerts: labels: - person - package genai: enabled: true alerts: true image_source: preview activity_context_prompt: \u0026gt;- Suburban residential home. Household includes family members and a dog. Common expected visitors: delivery drivers, mail carrier, neighbors. A few things worth noting:\nThe \u0026ldquo;ollama\u0026rdquo; service isn\u0026rsquo;t Ollama. It\u0026rsquo;s a Kubernetes Service with a manual Endpoints resource pointing at the Mac Studio\u0026rsquo;s IP, where llama-swap serves an OpenAI-compatible API. Frigate doesn\u0026rsquo;t know or care — it just hits the /v1/chat/completions endpoint. I wrote about this pattern in Mac Studio as a Kubernetes Compute Satellite.\nPer-object prompts make a big difference. Generic prompts produce generic descriptions. Telling the model exactly what to focus on for each object type produces much better output:\nobjects: genai: enabled: true prompt: \u0026gt;- This image is from the {camera} camera at a suburban residential home. In one sentence, describe what the {label} is doing. object_prompts: person: \u0026gt;- In one sentence, describe this person: their appearance (clothing, build), what they are carrying, and what action they are taking. package: \u0026gt;- In one sentence, describe the package: its size, shape, and where it was left. Note if a delivery person is still visible. The activity context prompt reduces false alarm anxiety. Without context, the model tends toward security-guard language: \u0026ldquo;an unidentified individual is approaching the premises.\u0026rdquo; With context (\u0026ldquo;suburban home, family, dog, delivery drivers\u0026rdquo;), it calibrates better: \u0026ldquo;a woman in a gray jacket is walking a golden retriever past the driveway.\u0026rdquo;\nThe doorbell shortcut # The GenAI descriptions are great for review events that I check later. But for real-time \u0026ldquo;someone\u0026rsquo;s at the door\u0026rdquo; situations, I also have an AppDaemon automation that bypasses the LLM entirely and just throws the Frigate snapshot onto a Google display:\nclass DoorbellPersonFrigateSnapshot(hass.Hass): def person_count_changed(self, entity, attribute, old, new, kwargs): # Only trigger on 0 → \u0026gt;0 transition old_num = float(old) if old not in (None, \u0026#34;unknown\u0026#34;, \u0026#34;unavailable\u0026#34;, \u0026#34;\u0026#34;) else 0.0 new_num = float(new) if not (old_num == 0 and new_num \u0026gt; 0): return ts = int(time.time()) snapshot_url = f\u0026#34;{self.frigate_base_url}/api/{self.frigate_camera}/person/snapshot.jpg?t={ts}\u0026#34; self.call_service( \u0026#34;media_player/play_media\u0026#34;, entity_id=self.media_player, media_content_id=snapshot_url, media_content_type=\u0026#34;image/jpeg\u0026#34;, ) self.run_in(self.stop_display, self.display_duration) When Frigate\u0026rsquo;s person count goes from 0 to any positive number, it fetches the latest person snapshot (cache-busted with a timestamp) and casts it to the office Google Nest Hub for two minutes. No LLM latency — the image is on the display within a second or two of detection.\nThe config injection problem # Frigate expects to own its config.yaml and writes to it at runtime (for UI-based config changes). But I want the config in Git, managed as a Kubernetes ConfigMap. These two requirements conflict — you can\u0026rsquo;t mount a ConfigMap as writable.\nThe workaround is a busybox init container that copies the ConfigMap onto the PVC before Frigate starts:\ninitContainers: - name: copy-config image: busybox:latest command: - sh - -c - | cp /config-source/config.yaml /config/config.yaml cp /config-source/config.yaml /config/backup_config.yaml volumeMounts: - name: config-configmap mountPath: /config-source - name: config-volume mountPath: /config This means every pod restart resets the config to what\u0026rsquo;s in Git. Any changes made through Frigate\u0026rsquo;s UI are ephemeral — which is exactly what I want. The Git repo is the source of truth.\nModel selection # I\u0026rsquo;ve tried several vision models for this:\nModel Size Speed Quality minicpm-v 3B ~1s Brief, sometimes misses details llama3.2-vision 11B ~5s Good but verbose qwen3.5-vl:9b 9B ~3s Best balance — concise and accurate qwen3.5-vl:9b has been the sweet spot. It\u0026rsquo;s fast enough that descriptions are ready by the time I check the notification, and accurate enough that I trust it to describe what\u0026rsquo;s happening without hallucinating details. The \u0026ldquo;one sentence\u0026rdquo; instruction in the prompt is critical — without it, every model produces a paragraph.\nWhat it actually looks like # A typical Frigate event now has:\nThumbnail: Cropped image of the detected person/package Snapshot: Full camera frame at time of detection GenAI description: \u0026ldquo;A delivery driver in a dark blue uniform placed a small brown box on the front porch steps and is returning to a white van in the driveway.\u0026rdquo; The description shows up in the Frigate UI timeline and in Home Assistant notifications. It turns \u0026ldquo;motion detected\u0026rdquo; into something I can act on without opening the camera feed.\nTotal cost: a $30 Coral USB stick and GPU time on a Mac I already owned. No cloud APIs, no subscriptions, no video leaving the network.\n","date":"July 28, 2026","externalUrl":null,"permalink":"/posts/frigate-coral-tpu-vision-llm-camera-alerts/","section":"Posts","summary":"A notification that says “person detected on doorbell” is useful. A notification that says “a delivery driver in a brown UPS uniform placed a medium-sized box on the porch and is walking back to the truck” is significantly more useful — especially when you’re in a meeting and trying to decide whether to get up.\n","title":"Teaching Security Cameras to Describe What They See","type":"posts"},{"content":"","date":"July 21, 2026","externalUrl":null,"permalink":"/tags/docker/","section":"Tags","summary":"","title":"Docker","type":"tags"},{"content":"","date":"July 21, 2026","externalUrl":null,"permalink":"/categories/gitops/","section":"Categories","summary":"","title":"Gitops","type":"categories"},{"content":"My K3s cluster runs on six Intel NUCs. They\u0026rsquo;re great for the 50+ containers that make up my homelab — Plex, Home Assistant, Frigate, Paperless, the usual suspects. But they\u0026rsquo;re terrible at machine learning. Four Skylake cores and 32 GB of RAM per node doesn\u0026rsquo;t get you far when Immich wants to classify 80,000 photos or Frigate wants a vision model to describe who\u0026rsquo;s at your door.\nThe Mac Studio sitting under my desk — M1 Ultra, 64 GB unified memory — is the opposite problem. Absurd single-machine ML performance, but I don\u0026rsquo;t want to migrate my entire cluster to it. I want the K8s cluster to stay the control plane for routing, TLS, service discovery, and GitOps, while the Mac handles the heavy compute.\nThe solution turned out to be one of Kubernetes\u0026rsquo; oldest and least glamorous features: manual Endpoints.\nThe pattern # In Kubernetes, a Service normally discovers its backends through label selectors — it finds pods with matching labels and routes traffic to them. But if you create a Service without a selector and pair it with a hand-written Endpoints resource, you can point that Service at any IP address. Including one outside the cluster.\napiVersion: v1 kind: Service metadata: name: ollama spec: type: ClusterIP ports: - port: 11434 targetPort: 11434 name: http # No selector — backends come from the Endpoints resource --- apiVersion: v1 kind: Endpoints metadata: name: ollama # Must match the Service name subsets: - addresses: - ip: 192.168.1.200 # Mac Studio\u0026#39;s LAN IP ports: - port: 11434 name: http That\u0026rsquo;s it. Every pod in the cluster can now reach ollama.ollama.svc.cluster.local:11434 and the traffic routes to the Mac Studio. Add an Ingress in front and it\u0026rsquo;s also available at ollama.example.com with TLS termination, external-dns, and Homepage dashboard integration — all the same infrastructure every other service gets.\nThe Mac Studio doesn\u0026rsquo;t know or care that Kubernetes exists. It\u0026rsquo;s just listening on a port.\nWhat runs on the Mac # Six services currently use this pattern:\nService Port What it does llama-swap 11434 LLM inference (OpenAI-compatible API) Immich ML 3003 Photo classification and face detection Pet Tagger 2287 Individual dog identification (YOLO + CLIP) ComfyUI 8188 Image generation (Stable Diffusion) Fooocus 7865 Image generation (simplified UI) Stable Diffusion 7860 Image generation (A1111 WebUI) The first three run as Docker Compose on the Mac. The image generation tools run natively. llama-swap runs as a macOS LaunchAgent via a plist:\n\u0026lt;plist version=\u0026#34;1.0\u0026#34;\u0026gt; \u0026lt;dict\u0026gt; \u0026lt;key\u0026gt;Label\u0026lt;/key\u0026gt; \u0026lt;string\u0026gt;com.llama-swap\u0026lt;/string\u0026gt; \u0026lt;key\u0026gt;ProgramArguments\u0026lt;/key\u0026gt; \u0026lt;array\u0026gt; \u0026lt;string\u0026gt;/usr/local/bin/llama-swap\u0026lt;/string\u0026gt; \u0026lt;string\u0026gt;--config\u0026lt;/string\u0026gt; \u0026lt;string\u0026gt;/opt/models/llama-swap.yaml\u0026lt;/string\u0026gt; \u0026lt;string\u0026gt;--listen\u0026lt;/string\u0026gt; \u0026lt;string\u0026gt;0.0.0.0:11434\u0026lt;/string\u0026gt; \u0026lt;/array\u0026gt; \u0026lt;key\u0026gt;RunAtLoad\u0026lt;/key\u0026gt; \u0026lt;true/\u0026gt; \u0026lt;key\u0026gt;KeepAlive\u0026lt;/key\u0026gt; \u0026lt;true/\u0026gt; \u0026lt;/dict\u0026gt; \u0026lt;/plist\u0026gt; Each K8s-side app follows the same three-resource template: a Service (no selector), an Endpoints pointing to the Mac\u0026rsquo;s IP, and an Ingress for external access. Adding a new proxied service takes about two minutes.\nKeeping the Mac in the GitOps loop # The obvious downside of running workloads outside the cluster is that ArgoCD can\u0026rsquo;t manage them. The Docker Compose files and configs aren\u0026rsquo;t Kubernetes resources. But they\u0026rsquo;re still code, and they still need version tracking and dependency updates.\nI solved this by committing the Mac\u0026rsquo;s Docker Compose files into the same repo under hosts/mac-studio/:\nhosts/mac-studio/ ├── README.md ├── immich-ml/ │ └── docker-compose.yml └── pet-tagger/ ├── docker-compose.yml └── .env.example Renovate\u0026rsquo;s docker-compose manager is scoped to hosts/**/docker-compose.yml, so image updates get the same automated PRs as everything else in the cluster. When Immich releases a new version, Renovate opens a single PR that bumps both the in-cluster immich-server and the Mac\u0026rsquo;s immich-machine-learning — keeping them in lockstep, which matters because their internal API isn\u0026rsquo;t versioned.\nDeploying a Mac-side update is still manual — SSH in, docker compose pull \u0026amp;\u0026amp; docker compose up -d — but at least the change is tracked in Git and the version is always visible.\nWhy not just run the Mac as a K8s node? # I tried this. K3s runs on macOS with some effort, and you can register a Mac as a worker node. The problems:\nDevice access is painful. The M1\u0026rsquo;s GPU is the whole point, but exposing Apple Silicon\u0026rsquo;s unified memory to a container runtime is an unsolved problem. Docker Desktop on Mac doesn\u0026rsquo;t pass through the GPU. Native containers barely exist. launchd vs. kubelet. macOS already has a solid process supervisor. Fighting it to let kubelet own everything creates more problems than it solves. The Mac reboots for updates. macOS updates are not optional in the way Linux unattended-upgrades are. Having a K8s node disappear for 10 minutes during a macOS update is disruptive. The Endpoints approach avoids all of this. The Mac is a black box that happens to speak HTTP on known ports. Kubernetes doesn\u0026rsquo;t need to schedule pods on it, manage its lifecycle, or understand its hardware.\nThe one service that skips the proxy # Immich ML is the exception. Rather than routing through a Service and Endpoints, the Immich server pod points directly at the Mac via an environment variable:\nenv: - name: IMMICH_MACHINE_LEARNING_URL value: http://192.168.1.200:3003 This was a pragmatic choice. Immich ML processes thousands of photos during bulk imports, and the extra hop through kube-proxy added enough latency to be noticeable. For a service that only talks to one consumer, the direct connection is simpler and faster.\nWhat I\u0026rsquo;d change # Static IPs are fragile. The Mac\u0026rsquo;s static IP is hardcoded in six Endpoints resources. If it changes, I need to update all of them. A DNS-based approach (ExternalName Service pointing at a hostname) would be cleaner, but ExternalName Services don\u0026rsquo;t support port remapping, which several of these services need.\nNo health checking. With real pod backends, Kubernetes removes unhealthy endpoints automatically. With manual Endpoints, if the Mac is down, traffic routes to a dead IP and times out. I could add a sidecar that periodically checks the Mac and updates the Endpoints resource, but it hasn\u0026rsquo;t been painful enough to build yet.\nDocker Compose deploys are manual. A webhook that triggers docker compose pull \u0026amp;\u0026amp; up -d on the Mac when Renovate merges a PR would close the GitOps loop completely. Watchtower could do this, but I\u0026rsquo;d rather build something that only acts on merged PRs, not on every image push.\nDespite these rough edges, the pattern has been running for about six months without issues. The Mac handles all the ML inference for the cluster — photo classification, LLM serving, image generation, pet identification — while the NUCs handle everything else. The Endpoints proxy is invisible to every consumer; they just call a .svc.cluster.local address and get an answer.\n","date":"July 21, 2026","externalUrl":null,"permalink":"/posts/mac-studio-kubernetes-compute-satellite/","section":"Posts","summary":"My K3s cluster runs on six Intel NUCs. They’re great for the 50+ containers that make up my homelab — Plex, Home Assistant, Frigate, Paperless, the usual suspects. But they’re terrible at machine learning. Four Skylake cores and 32 GB of RAM per node doesn’t get you far when Immich wants to classify 80,000 photos or Frigate wants a vision model to describe who’s at your door.\n","title":"Mac Studio as a Kubernetes Compute Satellite","type":"posts"},{"content":"","date":"July 21, 2026","externalUrl":null,"permalink":"/tags/mac-studio/","section":"Tags","summary":"","title":"Mac-Studio","type":"tags"},{"content":"","date":"July 16, 2026","externalUrl":null,"permalink":"/tags/appdaemon/","section":"Tags","summary":"","title":"Appdaemon","type":"tags"},{"content":"My Bird Buddy smart feeder takes a photo every time a bird visits. Over a few months, that adds up to thousands of images — all sitting in Bird Buddy\u0026rsquo;s cloud app with no good way to search, organize, or back them up. I wanted them in Immich, my self-hosted photo library, organized by species and properly timestamped.\nThe Bird Buddy app doesn\u0026rsquo;t have an export feature. But the pybirdbuddy Python library can authenticate to their API and walk the feed. So I built a two-stage pipeline: a CronJob that downloads new photos daily, and a second CronJob that syncs them into Immich. Both run in Kubernetes, and the whole thing is about 300 lines of Python.\nThe timestamp problem # Bird Buddy images arrive as plain JPEGs with no EXIF metadata. If you drop them into Immich as-is, every image gets dated to when the CronJob ran, not when the bird visited. A cardinal at sunrise and a chickadee at sunset both show up as \u0026ldquo;3:00 AM\u0026rdquo; because that\u0026rsquo;s when download.py ran.\nThe Bird Buddy API does return a createdAt timestamp for each postcard. The fix is to write that timestamp into the JPEG\u0026rsquo;s EXIF data before Immich ever sees the file:\ndef apply_timestamp(filepath, dt, is_video): \u0026#34;\u0026#34;\u0026#34;Embed capture time so Immich dates the asset correctly.\u0026#34;\u0026#34;\u0026#34; local = dt.astimezone(LOCAL_TZ) if not is_video: stamp = local.strftime(\u0026#34;%Y:%m:%d %H:%M:%S\u0026#34;).encode() raw_off = local.strftime(\u0026#34;%z\u0026#34;) offset = (raw_off[:3] + \u0026#34;:\u0026#34; + raw_off[3:]).encode() if raw_off else None exif = piexif.load(str(filepath)) exif[\u0026#34;Exif\u0026#34;][piexif.ExifIFD.DateTimeOriginal] = stamp exif[\u0026#34;Exif\u0026#34;][piexif.ExifIFD.DateTimeDigitized] = stamp if offset: exif[\u0026#34;Exif\u0026#34;][piexif.ExifIFD.OffsetTimeOriginal] = offset piexif.insert(piexif.dump(exif), str(filepath)) # Also set file mtime — Immich\u0026#39;s fallback when EXIF is absent (videos) ts = dt.timestamp() os.utime(filepath, (ts, ts)) Immich reads DateTimeOriginal first, which is the standard EXIF field for \u0026ldquo;when was this photo actually taken.\u0026rdquo; For videos (MP4s from the feeder\u0026rsquo;s motion clips), EXIF doesn\u0026rsquo;t apply, so we fall back to setting the file\u0026rsquo;s modification time — Immich uses that as a last resort.\nThe timezone offset matters too. Without it, Immich has to guess whether \u0026ldquo;2026-07-04 08:32:15\u0026rdquo; is UTC, local time, or something else. Writing the explicit offset (-05:00 for Central) removes the ambiguity.\nStage 1: Download # The downloader runs daily at 3 AM as a Kubernetes CronJob. It authenticates to Bird Buddy, walks the feed for the last 24 hours, and downloads any new media into species-organized directories on an NFS share:\n/photos/birdbuddy/ ├── American_Robin/ │ ├── abc123.jpg │ └── def456.mp4 ├── Black-capped_Chickadee/ │ ├── ghi789.jpg │ └── ... ├── Northern_Cardinal/ │ └── ... └── .downloaded.json ← state file for deduplication Deduplication is handled by a simple JSON state file that tracks every media ID we\u0026rsquo;ve already downloaded. The state file lives on the same NFS share as the photos, so it survives pod restarts.\nThe CronJob itself uses a stock python:3.12-slim image with pip-installed dependencies — no custom Docker image to maintain:\napiVersion: batch/v1 kind: CronJob metadata: name: birdbuddy-downloader spec: schedule: \u0026#34;0 3 * * *\u0026#34; timeZone: \u0026#34;America/Chicago\u0026#34; concurrencyPolicy: Forbid jobTemplate: spec: template: spec: containers: - name: downloader image: python:3.12-slim command: - /bin/sh - -c - | pip install --quiet pybirdbuddy aiohttp piexif tzdata \u0026amp;\u0026amp; \\ python /scripts/download.py envFrom: - secretRef: name: birdbuddy-credentials env: - name: OUTPUT_DIR value: /data/birdbuddy volumeMounts: - name: scripts mountPath: /scripts - name: data mountPath: /data/birdbuddy volumes: - name: scripts configMap: name: birdbuddy-downloader-scripts - name: data nfs: server: 192.168.1.10 path: /mnt/storage/media/photos/birdbuddy The Python script is delivered as a Kustomize-generated ConfigMap — the same pattern I use for AppDaemon automations. No Dockerfile, no container registry, no build pipeline. Change the Python, push to Git, ArgoCD syncs the ConfigMap.\nThe downloader supports two modes: an incremental daily sync (last 24 hours of the feed) and a full backfill that walks all collections. The initial backfill pulled about 2,000 images. Daily runs typically download 5-20 new photos.\nStage 2: Sync to Immich # Thirty minutes after the downloader finishes, a second CronJob reconciles the NFS directory with Immich:\nTrigger a library scan. Immich has an \u0026ldquo;external library\u0026rdquo; feature that watches a filesystem path. The sync script pokes the API to start a scan and waits 60 seconds for it to index new files.\nEnsure a \u0026ldquo;Bird Buddy\u0026rdquo; album exists. All bird photos get collected into a single album for easy browsing.\nAdd missing assets to the album. Pages through every asset Immich imported from the BirdBuddy path and adds any that aren\u0026rsquo;t already in the album.\nArchive everything. Sets each asset\u0026rsquo;s visibility to archive, which removes it from the main Photos timeline while keeping it visible in the album and Archive views.\nThe archiving step is the key UX decision. Without it, my family\u0026rsquo;s photo timeline would be 80% bird pictures. Archiving keeps them organized and searchable without drowning out the actual family photos.\ndef archive_assets(asset_ids): for batch in chunked(asset_ids): api(\u0026#34;PUT\u0026#34;, \u0026#34;/api/assets\u0026#34;, {\u0026#34;ids\u0026#34;: batch, \u0026#34;visibility\u0026#34;: \u0026#34;archive\u0026#34;}) The sync script uses only Python\u0026rsquo;s standard library — no requests, no third-party HTTP client. urllib.request is ugly but it means the CronJob doesn\u0026rsquo;t need a pip install step at all.\nSecrets management # Bird Buddy credentials and the Immich API key are stored as Sealed Secrets:\napiVersion: bitnami.com/v1alpha1 kind: SealedSecret metadata: name: birdbuddy-credentials namespace: birdbuddy-downloader spec: encryptedData: BIRDBUDDY_EMAIL: AgBTcv5LFLpb610j... BIRDBUDDY_PASSWORD: AgBipbrCt5fI3I8V... The download script reads them as environment variables via envFrom. Nothing sensitive touches Git in plaintext.\nWhat I learned # EXIF timestamps are non-negotiable for photo imports. Without them, bulk imports are useless — every photo shows up on the same date. This applies to any pipeline that downloads photos from an API and imports them into a photo manager. The few lines of piexif code saved hours of manual re-dating.\nTwo CronJobs are better than one. Separating download from sync means I can re-run either independently. If Immich has a hiccup, I re-run the sync without re-downloading everything. If Bird Buddy\u0026rsquo;s API is down, the next day\u0026rsquo;s sync picks up whatever was already downloaded.\nNFS as the integration layer. The downloader writes to NFS. Immich\u0026rsquo;s external library reads from NFS. The sync script talks to Immich\u0026rsquo;s API. No service-to-service coupling, no message queues, no shared databases. The filesystem is the contract.\nThe pipeline has been running unattended for several months now. Every morning, any new bird visitors appear in the Bird Buddy album, correctly dated and organized by species. The total infrastructure cost is two CronJobs and an NFS share that was already there.\n","date":"July 16, 2026","externalUrl":null,"permalink":"/posts/birdbuddy-immich-photo-pipeline/","section":"Posts","summary":"My Bird Buddy smart feeder takes a photo every time a bird visits. Over a few months, that adds up to thousands of images — all sitting in Bird Buddy’s cloud app with no good way to search, organize, or back them up. I wanted them in Immich, my self-hosted photo library, organized by species and properly timestamped.\n","title":"Automating a Smart Bird Feeder's Photo Archive with Kubernetes and Immich","type":"posts"},{"content":"","date":"July 16, 2026","externalUrl":null,"permalink":"/tags/birdbuddy/","section":"Tags","summary":"","title":"Birdbuddy","type":"tags"},{"content":"","date":"July 16, 2026","externalUrl":null,"permalink":"/tags/immich/","section":"Tags","summary":"","title":"Immich","type":"tags"},{"content":"","date":"July 16, 2026","externalUrl":null,"permalink":"/categories/photography/","section":"Categories","summary":"","title":"Photography","type":"categories"},{"content":"","date":"July 16, 2026","externalUrl":null,"permalink":"/categories/python/","section":"Categories","summary":"","title":"Python","type":"categories"},{"content":"Home automations are code that controls physical things. When they break, the consequences aren\u0026rsquo;t a 500 error — they\u0026rsquo;re a garage door that won\u0026rsquo;t close in a snowstorm, or window shades that open at 3 AM, or a tablet that charges to 100% and stays plugged in for months until the battery swells.\nI run about 20 AppDaemon automations in my homelab, all deployed as Kubernetes ConfigMaps and tested with appdaemontestframework. The testing isn\u0026rsquo;t academic — it\u0026rsquo;s caught real bugs that would have been annoying or expensive to discover in production.\nWhy AppDaemon over HA automations? # Home Assistant\u0026rsquo;s built-in automations are great for simple trigger-action rules. \u0026ldquo;When the sun sets, turn on the porch lights.\u0026rdquo; But anything with branching logic, state tracking, or multiple interacting conditions becomes hard to read and impossible to test.\nAppDaemon lets you write automations as Python classes. Each one gets an initialize() method where you set up listeners and timers, and callback methods that fire when things happen. It\u0026rsquo;s just Python — you can use conditionals, loops, data structures, and (critically) unit tests.\nThe tradeoff is deployment complexity: you need to run AppDaemon as a separate service alongside Home Assistant, manage the Python files, and wire up the configuration. Kubernetes and Kustomize make this manageable.\nThe deployment model # Every AppDaemon automation lives as a .py file in the Git repo. Kustomize bundles them into a ConfigMap and mounts them into the AppDaemon container:\n# environments/bowerhaus/apps/appdaemon/kustomization.yaml configMapGenerator: - name: appdaemon-config files: - appdaemon/apps.yaml - appdaemon/bedroom_shades.py - appdaemon/garage_door_auto_close.py - appdaemon/tablet_charger.py - appdaemon/doorbell_person_frigate_snapshot.py # ... 15+ more The deployment mounts this ConfigMap at /conf/apps, which is where AppDaemon looks for automation modules:\nvolumes: - name: config-volume configMap: name: appdaemon-config containers: - name: appdaemon volumeMounts: - name: config-volume mountPath: /conf/apps The workflow is: edit Python → push to Git → ArgoCD syncs the ConfigMap → AppDaemon picks up the changes. No SSH, no file copying, no manual restarts.\nExample: temperature-aware garage doors # This one has saved me real money on heating bills. In cold weather, if a garage door is left open, the automation starts a countdown based on the current temperature:\nTemperature Timeout Below 30°F 7 minutes Below 40°F 15 minutes Below 50°F 30 minutes 50°F+ No auto-close Before closing, it flashes the garage lights to warn anyone who might be in the way:\nclass GarageDoorAutoClose(hass.Hass): def get_timeout_for_temperature(self): temp = float(self.get_state(self.temp_sensor, attribute=\u0026#34;temperature\u0026#34;)) if temp \u0026lt; 30: return 7 * 60 elif temp \u0026lt; 40: return 15 * 60 elif temp \u0026lt; 50: return 30 * 60 return None def auto_close_door(self, kwargs): door = kwargs[\u0026#34;door\u0026#34;] # Re-check temperature — it might have warmed up during the wait if self.get_timeout_for_temperature() is None: self.log(f\u0026#34;Temperature now above 50°F — skipping auto-close\u0026#34;) return # Flash lights as warning, then close self.flash_lights_then_close(door, ...) The re-check on close is important. If the door opens at 29°F and warms to 52°F during the 7-minute wait, the auto-close cancels. Temperature thresholds are checked twice — once when the timer starts and once when it fires.\nExample: forecast-aware window shades # The bedroom shade automation is the most complex one. It manages motorized shades based on time of day, weather forecast, and thermostat state:\nMorning: Open at 8:30 AM, unless the forecast high is above 85°F (solar heat gain would fight the AC all day). Midday: If the thermostat starts cooling, close the shades to reduce heat gain. When cooling stops, re-open — but only if it\u0026rsquo;s before 5 PM. Evening: Close at sunset or 7 PM, whichever comes first. The same BedroomShades class is reused across rooms with different parameters:\n# apps.yaml bedroom_shades: module: bedroom_shades class: BedroomShades covers: - cover.upstairs_master_bedroom_valance - cover.upstairs_master_bedroom_south thermostat: climate.master_bedroom_thermostat forecast_high_threshold: 85 morning_open_enabled: true # Guest room: close-only (no morning open, no cooling re-open) guest_shades: module: bedroom_shades class: BedroomShades covers: - cover.upstairs_guest_bedroom_valance thermostat: climate.guest_bedroom_thermostat morning_open_enabled: false forecast_high_threshold: 78 That morning_open_enabled: false flag exists because of a bug. Some rooms\u0026rsquo; shades were opening unexpectedly when the AC cycling stopped — the \u0026ldquo;cooling stopped → re-open\u0026rdquo; path didn\u0026rsquo;t check whether the shades were supposed to be auto-opened in the first place. The test that caught this:\n@automation_fixture( (BedroomShades, { \u0026#34;covers\u0026#34;: [\u0026#34;cover.guest_south\u0026#34;], \u0026#34;thermostat\u0026#34;: \u0026#34;climate.guest\u0026#34;, \u0026#34;morning_open_enabled\u0026#34;: False, }) ) def guest_shades_app(): pass def test_no_reopen_when_morning_open_disabled(given_that, guest_shades_app): opens = [] guest_shades_app.open_shades = lambda reason: opens.append(reason) guest_shades_app.sun_down = lambda: False guest_shades_app.is_before_reopen_cutoff = lambda: True guest_shades_app.closed_for_cooling = True guest_shades_app.thermostat_changed( \u0026#34;climate.guest\u0026#34;, \u0026#34;hvac_action\u0026#34;, \u0026#34;cooling\u0026#34;, \u0026#34;idle\u0026#34;, {} ) assert opens == [] # Shades should NOT have opened The @automation_fixture decorator from appdaemontestframework instantiates the real app class with mock Home Assistant services injected. You can call any method directly and assert on the side effects. The test above monkey-patches open_shades to record calls instead of actually calling the cover service — pure behavioral testing.\nExample: tablet battery management # Three Fire tablets run Fully Kiosk Browser as Home Assistant dashboards around the house. Each one sits on a Shelly smart plug. The automation keeps batteries between 25% and 80%:\nclass TabletCharger(hass.Hass): def battery_check(self, entity, attribute, old, new, kwargs): battery = int(float(new)) plug_on = self.get_state(cfg[\u0026#34;plug\u0026#34;]) == \u0026#34;on\u0026#34; if battery \u0026lt;= 25 and not plug_on: self.turn_on(plug) elif battery \u0026gt;= 80 and plug_on: self.turn_off(plug) def periodic_check(self, kwargs): # Every 5 minutes, catch edge cases the state listener might miss for name, cfg in self.tablets.items(): battery = int(float(self.get_state(cfg[\u0026#34;battery\u0026#34;]))) if battery \u0026gt;= 100 and self.get_state(cfg[\u0026#34;plug\u0026#34;]) == \u0026#34;on\u0026#34;: self.turn_off(cfg[\u0026#34;plug\u0026#34;]) # Hard cutoff at 100% The state listener handles the normal case. The periodic check is a safety net — if a state change event is missed (HA restart, network hiccup), the 5-minute sweep catches it. The 100% hard cutoff is separate from the 80% target because a tablet that jumps from 79% to 100% between state reports would otherwise stay charging indefinitely.\nTesting with time travel # Some automations are timer-based — they do something after N hours. The test framework has a time_travel helper that fast-forwards timers without actually waiting:\ndef test_projector_sends_notification_after_6_hours( given_that, theatre_projector_app, time_travel, assert_that ): given_that.state_of(\u0026#34;media_player.theater\u0026#34;).is_set_to(\u0026#34;on\u0026#34;) theatre_projector_app.projector_turned_on( \u0026#34;media_player.theater\u0026#34;, \u0026#34;state\u0026#34;, \u0026#34;off\u0026#34;, \u0026#34;on\u0026#34;, {} ) time_travel.fast_forward(21600) # 6 hours in seconds theatre_projector_app.check_projector_6_hours( {\u0026#34;entity\u0026#34;: \u0026#34;media_player.theater\u0026#34;} ) assert_that().notification().was_sent_with_message( \u0026#34;Projector has been on for 6 hours\u0026#34; ) This tests that the projector automation sends a push notification after 6 hours and force-powers-off the projector after 12. Without time travel, you\u0026rsquo;d have to wait 12 actual hours to verify the behavior.\nThe testing payoff # The tests have caught three classes of bugs:\nFlag interactions. The morning_open_enabled bug above — a flag meant to disable one behavior accidentally gated a different behavior. Timer state leaks. A cancelled timer\u0026rsquo;s callback still firing because the handle wasn\u0026rsquo;t properly cleared. Edge case arithmetic. Temperature thresholds that didn\u0026rsquo;t handle None (sensor unavailable) or string values from HA. None of these would have been caught by \u0026ldquo;deploy it and watch what happens\u0026rdquo; — they\u0026rsquo;re the kind of bugs that only manifest under specific conditions (cold snap + garage door + recently restarted HA) and are nearly impossible to reproduce manually.\nRunning the tests is just pytest:\ncd environments/bowerhaus/apps/appdaemon pip install appdaemontestframework pytest tests/ No Kubernetes cluster needed, no Home Assistant instance, no real hardware. The tests run in CI alongside kustomize build validation.\nIs it over-engineered? # Maybe. Plenty of people run Home Assistant automations from the UI and never look back. But I manage 20+ automations across six thermostats, three garage doors, five window shade motors, three tablets, and a projector. At that scale, \u0026ldquo;edit YAML in the HA UI and hope for the best\u0026rdquo; stops working. Automations interact with each other in ways that are hard to predict without tests, and bugs in physical-world automations have real consequences.\nThe ConfigMap deployment model means every change goes through Git, gets reviewed in a diff, and can be rolled back. The tests mean I can refactor an automation without manually verifying every scenario. It\u0026rsquo;s the same discipline we apply to production services — it just happens to control window shades instead of API endpoints.\n","date":"July 16, 2026","externalUrl":null,"permalink":"/posts/unit-tested-appdaemon-automations-kubernetes/","section":"Posts","summary":"Home automations are code that controls physical things. When they break, the consequences aren’t a 500 error — they’re a garage door that won’t close in a snowstorm, or window shades that open at 3 AM, or a tablet that charges to 100% and stays plugged in for months until the battery swells.\n","title":"Unit-Testing Home Automations That Run in Kubernetes","type":"posts"},{"content":"My grandfather had about 10,000 35mm slides. I rented a SlideSnap X1 and spent a weekend feeding them through — 33 boxes worth, organized into folders by box. The scanner itself was great, but when you\u0026rsquo;re pushing through thousands of slides in a weekend, some inevitably go in upside down or backwards. No metadata, no EXIF orientation flags — just thousands of JPEGs, some right-side up, some not, sitting on a NAS.\nManually reviewing 10,000 images isn\u0026rsquo;t realistic. But a vision LLM running on local hardware can look at each one and answer a simple question: is this upside down?\nThe problem # Scanned slides can be wrong in four orientations:\nOrientation What happened Correct Slide was loaded properly Upside down (180°) Slide was inserted flipped vertically Mirrored Slide was scanned from the emulsion side Mirrored + upside down Both problems at once Upside-down images are by far the most common issue. Mirror flips are harder to detect without readable text in the image, so I focused on rotation first.\nWhy not digiKam or OpenCV? # I tried digiKam first. It has an auto-rotate feature that uses DNN-based orientation detection, and it kind of works — but it felt painfully laggy when processing thousands of images, and the accuracy wasn\u0026rsquo;t great. It would confidently leave obviously upside-down photos untouched. For a few hundred photos it\u0026rsquo;s probably fine, but at this scale I needed something I could script, run unattended, and verify afterwards.\nPure classical computer vision doesn\u0026rsquo;t help much either. You could try detecting faces and checking if they\u0026rsquo;re inverted, but many of these photos are landscapes, buildings, or candid shots without clear faces. Edge detection and gradient analysis can suggest a dominant \u0026ldquo;up\u0026rdquo; direction, but they\u0026rsquo;re unreliable on photos with ambiguous composition — a snow-covered mountain reflected in a lake, for example.\nThe task requires the kind of semantic understanding that a human uses: sky goes up, people stand on their feet, buildings point upward, text reads left-to-right. A vision language model does exactly this.\nThe setup # I already had Ollama running on a Mac Studio (M1 Ultra, 64 GB RAM) for Frigate\u0026rsquo;s camera event descriptions. The Mac Studio sits on the same network as the NAS, so the pipeline is:\nRead image from NAS (SMB mount) Send to Ollama\u0026rsquo;s vision model via HTTP API Parse the response Record the result For the model, I used llama3.2-vision (11B parameters). It fits comfortably in 64 GB of RAM and processes each image in about 15 seconds. The smaller minicpm-v was faster but significantly less accurate.\nWhat didn\u0026rsquo;t work: the grid approach # My first attempt was clever but wrong. I created a 2x2 grid showing all four possible orientations of each image (original, mirrored, rotated 180°, mirrored+rotated), labeled A through D, and asked the model to pick which one looked correct.\n┌─────────┬─────────┐ │ A: orig │ B: mirr │ ├─────────┼─────────┤ │ C: 180° │ D: both │ └─────────┴─────────┘ The model said \u0026ldquo;A\u0026rdquo; for every single image. The grid made each sub-image too small to reason about, and the model defaulted to the first option. Five for five wrong.\nWhat worked: one question at a time # Splitting the problem into a simple binary question on the full-resolution image worked much better. The prompt:\nLook at this scanned photograph. Is it UPSIDE DOWN? Check these things: - Are people\u0026#39;s heads at the BOTTOM of the image? That means upside down. - Is the sky or ceiling at the BOTTOM? That means upside down. - Are buildings, trees, or poles pointing DOWNWARD? That means upside down. - Is the ground or floor at the TOP of the image? That means upside down. Respond in EXACTLY this format (two lines, nothing else): UPSIDE_DOWN: YES or NO REASON: Brief explanation On a test set of 5 images (3 upside-down, 2 correct), this got all 5 right. The key insight: give the model the full image at a reasonable resolution, ask one simple question, and tell it exactly what format to respond in.\nBlur detection for free # While I was processing each image, I also wanted to flag blurry scans that might need to be re-done. This part doesn\u0026rsquo;t need an LLM at all — the Laplacian variance method works well:\ndef calculate_blur_score(image_path): img = cv2.imread(str(image_path), cv2.IMREAD_GRAYSCALE) # Normalize for different scan resolutions h, w = img.shape if max(h, w) \u0026gt; 1024: scale = 1024 / max(h, w) img = cv2.resize(img, (int(w * scale), int(h * scale))) return float(cv2.Laplacian(img, cv2.CV_64F).var()) The Laplacian operator approximates the second derivative of the image — sharp edges produce high values, blurry regions produce low values. The variance of the Laplacian across the whole image gives a single sharpness score. Higher is sharper.\nA threshold of 100 worked well for these scans. Anything below that was visibly soft.\nSampling before committing # Running 10,000 images through a vision model at 15 seconds each would take about 42 hours. Before committing to that, I sampled 3 random images from each of the 33 folders to estimate which ones had problems:\nFolder Total Sampled Upside↓ % ------------------------- ------ -------- -------- ------ Batch 01 81 3 2 67% \u0026lt;\u0026lt;\u0026lt; Batch 02 70 3 2 67% \u0026lt;\u0026lt;\u0026lt; Batch 03 79 3 2 67% \u0026lt;\u0026lt;\u0026lt; Batch 04 81 3 2 67% \u0026lt;\u0026lt;\u0026lt; Batch 05 78 3 1 33% Batch 06 541 3 1 33% Batch 07 441 3 1 33% Batch 08 420 3 0 0% Batch 09 1305 3 0 0% Batch 10 1085 3 0 0% Batch 11 1373 3 0 0% ... 99 sample images, 13 minutes, and I had a priority map. Four folders had 67% upside-down rates — those ~311 images should be processed first. The large folders (4,000+ images) looked clean. The sampling pass likely saved 30+ hours of unnecessary processing.\nThe script # The full script is about 400 lines of Python. The core loop is straightforward:\nfor img_path in image_files: # Blur score (instant, no LLM needed) blur_score = calculate_blur_score(img_path) # Orientation check (sends image to Ollama) img_b64 = image_to_b64(img_path, max_size=1024) text = ask_vision(ollama_url, model, UPSIDE_DOWN_PROMPT, img_b64) is_upside_down = parse_response(text) results.append(ImageResult( path=str(img_path), blur_score=blur_score, orientation=\u0026#34;C\u0026#34; if is_upside_down else \u0026#34;A\u0026#34;, correction=\u0026#34;rotate_180\u0026#34; if is_upside_down else \u0026#34;none\u0026#34;, )) It generates an HTML report with side-by-side thumbnails (as-scanned vs. corrected) so you can visually verify before applying fixes. The fixes themselves are non-destructive — each corrected image gets a .original backup:\n# Review first python slide_analyzer.py \u0026#34;/Volumes/slides/batch-01\u0026#34; # Apply corrections from the report python slide_analyzer.py --fix-from slide_report.json Performance # Metric Value Model llama3.2-vision (11B) Hardware Mac Studio M1 Ultra, 64 GB Time per image ~15 seconds Accuracy (test set) 5/5 Sample pass (99 images) 13 minutes Estimated full run (10,000 images) ~42 hours Estimated full run (priority folders only) ~1.3 hours What I\u0026rsquo;d do differently # Mirror detection is hard. I initially tried to detect horizontal flips too, but the model was unreliable — it flagged correctly-oriented images as mirrored. Without readable text in the photo, there often aren\u0026rsquo;t enough visual cues. Mirror detection probably needs a more capable model or a different approach (OCR on the image, then check if the text is backwards).\nA larger model might help. The 11B llama3.2-vision worked well for upside-down detection, which is a relatively easy spatial reasoning task. Mirror detection is subtler and might benefit from a 34B+ parameter model. With 64 GB of RAM, llava:34b would fit, but I haven\u0026rsquo;t tested it yet.\nBatch the Ollama calls. The current script processes images sequentially because Ollama handles one inference at a time on a single GPU. If you had multiple GPUs or were using a cloud API, you could parallelize this significantly.\nConclusion # The combination of \u0026ldquo;cheap classical CV for the easy stuff\u0026rdquo; (blur detection) and \u0026ldquo;local vision LLM for the semantic stuff\u0026rdquo; (orientation detection) turned a multi-day manual review into something that runs overnight. The sampling strategy — check a few images per folder before processing everything — is the kind of optimization that seems obvious in retrospect but saves enormous amounts of time.\nThe script, the Ollama model, and the NAS are all local. No images leave the network, no API costs, no rate limits. That matters when the photos are your grandfather\u0026rsquo;s personal history.\n","date":"February 23, 2026","externalUrl":null,"permalink":"/posts/fixing-upside-down-scanned-slides-vision-llm/","section":"Posts","summary":"My grandfather had about 10,000 35mm slides. I rented a SlideSnap X1 and spent a weekend feeding them through — 33 boxes worth, organized into folders by box. The scanner itself was great, but when you’re pushing through thousands of slides in a weekend, some inevitably go in upside down or backwards. No metadata, no EXIF orientation flags — just thousands of JPEGs, some right-side up, some not, sitting on a NAS.\n","title":"Fixing 10,000 Upside-Down Scanned Slides with a Local Vision LLM","type":"posts"},{"content":"","date":"February 23, 2026","externalUrl":null,"permalink":"/tags/ollama/","section":"Tags","summary":"","title":"Ollama","type":"tags"},{"content":"","date":"February 23, 2026","externalUrl":null,"permalink":"/tags/opencv/","section":"Tags","summary":"","title":"Opencv","type":"tags"},{"content":"","date":"February 14, 2026","externalUrl":null,"permalink":"/categories/data/","section":"Categories","summary":"","title":"Data","type":"categories"},{"content":"","date":"February 14, 2026","externalUrl":null,"permalink":"/tags/duckdb/","section":"Tags","summary":"","title":"Duckdb","type":"tags"},{"content":"CMS recently published provider-level Medicaid spending data from T-MSIS — every fee-for-service, managed care, and CHIP claim from 2018 through 2024, aggregated by billing provider, procedure code, and month. 227 million rows. $1.09 trillion in payments. I wanted to see what falls out when you run some basic fraud heuristics against it.\nThe dataset is available at opendata.hhs.gov/datasets/medicaid-provider-spending/.\nThe dataset # The download is a single 2.94 GB parquet file with seven columns:\nColumn Type Description BILLING_PROVIDER_NPI_NUM string NPI of the billing provider SERVICING_PROVIDER_NPI_NUM string NPI of the servicing provider HCPCS_CODE string Procedure code CLAIM_FROM_MONTH string Month (YYYY-MM) TOTAL_UNIQUE_BENEFICIARIES int64 Unique patients TOTAL_CLAIMS int64 Claim count TOTAL_PAID double Dollars paid by Medicaid Here\u0026rsquo;s what the first few rows look like — the data is sorted by TOTAL_PAID descending, so the biggest line items are at the top:\n┌──────────────────────────┬────────────────────────────┬────────────┬──────────────────┬────────────────────────────┬──────────────┬──────────────┐ │ BILLING_PROVIDER_NPI_NUM │ SERVICING_PROVIDER_NPI_NUM │ HCPCS_CODE │ CLAIM_FROM_MONTH │ TOTAL_UNIQUE_BENEFICIARIES │ TOTAL_CLAIMS │ TOTAL_PAID │ ├──────────────────────────┼────────────────────────────┼────────────┼──────────────────┼────────────────────────────┼──────────────┼──────────────┤ │ 1376609297 │ 1376609297 │ T1019 │ 2024-07 │ 39765 │ 1205701 │ 118887675.31 │ │ 1376609297 │ 1376609297 │ T1019 │ 2024-08 │ 39677 │ 1152534 │ 115561066.11 │ │ 1376609297 │ 1376609297 │ T1019 │ 2024-05 │ 39678 │ 1157235 │ 112823255.3 │ │ 1376609297 │ 1376609297 │ T1019 │ 2024-06 │ 39834 │ 1164582 │ 111449173.13 │ │ 1376609297 │ 1376609297 │ T1019 │ 2024-09 │ 39527 │ 1099808 │ 111199832.57 │ └──────────────────────────┴────────────────────────────┴────────────┴──────────────────┴────────────────────────────┴──────────────┴──────────────┘ Right away you notice NPI 1376609297 billing $118M in a single month for T1019 (personal care services) with 1.2 million claims. That\u0026rsquo;s going to come up again.\nA quick summary of the full dataset:\ncon.sql(f\u0026#34;\u0026#34;\u0026#34; SELECT COUNT(*) as total_rows, COUNT(DISTINCT BILLING_PROVIDER_NPI_NUM) as unique_billing_npis, COUNT(DISTINCT SERVICING_PROVIDER_NPI_NUM) as unique_servicing_npis, COUNT(DISTINCT HCPCS_CODE) as unique_procedures, MIN(CLAIM_FROM_MONTH) as earliest_month, MAX(CLAIM_FROM_MONTH) as latest_month, SUM(TOTAL_PAID) as total_paid FROM \u0026#39;{PATH}\u0026#39; \u0026#34;\u0026#34;\u0026#34;) 227M rows, 617,503 billing providers, 1.6M servicing providers, 10,881 procedure codes, spanning 84 months (Jan 2018 – Dec 2024), totaling $1.09 trillion.\nWhy parquet, not CSV # HHS offers this dataset as a ZIP download. Inside is parquet, not CSV. This matters a lot for a file this size.\nCompression. The parquet file is 2.94 GB. The same data in CSV would be roughly 15-20 GB. Parquet uses column-oriented compression (dictionary encoding for the repeated NPI strings, run-length encoding for sorted columns, and Snappy/ZSTD compression on top). The three NPI/code string columns have high cardinality but lots of repetition within row groups — exactly the pattern parquet compresses best.\nColumn pruning. If your query only touches BILLING_PROVIDER_NPI_NUM and TOTAL_PAID, parquet readers skip the other five columns entirely. With CSV, you always read every byte of every row. For a 7-column file, that\u0026rsquo;s up to 5/7ths of I/O you never have to do.\nPredicate pushdown. DuckDB can push WHERE clauses down into the parquet scan, skipping entire row groups whose min/max statistics prove no rows match. When I filter WHERE TOTAL_CLAIMS \u0026gt; 1000, DuckDB doesn\u0026rsquo;t even read row groups where the max claims value is under 1,000.\nType preservation. CSV makes you guess types and parse strings. Parquet knows TOTAL_PAID is a double and TOTAL_CLAIMS is an int64 — no parsing overhead, no type ambiguity.\nFor a dataset with 227M rows, these differences compound. The parquet file isn\u0026rsquo;t just smaller — it\u0026rsquo;s fundamentally faster to query.\nDon\u0026rsquo;t load 227M rows into pandas # My first attempt was naive:\nimport pandas as pd df = pd.read_parquet(\u0026#39;/data/medicaid-provider-spending.parquet\u0026#39;) The Jupyter kernel OOM-killed immediately on a 16 GB pod.\nThe math makes it obvious in hindsight. Parquet decompresses to a much larger in-memory representation, and pandas stores each string as a Python object with ~56 bytes of overhead regardless of string length. Three string columns across 227M rows:\n227M rows × 3 string columns × ~56 bytes/object ≈ 38 GB + 2 int64 columns × 8 bytes × 227M ≈ 3.6 GB + 1 float64 column × 8 bytes × 227M ≈ 1.8 GB ≈ 43 GB total Even a 64 GB pod would be tight once you start doing groupbys that create intermediate DataFrames.\nThe fix: DuckDB. It queries parquet files directly on disk using memory-mapped I/O, streaming through row groups without materializing the full dataset. Peak memory usage stays in the low single-digit GBs. Every query in this analysis ran in under 3 minutes on that same 16 GB pod.\nimport duckdb PATH = \u0026#39;/data/medicaid-provider-spending.parquet\u0026#39; con = duckdb.connect() con.sql(f\u0026#34;SELECT COUNT(*) FROM \u0026#39;{PATH}\u0026#39;\u0026#34;) # → (227083361,) No read_parquet(), no DataFrames, no OOM. DuckDB treats the parquet file as a table and pushes the full query plan — filters, aggregations, window functions — down to the scan.\nThe trick is to do all the heavy lifting in DuckDB SQL, then call .df() only on the final result sets (which are thousands of rows, not millions):\n# Heavy aggregation stays in DuckDB — result is ~68K rows, perfectly fine for pandas outliers = con.sql(f\u0026#34;\u0026#34;\u0026#34; SELECT ... FROM \u0026#39;{PATH}\u0026#39; GROUP BY ... HAVING z_score \u0026gt; 3 \u0026#34;\u0026#34;\u0026#34;).df() Five fraud signals # Healthcare fraud detection comes down to finding providers whose billing patterns deviate sharply from their peers. I built five independent signals and combined them into a composite risk score. Here\u0026rsquo;s every query.\n1. Cost-per-beneficiary outliers # For each provider and procedure code, I sum the total dollars paid and divide by unique beneficiaries across all months. Then z-score within each procedure code (restricting to procedures with 20+ providers for stable statistics). Anything above z=3 is flagged.\noutliers_cost = con.sql(f\u0026#34;\u0026#34;\u0026#34; WITH prov_proc AS ( SELECT BILLING_PROVIDER_NPI_NUM, HCPCS_CODE, SUM(TOTAL_PAID) as total_paid, SUM(TOTAL_UNIQUE_BENEFICIARIES) as total_benes, SUM(TOTAL_CLAIMS) as total_claims, COUNT(DISTINCT CLAIM_FROM_MONTH) as months_active, SUM(TOTAL_PAID) / GREATEST(SUM(TOTAL_UNIQUE_BENEFICIARIES), 1) as paid_per_bene FROM \u0026#39;{PATH}\u0026#39; GROUP BY BILLING_PROVIDER_NPI_NUM, HCPCS_CODE ), proc_stats AS ( SELECT HCPCS_CODE, COUNT(*) as num_providers, AVG(paid_per_bene) as mean_ppb, STDDEV_POP(paid_per_bene) as std_ppb FROM prov_proc GROUP BY HCPCS_CODE HAVING COUNT(*) \u0026gt;= 20 ), scored AS ( SELECT p.*, (p.paid_per_bene - s.mean_ppb) / NULLIF(s.std_ppb, 0) as z_paid_per_bene FROM prov_proc p JOIN proc_stats s USING (HCPCS_CODE) ) SELECT * FROM scored WHERE z_paid_per_bene \u0026gt; 3 ORDER BY z_paid_per_bene DESC \u0026#34;\u0026#34;\u0026#34;).df() This scans the full 2.94 GB parquet file, builds the two-level aggregation entirely in DuckDB, and returns only the outliers. It ran in 192 seconds.\nResult: 68,037 provider/procedure combinations flagged across 27,348 unique billing NPIs.\nThe top hit: NPI 1467653303 billing $10,416 per beneficiary for CPT 99213 — a standard 15-minute office visit that typically pays around $57. A z-score of 183. They billed $541K for 52 beneficiaries in a single month. Either those were the most expensive office visits in the history of medicine, or something\u0026rsquo;s wrong.\nOther standouts:\nNPI 1437412525: $22,064/beneficiary for 92507 (speech therapy), z=94 NPI 1073608998: $65,778/beneficiary for J3490 (unclassified drugs), z=68 NPI 1427138726: $8,817/beneficiary for 99232 (subsequent hospital care), z=62 — and they did it across 16 months, billing $6M total 2. Billing mill detection # In Medicaid, the billing provider is the entity that submits the claim, and the servicing provider is the one who actually performed the service. Most providers bill for their own services — BILLING_PROVIDER_NPI_NUM = SERVICING_PROVIDER_NPI_NUM. A billing mill funnels claims through many servicing providers under one billing entity, taking a cut.\nbilling_mill = con.sql(f\u0026#34;\u0026#34;\u0026#34; SELECT BILLING_PROVIDER_NPI_NUM, COUNT(DISTINCT SERVICING_PROVIDER_NPI_NUM) as num_servicing_npis, COUNT(DISTINCT HCPCS_CODE) as num_procedures, SUM(TOTAL_PAID) as total_paid, SUM(TOTAL_CLAIMS) as total_claims, SUM(TOTAL_UNIQUE_BENEFICIARIES) as total_benes, COUNT(DISTINCT CLAIM_FROM_MONTH) as months_active FROM \u0026#39;{PATH}\u0026#39; GROUP BY BILLING_PROVIDER_NPI_NUM ORDER BY num_servicing_npis DESC \u0026#34;\u0026#34;\u0026#34;).df() The distribution tells the story:\ncount 617,503 mean 4.6 std 33.0 25% 1.0 50% 1.0 ← median is 1 75% 1.0 ← 75th percentile is still 1 max 5,746 The median and 75th percentile are both 1. Most providers bill for themselves. Then there\u0026rsquo;s NPI 1679525919 with 5,746 servicing NPIs, billing $863 million across 84 months with 1,579 distinct procedure codes. That\u0026rsquo;s either a massive health system or something worth investigating.\nI flag the top 1% by servicing NPI count as potential billing mills.\n3. Volume impossibilities # Each row in this dataset is already aggregated to one billing-provider/servicing-provider/procedure/month combination. So when TOTAL_CLAIMS exceeds 1,000 for a single row, that means one provider billed over 1,000 claims for one procedure code in one month — averaging 50+ per working day with zero days off.\nhigh_volume = con.sql(f\u0026#34;\u0026#34;\u0026#34; SELECT BILLING_PROVIDER_NPI_NUM, SERVICING_PROVIDER_NPI_NUM, HCPCS_CODE, CLAIM_FROM_MONTH, TOTAL_CLAIMS, TOTAL_UNIQUE_BENEFICIARIES, TOTAL_PAID FROM \u0026#39;{PATH}\u0026#39; WHERE TOTAL_CLAIMS \u0026gt; 1000 ORDER BY TOTAL_CLAIMS DESC \u0026#34;\u0026#34;\u0026#34;).df() Result: 1,783,926 rows exceeded 1,000 claims/month across 30,143 billing NPIs, totaling $391 billion.\nThe single most extreme: NPI 1225163876 billed 1,607,071 claims in February 2022 for code 1286Z — but only to 1,906 beneficiaries, and was paid just $230K. That\u0026rsquo;s 842 claims per beneficiary in a single month. The low payment amount suggests these may be bulk capitation or encounter records rather than fee-for-service fraud, but the ratio is still bizarre.\nThe real volume monster is NPI 1376609297 (the same provider from the first row of the dataset) — they consistently bill 1.1-1.2 million claims per month for T1019, each month paying out $100M+. They occupy 19 of the top 20 highest-volume rows.\n4. Spending spike detection # Fraudulent providers often show a \u0026ldquo;ramp and run\u0026rdquo; pattern — billing spikes dramatically before the entity disappears or gets caught. I used LAG() to compute month-over-month spending growth per provider and flagged anything with \u0026gt;5x growth where the spike month exceeded $50K:\nspikes = con.sql(f\u0026#34;\u0026#34;\u0026#34; WITH monthly AS ( SELECT BILLING_PROVIDER_NPI_NUM, CLAIM_FROM_MONTH, SUM(TOTAL_PAID) as monthly_paid, SUM(TOTAL_CLAIMS) as monthly_claims, SUM(TOTAL_UNIQUE_BENEFICIARIES) as monthly_benes FROM \u0026#39;{PATH}\u0026#39; GROUP BY BILLING_PROVIDER_NPI_NUM, CLAIM_FROM_MONTH ), with_prev AS ( SELECT *, LAG(monthly_paid) OVER ( PARTITION BY BILLING_PROVIDER_NPI_NUM ORDER BY CLAIM_FROM_MONTH ) as prev_paid FROM monthly ) SELECT BILLING_PROVIDER_NPI_NUM, CLAIM_FROM_MONTH, prev_paid, monthly_paid, monthly_paid / GREATEST(prev_paid, 1) as growth_ratio, monthly_claims, monthly_benes FROM with_prev WHERE monthly_paid / GREATEST(prev_paid, 1) \u0026gt; 5 AND monthly_paid \u0026gt; 50000 AND prev_paid IS NOT NULL ORDER BY growth_ratio DESC \u0026#34;\u0026#34;\u0026#34;).df() This query aggregates 227M rows to provider-month level, computes lag-based growth ratios via a window function, and filters — all pushed down into DuckDB\u0026rsquo;s execution engine. It ran in 91 seconds.\nResult: 18,916 spike events across 13,527 providers.\nThe top spikes all share a pattern: prev_paid of $0 (or near-zero) followed by millions in the next month. NPI 1770700221 went from $0 to $4.76M in July 2023. NPI 1336117670 went from $0 to $4.74M in February 2018. These are providers that appear out of nowhere with massive billing.\nNPI 1144347824 is interesting — they appear four times in the top 20 with spikes in Dec 2020, Feb 2021, Apr 2021, and Jun 2021. Repeated $0-to-$550K+ cycles suggest an on-again-off-again billing pattern.\n5. Procedure concentration # Most providers bill across multiple procedure codes, even within a narrow specialty. A provider deriving \u0026gt;95% of all Medicaid revenue from a single HCPCS code at high dollar volumes can indicate upcoding (always picking the most expensive code) or phantom billing (billing for services never rendered using a single code).\nconcentrated = con.sql(f\u0026#34;\u0026#34;\u0026#34; WITH prov_proc AS ( SELECT BILLING_PROVIDER_NPI_NUM, HCPCS_CODE, SUM(TOTAL_PAID) as total_paid, SUM(TOTAL_CLAIMS) as total_claims, SUM(TOTAL_UNIQUE_BENEFICIARIES) as total_benes FROM \u0026#39;{PATH}\u0026#39; GROUP BY BILLING_PROVIDER_NPI_NUM, HCPCS_CODE ), prov_total AS ( SELECT BILLING_PROVIDER_NPI_NUM, SUM(total_paid) as provider_total FROM prov_proc GROUP BY BILLING_PROVIDER_NPI_NUM ), ranked AS ( SELECT p.*, t.provider_total, p.total_paid / t.provider_total as pct_of_total, ROW_NUMBER() OVER ( PARTITION BY p.BILLING_PROVIDER_NPI_NUM ORDER BY p.total_paid DESC ) as rn FROM prov_proc p JOIN prov_total t USING (BILLING_PROVIDER_NPI_NUM) WHERE t.provider_total \u0026gt; 500000 ) SELECT * FROM ranked WHERE rn = 1 AND pct_of_total \u0026gt; 0.95 ORDER BY provider_total DESC \u0026#34;\u0026#34;\u0026#34;).df() Result: 25,397 providers with \u0026gt;$500K total and \u0026gt;95% from one code, representing $193 billion.\nThe dominant code across the top 20 is T1019 (personal care services). NPI 1376609297 tops the list at $5.5 billion, 98% from T1019. The next five are also T1019 providers at $1-3B each. These are likely large home and community-based services (HCBS) agencies — the concentration on one code may be legitimate program design, but the scale demands scrutiny.\nNPI 1932341898 stands out: $997M, 99.99% from H0044 (supported employment), with only two procedure codes total. That\u0026rsquo;s extreme even for this list.\nComposite scoring # Each provider gets a binary flag for each of the five signals. The composite fraud score is just the sum.\nimport pandas as pd all_providers = con.sql(f\u0026#34;\u0026#34;\u0026#34; SELECT BILLING_PROVIDER_NPI_NUM, SUM(TOTAL_PAID) as total_spending FROM \u0026#39;{PATH}\u0026#39; GROUP BY BILLING_PROVIDER_NPI_NUM \u0026#34;\u0026#34;\u0026#34;).df() risk = all_providers.copy() flag1_npis = set(outliers_cost[\u0026#39;BILLING_PROVIDER_NPI_NUM\u0026#39;]) risk[\u0026#39;flag_cost_outlier\u0026#39;] = risk[\u0026#39;BILLING_PROVIDER_NPI_NUM\u0026#39;] \\ .isin(flag1_npis).astype(int) threshold_mill = billing_mill[\u0026#39;num_servicing_npis\u0026#39;].quantile(0.99) flag2_npis = set(billing_mill[ billing_mill[\u0026#39;num_servicing_npis\u0026#39;] \u0026gt;= threshold_mill ][\u0026#39;BILLING_PROVIDER_NPI_NUM\u0026#39;]) risk[\u0026#39;flag_billing_mill\u0026#39;] = risk[\u0026#39;BILLING_PROVIDER_NPI_NUM\u0026#39;] \\ .isin(flag2_npis).astype(int) flag3_npis = set(high_volume[\u0026#39;BILLING_PROVIDER_NPI_NUM\u0026#39;]) risk[\u0026#39;flag_high_volume\u0026#39;] = risk[\u0026#39;BILLING_PROVIDER_NPI_NUM\u0026#39;] \\ .isin(flag3_npis).astype(int) flag4_npis = set(spikes[\u0026#39;BILLING_PROVIDER_NPI_NUM\u0026#39;]) risk[\u0026#39;flag_spike\u0026#39;] = risk[\u0026#39;BILLING_PROVIDER_NPI_NUM\u0026#39;] \\ .isin(flag4_npis).astype(int) flag5_npis = set(concentrated[\u0026#39;BILLING_PROVIDER_NPI_NUM\u0026#39;]) risk[\u0026#39;flag_concentrated\u0026#39;] = risk[\u0026#39;BILLING_PROVIDER_NPI_NUM\u0026#39;] \\ .isin(flag5_npis).astype(int) flag_cols = [ \u0026#39;flag_cost_outlier\u0026#39;, \u0026#39;flag_billing_mill\u0026#39;, \u0026#39;flag_high_volume\u0026#39;, \u0026#39;flag_spike\u0026#39;, \u0026#39;flag_concentrated\u0026#39; ] risk[\u0026#39;fraud_score\u0026#39;] = risk[flag_cols].sum(axis=1) This is the one step where I use pandas — the DuckDB queries already reduced 227M rows down to manageable result sets (tens of thousands of NPIs), so the set-membership lookups and joins are trivial.\nDistribution:\nScore Providers 0 539,161 1 57,424 2 17,690 3 3,034 4 194 2+ 20,918 The 20,918 providers with 2+ flags account for $535 billion — nearly half of all Medicaid spending in the dataset.\nThe top 5 # For the highest-scoring providers, I pulled their full procedure breakdowns and monthly spending ranges:\nfor npi in top5: proc_breakdown = con.sql(f\u0026#34;\u0026#34;\u0026#34; SELECT HCPCS_CODE, SUM(TOTAL_PAID) as paid, SUM(TOTAL_CLAIMS) as claims, SUM(TOTAL_UNIQUE_BENEFICIARIES) as benes FROM \u0026#39;{PATH}\u0026#39; WHERE BILLING_PROVIDER_NPI_NUM = \u0026#39;{npi}\u0026#39; GROUP BY HCPCS_CODE ORDER BY paid DESC LIMIT 5 \u0026#34;\u0026#34;\u0026#34;).df() NPI 1700090834 — Score 4/5 ($1.13B) # Flags: cost outlier, billing mill, high volume, spike\nHCPCS_CODE paid claims benes pct H0019 553,957,700 2,884,496 155,197 53.2% H0004 188,211,147 1,572,012 505,021 18.1% H0005 108,477,327 1,623,733 203,179 10.4% H0020 103,053,473 7,020,805 276,149 9.9% H0006 88,021,893 1,166,081 255,743 8.4% Monthly spending range: $37,576 — $25,877,770 All H-codes (behavioral health). H0019 alone is $554M for day treatment services across 155K beneficiaries. The monthly range swinging from $38K to $25.9M means there were massive ramp-up periods. This has the profile of a large behavioral health managed care entity — but one that also trips cost outlier and billing mill flags.\nNPI 1932341898 — Score 4/5 ($997M) # Flags: cost outlier, high volume, spike, concentrated\nHCPCS_CODE paid claims benes pct H0044 996,688,991 304,663 283,937 100.0% T2028 112,131 1,468 36 0.0% Monthly spending range: $512,298 — $21,818,712 $997M from a single code. H0044 is supported employment — helping people with disabilities find and maintain jobs. 283,937 beneficiaries at an average of $3,511 each. Only two procedure codes ever billed. The 43x spread between minimum and maximum monthly spending ($512K to $21.8M) is hard to explain with normal program growth.\nNPI 1114931391 — Score 4/5 ($382M) # Flags: cost outlier, high volume, spike, concentrated\nHCPCS_CODE paid claims benes pct S3620 370,523,694 1,324,711 974,114 97.1% 83655 8,882,287 685,614 667,070 2.3% 85018 1,601,721 623,389 607,346 0.4% 80061 691,330 45,838 44,922 0.2% 82947 49,396 11,154 10,951 0.0% Monthly spending range: $47 — $15,497,422 97% from S3620 (newborn screening panel). The monthly range from $47 to $15.5M is the most extreme spread in the top 5. This likely represents a state-contracted newborn screening lab — the code and beneficiary counts support that. But a $47 month to $15.5M month means either the contract changed drastically or there are data quality issues. The secondary codes (83655, 85018) are basic lab tests, consistent with a lab operation.\nNPI 1629283197 — Score 4/5 ($306M) # Flags: billing mill, high volume, spike, concentrated\nHCPCS_CODE paid claims benes pct T1015 298,897,046 718,309 655,601 99.1% 99393 722,772 52,037 39,635 0.2% 99392 688,366 47,287 38,251 0.2% 99394 608,918 31,656 24,001 0.2% 92014 591,683 27,987 16,169 0.2% Monthly spending range: $594,623 — $5,028,749 99% from T1015 (clinic-based behavioral health). The secondary codes (99392-99394) are preventive care E\u0026amp;M visits, and 92014 is an eye exam — suggesting this billing entity has a clinic component too. But the billing mill flag means claims are flowing through many servicing NPIs under this one billing number.\nNPI 1619341716 — Score 4/5 ($303M) # Flags: cost outlier, billing mill, high volume, spike\nHCPCS_CODE paid claims benes pct 99211 45,853,871 488,964 412,175 34.6% 90832 37,594,923 253,130 125,580 28.4% 99213 29,286,349 255,935 204,037 22.1% 99214 9,967,573 88,118 73,463 7.5% 99212 9,740,312 66,604 55,192 7.4% Monthly spending range: $66,177 — $7,160,676 The most diversified billing pattern of the top 5. A mix of E\u0026amp;M visits (99211-99214) and psychotherapy (90832). The cost outlier flag means they\u0026rsquo;re charging significantly more per beneficiary than peers for these common codes. The billing mill flag means many servicing NPIs are operating under this one billing entity. The 108x spread between min and max monthly billing ($66K to $7.2M) indicates major scaling events.\nCaveats # These are flags, not findings. Several patterns have legitimate explanations:\nLarge health systems and MCOs naturally have many servicing NPIs — that\u0026rsquo;s not fraud, it\u0026rsquo;s organizational structure. The billing mill signal needs to be interpreted in context. State-contracted labs (like the newborn screening provider at NPI 1114931391) can have legitimate volume and spending spikes tied to contract awards and state program changes. T1019/T1015/H-codes are commonly used in home and community-based services (HCBS) and behavioral health, where high volumes per billing entity may reflect program design rather than fraud. States often contract with large managed care entities for these services. This data is aggregated — we can\u0026rsquo;t see individual claim details, diagnosis codes, patient demographics, or service locations. Many fraud patterns only become visible at the claim level. The real value is in combining signals. A provider that\u0026rsquo;s both a cost outlier and has billing mill patterns and shows spending spikes is far more suspicious than one that only trips a single wire. The 194 providers at score 4/5 are the ones I\u0026rsquo;d start investigating.\nTechnical notes # The full analysis runs as a Jupyter notebook backed by DuckDB on a 16 GB Kubernetes pod. Some implementation details worth noting:\nDuckDB\u0026rsquo;s parquet performance is remarkable. The z-score query (Signal 1) does a full scan → two-level aggregation → window function → filter, all on 227M rows of parquet. It completes in 192 seconds. The spending spike query uses LAG() window functions over the aggregated monthly data — 91 seconds. These are the kinds of queries that would require Spark or a data warehouse for most teams.\nThe pandas handoff is intentional. DuckDB returns result sets via .df() only after reducing 227M rows to thousands. The composite scoring step — five set-membership lookups and a sum — is trivial in pandas. There\u0026rsquo;s no reason to push that into SQL.\nGREATEST(x, 1) prevents division by zero. Several providers have zero beneficiaries or zero prior-month spending. Rather than filtering them out (and potentially missing interesting cases), I clamp the denominator to 1. This means zero-bene providers get a per-bene cost equal to their total paid — which is correct behavior for flagging purposes.\n","date":"February 14, 2026","externalUrl":null,"permalink":"/posts/medicaid-fraud-detection-duckdb-227m-rows/","section":"Posts","summary":"CMS recently published provider-level Medicaid spending data from T-MSIS — every fee-for-service, managed care, and CHIP claim from 2018 through 2024, aggregated by billing provider, procedure code, and month. 227 million rows. $1.09 trillion in payments. I wanted to see what falls out when you run some basic fraud heuristics against it.\n","title":"Finding Fraud in $1 Trillion of Medicaid Data with DuckDB","type":"posts"},{"content":"","date":"February 14, 2026","externalUrl":null,"permalink":"/tags/jupyter/","section":"Tags","summary":"","title":"Jupyter","type":"tags"},{"content":"","date":"February 14, 2026","externalUrl":null,"permalink":"/tags/parquet/","section":"Tags","summary":"","title":"Parquet","type":"tags"},{"content":"My Home Assistant dashboard has evolved significantly over the past year. What started as the default auto-generated cards has become a carefully organized interface that my whole family actually uses. Here\u0026rsquo;s how I built it, with a focus on the whole home audio system that ties everything together.\nDashboard Philosophy # Before diving into implementation, a few principles guided the design:\nFunction over flash - Every card earns its screen space One-tap actions - Common tasks shouldn\u0026rsquo;t require drilling into menus Status at a glance - The overview tells you what\u0026rsquo;s happening without interaction Mobile-first - Tablets mounted around the house are the primary interface The Stack # Lovelace YAML mode - Full control over layout and structure Mushroom cards - Clean, consistent UI components Custom Layout Card - Responsive grid layouts that work on any screen Stack-in-card - Grouping related controls together Overview Page # The overview page is designed to answer \u0026ldquo;what\u0026rsquo;s happening right now?\u0026rdquo; at a glance:\n- type: custom:mushroom-chips-card alignment: center chips: - type: weather entity: weather.katw - type: template entity: person.rusty_bower icon: mdi:account icon_color: \u0026#34;{{ \u0026#39;green\u0026#39; if is_state(\u0026#39;person.rusty_bower\u0026#39;, \u0026#39;home\u0026#39;) else \u0026#39;grey\u0026#39; }}\u0026#34; content: Rusty - type: template icon: mdi:lightbulb-group icon_color: \u0026#34;{{ \u0026#39;amber\u0026#39; if states.light | selectattr(\u0026#39;state\u0026#39;, \u0026#39;eq\u0026#39;, \u0026#39;on\u0026#39;) | list | count \u0026gt; 0 else \u0026#39;disabled\u0026#39; }}\u0026#34; content: \u0026#34;{{ states.light | selectattr(\u0026#39;state\u0026#39;, \u0026#39;eq\u0026#39;, \u0026#39;on\u0026#39;) | list | count }} lights\u0026#34; The chip bar at the top shows:\nCurrent weather Who\u0026rsquo;s home (person tracking) How many lights are on (tap to navigate to lighting) Garage door status (red = open, green = closed) HVAC status (heating/cooling active) Upcoming trash/recycling days Quick actions provide one-tap scenes: Goodnight, Spa Mode, All Lights Off, Close All Garage Doors.\nWhole Home Audio Architecture # The audio system uses a Xantech DAX66 multi-zone amplifier - a commercial-grade matrix amplifier that can route any of 6 sources to any of 18 zones independently. Each zone has its own volume control and can play different content simultaneously.\nThe Zones # The house is wired with in-ceiling speakers across 15 zones spanning three floors - main living areas, bedrooms/offices upstairs, and utility/recreation spaces in the basement. Plus a dedicated Theater with an Anthem AV receiver for serious listening.\nIntegration Approach # The Xantech communicates via RS-232, but I needed network access from Kubernetes. The solution is a socat bridge running as a sidecar:\ncontainers: - name: socat image: alpine/socat:latest args: - \u0026#34;-d\u0026#34; - \u0026#34;-d\u0026#34; - \u0026#34;TCP-LISTEN:7001,reuseaddr,fork\u0026#34; - \u0026#34;TCP:10.0.10.209:4999,keepalive,forever,interval=10\u0026#34; This bridges TCP connections from Home Assistant to the serial-to-ethernet adapter connected to the Xantech. The keepalive,forever ensures the connection recovers from network hiccups.\nHome Assistant sees each zone as a media_player entity with source selection and volume control.\nAudio Dashboard Design # Designing the audio UI was tricky. With 15 zones, a traditional media player card for each would be overwhelming. The solution is a two-column layout: streaming controls on the left, zone controls on the right.\nStreaming Column # The maxi-media-player card provides a full-featured player with album art as a blurred background:\n- type: custom:maxi-media-player entities: - media_player.spotify - media_player.zone_1 - media_player.zone_2 # ... all zones sections: - player artworkAsBackgroundBlur: true This gives you playback controls, track info, and the ability to cast to any zone - all in one card.\nZone Controls Column # For the individual zones, I use mushroom-media-player-card with a horizontal layout. Tap to toggle on/off, volume buttons for quick adjustments:\n- type: custom:stack-in-card cards: - type: custom:mushroom-title-card title: Whole Home Audio subtitle: Upstairs - type: custom:mushroom-media-player-card entity: media_player.zone_1 name: Zone 1 layout: horizontal volume_controls: - volume_buttons - volume_set media_controls: [] collapsible_controls: false tap_action: action: toggle - type: custom:mushroom-media-player-card entity: media_player.zone_2 name: Zone 2 layout: horizontal volume_controls: - volume_buttons - volume_set media_controls: [] collapsible_controls: false tap_action: action: toggle # ... more zones - type: custom:mushroom-title-card title: \u0026#34;\u0026#34; subtitle: Main Floor # ... main floor zones The key settings:\nlayout: horizontal - Compact single-line layout volume_controls: [volume_buttons, volume_set] - Both +/- buttons and a slider media_controls: [] - Hide play/pause since we control that from the streaming card tap_action: action: toggle - Tap the card to turn the zone on/off This puts all zones in a scrollable card organized by floor, with the streaming player always visible on the left.\nScene Integration # The real power comes from combining audio with other automations:\nSpa Mode # One tap: dims the outdoor lights, starts a relaxation playlist in the relevant zone, adjusts the HVAC.\n- type: custom:mushroom-template-card primary: Spa Mode icon: mdi:hot-tub icon_color: cyan tap_action: action: call-service service: scene.turn_on target: entity_id: scene.spa_mode Goodnight # Turns off all audio zones, sets overnight temperature setpoints, dims lights to 5% in hallways, locks doors.\nParty Mode # Syncs all main floor and outdoor zones to the same source at matched volumes.\nTheater Integration # The theater runs separately on an Anthem AV receiver, integrated via IP control. It gets its own section with proper media controls:\n- type: custom:mushroom-media-player-card entity: media_player.anthem_av name: Anthem AV use_media_info: true show_volume_level: true volume_controls: - volume_set - volume_mute The theater lighting also has dedicated scene buttons: Off, Movie (5% seating lights), Intermission (50%), and Full brightness for cleaning.\nTablet Deployment # Fire tablets run Fully Kiosk Browser showing the dashboard at key locations - one on the main floor for everyday use, one upstairs, and a portable tablet that docks at a charging station when not in use.\nThe Admin page shows tablet connectivity status so I know if one has gone offline.\nLessons Learned # YAML mode is worth it - The visual editor can\u0026rsquo;t do responsive layouts or template chips Group by function, not location - \u0026ldquo;Lighting\u0026rdquo; view is more useful than \u0026ldquo;Living Room\u0026rdquo; view Limit visible options - Show the 80% use case, hide the edge cases behind taps Test with family - My initial designs were too information-dense What\u0026rsquo;s Next # Music Assistant integration for better source management Voice control via local Whisper for \u0026ldquo;play jazz in the kitchen\u0026rdquo; Presence-based audio - automatically route music to follow occupancy The dashboard continues to evolve, but the core principle stays the same: make the smart home actually easier to use than the dumb version.\n","date":"January 26, 2026","externalUrl":null,"permalink":"/posts/home-assistant-dashboard-whole-home-audio/","section":"Posts","summary":"My Home Assistant dashboard has evolved significantly over the past year. What started as the default auto-generated cards has become a carefully organized interface that my whole family actually uses. Here’s how I built it, with a focus on the whole home audio system that ties everything together.\n","title":"Building a Home Assistant Dashboard with Whole Home Audio Control","type":"posts"},{"content":"","date":"January 26, 2026","externalUrl":null,"permalink":"/categories/dashboards/","section":"Categories","summary":"","title":"Dashboards","type":"categories"},{"content":"","date":"January 26, 2026","externalUrl":null,"permalink":"/tags/lovelace/","section":"Tags","summary":"","title":"Lovelace","type":"tags"},{"content":"","date":"January 26, 2026","externalUrl":null,"permalink":"/tags/mushroom/","section":"Tags","summary":"","title":"Mushroom","type":"tags"},{"content":"","date":"January 17, 2026","externalUrl":null,"permalink":"/tags/argocd/","section":"Tags","summary":"","title":"Argocd","type":"tags"},{"content":"Managing DNS records for a homelab with dozens of services is tedious. Every time you deploy something new, you have to remember to add a DNS record. I solved this by using external-dns to automatically create Pi-hole DNS entries from Kubernetes ingress resources.\nThe Goal # When I create an ingress like this:\napiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: frigate annotations: external-dns.alpha.kubernetes.io/hostname: frigate.bowerha.us spec: rules: - host: frigate.bowerha.us # ... I want Pi-hole to automatically create a DNS record pointing frigate.bowerha.us to my ingress controller\u0026rsquo;s IP. No manual steps, no forgetting to update DNS.\nComponents # Pi-hole - DNS server and ad blocker external-dns - Kubernetes controller that syncs DNS records Kubernetes ingress-nginx - Ingress controller with a known IP Setting Up external-dns # Deployment # apiVersion: apps/v1 kind: Deployment metadata: name: external-dns namespace: external-dns spec: replicas: 1 selector: matchLabels: app: external-dns template: metadata: labels: app: external-dns spec: containers: - name: external-dns image: registry.k8s.io/external-dns/external-dns:v0.14.0 args: - --source=ingress - --provider=pihole - --pihole-server=http://pihole-web.pihole.svc.cluster.local - --pihole-password=$(PIHOLE_PASSWORD) - --domain-filter=bowerha.us - --registry=noop - --policy=upsert-only - --interval=1m env: - name: PIHOLE_PASSWORD valueFrom: secretKeyRef: name: pihole-password key: password Key arguments:\n--source=ingress - Watch ingress resources for DNS records --provider=pihole - Use Pi-hole as the DNS backend --domain-filter=bowerha.us - Only manage records for this domain --policy=upsert-only - Create/update but never delete records --interval=1m - Check for changes every minute Pi-hole Password Secret # apiVersion: v1 kind: Secret metadata: name: pihole-password namespace: external-dns type: Opaque stringData: password: \u0026#34;your-pihole-admin-password\u0026#34; RBAC # external-dns needs permission to read ingresses:\napiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: external-dns rules: - apiGroups: [\u0026#34;\u0026#34;] resources: [\u0026#34;services\u0026#34;, \u0026#34;endpoints\u0026#34;, \u0026#34;pods\u0026#34;] verbs: [\u0026#34;get\u0026#34;, \u0026#34;watch\u0026#34;, \u0026#34;list\u0026#34;] - apiGroups: [\u0026#34;extensions\u0026#34;, \u0026#34;networking.k8s.io\u0026#34;] resources: [\u0026#34;ingresses\u0026#34;] verbs: [\u0026#34;get\u0026#34;, \u0026#34;watch\u0026#34;, \u0026#34;list\u0026#34;] - apiGroups: [\u0026#34;\u0026#34;] resources: [\u0026#34;nodes\u0026#34;] verbs: [\u0026#34;list\u0026#34;, \u0026#34;watch\u0026#34;] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: external-dns roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: external-dns subjects: - kind: ServiceAccount name: external-dns namespace: external-dns Annotating Ingresses # The key annotation is external-dns.alpha.kubernetes.io/hostname:\napiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: home-assistant namespace: homeassistant annotations: cert-manager.io/cluster-issuer: letsencrypt-prod external-dns.alpha.kubernetes.io/hostname: ha.bowerha.us spec: ingressClassName: nginx rules: - host: ha.bowerha.us http: paths: - path: / pathType: Prefix backend: service: name: home-assistant port: number: 8123 tls: - hosts: - ha.bowerha.us secretName: home-assistant-tls Within a minute of applying this ingress, Pi-hole will have a local DNS record for ha.bowerha.us pointing to the ingress controller\u0026rsquo;s LoadBalancer IP.\nHow It Works # external-dns watches for ingress resources with the hostname annotation It queries the ingress controller\u0026rsquo;s service to find its external IP It calls the Pi-hole API to create/update a Local DNS Record Pi-hole serves that record to all clients on the network The records appear in Pi-hole under Local DNS \u0026gt; DNS Records.\nTroubleshooting # external-dns CrashLoopBackOff # Usually means it can\u0026rsquo;t reach Pi-hole. Check:\nIs the Pi-hole service URL correct? Is Pi-hole actually running and responding on port 80? Is the password correct? I had an issue where Pi-hole\u0026rsquo;s FTL process got stuck after a volume issue. Restarting Pi-hole fixed external-dns.\nRecords Not Appearing # Check external-dns logs:\nkubectl logs -n external-dns deployment/external-dns Look for:\nConnection errors to Pi-hole Domain filter mismatches Ingress resources missing the annotation Wrong IP Address # external-dns gets the IP from the ingress controller\u0026rsquo;s service. If you\u0026rsquo;re using MetalLB or a cloud LoadBalancer, make sure the service has an external IP assigned:\nkubectl get svc -n ingress-nginx Making It Standard Practice # I added this to my team\u0026rsquo;s deployment guidelines in CLAUDE.md:\n## DNS Management When creating ingress resources for `bowerha.us` domains, always include the external-dns annotation: annotations: external-dns.alpha.kubernetes.io/hostname: myapp.bowerha.us This automatically creates a Pi-hole DNS record pointing to the ingress controller. No manual DNS configuration needed. Now DNS is just part of the deployment - no separate step to remember.\nResult # What used to be a manual process:\nDeploy app Create ingress Remember to add DNS Log into Pi-hole Add Local DNS record Test that it works Is now:\nDeploy app with annotated ingress Done The DNS record appears automatically within a minute. When I tear down services, I use --policy=upsert-only so records persist (useful for debugging), but you could use --policy=sync for automatic cleanup.\nThis small automation removes friction from deploying new services and eliminates a whole category of \u0026ldquo;why can\u0026rsquo;t I reach this?\u0026rdquo; debugging sessions.\n","date":"January 17, 2026","externalUrl":null,"permalink":"/posts/external-dns-pihole-kubernetes/","section":"Posts","summary":"Managing DNS records for a homelab with dozens of services is tedious. Every time you deploy something new, you have to remember to add a DNS record. I solved this by using external-dns to automatically create Pi-hole DNS entries from Kubernetes ingress resources.\n","title":"Auto-Populating Pi-hole DNS from Kubernetes Ingresses","type":"posts"},{"content":"One of the challenges of running a homelab with dozens of services is keeping track of what\u0026rsquo;s running and where. I recently deployed Homepage - a modern, fully static dashboard that automatically discovers services from Kubernetes ingresses.\nWhy Homepage? # I evaluated several dashboard options including Homarr and Heimdall. Homepage stood out for a few reasons:\nKubernetes-native service discovery - no manual configuration needed Real-time pod status - shows if services are actually running Clean, modern UI - dark theme, customizable layout Lightweight - just a static site with no database The Setup # Homepage runs as a simple deployment in Kubernetes with a ServiceAccount that has read access to the cluster. The magic happens through ingress annotations.\nBase Deployment # apiVersion: apps/v1 kind: Deployment metadata: name: homepage spec: replicas: 1 template: spec: serviceAccountName: homepage initContainers: - name: copy-config image: busybox:1.36 command: [\u0026#39;sh\u0026#39;, \u0026#39;-c\u0026#39;, \u0026#39;cp /config-source/* /config/\u0026#39;] volumeMounts: - name: config-source mountPath: /config-source - name: config mountPath: /config containers: - name: homepage image: ghcr.io/gethomepage/homepage:v0.10.9 volumeMounts: - name: config mountPath: /app/config volumes: - name: config-source configMap: name: homepage-config - name: config emptyDir: {} The init container pattern is important here - Homepage needs a writable config directory to create log files, but ConfigMaps are read-only. We copy the config to an emptyDir volume at startup.\nService Discovery via Annotations # The real power comes from ingress annotations. For each service you want on the dashboard, add these annotations:\napiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: frigate annotations: gethomepage.dev/enabled: \u0026#34;true\u0026#34; gethomepage.dev/name: \u0026#34;Frigate\u0026#34; gethomepage.dev/group: \u0026#34;Home Automation\u0026#34; gethomepage.dev/icon: \u0026#34;frigate.png\u0026#34; gethomepage.dev/pod-selector: \u0026#34;app=frigate\u0026#34; The pod-selector annotation is crucial for status monitoring. Homepage uses this to find the actual pods and show whether they\u0026rsquo;re running. Without it, you\u0026rsquo;ll see \u0026ldquo;not found\u0026rdquo; errors.\nConfigMap for Layout # The ConfigMap controls the dashboard layout and widgets:\napiVersion: v1 kind: ConfigMap metadata: name: homepage-config data: settings.yaml: | title: Homelab theme: dark color: slate headerStyle: clean layout: Home Automation: style: row columns: 4 Media: style: row columns: 4 Infrastructure: style: row columns: 4 widgets.yaml: | - kubernetes: cluster: show: true cpu: true memory: true showLabel: true label: \u0026#34;cluster\u0026#34; nodes: show: true cpu: true memory: true kubernetes.yaml: | mode: cluster services.yaml: | [] docker.yaml: | Note that services.yaml and docker.yaml must be valid YAML (even if empty). Setting services.yaml: \u0026quot;[]\u0026quot; prevents JavaScript errors from empty files.\nGotchas I Encountered # 1. Empty Config Files Cause JS Errors # If services.yaml or docker.yaml are empty or contain only comments, Homepage throws a classList null reference error. Always set them to valid YAML.\n2. Pod Selector Mismatches # Homepage defaults to looking for pods with app.kubernetes.io/name=\u0026lt;ingress-name\u0026gt;. Most of my deployments use simpler labels like app=frigate. The gethomepage.dev/pod-selector annotation fixes this.\n3. Icon Names # Icons come from the Dashboard Icons project. Some names aren\u0026rsquo;t obvious:\nArgoCD: argo-cd.png (not argocd.png) Z-Wave JS: mdi-z-wave (Material Design Icons format) Result # Now every time I add a new service with the proper annotations, it automatically appears on my dashboard with real-time status. No more manually updating bookmark pages or forgetting what port something runs on.\nThe dashboard shows CPU/memory usage for the cluster and nodes, groups services logically, and immediately tells me if something is down. It\u0026rsquo;s become my starting point for accessing everything in the homelab.\n","date":"January 17, 2026","externalUrl":null,"permalink":"/posts/homepage-kubernetes-service-discovery/","section":"Posts","summary":"One of the challenges of running a homelab with dozens of services is keeping track of what’s running and where. I recently deployed Homepage - a modern, fully static dashboard that automatically discovers services from Kubernetes ingresses.\n","title":"Building a Self-Updating Dashboard with Homepage and Kubernetes","type":"posts"},{"content":"","date":"January 17, 2026","externalUrl":null,"permalink":"/tags/drone/","section":"Tags","summary":"","title":"Drone","type":"tags"},{"content":"","date":"January 17, 2026","externalUrl":null,"permalink":"/tags/external-dns/","section":"Tags","summary":"","title":"External-Dns","type":"tags"},{"content":"For years I used Keel to automatically update container images in my Kubernetes clusters. It worked, but as I moved to GitOps with ArgoCD, Keel\u0026rsquo;s push-based approach became a liability. I migrated to Renovate for PR-based image updates, and it\u0026rsquo;s been a significant improvement.\nThe Problem with Keel # Keel watches for new container images and updates deployments directly in the cluster. You can configure it via annotations:\nmetadata: annotations: keel.sh/policy: major keel.sh/trigger: poll When a new image appears, Keel modifies the deployment in-place.\nWhy This Broke Down # GitOps drift - Keel updates the cluster, but not git. ArgoCD sees drift and wants to revert to what\u0026rsquo;s in git. You end up fighting your own tools.\nNo review process - Updates happen automatically. A bad image goes live immediately. Rolling back means finding the old tag and manually updating.\nNotification-only visibility - Keel can send Slack notifications, but there\u0026rsquo;s no audit trail, no PR history, no way to see what changed when.\nLimited version control - Keel\u0026rsquo;s policies (major/minor/patch) are annotation-based and hard to customize per-image.\nRenovate: PR-Based Updates # Renovate takes a different approach. It scans your repository, finds container images, checks for updates, and opens pull requests. The update only happens when you merge the PR.\nThis fits GitOps perfectly - git remains the source of truth.\nMy Renovate Setup # Self-Hosted CronJob # I run Renovate as a Kubernetes CronJob rather than using the hosted service:\napiVersion: batch/v1 kind: CronJob metadata: name: renovate namespace: renovate spec: schedule: \u0026#34;0 */4 * * *\u0026#34; # Every 4 hours concurrencyPolicy: Forbid jobTemplate: spec: template: spec: restartPolicy: Never containers: - name: renovate image: ghcr.io/renovatebot/renovate:42.19.5 env: - name: RENOVATE_CONFIG_FILE value: /config/renovate.json - name: RENOVATE_TOKEN valueFrom: secretKeyRef: name: renovate-env key: RENOVATE_TOKEN volumeMounts: - name: renovate-config mountPath: /config volumes: - name: renovate-config configMap: name: renovate-config Why self-hosted:\nFull control over scan frequency No rate limits from Renovate\u0026rsquo;s hosted service Works with private registries Runs inside my cluster, close to my git server Configuration # The repository-level renovate.json controls what gets updated:\n{ \u0026#34;$schema\u0026#34;: \u0026#34;https://docs.renovatebot.com/renovate-schema.json\u0026#34;, \u0026#34;extends\u0026#34;: [\u0026#34;config:recommended\u0026#34;], \u0026#34;enabledManagers\u0026#34;: [\u0026#34;helm-values\u0026#34;, \u0026#34;kustomize\u0026#34;, \u0026#34;kubernetes\u0026#34;], \u0026#34;kubernetes\u0026#34;: { \u0026#34;managerFilePatterns\u0026#34;: [ \u0026#34;/(^|/)base/.+/(?:[^/]*deployment)\\\\.ya?ml$/\u0026#34; ] }, \u0026#34;prConcurrentLimit\u0026#34;: 20, \u0026#34;prHourlyLimit\u0026#34;: 0, \u0026#34;packageRules\u0026#34;: [ // ... rules ] } The managerFilePatterns is crucial - it tells Renovate to only look at deployment files in my base/ directory. This prevents it from trying to update the same image in multiple overlay locations.\nPackage Rules: The Power Feature # This is where Renovate really shines. You can define complex rules for how different images should be handled.\nPinning Major Versions # Some apps I want to stay on a specific major version until I\u0026rsquo;m ready to upgrade:\n{ \u0026#34;description\u0026#34;: \u0026#34;Keep linuxserver/lidarr on 2.x\u0026#34;, \u0026#34;matchDatasources\u0026#34;: [\u0026#34;docker\u0026#34;], \u0026#34;matchPackageNames\u0026#34;: [\u0026#34;lscr.io/linuxserver/lidarr\u0026#34;], \u0026#34;allowedVersions\u0026#34;: \u0026#34;\u0026lt;3.0.0\u0026#34; } Automerging Safe Updates # Minor and patch updates for stable images can merge automatically:\n{ \u0026#34;description\u0026#34;: \u0026#34;Automerge minor/patch for Linuxserver images\u0026#34;, \u0026#34;matchDatasources\u0026#34;: [\u0026#34;docker\u0026#34;], \u0026#34;matchPackageNames\u0026#34;: [\u0026#34;lscr.io/linuxserver/*\u0026#34;], \u0026#34;matchUpdateTypes\u0026#34;: [\u0026#34;minor\u0026#34;, \u0026#34;patch\u0026#34;], \u0026#34;automerge\u0026#34;: true, \u0026#34;automergeType\u0026#34;: \u0026#34;pr\u0026#34; } The PR is still created (for visibility), but it merges without manual intervention.\nHandling Weird Versioning # LinuxServer images sometimes use date-based tags like 2021.12.15. These confuse semver parsing:\n{ \u0026#34;description\u0026#34;: \u0026#34;Ignore Linuxserver date-style 2021 tags\u0026#34;, \u0026#34;matchDatasources\u0026#34;: [\u0026#34;docker\u0026#34;], \u0026#34;matchPackageNames\u0026#34;: [ \u0026#34;lscr.io/linuxserver/tautulli\u0026#34;, \u0026#34;lscr.io/linuxserver/overseerr\u0026#34; ], \u0026#34;allowedVersions\u0026#34;: \u0026#34;!/^2021\\\\./\u0026#34; } Avoiding .0 Releases # Home Assistant .0 releases are often buggy. I skip them:\n{ \u0026#34;description\u0026#34;: \u0026#34;Home Assistant: avoid .0 releases\u0026#34;, \u0026#34;matchDatasources\u0026#34;: [\u0026#34;docker\u0026#34;], \u0026#34;matchPackageNames\u0026#34;: [\u0026#34;ghcr.io/home-assistant/home-assistant\u0026#34;], \u0026#34;allowedVersions\u0026#34;: \u0026#34;!/\\\\.0$/\u0026#34; } I also disable major/minor updates entirely for Home Assistant - I want to control those manually:\n{ \u0026#34;description\u0026#34;: \u0026#34;Home Assistant: disable major/minor updates\u0026#34;, \u0026#34;matchPackageNames\u0026#34;: [\u0026#34;ghcr.io/home-assistant/home-assistant\u0026#34;], \u0026#34;matchUpdateTypes\u0026#34;: [\u0026#34;major\u0026#34;, \u0026#34;minor\u0026#34;], \u0026#34;enabled\u0026#34;: false } The Workflow Now # Renovate scans every 4 hours PRs are created for available updates I review (or automerge handles it) Merge to master ArgoCD syncs the new image to the cluster Everything flows through git. I can see exactly when an image was updated, who approved it, and what the previous version was.\nDependency Dashboard # Renovate creates an issue called \u0026ldquo;Dependency Dashboard\u0026rdquo; that shows:\nPending updates waiting for PRs Open PRs awaiting merge Updates blocked by version constraints Errors from failed lookups This single issue gives you visibility into your entire update backlog.\nMigration Tips # 1. Start with Automerge Disabled # Get comfortable with the PR flow before enabling automerge:\n{ \u0026#34;extends\u0026#34;: [\u0026#34;config:recommended\u0026#34;, \u0026#34;:automergeDisabled\u0026#34;] } 2. Use Semantic Commits # Enable semantic commits for cleaner git history:\n{ \u0026#34;semanticCommits\u0026#34;: \u0026#34;enabled\u0026#34; } PRs get titles like fix(deps): update lscr.io/linuxserver/sonarr to v4.0.2.\n3. Limit Concurrent PRs Initially # Don\u0026rsquo;t flood yourself with PRs on the first run:\n{ \u0026#34;prConcurrentLimit\u0026#34;: 5 } Increase once you\u0026rsquo;ve caught up on the backlog.\n4. Remove Keel Gradually # I removed Keel annotations from one app at a time, verified Renovate was creating PRs, then moved to the next. Don\u0026rsquo;t rip out Keel all at once.\nKeel vs Renovate: Summary # Aspect Keel Renovate Update method Direct cluster modification Pull requests GitOps compatible No (causes drift) Yes Review process None (notification only) Full PR review Version constraints Basic annotation policies Powerful regex rules Rollback Manual Git revert Audit trail Logs only Full git history Automerge N/A (always auto) Optional per-package Result # My image updates now have:\nVisibility - PRs show exactly what\u0026rsquo;s changing Control - Rules prevent unwanted major updates History - Git log shows every update Safety - Automerge only for trusted packages Consistency - GitOps stays clean, no drift The slight delay (PR review vs instant deploy) is worth the reliability. When something breaks, I know exactly which merge caused it and can revert cleanly.\nIf you\u0026rsquo;re running GitOps with ArgoCD or Flux, Renovate is the right choice for container image updates. Keel solved a problem for the pre-GitOps era, but PR-based updates are the modern approach.\n","date":"January 17, 2026","externalUrl":null,"permalink":"/posts/keel-to-renovate-kubernetes-image-updates/","section":"Posts","summary":"For years I used Keel to automatically update container images in my Kubernetes clusters. It worked, but as I moved to GitOps with ArgoCD, Keel’s push-based approach became a liability. I migrated to Renovate for PR-based image updates, and it’s been a significant improvement.\n","title":"From Keel to Renovate: Better Container Image Updates for GitOps","type":"posts"},{"content":"","date":"January 17, 2026","externalUrl":null,"permalink":"/tags/gitea/","section":"Tags","summary":"","title":"Gitea","type":"tags"},{"content":"I manage two Kubernetes environments - a home cluster (bowerhaus) and a cloud cluster (rustycloud) - using GitOps with Kustomize and ArgoCD. After running this setup for a while, I\u0026rsquo;ve learned what works, what doesn\u0026rsquo;t, and some non-obvious gotchas.\nThe Architecture # kustomize/ ├── base/ # Shared, environment-agnostic configs │ ├── media/ │ │ ├── lidarr/ │ │ ├── radarr/ │ │ └── sonarr/ │ ├── home-automation/ │ │ ├── home-assistant/ │ │ └── frigate/ │ └── data-analytics/ │ ├── prometheus/ │ └── grafana/ ├── environments/ │ ├── bowerhaus/ │ │ ├── applicationsets/ # ArgoCD ApplicationSet │ │ └── apps/ # Per-app overlays │ │ ├── frigate/ │ │ ├── home-assistant/ │ │ └── prometheus/ │ └── rustycloud/ │ ├── applicationsets/ │ └── apps/ │ ├── plex/ │ ├── sonarr/ │ └── grafana/ The key principle: base contains environment-agnostic resources, environments contain overlays that customize for each cluster.\nArgoCD ApplicationSets # Instead of creating individual ArgoCD Applications for each service, I use ApplicationSets with a git directory generator:\napiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: name: bowerhaus-appset namespace: argocd spec: generators: - git: repoURL: https://github.com/RustyBower/kustomize.git revision: master directories: - path: environments/bowerhaus/apps/* template: metadata: name: \u0026#39;{{path.basename}}\u0026#39; spec: project: default source: repoURL: https://github.com/RustyBower/kustomize.git targetRevision: master path: \u0026#39;environments/bowerhaus/apps/{{path.basename}}\u0026#39; kustomize: {} destination: server: https://kubernetes.default.svc namespace: \u0026#39;{{path.basename}}\u0026#39; syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true Every directory under environments/bowerhaus/apps/ automatically becomes an ArgoCD Application. Add a new folder, push to git, and ArgoCD deploys it.\nThe kustomize: {} block forces ArgoCD to use Kustomize even if there\u0026rsquo;s no explicit kustomization.yaml (though you should always have one).\nBase/Overlay Pattern # Base: Generic, Reusable # The base contains the core deployment without environment-specific details:\n# base/media/lidarr/lidarr-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: lidarr spec: replicas: 1 template: spec: containers: - name: lidarr image: lscr.io/linuxserver/lidarr:2.8.2 ports: - containerPort: 8686 volumeMounts: - name: config mountPath: /config volumes: - name: config persistentVolumeClaim: claimName: lidarr-config No NFS paths, no environment-specific storage, no ingress hostnames.\nOverlay: Environment-Specific # The overlay adds what\u0026rsquo;s unique to each environment:\n# environments/rustycloud/apps/lidarr/kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ../../../../base/media/lidarr/ - ingress.yaml namespace: lidarr patches: - path: deployment-patch.yaml # environments/rustycloud/apps/lidarr/deployment-patch.yaml apiVersion: apps/v1 kind: Deployment metadata: name: lidarr spec: template: spec: volumes: - name: media nfs: server: 10.0.0.105 path: \u0026#34;/cephfs/media/\u0026#34; - name: download nfs: server: 10.0.0.105 path: \u0026#34;/cephfs/data/download/complete/music\u0026#34; containers: - name: lidarr volumeMounts: - mountPath: /download/complete/music name: download - mountPath: /data/music name: media subPath: Music The patch adds NFS volumes specific to rustycloud\u0026rsquo;s storage infrastructure.\nKustomize Challenges I\u0026rsquo;ve Hit # 1. ClusterRoleBinding Namespace Issues # ClusterRoleBindings reference ServiceAccounts with a namespace. The base might have:\nsubjects: - kind: ServiceAccount name: prometheus namespace: monitoring But your overlay uses namespace: prometheus. Kustomize\u0026rsquo;s namespace transformer doesn\u0026rsquo;t update references inside ClusterRoleBindings.\nSolution: Use an inline patch in your overlay:\npatches: - target: kind: ClusterRoleBinding name: prometheus patch: |- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: prometheus subjects: - kind: ServiceAccount name: prometheus namespace: prometheus 2. Strategic Merge vs JSON Patches # Kustomize\u0026rsquo;s default strategic merge patches work well for adding fields but struggle with:\nRemoving fields Modifying array items by index Complex nested structures When strategic merge fails, use JSON patches:\npatches: - target: kind: Deployment name: myapp patch: |- - op: remove path: /spec/template/spec/containers/0/resources/limits 3. ConfigMap/Secret Name Hashing # Kustomize appends hashes to ConfigMap and Secret names by default. This breaks references if you\u0026rsquo;re not careful.\ngeneratorOptions: disableNameSuffixHash: true I disable hashing for configs that need stable names (like Renovate\u0026rsquo;s config).\n4. Image Tags in Different Locations # Renovate and Kustomize both want to manage image tags. I keep images in the base deployments and let Renovate update them there. The kubernetes manager in Renovate handles this:\n{ \u0026#34;kubernetes\u0026#34;: { \u0026#34;managerFilePatterns\u0026#34;: [ \u0026#34;/(^|/)base/.+/(?:[^/]*deployment)\\\\.ya?ml$/\u0026#34; ] } } This tells Renovate to only look at deployment files in the base directory.\n5. ArgoCD Sync Waves # When deploying interconnected services, order matters. Use sync-wave annotations:\nmetadata: annotations: argocd.argoproj.io/sync-wave: \u0026#34;-1\u0026#34; # Deploy before wave 0 Negative waves deploy first. I use this for:\n-2: Namespaces and CRDs -1: Secrets and ConfigMaps 0: Main deployments (default) 1: Ingresses and monitoring Directory Structure Tips # One App Per Directory # Each app gets its own directory with a kustomization.yaml. This maps cleanly to ArgoCD Applications and makes it obvious what\u0026rsquo;s deployed where.\nConsistent Naming # I use the app name as:\nDirectory name: apps/frigate/ Namespace: namespace: frigate ArgoCD Application name: {{path.basename}} → frigate This consistency makes debugging easier.\nBase Categories # Group bases by function:\nbase/media/ - Plex, *arr stack base/home-automation/ - Home Assistant, Frigate base/data-analytics/ - Prometheus, Grafana, InfluxDB base/dev-tools/ - Gitea, Drone, Renovate Self-Healing and Pruning # I enable both in the ApplicationSet:\nsyncPolicy: automated: prune: true # Delete resources not in git selfHeal: true # Revert manual changes This ensures git is the source of truth. Any kubectl edits get reverted. Deleted files remove resources.\nWarning: prune: true means deleting a directory from git deletes the entire deployment. Good for cleanup, dangerous if you accidentally remove something.\nThe Workflow # Make changes in a branch Test with kustomize build environments/bowerhaus/apps/myapp Push and merge to master ArgoCD syncs within 3 minutes (or trigger manually) Watch the sync in ArgoCD UI No kubectl apply, no remembering which cluster you\u0026rsquo;re on, no drift between environments.\nTakeaways # ApplicationSets eliminate per-app boilerplate Base/overlay pattern enforces separation of concerns Strategic merge patches work 80% of the time; JSON patches handle the rest Self-heal + prune keeps clusters matching git Test locally with kustomize build before pushing The initial setup takes effort, but the ongoing maintenance is minimal. Adding a new service is just creating a directory and pushing to git.\n","date":"January 17, 2026","externalUrl":null,"permalink":"/posts/kustomize-argocd-homelab-gitops/","section":"Posts","summary":"I manage two Kubernetes environments - a home cluster (bowerhaus) and a cloud cluster (rustycloud) - using GitOps with Kustomize and ArgoCD. After running this setup for a while, I’ve learned what works, what doesn’t, and some non-obvious gotchas.\n","title":"GitOps for Homelabs: Kustomize + ArgoCD Patterns and Pitfalls","type":"posts"},{"content":"","date":"January 17, 2026","externalUrl":null,"permalink":"/tags/harbor/","section":"Tags","summary":"","title":"Harbor","type":"tags"},{"content":"","date":"January 17, 2026","externalUrl":null,"permalink":"/tags/homepage/","section":"Tags","summary":"","title":"Homepage","type":"tags"},{"content":"","date":"January 17, 2026","externalUrl":null,"permalink":"/tags/keel/","section":"Tags","summary":"","title":"Keel","type":"tags"},{"content":"","date":"January 17, 2026","externalUrl":null,"permalink":"/tags/kustomize/","section":"Tags","summary":"","title":"Kustomize","type":"tags"},{"content":"I have six Honeywell T6 Pro Z-Wave thermostats throughout my house controlling a multi-zone HVAC system. Out of the box, these thermostats have their own scheduling and \u0026ldquo;smart\u0026rdquo; features that conflict with Home Assistant automations. Here\u0026rsquo;s how I configured them to let HA take full control.\nThe Setup # 6x Honeywell T6 Pro Z-Wave thermostats (TH6320ZW2003) Z-Wave JS UI running in Kubernetes Home Assistant with Z-Wave JS integration Gas furnace + AC (not heat pump) with multi-zone dampers The zones:\nGreat Room \u0026amp; Music Room - shared outdoor unit, multi-zone Master Bedroom, Office, Kids Room - shared outdoor unit, multi-zone Basement - standalone (heat only, no AC) The Problem # The T6 Pro has built-in features that fight against Home Assistant control:\nBuilt-in scheduling - The thermostat tries to follow its own 5-2 or 5-1-1 schedule Adaptive Intelligent Recovery - Pre-heats/cools based on learned patterns Temperature resolution - Reports in 1°F increments by default, limiting precision When HA sends a setpoint change, these features can override it or cause unexpected behavior.\nKey Parameters to Change # After digging through the Z-Wave JS node configuration, here are the critical parameters:\nParameter 1: Schedule Type # Set to 0 (No schedule)\nParameter: 1 Value: 0 (None) Options: 0=None, 1=Occupancy, 2=Every Day, 3=5-2, 4=5-1-1 This disables the thermostat\u0026rsquo;s built-in scheduling entirely. HA will handle all scheduling via automations.\nParameter 27: Adaptive Intelligent Recovery # Set to 0 (Disabled)\nParameter: 27 Value: 0 (Disabled) Options: 0=Disabled, 1=Enabled AIR learns how long it takes to reach setpoint and starts heating/cooling early. Sounds nice, but it means the thermostat ignores your setpoint timing. Disable it and let HA handle pre-conditioning if needed.\nParameter 44: Temperature Display Resolution # Set to 0 (1°F)\nParameter: 44 Value: 0 (1°F) Options: 0=1°F, 1=0.5°F Actually, keep this at 0 for Fahrenheit. If you use Celsius, setting to 1 gives 0.5° resolution which can be useful for tighter control.\nParameter 6: Cool Stages (Basement Only) # Set to 0 for heat-only zones\nParameter: 6 Value: 0 (No cooling) Options: 0=None, 1=1 Stage, 2=2 Stages My basement has no AC, so I set cool stages to 0. This prevents the thermostat from ever calling for cooling.\nHow to Change Parameters # Via Z-Wave JS UI # Navigate to your thermostat node Go to Configuration Parameters Find the parameter number Set the new value and click Save The changes take effect immediately - you\u0026rsquo;ll see the thermostat update its display.\nVia Home Assistant Service Call # You can also use the zwave_js.set_config_parameter service:\nservice: zwave_js.set_config_parameter target: device_id: abc123... data: parameter: 1 value: 0 However, I found the Z-Wave JS UI more reliable for bulk changes across multiple devices.\nMulti-Zone Considerations # With multi-zone HVAC, the thermostats operate dampers rather than directly controlling the furnace/AC. A few things to keep in mind:\nZone coordination - If one zone calls for heat while another calls for cooling, you have a conflict. HA automations should prevent this. Minimum airflow - Most systems need at least one zone open. Configure a \u0026ldquo;dump zone\u0026rdquo; or ensure automations don\u0026rsquo;t close all dampers. Equipment protection - The furnace/AC has its own safeties, but avoid rapid cycling by adding delays between mode changes in HA. My HA Automation Strategy # With the thermostats now in \u0026ldquo;dumb\u0026rdquo; mode, HA handles:\nTime-based scheduling - Different setpoints for sleep, away, home Occupancy detection - Adjust zones based on presence sensors Weather integration - Pre-condition before temperature swings Energy optimization - Coordinate zones to minimize equipment runtime The thermostats become simple actuators - they receive a setpoint from HA and maintain it. No more fighting between two \u0026ldquo;smart\u0026rdquo; systems trying to be helpful.\nResult # After making these changes across all six thermostats:\nSetpoint changes from HA take effect immediately No more mysterious temperature swings from AIR Schedules are 100% controlled by HA automations The system behaves predictably The T6 Pro is a solid Z-Wave thermostat once you disable its autonomous features. It reports temperature accurately, responds quickly to commands, and the hardware is reliable. Just don\u0026rsquo;t let it think for itself.\n","date":"January 17, 2026","externalUrl":null,"permalink":"/posts/zwave-thermostats-home-assistant/","section":"Posts","summary":"I have six Honeywell T6 Pro Z-Wave thermostats throughout my house controlling a multi-zone HVAC system. Out of the box, these thermostats have their own scheduling and “smart” features that conflict with Home Assistant automations. Here’s how I configured them to let HA take full control.\n","title":"Optimizing Z-Wave Thermostats for Home Assistant Control","type":"posts"},{"content":"","date":"January 17, 2026","externalUrl":null,"permalink":"/tags/pihole/","section":"Tags","summary":"","title":"Pihole","type":"tags"},{"content":"","date":"January 17, 2026","externalUrl":null,"permalink":"/tags/renovate/","section":"Tags","summary":"","title":"Renovate","type":"tags"},{"content":"I run a fully self-hosted CI/CD pipeline using Drone for builds, Gitea for git hosting, and Harbor for container registry. No GitHub Actions, no Docker Hub, no external dependencies. Here\u0026rsquo;s how it all fits together.\nThe Stack # Gitea - Lightweight git server with OAuth2 support Drone - Container-native CI/CD platform Harbor - Enterprise container registry with vulnerability scanning BuildKit - Modern Docker builder for efficient image builds Why Self-Hosted? # Privacy - Code never leaves my network No rate limits - Build as often as needed Offline capability - Works during internet outages (for local images) Learning - Understanding the full DevOps stack Cost - No per-minute billing for CI runners Architecture # ┌─────────────────────────────────────────────────────────────┐ │ Kubernetes │ │ │ │ ┌──────────┐ ┌─────────────────────────────┐ │ │ │ Gitea │────▶│ Drone │ │ │ │ (git) │ │ ┌───────┐ ┌──────────┐ │ │ │ └──────────┘ │ │Server │ │ Runner │ │ │ │ │ └───────┘ └────┬─────┘ │ │ │ └────────────────────│────────┘ │ │ │ │ │ ┌──────────┐ ┌──────▼─────┐ │ │ │ Harbor │◀──────────────────│ BuildKit │ │ │ │(registry)│ │ (builds) │ │ │ └──────────┘ └────────────┘ │ │ │ └─────────────────────────────────────────────────────────────┘ Push code to Gitea Gitea webhook triggers Drone Drone spawns a build job via the Kubernetes runner BuildKit builds the container image Image is pushed to Harbor ArgoCD deploys the new image (separate workflow) Gitea Setup # Gitea is straightforward - a single deployment with PostgreSQL backend. The key configuration is creating an OAuth2 application for Drone:\nGo to Gitea Settings → Applications Create new OAuth2 application Set redirect URI to https://drone.rustybower.com/login Note the Client ID and Client Secret Drone Components # Drone Server # The main Drone server handles the web UI, webhook processing, and build coordination:\napiVersion: apps/v1 kind: Deployment metadata: name: drone spec: template: spec: containers: - name: drone image: drone/drone:2.26.0 envFrom: - configMapRef: name: drone-env volumeMounts: - mountPath: /data name: drone-data Configuration via ConfigMap:\napiVersion: v1 kind: ConfigMap metadata: name: drone-env data: DRONE_GITEA_SERVER: \u0026#34;https://gitea.rustybower.com\u0026#34; DRONE_GITEA_CLIENT_ID: \u0026#34;your-oauth-client-id\u0026#34; DRONE_GITEA_CLIENT_SECRET: \u0026#34;your-oauth-client-secret\u0026#34; DRONE_RPC_SECRET: \u0026#34;shared-secret-for-runners\u0026#34; DRONE_SERVER_HOST: \u0026#34;drone.rustybower.com\u0026#34; DRONE_SERVER_PROTO: \u0026#34;https\u0026#34; DRONE_USER_CREATE: \u0026#34;username:rusty,admin:true\u0026#34; Drone Kubernetes Runner # The runner executes pipeline steps as Kubernetes pods:\napiVersion: apps/v1 kind: Deployment metadata: name: runner spec: template: spec: containers: - name: runner image: drone/drone-runner-kube:latest envFrom: - configMapRef: name: drone-env serviceAccountName: drone-runner The runner needs RBAC permissions to create pods:\napiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: drone-runner rules: - apiGroups: [\u0026#34;\u0026#34;] resources: [\u0026#34;pods\u0026#34;, \u0026#34;pods/log\u0026#34;, \u0026#34;secrets\u0026#34;] verbs: [\u0026#34;get\u0026#34;, \u0026#34;list\u0026#34;, \u0026#34;watch\u0026#34;, \u0026#34;create\u0026#34;, \u0026#34;delete\u0026#34;] BuildKit for Efficient Builds # Instead of Docker-in-Docker (security nightmare), I use BuildKit as a separate service:\napiVersion: apps/v1 kind: Deployment metadata: name: buildkitd spec: template: spec: containers: - name: buildkitd image: moby/buildkit:v0.26.3 args: - --addr - tcp://0.0.0.0:1234 - --config - /etc/buildkit/buildkit.toml securityContext: privileged: true # Required for building volumeMounts: - name: config mountPath: /etc/buildkit - name: harbor-docker-config mountPath: /root/.docker BuildKit needs registry credentials to push images:\napiVersion: v1 kind: Secret metadata: name: harbor-docker-config type: kubernetes.io/dockerconfigjson data: .dockerconfigjson: \u0026lt;base64-encoded-docker-config\u0026gt; Harbor Configuration # Harbor provides:\nContainer image storage Vulnerability scanning with Trivy Robot accounts for CI/CD access Image replication between registries Create a robot account for Drone:\nHarbor → Administration → Robot Accounts Create with push/pull permissions Use credentials in Drone secrets Pipeline Example # Here\u0026rsquo;s a real pipeline for my Hugo blog:\n# .drone.yml kind: pipeline type: kubernetes name: default clone: depth: 1 submodules: true steps: - name: init submodules image: alpine/git commands: - git submodule update --init --recursive - name: docker build and push image: plugins/docker settings: registry: harbor.rustybower.com repo: harbor.rustybower.com/hugo/hugo-site tags: - latest dockerfile: Dockerfile build_args: - HUGO_BASEURL=https://www.rustybower.com/ username: from_secret: harbor_username password: from_secret: harbor_password The Dockerfile uses multi-stage builds:\n# Build Stage FROM hugomods/hugo:exts AS builder ARG HUGO_BASEURL= ENV HUGO_BASEURL=${HUGO_BASEURL} COPY . /src RUN hugo --minify --enableGitInfo # Final Stage FROM hugomods/hugo:nginx COPY --from=builder /src/public /site Secrets Management # Drone secrets are stored per-repository:\nGo to repository settings in Drone UI Add secrets for registry credentials Reference in pipeline with from_secret Never commit credentials to .drone.yml - always use secrets.\nTroubleshooting Tips # Build Not Starting # Check the runner logs:\nkubectl logs -n drone deployment/runner Common issues:\nRPC secret mismatch between server and runner Runner can\u0026rsquo;t reach Drone server RBAC permissions missing Registry Push Fails # Verify BuildKit has credentials:\nkubectl exec -n drone deployment/buildkitd -- cat /root/.docker/config.json Check Harbor robot account permissions.\nPipeline Stuck # The Kubernetes runner creates pods for each step. Check for stuck pods:\nkubectl get pods -n drone Look for ImagePullBackOff or resource constraints.\nIntegration with GitOps # After the image is pushed to Harbor, I use Renovate to create PRs updating the image tag in my Kustomize repo. ArgoCD then deploys the new image.\nThe full flow:\nPush code to Gitea Drone builds and pushes to Harbor Renovate detects new image, creates PR Merge PR to update manifest ArgoCD syncs new image to cluster This keeps the GitOps principle intact - images are built automatically, but deployment still flows through git.\nPerformance Optimizations # Layer Caching # BuildKit caches layers automatically. For faster builds:\nOrder Dockerfile commands from least to most changing Use multi-stage builds to minimize final image size Mount build caches for package managers Resource Limits # Set appropriate limits in Drone steps:\nsteps: - name: build image: node:20 resources: limits: memory: 2Gi cpu: 2 Parallel Steps # Independent steps can run in parallel:\nsteps: - name: test image: node:20 commands: - npm test - name: lint image: node:20 commands: - npm run lint Security Considerations # Network policies - Restrict what build pods can access Robot accounts - Use minimal permissions for registry access No privileged unless needed - Only BuildKit needs privileged mode Scan images - Harbor\u0026rsquo;s Trivy integration catches vulnerabilities Audit logs - Gitea, Drone, and Harbor all log activity Result # My self-hosted CI/CD:\nBuilds in ~2 minutes for most projects Zero external dependencies Full control over the pipeline Complete audit trail across all components The initial setup is more complex than GitHub Actions, but the understanding you gain of CI/CD internals is valuable. Plus, there\u0026rsquo;s something satisfying about git push triggering a build on your own infrastructure.\n","date":"January 17, 2026","externalUrl":null,"permalink":"/posts/drone-gitea-harbor-self-hosted-cicd/","section":"Posts","summary":"I run a fully self-hosted CI/CD pipeline using Drone for builds, Gitea for git hosting, and Harbor for container registry. No GitHub Actions, no Docker Hub, no external dependencies. Here’s how it all fits together.\n","title":"Self-Hosted CI/CD with Drone, Gitea, and Harbor","type":"posts"},{"content":"","date":"January 17, 2026","externalUrl":null,"permalink":"/tags/z-wave/","section":"Tags","summary":"","title":"Z-Wave","type":"tags"}]