Add periodic PIA VPN connectivity check

Oneshot service + timer (every 5 min) inside the VPN container that
verifies WireGuard handshake freshness and internet reachability.
Fails on VPN or internet outage, triggering ntfy alert via OnFailure.
Capped at 3 failures per day via StartLimitBurst.

README.md

@@ -1,16 +1,26 @@
 # NixOS Configuration
 
-A NixOS flake managing multiple machines with role-based configuration, agenix secrets, and sandboxed dev workspaces.
+A NixOS flake managing multiple machines with role-based configuration, agenix secrets, sandboxed dev workspaces, and self-hosted services.
 
 ## Layout
 
 - `/common` - shared configuration imported by all machines
   - `/boot` - bootloaders, CPU microcode, remote LUKS unlock over Tor
-  - `/network` - Tailscale, VPN tunneling via PIA
+  - `/network` - Tailscale, PIA VPN with leak-proof containers, sandbox networking
   - `/pc` - desktop/graphical config (enabled by the `personal` role)
-  - `/server` - service definitions and extensions
+  - `/server` - self-hosted service definitions (Gitea, Matrix, Nextcloud, media stack, etc.)
   - `/sandboxed-workspace` - isolated dev environments (VM, container, or Incus)
+  - `/ntfy` - push notification integration (service failures, SSH logins, ZFS alerts)
+  - `binary-cache.nix` - nix binary cache configuration (nixos.org, cachix, self-hosted atticd)
+  - `nix-builder.nix` - distributed build delegation across machines
+  - `backups.nix` - snapshot-aware restic backups to Backblaze B2
 - `/machines` - per-machine config (`default.nix`, `hardware-configuration.nix`, `properties.nix`)
+  - `fry` - personal desktop
+  - `howl` - personal laptop
+  - `ponyo` - web/mail server (Gitea, Nextcloud, LibreChat, mail)
+  - `storage/s0` - storage/media server (Jellyfin, Home Assistant, monitoring, productivity apps)
+  - `zoidberg` - media center
+  - `ephemeral` - minimal config for building install ISOs and kexec images
 - `/secrets` - agenix-encrypted secrets, decryptable by machines based on their roles
 - `/home` - Home Manager user config
 - `/lib` - custom library functions extending nixpkgs lib
@@ -25,8 +35,14 @@ A NixOS flake managing multiple machines with role-based configuration, agenix s
 
 **Remote LUKS unlock over Tor** — Machines with encrypted root disks can be unlocked remotely via SSH. An embedded Tor hidden service starts in the initrd so the machine is reachable even without a known IP, using a separate SSH host key for the boot environment.
 
-**VPN containers** — A `vpn-container` module spins up an ephemeral NixOS container with a PIA WireGuard tunnel. The host creates the WireGuard interface and authenticates with PIA, then hands it off to the container's network namespace. This ensures that the container can **never** have direct internet access. Leakage is impossible.
+**VPN containers** — A `pia-vpn` module provides leak-proof VPN networking for containers. The host creates a WireGuard interface and runs tinyproxy on a bridge network for PIA API bootstrap. A dedicated VPN container authenticates with PIA via the proxy, configures WireGuard, and masquerades bridge traffic through the tunnel. Service containers default-route exclusively through the VPN container — leakage is impossible by network topology. Supports port forwarding with automatic port assignment.
 
-**Sandboxed workspaces** — Isolated dev environments backed by microVMs (cloud-hypervisor), systemd-nspawn containers, or Incus. Each workspace gets a static IP on a NAT'd bridge, auto-generated SSH host keys, shell aliases for management, and comes pre-configured with Claude Code. The sandbox network blocks access to the local LAN while allowing internet.
+**Sandboxed workspaces** — Isolated dev environments backed by microVMs (cloud-hypervisor), systemd-nspawn containers, or Incus. Each workspace gets a static IP on a NAT'd bridge (`192.168.83.0/24`), auto-generated SSH host keys, shell aliases for management, and comes pre-configured with Claude Code. The sandbox network blocks access to the local LAN while allowing internet.
 
 **Snapshot-aware backups** — Restic backups to Backblaze B2 automatically create ZFS snapshots or btrfs read-only snapshots before backing up, using mount namespaces to bind-mount frozen data over the original paths so restic records correct paths. Each backup group gets a `restic_<group>` CLI wrapper. Supports `.nobackup` marker files.
+
+**Self-hosted services** — Comprehensive service stack across ponyo and s0: Gitea (git hosting + CI), Nextcloud (files/calendar), Matrix (chat), mail server, Jellyfin/Sonarr/Radarr/Lidarr (media), Home Assistant/Zigbee2MQTT/Frigate (home automation), LibreChat (AI), Gatus (monitoring), and productivity tools (Vikunja, Actual Budget, Outline, Linkwarden, Memos).
+
+**Push notifications** — ntfy integration alerts on systemd service failures, SSH logins, and ZFS pool issues. Gatus monitors all web-facing services and sends alerts via ntfy.
+
+**Remote deployment** — deploy-rs handles remote machine deployments with boot-only or immediate activation modes. A Makefile wraps common operations (`make deploy <host>`, `make deploy-activate <host>`).

common/network/pia-vpn/vpn-container.nix

@@ -226,6 +226,57 @@ in
               RandomizedDelaySec = "1m";
             };
           };
+
+          # Periodic VPN connectivity check — fails if VPN or internet is down,
+          # triggering ntfy alert via the OnFailure drop-in
+          systemd.services.pia-vpn-check = {
+            description = "Check PIA VPN connectivity";
+            after = [ "pia-vpn-setup.service" ];
+            requires = [ "pia-vpn-setup.service" ];
+
+            path = with pkgs; [ wireguard-tools iputils coreutils gawk ];
+
+            unitConfig = {
+              StartLimitBurst = 3;
+              StartLimitIntervalSec = "1d";
+            };
+
+            serviceConfig.Type = "oneshot";
+
+            script = ''
+              set -euo pipefail
+
+              # Check that WireGuard has a peer with a recent handshake (within 3 minutes)
+              handshake=$(wg show ${cfg.interfaceName} latest-handshakes | awk '{print $2}')
+              if [ -z "$handshake" ] || [ "$handshake" -eq 0 ]; then
+                echo "No WireGuard handshake recorded" >&2
+                exit 1
+              fi
+              now=$(date +%s)
+              age=$((now - handshake))
+              if [ "$age" -gt 180 ]; then
+                echo "WireGuard handshake is stale (''${age}s ago)" >&2
+                exit 1
+              fi
+
+              # Verify internet connectivity through VPN tunnel
+              if ! ping -c1 -W10 1.1.1.1 >/dev/null 2>&1; then
+                echo "Cannot reach internet through VPN" >&2
+                exit 1
+              fi
+
+              echo "PIA VPN connectivity OK (handshake ''${age}s ago)"
+            '';
+          };
+
+          systemd.timers.pia-vpn-check = {
+            description = "Periodic PIA VPN connectivity check";
+            wantedBy = [ "timers.target" ];
+            timerConfig = {
+              OnCalendar = "*:0/5";
+              RandomizedDelaySec = "30s";
+            };
+          };
         };
     };
   };

common/ntfy/default.nix

@@ -5,6 +5,7 @@
     ./service-failure.nix
     ./ssh-login.nix
     ./zfs.nix
+    ./dimm-temp.nix
   ];
 
   options.ntfy-alerts = {

common/ntfy/dimm-temp.nix

@@ -0,0 +1,69 @@
+{ config, lib, pkgs, ... }:
+
+let
+  cfg = config.ntfy-alerts;
+  hasNtfy = config.thisMachine.hasRole."ntfy";
+
+  checkScript = pkgs.writeShellScript "dimm-temp-check" ''
+    PATH="${lib.makeBinPath [ pkgs.lm_sensors pkgs.gawk pkgs.coreutils pkgs.curl ]}"
+
+    threshold=55
+    hot=""
+
+    while IFS= read -r line; do
+      case "$line" in
+        spd5118-*)
+          chip="$line"
+          ;;
+        *temp1_input:*)
+          temp="''${line##*: }"
+          whole="''${temp%%.*}"
+          if [ "$whole" -ge "$threshold" ]; then
+            hot="$hot"$'\n'"  $chip: ''${temp}°C"
+          fi
+          ;;
+      esac
+    done < <(sensors -u 'spd5118-*' 2>/dev/null)
+
+    if [ -n "$hot" ]; then
+      message="DIMM temperature above ''${threshold}°C on ${config.networking.hostName}:$hot"
+
+      curl \
+        --fail --silent --show-error \
+        --max-time 30 --retry 3 \
+        -H "Authorization: Bearer $NTFY_TOKEN" \
+        -H "Title: High DIMM temperature on ${config.networking.hostName}" \
+        -H "Priority: high" \
+        -H "Tags: thermometer" \
+        -d "$message" \
+        "${cfg.serverUrl}/service-failures"
+
+      echo "$message" >&2
+    fi
+  '';
+in
+{
+  options.ntfy-alerts.dimmTempCheck.enable = lib.mkEnableOption "DDR5 DIMM temperature monitoring via spd5118";
+
+  config = lib.mkIf (cfg.dimmTempCheck.enable && hasNtfy) {
+    systemd.services.dimm-temp-check = {
+      description = "Check DDR5 DIMM temperatures and alert on overheating";
+      wants = [ "network-online.target" ];
+      after = [ "network-online.target" ];
+      serviceConfig = {
+        Type = "oneshot";
+        EnvironmentFile = "/run/agenix/ntfy-token";
+        ExecStart = checkScript;
+      };
+    };
+
+    systemd.timers.dimm-temp-check = {
+      description = "Periodic DDR5 DIMM temperature check";
+      wantedBy = [ "timers.target" ];
+      timerConfig = {
+        OnCalendar = "*:0/5";
+        Persistent = true;
+      };
+    };
+  };
+}

common/server/gatus.nix

@@ -270,6 +270,16 @@ in
             ];
             alerts = [{ type = "ntfy"; }];
           }
+          {
+            name = "Music Assistant";
+            group = "s0";
+            url = "http://s0.koi-bebop.ts.net:8095";
+            interval = "5m";
+            conditions = [
+              "[STATUS] == 200"
+            ];
+            alerts = [{ type = "ntfy"; }];
+          }
           {
             name = "Vikunja";
             group = "s0";
@@ -320,16 +330,7 @@ in
             ];
             alerts = [{ type = "ntfy"; }];
           }
-          {
+
-            name = "LanguageTool";
-            group = "s0";
-            url = "https://languagetool.s0.neet.dev";
-            interval = "5m";
-            conditions = [
-              "[STATUS] == 200"
-            ];
-            alerts = [{ type = "ntfy"; }];
-          }
           {
             name = "Unifi";
             group = "s0";

machines/storage/s0/dashy.nix

@@ -401,6 +401,15 @@
             statusCheck = false;
             id = "5_4201_sandman";
           };
+          music-assistant = {
+            title = "Music Assistant";
+            description = "s0:8095";
+            icon = "hl-music-assistant";
+            url = "http://s0.koi-bebop.ts.net:8095";
+            target = "sametab";
+            statusCheck = false;
+            id = "6_4201_music-assistant";
+          };
         };
         haList = [
           haItems.home-assistant
@@ -409,6 +418,7 @@
           haItems.frigate
           haItems.valetudo
           haItems.sandman
+          haItems.music-assistant
         ];
       in
       {
@@ -474,15 +484,6 @@
             statusCheck = false;
             id = "4_5301_outline";
           };
-          languagetool = {
-            title = "LanguageTool";
-            description = "languagetool.s0.neet.dev";
-            icon = "hl-languagetool";
-            url = "https://languagetool.s0.neet.dev";
-            target = "sametab";
-            statusCheck = false;
-            id = "5_5301_languagetool";
-          };
         };
         prodList = [
           prodItems.vikunja
@@ -490,7 +491,6 @@
           prodItems.linkwarden
           prodItems.memos
           prodItems.outline
-          prodItems.languagetool
         ];
       in
       {

machines/storage/s0/default.nix

@@ -10,6 +10,7 @@
   networking.hostName = "s0";
 
   ntfy-alerts.ignoredUnits = [ "logrotate" ];
+  ntfy-alerts.dimmTempCheck.enable = true;
 
   # system.autoUpgrade.enable = true;
 
@@ -253,7 +254,6 @@
         (mkVirtualHost "linkwarden.s0.neet.dev" "http://localhost:${toString config.services.linkwarden.port}")
         (mkVirtualHost "memos.s0.neet.dev" "http://localhost:${toString config.services.memos.settings.MEMOS_PORT}")
         (mkVirtualHost "outline.s0.neet.dev" "http://localhost:${toString config.services.outline.port}")
-        (mkVirtualHost "languagetool.s0.neet.dev" "http://localhost:${toString config.services.languagetool.port}")
       ];
 
     tailscaleAuth = {
@@ -277,7 +277,6 @@
         "linkwarden.s0.neet.dev"
         # "memos.s0.neet.dev" # messes up memos /auth route
         # "outline.s0.neet.dev" # messes up outline /auth route
-        "languagetool.s0.neet.dev"
       ];
       expectedTailnet = "koi-bebop.ts.net";
     };
@@ -367,10 +366,5 @@
     owner = config.services.outline.user;
   };
 
-  services.languagetool = {
-    enable = true;
-    port = 60613;
-  };
-
   boot.binfmt.emulatedSystems = [ "aarch64-linux" "armv7l-linux" ];
 }