fix: add SWAP to AZ-PRM-1

First deploy failed: atrocore-db-init (User=postgres) could not read
/run/agenix/atrocore-env because agenix defaults to root:root:0400.
Follow the pg-cert/pg-key precedent: owner/group postgres, mode 0400.
Podman still reads the env-file and registry token as root (root
bypasses DAC), container pull/run unaffected. Agenix applies
ownership at activation time - no rekey needed. Validated on AZ-PRM-1.

.envrc

@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+# Activate the devshell from the Nix flake
+# This loads all tools and environment variables defined in flake.nix
+
+use flake

.gitattributes

@@ -0,0 +1,3 @@
+
+# Use bd merge for beads JSONL files
+.beads/issues.jsonl merge=beads

.gitignore

@@ -0,0 +1,48 @@
+# Sisyphus work session data
+.sisyphus/
+
+# Editor files
+*~
+.*.swp
+.*.swo
+.*.swx
+
+# Build artifacts
+result
+result-*
+.direnv/
+
+# IDE
+.vscode/
+.idea/
+*.iml
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Opencode rules
+.opencode-rules
+opencode.json
+
+# Local agent/coding-rule artifacts
+.pi/
+.pi-lens/
+coding-rules.json
+coding-rules/
+docs/
+
+# Git worktrees
+.worktrees/
+
+.todos/
+.sidecar/
+.claude/
+.codex/
+.agents/
+
+# Beads / Dolt files (added by bd init)
+.dolt/
+*.db
+.beads-credential-key
+.beads/proxieddb/

AGENTS.md

@@ -0,0 +1,125 @@
+# Agent Instructions
+
+## MANDATORY: Use td for Task Management
+
+You must run td usage --new-session at conversation start (or after /clear) to see current work.
+Use td usage -q for subsequent reads.
+
+This project uses **bd** (beads) for issue tracking. Run `bd onboard` to get started.
+
+## Quick Reference
+
+```bash
+bd ready              # Find available work
+bd show <id>          # View issue details
+bd update <id> --status in_progress  # Claim work
+bd close <id>         # Complete work
+bd sync               # Sync with git
+```
+
+## Landing the Plane (Session Completion)
+
+**When ending a work session**, you MUST complete ALL steps below. Work is NOT complete until `git push` succeeds.
+
+**MANDATORY WORKFLOW:**
+
+1. **File issues for remaining work** - Create issues for anything that needs follow-up
+2. **Run quality gates** (if code changed) - Tests, linters, builds
+3. **Update issue status** - Close finished work, update in-progress items
+4. **PUSH TO REMOTE** - This is MANDATORY:
+   ```bash
+   git pull --rebase
+   bd sync
+   git push
+   git status  # MUST show "up to date with origin"
+   ```
+5. **Clean up** - Clear stashes, prune remote branches
+6. **Verify** - All changes committed AND pushed
+7. **Hand off** - Provide context for next session
+
+**CRITICAL RULES:**
+- Work is NOT complete until `git push` succeeds
+- NEVER stop before pushing - that leaves work stranded locally
+- NEVER say "ready to push when you are" - YOU must push
+- If push fails, resolve and retry until it succeeds
+
+
+<!-- BEGIN BEADS INTEGRATION v:1 profile:minimal hash:970c3bf2 -->
+## Beads Issue Tracker
+
+This project uses **bd (beads)** for issue tracking. Run `bd prime` to see full workflow context and commands.
+
+### Quick Reference
+
+```bash
+bd ready              # Find available work
+bd show <id>          # View issue details
+bd update <id> --claim  # Claim work
+bd close <id>         # Complete work
+```
+
+### Rules
+
+- Use `bd` for ALL task tracking — do NOT use TodoWrite, TaskCreate, or markdown TODO lists
+- Run `bd prime` for detailed command reference and session close protocol
+- Use `bd remember` for persistent knowledge — do NOT use MEMORY.md files
+
+**Architecture in one line:** issues live in a local Dolt DB; sync uses `refs/dolt/data` on your git remote; `.beads/issues.jsonl` is a passive export. See https://github.com/gastownhall/beads/blob/main/docs/SYNC_CONCEPTS.md for details and anti-patterns.
+
+## Agent Context Profiles
+
+The managed Beads block is task-tracking guidance, not permission to override repository, user, or orchestrator instructions.
+
+- **Conservative (default)**: Use `bd` for task tracking. Do not run git commits, git pushes, or Dolt remote sync unless explicitly asked. At handoff, report changed files, validation, and suggested next commands.
+- **Minimal**: Keep tool instruction files as pointers to `bd prime`; use the same conservative git policy unless active instructions say otherwise.
+- **Team-maintainer**: Only when the repository explicitly opts in, agents may close beads, run quality gates, commit, and push as part of session close. A current "do not commit" or "do not push" instruction still wins.
+
+## Session Completion
+
+This protocol applies when ending a Beads implementation workflow. It is subordinate to explicit user, repository, and orchestrator instructions.
+
+1. **File issues for remaining work** - Create beads for anything that needs follow-up
+2. **Run quality gates** (if code changed) - Tests, linters, builds
+3. **Update issue status** - Close finished work, update in-progress items
+4. **Handle git/sync by active profile**:
+   ```bash
+   # Conservative/minimal/default: report status and proposed commands; wait for approval.
+   git status
+
+   # Team-maintainer opt-in only, unless current instructions forbid it:
+   git pull --rebase
+   bd dolt push
+   git push
+   git status
+   ```
+5. **Hand off** - Summarize changes, validation, issue status, and any blocked sync/commit/push step
+
+**Critical rules:**
+- Explicit user or orchestrator instructions override this Beads block.
+- Do not commit or push without clear authority from the active profile or the current user request.
+- If a required sync or push is blocked, stop and report the exact command and error.
+<!-- END BEADS INTEGRATION -->
+
+<!-- BEGIN BEADS CODEX SETUP: generated by bd setup codex -->
+## Beads Issue Tracker
+
+Use Beads (`bd`) for durable task tracking in repositories that include it. Use the `beads` skill at `.agents/skills/beads/SKILL.md` (project install) or `~/.agents/skills/beads/SKILL.md` (global install) for Beads workflow guidance, then use the `bd` CLI for issue operations.
+
+### Quick Reference
+
+```bash
+bd ready                # Find available work
+bd show <id>            # View issue details
+bd update <id> --claim  # Claim work
+bd close <id>           # Complete work
+bd prime                # Refresh Beads context
+```
+
+### Rules
+
+- Use `bd` for all task tracking; do not create markdown TODO lists.
+- Run `bd prime` when Beads context is missing or stale. Codex 0.129.0+ can load Beads context automatically through native hooks; use `/hooks` to inspect or toggle them.
+- Keep persistent project memory in Beads via `bd remember`; do not create ad hoc memory files.
+
+**Architecture in one line:** issues live in a local Dolt DB; sync uses `refs/dolt/data` on your git remote; `.beads/issues.jsonl` is a passive export. See https://github.com/gastownhall/beads/blob/main/docs/SYNC_CONCEPTS.md for details and anti-patterns.
+<!-- END BEADS CODEX SETUP -->

DOLT_REMOTE.md

@@ -1,7 +0,0 @@
-This repository is being used as a Dolt remote.
-
-ref=refs/dolt/data
-
-head=fff4640143b0c0612b490c2a55c9b322173bf64e
-
-timestamp=2026-08-24T04:46:20Z

flake.nix

@@ -0,0 +1,203 @@
+{
+  description = ''
+    For questions just DM me on X: https://twitter.com/@m3tam3re
+    There is also some NIXOS content on my YT channel: https://www.youtube.com/@m3tam3re
+
+    One of the best ways to learn NIXOS is to read other peoples configurations. I have personally learned a lot from Gabriel Fontes configs:
+    https://github.com/Misterio77/nix-starter-configs
+    https://github.com/Misterio77/nix-config
+
+    Please also check out the starter configs mentioned above.
+  '';
+
+  inputs = {
+    home-manager = {
+      url = "github:nix-community/home-manager/release-26.05";
+      inputs.nixpkgs.follows = "nixpkgs";
+    };
+
+    nixpkgs.url = "github:nixos/nixpkgs/nixos-26.05";
+    nixpkgs-unstable.url = "github:nixos/nixpkgs/nixos-unstable";
+
+    m3ta-nixpkgs.url = "git+https://code.m3ta.dev/m3tam3re/nixpkgs";
+
+    m3ta-home = {
+      url = "git+ssh://gitea@code.m3ta.dev/m3tam3re/m3ta-home";
+      inputs.nixpkgs.follows = "nixpkgs";
+    };
+
+    llm-agents.url = "github:numtide/llm-agents.nix";
+
+    nur = {
+      url = "github:nix-community/NUR";
+      inputs.nixpkgs.follows = "nixpkgs";
+    };
+
+    disko = {
+      url = "github:nix-community/disko";
+      inputs.nixpkgs.follows = "nixpkgs";
+    };
+
+    agenix.url = "github:ryantm/agenix";
+
+    nixos-anywhere = {
+      url = "github:nix-community/nixos-anywhere";
+      inputs.nixpkgs.follows = "nixpkgs";
+    };
+
+    nix-colors.url = "github:misterio77/nix-colors";
+
+    agents = {
+      url = "git+https://code.m3ta.dev/m3tam3re/AGENTS";
+      flake = false;
+    };
+
+    zugferd-service = {
+      url = "git+https://git.az-gruppe.com/AZ-Intec-GmbH/zugferd-service";
+      # url = "path:/home/sascha.koenig/p/CODE/python/zugferd-service";
+    };
+
+    azion-scheduler = {
+      url = "git+https://git.az-gruppe.com/AZ-Intec-GmbH/AZion";
+      inputs.nixpkgs.follows = "nixpkgs";
+    };
+
+    azess = {
+      url = "git+https://git.az-gruppe.com/AZ-Intec-GmbH/AZess";
+      inputs.nixpkgs.follows = "nixpkgs";
+    };
+    phishboard = {
+      url = "git+https://git.az-gruppe.com/AZ-Intec-GmbH/phishboard.git";
+      inputs.nixpkgs.follows = "nixpkgs";
+    };
+
+    atrocore-docker = {
+      # Rev-pinned vendor Dockerfiles for the AtroPIM image pipeline (AZ-NIX-ava.1)
+      url = "github:atrocore/docker/e1e9bed1f6ae983d1aac474657cbeba1c004e313";
+      flake = false;
+    };
+  };
+
+  outputs = {
+    self,
+    agenix,
+    agents,
+    nixpkgs,
+    m3ta-nixpkgs,
+    ...
+  } @ inputs: let
+    inherit (self) outputs;
+    systems = [
+      "aarch64-linux"
+      "i686-linux"
+      "x86_64-linux"
+      "aarch64-darwin"
+      "x86_64-darwin"
+    ];
+    forAllSystems = nixpkgs.lib.genAttrs systems;
+  in {
+    packages = forAllSystems (system:
+      import ./pkgs {
+        pkgs = nixpkgs.legacyPackages.${system};
+        atrocore-docker = inputs.atrocore-docker;
+      });
+    overlays = let
+      all = import ./overlays {inherit inputs;};
+    in
+      removeAttrs all ["mkLlmAgentsOverlay"];
+    lib.mkLlmAgentsOverlay = (import ./overlays {inherit inputs;}).mkLlmAgentsOverlay;
+
+    devShells = forAllSystems (system: let
+      pkgs = import nixpkgs {
+        inherit system;
+        config.allowUnfree = true; # Allow unfree packages in devShell
+      };
+      m3taLib = m3ta-nixpkgs.lib.${system};
+      rules = m3taLib.coding-rules.mkCodingRules {
+        inherit agents;
+        languages = ["nix"];
+      };
+    in {
+      default = pkgs.mkShell {
+        buildInputs = with pkgs; [
+          alejandra
+          nixd
+          opencode
+          # pi-coding-agent
+          inputs.agenix.packages.${system}.default
+          outputs.packages.${system}.atropim-tools
+        ];
+
+        shellHook = ''
+          ${rules.shellHook}
+          echo "🚀 NixOS Infrastructure Development Shell with Opencode Rules"
+          echo ""
+          echo "Active rules:"
+          echo "  - Nix language conventions"
+          echo "  - Coding-style best practices"
+          echo "  - Naming conventions"
+          echo "  - Documentation standards"
+          echo "  - Testing guidelines"
+          echo "  - Git workflow patterns"
+          echo "  - Project structure guidelines"
+          echo ""
+          echo "Generated files:"
+          echo "  - .opencode-rules/ (symlink to AGENTS repo rules)"
+          echo "  - coding-rules.json (configuration file)"
+          echo ""
+          echo "Useful commands:"
+          echo "  - cat coding-rules.json    View rules configuration"
+          echo "  - ls .opencode-rules/      Browse available rules"
+          echo "  - nix develop              Re-enter this shell"
+          echo ""
+          echo "🐘 AtroPIM Image Pipeline (AZ-NIX-ava)"
+          echo "  Commands:"
+          echo "    atropim-build            Build secret-free AtroPIM image (podman, 2 stages)"
+          echo "    atropim-push [version]   Tag + push to Gitea registry (default tag: YYYYMMDD)"
+          echo "  Build parameters (pkgs/atropim-image/default.nix):"
+          echo "    SKELETON_VARIANT=pim-no-demo  BUILD_VARIANT=pdf  PRODUCTION_STABILITY=stable"
+          echo "    PRODUCTION_DOMAIN=pim.l.az-gruppe.com  DB build args deliberately EMPTY"
+          echo "  Runtime ENV (written to data/config.php at container start):"
+          echo "    ATRO_DB_HOST  ATRO_DB_NAME  ATRO_DB_USER  ATRO_DB_PASSWORD"
+          echo "  Registry: git.az-gruppe.com/az-intec-gmbh/atrocore-web (podman login git.az-gruppe.com)"
+          echo "  Quick test:"
+          echo "    podman run -d --name atropim -p 127.0.0.1:8080:80 <image>"
+          echo ""
+          echo "Remember to add to .gitignore:"
+          echo "  .opencode-rules"
+          echo "  coding-rules.json"
+          echo "======================================"
+        '';
+      };
+    });
+
+    nixosConfigurations = {
+      AZ-CLD-1 = nixpkgs.lib.nixosSystem {
+        specialArgs = {
+          inherit inputs outputs;
+          system = "x86_64-linux";
+        };
+        modules = [
+          ./hosts/AZ-CLD-1
+          agenix.nixosModules.default
+          inputs.disko.nixosModules.disko
+        ];
+      };
+      AZ-PRM-1 = nixpkgs.lib.nixosSystem {
+        specialArgs = {
+          inherit inputs outputs;
+          system = "x86_64-linux";
+        };
+        modules = [
+          ./hosts/AZ-PRM-1
+          agenix.nixosModules.default
+          inputs.disko.nixosModules.disko
+          inputs.azion-scheduler.nixosModules.default
+          inputs.zugferd-service.nixosModules.default
+          inputs.azess.nixosModules.default
+          inputs.phishboard.nixosModules.default
+        ];
+      };
+    };
+  };
+}

hosts/AZ-CLD-1/configuration.nix

@@ -0,0 +1,137 @@
+# Edit this configuration file to define what should be installed on
+# your system. Help is available in the configuration.nix(5) man page, on
+# https://search.nixos.org/options and in the NixOS manual (`nixos-help`).
+{
+  config,
+  lib,
+  pkgs,
+  ...
+}: {
+  imports = [
+    # Include the results of the hardware scan.
+    ./hardware-configuration.nix
+    ./disko-config.nix
+  ];
+
+  boot.loader.grub = {
+    efiSupport = true;
+    efiInstallAsRemovable = true;
+  };
+
+  swapDevices = [
+    {
+      device = "/var/lib/swapfile";
+      size = 16 * 1024;
+    }
+  ];
+
+  networking.hostName = "AZ-CLD-1"; # Define your hostname.
+  # Pick only one of the below networking options.
+  # networking.wireless.enable = true;  # Enables wireless support via wpa_supplicant.
+  networking.networkmanager.enable = true; # Easiest to use and most distros use this by default.
+
+  # Set your time zone.
+  time.timeZone = "Europe/Berlin";
+
+  # Configure network proxy if necessary
+  # networking.proxy.default = "http://user:password@proxy:port/";
+  # networking.proxy.noProxy = "127.0.0.1,localhost,internal.domain";
+
+  # Select internationalisation properties.
+  i18n.defaultLocale = "de_DE.UTF-8";
+  # console = {
+  #   font = "Lat2-Terminus16";
+  #   keyMap = "us";
+  #   useXkbConfig = true; # use xkb.options in tty.
+  # };
+
+  # Enable the X11 windowing system.
+  # services.xserver.enable = true;
+
+  # Configure keymap in X11
+  # services.xserver.xkb.layout = "us";
+  # services.xserver.xkb.options = "eurosign:e,caps:escape";
+
+  # Enable CUPS to print documents.
+  # services.printing.enable = true;
+
+  # Enable sound.
+  # services.pulseaudio.enable = true;
+  # OR
+  # services.pipewire = {
+  #   enable = true;
+  #   pulse.enable = true;
+  # };
+
+  # Enable touchpad support (enabled default in most desktopManager).
+  # services.libinput.enable = true;
+
+  # Define a user account. Don't forget to set a password with ‘passwd’.
+  # users.users.alice = {
+  #   isNormalUser = true;
+  #   extraGroups = [ "wheel" ]; # Enable ‘sudo’ for the user.
+  #   packages = with pkgs; [
+  #     tree
+  #   ];
+  # };
+
+  # programs.firefox.enable = true;
+
+  # List packages installed in system profile.
+  # You can use https://search.nixos.org/ to find more packages (and options).
+  environment.systemPackages = with pkgs; [
+    neovim # Do not forget to add an editor to edit configuration.nix! The Nano editor is also installed by default.
+    git
+    ghostty
+  ];
+
+  # Some programs need SUID wrappers, can be configured further or are
+  # started in user sessions.
+  # programs.mtr.enable = true;
+  programs.gnupg.agent = {
+    enable = true;
+    enableSSHSupport = true;
+  };
+
+  # List services that you want to enable:
+
+  # Enable the OpenSSH daemon.
+  services.openssh = {
+    enable = true;
+    ports = [2022];
+    settings = {
+      PermitRootLogin = "no";
+      PasswordAuthentication = false;
+    };
+  };
+
+  # Open ports in the firewall.
+  networking.firewall.allowedTCPPorts = [587];
+  # networking.firewall.allowedUDPPorts = [ ... ];
+  # Or disable the firewall altogether.
+  # networking.firewall.enable = false;
+
+  # Copy the NixOS configuration file and link it from the resulting system
+  # (/run/current-system/configuration.nix). This is useful in case you
+  # accidentally delete configuration.nix.
+  # system.copySystemConfiguration = true;
+
+  # This option defines the first version of NixOS you have installed on this particular machine,
+  # and is used to maintain compatibility with application data (e.g. databases) created on older NixOS versions.
+  #
+  # Most users should NEVER change this value after the initial install, for any reason,
+  # even if you've upgraded your system to a new NixOS release.
+  #
+  # This value does NOT affect the Nixpkgs version your packages and OS are pulled from,
+  # so changing it will NOT upgrade your system - see https://nixos.org/manual/nixos/stable/#sec-upgrading for how
+  # to actually do that.
+  #
+  # This value being lower than the current NixOS release does NOT mean your system is
+  # out of date, out of support, or vulnerable.
+  #
+  # Do NOT change this value unless you have manually inspected all the changes it would make to your configuration,
+  # and migrated your data accordingly.
+  #
+  # For more information, see `man configuration.nix` or https://nixos.org/manual/nixos/stable/options#opt-system.stateVersion .
+  system.stateVersion = "25.05"; # Did you read the comment?
+}

hosts/AZ-CLD-1/default.nix

@@ -0,0 +1,12 @@
+{
+  imports = [
+    ../common
+    ./configuration.nix
+    ./secrets.nix
+    ./services
+  ];
+
+  extraServices = {
+    podman.enable = true;
+  };
+}

hosts/AZ-CLD-1/disko-config.nix

@@ -0,0 +1,39 @@
+{
+  disko.devices = {
+    disk = {
+      main = {
+        type = "disk";
+        device = "/dev/vda"; # CHANGE ME
+        content = {
+          type = "gpt";
+          partitions = {
+            boot = {
+              size = "1M";
+              type = "EF02"; # for GRUB MBR
+              priority = 1;
+            };
+            esp = {
+              size = "512M";
+              type = "EF00";
+              content = {
+                type = "filesystem";
+                format = "vfat";
+                mountpoint = "/boot";
+                mountOptions = ["defaults" "umask=0077"];
+              };
+            };
+            root = {
+              size = "100%";
+              content = {
+                type = "filesystem";
+                format = "ext4";
+                mountpoint = "/";
+                mountOptions = ["noatime" "nodiratime" "discard"];
+              };
+            };
+          };
+        };
+      };
+    };
+  };
+}

hosts/AZ-CLD-1/hardware-configuration.nix

@@ -0,0 +1,28 @@
+# Do not modify this file!  It was generated by ‘nixos-generate-config’
+# and may be overwritten by future invocations.  Please make changes
+# to /etc/nixos/configuration.nix instead.
+{
+  config,
+  lib,
+  pkgs,
+  modulesPath,
+  ...
+}: {
+  imports = [
+    (modulesPath + "/profiles/qemu-guest.nix")
+  ];
+
+  boot.initrd.availableKernelModules = ["ata_piix" "uhci_hcd" "virtio_pci" "virtio_scsi" "sd_mod" "sr_mod"];
+  boot.initrd.kernelModules = [];
+  boot.kernelModules = [];
+  boot.extraModulePackages = [];
+
+  # Enables DHCP on each ethernet and wireless interface. In case of scripted networking
+  # (the default) this is the recommended approach. When using systemd-networkd it's
+  # still possible to use this option, but it's recommended to use it in conjunction
+  # with explicit per-interface declarations with `networking.interfaces.<interface>.useDHCP`.
+  networking.useDHCP = lib.mkDefault true;
+  # networking.interfaces.ens18.useDHCP = lib.mkDefault true;
+
+  nixpkgs.hostPlatform = lib.mkDefault "x86_64-linux";
+}

hosts/AZ-CLD-1/secrets.nix

@@ -0,0 +1,127 @@
+{
+  age = {
+    secrets = {
+      traefik-env = {
+        file = ../../secrets/traefik-env.age;
+      };
+      monitoring-loki-remote-env = {
+        file = ../../secrets/monitoring-loki-remote-env.age;
+        owner = "root";
+        group = "root";
+        mode = "0400";
+      };
+      baserow-env = {
+        file = ../../secrets/baserow-env.age;
+      };
+      homarr-env = {
+        file = ../../secrets/homarr-env.age;
+      };
+      librechat = {
+        file = ../../secrets/librechat.age;
+      };
+      librechat-env = {
+        file = ../../secrets/librechat-env.age;
+      };
+      librechat-env-dev = {
+        file = ../../secrets/librechat-env-dev.age;
+      };
+      librechat-env-prod = {
+        file = ../../secrets/librechat-env-prod.age;
+      };
+      litellm-env = {
+        file = ../../secrets/litellm-env.age;
+      };
+      metabase-env = {
+        file = ../../secrets/metabase-env.age;
+      };
+      n8n-env = {
+        file = ../../secrets/n8n-env.age;
+      };
+      n8n-runner-auth-token = {
+        file = ../../secrets/n8n-runner-auth-token-cld.age;
+      };
+      cld-var-backup-sync-ssh-key = {
+        file = ../../secrets/cld-var-backup-sync-ssh-key.age;
+        owner = "root";
+        group = "root";
+        mode = "0400";
+      };
+      netbird-auth-secret = {
+        file = ../../secrets/netbird-auth-secret.age;
+      };
+      netbird-db-password = {
+        file = ../../secrets/netbird-db-password.age;
+      };
+      netbird-encryption-key = {
+        file = ../../secrets/netbird-encryption-key.age;
+      };
+      netbird-dashboard-env = {
+        file = ../../secrets/netbird-dashboard-env.age;
+      };
+      netbird-server-env = {
+        file = ../../secrets/netbird-server-env.age;
+      };
+      netbird-proxy-env = {
+        file = ../../secrets/netbird-proxy-env.age;
+      };
+      outline-env = {
+        file = ../../secrets/outline-env.age;
+        owner = "outline";
+      };
+      snipe-it-app-key = {
+        file = ../../secrets/snipe-it-app-key.age;
+        owner = "snipeit";
+      };
+      snipe-it-db-password = {
+        file = ../../secrets/snipe-it-db-password.age;
+        owner = "snipeit";
+      };
+      snipe-it-mail-password = {
+        file = ../../secrets/snipe-it-mail-password.age;
+        owner = "snipeit";
+      };
+      pgadmin-pw = {
+        file = ../../secrets/pgadmin-pw.age;
+        owner = "pgadmin";
+      };
+      pgbouncer-userlist = {
+        file = ../../secrets/pgbouncer-userlist.age;
+        owner = "pgbouncer";
+      };
+      pgbouncer-tls-cert = {
+        file = ../../secrets/server.crt.age;
+        owner = "pgbouncer";
+        group = "pgbouncer";
+        mode = "0444";
+      };
+      pgbouncer-tls-key = {
+        file = ../../secrets/server.key.age;
+        owner = "pgbouncer";
+        group = "pgbouncer";
+        mode = "0400";
+      };
+      vaultwarden-env = {
+        file = ../../secrets/vaultwarden-env.age;
+      };
+      hetzner-s3-az-intern-secret-key = {
+        file = ../../secrets/hetzner-s3-az-intern-secret-key.age;
+        owner = "outline";
+      };
+      hetzner-s3-az-intern-access-key = {
+        file = ../../secrets/hetzner-s3-az-intern-access-key.age;
+      };
+      zammad-pw = {
+        file = ../../secrets/zammad-pw.age;
+      };
+      zammad-secret = {
+        file = ../../secrets/zammad-secret.age;
+      };
+      zammad-hr-env-prod = {
+        file = ../../secrets/zammad-hr-env-prod.age;
+      };
+      zammad-hr-env = {
+        file = ../../secrets/zammad-hr-env.age;
+      };
+    };
+  };
+}

hosts/AZ-CLD-1/services/containers/baserow.nix

@@ -0,0 +1,70 @@
+{config, ...}: let
+  serviceName = "baserow";
+  servicePort = config.m3ta.ports.get serviceName;
+in {
+  virtualisation.oci-containers.containers.${serviceName} = {
+    image = "docker.io/baserow/baserow:2.3.3";
+    environment = {
+      # BASEROW_AMOUNT_OF_GUNICORN_WORKERS = "4";
+      # BASEROW_AMOUNT_OF_WORKERS = "2";
+      DATABASE_CONN_MAX_AGE = "0";
+      # Proxy: tell Django the connection is HTTPS so cookies get Secure flag
+      BASEROW_ENABLE_SECURE_PROXY_SSL_HEADER = "yes";
+      # Published apps run on different origins — allow cross-origin cookie delivery
+      BASEROW_FRONTEND_SAME_SITE_COOKIE = "none";
+      # Valid base domain for published app subdomains
+      BASEROW_BUILDER_DOMAINS = "az-gruppe.com";
+      # Disable Caddy's on_demand TLS — Traefik handles TLS termination
+      BASEROW_CADDY_GLOBAL_CONF = "auto_https off";
+    };
+    environmentFiles = [config.age.secrets.baserow-env.path];
+    ports = ["127.0.0.1:${toString servicePort}:80"];
+    volumes = ["baserow_data:/baserow/data"];
+    extraOptions = ["--add-host=postgres:10.89.0.1" "--ip=10.89.0.10" "--network=web"];
+  };
+
+  # Traefik configuration
+  services.traefik.dynamicConfigOptions.http = {
+    services.${serviceName}.loadBalancer.servers = [
+      {
+        url = "http://localhost:${toString servicePort}/";
+      }
+    ];
+
+    middlewares."${serviceName}-headers".headers = {
+      customRequestHeaders = {
+        X-Forwarded-Proto = "https";
+        X-Forwarded-Port = "443";
+      };
+    };
+
+    routers.${serviceName} = {
+      rule = "Host(`br.az-gruppe.com`)";
+      tls = {
+        certResolver = "ionos";
+      };
+      service = serviceName;
+      entrypoints = "websecure";
+      middlewares = ["${serviceName}-headers"];
+    };
+
+    routers.azubi = {
+      rule = "Host(`azubi.az-gruppe.com`)";
+      tls = {
+        certResolver = "ionos";
+      };
+      service = serviceName;
+      entrypoints = "websecure";
+      middlewares = ["${serviceName}-headers"];
+    };
+    routers.ausbilder = {
+      rule = "Host(`ausbilder.az-gruppe.com`)";
+      tls = {
+        certResolver = "ionos";
+      };
+      service = serviceName;
+      entrypoints = "websecure";
+      middlewares = ["${serviceName}-headers"];
+    };
+  };
+}

hosts/AZ-CLD-1/services/containers/default.nix

@@ -0,0 +1,21 @@
+{lib, ...}: {
+  imports = [
+    ./baserow.nix
+    ./homarr.nix
+    ./it-tools.nix
+    ./librechat.nix
+    ./litellm.nix
+    ./librechat-dev.nix
+    ./netbird.nix
+    # ./portainer.nix
+    ./zammad-hr.nix
+  ];
+  system.activationScripts.createPodmanNetworkWeb = lib.mkAfter ''
+    if ! /run/current-system/sw/bin/podman network exists web; then
+      /run/current-system/sw/bin/podman network create web --subnet=10.89.0.0/24
+    fi
+    if ! /run/current-system/sw/bin/podman network exists web-dev; then
+      /run/current-system/sw/bin/podman network create web-dev --subnet=10.89.1.0/24
+    fi
+  '';
+}

hosts/AZ-CLD-1/services/containers/homarr.nix

@@ -0,0 +1,49 @@
+{config, ...}: let
+  serviceName = "homarr";
+  servicePort = config.m3ta.ports.get serviceName;
+in {
+  virtualisation.oci-containers.containers.${serviceName} = {
+    image = "ghcr.io/homarr-labs/homarr:latest";
+    environment = {
+      TZ = "Europe/Berlin";
+
+      BASE_URL = "https://dash.az-gruppe.com";
+      NEXTAUTH_URL = "https://dash.az-gruppe.com";
+      AUTH_PROVIDERS = "oidc";
+      AUTH_OIDC_CLIENT_NAME = "Azure";
+      AUTH_OIDC_SCOPE_OVERWRITE = "openid email profile";
+      AUTH_OIDC_GROUPS_ATTRIBUTE = "roles";
+      AUTH_OIDC_GROUPS_LOCAL_MANAGEMENT = "true";
+    };
+    environmentFiles = [config.age.secrets.homarr-env.path];
+    ports = ["127.0.0.1:${toString servicePort}:7575"];
+    volumes = ["homarr_data:/appdata"];
+    extraOptions = ["--ip=10.89.0.13" "--network=web" "--dns=8.8.8.8" "--dns=8.8.4.4"];
+  };
+
+  # Traefik configuration
+  services.traefik.dynamicConfigOptions.http = {
+    services.${serviceName}.loadBalancer.servers = [
+      {
+        url = "http://localhost:${toString servicePort}/";
+      }
+    ];
+
+    middlewares."${serviceName}-headers".headers = {
+      customRequestHeaders = {
+        X-Forwarded-Proto = "https";
+        X-Forwarded-Port = "443";
+      };
+    };
+
+    routers.${serviceName} = {
+      rule = "Host(`dash.az-gruppe.com`)";
+      tls = {
+        certResolver = "ionos";
+      };
+      service = serviceName;
+      entrypoints = "websecure";
+      middlewares = ["${serviceName}-headers"];
+    };
+  };
+}

hosts/AZ-CLD-1/services/containers/it-tools.nix

@@ -0,0 +1,27 @@
+{config, ...}: let
+  serviceName = "it-tools";
+  servicePort = config.m3ta.ports.get serviceName;
+in {
+  virtualisation.oci-containers.containers.${serviceName} = {
+    image = "docker.io/sharevb/it-tools:latest";
+    ports = ["127.0.0.1:${toString servicePort}:8080"];
+  };
+
+  # Traefik configuration
+  services.traefik.dynamicConfigOptions.http = {
+    services.${serviceName}.loadBalancer.servers = [
+      {
+        url = "http://localhost:${toString servicePort}/";
+      }
+    ];
+
+    routers.${serviceName} = {
+      rule = "Host(`tools.az-gruppe.com`)";
+      tls = {
+        certResolver = "ionos";
+      };
+      service = serviceName;
+      entrypoints = "websecure";
+    };
+  };
+}

hosts/AZ-CLD-1/services/containers/librechat-dev.nix

@@ -0,0 +1,133 @@
+{
+  config,
+  pkgs,
+  ...
+}: let
+  serviceName = "librechat-dev";
+  servicePort = config.m3ta.ports.get serviceName;
+  ragApiDevServiceName = "rag-api-dev";
+  ragApiDevPort = config.m3ta.ports.get ragApiDevServiceName;
+  envFileDev = config.age.secrets.librechat-env-dev.path;
+  envFileCommon = config.age.secrets.librechat.path;
+in {
+  virtualisation.oci-containers = {
+    containers.meilisearch-dev = {
+      image = "getmeili/meilisearch:v1.12.3";
+      autoStart = false;
+      volumes = ["librechat_dev_meili:/meili_data"];
+      environment = {
+        MEILI_HTTP_ADDR = "0.0.0.0:7700";
+        MEILI_NO_ANALYTICS = "true";
+      };
+      environmentFiles = [envFileDev envFileCommon];
+      extraOptions = ["--ip=10.89.1.20" "--network=web-dev"];
+    };
+
+    containers.rag_api-dev = {
+      image = "ghcr.io/danny-avila/librechat-rag-api-dev-lite:latest";
+      autoStart = false;
+      environment = {
+        RAG_PORT = "8000";
+        DB_HOST = "10.89.1.1";
+        DB_PORT = "6432";
+      };
+      environmentFiles = [envFileDev envFileCommon];
+      dependsOn = ["meilisearch-dev"];
+      extraOptions = ["--add-host=postgres:10.89.1.1" "--ip=10.89.1.21" "--network=web-dev"];
+      ports = ["127.0.0.1:${toString ragApiDevPort}:8000"];
+    };
+
+    containers.mongodb-dev = {
+      image = "mongo:7";
+      autoStart = false;
+      volumes = [
+        "librechat_dev_mongo:/data/db"
+        "/var/backup/mongodb-dev:/data/backups"
+      ];
+      extraOptions = ["--ip=10.89.1.22" "--network=web-dev"];
+    };
+
+    containers.${serviceName} = {
+      image = "ghcr.io/danny-avila/librechat-dev-api:latest";
+      autoStart = false;
+      ports = ["127.0.0.1:${toString servicePort}:3080"];
+      dependsOn = ["mongodb-dev" "rag_api-dev" "meilisearch-dev"];
+      environment = {
+        HOST = "0.0.0.0";
+        NODE_ENV = "development";
+        MONGO_URI = "mongodb://mongodb-dev:27017/LibreChatDev";
+        MEILI_HOST = "http://meilisearch-dev:7700";
+        RAG_PORT = "8000";
+        RAG_API_URL = "http://rag_api-dev:8000";
+      };
+      environmentFiles = [envFileDev envFileCommon];
+      volumes = [
+        "/var/lib/librechat-dev/librechat.yaml:/app/librechat.yaml:ro"
+        "librechat_dev_images:/app/client/public/images"
+        "librechat_dev_uploads:/app/uploads"
+        "librechat_dev_logs:/app/api/logs"
+      ];
+      extraOptions = ["--ip=10.89.1.23" "--network=web-dev"];
+    };
+  };
+
+  # Traefik configuration
+  services.traefik.dynamicConfigOptions.http = {
+    services.${serviceName}.loadBalancer.servers = [
+      {
+        url = "http://localhost:${toString servicePort}/";
+      }
+    ];
+
+    routers.${serviceName} = {
+      rule = "Host(`chat-dev.az-gruppe.com`)";
+      tls = {
+        certResolver = "ionos";
+      };
+      service = serviceName;
+      entrypoints = "websecure";
+    };
+  };
+
+  environment.systemPackages = [
+    (pkgs.writeShellScriptBin "librechat-dev" ''
+      #!/usr/bin/env bash
+      set -e
+
+      SERVICES=(
+        podman-meilisearch-dev
+        podman-mongodb-dev
+        podman-rag_api-dev
+        podman-librechat-dev
+      )
+
+      case "$1" in
+        up)
+          echo "🚀 Starte LibreChat-Dev-Umgebung..."
+          for svc in "''${SERVICES[@]}"; do
+            sudo systemctl start "$svc"
+          done
+          ;;
+        down)
+          echo "🛑 Stoppe LibreChat-Dev-Umgebung..."
+          for svc in "''${SERVICES[@]}"; do
+            sudo systemctl stop "$svc"
+          done
+          ;;
+        restart)
+          echo "🔄 Neustart der LibreChat-Dev-Umgebung..."
+          for svc in "''${SERVICES[@]}"; do
+            sudo systemctl restart "$svc"
+          done
+          ;;
+        status)
+          systemctl status "''${SERVICES[@]}"
+          ;;
+        *)
+          echo "Usage: librechat-dev {up|down|restart|status}"
+          exit 1
+          ;;
+      esac
+    '')
+  ];
+}

hosts/AZ-CLD-1/services/containers/librechat.nix

@@ -0,0 +1,170 @@
+{
+  config,
+  pkgs,
+  ...
+}: let
+  serviceName = "librechat";
+  servicePort = config.m3ta.ports.get serviceName;
+  ragApiServiceName = "rag-api";
+  ragApiPort = config.m3ta.ports.get ragApiServiceName;
+  envFileProd = config.age.secrets.librechat-env-prod.path;
+  envFileCommon = config.age.secrets.librechat.path;
+in {
+  virtualisation.oci-containers = {
+    containers.meilisearch = {
+      image = "getmeili/meilisearch:v1.35.1";
+      autoStart = true;
+      volumes = ["librechat_meili:/meili_data"];
+      environment = {
+        MEILI_HTTP_ADDR = "0.0.0.0:7700";
+        MEILI_NO_ANALYTICS = "true";
+      };
+      environmentFiles = [envFileCommon envFileProd];
+      extraOptions = ["--ip=10.89.0.20" "--network=web"];
+    };
+
+    containers.rag_api = {
+      image = "registry.librechat.ai/danny-avila/librechat-rag-api-dev-lite:latest";
+      autoStart = true;
+      environment = {
+        RAG_PORT = "8000";
+        DB_HOST = "10.89.0.1";
+        DB_PORT = "6432";
+      };
+      environmentFiles = [envFileCommon envFileProd];
+      dependsOn = ["meilisearch"];
+      extraOptions = ["--add-host=postgres:10.89.0.1" "--ip=10.89.0.21" "--network=web"];
+      ports = ["127.0.0.1:${toString ragApiPort}:8000"];
+    };
+
+    containers.mongodb = {
+      image = "mongo:8.0.20";
+      autoStart = true;
+      cmd = ["mongod" "--noauth"];
+      volumes = [
+        "librechat_mongo:/data/db"
+        "/var/backup/mongodb:/data/backups"
+      ];
+      # Enable auth once users exist; see Mongo auth doc.
+      # command = [ "mongod", "--auth" ];
+      extraOptions = ["--ip=10.89.0.22" "--network=web"];
+    };
+
+    containers.${serviceName} = {
+      image = "registry.librechat.ai/danny-avila/librechat-dev:latest";
+      autoStart = true;
+      user = "1000:1000";
+      ports = ["127.0.0.1:${toString servicePort}:3080"];
+      dependsOn = ["mongodb" "rag_api" "meilisearch"];
+      environment = {
+        HOST = "0.0.0.0";
+        NODE_ENV = "production";
+        # Mongo URI (start without auth; switch to mongodb://user:pass@mongodb:27017/LibreChat after Step 4)
+        MONGO_URI = "mongodb://mongodb:27017/LibreChat";
+        MEILI_HOST = "http://meilisearch:7700";
+        RAG_PORT = "8000";
+        RAG_API_URL = "http://rag_api:8000";
+      };
+      environmentFiles = [envFileCommon envFileProd];
+      volumes = [
+        # Config file still needs to be a bind mount for host management
+        "/var/lib/librechat/librechat.yaml:/app/librechat.yaml:ro"
+        # Use named volumes for application data
+        "librechat_images:/app/client/public/images"
+        "librechat_uploads:/app/uploads"
+        "librechat_logs:/app/logs"
+      ];
+      extraOptions = ["--ip=10.89.0.23" "--network=web" "--dns=8.8.8.8" "--dns=8.8.4.4"];
+    };
+  };
+
+  systemd.services."mongo-backup" = {
+    serviceConfig = {
+      Type = "oneshot";
+      User = "root";
+      Group = "root";
+    };
+    script = ''
+      set -euo pipefail
+
+      BACKUP_DIR="/var/backup/mongodb"
+      TIMESTAMP=$(date +%Y%m%d_%H%M%S)
+      TEMP_BACKUP="mongodb_backup_$TIMESTAMP"
+      ARCHIVE_NAME="mongodb_backup_$TIMESTAMP.tar.gz"
+
+      # Ensure backup directory exists with proper permissions
+      mkdir -p "$BACKUP_DIR"
+      chown root:root "$BACKUP_DIR"
+      chmod 750 "$BACKUP_DIR"
+
+      echo "Starting MongoDB backup at $(date)"
+
+      # Create the backup dump in container
+      if ${pkgs.podman}/bin/podman exec mongodb mongodump --out "/data/backups/$TEMP_BACKUP"; then
+        echo "MongoDB dump completed successfully"
+
+        # Create compressed archive from the backup
+        cd "$BACKUP_DIR"
+        if [ -d "$TEMP_BACKUP" ]; then
+          echo "Creating compressed archive: $ARCHIVE_NAME"
+          ${pkgs.gnutar}/bin/tar --use-compress-program=${pkgs.gzip}/bin/gzip -cf "$ARCHIVE_NAME" -C . "$TEMP_BACKUP"
+
+          # Remove the uncompressed backup directory
+          rm -rf "$TEMP_BACKUP"
+
+          # Verify archive was created
+          if [ -f "$ARCHIVE_NAME" ]; then
+            ARCHIVE_SIZE=$(${pkgs.coreutils}/bin/du -sh "$ARCHIVE_NAME" | cut -f1)
+            echo "Compressed backup created: $ARCHIVE_NAME (Size: $ARCHIVE_SIZE)"
+
+            # Keep only the 2 most recent backup archives
+            ls -1t mongodb_backup_*.tar.gz | tail -n +3 | xargs -r rm -f
+            echo "Old backup archives cleaned up, keeping 2 most recent"
+
+            # List current backups
+            echo "Current backups:"
+            ls -lah mongodb_backup_*.tar.gz 2>/dev/null || echo "No previous backups found"
+          else
+            echo "ERROR: Failed to create compressed archive" >&2
+            exit 1
+          fi
+        else
+          echo "ERROR: Backup directory not found at $BACKUP_DIR/$TEMP_BACKUP" >&2
+          exit 1
+        fi
+      else
+        echo "ERROR: MongoDB backup failed" >&2
+        exit 1
+      fi
+
+      echo "MongoDB backup completed successfully at $(date)"
+    '';
+  };
+
+  systemd.timers."mongo-backup" = {
+    wantedBy = ["timers.target"];
+    timerConfig = {
+      OnCalendar = "*-*-* 02:00:00";
+      RandomizedDelaySec = "30m";
+      Persistent = true;
+    };
+  };
+
+  # Traefik configuration
+  services.traefik.dynamicConfigOptions.http = {
+    services.${serviceName}.loadBalancer.servers = [
+      {
+        url = "http://localhost:${toString servicePort}/";
+      }
+    ];
+
+    routers.${serviceName} = {
+      rule = "Host(`chat.az-gruppe.com`)";
+      tls = {
+        certResolver = "ionos";
+      };
+      service = serviceName;
+      entrypoints = "websecure";
+    };
+  };
+}

hosts/AZ-CLD-1/services/containers/litellm.nix

@@ -0,0 +1,137 @@
+{
+  config,
+  pkgs,
+  ...
+}: let
+  serviceName = "litellm";
+  servicePort = config.m3ta.ports.get serviceName;
+  prmNetbirdIP = "100.91.49.26";
+  cldNetbirdIP = "100.91.203.184";
+
+  python = pkgs.python3.withPackages (ps: [ps.pyyaml]);
+  litellmConfigGenerator = pkgs.writeText "litellm-config-generator.py" ''
+    import sys
+    from pathlib import Path
+
+    import yaml
+
+    base_path = Path(sys.argv[1])
+    target_path = Path(sys.argv[2])
+
+    with base_path.open("r", encoding="utf-8") as fh:
+        config = yaml.safe_load(fh) or {}
+
+    settings = config.get("litellm_settings")
+    if not isinstance(settings, dict):
+        settings = {}
+        config["litellm_settings"] = settings
+
+    def ensure_list(value):
+        if value is None:
+            return []
+        if isinstance(value, list):
+            return value
+        return [value]
+
+    callbacks = ensure_list(settings.get("callbacks"))
+    if "prometheus" not in callbacks:
+        callbacks.append("prometheus")
+    settings["callbacks"] = callbacks
+
+    service_callbacks = ensure_list(settings.get("service_callback"))
+    if "prometheus_system" not in service_callbacks:
+        service_callbacks.append("prometheus_system")
+    settings["service_callback"] = service_callbacks
+
+    # LiteLLM >= 1.85 protects /metrics by default. Keep auth enabled;
+    # Prometheus scrapes with a bearer token from an agenix secret on AZ-PRM-1.
+    settings["require_auth_for_metrics_endpoint"] = True
+
+    target_path.parent.mkdir(parents=True, exist_ok=True)
+    with target_path.open("w", encoding="utf-8") as fh:
+        yaml.safe_dump(config, fh, sort_keys=False)
+  '';
+in {
+  systemd.services."podman-${serviceName}".preStart = ''
+    set -euo pipefail
+
+    base_config="/var/lib/${serviceName}/config.yaml"
+    target_config="/run/${serviceName}/config.yaml"
+    multiproc_dir="/var/lib/${serviceName}/prometheus-multiproc"
+
+    if [ ! -f "$base_config" ]; then
+      echo "Missing LiteLLM base config: $base_config" >&2
+      exit 1
+    fi
+
+    mkdir -p "$(dirname "$target_config")" "$multiproc_dir"
+    rm -f "$multiproc_dir"/*
+    chmod 0777 "$multiproc_dir"
+
+    ${python}/bin/python ${litellmConfigGenerator} "$base_config" "$target_config"
+    chmod 0644 "$target_config"
+  '';
+
+  virtualisation.oci-containers.containers.${serviceName} = {
+    #image = "ghcr.io/berriai/litellm:v1.78.5-stable";
+    image = "docker.litellm.ai/berriai/litellm:1.95.0";
+    ports = [
+      "127.0.0.1:${toString servicePort}:4000"
+      "${cldNetbirdIP}:${toString servicePort}:4000"
+    ];
+    cmd = ["--config" "/app/config.yaml" "--port" "4000"];
+    environmentFiles = [config.age.secrets.litellm-env.path];
+    environment = {
+      ANONYMIZED_TELEMETRY = "False";
+      DO_NOT_TRACK = "True";
+      SCARF_NO_ANALYTICS = "True";
+      STORE_MODEL_IN_DB = "True";
+      PROMETHEUS_MULTIPROC_DIR = "/prometheus_multiproc";
+    };
+    volumes = [
+      "/run/${serviceName}/config.yaml:/app/config.yaml:ro"
+      "/var/lib/${serviceName}/prometheus-multiproc:/prometheus_multiproc"
+    ];
+    extraOptions = ["--add-host=postgres:10.89.0.1" "--ip=10.89.0.30" "--network=web"];
+  };
+
+  # Traefik configuration
+  services.traefik.dynamicConfigOptions.http = {
+    services.${serviceName}.loadBalancer.servers = [
+      {
+        url = "http://localhost:${toString servicePort}/";
+      }
+    ];
+
+    middlewares."${serviceName}-metrics-local-only".ipAllowList.sourceRange = [
+      "127.0.0.1/32"
+      "::1/128"
+      "${prmNetbirdIP}/32"
+    ];
+
+    routers."${serviceName}-metrics" = {
+      rule = "Host(`llm.az-gruppe.com`) && PathPrefix(`/metrics`)";
+      tls = {
+        certResolver = "ionos";
+      };
+      service = serviceName;
+      entrypoints = "websecure";
+      middlewares = ["${serviceName}-metrics-local-only"];
+      priority = 200;
+    };
+
+    routers.${serviceName} = {
+      rule = "Host(`llm.az-gruppe.com`)";
+      tls = {
+        certResolver = "ionos";
+      };
+      service = serviceName;
+      entrypoints = "websecure";
+      priority = 1;
+    };
+  };
+
+  networking.firewall.extraCommands = ''
+    iptables -A INPUT -p tcp -s ${prmNetbirdIP} --dport ${toString servicePort} -j ACCEPT
+  '';
+}

hosts/AZ-CLD-1/services/containers/netbird.nix

@@ -0,0 +1,251 @@
+{
+  config,
+  pkgs,
+  ...
+}: let
+  serviceName = "netbird";
+
+  servicePort = config.m3ta.ports.get serviceName;
+
+  domain = "v.az-gruppe.com";
+  proxyDomain = "p.az-gruppe.com";
+
+  ipBase = "10.89.0";
+  ipOffset = 50;
+
+  # Derived IPs
+  gatewayIp = "${ipBase}.1";
+  dashboardIp = "${ipBase}.${toString ipOffset}";
+  serverIp = "${ipBase}.${toString (ipOffset + 1)}";
+  proxyIp = "${ipBase}.${toString (ipOffset + 2)}";
+
+  # Database configuration
+  dbName = "netbird";
+  dbUser = "netbird";
+  dbHost = gatewayIp;
+
+  # NetBird config as Nix attribute set
+  netbirdConfig = {
+    server = {
+      listenAddress = ":80";
+      exposedAddress = "https://${domain}:443";
+      stunPorts = [3478];
+      metricsPort = 9090;
+      healthcheckAddress = ":9000";
+      logLevel = "info";
+      logFile = "console";
+      dataDir = "/var/lib/netbird";
+
+      auth = {
+        issuer = "https://${domain}/oauth2";
+        localAuthDisabled = true;
+        signKeyRefreshEnabled = true;
+        dashboardRedirectURIs = [
+          "https://${domain}/nb-auth"
+          "https://${domain}/nb-silent-auth"
+        ];
+        cliRedirectURIs = ["http://localhost:53000/"];
+      };
+
+      reverseProxy = {
+        trustedHTTPProxies = ["${gatewayIp}/32"];
+      };
+
+      # Proxy Feature
+      proxy = {
+        enabled = true;
+        domain = proxyDomain;
+      };
+
+      store = {
+        engine = "postgres";
+        postgres = {
+          host = dbHost;
+          port = 5432;
+          database = dbName;
+          username = dbUser;
+        };
+      };
+    };
+  };
+
+  # Generate YAML config
+  yamlFormat = pkgs.formats.yaml {};
+  configYamlBase = yamlFormat.generate "netbird-config-base.yaml" netbirdConfig;
+
+  # Script to inject secrets at runtime
+  configGenScript = pkgs.writeShellScript "netbird-gen-config" ''
+    set -euo pipefail
+
+    AUTH_SECRET=$(cat "$1")
+    DB_PASSWORD=$(cat "$2")
+    ENCRYPTION_KEY=$(cat "$3")
+
+    ${pkgs.yq-go}/bin/yq eval "
+      .server.authSecret = \"$AUTH_SECRET\" |
+      .server.store.encryptionKey = \"$ENCRYPTION_KEY\" |
+      .server.store.postgres.password = \"$DB_PASSWORD\"
+    " ${configYamlBase}
+  '';
+in {
+  age.secrets."${serviceName}-auth-secret".file = ../../../../secrets/${serviceName}-auth-secret.age;
+  age.secrets."${serviceName}-db-password".file = ../../../../secrets/${serviceName}-db-password.age;
+  age.secrets."${serviceName}-encryption-key".file = ../../../../secrets/${serviceName}-encryption-key.age;
+  age.secrets."${serviceName}-dashboard-env".file = ../../../../secrets/${serviceName}-dashboard-env.age;
+  age.secrets."${serviceName}-server-env".file = ../../../../secrets/${serviceName}-server-env.age;
+  age.secrets."${serviceName}-proxy-env".file = ../../../../secrets/${serviceName}-proxy-env.age;
+
+  # Systemd oneshot service to generate config with secrets
+  systemd.services."${serviceName}-config" = {
+    description = "Generate NetBird config with secrets";
+    wantedBy = ["multi-user.target"];
+    before = ["podman-${serviceName}-server.service"];
+    requiredBy = ["podman-${serviceName}-server.service"];
+
+    serviceConfig = {
+      Type = "oneshot";
+      RemainAfterExit = true;
+      ExecStart = pkgs.writeShellScript "netbird-write-config" ''
+        mkdir -p /var/lib/${serviceName}
+        ${configGenScript} \
+          ${config.age.secrets."${serviceName}-auth-secret".path} \
+          ${config.age.secrets."${serviceName}-db-password".path} \
+          ${config.age.secrets."${serviceName}-encryption-key".path} \
+          > /var/lib/${serviceName}/config.yaml
+        chmod 600 /var/lib/${serviceName}/config.yaml
+      '';
+    };
+  };
+
+  virtualisation.oci-containers.containers = {
+    "${serviceName}-dashboard" = {
+      image = "netbirdio/dashboard:latest";
+      autoStart = true;
+      ports = ["127.0.0.1:${toString servicePort}:80"];
+      environmentFiles = [config.age.secrets."${serviceName}-dashboard-env".path];
+      extraOptions = [
+        "--ip=${dashboardIp}"
+        "--network=web"
+      ];
+    };
+
+    "${serviceName}-server" = {
+      image = "netbirdio/netbird-server:latest";
+      autoStart = true;
+      ports = ["3478:3478/udp"];
+      environmentFiles = [config.age.secrets."${serviceName}-server-env".path];
+      volumes = [
+        "${serviceName}_data:/var/lib/netbird"
+        "/var/lib/${serviceName}/config.yaml:/etc/netbird/config.yaml:ro"
+      ];
+      cmd = ["--config" "/etc/netbird/config.yaml"];
+      extraOptions = [
+        "--ip=${serverIp}"
+        "--network=web"
+      ];
+    };
+
+    "${serviceName}-proxy" = {
+      image = "netbirdio/reverse-proxy:latest";
+      autoStart = true;
+      ports = ["51820:51820/udp"];
+      volumes = [
+        "${serviceName}_proxy_certs:/certs"
+      ];
+      environment = {
+        # Enable private services (NetBird Agent Network). Reachable only
+        # over the WireGuard overlay; endpoints are auto-generated under the
+        # proxy domain (e.g. <name>.p.az-gruppe.com) with a MagicDNS record
+        # pointing at this proxy's overlay peer IP. TLS is still terminated
+        # here via the existing ACME wildcard cert.
+        NB_PROXY_PRIVATE = "true";
+      };
+      environmentFiles = [config.age.secrets."${serviceName}-proxy-env".path];
+      cmd = [
+        "--domain=${proxyDomain}"
+        "--mgmt=https://${domain}:443"
+        "--addr=:8443"
+        "--cert-dir=/certs"
+        "--acme-certs"
+        "--trusted-proxies=${gatewayIp}/32"
+      ];
+      dependsOn = ["${serviceName}-server"];
+      extraOptions = [
+        "--ip=${proxyIp}"
+        "--network=web"
+      ];
+    };
+  };
+
+  services.traefik.dynamicConfigOptions = {
+    # HTTP services and routers
+    http = {
+      services = {
+        "${serviceName}-dashboard".loadBalancer.servers = [
+          {url = "http://localhost:${toString servicePort}/";}
+        ];
+
+        "${serviceName}-server".loadBalancer.servers = [
+          {url = "http://${serverIp}:80/";}
+        ];
+
+        "${serviceName}-server-h2c".loadBalancer.servers = [
+          {url = "h2c://${serverIp}:80";}
+        ];
+      };
+
+      routers = {
+        # gRPC (Signal + Management)
+        "${serviceName}-grpc" = {
+          rule = "Host(`${domain}`) && (PathPrefix(`/signalexchange.SignalExchange/`) || PathPrefix(`/management.ManagementService/`) || PathPrefix(`/management.ProxyService/`))";
+          entrypoints = "websecure";
+          tls.certResolver = "ionos";
+          service = "${serviceName}-server-h2c";
+          priority = 100;
+        };
+        # Backend (relay, WebSocket, API, OAuth2)
+        "${serviceName}-backend" = {
+          rule = "Host(`${domain}`) && (PathPrefix(`/relay`) || PathPrefix(`/ws-proxy/`) || PathPrefix(`/api`) || PathPrefix(`/oauth2`))";
+          entrypoints = "websecure";
+          tls.certResolver = "ionos";
+          service = "${serviceName}-server";
+          priority = 100;
+        };
+
+        # Dashboard (catch-all, lowest priority)
+        "${serviceName}-dashboard" = {
+          rule = "Host(`${domain}`)";
+          entrypoints = "websecure";
+          tls.certResolver = "ionos";
+          service = "${serviceName}-dashboard";
+          priority = 1;
+        };
+      };
+    };
+
+    # TCP for proxy TLS passthrough
+    tcp = {
+      services."${serviceName}-proxy-tls".loadBalancer.servers = [
+        {address = "${proxyIp}:8443";}
+      ];
+
+      routers."${serviceName}-proxy-passthrough" = {
+        entryPoints = ["websecure"];
+        rule = "HostSNI(`*`)";
+        service = "${serviceName}-proxy-tls";
+        priority = 1;
+        tls.passthrough = true;
+      };
+    };
+
+    # ServersTransport for proxy protocol v2 (optional)
+    serversTransports."pp-v2" = {
+      proxyProtocol.version = 2;
+    };
+  };
+
+  networking.firewall.allowedUDPPorts = [
+    3478 # STUN
+    51820 # WireGuard for public proxy
+  ];
+}

hosts/AZ-CLD-1/services/containers/portainer.nix

@@ -0,0 +1,32 @@
+{config, ...}: let
+  serviceName = "portainer";
+  servicePort = config.m3ta.ports.get serviceName;
+in {
+  virtualisation.oci-containers.containers.${serviceName} = {
+    image = "docker.io/portainer/portainer-ce:latest";
+    ports = ["127.0.0.1:${toString servicePort}:9000"];
+    volumes = [
+      "/etc/localtime:/etc/localtime:ro"
+      "/run/podman/podman.sock:/var/run/docker.sock:ro"
+      "portainer_data:/data"
+    ];
+  };
+
+  # Traefik configuration
+  services.traefik.dynamicConfigOptions.http = {
+    services.${serviceName}.loadBalancer.servers = [
+      {
+        url = "http://localhost:${toString servicePort}/";
+      }
+    ];
+
+    routers.${serviceName} = {
+      rule = "Host(`pt.az-gruppe.com`)";
+      tls = {
+        certResolver = "ionos";
+      };
+      service = serviceName;
+      entrypoints = "websecure";
+    };
+  };
+}

hosts/AZ-CLD-1/services/containers/zammad-hr.nix

@@ -0,0 +1,342 @@
+{
+  config,
+  pkgs,
+  lib,
+  ...
+}: let
+  instanceName = "hr";
+  serviceName = "zammad-${instanceName}";
+
+  servicePort = config.m3ta.ports.get serviceName;
+  elasticsearchServiceName = "${serviceName}-elasticsearch";
+  elasticsearchPort = config.m3ta.ports.get elasticsearchServiceName;
+
+  envFileProd = config.age.secrets."${serviceName}-env-prod".path;
+  envFileCommon = config.age.secrets."${serviceName}-env".path;
+
+  zammadVersion = "7.1.0";
+  zammadImage = "ghcr.io/zammad/zammad:${zammadVersion}";
+
+  ipBase = "10.89.0";
+  ipOffset = 40;
+
+  # Domain-Konfiguration
+  zammadDomain = "tickets.az-gruppe.com";
+  # Alte Domain wird per 301 auf zammadDomain weitergeleitet (siehe Traefik-Config unten)
+  zammadDomainOld = "hr-ticket.az-gruppe.com";
+
+  sharedEnvironment = {
+    MEMCACHE_SERVERS = "zammad-memcached:11211";
+    POSTGRESQL_DB = "zammad_${instanceName}";
+    POSTGRESQL_HOST = "10.89.0.1";
+    POSTGRESQL_USER = "zammad_${instanceName}";
+    POSTGRESQL_PORT = "6433";
+    # pool=50 entspricht dem Zammad-Default (config/database/database.yml).
+    # Der Scheduler startet mehrere BackgroundServices-Threads gleichzeitig
+    # (ProcessSessions/Scheduled/Delayed/DelayedAI + Cleanup/fetch); pool=5
+    # war zu klein -> ActiveRecord::ConnectionTimeoutError beim Start.
+    # Wird durch PgBouncer-Session (default_pool_size=30) server-seitig gedeckelt.
+    POSTGRESQL_OPTIONS = "?pool=50";
+    REDIS_URL = "redis://zammad-redis:6379";
+    TZ = "Europe/Berlin";
+    BACKUP_DIR = "/var/tmp/zammad";
+    BACKUP_TIME = "03:00";
+    HOLD_DAYS = "10";
+    ELASTICSEARCH_ENABLED = "true";
+    ELASTICSEARCH_HOST = "zammad-elasticsearch";
+    ELASTICSEARCH_PORT = "9200";
+    ELASTICSEARCH_NAMESPACE = "zammad_${instanceName}";
+    NGINX_PORT = "8080";
+
+    # CSRF & Reverse Proxy Settings
+    NGINX_SERVER_SCHEME = "https";
+    NGINX_SERVER_NAME = zammadDomain;
+    ZAMMAD_HTTP_TYPE = "https";
+    ZAMMAD_FQDN = zammadDomain;
+    RAILS_TRUSTED_PROXIES = "127.0.0.1,::1,${ipBase}.1,${ipBase}.${toString (ipOffset + 7)}";
+  };
+
+  alwaysRestart = {
+    serviceConfig = {
+      Restart = lib.mkOverride 90 "always";
+      RestartSec = lib.mkOverride 90 "10s";
+      StartLimitIntervalSec = lib.mkOverride 90 0;
+    };
+  };
+in {
+  virtualisation.oci-containers = {
+    containers."${serviceName}-elasticsearch" = {
+      image = "elasticsearch:9.4.2";
+      autoStart = true;
+      volumes = ["${serviceName}_elasticsearch:/usr/share/elasticsearch/data"];
+      environment = {
+        "discovery.type" = "single-node";
+        "xpack.security.enabled" = "false";
+        ES_JAVA_OPTS = "-Xms1g -Xmx1g";
+      };
+      extraOptions = [
+        "--ip=${ipBase}.${toString ipOffset}"
+        "--network=web"
+        "--network-alias=zammad-elasticsearch"
+      ];
+      ports = ["127.0.0.1:${toString elasticsearchPort}:9200"];
+    };
+
+    containers."${serviceName}-memcached" = {
+      image = "memcached:1.6.42-alpine";
+      autoStart = true;
+      cmd = ["memcached" "-m" "256M"];
+      extraOptions = [
+        "--ip=${ipBase}.${toString (ipOffset + 1)}"
+        "--network=web"
+        "--network-alias=zammad-memcached"
+      ];
+    };
+
+    containers."${serviceName}-redis" = {
+      image = "redis:8.8-alpine";
+      autoStart = true;
+      volumes = ["${serviceName}_redis:/data"];
+      extraOptions = [
+        "--ip=${ipBase}.${toString (ipOffset + 2)}"
+        "--network=web"
+        "--network-alias=zammad-redis"
+      ];
+    };
+
+    containers."${serviceName}-railsserver" = {
+      image = zammadImage;
+      autoStart = true;
+      cmd = ["zammad-railsserver"];
+      environment = sharedEnvironment;
+      environmentFiles = [envFileCommon envFileProd];
+      volumes = ["${serviceName}_storage:/opt/zammad/storage"];
+      dependsOn = ["${serviceName}-memcached" "${serviceName}-redis" "${serviceName}-elasticsearch"];
+      extraOptions = [
+        "--ip=${ipBase}.${toString (ipOffset + 4)}"
+        "--network=web"
+        "--add-host=postgres:10.89.0.1"
+        "--network-alias=zammad-railsserver"
+      ];
+    };
+
+    containers."${serviceName}-scheduler" = {
+      image = zammadImage;
+      autoStart = true;
+      cmd = ["zammad-scheduler"];
+      environment = sharedEnvironment;
+      environmentFiles = [envFileCommon envFileProd];
+      volumes = ["${serviceName}_storage:/opt/zammad/storage"];
+      dependsOn = ["${serviceName}-memcached" "${serviceName}-redis"];
+      extraOptions = [
+        "--ip=${ipBase}.${toString (ipOffset + 5)}"
+        "--network=web"
+        "--add-host=postgres:10.89.0.1"
+      ];
+    };
+
+    containers."${serviceName}-websocket" = {
+      image = zammadImage;
+      autoStart = true;
+      cmd = ["zammad-websocket"];
+      environment = sharedEnvironment;
+      environmentFiles = [envFileCommon envFileProd];
+      volumes = ["${serviceName}_storage:/opt/zammad/storage"];
+      dependsOn = ["${serviceName}-memcached" "${serviceName}-redis"];
+      extraOptions = [
+        "--ip=${ipBase}.${toString (ipOffset + 6)}"
+        "--network=web"
+        "--add-host=postgres:10.89.0.1"
+        "--network-alias=zammad-websocket"
+      ];
+    };
+
+    containers."${serviceName}-nginx" = {
+      image = zammadImage;
+      autoStart = true;
+      cmd = ["zammad-nginx"];
+      environment = sharedEnvironment;
+      environmentFiles = [envFileCommon envFileProd];
+      volumes = ["${serviceName}_storage:/opt/zammad/storage"];
+      ports = ["127.0.0.1:${toString servicePort}:8080"];
+      extraOptions = [
+        "--ip=${ipBase}.${toString (ipOffset + 7)}"
+        "--network=web"
+        "--add-host=postgres:10.89.0.1"
+      ];
+    };
+
+    containers."${serviceName}-backup" = {
+      image = zammadImage;
+      autoStart = true;
+      cmd = ["zammad-backup"];
+      environment = sharedEnvironment;
+      environmentFiles = [envFileCommon envFileProd];
+      volumes = [
+        "${serviceName}_storage:/opt/zammad/storage:ro"
+        "/var/backup/${serviceName}:/var/tmp/zammad:rw"
+      ];
+      dependsOn = ["${serviceName}-memcached" "${serviceName}-redis"];
+      extraOptions = [
+        "--ip=${ipBase}.${toString (ipOffset + 8)}"
+        "--network=web"
+        "--add-host=postgres:10.89.0.1"
+        "--user=0:0"
+      ];
+    };
+  };
+
+  systemd.services."podman-${serviceName}-elasticsearch" = alwaysRestart;
+  systemd.services."podman-${serviceName}-memcached" = alwaysRestart;
+  systemd.services."podman-${serviceName}-redis" = alwaysRestart;
+  systemd.services."podman-${serviceName}-railsserver" = alwaysRestart;
+  systemd.services."podman-${serviceName}-scheduler" = alwaysRestart;
+  systemd.services."podman-${serviceName}-websocket" = alwaysRestart;
+
+  systemd.services."podman-${serviceName}-nginx" =
+    alwaysRestart
+    // {
+      after = ["podman-${serviceName}-railsserver.service"];
+      wants = ["podman-${serviceName}-railsserver.service"];
+    };
+
+  # Init als oneshot systemd-Service
+  systemd.services."${serviceName}-init" = {
+    description = "Zammad ${instanceName} Database Initialization";
+    after = [
+      "podman-${serviceName}-memcached.service"
+      "podman-${serviceName}-redis.service"
+      "podman-${serviceName}-elasticsearch.service"
+    ];
+    requires = [
+      "podman-${serviceName}-memcached.service"
+      "podman-${serviceName}-redis.service"
+    ];
+    wantedBy = [];
+
+    serviceConfig = {
+      Type = "oneshot";
+      RemainAfterExit = true;
+      User = "root";
+      Group = "root";
+    };
+
+    script = ''
+      set -euo pipefail
+
+      echo "Starting Zammad ${instanceName} database initialization..."
+
+      ${pkgs.podman}/bin/podman run --rm \
+        --name ${serviceName}-init-oneshot \
+        --network web \
+        --ip ${ipBase}.${toString (ipOffset + 3)} \
+        --add-host=postgres:10.89.0.1 \
+        --user 0:0 \
+        --env-file ${envFileCommon} \
+        --env-file ${envFileProd} \
+        --env MEMCACHE_SERVERS=zammad-memcached:11211 \
+        --env POSTGRESQL_DB=zammad_${instanceName} \
+        --env POSTGRESQL_HOST=10.89.0.1 \
+        --env POSTGRESQL_USER=zammad_${instanceName} \
+        --env POSTGRESQL_PORT=6433 \
+        --env POSTGRESQL_OPTIONS='?pool=50' \
+        --env REDIS_URL=redis://zammad-redis:6379 \
+        --env TZ=Europe/Berlin \
+        --env ELASTICSEARCH_ENABLED=true \
+        --env ELASTICSEARCH_HOST=zammad-elasticsearch \
+        --env ELASTICSEARCH_PORT=9200 \
+        --env ELASTICSEARCH_NAMESPACE=zammad_${instanceName} \
+        --env NGINX_SERVER_SCHEME=https \
+        --env NGINX_SERVER_NAME=${zammadDomain} \
+        --env ZAMMAD_HTTP_TYPE=https \
+        --env ZAMMAD_FQDN=${zammadDomain} \
+        -v ${serviceName}_storage:/opt/zammad/storage \
+        ${zammadImage} \
+        zammad-init
+
+      echo "Zammad ${instanceName} initialization completed successfully"
+    '';
+  };
+
+  # Backup retention service
+  systemd.services."${serviceName}-backup-cleanup" = {
+    serviceConfig = {
+      Type = "oneshot";
+      User = "root";
+      Group = "root";
+    };
+    script = ''
+      set -euo pipefail
+
+      BACKUP_DIR="/var/backup/${serviceName}"
+      HOLD_DAYS=10
+
+      echo "Starting ${serviceName} backup cleanup at $(date)"
+
+      mkdir -p "$BACKUP_DIR"
+      chown root:root "$BACKUP_DIR"
+      chmod 750 "$BACKUP_DIR"
+
+      ${pkgs.findutils}/bin/find "$BACKUP_DIR" -type f -name "*.gz" -mtime +$HOLD_DAYS -delete
+
+      echo "Current backups:"
+      ls -lah "$BACKUP_DIR" || echo "No backups found"
+
+      echo "${serviceName} backup cleanup completed at $(date)"
+    '';
+  };
+
+  systemd.timers."${serviceName}-backup-cleanup" = {
+    wantedBy = ["timers.target"];
+    timerConfig = {
+      OnCalendar = "*-*-* 04:00:00";
+      RandomizedDelaySec = "30m";
+      Persistent = true;
+    };
+  };
+
+  # Traefik configuration with proper headers
+  services.traefik.dynamicConfigOptions.http = {
+    services.${serviceName}.loadBalancer.servers = [
+      {
+        url = "http://localhost:${toString servicePort}/";
+      }
+    ];
+
+    middlewares."${serviceName}-headers".headers = {
+      customRequestHeaders = {
+        X-Forwarded-Proto = "https";
+        X-Forwarded-Port = "443";
+        X-Forwarded-Host = zammadDomain;
+        X-Real-IP = "";
+      };
+    };
+
+    # Permanenter Redirect von der alten Domain auf die neue
+    middlewares."${serviceName}-redirect-old-domain".redirectRegex = {
+      permanent = true;
+      regex = "^https?://${lib.replaceStrings ["."] ["\\."] zammadDomainOld}/(.*)";
+      replacement = "https://${zammadDomain}/\${1}";
+    };
+
+    routers.${serviceName} = {
+      rule = "Host(`${zammadDomain}`)";
+      tls = {
+        certResolver = "ionos";
+      };
+      service = serviceName;
+      entrypoints = "websecure";
+      middlewares = ["${serviceName}-headers"];
+    };
+
+    routers."${serviceName}-old-domain-redirect" = {
+      rule = "Host(`${zammadDomainOld}`)";
+      tls = {
+        certResolver = "ionos";
+      };
+      service = serviceName;
+      entrypoints = "websecure";
+      middlewares = ["${serviceName}-redirect-old-domain"];
+    };
+  };
+}

hosts/AZ-CLD-1/services/default.nix

@@ -0,0 +1,22 @@
+{
+  imports = [
+    ./containers
+    ./monitoring
+
+    ./gitea.nix
+    # ./gotenberg.nix
+    ./metabase.nix
+    ./mysql.nix
+    ./n8n.nix
+    ./netbird.nix
+    ./var-backup-sync.nix
+    ./ntfy.nix
+    ./outline.nix
+    ./pgbouncer.nix
+    ./postgres.nix
+    ./snipe-it.nix
+    ./traefik.nix
+    ./vaultwarden.nix
+    # ./zammad.nix
+  ];
+}

hosts/AZ-CLD-1/services/gitea.nix

@@ -0,0 +1,41 @@
+{config, ...}: let
+  serviceName = "gitea";
+  servicePort = config.m3ta.ports.get serviceName;
+in {
+  services.${serviceName} = {
+    enable = true;
+    settings = {
+      server = {
+        ROOT_URL = "https://git.az-gruppe.com";
+        HTTP_PORT = servicePort;
+      };
+      mailer.SENDMAIL_PATH = "/run/wrappers/bin/sendmail";
+      service.DISABLE_REGISTRATION = true;
+    };
+    lfs.enable = true;
+    dump = {
+      enable = true;
+      type = "tar.gz";
+      interval = "03:30:00";
+      backupDir = "/var/backup/gitea";
+    };
+  };
+
+  # Traefik configuration
+  services.traefik.dynamicConfigOptions.http = {
+    services.${serviceName}.loadBalancer.servers = [
+      {
+        url = "http://localhost:${toString servicePort}/";
+      }
+    ];
+
+    routers.${serviceName} = {
+      rule = "Host(`git.az-gruppe.com`)";
+      tls = {
+        certResolver = "ionos";
+      };
+      service = serviceName;
+      entrypoints = "websecure";
+    };
+  };
+}

hosts/AZ-CLD-1/services/gotenberg.nix

@@ -0,0 +1,10 @@
+{config, ...}: let
+  serviceName = "gotenberg";
+  servicePort = config.m3ta.ports.get serviceName;
+in {
+  services.gotenberg = {
+    enable = true;
+    port = servicePort;
+    bindIP = "127.0.0.1";
+  };
+}

hosts/AZ-CLD-1/services/metabase.nix

@@ -0,0 +1,36 @@
+{
+  config,
+  pkgs,
+  ...
+}: let
+  serviceName = "metabase";
+  servicePort = config.m3ta.ports.get serviceName;
+in {
+  services.${serviceName} = {
+    enable = true;
+    package = pkgs.unstable.metabase;
+    listen.port = servicePort;
+  };
+
+  systemd.services.${serviceName}.serviceConfig = {
+    EnvironmentFile = config.age.secrets.metabase-env.path;
+  };
+
+  # Traefik configuration
+  services.traefik.dynamicConfigOptions.http = {
+    services.${serviceName}.loadBalancer.servers = [
+      {
+        url = "http://localhost:${toString servicePort}/";
+      }
+    ];
+
+    routers.${serviceName} = {
+      rule = "Host(`kpi.az-gruppe.com`)";
+      tls = {
+        certResolver = "ionos";
+      };
+      service = serviceName;
+      entrypoints = "websecure";
+    };
+  };
+}

hosts/AZ-CLD-1/services/monitoring/alloy.nix

@@ -0,0 +1,100 @@
+{
+  config,
+  lib,
+  ...
+}: let
+  lokiPort = config.m3ta.ports.get "loki";
+  alloyPort = config.m3ta.ports.get "alloy";
+  prmNetbirdIP = "100.91.49.26";
+in {
+  services.alloy = {
+    enable = true;
+    configPath = "/etc/alloy";
+    environmentFile = config.age.secrets.monitoring-loki-remote-env.path;
+    extraFlags = [
+      "--server.http.listen-addr=127.0.0.1:${toString alloyPort}"
+      "--disable-reporting"
+    ];
+  };
+
+  environment.etc."alloy/config.alloy".text = ''
+    loki.relabel "journal" {
+      forward_to = []
+
+      rule {
+        source_labels = ["__journal__systemd_unit"]
+        target_label  = "unit"
+      }
+
+      rule {
+        source_labels = ["__journal_priority_keyword"]
+        target_label  = "level"
+      }
+
+      rule {
+        source_labels = ["__journal_syslog_identifier"]
+        target_label  = "syslog_identifier"
+      }
+    }
+
+    loki.source.journal "system" {
+      forward_to    = [loki.process.journal.receiver]
+      relabel_rules = loki.relabel.journal.rules
+      labels        = {
+        host        = "AZ-CLD-1",
+        environment = "prod",
+      }
+    }
+
+    loki.process "journal" {
+      forward_to = [loki.write.prm.receiver]
+
+      stage.json {
+        expressions = {
+          app        = "app",
+          service    = "service",
+          level      = "level",
+          trace_id   = "trace_id",
+          session_id = "session_id",
+          request_id = "request_id",
+          model      = "model",
+          message    = "message",
+        }
+        drop_malformed = false
+      }
+
+      stage.labels {
+        values = {
+          app     = "",
+          service = "",
+          level   = "",
+        }
+      }
+
+      stage.structured_metadata {
+        values = {
+          trace_id   = "",
+          session_id = "",
+          request_id = "",
+          model      = "",
+        }
+      }
+    }
+
+    loki.write "prm" {
+      endpoint {
+        url = "http://${prmNetbirdIP}:${toString lokiPort}/loki/api/v1/push"
+        basic_auth {
+          username = "alloy"
+          password = sys.env("LOKI_BASIC_AUTH_PASSWORD")
+        }
+      }
+    }
+  '';
+
+  systemd.services.alloy = {
+    wants = ["network-online.target"];
+    after = ["network-online.target"];
+    serviceConfig.SupplementaryGroups = lib.mkAfter ["adm"];
+  };
+}

hosts/AZ-CLD-1/services/monitoring/default.nix

@@ -0,0 +1,7 @@
+# Phase 2 — activate by adding `./monitoring` to hosts/AZ-CLD-1/services/default.nix imports.
+{...}: {
+  imports = [
+    ./alloy.nix
+    ./node-exporter.nix
+  ];
+}

hosts/AZ-CLD-1/services/monitoring/node-exporter.nix

@@ -0,0 +1,26 @@
+# Phase 2 — not active until imported.
+# Activate by:
+#   1. Create hosts/AZ-CLD-1/services/monitoring/default.nix with `imports = [ ./node-exporter.nix ];`
+#   2. Add `./monitoring` to hosts/AZ-CLD-1/services/default.nix imports.
+#
+# binds to netbird interface so PRM prometheus can scrape it over VPN.
+{config, ...}: let
+  nodeExporterPort = config.m3ta.ports.get "node-exporter";
+  prmNetbirdIP = "100.91.49.26";
+in {
+  services.prometheus.exporters.node = {
+    enable = true;
+    port = nodeExporterPort;
+    listenAddress = "100.91.203.184"; # CLD netbird IP — overwrite if changed
+    enabledCollectors = [
+      "systemd"
+      "diskstats"
+      "filesystem"
+    ];
+    openFirewall = false;
+  };
+
+  networking.firewall.extraCommands = ''
+    iptables -A INPUT -p tcp -s ${prmNetbirdIP} --dport ${toString nodeExporterPort} -j ACCEPT
+  '';
+}

hosts/AZ-CLD-1/services/mysql.nix

@@ -0,0 +1,42 @@
+{
+  config,
+  pkgs,
+  ...
+}: let
+  mysqlPort = config.m3ta.ports.get "mysql";
+in {
+  services.mysql = {
+    enable = true;
+    package = pkgs.mariadb;
+    settings = {
+      mysqld = {
+        "bind-address" = "127.0.0.1";
+        port = mysqlPort;
+        max_connections = 200;
+        innodb_buffer_pool_size = "1G";
+        "character-set-server" = "utf8mb4";
+        "collation-server" = "utf8mb4_unicode_ci";
+        "skip-name-resolve" = true;
+      };
+      client = {
+        port = mysqlPort;
+        socket = "/run/mysqld/mysqld.sock";
+      };
+      mysqldump = {
+        quick = true;
+        "single-transaction" = true;
+      };
+    };
+  };
+
+  services.mysqlBackup = {
+    enable = true;
+    calendar = "03:40:00";
+    databases = ["snipeit"];
+    singleTransaction = true;
+  };
+
+  networking.firewall.extraCommands = ''
+    iptables -A INPUT -p tcp -s 127.0.0.1 --dport ${toString mysqlPort} -j ACCEPT
+  '';
+}

hosts/AZ-CLD-1/services/n8n.nix

@@ -0,0 +1,44 @@
+{config, ...}: let
+  serviceName = "n8n";
+  servicePort = config.m3ta.ports.get serviceName;
+in {
+  services.${serviceName} = {
+    enable = true;
+    environment = {
+      WEBHOOK_URL = "https://wf.az-gruppe.com";
+      N8N_RUNNERS_ENABLED = true;
+      N8N_NATIVE_PYTHON_RUNNER = true;
+      N8N_RUNNERS_AUTH_TOKEN_FILE = config.age.secrets.n8n-runner-auth-token.path;
+      NODE_FUNCTION_ALLOW_EXTERNAL = "*";
+      N8N_RUNNERS_STDLIB_ALLOW = "*";
+    };
+    taskRunners = {
+      enable = true;
+      environment = {
+        N8N_RUNNERS_STDLIB_ALLOW = "*";
+      };
+    };
+  };
+
+  systemd.services.${serviceName}.serviceConfig = {
+    EnvironmentFile = config.age.secrets.n8n-env.path;
+  };
+
+  # Traefik configuration
+  services.traefik.dynamicConfigOptions.http = {
+    services.${serviceName}.loadBalancer.servers = [
+      {
+        url = "http://localhost:${toString servicePort}/";
+      }
+    ];
+
+    routers.${serviceName} = {
+      rule = "Host(`wf.az-gruppe.com`)";
+      tls = {
+        certResolver = "ionos";
+      };
+      service = serviceName;
+      entrypoints = "websecure";
+    };
+  };
+}

hosts/AZ-CLD-1/services/netbird.nix

@@ -0,0 +1,31 @@
+{pkgs, ...}: {
+  services.netbird = {
+    enable = true;
+    package = pkgs.unstable.netbird;
+  };
+
+  systemd.services.netbird = {
+    environment = {
+      NB_DISABLE_SSH_CONFIG = "true";
+    };
+    path = [
+      pkgs.shadow
+      pkgs.util-linux
+    ];
+  };
+
+  programs.ssh.extraConfig = ''
+    Match exec "${pkgs.netbird}/bin/netbird ssh detect %h %p"
+        PreferredAuthentications password,publickey,keyboard-interactive
+        PasswordAuthentication yes
+        PubkeyAuthentication yes
+        BatchMode no
+        ProxyCommand ${pkgs.netbird}/bin/netbird ssh proxy %h %p
+        StrictHostKeyChecking no
+        UserKnownHostsFile /dev/null
+        CheckHostIP no
+        LogLevel ERROR
+  '';
+
+  networking.firewall.checkReversePath = "loose";
+}

hosts/AZ-CLD-1/services/ntfy.nix

@@ -0,0 +1,32 @@
+{config, ...}: let
+  serviceName = "ntfy-sh";
+  servicePort = config.m3ta.ports.get serviceName;
+in {
+  services.${serviceName} = {
+    enable = true;
+    settings = {
+      base-url = "https://ping.az-gruppe.com";
+      listen-http = ":${toString servicePort}";
+      auth-file = "/var/lib/ntfy-sh/user.db";
+      auth-default-access = "deny-all";
+    };
+  };
+
+  # Traefik configuration
+  services.traefik.dynamicConfigOptions.http = {
+    services.${serviceName}.loadBalancer.servers = [
+      {
+        url = "http://localhost:${toString servicePort}/";
+      }
+    ];
+
+    routers.${serviceName} = {
+      rule = "Host(`ping.az-gruppe.com`)";
+      tls = {
+        certResolver = "ionos";
+      };
+      service = serviceName;
+      entrypoints = "websecure";
+    };
+  };
+}

hosts/AZ-CLD-1/services/outline.nix

@@ -0,0 +1,41 @@
+{config, ...}: let
+  serviceName = "outline";
+  servicePort = config.m3ta.ports.get serviceName;
+in {
+  services.${serviceName} = {
+    enable = true;
+    port = servicePort;
+    publicUrl = "https://wiki.az-gruppe.com";
+    databaseUrl = "postgresql://outline:outline@127.0.0.1:5432/outline";
+    storage = {
+      storageType = "s3";
+      region = "eu-central";
+      uploadBucketUrl = "https://nbg1.your-objectstorage.com";
+      uploadBucketName = "az-wiki";
+      secretKeyFile = config.age.secrets.hetzner-s3-az-intern-secret-key.path;
+      accessKey = "CRT7V4HR5CG9NHICD2WW";
+    };
+  };
+
+  systemd.services.${serviceName}.serviceConfig = {
+    EnvironmentFile = config.age.secrets.outline-env.path;
+  };
+
+  # Traefik configuration
+  services.traefik.dynamicConfigOptions.http = {
+    services.${serviceName}.loadBalancer.servers = [
+      {
+        url = "http://localhost:${toString servicePort}/";
+      }
+    ];
+
+    routers.${serviceName} = {
+      rule = "Host(`wiki.az-gruppe.com`)";
+      tls = {
+        certResolver = "ionos";
+      };
+      service = serviceName;
+      entrypoints = "websecure";
+    };
+  };
+}

hosts/AZ-CLD-1/services/pgbouncer.nix

@@ -0,0 +1,156 @@
+{
+  config,
+  lib,
+  pkgs,
+  ...
+}: let
+  txPort = config.m3ta.ports.get "pgbouncer-tx";
+  sessionPort = config.m3ta.ports.get "pgbouncer-session";
+  pgPort = config.m3ta.ports.get "postgres";
+
+  # Podman-Bridge-IP des Hosts
+  hostBridgeIP = "10.89.0.1";
+
+  # pgbouncer (1.24+ als auch 1.25.2) lädt bei JEDEM TLS-Setup eine CA — auch
+  # bei sslmode=prefer/require ohne Verifikation. Ohne explizite Datei fällt
+  # es auf die OpenSSL-Default-Verify-Paths zurück, die hier fehlschlagen:
+  #   "TLS setup failed: failed to load CA: (null)"  → Startabbruch.
+  # Explizites Bundle umgeht das (verifiziert 2026-08-20 gegen 1.24.1 + 1.25.2).
+  caBundle = "${pkgs.cacert}/etc/ssl/certs/ca-bundle.crt";
+
+  commonSettings = {
+    listen_addr = "127.0.0.1,${hostBridgeIP}";
+    auth_type = "scram-sha-256";
+    auth_file = config.age.secrets.pgbouncer-userlist.path;
+
+    admin_users = "sascha_koenig";
+    stats_users = "sascha_koenig";
+
+    log_connections = 1;
+    log_disconnections = 1;
+
+    client_tls_sslmode = "prefer";
+    client_tls_cert_file = config.age.secrets.pgbouncer-tls-cert.path;
+    client_tls_key_file = config.age.secrets.pgbouncer-tls-key.path;
+    client_tls_protocols = "secure";
+    client_tls_ca_file = caBundle;
+
+    server_reset_query = "DISCARD ALL";
+    server_check_query = "SELECT 1";
+    server_check_delay = 30;
+
+    server_tls_sslmode = "prefer";
+    server_tls_protocols = "secure";
+    server_tls_ca_file = caBundle;
+  };
+
+  mkDatabases = dbs:
+    lib.listToAttrs (map (db:
+      lib.nameValuePair db "host=127.0.0.1 port=${toString pgPort} dbname=${db}")
+    dbs);
+in {
+  services.pgbouncer = {
+    enable = true;
+
+    settings = {
+      pgbouncer =
+        commonSettings
+        // {
+          listen_port = txPort;
+          pool_mode = "transaction";
+
+          max_client_conn = 1000;
+          default_pool_size = 10;
+          min_pool_size = 2;
+          reserve_pool_size = 3;
+          reserve_pool_timeout = 3;
+
+          max_prepared_statements = 200;
+        };
+      databases = mkDatabases [
+        "baserow"
+        "litellm"
+        "librechat_rag"
+        "librechat_rag_dev"
+        "metabase"
+        "az_kpi_raw"
+      ];
+    };
+  };
+
+  systemd.services.pgbouncer = {
+    after = ["postgresql.service"];
+    requires = ["postgresql.service"];
+  };
+
+  environment.etc."pgbouncer/pgbouncer-session.ini".text = let
+    dbLines =
+      lib.concatStringsSep "\n"
+      (lib.mapAttrsToList (name: conn: "${name} = ${conn}")
+        (mkDatabases [
+          "outline"
+          "zammad"
+          "zammad_hr"
+          "vaultwarden"
+          "dash"
+        ]));
+  in ''
+    [databases]
+    ${dbLines}
+
+    [pgbouncer]
+    listen_addr = ${commonSettings.listen_addr}
+    listen_port = ${toString sessionPort}
+    pool_mode = session
+    auth_type = ${commonSettings.auth_type}
+    auth_file = ${config.age.secrets.pgbouncer-userlist.path}
+    admin_users = ${commonSettings.admin_users}
+    stats_users = ${commonSettings.stats_users}
+
+    client_tls_sslmode = ${commonSettings.client_tls_sslmode}
+    client_tls_cert_file = ${commonSettings.client_tls_cert_file}
+    client_tls_key_file = ${commonSettings.client_tls_key_file}
+    client_tls_protocols = ${commonSettings.client_tls_protocols}
+    client_tls_ca_file = ${commonSettings.client_tls_ca_file}
+
+    max_client_conn = 300
+    default_pool_size = 30
+    min_pool_size = 1
+    query_wait_timeout = 30
+
+    log_connections = 1
+    log_disconnections = 1
+    server_reset_query = DISCARD ALL
+    server_check_query = SELECT 1
+    server_check_delay = 30
+    server_tls_sslmode = ${commonSettings.server_tls_sslmode}
+    server_tls_protocols = ${commonSettings.server_tls_protocols}
+    server_tls_ca_file = ${commonSettings.server_tls_ca_file}
+
+    # PID/Socket getrennt von der Haupt-Instanz halten.
+    pidfile = /run/pgbouncer-session/pgbouncer.pid
+    unix_socket_dir = /run/pgbouncer-session
+  '';
+
+  systemd.services.pgbouncer-session = {
+    description = "PgBouncer (session pool) for prepared-statement apps";
+    after = ["postgresql.service"];
+    requires = ["postgresql.service"];
+    wantedBy = ["multi-user.target"];
+    serviceConfig = {
+      User = "pgbouncer";
+      Group = "pgbouncer";
+      RuntimeDirectory = "pgbouncer-session";
+      ExecStart = "${pkgs.pgbouncer}/bin/pgbouncer /etc/pgbouncer/pgbouncer-session.ini";
+      Restart = "on-failure";
+      RestartSec = 5;
+    };
+  };
+
+  networking.firewall.extraCommands = ''
+    iptables -A INPUT -p tcp -s 127.0.0.1     --dport ${toString txPort}      -j ACCEPT
+    iptables -A INPUT -p tcp -s 10.89.0.0/24  --dport ${toString txPort}      -j ACCEPT
+    iptables -A INPUT -p tcp -s 127.0.0.1     --dport ${toString sessionPort} -j ACCEPT
+    iptables -A INPUT -p tcp -s 10.89.0.0/24  --dport ${toString sessionPort} -j ACCEPT
+  '';
+}

hosts/AZ-CLD-1/services/postgres.nix

@@ -0,0 +1,190 @@
+{
+  config,
+  pkgs,
+  ...
+}: let
+  serviceName = "pgadmin";
+  pgadminPort = config.m3ta.ports.get serviceName;
+in {
+  services.postgresql = {
+    enable = true;
+    enableTCPIP = true;
+    package = pkgs.postgresql_17;
+    settings = {
+      ssl = true;
+      max_connections = 100;
+      shared_buffers = "4GB";
+      work_mem = "8MB";
+      superuser_reserved_connections = 5;
+
+      idle_in_transaction_session_timeout = "10min";
+      idle_session_timeout = "2h";
+
+      tcp_keepalives_idle = 60;
+      tcp_keepalives_interval = 10;
+      tcp_keepalives_count = 6;
+
+      deadlock_timeout = "1s";
+
+      authentication_timeout = "30s";
+
+      log_connections = true;
+      log_disconnections = true;
+      log_lock_waits = true;
+    };
+    extensions = with pkgs.postgresql17Packages; [
+      pgvector
+    ];
+    initialScript = pkgs.writeText "backend-initScript" ''
+      CREATE USER baserow WITH ENCRYPTED PASSWORD 'baserow';
+        CREATE DATABASE baserow;
+        ALTER DATABASE baserow OWNER to baserow;
+        ALTER DATABASE baserow CONNECTION LIMIT 60;
+
+        CREATE USER kestra WITH ENCRYPTED PASSWORD 'kestra';
+        CREATE DATABASE kestra;
+        ALTER DATABASE kestra OWNER to kestra;
+        ALTER DATABASE kestra CONNECTION LIMIT 10;
+
+        CREATE USER librechat_rag WITH ENCRYPTED PASSWORD 'librechat_rag';
+        CREATE DATABASE librechat_rag;
+        ALTER DATABASE librechat_rag OWNER to librechat_rag;
+        ALTER DATABASE librechat_rag CONNECTION LIMIT 20;
+
+        CREATE USER librechat_rag_dev WITH ENCRYPTED PASSWORD 'librechat_rag_dev';
+        CREATE DATABASE librechat_rag_dev;
+        ALTER DATABASE librechat_rag_dev OWNER to librechat_rag_dev;
+        ALTER DATABASE librechat_rag_dev CONNECTION LIMIT 10;
+
+        CREATE USER metabase WITH ENCRYPTED PASSWORD 'metabase';
+        CREATE DATABASE metabase;
+        ALTER DATABASE metabase OWNER to metabase;
+        ALTER DATABASE metabase CONNECTION LIMIT 15;
+
+        CREATE USER n8n WITH ENCRYPTED PASSWORD 'n8n';
+        CREATE DATABASE n8n;
+        ALTER DATABASE n8n OWNER to n8n;
+        ALTER DATABASE n8n CONNECTION LIMIT 5;
+
+        CREATE USER outline WITH ENCRYPTED PASSWORD 'outline';
+        CREATE DATABASE outline;
+        ALTER DATABASE outline OWNER to outline;
+        ALTER DATABASE outline CONNECTION LIMIT 5;
+
+        CREATE USER vaultwarden WITH ENCRYPTED PASSWORD 'vaultwarden';
+        CREATE DATABASE vaultwarden;
+        ALTER DATABASE vaultwarden OWNER to vaultwarden;
+        ALTER DATABASE vaultwarden CONNECTION LIMIT 20;
+
+        CREATE USER zammad-hr WITH ENCRYPTED PASSWORD 'zammad-hr';
+        CREATE DATABASE zammad-hr;
+        ALTER DATABASE zammad-hr OWNER to zammad-hr;
+        ALTER DATABASE zammad-hr CONNECTION LIMIT 50;
+
+        -- Group roles (NOLOGIN, for permission management)
+        CREATE ROLE admin NOLOGIN;
+        CREATE ROLE dba NOLOGIN;
+
+        -- Personal login roles
+        CREATE USER sascha_koenig WITH ENCRYPTED PASSWORD 'sascha_koenig';
+        GRANT admin TO sascha_koenig;
+
+        CREATE USER jannik_mueller WITH ENCRYPTED PASSWORD 'jannik_mueller';
+        GRANT admin TO jannik_mueller;
+    '';
+    authentication = pkgs.lib.mkOverride 10 ''
+      # Local connections (Unix socket)
+      local   all                 postgres            peer
+      local   all                 sascha_koenig       scram-sha-256
+      local   all                 jannik_mueller      scram-sha-256
+      local   az_test             az_test             scram-sha-256
+      local   metabase,az_kpi_raw metabase            scram-sha-256
+      local   all                 n8n                 scram-sha-256
+      local   outline             outline             scram-sha-256
+      local   vaultwarden         vaultwarden         scram-sha-256
+      local   zammad              zammad              scram-sha-256
+
+      # Localhost connections (IPv4 and IPv6)
+      host    all                 postgres            127.0.0.1/32       scram-sha-256
+      host    all                 postgres            ::1/128            scram-sha-256
+
+      host    all                 sascha_koenig       127.0.0.1/32       scram-sha-256
+      host    all                 sascha_koenig       ::1/128            scram-sha-256
+
+      host    all                 jannik_mueller      127.0.0.1/32       scram-sha-256
+      host    all                 jannik_mueller      ::1/128            scram-sha-256
+
+      host    az_test             az_test             127.0.0.1/32       scram-sha-256
+      host    az_test             az_test             ::1/128            scram-sha-256
+
+      host    baserow             baserow             127.0.0.1/32       scram-sha-256
+      host    baserow             baserow             ::1/128            scram-sha-256
+
+      host    librechat_rag       librechat_rag       127.0.0.1/32       scram-sha-256
+      host    librechat_rag       librechat_rag       ::1/128            scram-sha-256
+      host    librechat_rag_dev   librechat_rag_dev   127.0.0.1/32       scram-sha-256
+      host    librechat_rag_dev   librechat_rag_dev   ::1/128            scram-sha-256
+
+      host    litellm             litellm             127.0.0.1/32       scram-sha-256
+      host    litellm             litellm             ::1/128            scram-sha-256
+
+      host    outline             outline             127.0.0.1/32       scram-sha-256
+      host    outline             outline             ::1/128            scram-sha-256
+
+      host    metabase,az_kpi_raw metabase            127.0.0.1/32       scram-sha-256
+      host    metabase,az_kpi_raw metabase            ::1/128            scram-sha-256
+
+      host    all                 n8n                 127.0.0.1/32       scram-sha-256
+      host    all                 n8n                 ::1/128            scram-sha-256
+
+      host    vaultwarden         vaultwarden         127.0.0.1/32       scram-sha-256
+      host    vaultwarden         vaultwarden         ::1/128            scram-sha-256
+
+      host    zammad_hr           zammad_hr           127.0.0.1/32       scram-sha-256
+      host    zammad_hr           zammad_hr           ::1/128            scram-sha-256
+
+      # Podman network connections for Baserow
+      # host    baserow             baserow             10.89.0.0/24       scram-sha-256
+      host    kestra              kestra              10.89.0.0/24       scram-sha-256
+      host    litellm             litellm             10.89.0.0/24       scram-sha-256
+      host    netbird             netbird             10.89.0.0/24       scram-sha-256
+
+      # Netbird network connections
+      host    az_kpi_raw          kestra_prm          100.91.49.26/32    scram-sha-256
+
+      # Deny all other connections
+      local   all                 all                 reject
+      host    all                 all                 0.0.0.0/0          reject
+      host    all                 all                 ::/0               reject
+    '';
+  };
+  services.postgresqlBackup = {
+    enable = true;
+    startAt = "03:10:00";
+    databases = ["az_kpi_raw" "baserow" "kestra" "librechat_rag" "litellm" "metabase" "n8n" "outline" "vaultwarden" "zammad" "zammad_hr"];
+  };
+  services.pgadmin = {
+    enable = true;
+    initialPasswordFile = "${config.age.secrets.pgadmin-pw.path}";
+    initialEmail = "sascha.koenig@azintec.com";
+  };
+
+  # Traefik configuration specific to pgadmin
+  services.traefik.dynamicConfigOptions.http = {
+    services.pgadmin.loadBalancer.servers = [{url = "http://localhost:${toString pgadminPort}/";}];
+    routers.pgadmin = {
+      rule = "Host(`pg.az-gruppe.com`)";
+      tls.certResolver = "ionos";
+      service = "pgadmin";
+      entrypoints = "websecure";
+    };
+  };
+  networking.firewall = {
+    extraCommands = ''
+      iptables -A INPUT -p tcp -s 127.0.0.1 --dport 5432 -j ACCEPT
+      iptables -A INPUT -p tcp -s 10.89.0.0/24 --dport 5432 -j ACCEPT
+      iptables -A INPUT -p tcp -s 10.89.1.0/24 --dport 5432 -j ACCEPT
+      iptables -A INPUT -p tcp -s 100.91.49.26/32 --dport 5432 -j ACCEPT
+    '';
+  };
+}

hosts/AZ-CLD-1/services/snipe-it.nix

@@ -0,0 +1,95 @@
+{
+  config,
+  lib,
+  pkgs,
+  ...
+}: let
+  serviceName = "snipe-it";
+  servicePort = config.m3ta.ports.get serviceName;
+  mysqlPort = config.m3ta.ports.get "mysql";
+  hostName = "am.az-gruppe.com";
+
+  restoreDefaults = pkgs.writeShellScript "snipe-it-restore-defaults" ''
+    set -euo pipefail
+    src="${pkgs.snipe-it}/share/snipe-it/uploads"
+    dst="/var/lib/snipe-it/public/uploads"
+
+    if [ -d "$src" ] && [ -d "$dst" ]; then
+      # -n = kein Überschreiben existierender Dateien (User-Uploads bleiben erhalten)
+      cp -rn "$src/." "$dst/"
+    fi
+  '';
+in {
+  services.${serviceName} = {
+    enable = true;
+    inherit hostName;
+    appURL = "https://${hostName}";
+    appKeyFile = config.age.secrets.snipe-it-app-key.path;
+
+    database = {
+      host = "127.0.0.1";
+      port = mysqlPort;
+      name = "snipeit";
+      user = "snipeit";
+      passwordFile = config.age.secrets.snipe-it-db-password.path;
+      createLocally = false;
+    };
+
+    mail = {
+      driver = "smtp";
+      host = "smtp.eu.mailgun.org";
+      port = 587;
+      encryption = "tls";
+      user = "bot@az-gruppe.com";
+      passwordFile = config.age.secrets.snipe-it-mail-password.path;
+      from = {
+        name = "AZ - Asset Management";
+        address = "bot@az-gruppe.com";
+      };
+      replyTo = {
+        name = "Snipe-IT Asset Management";
+        address = "bot@az-gruppe.com";
+      };
+      backupNotificationAddress = "sascha.koenig@azintec.com";
+    };
+
+    nginx.listen = [
+      {
+        addr = "127.0.0.1";
+        port = servicePort;
+      }
+    ];
+
+    config = {
+      APP_TRUSTED_PROXIES = "127.0.0.1";
+      SECURE_COOKIES = lib.mkForce true;
+    };
+  };
+
+  systemd.services.snipe-it-setup = {
+    after = ["mysql.service"];
+    requires = ["mysql.service"];
+    unitConfig.ConditionPathExists = "${config.services.mysql.dataDir}/snipeit";
+    serviceConfig = {
+      ExecStartPre = ["+${restoreDefaults}"];
+    };
+  };
+
+  # Traefik configuration
+  services.traefik.dynamicConfigOptions.http = {
+    services.${serviceName}.loadBalancer.servers = [
+      {
+        url = "http://localhost:${toString servicePort}/";
+      }
+    ];
+
+    routers.${serviceName} = {
+      rule = "Host(`${hostName}`)";
+      tls = {
+        certResolver = "ionos";
+      };
+      service = serviceName;
+      entrypoints = "websecure";
+    };
+  };
+}

hosts/AZ-CLD-1/services/traefik.nix

@@ -0,0 +1,77 @@
+{config, ...}: let
+  httpPort = config.m3ta.ports.get "traefik";
+  httpsPort = config.m3ta.ports.get "traefik-ssl";
+in {
+  services.traefik = {
+    enable = true;
+    staticConfigOptions = {
+      log = {level = "WARN";};
+      certificatesResolvers = {
+        ionos = {
+          acme = {
+            email = "sascha.koenig@azintec.com";
+            storage = "/var/lib/traefik/acme.json";
+            caserver = "https://acme-v02.api.letsencrypt.org/directory";
+            dnsChallenge = {
+              provider = "ionos";
+              resolvers = ["1.1.1.1:53" "8.8.8.8:53"];
+              propagation = {
+                delayBeforeChecks = 60;
+                disableChecks = true;
+              };
+            };
+          };
+        };
+      };
+      api = {};
+      entryPoints = {
+        web = {
+          address = ":${toString httpPort}";
+          http.redirections.entryPoint = {
+            to = "websecure";
+            scheme = "https";
+          };
+        };
+        websecure = {
+          address = ":${toString httpsPort}";
+        };
+      };
+    };
+    dynamicConfigOptions = {
+      http = {
+        services = {
+          dummy = {
+            loadBalancer.servers = [
+              {url = "http://192.168.0.1";} # Diese URL wird nie verwendet
+            ];
+          };
+        };
+        middlewares = {
+          auth = {
+            basicAuth = {
+              users = ["sascha.koenig:$apr1$1xqdta2b$DIVNvvp5iTUGNccJjguKh."];
+            };
+          };
+        };
+
+        routers = {
+          api = {
+            rule = "Host(`r.az-gruppe.com`)";
+            service = "api@internal";
+            middlewares = ["auth"];
+            entrypoints = ["websecure"];
+            tls = {
+              certResolver = "ionos";
+            };
+          };
+        };
+      };
+    };
+  };
+
+  systemd.services.traefik.serviceConfig = {
+    EnvironmentFile = ["${config.age.secrets.traefik-env.path}"];
+  };
+
+  networking.firewall.allowedTCPPorts = [httpPort httpsPort];
+}

hosts/AZ-CLD-1/services/var-backup-sync.nix

@@ -0,0 +1,93 @@
+{
+  config,
+  pkgs,
+  ...
+}: let
+  serviceName = "cld-var-backup-sync";
+  sourceDir = "/var/backup/";
+  targetHost = "100.91.49.26";
+  targetPort = "2022";
+  targetUser = "backup-ingest";
+  targetDir = "/srv/veeam-ingest/AZ-CLD-1/var-backup/";
+  sshKey = config.age.secrets.cld-var-backup-sync-ssh-key.path;
+in {
+  environment.systemPackages = with pkgs; [
+    rsync
+    openssh
+  ];
+
+  systemd.tmpfiles.rules = [
+    "d /var/lib/${serviceName} 0700 root root - -"
+  ];
+
+  systemd.services.${serviceName} = {
+    description = "Mirror /var/backup from AZ-CLD-1 to AZ-PRM-1 for VEEAM";
+    after = ["network-online.target" "netbird.service"];
+    wants = ["network-online.target" "netbird.service"];
+
+    path = with pkgs; [
+      coreutils
+      findutils
+      openssh
+      rsync
+      util-linux
+    ];
+
+    serviceConfig = {
+      Type = "oneshot";
+      User = "root";
+      Group = "root";
+      RuntimeDirectory = serviceName;
+      RuntimeDirectoryMode = "0700";
+      LockPersonality = true;
+      NoNewPrivileges = true;
+      PrivateTmp = true;
+    };
+
+    script = ''
+      set -euo pipefail
+
+      lock_file="/run/${serviceName}/${serviceName}.lock"
+
+      if ! ssh-keygen -y -f ${sshKey} >/dev/null; then
+        echo "Invalid SSH private key secret at ${sshKey}" >&2
+        exit 1
+      fi
+
+      (
+        flock -n 9
+
+        rsync \
+          -aH \
+          --no-owner \
+          --no-group \
+          --no-devices \
+          --no-specials \
+          --numeric-ids \
+          --delete \
+          --partial \
+          --delay-updates \
+          --human-readable \
+          --stats \
+          -e "ssh -i ${sshKey} -p ${targetPort} -o IdentitiesOnly=yes -o BatchMode=yes -o StrictHostKeyChecking=accept-new -o UserKnownHostsFile=/var/lib/${serviceName}/known_hosts" \
+          ${sourceDir} \
+          ${targetUser}@${targetHost}:${targetDir}
+
+        echo "Removing /var/backup files older than 7 days"
+        find ${sourceDir} -type f -mtime +7 -print -delete
+
+        echo "Removing empty directories under /var/backup"
+        find ${sourceDir} -mindepth 1 -depth -type d -empty -print -delete
+      ) 9>"$lock_file"
+    '';
+  };
+
+  systemd.timers.${serviceName} = {
+    wantedBy = ["timers.target"];
+    timerConfig = {
+      OnCalendar = "*-*-* 05:00:00";
+      Persistent = true;
+      Unit = "${serviceName}.service";
+    };
+  };
+}

hosts/AZ-CLD-1/services/vaultwarden.nix

@@ -0,0 +1,32 @@
+{config, ...}: let
+  serviceName = "vaultwarden";
+  servicePort = config.m3ta.ports.get serviceName;
+in {
+  services.${serviceName} = {
+    enable = true;
+    dbBackend = "postgresql";
+    config = {
+      ROCKET_ADDRESS = "127.0.0.1";
+      ROCKET_PORT = servicePort;
+    };
+    environmentFile = config.age.secrets.vaultwarden-env.path;
+  };
+
+  # Traefik configuration
+  services.traefik.dynamicConfigOptions.http = {
+    services.${serviceName}.loadBalancer.servers = [
+      {
+        url = "http://localhost:${toString servicePort}/";
+      }
+    ];
+
+    routers.${serviceName} = {
+      rule = "Host(`pw.az-gruppe.com`)";
+      tls = {
+        certResolver = "ionos";
+      };
+      service = serviceName;
+      entrypoints = "websecure";
+    };
+  };
+}

hosts/AZ-CLD-1/services/zammad.nix

@@ -0,0 +1,35 @@
+{config, ...}: let
+  serviceName = "zammad";
+  servicePort = config.m3ta.ports.get serviceName;
+in {
+  services.${serviceName} = {
+    enable = true;
+    openPorts = false;
+    port = servicePort;
+    secretKeyBaseFile = config.age.secrets.zammad-secret.path;
+    database = {
+      createLocally = false;
+      port = 5432;
+      host = "127.0.0.1";
+      passwordFile = config.age.secrets.zammad-pw.path;
+    };
+  };
+
+  # Traefik configuration
+  services.traefik.dynamicConfigOptions.http = {
+    services.${serviceName}.loadBalancer.servers = [
+      {
+        url = "http://localhost:${toString servicePort}/";
+      }
+    ];
+
+    routers.${serviceName} = {
+      rule = "Host(`help.az-gruppe.com`)";
+      tls = {
+        certResolver = "ionos";
+      };
+      service = serviceName;
+      entrypoints = "websecure";
+    };
+  };
+}

hosts/AZ-PRM-1/configuration.nix

@@ -0,0 +1,121 @@
+{
+  config,
+  pkgs,
+  ...
+}: {
+  imports = [
+    ./hardware-configuration.nix
+    ./disko-config.nix
+  ];
+
+  boot.loader.systemd-boot.enable = true;
+  boot.loader.efi.canTouchEfiVariables = true;
+
+  swapDevices = [
+    {
+      device = "/var/lib/swapfile";
+      size = 16 * 1024;
+    }
+  ];
+
+  networking.hostName = "AZ-PRM-1";
+  networking.networkmanager.enable = true;
+
+  time.timeZone = "Europe/Berlin";
+
+  i18n.defaultLocale = "de_DE.UTF-8";
+
+  environment.systemPackages = with pkgs; [
+    neovim
+    ghostty
+    git
+    python3
+    python3Packages.pysmb
+  ];
+
+  programs.gnupg.agent = {
+    enable = true;
+    enableSSHSupport = true;
+  };
+
+  services.openssh = {
+    enable = true;
+    ports = [2022];
+    settings = {
+      PermitRootLogin = "no";
+      PasswordAuthentication = false;
+    };
+  };
+
+  networking.firewall.allowedTCPPorts = [587];
+
+  fileSystems."/mnt/AutoAblage" = {
+    device = "//192.168.152.97/AutoAblage";
+    fsType = "cifs";
+    options = [
+      "credentials=${config.age.secrets.smb-autoablage.path}"
+      "domain=az-group.local"
+      "uid=0"
+      "gid=0"
+      "file_mode=0777"
+      "dir_mode=0777"
+      "iocharset=utf8"
+      "nofail"
+      "x-systemd.automount"
+      "x-systemd.idle-timeout=60"
+    ];
+  };
+
+  fileSystems."/mnt/SkriptHelper" = {
+    device = "//192.168.152.98/SkriptHelper";
+    fsType = "cifs";
+    options = [
+      "credentials=${config.age.secrets.smb-autoablage.path}"
+      "domain=az-group.local"
+      "uid=0"
+      "gid=0"
+      "file_mode=0777"
+      "dir_mode=0777"
+      "iocharset=utf8"
+      "nofail"
+      "x-systemd.automount"
+      "x-systemd.idle-timeout=60"
+    ];
+  };
+
+  fileSystems."/mnt/DMS-ALT-INBOX" = {
+    device = "//192.168.152.104/01-E-RECHNUNG-DMS-ALT";
+    fsType = "cifs";
+    options = [
+      "credentials=${config.age.secrets.smb-autoablage.path}"
+      "domain=az-group.local"
+      "uid=0"
+      "gid=0"
+      "file_mode=0777"
+      "dir_mode=0777"
+      "iocharset=utf8"
+      "nofail"
+      "x-systemd.automount"
+      "x-systemd.idle-timeout=60"
+    ];
+  };
+
+  fileSystems."/mnt/DMS-INBOX" = {
+    device = "//192.168.152.97/01-E-RECHNUNG-DMS";
+    fsType = "cifs";
+    options = [
+      "credentials=${config.age.secrets.smb-autoablage.path}"
+      "domain=az-group.local"
+      "uid=0"
+      "gid=0"
+      "file_mode=0777"
+      "dir_mode=0777"
+      "iocharset=utf8"
+      "nofail"
+      "x-systemd.automount"
+      "x-systemd.idle-timeout=60"
+    ];
+  };
+
+  system.stateVersion = "25.05";
+}

hosts/AZ-PRM-1/default.nix

@@ -0,0 +1,12 @@
+{
+  imports = [
+    ../common
+    ./configuration.nix
+    ./secrets.nix
+    ./services
+  ];
+
+  extraServices = {
+    podman.enable = true;
+  };
+}

hosts/AZ-PRM-1/disko-config.nix

@@ -0,0 +1,34 @@
+{
+  disko.devices = {
+    disk = {
+      main = {
+        type = "disk";
+        device = "/dev/sda";
+        content = {
+          type = "gpt";
+          partitions = {
+            esp = {
+              size = "512M";
+              type = "EF00";
+              content = {
+                type = "filesystem";
+                format = "vfat";
+                mountpoint = "/boot";
+                mountOptions = ["defaults" "umask=0077"];
+              };
+            };
+            root = {
+              size = "100%";
+              content = {
+                type = "filesystem";
+                format = "ext4";
+                mountpoint = "/";
+                mountOptions = ["noatime" "nodiratime" "discard"];
+              };
+            };
+          };
+        };
+      };
+    };
+  };
+}

hosts/AZ-PRM-1/hardware-configuration.nix

@@ -0,0 +1,17 @@
+{
+  config,
+  lib,
+  pkgs,
+  ...
+}: {
+  virtualisation.hypervGuest.enable = true;
+
+  boot.initrd.availableKernelModules = ["sd_mod" "sr_mod" "hv_storvsc"];
+  boot.initrd.kernelModules = [];
+  boot.kernelModules = [];
+  boot.extraModulePackages = [];
+
+  networking.useDHCP = lib.mkDefault true;
+
+  nixpkgs.hostPlatform = lib.mkDefault "x86_64-linux";
+}

hosts/AZ-PRM-1/secrets.nix

@@ -0,0 +1,109 @@
+{lib, ...}: let
+  ntfyGrafanaWebhookSecret = ../../secrets/ntfy-grafana-webhook.age;
+  litellmPrometheusBearerSecret = ../../secrets/litellm-prometheus-bearer.age;
+in {
+  age = {
+    secrets =
+      {
+        atrocore-env = {
+          file = ../../secrets/atrocore-env.age;
+          owner = "postgres";
+          group = "postgres";
+          mode = "0400";
+        };
+        atrocore-registry-token = {
+          file = ../../secrets/atrocore-registry-token.age;
+        };
+        azion-env = {
+          file = ../../secrets/azion-env.age;
+        };
+        grafana-admin-pw = {
+          file = ../../secrets/grafana-admin-pw.age;
+          owner = "grafana";
+        };
+        grafana-db-password = {
+          file = ../../secrets/grafana-db-password.age;
+          owner = "grafana";
+        };
+        grafana-secret-key = {
+          file = ../../secrets/grafana-secret-key.age;
+          owner = "grafana";
+        };
+        monitoring-loki-htpasswd = {
+          file = ../../secrets/monitoring-loki-htpasswd.age;
+          owner = "traefik";
+          mode = "0400";
+        };
+        netbox-secret-key = {
+          file = ../../secrets/netbox-secret-key.age;
+          owner = "netbox";
+          mode = "0400";
+        };
+        netbox-api-token-pepper = {
+          file = ../../secrets/netbox-api-token-pepper.age;
+          owner = "netbox";
+          mode = "0400";
+        };
+        traefik-env = {
+          file = ../../secrets/traefik-env.age;
+        };
+        kestra-config = {
+          file = ../../secrets/kestra-config.age;
+          mode = "644";
+        };
+        kestra-env = {file = ../../secrets/kestra-env.age;};
+        kestra-secrets = {file = ../../secrets/kestra-secrets.age;};
+        n8n-env = {
+          file = ../../secrets/n8n-env-prm.age;
+        };
+        n8n-runner-auth-token = {
+          file = ../../secrets/n8n-runner-auth-token-prm.age;
+        };
+        n8n-sandbox-env = {
+          file = ../../secrets/n8n-sandbox-env-prm.age;
+        };
+        pgadmin-pw = {
+          file = ../../secrets/pgadmin-pw.age;
+          owner = "pgadmin";
+        };
+        semaphore-env = {
+          file = ../../secrets/semaphore-env.age;
+          owner = "postgres";
+          group = "postgres";
+          mode = "0400";
+        };
+        pg-cert = {
+          file = ../../secrets/server.crt.age;
+          owner = "postgres";
+          group = "postgres";
+          mode = "0644";
+        };
+        pg-key = {
+          file = ../../secrets/server.key.age;
+          owner = "postgres";
+          group = "postgres";
+          mode = "0600";
+        };
+        phishboard-env = {
+          file = ../../secrets/phishboard-env.age;
+        };
+        smb-autoablage = {
+          file = ../../secrets/smb-autoablage.age;
+        };
+      }
+      // lib.optionalAttrs (builtins.pathExists ntfyGrafanaWebhookSecret) {
+        ntfy-grafana-webhook = {
+          file = ntfyGrafanaWebhookSecret;
+          mode = "0400";
+        };
+      }
+      // lib.optionalAttrs (builtins.pathExists litellmPrometheusBearerSecret) {
+        litellm-prometheus-bearer = {
+          file = litellmPrometheusBearerSecret;
+          owner = "prometheus";
+          group = "prometheus";
+          mode = "0400";
+        };
+      };
+  };
+}

hosts/AZ-PRM-1/services/azess.nix

@@ -0,0 +1,26 @@
+{config, ...}: let
+  serviceName = "azess";
+  servicePort = config.m3ta.ports.get serviceName;
+in {
+  services.azess = {
+    enable = true;
+    host = "127.0.0.1";
+    port = servicePort;
+    workers = 4;
+  };
+
+  services.traefik.dynamicConfigOptions.http = {
+    services.${serviceName}.loadBalancer.servers = [
+      {url = "http://localhost:${toString servicePort}/";}
+    ];
+
+    routers.${serviceName} = {
+      rule = "Host(`azess.l.az-gruppe.com`)";
+      tls = {
+        certResolver = "ionos";
+      };
+      service = serviceName;
+      entrypoints = "websecure";
+    };
+  };
+}

hosts/AZ-PRM-1/services/azion-scheduler.nix

@@ -0,0 +1,36 @@
+{
+  config,
+  lib,
+  inputs,
+  pkgs,
+  ...
+}: let
+  serviceName = "azion-scheduler";
+  proxyServiceName = "${serviceName}-proxy";
+  servicePort = config.m3ta.ports.get serviceName;
+  schedulerProxyPort = config.m3ta.ports.get proxyServiceName;
+in {
+  services.azion-scheduler = {
+    enable = true;
+    package = inputs.azion-scheduler.packages.${pkgs.stdenv.hostPlatform.system}.default;
+    port = servicePort;
+    proxyPort = schedulerProxyPort;
+    environmentFile = config.age.secrets.azion-env.path;
+  };
+
+  # Traefik configuration
+  services.traefik.dynamicConfigOptions.http = {
+    services.${serviceName}.loadBalancer.servers = [
+      {url = "http://localhost:${toString servicePort}/";}
+    ];
+
+    routers.${serviceName} = {
+      rule = "Host(`azion.l.az-gruppe.com`)";
+      tls = {
+        certResolver = "ionos";
+      };
+      service = serviceName;
+      entrypoints = "websecure";
+    };
+  };
+}

hosts/AZ-PRM-1/services/backup-ingest.nix

@@ -0,0 +1,25 @@
+{pkgs, ...}: let
+  backupUser = "backup-ingest";
+  backupGroup = "backup-ingest";
+  backupRoot = "/srv/veeam-ingest";
+  cldBackupTarget = "${backupRoot}/AZ-CLD-1/var-backup";
+in {
+  users.groups.${backupGroup} = {};
+
+  users.users.${backupUser} = {
+    isSystemUser = true;
+    group = backupGroup;
+    home = backupRoot;
+    createHome = false;
+    shell = pkgs.bashInteractive;
+    openssh.authorizedKeys.keys = [
+      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBAbgaMFD8U+NrhA4P7BzVOsAbuu82ebkAOfQA09u12v cld-var-backup-sync@AZ-CLD-1-to-AZ-PRM-1"
+    ];
+  };
+
+  systemd.tmpfiles.rules = [
+    "d ${backupRoot} 0750 ${backupUser} ${backupGroup} - -"
+    "d ${backupRoot}/AZ-CLD-1 0750 ${backupUser} ${backupGroup} - -"
+    "d ${cldBackupTarget} 0750 ${backupUser} ${backupGroup} - -"
+  ];
+}

hosts/AZ-PRM-1/services/bpi.nix

@@ -0,0 +1,80 @@
+{
+  config,
+  lib,
+  pkgs,
+  ...
+}: let
+  serviceName = "bpi";
+  servicePort = config.m3ta.ports.get serviceName;
+  appDir = "/var/lib/bpi/app";
+in {
+  users.users.bpi = {
+    isSystemUser = true;
+    group = "bpi";
+    home = "/var/lib/bpi";
+    createHome = true;
+  };
+  users.groups.bpi = {};
+
+  systemd.services.bpi = {
+    description = "AZ INTEC Basispreis Index";
+    after = ["network-online.target"];
+    wants = ["network-online.target"];
+    wantedBy = ["multi-user.target"];
+
+    path = with pkgs; [
+      git
+      nodejs
+      openssh
+    ];
+
+    environment = {
+      PORT = toString servicePort;
+      HOME = "/var/lib/bpi";
+      BPI_BACKUP_DIR = "/var/lib/bpi/backups";
+      BPI_MAX_BACKUPS = "10";
+    };
+
+    preStart = ''
+      set -euo pipefail
+
+      if [ ! -d "${appDir}/.git" ]; then
+        rm -rf "${appDir}"
+        git clone --depth=1 https://git.az-gruppe.com/AZ-Intec-GmbH/BPI.git "${appDir}"
+      else
+        git -C "${appDir}" pull --ff-only
+      fi
+
+      if [ ! -f "${appDir}/server.js" ]; then
+        echo "${appDir}/server.js fehlt. Bitte server.js in das BPI Repository committen."
+        exit 1
+      fi
+    '';
+
+    serviceConfig = {
+      Type = "simple";
+      User = "bpi";
+      Group = "bpi";
+      StateDirectory = "bpi";
+      WorkingDirectory = "/var/lib/bpi";
+      ExecStart = "${pkgs.nodejs}/bin/node ${appDir}/server.js";
+      Restart = "on-failure";
+      RestartSec = "10s";
+    };
+  };
+
+  services.traefik.dynamicConfigOptions.http = {
+    services.bpi.loadBalancer.servers = [
+      {url = "http://localhost:${toString servicePort}/";}
+    ];
+
+    routers.bpi = {
+      rule = "Host(`bpi.l.az-gruppe.com`)";
+      tls = {
+        certResolver = "ionos";
+      };
+      service = "bpi";
+      entrypoints = "websecure";
+    };
+  };
+}

hosts/AZ-PRM-1/services/containers/atrocore.nix

@@ -0,0 +1,87 @@
+{config, ...}: let
+  serviceName = "atrocore";
+  servicePort = config.m3ta.ports.get serviceName;
+  instanceDir = "/var/www/pim.l.az-gruppe.com";
+in {
+  virtualisation.oci-containers.containers."${serviceName}" = {
+    # Secret-free image from the AZ-NIX-ava.1 pipeline (Gitea registry).
+    image = "git.az-gruppe.com/az-intec-gmbh/atrocore-web:latest";
+    # DB host is the alias for the host PostgreSQL on the podman web network;
+    # ATRO_DB_NAME/ATRO_DB_USER/ATRO_DB_PASSWORD come from the agenix secret
+    # (podman env-file) and are written to data/config.php by the entrypoint.
+    environment = {
+      ATRO_DB_HOST = "db";
+      ATRO_INSTANCE_DIR = "pim.l.az-gruppe.com";
+    };
+    environmentFiles = [config.age.secrets.atrocore-env.path];
+    # Registry pull credentials for the Gitea package registry (token from
+    # agenix secret, so nothing lands in the Nix store).
+    login = {
+      registry = "git.az-gruppe.com";
+      username = "sascha.koenig";
+      passwordFile = config.age.secrets.atrocore-registry-token.path;
+    };
+    ports = ["127.0.0.1:${toString servicePort}:80"];
+    volumes = [
+      "atrocore_data:${instanceDir}/data"
+    ];
+    extraOptions = ["--network=web" "--ip=10.89.0.16" "--add-host=db:10.89.0.1"];
+  };
+
+  # Idempotent provisioning of the atrocore role/database on the host
+  # PostgreSQL 17. initialScript cannot be used here (cluster is already
+  # initialized and the password must never land in the Nix store), so the
+  # secret is read at runtime and applied via psql peer auth.
+  systemd.services.atrocore-db-init = {
+    description = "Provision atrocore role/database from agenix secret";
+    after = ["postgresql.service"];
+    before = ["podman-${serviceName}.service"];
+    wants = ["postgresql.service"];
+    wantedBy = ["multi-user.target"];
+    serviceConfig = {
+      Type = "oneshot";
+      User = "postgres";
+      Group = "postgres";
+      RemainAfterExit = true;
+    };
+    path = [config.services.postgresql.package];
+    # Env-file format: plain KEY=value lines (no quotes), same file the
+    # container consumes.
+    script = ''
+      set -euo pipefail
+      ENV_FILE="${config.age.secrets.atrocore-env.path}"
+      get_val() {
+        grep -m1 "^$1=" "$ENV_FILE" | head -n1 | cut -d= -f2- | tr -d '\r'
+      }
+      db_name="$(get_val ATRO_DB_NAME)"
+      db_user="$(get_val ATRO_DB_USER)"
+      db_pass="$(get_val ATRO_DB_PASSWORD)"
+      if [ -z "$db_name" ] || [ -z "$db_user" ] || [ -z "$db_pass" ]; then
+        echo "atrocore-db-init: ATRO_DB_NAME/ATRO_DB_USER/ATRO_DB_PASSWORD missing in $ENV_FILE" >&2
+        exit 1
+      fi
+      psql -v ON_ERROR_STOP=1 -d postgres \
+        -v db_name="$db_name" -v db_user="$db_user" -v db_pass="$db_pass" <<'SQL'
+      SELECT format('CREATE ROLE %I LOGIN', :'db_user')
+      WHERE NOT EXISTS (SELECT FROM pg_roles WHERE rolname = :'db_user') \gexec
+      ALTER ROLE :"db_user" WITH LOGIN PASSWORD :'db_pass';
+      SELECT format('CREATE DATABASE %I OWNER %I', :'db_name', :'db_user')
+      WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = :'db_name') \gexec
+      ALTER DATABASE :"db_name" OWNER TO :"db_user";
+      SQL
+      echo "atrocore-db-init: role/database '$db_name' ready"
+    '';
+  };
+
+  # Traefik configuration specific to atrocore (AtroPIM)
+  services.traefik.dynamicConfigOptions.http = {
+    services.${serviceName}.loadBalancer.servers = [{url = "http://localhost:${toString servicePort}/";}];
+
+    routers.${serviceName} = {
+      rule = "Host(`pim.l.az-gruppe.com`)";
+      tls = {certResolver = "ionos";};
+      service = serviceName;
+      entrypoints = "websecure";
+    };
+  };
+}

hosts/AZ-PRM-1/services/containers/default.nix

@@ -0,0 +1,32 @@
+{lib, ...}: {
+  imports = [
+    #./baserow.nix
+    ./atrocore.nix
+    ./excalidraw.nix
+    ./kestra.nix
+    ./n8n-sandbox.nix
+    ./online3dviewer.nix
+    ./semaphore.nix
+    ./stirling-pdf.nix
+  ];
+  system.activationScripts.createPodmanNetworkWeb = lib.mkAfter ''
+    if ! /run/current-system/sw/bin/podman network exists web; then
+      /run/current-system/sw/bin/podman network create web --subnet=10.89.0.0/24 --internal
+    fi
+    if ! /run/current-system/sw/bin/podman network exists web-dev; then
+      /run/current-system/sw/bin/podman network create web-dev --subnet=10.89.1.0/24 --internal
+    fi
+    # Routed egress network: unlike web/web-dev (isolated), netavark sets up
+    # a default gateway plus NAT/masquerade for attached containers —
+    # outbound via host, incl. the NetBird overlay (wt0, 100.91.0.0/16).
+    if ! /run/current-system/sw/bin/podman network exists vpn-egress; then
+      /run/current-system/sw/bin/podman network create vpn-egress --subnet=10.89.9.0/24
+    fi
+    # n8n AI-Assistant sandbox stack (n8n-sandbox.nix): routed, NOT --internal
+    # — the privileged DinD runner must pull its sandbox images from ghcr.io.
+    # DNS between sandbox-api/sandbox-runner-1 via netavark/aardvark.
+    if ! /run/current-system/sw/bin/podman network exists n8n-sandbox; then
+      /run/current-system/sw/bin/podman network create n8n-sandbox --subnet=10.89.10.0/24
+    fi
+  '';
+}

hosts/AZ-PRM-1/services/containers/excalidraw.nix

@@ -0,0 +1,32 @@
+{config, ...}: let
+  serviceName = "excalidraw";
+  servicePort = config.m3ta.ports.get serviceName;
+in {
+  virtualisation.oci-containers.containers."${serviceName}" = {
+    image = "docker.io/excalidraw/excalidraw:latest";
+    cmd = [
+      "/bin/sh"
+      "-c"
+      ''
+        set -e
+        if ! grep -q "az-hide-excalidraw-plus" /usr/share/nginx/html/index.html; then
+          sed -i 's#</head>#<style id="az-hide-excalidraw-plus">.plus-banner{display:none!important}</style></head>#' /usr/share/nginx/html/index.html
+        fi
+        exec nginx -g 'daemon off;'
+      ''
+    ];
+    ports = ["127.0.0.1:${toString servicePort}:80"];
+    extraOptions = ["--ip=10.89.0.14" "--network=web"];
+  };
+
+  services.traefik.dynamicConfigOptions.http = {
+    services.${serviceName}.loadBalancer.servers = [{url = "http://localhost:${toString servicePort}/";}];
+
+    routers.${serviceName} = {
+      rule = "Host(`draw.l.az-gruppe.com`)";
+      tls = {certResolver = "ionos";};
+      service = serviceName;
+      entrypoints = "websecure";
+    };
+  };
+}

hosts/AZ-PRM-1/services/containers/kestra.nix

@@ -0,0 +1,42 @@
+{config, ...}: let
+  serviceName = "kestra";
+  servicePort = config.m3ta.ports.get serviceName;
+  metricsPort = config.m3ta.ports.get "kestra-metrics";
+in {
+  virtualisation.oci-containers.containers."${serviceName}" = {
+    image = "docker.io/kestra/kestra:latest";
+    environmentFiles = [
+      config.age.secrets.kestra-env.path
+      config.age.secrets.kestra-secrets.path
+    ];
+    cmd = ["server" "standalone" "--config" "/etc/config/application.yaml"];
+    ports = [
+      "127.0.0.1:${toString servicePort}:8080"
+      "127.0.0.1:${toString metricsPort}:8081"
+    ];
+    user = "root";
+    volumes = [
+      "/var/run/podman/podman.sock:/var/run/docker.sock"
+      "${config.age.secrets.kestra-config.path}:/etc/config/application.yaml"
+      "kestra_data:/app/storage"
+      "/tmp/kestra-wd:/tmp/kestra-wd"
+    ];
+    extraOptions = ["--add-host=postgres:10.89.0.1" "--ip=10.89.0.12" "--network=web"];
+  };
+
+  systemd.tmpfiles.rules = [
+    "d /tmp/kestra-wd 0750 1000 1000 - -"
+  ];
+
+  # Traefik configuration specific to kestra
+  services.traefik.dynamicConfigOptions.http = {
+    services.kestra.loadBalancer.servers = [{url = "http://localhost:${toString servicePort}/";}];
+
+    routers.kestra = {
+      rule = "Host(`k.l.az-gruppe.com`)";
+      tls = {certResolver = "ionos";};
+      service = "kestra";
+      entrypoints = "websecure";
+    };
+  };
+}

hosts/AZ-PRM-1/services/containers/n8n-sandbox.nix

@@ -0,0 +1,90 @@
+{config, ...}: let
+  sandboxApiPort = config.m3ta.ports.get "n8n-sandbox-api";
+  tlsDir = "/var/lib/n8n-sandbox/tls";
+  apiImage = "ghcr.io/n8n-io/n8n-sandbox-service-api:1.2.0";
+  runnerImage = "ghcr.io/n8n-io/n8n-sandbox-service-runner-dind:1.2.0";
+  sandboxImage = "ghcr.io/n8n-io/n8n-sandbox-service-sandbox:latest";
+in {
+  virtualisation.oci-containers.containers = {
+    sandbox-api = {
+      image = apiImage;
+      environmentFiles = [config.age.secrets.n8n-sandbox-env.path];
+      environment = {
+        SANDBOX_API_GRPC_TLS_CERT_FILE = "/tls/api/grpc-server.crt";
+        SANDBOX_API_GRPC_TLS_KEY_FILE = "/tls/api/grpc-server.key";
+        SANDBOX_API_GRPC_TLS_CLIENT_CA_FILE = "/tls/api/ca.crt";
+        SANDBOX_API_RUNNER_CONTROL_GRPC_TLS_CA_FILE = "/tls/api/ca.crt";
+        SANDBOX_API_RUNNER_CONTROL_GRPC_TLS_CERT_FILE = "/tls/api/control-grpc-api-client.crt";
+        SANDBOX_API_RUNNER_CONTROL_GRPC_TLS_KEY_FILE = "/tls/api/control-grpc-api-client.key";
+        SANDBOX_API_RUNNER_CONTROL_GRPC_TLS_SERVER_NAME = "sandbox-runner-1";
+      };
+      volumes = ["${tlsDir}:/tls:ro"];
+      ports = ["127.0.0.1:${toString sandboxApiPort}:8080"];
+      extraOptions = ["--network=n8n-sandbox"];
+    };
+
+    sandbox-runner-1 = {
+      image = runnerImage;
+      environmentFiles = [config.age.secrets.n8n-sandbox-env.path];
+      environment = {
+        SANDBOX_RUNNER_API_GRPC_ADDR = "sandbox-api:9090";
+        SANDBOX_RUNNER_HTTP_BASE_URL = "http://sandbox-runner-1:8080";
+        SANDBOX_RUNNER_CONTROL_GRPC_LISTEN_ADDR = ":9091";
+        SANDBOX_RUNNER_CONTROL_GRPC_ADVERTISE_ADDR = "sandbox-runner-1:9091";
+        SANDBOX_RUNNER_ID = "runner-1";
+        SANDBOX_RUNNER_DOCKER_SANDBOX_IMAGE = sandboxImage;
+        SANDBOX_RUNNER_REGISTRATION_GRPC_CA_FILE = "/tls/runner/ca.crt";
+        SANDBOX_RUNNER_REGISTRATION_GRPC_CERT_FILE = "/tls/runner/grpc-client.crt";
+        SANDBOX_RUNNER_REGISTRATION_GRPC_KEY_FILE = "/tls/runner/grpc-client.key";
+        SANDBOX_RUNNER_REGISTRATION_GRPC_SERVER_NAME = "sandbox-api";
+        SANDBOX_RUNNER_CONTROL_GRPC_TLS_CERT_FILE = "/tls/runner/control-grpc-server.crt";
+        SANDBOX_RUNNER_CONTROL_GRPC_TLS_KEY_FILE = "/tls/runner/control-grpc-server.key";
+        SANDBOX_RUNNER_CONTROL_GRPC_TLS_CLIENT_CA_FILE = "/tls/runner/ca.crt";
+      };
+      volumes = ["${tlsDir}:/tls:ro"];
+      dependsOn = ["sandbox-api"];
+      extraOptions = ["--network=n8n-sandbox" "--privileged"];
+    };
+  };
+
+  systemd.services.n8n-sandbox-certs = {
+    description = "n8n sandbox: mTLS certificate bootstrap";
+    wantedBy = ["multi-user.target"];
+    after = ["network-online.target"];
+    wants = ["network-online.target"];
+    serviceConfig = {
+      Type = "oneshot";
+      RemainAfterExit = true;
+      ConditionPathExists = "!${tlsDir}/ca.crt";
+    };
+    path = [config.virtualisation.podman.package];
+    preStart = ''
+      mkdir -p ${tlsDir}
+    '';
+    script = ''
+      podman run --rm \
+        --name n8n-sandbox-certs \
+        --user 0:0 \
+        -e NUM_RUNNERS=1 \
+        -v ${tlsDir}:/tls \
+        --entrypoint sh \
+        ${apiImage} \
+        -c 'bootstrap-mtls.sh --out-dir /tls --api-san sandbox-api --control-san-prefix sandbox-runner --world-readable && chown -R sandbox-api:sandbox-api /tls/api && chmod -R a+rX /tls'
+    '';
+  };
+
+  systemd.services."podman-sandbox-api" = {
+    after = ["n8n-sandbox-certs.service"];
+    requires = ["n8n-sandbox-certs.service"];
+    preStart = ''
+      ${config.virtualisation.podman.package}/bin/podman rm -f sandbox-api 2>/dev/null || true
+    '';
+  };
+  systemd.services."podman-sandbox-runner-1" = {
+    after = ["n8n-sandbox-certs.service"];
+    requires = ["n8n-sandbox-certs.service"];
+    preStart = ''
+      ${config.virtualisation.podman.package}/bin/podman rm -f sandbox-runner-1 2>/dev/null || true
+    '';
+  };
+}

hosts/AZ-PRM-1/services/containers/online3dviewer.nix

@@ -0,0 +1,31 @@
+{
+  config,
+  pkgs,
+  ...
+}: let
+  serviceName = "online3dviewer";
+  servicePort = config.m3ta.ports.get serviceName;
+in {
+  virtualisation.oci-containers.containers."${serviceName}" = {
+    image = "docker.io/nginxinc/nginx-unprivileged:stable-alpine";
+    ports = ["127.0.0.1:${toString servicePort}:8080"];
+    volumes = ["${pkgs.online3dviewer}:/usr/share/nginx/html:ro"];
+    extraOptions = [
+      "--ip=10.89.0.15"
+      "--network=web"
+      "--security-opt=no-new-privileges"
+      "--cap-drop=ALL"
+    ];
+  };
+
+  services.traefik.dynamicConfigOptions.http = {
+    services.${serviceName}.loadBalancer.servers = [{url = "http://localhost:${toString servicePort}/";}];
+
+    routers.${serviceName} = {
+      rule = "Host(`3dv.l.az-gruppe.com`)";
+      tls = {certResolver = "ionos";};
+      service = serviceName;
+      entrypoints = "websecure";
+    };
+  };
+}

hosts/AZ-PRM-1/services/containers/semaphore.nix

@@ -0,0 +1,85 @@
+{
+  config,
+  lib,
+  ...
+}: let
+  serviceName = "semaphore";
+  servicePort = config.m3ta.ports.get serviceName;
+in {
+  virtualisation.oci-containers.containers."${serviceName}" = {
+    image = "docker.io/semaphoreui/semaphore:latest";
+    environment = {
+      SEMAPHORE_DB_DIALECT = "postgres";
+      SEMAPHORE_DB_HOST = "postgres";
+      SEMAPHORE_DB_PORT = "5432";
+      SEMAPHORE_PLAYBOOK_PATH = "/tmp/semaphore/";
+    };
+    environmentFiles = [config.age.secrets.semaphore-env.path];
+    ports = ["127.0.0.1:${toString servicePort}:3000"];
+    volumes = [
+      "semaphore_data:/var/lib/semaphore"
+    ];
+    extraOptions = [
+      "--network=web:ip=10.89.0.17"
+      "--network=vpn-egress"
+      "--add-host=postgres:10.89.0.1"
+      "--dns=8.8.8.8"
+      "--dns=8.8.4.4"
+    ];
+  };
+
+  networking.firewall.extraForwardRules = ''
+    iifname "vpn-egress" oifname "wt0" accept
+  '';
+
+  systemd.services.semaphore-db-init = {
+    description = "Provision semaphore role/database from agenix secret";
+    after = ["postgresql.service"];
+    before = ["podman-${serviceName}.service"];
+    wants = ["postgresql.service"];
+    wantedBy = ["multi-user.target"];
+    serviceConfig = {
+      Type = "oneshot";
+      User = "postgres";
+      Group = "postgres";
+      RemainAfterExit = true;
+    };
+    path = [config.services.postgresql.package];
+    script = ''
+      set -euo pipefail
+      ENV_FILE="${config.age.secrets.semaphore-env.path}"
+      get_val() {
+        grep -m1 "^$1=" "$ENV_FILE" | head -n1 | cut -d= -f2- | tr -d '\r'
+      }
+      db_name="$(get_val SEMAPHORE_DB)"
+      db_user="$(get_val SEMAPHORE_DB_USER)"
+      db_pass="$(get_val SEMAPHORE_DB_PASS)"
+      if [ -z "$db_name" ] || [ -z "$db_user" ] || [ -z "$db_pass" ]; then
+        echo "semaphore-db-init: SEMAPHORE_DB/SEMAPHORE_DB_USER/SEMAPHORE_DB_PASS missing in $ENV_FILE" >&2
+        exit 1
+      fi
+      psql -v ON_ERROR_STOP=1 -d postgres \
+        -v db_name="$db_name" -v db_user="$db_user" -v db_pass="$db_pass" <<'SQL'
+      SELECT format('CREATE ROLE %I LOGIN', :'db_user')
+      WHERE NOT EXISTS (SELECT FROM pg_roles WHERE rolname = :'db_user') \gexec
+      ALTER ROLE :"db_user" WITH LOGIN PASSWORD :'db_pass';
+      SELECT format('CREATE DATABASE %I OWNER %I', :'db_name', :'db_user')
+      WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = :'db_name') \gexec
+      ALTER DATABASE :"db_name" OWNER TO :"db_user";
+      SQL
+      echo "semaphore-db-init: role/database '$db_name' ready"
+    '';
+  };
+
+  # Traefik configuration specific to semaphore
+  services.traefik.dynamicConfigOptions.http = {
+    services.${serviceName}.loadBalancer.servers = [{url = "http://localhost:${toString servicePort}/";}];
+
+    routers.${serviceName} = {
+      rule = "Host(`sem.l.az-gruppe.com`)";
+      tls = {certResolver = "ionos";};
+      service = serviceName;
+      entrypoints = "websecure";
+    };
+  };
+}

hosts/AZ-PRM-1/services/containers/stirling-pdf.nix

@@ -0,0 +1,30 @@
+{config, ...}: let
+  serviceName = "stirling-pdf";
+  servicePort = config.m3ta.ports.get serviceName;
+in {
+  virtualisation.oci-containers.containers."${serviceName}" = {
+    image = "docker.stirlingpdf.com/stirlingtools/stirling-pdf:latest-fat";
+    ports = ["127.0.0.1:${toString servicePort}:8080"];
+    environment = {
+      SECURITY_ENABLELOGIN = "False";
+      DISABLE_ADDITIONAL_FEATURES = "False";
+    };
+
+    volumes = [
+      "stirling_pdf_data:/usr/share/tessdata"
+      "stirling_pdf_configs:/configs"
+    ];
+    extraOptions = ["--ip=10.89.0.13" "--network=web"];
+  };
+
+  services.traefik.dynamicConfigOptions.http = {
+    services.${serviceName}.loadBalancer.servers = [{url = "http://localhost:${toString servicePort}/";}];
+
+    routers.${serviceName} = {
+      rule = "Host(`pdf.l.az-gruppe.com`)";
+      tls = {certResolver = "ionos";};
+      service = serviceName;
+      entrypoints = "websecure";
+    };
+  };
+}

hosts/AZ-PRM-1/services/default.nix

@@ -0,0 +1,20 @@
+{
+  imports = [
+    ./containers
+    ./backup-ingest.nix
+    ./azess.nix
+    ./azion-scheduler.nix
+    ./bpi.nix
+    ./monitoring
+    ./n8n.nix
+    ./netbird.nix
+    ./netbox.nix
+    ./pgadmin.nix
+    ./phishboard.nix
+    ./postgres.nix
+    ./printing.nix
+    ./traefik.nix
+    ./traefik-routing.nix
+    ./zugferd.nix
+  ];
+}

hosts/AZ-PRM-1/services/monitoring/alerting.nix

@@ -0,0 +1,318 @@
+{
+  config,
+  lib,
+  ...
+}: let
+  prometheusDatasourceUid = "prometheus";
+  ntfySecretAvailable = builtins.hasAttr "ntfy-grafana-webhook" config.age.secrets;
+
+  prometheusQuery = refId: expr: {
+    inherit refId;
+    datasourceUid = prometheusDatasourceUid;
+    relativeTimeRange = {
+      from = 600;
+      to = 0;
+    };
+    model = {
+      datasource = {
+        type = "prometheus";
+        uid = prometheusDatasourceUid;
+      };
+      editorMode = "code";
+      inherit expr refId;
+      hide = false;
+      instant = true;
+      intervalMs = 1000;
+      legendFormat = "__auto";
+      maxDataPoints = 43200;
+      range = false;
+    };
+  };
+
+  thresholdExpression = {
+    refId,
+    expressionRefId,
+    evaluator,
+  }: {
+    inherit refId;
+    datasourceUid = "__expr__";
+    model = {
+      conditions = [
+        {
+          inherit evaluator;
+          operator.type = "and";
+          query.params = [];
+          reducer = {
+            params = [];
+            type = "avg";
+          };
+          type = "query";
+        }
+      ];
+      datasource = {
+        name = "Expression";
+        type = "__expr__";
+        uid = "__expr__";
+      };
+      expression = expressionRefId;
+      hide = false;
+      inherit refId;
+      type = "threshold";
+    };
+  };
+
+  mkAlertRule = {
+    uid,
+    title,
+    expr,
+    for ? "5m",
+    noDataState ? "Alerting",
+    execErrState ? "Error",
+    evaluatorType ? "gt",
+    evaluatorParams ? [0 0],
+    labels ? {},
+    annotations ? {},
+  }: {
+    inherit uid title for noDataState execErrState;
+    condition = "B";
+    data = [
+      (prometheusQuery "A" expr)
+      (thresholdExpression {
+        refId = "B";
+        expressionRefId = "A";
+        evaluator = {
+          type = evaluatorType;
+          params = evaluatorParams;
+        };
+      })
+    ];
+    annotations =
+      {
+        summary = title;
+      }
+      // annotations;
+    labels =
+      {
+        service = "monitoring";
+      }
+      // labels;
+    isPaused = false;
+  };
+in {
+  warnings = lib.optional (!ntfySecretAvailable) ''
+    Grafana alert rules are provisioned, but ntfy notifications are disabled because secrets/ntfy-grafana-webhook.age is missing.
+    Create it as an EnvironmentFile containing: GRAFANA_NTFY_WEBHOOK_URL=https://<ntfy-webhook-url>
+  '';
+
+  systemd.services.grafana.serviceConfig.EnvironmentFile = lib.mkIf ntfySecretAvailable [
+    config.age.secrets."ntfy-grafana-webhook".path
+  ];
+
+  services.grafana.provision.alerting = {
+    rules.settings = {
+      apiVersion = 1;
+      groups = [
+        {
+          orgId = 1;
+          name = "az-infrastructure";
+          folder = "Infrastructure";
+          interval = "60s";
+          rules = [
+            (mkAlertRule {
+              uid = "az_host_down";
+              title = "Host down";
+              expr = ''up{job=~"node|node-cld"}'';
+              for = "2m";
+              evaluatorType = "lt";
+              evaluatorParams = [1 0];
+              labels.severity = "critical";
+              annotations.description = "Prometheus cannot scrape a node_exporter target.";
+            })
+            (mkAlertRule {
+              uid = "az_cpu_high";
+              title = "CPU usage high";
+              expr = ''100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)'';
+              for = "10m";
+              evaluatorType = "gt";
+              evaluatorParams = [90 0];
+              labels.severity = "warning";
+              annotations.description = "Average CPU usage has been above 90% for 10 minutes.";
+            })
+            (mkAlertRule {
+              uid = "az_memory_high";
+              title = "Memory usage high";
+              expr = ''100 * (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes))'';
+              for = "10m";
+              evaluatorType = "gt";
+              evaluatorParams = [90 0];
+              labels.severity = "warning";
+              annotations.description = "Memory usage has been above 90% for 10 minutes.";
+            })
+            (mkAlertRule {
+              uid = "az_rootfs_high";
+              title = "Root filesystem usage high";
+              expr = ''100 * (1 - (node_filesystem_avail_bytes{mountpoint="/",fstype!~"tmpfs|overlay|squashfs|ramfs"} / node_filesystem_size_bytes{mountpoint="/",fstype!~"tmpfs|overlay|squashfs|ramfs"}))'';
+              for = "15m";
+              evaluatorType = "gt";
+              evaluatorParams = [90 0];
+              labels.severity = "warning";
+              annotations.description = "Root filesystem usage has been above 90% for 15 minutes.";
+            })
+            (mkAlertRule {
+              uid = "az_systemd_failed";
+              title = "Systemd units failed";
+              expr = ''sum by(instance) (node_systemd_units{state="failed"})'';
+              for = "5m";
+              evaluatorType = "gt";
+              evaluatorParams = [0 0];
+              labels.severity = "warning";
+              annotations.description = "One or more systemd units are failed on the host.";
+            })
+            (mkAlertRule {
+              uid = "az_kestra_down";
+              title = "Kestra metrics unreachable";
+              expr = ''up{job="kestra"}'';
+              for = "2m";
+              evaluatorType = "lt";
+              evaluatorParams = [1 0];
+              labels = {
+                service = "kestra";
+                severity = "critical";
+              };
+              annotations.description = "Prometheus cannot scrape Kestra metrics.";
+            })
+            (mkAlertRule {
+              uid = "az_kestra_execution_failed";
+              title = "Kestra execution failed";
+              expr = ''sum by(namespace_id, flow_id, state) (increase(kestra_executor_execution_end_count_total{job="kestra",state=~"FAILED|WARNING|KILLED"}[5m]))'';
+              for = "1m";
+              noDataState = "OK";
+              evaluatorType = "gt";
+              evaluatorParams = [0 0];
+              labels = {
+                service = "kestra";
+                severity = "critical";
+              };
+              annotations.description = "A Kestra flow execution ended with FAILED, WARNING, or KILLED.";
+            })
+            (mkAlertRule {
+              uid = "az_litellm_down";
+              title = "LiteLLM metrics unreachable";
+              expr = ''up{job="litellm"}'';
+              for = "2m";
+              evaluatorType = "lt";
+              evaluatorParams = [1 0];
+              labels = {
+                service = "litellm";
+                severity = "critical";
+              };
+              annotations.description = "Prometheus cannot scrape LiteLLM /metrics on AZ-CLD-1 over Netbird.";
+            })
+            (mkAlertRule {
+              uid = "az_litellm_proxy_errors";
+              title = "LiteLLM proxy errors";
+              expr = ''sum(increase(litellm_proxy_failed_requests_metric_total{job="litellm"}[5m]))'';
+              for = "1m";
+              noDataState = "OK";
+              evaluatorType = "gt";
+              evaluatorParams = [0 0];
+              labels = {
+                service = "litellm";
+                severity = "warning";
+              };
+              annotations.description = "LiteLLM reported one or more failed proxy responses in the last 5 minutes.";
+            })
+            (mkAlertRule {
+              uid = "az_litellm_latency_high";
+              title = "LiteLLM latency high";
+              expr = ''histogram_quantile(0.95, sum(rate(litellm_request_total_latency_metric_bucket{job="litellm"}[5m])) by (le))'';
+              for = "10m";
+              noDataState = "OK";
+              evaluatorType = "gt";
+              evaluatorParams = [30 0];
+              labels = {
+                service = "litellm";
+                severity = "warning";
+              };
+              annotations.description = "LiteLLM p95 total request latency has been above 30 seconds for 10 minutes.";
+            })
+            (mkAlertRule {
+              uid = "az_http_probe_failed";
+              title = "HTTP probe failed";
+              expr = ''probe_success{job="blackbox_http"}'';
+              for = "2m";
+              evaluatorType = "lt";
+              evaluatorParams = [1 0];
+              labels.severity = "critical";
+              annotations.description = "A blackbox HTTP probe is failing.";
+            })
+            (mkAlertRule {
+              uid = "az_icmp_probe_failed";
+              title = "ICMP probe failed";
+              expr = ''probe_success{job="blackbox_icmp"}'';
+              for = "2m";
+              evaluatorType = "lt";
+              evaluatorParams = [1 0];
+              labels.severity = "warning";
+              annotations.description = "A blackbox ICMP probe is failing.";
+            })
+            (mkAlertRule {
+              uid = "az_tls_cert_expiring";
+              title = "TLS certificate expires soon";
+              expr = ''(probe_ssl_earliest_cert_expiry{job="blackbox_http"} - time()) / 86400'';
+              for = "1h";
+              noDataState = "OK";
+              evaluatorType = "lt";
+              evaluatorParams = [14 0];
+              labels.severity = "warning";
+              annotations.description = "A probed TLS certificate expires in less than 14 days.";
+            })
+          ];
+        }
+      ];
+    };
+
+    contactPoints.settings = lib.mkIf ntfySecretAvailable {
+      apiVersion = 1;
+      contactPoints = [
+        {
+          orgId = 1;
+          name = "ntfy";
+          receivers = [
+            {
+              uid = "ntfy-webhook";
+              type = "webhook";
+              disableResolveMessage = false;
+              settings = {
+                url = "$GRAFANA_NTFY_WEBHOOK_URL";
+                httpMethod = "POST";
+                # Let Grafana render its default title/message. The default message
+                # includes all alert annotations via .Annotations.SortedPairs;
+                # ntfy's grafana template then displays only title/message instead
+                # of the full webhook JSON body.
+                headers = {
+                  Template = "grafana";
+                  Markdown = "yes";
+                };
+              };
+            }
+          ];
+        }
+      ];
+    };
+
+    policies.settings = lib.mkIf ntfySecretAvailable {
+      apiVersion = 1;
+      policies = [
+        {
+          orgId = 1;
+          receiver = "ntfy";
+          group_by = ["alertname" "instance" "target" "severity"];
+          group_wait = "30s";
+          group_interval = "5m";
+          repeat_interval = "4h";
+        }
+      ];
+    };
+  };
+}

hosts/AZ-PRM-1/services/monitoring/alloy.nix

@@ -0,0 +1,92 @@
+{
+  config,
+  lib,
+  ...
+}: let
+  lokiPort = config.m3ta.ports.get "loki";
+  alloyPort = config.m3ta.ports.get "alloy";
+in {
+  services.alloy = {
+    enable = true;
+    configPath = "/etc/alloy";
+    extraFlags = [
+      "--server.http.listen-addr=127.0.0.1:${toString alloyPort}"
+      "--disable-reporting"
+    ];
+  };
+
+  environment.etc."alloy/config.alloy".text = ''
+    loki.relabel "journal" {
+      forward_to = []
+
+      rule {
+        source_labels = ["__journal__systemd_unit"]
+        target_label  = "unit"
+      }
+
+      rule {
+        source_labels = ["__journal_priority_keyword"]
+        target_label  = "level"
+      }
+
+      rule {
+        source_labels = ["__journal_syslog_identifier"]
+        target_label  = "syslog_identifier"
+      }
+    }
+
+    loki.source.journal "system" {
+      forward_to    = [loki.process.journal.receiver]
+      relabel_rules = loki.relabel.journal.rules
+      labels        = {
+        host        = "AZ-PRM-1",
+        environment = "prod",
+      }
+    }
+
+    loki.process "journal" {
+      forward_to = [loki.write.local.receiver]
+
+      stage.json {
+        expressions = {
+          app              = "app",
+          service          = "service",
+          level            = "level",
+          trace_id         = "trace_id",
+          session_id       = "session_id",
+          n8n_execution_id = "execution_id",
+          workflow_id      = "workflow_id",
+          execution_id     = "execution_id",
+          message          = "message",
+        }
+        drop_malformed = false
+      }
+
+      stage.labels {
+        values = {
+          app     = "",
+          service = "",
+          level   = "",
+        }
+      }
+
+      stage.structured_metadata {
+        values = {
+          trace_id         = "",
+          session_id       = "",
+          n8n_execution_id = "",
+          workflow_id      = "",
+          execution_id     = "",
+        }
+      }
+    }
+
+    loki.write "local" {
+      endpoint {
+        url = "http://127.0.0.1:${toString lokiPort}/loki/api/v1/push"
+      }
+    }
+  '';
+
+  systemd.services.alloy.serviceConfig.SupplementaryGroups = lib.mkAfter ["adm"];
+}

hosts/AZ-PRM-1/services/monitoring/blackbox.nix

@@ -0,0 +1,100 @@
+# Phase 2 — not active until imported in services/default.nix.
+# Activate by adding `./blackbox.nix` to hosts/AZ-PRM-1/services/monitoring/default.nix imports.
+{config, ...}: let
+  blackboxPort = config.m3ta.ports.get "blackbox-exporter";
+  prometheusPort = config.m3ta.ports.get "prometheus";
+in {
+  services.prometheus.exporters.blackbox = {
+    enable = true;
+    port = blackboxPort;
+    listenAddress = "127.0.0.1";
+    openFirewall = false;
+    configFile = builtins.toFile "blackbox.yml" ''
+      modules:
+        http_2xx:
+          prober: http
+          timeout: 5s
+        http_post_2xx:
+          prober: http
+          timeout: 5s
+          http:
+            method: POST
+        icmp_ping:
+          prober: icmp
+          timeout: 5s
+        tcp_connect:
+          prober: tcp
+          timeout: 5s
+    '';
+  };
+
+  services.prometheus.scrapeConfigs = [
+    {
+      job_name = "blackbox_http";
+      metrics_path = "/probe";
+      params.module = ["http_2xx"];
+      static_configs = [
+        {
+          targets = [
+            "https://g.l.az-gruppe.com"
+            "https://wf.l.az-gruppe.com"
+            "https://k.l.az-gruppe.com"
+            "https://sem.l.az-gruppe.com"
+            "https://git.az-gruppe.com"
+            "https://ping.az-gruppe.com"
+            "https://llm.az-gruppe.com"
+            "https://r.az-gruppe.com"
+          ];
+          labels = {
+            instance = "AZ-PRM-1-blackbox";
+          };
+        }
+      ];
+      relabel_configs = [
+        {
+          source_labels = ["__address__"];
+          target_label = "__param_target";
+        }
+        {
+          source_labels = ["__param_target"];
+          target_label = "target";
+        }
+        {
+          target_label = "__address__";
+          replacement = "localhost:${toString blackboxPort}";
+        }
+      ];
+    }
+    {
+      job_name = "blackbox_icmp";
+      metrics_path = "/probe";
+      params.module = ["icmp_ping"];
+      static_configs = [
+        {
+          targets = [
+            "100.91.203.184" # AZ-CLD-1 netbird
+            "192.168.152.97" # SMB/DMS share
+            "192.168.152.98" # SkriptHelper
+            "192.168.152.102" # legacy PRTG (until decommissioned)
+            "1.1.1.1" # upstream DNS reachability
+            "8.8.8.8" # upstream DNS reachability
+          ];
+        }
+      ];
+      relabel_configs = [
+        {
+          source_labels = ["__address__"];
+          target_label = "__param_target";
+        }
+        {
+          source_labels = ["__param_target"];
+          target_label = "target";
+        }
+        {
+          target_label = "__address__";
+          replacement = "localhost:${toString blackboxPort}";
+        }
+      ];
+    }
+  ];
+}

hosts/AZ-PRM-1/services/monitoring/cld-scrape.nix

@@ -0,0 +1,55 @@
+{
+  config,
+  lib,
+  ...
+}: let
+  nodeExporterPort = config.m3ta.ports.get "node-exporter";
+  litellmPort = config.m3ta.ports.get "litellm";
+  cldNetbirdIP = "100.91.203.184";
+  litellmPrometheusBearerSecretAvailable = builtins.hasAttr "litellm-prometheus-bearer" config.age.secrets;
+  litellmPrometheusAuthorization = lib.optionalAttrs litellmPrometheusBearerSecretAvailable {
+    authorization = {
+      type = "Bearer";
+      credentials_file = config.age.secrets."litellm-prometheus-bearer".path;
+    };
+  };
+in {
+  # Prometheus' config checker validates credentials_file paths at build time,
+  # but agenix secrets only exist at runtime under /run/agenix.
+  services.prometheus.checkConfig = lib.mkIf litellmPrometheusBearerSecretAvailable false;
+
+  services.prometheus.scrapeConfigs = [
+    {
+      job_name = "node-cld";
+      static_configs = [
+        {
+          targets = ["${cldNetbirdIP}:${toString nodeExporterPort}"];
+          labels = {
+            instance = "AZ-CLD-1";
+          };
+        }
+      ];
+    }
+    ({
+        job_name = "litellm";
+        metrics_path = "/metrics/";
+        scrape_interval = "15s";
+        static_configs = [
+          {
+            targets = ["${cldNetbirdIP}:${toString litellmPort}"];
+            labels = {
+              instance = "AZ-CLD-1";
+              service = "litellm";
+            };
+          }
+        ];
+      }
+      // litellmPrometheusAuthorization)
+  ];
+
+  # Allow CLD to reach PRM prometheus scrapes (prometheus initiates connection TO CLD exporters)
+  networking.firewall.extraCommands = ''
+    # No PRM-side firewall change needed: PRM scrapes outbound to CLD:9100/4000.
+    # CLD-side must allow inbound from 100.91.49.26 (PRM netbird IP) — see CLD config.
+  '';
+}

hosts/AZ-PRM-1/services/monitoring/dashboards/az-blackbox-probe-demo.json

@@ -0,0 +1,722 @@
+{
+  "annotations": {
+    "list": [
+      {
+        "builtIn": 1,
+        "datasource": {
+          "type": "grafana",
+          "uid": "-- Grafana --"
+        },
+        "enable": true,
+        "hide": true,
+        "iconColor": "rgba(0, 211, 255, 1)",
+        "name": "Annotations & Alerts",
+        "target": {
+          "limit": 100,
+          "matchAny": false,
+          "tags": [],
+          "type": "dashboard"
+        },
+        "type": "dashboard"
+      }
+    ]
+  },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 1,
+  "links": [],
+  "liveNow": false,
+  "schemaVersion": 39,
+  "style": "dark",
+  "tags": [
+    "az",
+    "demo",
+    "provisioned"
+  ],
+  "templating": {
+    "list": []
+  },
+  "time": {
+    "from": "now-24h",
+    "to": "now"
+  },
+  "timepicker": {
+    "refresh_intervals": [
+      "10s",
+      "30s",
+      "1m",
+      "5m",
+      "15m",
+      "30m",
+      "1h"
+    ]
+  },
+  "timezone": "browser",
+  "version": 1,
+  "weekStart": "",
+  "uid": "az-blackbox-demo",
+  "title": "AZ Blackbox Probe Demo",
+  "refresh": "30s",
+  "description": "Demo dashboard for HTTP and ICMP probes from Prometheus blackbox_exporter.",
+  "panels": [
+    {
+      "id": 1,
+      "title": "HTTP probe success",
+      "type": "stat",
+      "datasource": null,
+      "gridPos": {
+        "h": 5,
+        "w": 8,
+        "x": 0,
+        "y": 0
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {},
+          "mappings": [],
+          "unit": "short",
+          "decimals": 0,
+          "min": 0,
+          "max": 1,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "red",
+                "value": null
+              },
+              {
+                "color": "green",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "colorMode": "background",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "targets": [
+        {
+          "expr": "probe_success{job=\"blackbox_http\"}",
+          "legendFormat": "{{target}}",
+          "refId": "A"
+        }
+      ],
+      "description": "1 means the last HTTP probe matched the http_2xx module."
+    },
+    {
+      "id": 2,
+      "title": "HTTP 24h availability",
+      "type": "stat",
+      "datasource": null,
+      "gridPos": {
+        "h": 5,
+        "w": 8,
+        "x": 8,
+        "y": 0
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {},
+          "mappings": [],
+          "unit": "percent",
+          "decimals": 2,
+          "min": 0,
+          "max": 100,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "red",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 95
+              },
+              {
+                "color": "green",
+                "value": 99
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "colorMode": "background",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "targets": [
+        {
+          "expr": "avg_over_time(probe_success{job=\"blackbox_http\"}[24h]) * 100",
+          "legendFormat": "{{target}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 3,
+      "title": "TLS cert days left",
+      "type": "stat",
+      "datasource": null,
+      "gridPos": {
+        "h": 5,
+        "w": 8,
+        "x": 16,
+        "y": 0
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {},
+          "mappings": [],
+          "unit": "d",
+          "decimals": 0,
+          "min": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "red",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 14
+              },
+              {
+                "color": "green",
+                "value": 30
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "colorMode": "background",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "targets": [
+        {
+          "expr": "(probe_ssl_earliest_cert_expiry{job=\"blackbox_http\"} - time()) / 86400",
+          "legendFormat": "{{target}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 4,
+      "title": "HTTP latency",
+      "type": "timeseries",
+      "datasource": null,
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 5
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {},
+          "mappings": [],
+          "unit": "s",
+          "decimals": 3,
+          "min": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 1
+              },
+              {
+                "color": "red",
+                "value": 3
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "expr": "probe_duration_seconds{job=\"blackbox_http\"}",
+          "legendFormat": "{{target}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 5,
+      "title": "HTTP status code",
+      "type": "timeseries",
+      "datasource": null,
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 5
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {},
+          "mappings": [],
+          "unit": "short",
+          "decimals": 0,
+          "min": 0
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "expr": "probe_http_status_code{job=\"blackbox_http\"}",
+          "legendFormat": "{{target}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 6,
+      "title": "ICMP probe success",
+      "type": "stat",
+      "datasource": null,
+      "gridPos": {
+        "h": 5,
+        "w": 8,
+        "x": 0,
+        "y": 13
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {},
+          "mappings": [],
+          "unit": "short",
+          "decimals": 0,
+          "min": 0,
+          "max": 1,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "red",
+                "value": null
+              },
+              {
+                "color": "green",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "colorMode": "background",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "targets": [
+        {
+          "expr": "probe_success{job=\"blackbox_icmp\"}",
+          "legendFormat": "{{target}}",
+          "refId": "A"
+        }
+      ],
+      "description": "1 means the last ICMP probe succeeded."
+    },
+    {
+      "id": 7,
+      "title": "ICMP 24h availability",
+      "type": "stat",
+      "datasource": null,
+      "gridPos": {
+        "h": 5,
+        "w": 8,
+        "x": 8,
+        "y": 13
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {},
+          "mappings": [],
+          "unit": "percent",
+          "decimals": 2,
+          "min": 0,
+          "max": 100,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "red",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 95
+              },
+              {
+                "color": "green",
+                "value": 99
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "colorMode": "background",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "targets": [
+        {
+          "expr": "avg_over_time(probe_success{job=\"blackbox_icmp\"}[24h]) * 100",
+          "legendFormat": "{{target}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 8,
+      "title": "ICMP last latency",
+      "type": "stat",
+      "datasource": null,
+      "gridPos": {
+        "h": 5,
+        "w": 8,
+        "x": 16,
+        "y": 13
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {},
+          "mappings": [],
+          "unit": "s",
+          "decimals": 3,
+          "min": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 0.1
+              },
+              {
+                "color": "red",
+                "value": 0.3
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "colorMode": "background",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "targets": [
+        {
+          "expr": "probe_duration_seconds{job=\"blackbox_icmp\"}",
+          "legendFormat": "{{target}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 9,
+      "title": "ICMP latency",
+      "type": "timeseries",
+      "datasource": null,
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 18
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {},
+          "mappings": [],
+          "unit": "s",
+          "decimals": 3,
+          "min": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 0.1
+              },
+              {
+                "color": "red",
+                "value": 0.3
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "expr": "probe_duration_seconds{job=\"blackbox_icmp\"}",
+          "legendFormat": "{{target}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 10,
+      "title": "Probe success timeline",
+      "type": "timeseries",
+      "datasource": null,
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 18
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {},
+          "mappings": [],
+          "unit": "short",
+          "decimals": 0,
+          "min": 0,
+          "max": 1,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "red",
+                "value": null
+              },
+              {
+                "color": "green",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "expr": "probe_success{job=\"blackbox_http\"}",
+          "legendFormat": "HTTP {{target}}",
+          "refId": "A"
+        },
+        {
+          "expr": "probe_success{job=\"blackbox_icmp\"}",
+          "legendFormat": "ICMP {{target}}",
+          "refId": "B"
+        }
+      ]
+    },
+    {
+      "id": 11,
+      "title": "DNS + connect + TLS phase timing",
+      "type": "timeseries",
+      "datasource": null,
+      "gridPos": {
+        "h": 8,
+        "w": 24,
+        "x": 0,
+        "y": 26
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {},
+          "mappings": [],
+          "unit": "s",
+          "decimals": 3,
+          "min": 0
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "expr": "probe_http_duration_seconds{job=\"blackbox_http\", phase=\"resolve\"}",
+          "legendFormat": "{{target}} resolve",
+          "refId": "A"
+        },
+        {
+          "expr": "probe_http_duration_seconds{job=\"blackbox_http\", phase=\"connect\"}",
+          "legendFormat": "{{target}} connect",
+          "refId": "B"
+        },
+        {
+          "expr": "probe_http_duration_seconds{job=\"blackbox_http\", phase=\"tls\"}",
+          "legendFormat": "{{target}} tls",
+          "refId": "C"
+        },
+        {
+          "expr": "probe_http_duration_seconds{job=\"blackbox_http\", phase=\"processing\"}",
+          "legendFormat": "{{target}} processing",
+          "refId": "D"
+        }
+      ]
+    }
+  ]
+}

hosts/AZ-PRM-1/services/monitoring/dashboards/az-litellm-observability.json

@@ -0,0 +1,568 @@
+{
+  "annotations": {
+    "list": [
+      {
+        "builtIn": 1,
+        "datasource": {
+          "type": "grafana",
+          "uid": "-- Grafana --"
+        },
+        "enable": true,
+        "hide": true,
+        "iconColor": "rgba(0, 211, 255, 1)",
+        "name": "Annotations & Alerts",
+        "target": {
+          "limit": 100,
+          "matchAny": false,
+          "tags": [],
+          "type": "dashboard"
+        },
+        "type": "dashboard"
+      }
+    ]
+  },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 1,
+  "links": [],
+  "liveNow": false,
+  "panels": [
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "mappings": [
+            {
+              "options": {
+                "0": {
+                  "color": "red",
+                  "index": 1,
+                  "text": "down"
+                },
+                "1": {
+                  "color": "green",
+                  "index": 0,
+                  "text": "up"
+                }
+              },
+              "type": "value"
+            }
+          ],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "red",
+                "value": null
+              },
+              {
+                "color": "green",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 6,
+        "x": 0,
+        "y": 0
+      },
+      "id": 1,
+      "options": {
+        "colorMode": "background",
+        "graphMode": "none",
+        "justifyMode": "center",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "pluginVersion": "11.0.0",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "editorMode": "code",
+          "expr": "up{job=\"litellm\", instance=~\"$instance\"}",
+          "legendFormat": "{{instance}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Scrape status",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "decimals": 3,
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 6,
+        "x": 6,
+        "y": 0
+      },
+      "id": 2,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "pluginVersion": "11.0.0",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "editorMode": "code",
+          "expr": "sum(rate(litellm_proxy_total_requests_metric_total{job=\"litellm\", instance=~\"$instance\"}[$__rate_interval]))",
+          "legendFormat": "requests/s",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Proxy request rate",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "decimals": 3,
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 6,
+        "x": 12,
+        "y": 0
+      },
+      "id": 3,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "pluginVersion": "11.0.0",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "editorMode": "code",
+          "expr": "sum(rate(litellm_proxy_failed_requests_metric_total{job=\"litellm\", instance=~\"$instance\"}[$__rate_interval])) or vector(0)",
+          "legendFormat": "errors/s",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Proxy error rate",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "decimals": 2,
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 6,
+        "x": 18,
+        "y": 0
+      },
+      "id": 4,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "pluginVersion": "11.0.0",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "editorMode": "code",
+          "expr": "histogram_quantile(0.95, sum(rate(litellm_request_total_latency_metric_bucket{job=\"litellm\", instance=~\"$instance\"}[$__rate_interval])) by (le))",
+          "legendFormat": "p95",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "p95 total latency",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "lineInterpolation": "linear",
+            "lineWidth": 2,
+            "showPoints": "never",
+            "spanNulls": false
+          },
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 4
+      },
+      "id": 5,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "displayMode": "table",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "single",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "editorMode": "code",
+          "expr": "sum by (requested_model) (rate(litellm_proxy_total_requests_metric_total{job=\"litellm\", instance=~\"$instance\"}[$__rate_interval]))",
+          "legendFormat": "{{requested_model}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Requests by requested model",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "lineInterpolation": "linear",
+            "lineWidth": 2,
+            "showPoints": "never",
+            "spanNulls": false
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 4
+      },
+      "id": 6,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "displayMode": "table",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "single",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "editorMode": "code",
+          "expr": "sum by (model) (rate(litellm_total_tokens_metric_total{job=\"litellm\", instance=~\"$instance\"}[$__rate_interval]))",
+          "legendFormat": "{{model}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Token rate by model",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "currencyUSD"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 12
+      },
+      "id": 7,
+      "options": {
+        "displayMode": "gradient",
+        "legend": {
+          "calcs": [],
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": false
+        },
+        "orientation": "horizontal",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": true
+        },
+        "showUnfilled": true,
+        "valueMode": "color"
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "editorMode": "code",
+          "expr": "topk(10, sum by (model) (increase(litellm_spend_metric_total{job=\"litellm\", instance=~\"$instance\"}[$__range])))",
+          "legendFormat": "{{model}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Spend by model in selected range",
+      "type": "bargauge"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "lineInterpolation": "linear",
+            "lineWidth": 2,
+            "showPoints": "never",
+            "spanNulls": false
+          },
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 12
+      },
+      "id": 8,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "displayMode": "table",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "single",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "editorMode": "code",
+          "expr": "histogram_quantile(0.95, sum by (le, model) (rate(litellm_llm_api_latency_metric_bucket{job=\"litellm\", instance=~\"$instance\"}[$__rate_interval])))",
+          "legendFormat": "{{model}} p95",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "LLM API latency by model",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "gridPos": {
+        "h": 10,
+        "w": 24,
+        "x": 0,
+        "y": 20
+      },
+      "id": 9,
+      "options": {
+        "dedupStrategy": "none",
+        "enableLogDetails": true,
+        "prettifyLogMessage": false,
+        "showCommonLabels": false,
+        "showLabels": false,
+        "showTime": true,
+        "sortOrder": "Descending",
+        "wrapLogMessage": true
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "loki"
+          },
+          "editorMode": "code",
+          "expr": "{host=\"AZ-CLD-1\", unit=~\"podman-litellm.service|docker-litellm.service|litellm.service\"}",
+          "queryType": "range",
+          "refId": "A"
+        }
+      ],
+      "title": "LiteLLM logs from Loki",
+      "type": "logs"
+    }
+  ],
+  "refresh": "30s",
+  "schemaVersion": 39,
+  "style": "dark",
+  "tags": [
+    "az",
+    "litellm",
+    "prometheus",
+    "loki",
+    "provisioned"
+  ],
+  "templating": {
+    "list": [
+      {
+        "current": {
+          "selected": true,
+          "text": "All",
+          "value": "$__all"
+        },
+        "datasource": {
+          "type": "prometheus",
+          "uid": "prometheus"
+        },
+        "definition": "label_values(up{job=\"litellm\"}, instance)",
+        "hide": 0,
+        "includeAll": true,
+        "label": "Instance",
+        "multi": true,
+        "name": "instance",
+        "options": [],
+        "query": {
+          "query": "label_values(up{job=\"litellm\"}, instance)",
+          "refId": "PrometheusVariableQueryEditor-VariableQuery"
+        },
+        "refresh": 1,
+        "regex": "",
+        "skipUrlSync": false,
+        "sort": 1,
+        "type": "query"
+      }
+    ]
+  },
+  "time": {
+    "from": "now-6h",
+    "to": "now"
+  },
+  "timepicker": {},
+  "timezone": "browser",
+  "title": "AZ LiteLLM Observability",
+  "uid": "az-litellm-observability",
+  "version": 1,
+  "weekStart": ""
+}

hosts/AZ-PRM-1/services/monitoring/dashboards/az-loki-usage-demo.json

@@ -0,0 +1,764 @@
+{
+  "annotations": {
+    "list": [
+      {
+        "builtIn": 1,
+        "datasource": {
+          "type": "grafana",
+          "uid": "-- Grafana --"
+        },
+        "enable": true,
+        "hide": true,
+        "iconColor": "rgba(0, 211, 255, 1)",
+        "name": "Annotations & Alerts",
+        "target": {
+          "limit": 100,
+          "matchAny": false,
+          "tags": [],
+          "type": "dashboard"
+        },
+        "type": "dashboard"
+      }
+    ]
+  },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 1,
+  "links": [],
+  "liveNow": false,
+  "schemaVersion": 39,
+  "style": "dark",
+  "tags": [
+    "az",
+    "demo",
+    "loki",
+    "logs",
+    "provisioned"
+  ],
+  "templating": {
+    "list": [
+      {
+        "name": "host",
+        "label": "Host",
+        "type": "query",
+        "datasource": {
+          "type": "loki",
+          "uid": "loki"
+        },
+        "definition": "label_values(host)",
+        "query": "label_values(host)",
+        "refresh": 1,
+        "sort": 1,
+        "includeAll": true,
+        "multi": true,
+        "allValue": ".+",
+        "current": {
+          "selected": true,
+          "text": "All",
+          "value": "$__all"
+        },
+        "hide": 0,
+        "skipUrlSync": false
+      },
+      {
+        "name": "unit",
+        "label": "Systemd Unit",
+        "type": "query",
+        "datasource": {
+          "type": "loki",
+          "uid": "loki"
+        },
+        "definition": "label_values({host=~\"$host\"}, unit)",
+        "query": "label_values({host=~\"$host\"}, unit)",
+        "refresh": 1,
+        "sort": 1,
+        "includeAll": true,
+        "multi": true,
+        "allValue": ".*",
+        "current": {
+          "selected": true,
+          "text": "All",
+          "value": "$__all"
+        },
+        "hide": 0,
+        "skipUrlSync": false
+      },
+      {
+        "name": "app",
+        "label": "App",
+        "type": "query",
+        "datasource": {
+          "type": "loki",
+          "uid": "loki"
+        },
+        "definition": "label_values({host=~\"$host\", unit=~\"$unit\"}, app)",
+        "query": "label_values({host=~\"$host\", unit=~\"$unit\"}, app)",
+        "refresh": 1,
+        "sort": 1,
+        "includeAll": true,
+        "multi": true,
+        "allValue": ".*",
+        "current": {
+          "selected": true,
+          "text": "All",
+          "value": "$__all"
+        },
+        "hide": 0,
+        "skipUrlSync": false
+      },
+      {
+        "name": "level",
+        "label": "Level",
+        "type": "query",
+        "datasource": {
+          "type": "loki",
+          "uid": "loki"
+        },
+        "definition": "label_values(level)",
+        "query": "label_values(level)",
+        "refresh": 1,
+        "sort": 1,
+        "includeAll": true,
+        "multi": true,
+        "allValue": ".+",
+        "current": {
+          "selected": true,
+          "text": "All",
+          "value": "$__all"
+        },
+        "hide": 0,
+        "skipUrlSync": false
+      }
+    ]
+  },
+  "time": {
+    "from": "now-6h",
+    "to": "now"
+  },
+  "timepicker": {
+    "refresh_intervals": [
+      "10s",
+      "30s",
+      "1m",
+      "5m",
+      "15m",
+      "30m",
+      "1h"
+    ]
+  },
+  "timezone": "browser",
+  "version": 1,
+  "weekStart": "",
+  "uid": "az-loki-usage-demo",
+  "title": "AZ Loki Usage Demo",
+  "refresh": "30s",
+  "description": "Provisioned demo dashboard for Loki ingestion health, log volume, noisy units/apps, and recent warnings/errors from Alloy journald streams.",
+  "panels": [
+    {
+      "id": 1,
+      "title": "Log ingest rate",
+      "type": "stat",
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 6,
+        "x": 0,
+        "y": 0
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {},
+          "mappings": [],
+          "unit": "logs/s",
+          "decimals": 2,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 100
+              },
+              {
+                "color": "red",
+                "value": 500
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "colorMode": "background",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "loki"
+          },
+          "expr": "sum(rate({host=~\"$host\", unit=~\"$unit\", app=~\"$app\"}[5m]))",
+          "legendFormat": "logs/sec",
+          "queryType": "range",
+          "refId": "A"
+        }
+      ],
+      "description": "Total Loki log ingestion rate for the selected hosts."
+    },
+    {
+      "id": 2,
+      "title": "Warn/Error rate",
+      "type": "stat",
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 6,
+        "x": 6,
+        "y": 0
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {},
+          "mappings": [],
+          "unit": "logs/s",
+          "decimals": 2,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 0.1
+              },
+              {
+                "color": "red",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "colorMode": "background",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "loki"
+          },
+          "expr": "sum(rate({host=~\"$host\", unit=~\"$unit\", app=~\"$app\", level=~\"warning|err|error|crit|alert|emerg\"}[5m]))",
+          "legendFormat": "warn+error/sec",
+          "queryType": "range",
+          "refId": "A"
+        }
+      ],
+      "description": "Rate of warning and higher priority logs."
+    },
+    {
+      "id": 3,
+      "title": "Logs last hour",
+      "type": "stat",
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 6,
+        "x": 12,
+        "y": 0
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {},
+          "mappings": [],
+          "unit": "short",
+          "decimals": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "colorMode": "background",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "loki"
+          },
+          "expr": "sum(count_over_time({host=~\"$host\", unit=~\"$unit\", app=~\"$app\"}[1h]))",
+          "legendFormat": "logs",
+          "queryType": "range",
+          "refId": "A"
+        }
+      ],
+      "description": "Total log lines ingested in the last hour."
+    },
+    {
+      "id": 4,
+      "title": "Noisy units",
+      "type": "stat",
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 6,
+        "x": 18,
+        "y": 0
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {},
+          "mappings": [],
+          "unit": "short",
+          "decimals": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "colorMode": "background",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "loki"
+          },
+          "expr": "count(topk(10, sum by (unit) (count_over_time({host=~\"$host\", unit=~\"$unit\", app=~\"$app\"}[1h]))))",
+          "legendFormat": "units",
+          "queryType": "range",
+          "refId": "A"
+        }
+      ],
+      "description": "How many systemd units are present in the top-10 noisy-unit set."
+    },
+    {
+      "id": 5,
+      "title": "Log rate by host",
+      "type": "timeseries",
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 4
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {},
+          "mappings": [],
+          "unit": "logs/s",
+          "decimals": 2,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "loki"
+          },
+          "expr": "sum by (host) (rate({host=~\"$host\", unit=~\"$unit\", app=~\"$app\"}[5m]))",
+          "legendFormat": "{{host}}",
+          "queryType": "range",
+          "refId": "A"
+        }
+      ],
+      "description": "Compares PRM and CLD log volume over time."
+    },
+    {
+      "id": 6,
+      "title": "Warnings/errors by host and level",
+      "type": "timeseries",
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 4
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {},
+          "mappings": [],
+          "unit": "logs/s",
+          "decimals": 2,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "loki"
+          },
+          "expr": "sum by (host, level) (rate({host=~\"$host\", unit=~\"$unit\", app=~\"$app\", level=~\"warning|err|error|crit|alert|emerg\"}[5m]))",
+          "legendFormat": "{{host}} {{level}}",
+          "queryType": "range",
+          "refId": "A"
+        }
+      ],
+      "description": "Highlights noisy warning/error streams before they become incidents."
+    },
+    {
+      "id": 7,
+      "title": "Top systemd units by log volume (1h)",
+      "type": "table",
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 12
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {},
+          "mappings": [],
+          "unit": "short",
+          "decimals": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "cellHeight": "sm",
+        "footer": {
+          "countRows": false,
+          "fields": "",
+          "reducer": [
+            "sum"
+          ],
+          "show": false
+        },
+        "showHeader": true
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "loki"
+          },
+          "expr": "topk(10, sum by (unit) (count_over_time({host=~\"$host\", unit=~\"$unit\", app=~\"$app\"}[1h])))",
+          "legendFormat": "{{unit}}",
+          "queryType": "range",
+          "refId": "A"
+        }
+      ],
+      "description": "Top systemd units by raw log-line count in the last hour."
+    },
+    {
+      "id": 8,
+      "title": "Top warning/error units (1h)",
+      "type": "table",
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 12
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {},
+          "mappings": [],
+          "unit": "short",
+          "decimals": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "cellHeight": "sm",
+        "footer": {
+          "countRows": false,
+          "fields": "",
+          "reducer": [
+            "sum"
+          ],
+          "show": false
+        },
+        "showHeader": true
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "loki"
+          },
+          "expr": "topk(10, sum by (unit, level) (count_over_time({host=~\"$host\", unit=~\"$unit\", app=~\"$app\", level=~\"warning|err|error|crit|alert|emerg\"}[1h])))",
+          "legendFormat": "{{unit}} {{level}}",
+          "queryType": "range",
+          "refId": "A"
+        }
+      ],
+      "description": "Top warning/error-producing units in the last hour."
+    },
+    {
+      "id": 9,
+      "title": "Recent warnings and errors",
+      "type": "logs",
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 24,
+        "x": 0,
+        "y": 20
+      },
+      "fieldConfig": {
+        "defaults": {
+          "custom": {}
+        },
+        "overrides": []
+      },
+      "options": {
+        "dedupStrategy": "none",
+        "enableLogDetails": true,
+        "prettifyLogMessage": false,
+        "showCommonLabels": false,
+        "showLabels": false,
+        "showTime": true,
+        "sortOrder": "Descending",
+        "wrapLogMessage": true
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "loki"
+          },
+          "expr": "{host=~\"$host\", unit=~\"$unit\", app=~\"$app\", level=~\"warning|err|error|crit|alert|emerg\"}",
+          "queryType": "range",
+          "refId": "A",
+          "maxLines": 500
+        }
+      ],
+      "description": "Latest warning and higher priority logs. Use host/level variables to narrow the stream."
+    },
+    {
+      "id": 10,
+      "title": "Live filtered log stream",
+      "type": "logs",
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "gridPos": {
+        "h": 10,
+        "w": 24,
+        "x": 0,
+        "y": 28
+      },
+      "fieldConfig": {
+        "defaults": {
+          "custom": {}
+        },
+        "overrides": []
+      },
+      "options": {
+        "dedupStrategy": "none",
+        "enableLogDetails": true,
+        "prettifyLogMessage": false,
+        "showCommonLabels": false,
+        "showLabels": false,
+        "showTime": true,
+        "sortOrder": "Descending",
+        "wrapLogMessage": true
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "loki"
+          },
+          "expr": "{host=~\"$host\", unit=~\"$unit\", app=~\"$app\", level=~\"$level\"}",
+          "queryType": "range",
+          "refId": "A",
+          "maxLines": 1000
+        }
+      ],
+      "description": "General Loki stream for selected host and level values."
+    }
+  ]
+}

hosts/AZ-PRM-1/services/monitoring/dashboards/az-node-fleet-demo.json

@@ -0,0 +1,731 @@
+{
+  "annotations": {
+    "list": [
+      {
+        "builtIn": 1,
+        "datasource": {
+          "type": "grafana",
+          "uid": "-- Grafana --"
+        },
+        "enable": true,
+        "hide": true,
+        "iconColor": "rgba(0, 211, 255, 1)",
+        "name": "Annotations & Alerts",
+        "target": {
+          "limit": 100,
+          "matchAny": false,
+          "tags": [],
+          "type": "dashboard"
+        },
+        "type": "dashboard"
+      }
+    ]
+  },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 1,
+  "links": [],
+  "liveNow": false,
+  "schemaVersion": 39,
+  "style": "dark",
+  "tags": [
+    "az",
+    "demo",
+    "provisioned"
+  ],
+  "templating": {
+    "list": []
+  },
+  "time": {
+    "from": "now-6h",
+    "to": "now"
+  },
+  "timepicker": {
+    "refresh_intervals": [
+      "10s",
+      "30s",
+      "1m",
+      "5m",
+      "15m",
+      "30m",
+      "1h"
+    ]
+  },
+  "timezone": "browser",
+  "version": 1,
+  "weekStart": "",
+  "uid": "az-node-fleet-demo",
+  "title": "AZ Node Fleet Demo",
+  "refresh": "30s",
+  "description": "Demo dashboard for node_exporter metrics on AZ-PRM-1 and AZ-CLD-1. Uses the default Prometheus datasource provisioned by NixOS.",
+  "panels": [
+    {
+      "id": 1,
+      "title": "Scrape up",
+      "type": "stat",
+      "datasource": null,
+      "gridPos": {
+        "h": 4,
+        "w": 6,
+        "x": 0,
+        "y": 0
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {},
+          "mappings": [],
+          "unit": "short",
+          "decimals": 0,
+          "min": 0,
+          "max": 1,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "red",
+                "value": null
+              },
+              {
+                "color": "green",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "colorMode": "background",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "targets": [
+        {
+          "expr": "up{job=~\"node|node-cld\"}",
+          "legendFormat": "{{instance}}",
+          "refId": "A"
+        }
+      ],
+      "description": "1 means Prometheus can scrape the host."
+    },
+    {
+      "id": 2,
+      "title": "Uptime",
+      "type": "stat",
+      "datasource": null,
+      "gridPos": {
+        "h": 4,
+        "w": 6,
+        "x": 6,
+        "y": 0
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {},
+          "mappings": [],
+          "unit": "d",
+          "decimals": 1,
+          "min": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "colorMode": "background",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "targets": [
+        {
+          "expr": "(time() - node_boot_time_seconds) / 86400",
+          "legendFormat": "{{instance}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 3,
+      "title": "CPU busy",
+      "type": "gauge",
+      "datasource": null,
+      "gridPos": {
+        "h": 4,
+        "w": 6,
+        "x": 12,
+        "y": 0
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {},
+          "mappings": [],
+          "unit": "percent",
+          "decimals": 1,
+          "min": 0,
+          "max": 100,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 70
+              },
+              {
+                "color": "red",
+                "value": 90
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "showThresholdLabels": false,
+        "showThresholdMarkers": true
+      },
+      "targets": [
+        {
+          "expr": "100 - (avg by(instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[$__rate_interval])) * 100)",
+          "legendFormat": "{{instance}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 4,
+      "title": "Memory used",
+      "type": "gauge",
+      "datasource": null,
+      "gridPos": {
+        "h": 4,
+        "w": 6,
+        "x": 18,
+        "y": 0
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {},
+          "mappings": [],
+          "unit": "percent",
+          "decimals": 1,
+          "min": 0,
+          "max": 100,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 75
+              },
+              {
+                "color": "red",
+                "value": 90
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "showThresholdLabels": false,
+        "showThresholdMarkers": true
+      },
+      "targets": [
+        {
+          "expr": "100 * (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes))",
+          "legendFormat": "{{instance}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 5,
+      "title": "CPU busy by host",
+      "type": "timeseries",
+      "datasource": null,
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 4
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {},
+          "mappings": [],
+          "unit": "percent",
+          "decimals": 1,
+          "min": 0,
+          "max": 100,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 70
+              },
+              {
+                "color": "red",
+                "value": 90
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "expr": "100 - (avg by(instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[$__rate_interval])) * 100)",
+          "legendFormat": "{{instance}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 6,
+      "title": "Load average (5m)",
+      "type": "timeseries",
+      "datasource": null,
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 4
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {},
+          "mappings": [],
+          "unit": "short",
+          "decimals": 2,
+          "min": 0
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "expr": "node_load5",
+          "legendFormat": "{{instance}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 7,
+      "title": "Memory used",
+      "type": "timeseries",
+      "datasource": null,
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 12
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {},
+          "mappings": [],
+          "unit": "percent",
+          "decimals": 1,
+          "min": 0,
+          "max": 100,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 75
+              },
+              {
+                "color": "red",
+                "value": 90
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "expr": "100 * (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes))",
+          "legendFormat": "{{instance}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 8,
+      "title": "Root filesystem used",
+      "type": "timeseries",
+      "datasource": null,
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 12
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {},
+          "mappings": [],
+          "unit": "percent",
+          "decimals": 1,
+          "min": 0,
+          "max": 100,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 80
+              },
+              {
+                "color": "red",
+                "value": 92
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "expr": "100 * (1 - (node_filesystem_avail_bytes{mountpoint=\"/\",fstype!~\"tmpfs|overlay|squashfs|ramfs\"} / node_filesystem_size_bytes{mountpoint=\"/\",fstype!~\"tmpfs|overlay|squashfs|ramfs\"}))",
+          "legendFormat": "{{instance}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 9,
+      "title": "Network throughput",
+      "type": "timeseries",
+      "datasource": null,
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 20
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {},
+          "mappings": [],
+          "unit": "Bps",
+          "decimals": 1,
+          "min": 0
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "expr": "sum by(instance) (rate(node_network_receive_bytes_total{device!~\"lo|veth.*|docker.*|br-.*\"}[$__rate_interval]))",
+          "legendFormat": "{{instance}} RX",
+          "refId": "A"
+        },
+        {
+          "expr": "sum by(instance) (rate(node_network_transmit_bytes_total{device!~\"lo|veth.*|docker.*|br-.*\"}[$__rate_interval]))",
+          "legendFormat": "{{instance}} TX",
+          "refId": "B"
+        }
+      ]
+    },
+    {
+      "id": 10,
+      "title": "Disk IO",
+      "type": "timeseries",
+      "datasource": null,
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 20
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {},
+          "mappings": [],
+          "unit": "Bps",
+          "decimals": 1,
+          "min": 0
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "expr": "sum by(instance) (rate(node_disk_read_bytes_total[$__rate_interval]))",
+          "legendFormat": "{{instance}} read",
+          "refId": "A"
+        },
+        {
+          "expr": "sum by(instance) (rate(node_disk_written_bytes_total[$__rate_interval]))",
+          "legendFormat": "{{instance}} write",
+          "refId": "B"
+        }
+      ]
+    },
+    {
+      "id": 11,
+      "title": "Failed systemd units",
+      "type": "stat",
+      "datasource": null,
+      "gridPos": {
+        "h": 4,
+        "w": 12,
+        "x": 0,
+        "y": 28
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {},
+          "mappings": [],
+          "unit": "short",
+          "decimals": 0,
+          "min": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "colorMode": "background",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "targets": [
+        {
+          "expr": "sum by(instance) (node_systemd_units{state=\"failed\"})",
+          "legendFormat": "{{instance}}",
+          "refId": "A"
+        }
+      ],
+      "description": "Requires node_exporter systemd collector."
+    },
+    {
+      "id": 12,
+      "title": "Filesystem free bytes",
+      "type": "stat",
+      "datasource": null,
+      "gridPos": {
+        "h": 4,
+        "w": 12,
+        "x": 12,
+        "y": 28
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {},
+          "mappings": [],
+          "unit": "bytes",
+          "decimals": 1,
+          "min": 0
+        },
+        "overrides": []
+      },
+      "options": {
+        "colorMode": "background",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "targets": [
+        {
+          "expr": "node_filesystem_avail_bytes{mountpoint=\"/\",fstype!~\"tmpfs|overlay|squashfs|ramfs\"}",
+          "legendFormat": "{{instance}}",
+          "refId": "A"
+        }
+      ]
+    }
+  ]
+}

hosts/AZ-PRM-1/services/monitoring/default.nix

@@ -0,0 +1,12 @@
+{...}: {
+  imports = [
+    ./alerting.nix
+    ./blackbox.nix
+    ./cld-scrape.nix
+    ./exporters.nix
+    ./loki.nix
+    ./alloy.nix
+    ./grafana.nix
+    ./prometheus.nix
+  ];
+}

hosts/AZ-PRM-1/services/monitoring/exporters.nix

@@ -0,0 +1,15 @@
+{config, ...}: let
+  nodeExporterPort = config.m3ta.ports.get "node-exporter";
+in {
+  services.prometheus.exporters.node = {
+    enable = true;
+    port = nodeExporterPort;
+    listenAddress = "127.0.0.1";
+    enabledCollectors = [
+      "systemd"
+      "diskstats"
+      "filesystem"
+    ];
+    openFirewall = false; # localhost only; prometheus scrapes via 127.0.0.1
+  };
+}

hosts/AZ-PRM-1/services/monitoring/grafana.nix

@@ -0,0 +1,90 @@
+{config, ...}: let
+  serviceName = "grafana";
+  servicePort = config.m3ta.ports.get serviceName;
+  prometheusPort = config.m3ta.ports.get "prometheus";
+  lokiPort = config.m3ta.ports.get "loki";
+in {
+  services.grafana = {
+    enable = true;
+    settings = {
+      server = {
+        http_addr = "127.0.0.1";
+        http_port = servicePort;
+        domain = "g.l.az-gruppe.com";
+        root_url = "https://g.l.az-gruppe.com";
+        serve_from_sub_path = false;
+      };
+      database = {
+        type = "postgres";
+        host = "127.0.0.1";
+        name = "grafana";
+        user = "grafana";
+        password = "$__file{${config.age.secrets.grafana-db-password.path}}";
+        ssl_mode = "disable";
+      };
+      security = {
+        admin_user = "admin";
+        # Grafana file-provider: $__file{<path>} reads the raw secret from the file.
+        # The agenix secrets must contain ONLY the raw value (no KEY= prefix).
+        admin_password = "$__file{${config.age.secrets.grafana-admin-pw.path}}";
+        secret_key = "$__file{${config.age.secrets.grafana-secret-key.path}}";
+        disable_gravatar = true;
+      };
+    };
+    provision = {
+      datasources.settings.datasources = [
+        {
+          name = "Prometheus";
+          uid = "prometheus";
+          type = "prometheus";
+          url = "http://localhost:${toString prometheusPort}";
+          isDefault = true;
+          access = "proxy";
+          jsonData.timeInterval = "15s";
+        }
+        {
+          name = "Loki";
+          uid = "loki";
+          type = "loki";
+          url = "http://localhost:${toString lokiPort}";
+          access = "proxy";
+          jsonData = {
+            maxLines = 1000;
+          };
+        }
+      ];
+      dashboards.settings.providers = [
+        {
+          name = "default";
+          options.path = "/etc/grafana-dashboards";
+        }
+      ];
+    };
+  };
+
+  # Dashboards directory (Phase 1: empty placeholder, JSON files added later)
+  environment.etc."grafana-dashboards".source = ./. + "/dashboards";
+
+  systemd.services.grafana = {
+    after = ["postgresql.service"];
+    wants = ["postgresql.service"];
+  };
+
+  # Traefik configuration specific to grafana
+  services.traefik.dynamicConfigOptions.http = {
+    services.${serviceName}.loadBalancer.servers = [
+      {
+        url = "http://localhost:${toString servicePort}/";
+      }
+    ];
+
+    routers.${serviceName} = {
+      rule = "Host(`g.l.az-gruppe.com`)";
+      tls = {
+        certResolver = "ionos";
+      };
+      service = serviceName;
+      entrypoints = "websecure";
+    };
+  };
+}

hosts/AZ-PRM-1/services/monitoring/loki.nix

@@ -0,0 +1,91 @@
+{config, ...}: let
+  lokiPort = config.m3ta.ports.get "loki";
+  prmNetbirdIP = "100.91.49.26";
+  cldNetbirdIP = "100.91.203.184";
+in {
+  services.loki = {
+    enable = true;
+    dataDir = "/var/lib/loki";
+    configuration = {
+      auth_enabled = false;
+
+      server = {
+        http_listen_address = "127.0.0.1";
+        http_listen_port = lokiPort;
+      };
+
+      common = {
+        path_prefix = config.services.loki.dataDir;
+        replication_factor = 1;
+        ring = {
+          instance_addr = "127.0.0.1";
+          kvstore.store = "inmemory";
+        };
+      };
+
+      schema_config.configs = [
+        {
+          from = "2024-04-01";
+          store = "tsdb";
+          object_store = "filesystem";
+          schema = "v13";
+          index = {
+            prefix = "index_";
+            period = "24h";
+          };
+        }
+      ];
+
+      storage_config = {
+        filesystem.directory = "${config.services.loki.dataDir}/chunks";
+        tsdb_shipper = {
+          active_index_directory = "${config.services.loki.dataDir}/tsdb-index";
+          cache_location = "${config.services.loki.dataDir}/tsdb-cache";
+        };
+      };
+
+      limits_config = {
+        retention_period = "336h";
+        max_query_lookback = "336h";
+        allow_structured_metadata = true;
+        volume_enabled = true;
+        reject_old_samples = true;
+        reject_old_samples_max_age = "168h";
+      };
+
+      compactor = {
+        working_directory = "${config.services.loki.dataDir}/compactor";
+        compaction_interval = "10m";
+        retention_enabled = true;
+        retention_delete_delay = "2h";
+        retention_delete_worker_count = 50;
+        delete_request_store = "filesystem";
+      };
+
+      analytics.reporting_enabled = false;
+    };
+  };
+
+  services.traefik.staticConfigOptions.entryPoints.loki = {
+    address = "${prmNetbirdIP}:${toString lokiPort}";
+  };
+
+  services.traefik.dynamicConfigOptions.http = {
+    middlewares.loki-basic-auth.basicAuth.usersFile = config.age.secrets.monitoring-loki-htpasswd.path;
+
+    services.loki.loadBalancer.servers = [
+      {url = "http://127.0.0.1:${toString lokiPort}/";}
+    ];
+
+    routers.loki-push = {
+      rule = "PathPrefix(`/loki/api/v1/push`)";
+      entrypoints = ["loki"];
+      service = "loki";
+      middlewares = ["loki-basic-auth"];
+    };
+  };
+
+  networking.firewall.extraCommands = ''
+    iptables -A INPUT -p tcp -s ${cldNetbirdIP} --dport ${toString lokiPort} -j ACCEPT
+  '';
+}

hosts/AZ-PRM-1/services/monitoring/prometheus.nix

@@ -0,0 +1,43 @@
+{config, ...}: let
+  prometheusPort = config.m3ta.ports.get "prometheus";
+  nodeExporterPort = config.m3ta.ports.get "node-exporter";
+  kestraMetricsPort = config.m3ta.ports.get "kestra-metrics";
+in {
+  services.prometheus = {
+    enable = true;
+    port = prometheusPort;
+    listenAddress = "127.0.0.1";
+    retentionTime = "30d";
+
+    scrapeConfigs = [
+      {
+        job_name = "node";
+        static_configs = [
+          {
+            targets = ["localhost:${toString nodeExporterPort}"];
+            labels = {
+              instance = "AZ-PRM-1";
+            };
+          }
+        ];
+      }
+      {
+        job_name = "kestra";
+        metrics_path = "/prometheus";
+        scrape_interval = "15s";
+        static_configs = [
+          {
+            targets = ["localhost:${toString kestraMetricsPort}"];
+            labels = {
+              instance = "AZ-PRM-1";
+              service = "kestra";
+            };
+          }
+        ];
+      }
+      # Phase 2: add AZ-CLD-1 node target (scrape over netbird 100.x).
+      # Phase 2: add blackbox_exporter, podman/cadvisor targets.
+      # Phase 2: enable remote-write receiver (extraFlags) for Windows clients.
+    ];
+  };
+}

hosts/AZ-PRM-1/services/n8n.nix

@@ -0,0 +1,61 @@
+{config, ...}: let
+  serviceName = "n8n";
+  servicePort = config.m3ta.ports.get serviceName;
+  sandboxApiPort = config.m3ta.ports.get "n8n-sandbox-api";
+  sambaMounts = [
+    "/mnt/AutoAblage"
+    "/mnt/SkriptHelper"
+    "/mnt/DMS-ALT-INBOX"
+    "/mnt/DMS-INBOX"
+  ];
+in {
+  services.n8n = {
+    enable = true;
+    environment = {
+      WEBHOOK_URL = "https://wf.l.az-gruppe.com";
+      NODES_EXCLUDE = "[]";
+      N8N_RESTRICT_FILE_ACCESS_TO = builtins.concatStringsSep ";" sambaMounts;
+      N8N_RUNNERS_ENABLED = true;
+      N8N_NATIVE_PYTHON_RUNNER = true;
+      N8N_RUNNERS_AUTH_TOKEN_FILE = config.age.secrets.n8n-runner-auth-token.path;
+      N8N_LOG_OUTPUT = "json";
+      N8N_LOG_LEVEL = "info";
+      N8N_ENABLED_MODULES = "instance-ai";
+      N8N_INSTANCE_AI_SANDBOX_ENABLED = true;
+      N8N_INSTANCE_AI_SANDBOX_PROVIDER = "n8n-sandbox";
+      N8N_INSTANCE_AI_SANDBOX_IMAGE = "ghcr.io/n8n-io/n8n-sandbox-service-sandbox:latest";
+      N8N_INSTANCE_AI_SANDBOX_API_URL = "http://127.0.0.1:${toString sandboxApiPort}";
+      N8N_SANDBOX_SERVICE_URL = "http://127.0.0.1:${toString sandboxApiPort}";
+    };
+    taskRunners.enable = true;
+  };
+
+  systemd.services.n8n = {
+    serviceConfig = {
+      EnvironmentFile = ["${config.age.secrets.n8n-env.path}" "${config.age.secrets.n8n-sandbox-env.path}"];
+      ReadWritePaths = sambaMounts;
+    };
+    wants = ["podman-sandbox-api.service"];
+    after = ["podman-sandbox-api.service"];
+  };
+
+  systemd.services.n8n-task-runner.serviceConfig.ReadWritePaths = sambaMounts;
+
+  # Traefik configuration specific to n8n
+  services.traefik.dynamicConfigOptions.http = {
+    services.${serviceName}.loadBalancer.servers = [
+      {
+        url = "http://localhost:${toString servicePort}/";
+      }
+    ];
+
+    routers.${serviceName} = {
+      rule = "Host(`wf.l.az-gruppe.com`)";
+      tls = {
+        certResolver = "ionos";
+      };
+      service = serviceName;
+      entrypoints = "websecure";
+    };
+  };
+}

hosts/AZ-PRM-1/services/netbird.nix

@@ -0,0 +1,31 @@
+{pkgs, ...}: {
+  services.netbird = {
+    enable = true;
+    package = pkgs.unstable.netbird;
+  };
+
+  systemd.services.netbird = {
+    environment = {
+      NB_DISABLE_SSH_CONFIG = "true";
+    };
+    path = [
+      pkgs.shadow
+      pkgs.util-linux
+    ];
+  };
+
+  programs.ssh.extraConfig = ''
+    Match exec "${pkgs.netbird}/bin/netbird ssh detect %h %p"
+        PreferredAuthentications password,publickey,keyboard-interactive
+        PasswordAuthentication yes
+        PubkeyAuthentication yes
+        BatchMode no
+        ProxyCommand ${pkgs.netbird}/bin/netbird ssh proxy %h %p
+        StrictHostKeyChecking no
+        UserKnownHostsFile /dev/null
+        CheckHostIP no
+        LogLevel ERROR
+  '';
+
+  networking.firewall.checkReversePath = "loose";
+}

hosts/AZ-PRM-1/services/netbox.nix

@@ -0,0 +1,120 @@
+{
+  config,
+  pkgs,
+  ...
+}: let
+  serviceName = "netbox";
+  servicePort = config.m3ta.ports.get serviceName;
+  staticPort = config.m3ta.ports.get "netbox-static";
+  hostName = "nb.l.az-gruppe.com";
+  staticRoot = "${config.services.netbox.dataDir}/static";
+  # nixpkgs >= 25.11: Plugin-Pakete liegen im passthru-Set netbox.plugins
+  # (python3Packages.netbox-* sind Throw-Stubs); pluginName = NetBox-Modulname
+  netboxPlugins = with pkgs.unstable.netbox.plugins; [
+    netbox-dns
+    netbox-qrcode
+    netbox-routing
+    netbox-topology-views
+    netbox-floorplan-plugin
+  ];
+in {
+  services.netbox = {
+    enable = true;
+    package = pkgs.unstable.netbox;
+    listenAddress = "127.0.0.1";
+    port = servicePort;
+    secretKeyFile = config.age.secrets.netbox-secret-key.path;
+    apiTokenPeppersFile = config.age.secrets.netbox-api-token-pepper.path;
+
+    settings = {
+      ALLOWED_HOSTS = [hostName "localhost" "127.0.0.1"];
+      SECURE_SSL_REDIRECT = true;
+      SESSION_COOKIE_SECURE = true;
+      CSRF_COOKIE_SECURE = true;
+      # NetBox aktiviert Plugins erst über PLUGINS in configuration.py —
+      # der NixOS-NetBox-Modul trägt sie NICHT automatisch ein.
+      PLUGINS = map (p: p.pluginName) netboxPlugins;
+    };
+    # installiert die Pakete in NetBox' Python-Environment (extraBuildInputs)
+    plugins = _: netboxPlugins;
+    extraConfig = ''
+      SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")
+    '';
+  };
+
+  # Der Upgrade-Guard im NetBox-Modul (preStart) vergleicht /var/lib/netbox/version
+  # nur mit dem BASISPAKET (services.netbox.package). Reine Config-/Plugin-
+  # Änderungen bei gleicher NetBox-Version überspringen damit migrate +
+  # collectstatic → Plugin-Tabellen fehlen ("database migration missing").
+  # Deshalb Migrationen/Statics hier VOR jedem netbox.service-Start ausführen
+  # (idempotent, entspricht dem offiziellen NetBox-Upgrade-Pfad):
+  systemd.services.netbox = {
+    wants = ["netbox-migrate.service"];
+    after = ["netbox-migrate.service"];
+  };
+
+  systemd.services.netbox-migrate = {
+    description = "NetBox: DB-Migrationen & Statics (v. a. Plugins)";
+    wants = ["network-online.target"];
+    after = ["network-online.target" "postgresql.service" "redis-netbox.service"];
+    serviceConfig = {
+      Type = "oneshot";
+      User = "netbox";
+      Group = "netbox";
+      StateDirectory = "netbox";
+      TimeoutStartSec = "10min";
+    };
+    script = ''
+      /run/current-system/sw/bin/netbox-manage migrate --no-input
+      /run/current-system/sw/bin/netbox-manage collectstatic --no-input
+    '';
+  };
+
+  services.nginx = {
+    enable = true;
+    virtualHosts.netbox-static = {
+      listen = [
+        {
+          addr = "127.0.0.1";
+          port = staticPort;
+        }
+      ];
+      locations."/static/" = {
+        alias = "${staticRoot}/";
+        extraConfig = ''
+          expires 1h;
+          access_log off;
+        '';
+      };
+    };
+  };
+
+  users.users.nginx.extraGroups = ["netbox"];
+
+  # Traefik-Routing: zwei Backends (App + Static), ein Host
+  services.traefik.dynamicConfigOptions.http = {
+    services = {
+      ${serviceName}.loadBalancer.servers = [
+        {url = "http://127.0.0.1:${toString servicePort}/";}
+      ];
+      "${serviceName}-static".loadBalancer.servers = [
+        {url = "http://127.0.0.1:${toString staticPort}/";}
+      ];
+    };
+
+    routers = {
+      ${serviceName} = {
+        rule = "Host(`${hostName}`) && !PathPrefix(`/static`)";
+        tls.certResolver = "ionos";
+        service = serviceName;
+        entrypoints = "websecure";
+      };
+      "${serviceName}-static" = {
+        rule = "Host(`${hostName}`) && PathPrefix(`/static`)";
+        tls.certResolver = "ionos";
+        service = "${serviceName}-static";
+        entrypoints = "websecure";
+      };
+    };
+  };
+}

hosts/AZ-PRM-1/services/pgadmin.nix

@@ -0,0 +1,21 @@
+{config, ...}: let
+  serviceName = "pgadmin";
+  servicePort = config.m3ta.ports.get serviceName;
+in {
+  services.pgadmin = {
+    enable = true;
+    initialPasswordFile = "${config.age.secrets.pgadmin-pw.path}";
+    initialEmail = "sascha.koenig@azintec.com";
+  };
+
+  # Traefik configuration specific to pgadmin
+  services.traefik.dynamicConfigOptions.http = {
+    services.pgadmin.loadBalancer.servers = [{url = "http://localhost:${toString servicePort}/";}];
+    routers.pgadmin = {
+      rule = "Host(`pg.l.az-gruppe.com`)";
+      tls.certResolver = "ionos";
+      service = "pgadmin";
+      entrypoints = "websecure";
+    };
+  };
+}

hosts/AZ-PRM-1/services/phishboard.nix

@@ -0,0 +1,32 @@
+{
+  config,
+  inputs,
+  pkgs,
+  ...
+}: let
+  serviceName = "phishboard";
+  servicePort = config.m3ta.ports.get serviceName;
+in {
+  services.phishboard = {
+    enable = true;
+    package = inputs.phishboard.packages.${pkgs.stdenv.hostPlatform.system}.default;
+    host = "127.0.0.1";
+    port = servicePort;
+    environmentFile = config.age.secrets.phishboard-env.path;
+  };
+
+  services.traefik.dynamicConfigOptions.http = {
+    services.${serviceName}.loadBalancer.servers = [
+      {url = "http://localhost:${toString servicePort}/";}
+    ];
+
+    routers.${serviceName} = {
+      rule = "Host(`pb.l.az-gruppe.com`)";
+      tls = {
+        certResolver = "ionos";
+      };
+      service = serviceName;
+      entrypoints = "websecure";
+    };
+  };
+}

hosts/AZ-PRM-1/services/postgres.nix

@@ -0,0 +1,78 @@
+{
+  config,
+  pkgs,
+  ...
+}: {
+  services.postgresql = {
+    enable = true;
+    enableTCPIP = true;
+    package = pkgs.postgresql_17;
+    settings = {
+      ssl = true;
+      ssl_cert_file = config.age.secrets.pg-cert.path;
+      ssl_key_file = config.age.secrets.pg-key.path;
+    };
+    extensions = with pkgs.postgresql17Packages; [
+      pgvector
+    ];
+    initialScript = pkgs.writeText "backend-initScript" ''
+      CREATE USER baserow WITH ENCRYPTED PASSWORD 'baserow';
+      CREATE DATABASE baserow;
+      ALTER DATABASE baserow OWNER to baserow;
+
+      CREATE USER kestra WITH ENCRYPTED PASSWORD 'kestra';
+      CREATE DATABASE kestra;
+      ALTER DATABASE kestra OWNER to kestra;
+
+      CREATE USER n8n WITH ENCRYPTED PASSWORD 'n8n';
+      CREATE DATABASE n8n;
+      ALTER DATABASE n8n OWNER to n8n;
+
+      CREATE USER vaultwarden WITH ENCRYPTED PASSWORD 'n8n';
+      CREATE DATABASE vaultwarden;
+      ALTER DATABASE vaultwarden OWNER to vaultwarden;
+
+      CREATE USER grafana WITH ENCRYPTED PASSWORD 'grafana';
+      CREATE DATABASE grafana;
+      ALTER DATABASE grafana OWNER to grafana;
+    '';
+    authentication = pkgs.lib.mkOverride 10 ''
+      # Local connections (Unix socket)
+      local   all         postgres    peer
+      local   netbox      netbox      peer
+      local   n8n         n8n         scram-sha-256
+
+      # Localhost connections (IPv4 and IPv6)
+      host    all          postgres     127.0.0.1/32    scram-sha-256
+      host    all          postgres     ::1/128         scram-sha-256
+
+      host    n8n          n8n          127.0.0.1/32    scram-sha-256
+      host    n8n          n8n          ::1/128         scram-sha-256
+
+      # Podman network connections
+      host    baserow      baserow      10.89.0.0/24    scram-sha-256
+      host    kestra       kestra       10.89.0.0/24    scram-sha-256
+      host    atrocore     atrocore     10.89.0.0/24    scram-sha-256
+      host    semaphore    semaphore    10.89.0.0/24    scram-sha-256
+
+      # Grafana (local socket / localhost only)
+      host    grafana      grafana      127.0.0.1/32    scram-sha-256
+      host    grafana      grafana      ::1/128         scram-sha-256
+
+      # Deny all other connections
+      host    all       all       0.0.0.0/0      reject
+      host    all       all       ::/0           reject
+    '';
+  };
+  services.postgresqlBackup = {
+    enable = true;
+    startAt = "03:10:00";
+    databases = ["atrocore" "baserow" "kestra" "n8n" "netbox" "semaphore"];
+  };
+  networking.firewall = {
+    extraCommands = ''
+      iptables -A INPUT -p tcp -s 127.0.0.1 --dport 5432 -j ACCEPT
+      iptables -A INPUT -p tcp -s 10.89.0.0/24 --dport 5432 -j ACCEPT
+    '';
+  };
+}

hosts/AZ-PRM-1/services/printing.nix

@@ -0,0 +1,40 @@
+{pkgs, ...}: {
+  # CUPS Druckdienst für PDF-Druck aus n8n
+  # Drucker: Kyocera TASKalfa 4054ci @ 192.168.152.137
+  services.printing = {
+    enable = true;
+    drivers = with pkgs; [
+      cups-filters # driverless IPP Everywhere Support
+    ];
+  };
+
+  # Avahi für mDNS/IPP-Druckererkennung
+  services.avahi = {
+    enable = true;
+    nssmdns4 = true;
+    openFirewall = true;
+  };
+
+  # Kyocera TASKalfa 4054ci deklarativ einrichten
+  hardware.printers = {
+    ensurePrinters = [
+      {
+        name = "JW2OG";
+        location = "Buero";
+        description = "Kyocera TASKalfa 4054ci";
+        deviceUri = "ipps://192.168.152.137:443/ipp/print";
+        model = "everywhere";
+        ppdOptions = {
+          PageSize = "A4";
+        };
+      }
+    ];
+    ensureDefaultPrinter = "JW2OG";
+  };
+
+  # n8n braucht Zugriff auf lp/lpr zum Drucken
+  systemd.services.n8n = {
+    path = [pkgs.cups];
+    serviceConfig.SupplementaryGroups = ["lp"];
+  };
+}

hosts/AZ-PRM-1/services/samba.nix

@@ -0,0 +1,7 @@
+{pkgs, ...}: {
+  services.samba = {
+    enable = true;
+    package = pkgs.samba4Full;
+    openFirewall = true;
+  };
+}

hosts/AZ-PRM-1/services/traefik-routing.nix

@@ -0,0 +1,31 @@
+{
+  services.traefik.dynamicConfigOptions.http = {
+    services.ptrg.loadBalancer.servers = [{url = "http://192.168.152.102:7784/";}];
+
+    routers.prtg = {
+      rule = "Host(`m.l.az-gruppe.com`)";
+      tls = {certResolver = "ionos";};
+      service = "ptrg";
+      entrypoints = "websecure";
+    };
+
+    services.AZHA.loadBalancer.servers = [{url = "http://192.168.152.47:8123/";}];
+    routers.AZHA = {
+      rule = "Host(`ha.l.az-gruppe.com`)";
+      tls = {certResolver = "ionos";};
+      service = "AZHA";
+      entrypoints = "websecure";
+    };
+    services.AZDESK.loadBalancer.servers = [
+      {
+        url = "https://azdesk.az-group.local:443/";
+      }
+    ];
+    routers.AZDESK = {
+      rule = "Host(`it-ticket.l.az-gruppe.com`)";
+      tls = {certResolver = "ionos";};
+      service = "AZDESK";
+      entrypoints = "websecure";
+    };
+  };
+}

hosts/AZ-PRM-1/services/traefik.nix

@@ -0,0 +1,92 @@
+{config, ...}: let
+  httpPort = config.m3ta.ports.get "traefik";
+  httpsPort = config.m3ta.ports.get "traefik-ssl";
+in {
+  services.traefik = {
+    enable = true;
+    staticConfigOptions = {
+      log = {level = "WARN";};
+      serversTransport.insecureSkipVerify = true;
+      certificatesResolvers = {
+        ionos = {
+          acme = {
+            email = "sascha.koenig@azintec.com";
+            storage = "/var/lib/traefik/acme.json";
+            caserver = "https://acme-v02.api.letsencrypt.org/directory";
+            dnsChallenge = {
+              provider = "ionos";
+              resolvers = ["1.1.1.1:53" "8.8.8.8:53"];
+              propagation = {
+                delayBeforeChecks = 60;
+                disableChecks = true;
+              };
+            };
+          };
+        };
+      };
+      api = {};
+      entryPoints = {
+        web = {
+          address = ":${toString httpPort}";
+          http.redirections.entryPoint = {
+            to = "websecure";
+            scheme = "https";
+          };
+        };
+        websecure = {
+          address = ":${toString httpsPort}";
+          http.tls = {
+            certResolver = "ionos";
+            domains = [
+              {
+                main = "l.az-gruppe.com";
+                sans = ["*.l.az-gruppe.com"];
+              }
+            ];
+          };
+        };
+      };
+    };
+    dynamicConfigOptions = {
+      http = {
+        services = {
+          dummy = {
+            loadBalancer.servers = [
+              {url = "http://192.168.0.1";}
+            ];
+          };
+        };
+        middlewares = {
+          auth = {
+            basicAuth = {
+              users = ["sascha.koenig:$apr1$1xqdta2b$DIVNvvp5iTUGNccJjguKh."];
+            };
+          };
+        };
+        routers = {
+          api = {
+            rule = "Host(`r.l.az-gruppe.com`)";
+            service = "api@internal";
+            middlewares = ["auth"];
+            entrypoints = ["websecure"];
+            tls = {
+              certResolver = "ionos";
+              domains = [
+                {
+                  main = "l.az-gruppe.com";
+                  sans = ["*.l.az-gruppe.com"];
+                }
+              ];
+            };
+          };
+        };
+      };
+    };
+  };
+
+  systemd.services.traefik.serviceConfig = {
+    EnvironmentFile = ["${config.age.secrets.traefik-env.path}"];
+  };
+
+  networking.firewall.allowedTCPPorts = [httpPort httpsPort];
+}

hosts/AZ-PRM-1/services/zugferd.nix

@@ -0,0 +1,10 @@
+{config, ...}: let
+  serviceName = "zugferd-service";
+  zugferdPort = config.m3ta.ports.get serviceName;
+in {
+  services.${serviceName} = {
+    enable = true;
+    port = zugferdPort;
+    host = "127.0.0.1";
+  };
+}

hosts/common/default.nix

@@ -0,0 +1,76 @@
+# Common configuration for all hosts
+{
+  config,
+  pkgs,
+  lib,
+  inputs,
+  outputs,
+  system,
+  ...
+}: {
+  imports = [
+    ./extraServices
+    ./users
+    ./ports.nix
+    inputs.m3ta-nixpkgs.nixosModules.ports
+    inputs.home-manager.nixosModules.home-manager
+  ];
+
+  home-manager = {
+    useGlobalPkgs = true;
+    useUserPackages = true;
+    extraSpecialArgs = {
+      inherit inputs outputs system;
+      videoDrivers = config.services.xserver.videoDrivers or [];
+    };
+  };
+
+  nixpkgs = {
+    # You can add overlays here
+    overlays = [
+      # Add overlays your own flake exports (from overlays and pkgs dir):
+      outputs.overlays.additions
+      outputs.overlays.modifications
+      outputs.overlays.unstable-packages
+
+      inputs.nur.overlays.default
+      inputs.m3ta-nixpkgs.overlays.default
+
+      (outputs.lib.mkLlmAgentsOverlay system)
+      # You can also add overlays exported from other flakes:
+      # neovim-nightly-overlay.overlays.default
+
+      # Or define it inline, for example:
+      # (final: prev: {
+      #   hi = final.hello.overrideAttrs (oldAttrs: {
+      #     patches = [ ./change-hello-to-hi.patch ];
+      #   });
+      # })
+    ];
+    # Configure your nixpkgs instance
+    config = {
+      # Disable if you don't want unfree packages
+      allowUnfree = true;
+    };
+  };
+
+  nix = {
+    settings = {
+      experimental-features = "nix-command flakes";
+      trusted-users = [
+        "root"
+        "sascha.koenig"
+        "jannik.mueller"
+      ]; # Set users that are allowed to use the flake command
+    };
+    gc = {
+      automatic = true;
+      options = "--delete-older-than 30d";
+    };
+    optimise.automatic = true;
+    registry =
+      (lib.mapAttrs (_: flake: {inherit flake;}))
+      ((lib.filterAttrs (_: lib.isType "flake")) inputs);
+    nixPath = ["/etc/nix/path"];
+  };
+}

hosts/common/extraServices/default.nix

@@ -0,0 +1,8 @@
+{
+  imports = [
+    ./flatpak.nix
+    ./ollama.nix
+    ./podman.nix
+    ./virtualisation.nix
+  ];
+}

hosts/common/extraServices/flatpak.nix

@@ -0,0 +1,23 @@
+{
+  config,
+  lib,
+  pkgs,
+  ...
+}:
+with lib; let
+  cfg = config.extraServices.flatpak;
+in {
+  options.extraServices.flatpak.enable = mkEnableOption "enable flatpak";
+
+  config = mkIf cfg.enable {
+    services.flatpak.enable = true;
+    xdg.portal = {
+      # xdg desktop intergration (required for flatpak)
+      enable = true;
+      extraPortals = with pkgs; [
+        xdg-desktop-portal-hyprland
+      ];
+      config.common.default = "*";
+    };
+  };
+}

hosts/common/extraServices/ollama.nix

@@ -0,0 +1,27 @@
+{
+  config,
+  lib,
+  pkgs,
+  ...
+}:
+with lib; let
+  cfg = config.extraServices.ollama;
+in {
+  options.extraServices.ollama.enable = mkEnableOption "enable ollama";
+
+  config = mkIf cfg.enable {
+    services.ollama = {
+      enable = true;
+      package = pkgs.ollama-vulkan;
+      host = "[::]";
+      openFirewall = true;
+      environmentVariables = {
+        OLLAMA_HOST = "0.0.0.0";
+      };
+    };
+    nixpkgs.config = {
+      rocmSupport = config.services.xserver.videoDrivers == ["amdgpu"];
+      cudaSupport = config.services.xserver.videoDrivers == ["nvidia"];
+    };
+  };
+}

hosts/common/extraServices/podman.nix

@@ -0,0 +1,33 @@
+{
+  config,
+  lib,
+  pkgs,
+  ...
+}:
+with lib; let
+  cfg = config.extraServices.podman;
+in {
+  options.extraServices.podman.enable = mkEnableOption "enable podman";
+
+  config = mkIf cfg.enable {
+    virtualisation = {
+      podman = {
+        enable = true;
+        dockerCompat = true;
+        dockerSocket.enable = true;
+        autoPrune = {
+          enable = true;
+          dates = "weekly";
+          flags = [
+            "--filter=until=24h"
+            "--filter=label!=important"
+          ];
+        };
+        defaultNetwork.settings.dns_enabled = true;
+      };
+    };
+    environment.systemPackages = with pkgs; [
+      podman-compose
+    ];
+  };
+}

hosts/common/extraServices/virtualisation.nix

@@ -0,0 +1,42 @@
+{
+  config,
+  lib,
+  pkgs,
+  ...
+}:
+with lib; let
+  cfg = config.extraServices.virtualisation;
+in {
+  options.extraServices.virtualisation.enable = mkEnableOption "enable virtualisation";
+
+  config = mkIf cfg.enable {
+    virtualisation = {
+      libvirtd = {
+        enable = true;
+        qemu = {
+          package = pkgs.qemu_kvm;
+          runAsRoot = true;
+          swtpm.enable = true;
+        };
+      };
+    };
+    programs.virt-manager.enable = true;
+    environment = {
+      systemPackages = [pkgs.qemu];
+      etc = {
+        "ovmf/OVMF_CODE.fd" = {
+          source = "${(pkgs.OVMF.override {
+            secureBoot = true;
+            tpmSupport = true;
+          }).fd}/FV/OVMF_CODE.fd";
+        };
+        "ovmf/OVMF_VARS.fd" = {
+          source = "${(pkgs.OVMF.override {
+            secureBoot = true;
+            tpmSupport = true;
+          }).fd}/FV/OVMF_VARS.fd";
+        };
+      };
+    };
+  };
+}

hosts/common/ports.nix

@@ -0,0 +1,87 @@
+{config, ...}: {
+  m3ta.ports = {
+    enable = true;
+
+    definitions = {
+      ssh = 2022;
+
+      traefik = 80;
+      traefik-ssl = 443;
+
+      gitea = 3030;
+      outline = 3031;
+      vaultwarden = 3032;
+      ntfy-sh = 3033;
+      zammad = 3034;
+      it-tools = 3035;
+      zammad-hr = 3036;
+      zammad-hr-elasticsearch = 3037;
+      netbird = 3038;
+      azion-scheduler = 3039;
+      azion-scheduler-proxy = 3049;
+      phishboard = 3055;
+      homarr = 3057;
+      atrocore = 3058;
+      semaphore = 3059;
+
+      metabase = 3013;
+      baserow = 3050;
+      frappe-lms = 3052;
+      snipe-it = 3053;
+      azess = 3054;
+
+      librechat = 3040;
+      librechat-dev = 3141;
+      rag-api = 8000;
+      rag-api-dev = 8100;
+      litellm = 4000;
+
+      n8n = 5678;
+      n8n-sandbox-api = 5082;
+      netbox = 8001;
+      netbox-static = 8002;
+      kestra = 5080;
+      kestra-metrics = 5081;
+      zugferd-service = 5060;
+      gotenberg = 5070;
+
+      portainer = 9000;
+
+      postgres = 5432;
+      pgbouncer-tx = 6432;
+      pgbouncer-session = 6433;
+      pgadmin = 5050;
+      mysql = 3306;
+
+      # Observability stack (AZ-PRM-1)
+      grafana = 3060;
+      prometheus = 9090;
+      loki = 3100;
+      alloy = 12345;
+      node-exporter = 9100;
+      blackbox-exporter = 9115;
+    };
+
+    hostOverrides = {
+      AZ-CLD-1 = {
+        baserow = 3050;
+        librechat-dev = 3141;
+        rag-api-dev = 8100;
+      };
+
+      AZ-PRM-1 = {
+        baserow = 3051;
+        kestra = 5080;
+        stirling-pdf = 3032;
+        bpi = 3033;
+        excalidraw = 3034;
+        online3dviewer = 3035;
+      };
+    };
+  };
+
+  environment.etc."info/all-ports.json".text = builtins.toJSON {
+    hostname = config.networking.hostName;
+    ports = config.m3ta.ports.all;
+  };
+}

hosts/common/users/default.nix

@@ -0,0 +1,6 @@
+{
+  imports = [
+    ./jannik.mueller.nix
+    ./sascha.koenig.nix
+  ];
+}

hosts/common/users/jannik.mueller.nix

@@ -0,0 +1,25 @@
+{
+  config,
+  pkgs,
+  inputs,
+  ...
+}: {
+  users.users."jannik.mueller" = {
+    hashedPassword = "$y$j9T$09RgD3AU3PK9Oi6JGLe0V1$i8J2ZOD1h1b6Zpw28ub.kExujoDKHzokeXzkM23Tfd/";
+    isNormalUser = true;
+    extraGroups = [
+      "wheel"
+      "networkmanager"
+      "libvirtd"
+      "flatpak"
+      "plugdev"
+      "input"
+      "kvm"
+      "qemu-libvirtd"
+    ];
+    openssh.authorizedKeys.keys = [
+      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPvZazSuIoWoRWhkAqQDMLeurxVUyy1MTllp1wfw1tzq"
+    ];
+    packages = [inputs.home-manager.packages.${pkgs.stdenv.hostPlatform.system}.default];
+  };
+}

hosts/common/users/sascha.koenig.nix

@@ -0,0 +1,124 @@
+# hosts/common/users/m3tam3re.nix — Central user definition with m3ta-home integration.
+#
+# This module:
+#   1. Creates the m3tam3re NixOS user
+#   2. Loads the m3ta-home profile system via mkHome
+#   3. Sets per-host feature flags based on a host profile mapping
+#   4. Imports per-host home.nix overrides (monitors, HW-specific config)
+#
+# To add a new host:
+#   1. Add entry to hostProfiles below
+#   2. Add feature flags in the hostFlags section
+#   3. Create hosts/<hostname>/home.nix if the host needs overrides (monitors, etc.)
+{
+  config,
+  pkgs,
+  inputs,
+  ...
+}: let
+  hostname = config.networking.hostName;
+
+  # ── Per-host profile mapping ──
+  # Determines which m3ta-home context and sets each host gets.
+  hostProfiles = {
+    # ── Server hosts ──
+    AZ-CLD-1 = {
+      context = "server";
+      sets = [];
+    };
+    AZ-PRM-1 = {
+      context = "server";
+      sets = [];
+    };
+  };
+
+  profile =
+    hostProfiles.${
+      hostname
+    } or {
+      context = "server";
+      sets = [];
+    };
+  m3ta-lib = inputs.m3ta-home.lib;
+
+  # Check if a per-host home.nix exists
+  hostHomeFile = ./../../${hostname}/home.nix;
+  hostHomeExists = builtins.pathExists hostHomeFile;
+
+  # ── Per-host feature flags ──
+  # These enable/disable specific m3ta-home modules per host.
+  hostFlags =
+    if hostname == "AZ-CLD-1"
+    then {
+      # Full desktop workstation
+      base = {
+        shell = {
+          fish.enable = true;
+          nushell.enable = true;
+          starship.enable = true;
+        };
+        cliTools = {
+          fzf.enable = true;
+          nitch.enable = true;
+          television.enable = true;
+        };
+        secrets.enable = false;
+      };
+    }
+    else {
+      base = {
+        shell = {
+          fish.enable = true;
+          starship.enable = true;
+        };
+        cliTools = {
+          fzf.enable = true;
+          nitch.enable = true;
+        };
+      };
+    };
+in {
+  # ── NixOS user definition ──
+  users.users."sascha.koenig" = {
+    hashedPassword = "$y$j9T$ORX4btVZgs9Xjq2oIvzJm0$lXiPwaa0D6t.eMDIx1UBesEAMOkWXBoGwpeI7X0aS8D";
+    isNormalUser = true;
+    shell = pkgs.nushell;
+    extraGroups = [
+      "wheel"
+      "networkmanager"
+      "libvirtd"
+      "flatpak"
+      "plugdev"
+      "input"
+      "kvm"
+      "qemu-libvirtd"
+    ];
+    openssh.authorizedKeys.keys = [
+      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEZbg/Z9mnflXuLahGY8WOSBMqbgeqVIkIwRkquys1Ml sascha.koenig@azintec.com"
+      "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3YEmpYbM+cpmyD10tzNRHEn526Z3LJOzYpWEKdJg8DaYyPbDn9iyVX30Nja2SrW4Wadws0Y8DW+Urs25/wVB6mKl7jgPJVkMi5hfobu3XAz8gwSdjDzRSWJrhjynuaXiTtRYED2INbvjLuxx3X8coNwMw58OuUuw5kNJp5aS2qFmHEYQErQsGT4MNqESe3jvTP27Z5pSneBj45LmGK+RcaSnJe7hG+KRtjuhjI7RdzMeDCX73SfUsal+rHeuEw/mmjYmiIItXhFTDn8ZvVwpBKv7xsJG90DkaX2vaTk0wgJdMnpVIuIRBa4EkmMWOQ3bMLGkLQeK/4FUkNcvQ/4+zcZsg4cY9Q7Fj55DD41hAUdF6SYODtn5qMPsTCnJz44glHt/oseKXMSd556NIw2HOvihbJW7Rwl4OEjGaO/dF4nUw4c9tHWmMn9dLslAVpUuZOb7ykgP0jk79ldT3Dv+2Hj0CdAWT2cJAdFX58KQ9jUPT3tBnObSF1lGMI7t77VU= m3tam3re@MBP-Sascha.fritz.box"
+    ];
+    packages = [inputs.home-manager.packages.${pkgs.stdenv.hostPlatform.system}.default];
+  };
+
+  # ── Home-Manager configuration via m3ta-home ──
+  home-manager.users."sascha.koenig" = {
+    home.stateVersion = "25.11";
+    imports =
+      [
+        # Load m3ta-home composition engine
+        (m3ta-lib.mkHome {
+          user = "m3tam3re";
+          identity = "work";
+          inherit (profile) context sets;
+        })
+        # Per-host feature flags
+        hostFlags
+      ]
+      # Per-host home.nix (Hyprland monitors, XDG/MIME, HW-specific overrides)
+      ++ (
+        if hostHomeExists
+        then [hostHomeFile]
+        else []
+      );
+  };
+}

overlays/default.nix

@@ -0,0 +1,52 @@
+{inputs, ...}: {
+  # This one brings our custom packages from the 'pkgs' directory
+  additions = final: prev:
+    (import ../pkgs {
+      pkgs = final;
+      atrocore-docker = inputs.atrocore-docker;
+    })
+    // {
+      zugferd-service = inputs.zugferd-service.packages.${prev.stdenv.hostPlatform.system}.default;
+    };
+
+  # This one contains whatever you want to overlay
+  # You can change versions, add patches, set compilation flags, anything really.
+  # https://nixos.wiki/wiki/Overlays
+  modifications = final: prev: {
+    # n8n = import ./mods/n8n.nix {inherit prev;};
+    snipe-it = import ./mods/snipe-it.nix {inherit prev;};
+    vivaldi = prev.vivaldi.override {
+      commandLineArgs = "--enable-features=UseOzonePlatform --ozone-platform=wayland";
+    };
+    # example = prev.example.overrideAttrs (oldAttrs: rec {
+    # ...
+    # });
+  };
+
+  stable-packages = final: _prev: {
+    stable = import inputs.nixpkgs {
+      system = final.stdenv.hostPlatform.system;
+      config.allowUnfree = true;
+    };
+  };
+  unstable-packages = final: _prev: {
+    unstable = import inputs.nixpkgs-unstable {
+      system = final.stdenv.hostPlatform.system;
+      config.allowUnfree = true;
+    };
+  };
+  # Flatten llm-agents packages into top-level pkgs namespace.
+  # Keep plain derivations only. llm-agents exports `buildNpmPackage` as an
+  # emptyDirectory derivation ("buildNpmPackage-guard") carrying a __functor
+  # that forwards to its OWN nixpkgs pin's builder. Flattening it would shadow
+  # nixpkgs' pkgs.buildNpmPackage and break unrelated packages — e.g.
+  # vaultwarden-webvault (npmDepsFetcherVersion = 3) failed with
+  # "fetchNpmDeps: fetcher version must be one of: 1, 2." because llm-agents'
+  # pinned fetchNpmDeps only allows 1 and 2. Real packages never have a
+  # __functor, so this filter drops such traps generically.
+  mkLlmAgentsOverlay = system: _final: _prev:
+    inputs.nixpkgs.lib.filterAttrs (
+      _: v: inputs.nixpkgs.lib.isDerivation v && !v ? __functor
+    )
+    (inputs.llm-agents.packages.${system} or {});
+}

overlays/mods/n8n.nix

@@ -0,0 +1,40 @@
+# Adds `libphonenumber-js` to n8n's node_modules so it can be require()'d
+# from Code nodes running in the Task Runner.
+#
+# Prerequisite on the n8n service:
+#   N8N_RUNNERS_EXTERNAL_ALLOW = "libphonenumber-js";
+#
+# libphonenumber-js has zero runtime + peer dependencies, so a plain tarball
+# unpack into the shared node_modules hierarchy is sufficient.
+{prev}: let
+  libphonenumber-js = prev.stdenv.mkDerivation rec {
+    pname = "libphonenumber-js";
+    version = "1.13.8";
+
+    src = prev.fetchurl {
+      url = "https://registry.npmjs.org/${pname}/-/${pname}-${version}.tgz";
+      hash = "sha256-SysWDKlbXgbe441Sd4pO+k+F1y/UC49a1KL+DcFWBIA=";
+    };
+
+    dontConfigure = true;
+    dontBuild = true;
+
+    installPhase = ''
+      runHook preInstall
+      mkdir -p $out/lib/node_modules/${pname}
+      cp -r * $out/lib/node_modules/${pname}/
+      runHook postInstall
+    '';
+  };
+in
+  prev.n8n.overrideAttrs (oldAttrs: {
+    postInstall =
+      (oldAttrs.postInstall or "")
+      + ''
+        # n8n ships a pnpm stub symlink (libphonenumber-js -> empty-npm-package).
+        # Remove it and place the real package there instead.
+        rm -rf $out/lib/n8n/node_modules/libphonenumber-js
+        cp -r ${libphonenumber-js}/lib/node_modules/libphonenumber-js \
+          $out/lib/n8n/node_modules/
+      '';
+  })

overlays/mods/snipe-it.nix

@@ -0,0 +1,24 @@
+{prev}:
+prev.snipe-it.overrideAttrs (oldAttrs: rec {
+  version = "8.6.3";
+
+  src = prev.fetchFromGitHub {
+    owner = "grokability";
+    repo = "snipe-it";
+    tag = "v${version}";
+    hash = "sha256-Y1TNZ4uMK9ryKjKzFwW6Im1I2oRspFUXQI88lVi+FKg=";
+  };
+
+  vendorHash = "sha256-SJWWV1XWnwKe1XShTJfhWrEWTpTtrKFoNb27RGwW6v8=";
+
+  postInstall =
+    ''
+      snipe_it_out="$out/share/php/snipe-it"
+      mkdir -p $out/share/snipe-it
+
+      # Alle Default-Uploads sichern (z.B. default.png) bevor das
+      # ursprüngliche postInstall das Verzeichnis mit rm -R löscht.
+      cp -r $snipe_it_out/public/uploads $out/share/snipe-it/uploads
+    ''
+    + oldAttrs.postInstall;
+})

pkgs/atropim-image/default.nix

@@ -0,0 +1,95 @@
+{
+  lib,
+  pkgs,
+  atrocore-docker,
+  registryHost ? "git.az-gruppe.com",
+  registryNamespace ? "az-intec-gmbh",
+  imageName ? "atrocore-web",
+  skeletonVariant ? "pim-no-demo",
+  buildVariant ? "pdf",
+  productionDomain ? "pim.l.az-gruppe.com",
+  productionStability ? "stable",
+}: let
+  imageRef = "${registryHost}/${registryNamespace}/${imageName}";
+  baseTag = "localhost/${imageName}-base:build";
+  entrypointContext = ./entrypoint;
+
+  # The vendor Dockerfile uses the unqualified FROM "php:8.4-apache-bookworm";
+  # resolve it against docker.io without touching the host's registries.conf.
+  registriesConf = pkgs.writeText "atropim-registries.conf" ''
+    unqualified-search-registries = ["docker.io"]
+  '';
+
+  podman = lib.getExe pkgs.podman;
+
+  atropim-build = pkgs.writeShellApplication {
+    name = "atropim-build";
+    runtimeInputs = with pkgs; [coreutils];
+    text = ''
+      # Two-stage build: stage 1 builds the untouched vendor image from the
+      # rev-pinned atrocore/docker flake input, stage 2 layers the secret-free
+      # entrypoint wrapper on top. All DB build args stay empty on purpose:
+      # no credentials are ever baked into any layer.
+      export CONTAINERS_REGISTRIES_CONF="${registriesConf}"
+
+      echo "[atropim-build] stage 1: vendor image from ${atrocore-docker} (target ${buildVariant})"
+      ${podman} build \
+        --target "${buildVariant}" \
+        --build-arg "SKELETON_VARIANT=${skeletonVariant}" \
+        --build-arg "PRODUCTION_DOMAIN=${productionDomain}" \
+        --build-arg "PRODUCTION_STABILITY=${productionStability}" \
+        --build-arg "PRODUCTION_DB=" \
+        --build-arg "DB_USER=" \
+        --build-arg "DB_PASSWORD=" \
+        --tag "${baseTag}" \
+        --file "${atrocore-docker}/.docker/php/Dockerfile" \
+        "${atrocore-docker}/.docker"
+
+      echo "[atropim-build] stage 2: entrypoint wrapper -> ${imageRef}:latest"
+      ${podman} build \
+        --build-arg "BASE_IMAGE=${baseTag}" \
+        --label "org.opencontainers.image.title=${imageName}" \
+        --label "org.opencontainers.image.description=AtroPIM (${skeletonVariant}) web image, secret-free build with runtime DB config" \
+        --tag "${imageRef}:latest" \
+        --file "${entrypointContext}/Dockerfile" \
+        "${entrypointContext}"
+
+      echo "[atropim-build] done: ${imageRef}:latest"
+      echo "[atropim-build] verify the image is secret-free:"
+      echo "  podman run --rm ${imageRef}:latest ls /var/www/${productionDomain}/data"
+    '';
+  };
+
+  atropim-push = pkgs.writeShellApplication {
+    name = "atropim-push";
+    runtimeInputs = with pkgs; [coreutils];
+    text = ''
+      VERSION="''${1:-$(date +%Y%m%d)}"
+
+      if ! ${podman} image exists "${imageRef}:latest"; then
+        echo "error: ${imageRef}:latest not found - run atropim-build first" >&2
+        exit 1
+      fi
+
+      if ! ${podman} login --get-login "${registryHost}" >/dev/null 2>&1; then
+        echo "not logged in to ${registryHost} - run: podman login ${registryHost}" >&2
+        exit 1
+      fi
+
+      echo "[atropim-push] pushing ${imageRef}:{latest,''${VERSION}}"
+      ${podman} tag "${imageRef}:latest" "${imageRef}:''${VERSION}"
+      ${podman} push "${imageRef}:''${VERSION}"
+      ${podman} push "${imageRef}:latest"
+
+      echo "[atropim-push] done - package visible at https://${registryHost}/${registryNamespace}/-/packages"
+    '';
+  };
+in
+  pkgs.symlinkJoin {
+    name = "atropim-tools";
+    paths = [atropim-build atropim-push];
+    meta = {
+      description = "Build/push tooling for the secret-free AtroPIM image (${imageRef})";
+      platforms = lib.platforms.linux;
+    };
+  }

pkgs/atropim-image/entrypoint/Dockerfile

@@ -0,0 +1,13 @@
+# Stage 2 of the AtroPIM image pipeline: takes the vendor-built base image
+# (atrocore/docker, target "pdf") and layers the secret-free entrypoint
+# wrapper on top. The base image reference is injected via BASE_IMAGE so the
+# vendor Dockerfile stays untouched (pinned flake input).
+ARG BASE_IMAGE
+FROM ${BASE_IMAGE}
+
+COPY entrypoint.sh /entrypoint.sh
+COPY entrypoint-prepare-pim.php /entrypoint-prepare-pim.php
+
+RUN chmod +x /entrypoint.sh
+
+CMD ["/entrypoint.sh"]

pkgs/atropim-image/entrypoint/entrypoint-prepare-pim.php

@@ -0,0 +1,32 @@
+<?php
+/* Runtime counterpart of the vendor's prepare-pim.php (atrocore/docker). */
+
+if (empty($argv[5])) {
+    exit("Usage: php entrypoint-prepare-pim.php <instance-dir> <db-host> <db-name> <db-user> <db-password>\n");
+}
+
+$instanceDir = $argv[1];
+
+chdir($instanceDir);
+set_include_path($instanceDir);
+
+require_once 'vendor/autoload.php';
+
+$app = new \Atro\Core\Application();
+$config = $app->getContainer()->get('config');
+
+if ($config->get('isInstalled')) {
+    exit("[entrypoint] instance already installed - keeping existing data/config.php\n");
+}
+
+$config->set('database', [
+    'driver' => 'pdo_pgsql',
+    'host' => $argv[2],
+    'port' => '',
+    'charset' => 'utf8',
+    'dbname' => $argv[3],
+    'user' => $argv[4],
+    'password' => $argv[5],
+]);
+$config->set('useChromeNoSandbox', true);
+$config->save();

pkgs/atropim-image/entrypoint/entrypoint.sh

@@ -0,0 +1,24 @@
+#!/bin/sh
+# Secret-free AtroPIM image entrypoint.
+#
+# Before cron/apache start, the ATRO_DB_* environment variables are written
+# into data/config.php of the instance directory (same pattern as the vendor's
+# prepare-pim.php). The PHP side skips the write once the instance is
+# installed (isInstalled), which makes the wrapper idempotent: a config
+# completed by the web installer is never overwritten.
+
+set -e
+
+INSTANCE_DIR="/var/www/${ATRO_INSTANCE_DIR:-pim.l.az-gruppe.com}"
+
+if [ -n "${ATRO_DB_HOST:-}" ] && [ -n "${ATRO_DB_NAME:-}" ] && [ -n "${ATRO_DB_USER:-}" ] && [ -n "${ATRO_DB_PASSWORD:-}" ]; then
+  echo "[entrypoint] applying ATRO_DB_* variables to ${INSTANCE_DIR}/data/config.php"
+  mkdir -p "${INSTANCE_DIR}/data"
+  php /entrypoint-prepare-pim.php "${INSTANCE_DIR}" "${ATRO_DB_HOST}" "${ATRO_DB_NAME}" "${ATRO_DB_USER}" "${ATRO_DB_PASSWORD}"
+  # The web installer (running as www-data) must stay able to update the config later.
+  chown www-data:www-data "${INSTANCE_DIR}/data/config.php"
+else
+  echo "[entrypoint] no ATRO_DB_* variables set - starting web installer"
+fi
+
+exec /startup.sh