Observability stack on top of nixos-homelab: Grafana, long-term metrics storage, and the exporters/collectors that feed it, for both cluster and host-level monitoring.
For the full list of module options, see the documentation, under Options.
{
inputs = {
...
homelab-monitoring = {
url = "github:nixos-homelab/monitoring";
inputs.nixpkgs.follows = "nixpkgs";
};
...
};
}{ inputs, ... }:
{
imports = [
inputs.homelab-monitoring.nixosModules.grafana
inputs.homelab-monitoring.nixosModules.mimir
];
config = {
homelab.grafana.enable = true;
homelab.mimir.enable = true;
};
}- grafana: dashboards, provisioned with datasources for Mimir and Loki.
homelab.grafana.dashboards.<name>provisions a dashboard (read-only in the UI), andhomelab.grafana.homeDashboardmakes one Grafana's home. The modules' tiles (homelab.grafana.home.tiles) make a Home dashboard, which is the home by default.homelab.grafana.alertingprovisions alert rules and where they notify (see Alerting below). - mimir: long-term Prometheus-compatible metrics storage.
- loki: log storage, one binary on its volume, with
homelab.loki.retention(30 days by default). On, it has Alloy ship the logs (see Logs below). - alloy: the Grafana Alloy agent, a DaemonSet that scrapes the metrics of
homelab.alloy.monitorsinto Mimir and, with Loki on, ships the logs of the pods and the journal of its node. - kube-state-metrics: metrics about the cluster's objects (Deployments, pods, nodes, ...), Secrets excepted.
- node-exporter: host-level metrics (CPU, memory, disk, network).
- smartctl-exporter: disk S.M.A.R.T. health metrics, from a NixOS service on each host.
- systemd-exporter: the state of each host's systemd units, failed ones included, from a NixOS service on each host.
- zfs-exporter: ZFS pool and dataset metrics, from a NixOS service on each host.
Alloy tails the containers of the pods on its node through the API server and the journal of the node through a read-only hostPath, and writes one Loki record per log record:
- the line is the message alone; a stack trace, a goroutine dump or a PostgreSQL ERROR with its DETAIL is one record;
- labels:
source(podorjournal),node,namespaceandcontainer(pods) orunit(journal),service_name(the application: theapp.kubernetes.io/nameof the pod, the unit),program(what wrote the line: the container, a process a format picks out, theSYSLOG_IDENTIFIERin the journal) andlevel; levelis one of the names ofhomelab.alloy.logs.levels(error, warning, info, verbose, debug) orunknown, mapped from whatever the program writes (#from Redis,30from pino,Wfrom klog, the priority in the journal);- fields under the names of
homelab.alloy.logs.fields(component,caller,error,request_id, ...) as structured metadata, with the pod and, where the message is something else, the original line asraw.
A module says how its containers write: homelab.alloy.logs.pods.<name> names a format of
homelab.alloy.logs.formats (logfmt, json, klog, go, golog, nest and plain from this repo, or a dialect the module adds) and
gives example lines with the records they must become; homelab.alloy.logs.journal.<program> does the same for the
journal. Containers no module describes get a format by the look of each line. lib.logs.check runs the pipeline on
the examples with the pinned Alloy and Loki in a build:
checks.logs = inputs.homelab-monitoring.lib.logs.check {
inherit pkgs;
modules = lib.attrValues self.nixosModules ++ [ { homelab.<module>.enable = true; } ];
};lib.grafana builds dashboards in Grafana's v2 resource format, with the
Mimir datasource set on every query. Panels go where they are shown, in
the layout; dashboard numbers them.
{ inputs, ... }:
{
homelab.grafana.dashboards.nas.dashboard =
with inputs.homelab-monitoring.lib.grafana;
dashboard {
name = "nas";
title = "NAS";
layout = rows [
(row { title = "CPU"; } (autoGrid { } [
(timeseries {
title = "Busy";
unit = "percentunit";
queries = [
(prom ''1 - avg(rate(node_cpu_seconds_total{mode="idle"}[$__rate_interval]))'' { })
];
})
]))
];
};
}Builders: dashboard; layouts grid, autoGrid, rows/row,
tabs/tab; panels timeseries, stat, gauge, bargauge, table,
stateTimeline, text, or panel "<plugin>" for any other; prom for a
query, queryVariable, link, and whenVariable to show a row, tab or
item only for a variable's value. A dashboard can also be a JSON file
(dashboards.<name>.dashboard = ./board.json;).
lib.screenshots renders dashboards without a cluster: the pinned Grafana
on a dashboard's JSON, its Mimir datasource pointed at a Prometheus that
serves samples generated for the dashboard's queries, and headless Chromium
taking the whole page as a PNG. Each dashboard is a derivation of its own,
so a changed one renders again on its own, in seconds.
$ nix build .#screenshots.nas # the mixed scenario
$ nix build .#screenshots.nas.failing # or healthy
$ nix build .#screenshots.all # every dashboard, mixedNext to <name>.png, <name>.log lists every query the page ran, with
what came back (frames and rows, no data, or the error), and what the page
reported.
The data comes from a scenario (lib.screenshots.scenarios: healthy,
mixed, failing): label tables that keep a workload's namespace, pod and
job together across the metrics a query joins, and per-metric specs where a
metric's name says too little. scenarios.nix documents the format; a repo
with metrics of its own extends one with lib.recursiveUpdate. A machine's
own dashboards:
legacyPackages.screenshots = inputs.homelab-monitoring.lib.screenshots.forDashboards {
inherit pkgs;
inherit (self.nixosConfigurations.nas.config.homelab.grafana) dashboards;
# A render with the dashboard's variables set
variants.workload-sonarr = {
dashboard = "workload";
variables.workload = "sonarr";
};
};Alert rules come with the modules they watch, as dashboards do: a module adds a group named after it to
homelab.grafana.alerting.groups, built with lib.grafana.alert. A rule is an instant query against Mimir (or Loki
with datasource = "loki"), its last value per series held against a threshold for a while. Thresholds live in rules,
not on panels.
{ inputs, ... }:
{
homelab.grafana.alerting.groups.zfs.rules.pool-degraded = inputs.homelab-monitoring.lib.grafana.alert {
title = "Pool not online";
query = ''max by (zpool) (node_zfs_zpool_state{state!="online"})'';
above = 0;
for = "1m";
severity = "critical";
summary = "Pool {{ $labels.zpool }} is not online";
};
}This flake's modules bring their own, each behind homelab.<module>.alerts.enable (on with the module and
alerting):
| Module | Critical | Warning |
|---|---|---|
| zfs-exporter | pool not ONLINE, over 90 %, ZED data errors | over 80 %, full within a week, ZED checksum or I/O errors, a vdev leaving ONLINE, no metrics |
| smartctl-exporter | SMART failed | a disk over 55 °C for 15 minutes, no metrics |
| systemd-exporter | a unit failed for 5 minutes | |
| kube-state-metrics | a node NotReady | a container stuck waiting or restarting, a workload short of replicas, a failed Job |
| alloy | a scrape target down, kernel I/O errors and OOM kills in the journal |
The ZED rules read ZED's journal lines (all-syslog.sh, tagged zed) through Loki, so they need ZED logging to
syslog on the host.
severity routes an alert: critical and warning notify the contact point homelab.grafana.alerting.notify
names for them, info only shows in the Firing list on Home. Contact points are Grafana integrations of any type;
their secrets come from Secrets in Grafana's namespace (environmentSecrets), named in the settings as $NAME:
homelab.grafana.alerting = {
contactPoints.phone = [
{
type = "webhook";
settings.url = "$NOTIFY_URL";
}
];
environmentSecrets = [ "notify" ];
notify.critical = "phone";
notify.warning = "phone";
};Grafana provisions alerting from files at start only and deletes nothing that drops out of them, so an init
container in its pod compares what this configuration provisions with what Grafana's database says was provisioned
from files before, and has Grafana delete the difference. Only what carries the module's hl- prefix: rules made in
the UI stay. A change to the alerting restarts Grafana. lib.alerts.check runs a configuration's PromQL through
promtool, and lib.logs.queries its LogQL through the pinned Loki.