Internal observability stack for the Bulwark org: Grafana + Prometheus + Postgres + Umami, plus a GitHub data collector.
- Grafana - dashboards, host port
3006. The admin vhost has no nginx basic auth; Grafana's own login is the gate: the local admin account, or authentik with thegrafana-adminrole (Admin) orgrafana-viewerrole (Viewer). Anyone authentik lets through without one of those roles is refused. Its Postgres data sources log in with read-only roles, see below. - Prometheus - scrapes
/metricsfromrelay,stalwart, and the host. Internal docker network only. The relay wants its metrics token: put it insecrets/relay-metrics-token(one line, no newline needed);deploy.shmakes the file readable by Prometheus only. - Postgres 16 - backing store for Grafana, Umami, and the GitHub collector. Bound to
127.0.0.1:5432. - Umami - privacy-friendly web analytics. Public vhost; tracking script reachable from end-user browsers. On each deploy,
deploy.shreplaces Umami's default admin login withUMAMI_ADMIN_PASSWORDfrom.envthrough Umami's API (a password changed by hand stays); until that worked, the vhost serves only/script.jsand/api/send. Umami takes the visitor's address fromX-Real-IP, which nginx sets. - collector (pm2) - periodically pulls per-repo GitHub stats (stars, issues, traffic, releases) and computes commits + LOC by cloning into
data/repos/. Writes to thedashboardPostgres DB. Runs as the system accountbulwark-collector, whichdeploy.shcreates; it ownsdata/reposand readscollector/.env(its own variables, whichdeploy.shcopies there from.env) and the App Store Connect key through its group. As long ascollector/src/env.tsstill reads the stack's.env,deploy.shleaves that file readable for the group too and says so; once it readscollector/.env,.envis root's alone. pm2 starts every app withPATHandHOMEonly (the apps read their.envthemselves), and its saved list (~/.pm2/dump.pm2) is mode 600 and holds no secret. The scan takes only rows ofreposthat name a repository ofGITHUB_ORG(defaultbulwarkmail) with an id of<owner>/<name>; any other row fails in that run. A renamed or moved repository fails too: its old row stays inrepos, and GitHub answers the old name with a redirect, which the scan doesn't follow. Setarchived = trueon the old row (the one whoselast_seen_atstopped moving). One repository's scan stops afterCOLLECTOR_SCAN_DEADLINE_MS(30 minutes), and its line count unpacks at mostCOLLECTOR_UNPACK_MAX_BYTES(1 GiB) andCOLLECTOR_UNPACK_MAX_ENTRIES(200000 files, folders and links); a repository past a limit fails in that run. Failed repositories are in the log (git.repo_failed), the first one incollector_runs.error. - telemetry-collector (pm2) - receives anonymous heartbeats from self-hosted instances, rolls them up, writes to the
telemetryPostgres DB. Schema 2 heartbeats add error counts (error_counts, 90 days); withBUGSINK_TELEMETRY_DSNset, complete days are forwarded to Bugsink, one event per fingerprint, version and day, without instance ids. Instances in instant mode also post the current day's changed counts toPOST /v1/errors(once per 30 s per instance); one instance adds at most 200 different errors per day of counts there (the webmail itself counts no more than that), and new counts for an error it already sent are always taken; the first report of each error per day goes to Bugsink at once (error_instant_forwards, at most 30 a minute), but only from an instance the collector first stored more than a day before, by its own clock (instance_arrivals). At most 100 error groups a day go to Bugsink in all (TELEMETRY_MAX_FORWARDED_GROUPS_PER_DAY): instant forwards take at most half of them, and the daily forward of complete days, which sends widely reported groups first, has the rest. One instance brings in at most 10 of the day's groups (TELEMETRY_MAX_FORWARDS_PER_INSTANCE_PER_DAY); what doesn't fit waits for the next day. An error batch from an id the collector doesn't know takes one of the day's places for new ids (below), as a heartbeat does, and the id counts as known from then on: a desktop app with crash reports on and telemetry off never sends a heartbeat. With no place left the batch gets 429, and the webmail and desktop apps send it again later. Both routes read only bodies sent asapplication/json(415 otherwise). A heartbeat counts for the UTC day it arrives, or the day before when it says it was sent then. Versions are Bulwark's release shape only:N.N.Nwith at most 4 digits per number, optionally-beta.N, optionally+desktop(or+desktop.<commit>), 64 characters in all. Of the instance ids the collector hasn't seen before, at most 500 a day are stored at once (TELEMETRY_MAX_NEW_INSTANCES_PER_DAY, counted innew_instance_counts, heartbeats and error batches together); the next 5000 waiting rows that day go topending_heartbeats(TELEMETRY_MAX_DEFERRED_PER_DAY): one row per id and heartbeat day, counted when it is made, without its error counts, kept 7 days. A heartbeat names the day it arrives on or the day before, so one id can wait under both days and take two of the 5000. Once they are used up, heartbeats of further new ids that day are dropped, each step with one warning a day in the log: those installations are not counted that day. Every hour up to 500 waiting ids a day are stored under their own days: ids that sent again on another day first, then ids waiting two days or more, oldest first. The public version table ranks versions by installations known for more than a day, counted from the day the collector first stored them by its own clock (instance_arrivals), not from a day a heartbeat names; then by all installations, and it leaves out a version only one new installation reports. nginx lets each address post 6 times a minute (burst 10) and keeps the addresses in memory only. - version-server (pm2) - answers the apps' update check from the GitHub releases of
VERSION_SERVER_REPOand fromversion-server/overrides.json(minimumSupported, and versions flaggedsecurityordeprecated). It doesn't start when that file is missing, isn't JSON, holds a key it doesn't know (a misspelt one, say), names a key twice in one object, or has a malformed entry. A file that turns bad later is logged at the next hourly refresh, and the overrides read before stay in use;/healththen shows"overridesStale": trueandoverridesProblem(missing,unreadableorinvalid) until the file is good again. While the file is good but GitHub's release list hasn't been read since the start or since the file changed,/healthshows"overridesProblem": "releases-unavailable"until a refresh reaches GitHub.
$APP_DIR/
├── docker-compose.yml # grafana, prometheus, postgres, umami
├── .env # secrets + site config, chmod 600
├── secrets/ # relay-metrics-token, the App Store Connect key (not in git)
├── nginx/*.conf.template # rendered with envsubst by deploy.sh
├── prometheus/prometheus.yml
├── grafana/provisioning/...
├── grafana/dashboards/*.json
├── postgres/init.sql # creates grafana, umami, dashboard, telemetry DBs
├── postgres/grafana-read-only.sql # Grafana's read-only logins, run on every deploy
├── postgres/watchdog/ # the query-watchdog service's script
├── postgres/test/ # checks against a throwaway Postgres
├── collector/ # pm2 service
├── telemetry-collector/ # pm2 service
├── version-server/ # pm2 service, overrides.json
└── data/
├── postgres/
├── prometheus/
├── grafana/
└── repos/ # bare clones for LOC/commit scans
sudo bash deploy.shFirst run: generates .env with random secrets and exits - fill in GRAFANA_DOMAIN, UMAMI_DOMAIN, TELEMETRY_DOMAIN, LE_EMAIL and GRAFANA_OIDC_SECRET, then re-run.
Subsequent runs render every nginx vhost from nginx/*.conf.template and overwrite what is in /etc/nginx/sites-available, so a change made on the host by hand belongs in the template first. Then they request any missing Let's Encrypt certs, pull compose images, restart services, rebuild + reload the pm2 collectors, apply Grafana's read-only database logins.
A host can hold the stack as copied files instead of a clone. There the update copies main's tree
over the stack's folder: every file main has is replaced, every other file stays. What stays is
what the host made: .env, data/, secrets/, collector/.env, node_modules/ and dist/
(deploy.sh rebuilds them), backups, a token file outside secrets/. A change made on the host by
hand to a file main has (a dashboard, a template, version-server/overrides.json) goes into the
repo first, or the copy takes it away.
On a machine with the repo, main's tree with LF line endings, then onto the host:
git fetch origin
git -c core.autocrlf=false archive -o dashboard-main.tar origin/main
scp dashboard-main.tar <host>:/tmp/On the host, first what the copy would change, by name; nothing changes yet:
cd /opt/bulwark-dashboard
tmp=$(mktemp -d) && tar -x -f /tmp/dashboard-main.tar -C "$tmp"
(cd "$tmp" && find . -type f) | while read -r f; do
if ! sudo test -e "$f"; then echo "new $f"; elif ! sudo cmp -s "$tmp/$f" "$f"; then echo "replaced $f"; fi
done
rm -rf "$tmp"A replaced file loses what the host has in it. For each one whose change isn't in the repo, look
at it (lines with a password, secret or token left out) and carry what should stay into the repo,
then make a new archive:
f=<file>; tar -xOf /tmp/dashboard-main.tar "$f" | sudo diff -u "$f" - | grep -viE 'password|secret|token'Then the copy, as the folder's owner, and the check, which prints nothing once every file of main is there with main's content:
sudo -u "$(stat -c %U /opt/bulwark-dashboard)" tar -x -f /tmp/dashboard-main.tar -C /opt/bulwark-dashboard
tar -tf /tmp/dashboard-main.tar | grep -v '/$' | while read -r f; do
tar -xOf /tmp/dashboard-main.tar "$f" | sudo cmp -s - "/opt/bulwark-dashboard/$f" || echo "differs from main: $f"
done
rm /tmp/dashboard-main.tarA file main has deleted stays on the host; a dashboard JSON left in grafana/dashboards/ is still
provisioned, so remove such a file by hand. Then sudo bash deploy.sh.
Grafana runs any SQL a signed-in user sends to a data source, Viewers included. So its four Postgres data sources don't log in as the owners of the databases but as roles that can only read:
grafana_dashboard_ro- every table of the GitHub collector indashboard, also tables it adds later, exceptcollector_runs.grafana_umami_ro- onlywebsite,website_event,sessionandevent_datainumami. Umami's other tables (logins, API keys, two-factor secrets) stay closed.grafana_telemetry_ro- every table of the telemetry collector intelemetry.grafana_warden_ro- onlywarden_runsinwarden, for the Warden dashboard. The Warden bot (github-discord-bot) creates and owns that database where it runs next to this stack; the stack doesn't, and without it the Warden data source fails its queries and the grants file leaves it out.
They own nothing, can't create tables, temporary tables, functions or large objects, connect only to
their own database, take at most 10 connections and write at most 100 MB of temporary files per
query (temp_file_limit, which only a superuser can change). Nothing in the four databases is
granted to PUBLIC; the Warden bot's role keeps its rights in its own database as its owner. postgres/grafana-read-only.sql sets all of this. It is idempotent: a new data
directory runs it after init.sql, and deploy.sh runs it on every deploy, right after Postgres is
up and again after the apps started. Each part runs in one transaction, so a Grafana session that
starts during a deploy gets the logins as they were or as they are after it, always with their
limits. Each run also puts back the logins' passwords from .env
(POSTGRES_GRAFANA_DASHBOARD_RO_PASSWORD, POSTGRES_GRAFANA_UMAMI_RO_PASSWORD,
POSTGRES_GRAFANA_TELEMETRY_RO_PASSWORD, POSTGRES_GRAFANA_WARDEN_RO_PASSWORD; deploy.sh
generates missing ones) and their settings:
Postgres lets every role change its own password and session defaults, so a Viewer can break its
own data source until the next deploy, but not keep it broken. The file stops, naming them, when a
database holds a schema other than public or a function that runs as its owner: the apps make
neither, so look at them and drop them. By hand:
docker compose exec -T postgres psql -X -q -U postgres -d postgres < postgres/grafana-read-only.sqlA statement of these logins gets 15 seconds by default (statement_timeout), but a session can
switch that off. The query-watchdog service (postgres/watchdog/) runs every 5 seconds as
grafana_query_watchdog (password POSTGRES_QUERY_WATCHDOG_PASSWORD), a login that may only see
sessions, cancel non-superuser statements and end their sessions: it cancels their statements after
20 seconds and ends their sessions idle in a transaction for more than 60, whatever the session set.
A statement can catch the cancel (PL/pgSQL traps it), so one still running at 30 seconds is ended
with its session. A session can still raise its own work_mem until then.
postgres/test/effective-privileges.sh checks what the roles can do. It needs Docker and node,
starts a throwaway postgres:16-alpine with init.sql, lets the owners create their tables (and
leave a grant to PUBLIC behind), runs the grants file twice, asks Postgres what each role may do,
changes a login's settings and password as the login and checks the grants file puts them back,
lets the watchdog cancel a long statement, and runs every dashboard panel's SQL as its data
source's role. It removes its container at the end and prints "all ok" or "FAILED":
bash postgres/test/effective-privileges.sh .Added 2026-10-19. The collector feeds three download sources into the dashboard DB:
- GitHub release assets -
release_assets(current count per asset, classified inkind:android-apk,standalone-amd64,standalone-arm64,lite,lite-stalwart,checksum,mapping,other) andrelease_asset_snapshots(a row whenever the count changes). Classification lives incollector/src/github.ts(classifyAsset). - ghcr.io pulls -
container_packages/container_pull_snapshots. There is no API for pull counts;collector/src/ghcr.tsscrapes the "Total downloads" number from the public package page. Package list:GHCR_PACKAGESin.env(repo/package,...), default webmail, webmail-lite, webmail-beta, webmail-lite-beta, legacy-proxy. - TestFlight installs -
testflight_builds/testflight_build_snapshotsfrom the App Store ConnectbetaBuildUsagesmetrics (collector/src/testflight.ts). Enabled only whenASC_KEY_FILE(the.p8key, insecrets/;deploy.shmakes it readable by the collector's group only),ASC_KEY_IDandASC_ISSUER_IDare set in.envand the key file exists; otherwise it is skipped silently. Restart the collector after placing the key.
All three run on the fast (15 min) cycle. Per-day and "time range" panels are computed from
the change-only snapshots with lag(), so history starts at the first run (2026-10-19);
the all-time tiles use the live totals.