because claims are cheap
The substrate is watchable
Anyone can say a crew of agents is supervised. The infrastructure is the part a reader can actually watch: dashboards that update without an audience, and alerts that reach a person.
trust chapter
What the record runs on
Infrastructure is not the headline on this site; it is the reason the record can be believed. A small number of layers, each doing one job, all declared in version control — one where every moving part has an owner, a rollback, and a dashboard watching it when nobody else is.
Why infrastructure is on this site at all
because the failures were real
This is what failed
The post-mortems describe production failures in these layers, with detection times and root causes. Reading them without the system underneath would be reading fiction.
because trust has to be checkable
Evidence, not assurance
Public dashboards, a frozen and hash-checked deploy tree, and read-only audits that end in a verdict per host. The point is not that the system is perfect; it is that its state can be inspected without asking me.
The layers
Top to bottom, in the order a request meets them.
edge
Traefik terminates TLS for every hostname, with ACME
certificates renewed automatically. Routes come from two sources only:
service registration and reviewed configuration files. No dashboard-only
routes — if it isn't in the repo, it doesn't survive a restart.
orchestration
Nomad schedules workloads on a small cluster with
Consul for service discovery and mutual TLS between
services. Workloads are pinned where their data lives, so a reschedule
never lands a service on a node that doesn't have its state.
secrets
Vault in high-availability mode holds every credential.
Applications read them at start-up through templates; nothing sensitive is
committed, and rotating a secret is a single write rather than a redeploy.
observability
Prometheus scrapes the cluster, Grafana
renders it, and an alert path escalates to a phone. Alert rules are versioned
like everything else, so an alert that fires is reproducible.
delivery
Gitea with self-hosted CI runners, and an internal registry
for images. Every deployment is traceable to a commit, and the same commit
can be rebuilt.
data & mail
PostgreSQL for application state, maddy for mail, and a
backup path that leaves the building: nightly dumps, weekly off-site copies,
restores rehearsed on a schedule rather than assumed.
Operating principles
01
Declared, not clicked
If a change can't be reviewed as a diff, it's not done. Manual fixes are allowed in an incident and expected to be written back the same week.
02
Least privilege by default
Services reach each other over an authenticated mesh, not a flat network. Anything that doesn't need to be reachable from outside isn't.
03
Restore is the test
A backup counts only once it has been restored into a clean environment and verified row-by-row. Until then it's an unverified copy.
04
Alert on symptoms
Pages are for things users feel. Everything else is a dashboard, a ticket, or a note — and sleeping badly is not a monitoring strategy.
05
Boring by preference
Fewer moving parts, one way to do each thing, and no component that only one person on the planet can debug. Excitement is a cost.
06
Write it down
Incident notes, runbooks and decision records live next to the code. The notes on incidents are the visible half of that.
Public dashboards
Read-only Grafana views. They refresh on their own — nothing on this page is a screenshot.
Framing is allowlisted to codalume.com. If this panel stays blank, the origin is not on that list — use open above.
Framing is allowlisted to codalume.com. If this panel stays blank, the origin is not on that list — use open above.
Source: liveproof/urls.md. Dashboards are published read-only
at the Grafana level — the embeds cannot query anything that isn't already public.
disclosure
what is not here
Published on purpose, withheld on purpose
This page describes architecture, not attack surface. It deliberately omits internal addressing, node names, credential paths, port maps and version pins — the details that turn a description into a map. Anything operational that is genuinely useful to a reader is published as a note in the post-mortems instead. The server enforces the same rule: story titles and summaries are scanned for internal addressing before they are rendered.
Found something that shouldn't be public? Please report it to accounts@codalume.com with [security] in the subject.