live operations

Live, in the open

This page is served by the stack it describes. The dashboards below are the lab's own observability watching the very services that deliver this page — and the history below them is the honest record of when they didn't hold.

serving stack self-hosted · grey DNS · no trackers · frames refresh live — nothing here is a screenshot

Live dashboards

The two public Grafana views the stack publishes about itself: service health as its probes see it, and the fleet's load. They refresh on their own.

2 dashboards
Fleet health — live status open

Framing is allowlisted to codalume.com. If this panel stays blank, the origin is not on that list — use open above.

Node load — fleet resources open

Framing is allowlisted to codalume.com. If this panel stays blank, the origin is not on that list — use open above.

Source: liveproof/urls.md. The two frames above are the only non-page requests this page makes — and they go to the same self-hosted observability stack described on the infra page, nothing external.

Incident history

The complete published post-incident record since the lab went live: four notes, each with the timeline, the root cause and the change that followed. One line here — the whole note, with evidence, on the incident log.

full incident log →
  1. Ransomware hunt across 11 machines

    A test database exposed by a bare port mapping was brute-forced and a crypto-miner installed. Containment took sixteen minutes; the 11-machine sweep a month later came back clean.

    note
  2. The cert that killed 215 allocations

    A renewed certificate stayed stale in memory for sixteen hours: 215 and 216 failed placements took the public edge down. The renewal now reloads its consumer — and the DNS-shaped recurrence six days later was fixed at the resolver.

    note
  3. The zombie handshake storm

    Four hosts with expired client certificates retried forever — 356 rejected handshakes every three minutes at peak. The sources were stopped or re-certified, and rejection sweeps became a standing audit item.

    note
  4. The day the mesh died from an expired certificate

    Two silently failing renewals — a dead credential, a shadowed certificate — took the mesh's control plane down. Fixed with a scoped periodic token and ACME over DNS behind always-grey records.

    note

Last recorded incident: . Uptime as a percentage lives in the dashboard above — probe results the stack records itself, over a 24-hour window. What this page publishes for the long run is the dated record, so availability can be judged rather than claimed.

How this page is served

Described by class, not by vendor. Everything below runs on hardware we operate, in a stack declared in version control — the same stack the dashboards above are watching.

self-hosted edge

TLS is terminated by an edge proxy on our own hardware, with certificates that renew automatically. Routes are files in version control — if it isn't in the repo, it doesn't survive a restart.

grey DNS

Every record is DNS-only. No proxying layer sits between the visitor and the origin: no rewriting, no origin-hiding, no second TLS terminator to keep in sync.

no third-party JS

The only script this page loads is the site's own navigation toggle. Nothing is fetched from anyone else's CDN, and nothing runs that wasn't shipped from this repository.

no trackers

No cookies, no pixels, no fingerprinting. Requests are logged by our own server, read by us alone, and kept for debugging — not for profiling visitors.

disclosure what this page does not show

The proof is live; the details stay operational

The frames above are real Grafana views refreshing themselves — not screenshots. What is deliberately absent is the same as everywhere on this site: node names, internal addressing, credential paths and version pins. The dashboards show fleet state through generic names; the mapping to real hosts lives inside the stack, never in a page.

This page is itself part of the stack it shows: server-rendered by a small Rust service, behind the same self-hosted edge, on the same hardware the dashboards are watching.