Better with Kent · draft

Hook

Every backup my system ever took

failed the restore drill

Rows up to 1.28 MB · import limit ~100 KB · oldest bad row from April

The good part

A drill found it — not an incident

• Found on purpose, before public launch
• Zero users harmed
• That is what hardening buys

Thesis

Launch is a threat-model change

not a marketing event

What changes

Three forces

Each breaks something different · one deliberate pass over each

Receipts

One month of hardening

• ~630 PRs in 31 days — a fleet of cloud agents
• 40+ PRs of alert hygiene alone
• 97 orphaned objects purged
• 1 silently failing backup lane

Force 1

Strangers

Abuse stops being hypothetical

Move 1

Ramp the front door

Launch is a progressive rollout of exposure — a dimmer, not a switch

Move 1 · also

Baseline before you touch the door

• Measure the activation funnel first
• “Baseline is perishable”
• Separate human actions from agent actions

Strangers include their agents

Paperwork

Strangers are customers

• Terms + pricing pages
• Billing that cancels on account delete
• Data export + deletion that actually covers everything

Boring until it’s an emergency

Move 2

Assume the bad actor

• Suspension kill switch — fail-closed everywhere
• Email pause: 1 complaint or 5 bounces/day
• Quotas charged before the work runs

“One bad actor can poison shared platform identity”

Move 2 · isolation

User code gets its own origin

• Hosted user apps on a separate domain — fail closed
• Tenant scoping in the query, not the caller’s manners
• Least privilege for agents and users alike

Force 2

Scale

Your busiest table × N

Move 3

Meter before you limit

Denial-of-wallet: on usage-billed infra, attackers just make you pay

Move 3 · tuning

Set limits from data

• Metering landed before the limits
• Quotas were placeholders until data arrived
• Free tier sized to reach a working automation

Guessing strangles real users or subsidizes abusers

Move 4

Kill the shared hot paths

• The table everyone writes falls over first
• Per-user stores for run-hot state
• Rate-limit at the edge, not in the database

Move 4 · how

No faith-based cutovers

Dozens of small PRs · every cutover behind a verified parity gate

Force 3

Stakes

Their data · their money · your 2am

Move 5

Only page on truth

• Context in: traces, user context, diagnostics
• Noise out: 40+ PRs deleting non-incidents
• Caller errors → audit log, not pager

An alert channel that cries wolf is worse than none

Move 5 · outside view

A status page that outlives you

• Separate worker — probes from outside
• Up even when deploys or the database are broken
• One email at outage start · one reminder per day

Alert policy, not alert firehose

Move 6

Drill the worst day

• Restore drills found every gap alerting missed
• Unrestorable backup = failure, not checkbox
• Guardrails: one wrong migration line can’t erase prod
• When data moves, backups follow

Meta

The checklist is old — the affordability is new

• SRE-era wisdom, all of it
• Drills + isolation used to be “someday” work
• A fleet of agents made them a Tuesday

The threat model is still the human’s call

Homework

Run one restore drill this week

• Restore your latest backup somewhere disposable
• Open the app against it
• Write down what broke — that’s your launch list

No backups? That is the drill result.

Better with Kent

Drill before the strangers arrive

Strangers · scale · stakes — one honest pass over each

Like · comment · subscribe · get better with me