Disaster recovery runbook

Written to be followed during an incident, by someone who did not write it. Every step states what to run and how to know it worked.

Recovery objectives

These are the targets the runbook is designed to meet. Confirm them against your own hosting arrangement before quoting them to a customer — an objective nobody has rehearsed is an aspiration, not a commitment.

Objective	Target	What sets it
RPO — maximum data loss	15 minutes	Database point-in-time recovery window (binlog / WAL retention).
RTO — redirect service	1 hour	Time to restore the database and bring the redirect path back. Restored first: a broken redirect is a broken customer link, while a delayed dashboard is an inconvenience.
RTO — dashboard and API	4 hours	Full application restore including queue workers and scheduled jobs.
Analytics backfill	Best effort	Scans queued but not yet written are lost. They are deliberately not part of the RPO: retrying them would double-count redirects that already happened.

What is backed up

Asset	Backed up	Notes
Database	Yes — the only irreplaceable asset	Links, workspaces, scans, audit log, SSO connections, tokens.
Uploaded assets (logos, bio-page images)	Yes	Object storage; restore alongside the database or links render without branding.
APP_KEY and environment secrets	Yes — separately, in a secrets manager	Without the original APP_KEY, encrypted SSO client secrets cannot be decrypted and the audit log hash chain cannot be verified. A database backup restored under a new key is only partially usable.
Redis (cache, sessions, queues)	No, by design	Caches rebuild on demand and sessions are safe to lose — everyone signs in again. Queued jobs are lost; see the analytics note above.

Restore procedure

Declare and communicate. Post the incident before starting work. Customers discovering an outage themselves is worse than the outage.
Restore the database to the last good point. Prefer point-in-time recovery over the nightly snapshot; the difference is usually most of a day of link edits.
Restore APP_KEY and environment secrets from the secrets manager. Do this before starting the application: booting with a fresh key will fail to decrypt stored SSO credentials.
Bring up the redirect path first. Web nodes only, without queue workers. Verify with a known short code and confirm it 302s to the right destination.
```
curl -sI https://link.iutils.net/t/<known-code> | head -2
```
Verify the platform can serve traffic.
```
php artisan platform:readiness-check --strict
php artisan links:redirect-slo --requests=50
```
The SLO command fails if a warm redirect costs more than its query budget, which is the usual sign the cache layer came back empty or misconfigured.
Start queue workers. Analytics, webhooks and reports resume. Check Horizon for a backlog that is draining rather than growing.
Verify the audit log survived intact.
```
php artisan privacy:verify-audit-log
```
A break here means the restore was partial or rows were lost — worth knowing before a customer asks for an access record.
Restore the dashboard and API. Confirm sign-in, including any workspace using its own identity provider, since those depend on the restored secrets.
Re-verify custom domains.
```
php artisan domains:reverify
```
White-label hosts are cached; a stale cache after restore serves the not-found page on a customer’s own domain.
Close the incident and record what was lost. State the actual data gap, not the target. A customer can plan around a known gap; they cannot plan around a reassurance.

Rehearsal

An untested backup is a hypothesis. Restore into a scratch environment at least quarterly and time it end to end — the objectives above are only credible once someone has actually met them.

Record the measured RPO and RTO from each rehearsal. If they drift past the targets, change the targets or change the architecture; do not leave the published numbers standing.