Ops runbook · RB-014

Restore the booking database from snapshot

Applies to: booking-prod cluster only Causes downtime Last verified Jul 12 2026 by Ines Okonkwo Estimated duration 35–50 min

Preconditions

You are on-call and named as incident driver in the incident channel. If you are not, stop and find them.
You have ops role credentials for booking-prod and can run blctl from the bastion.
A verified snapshot exists from within the last two hours — confirm before you begin, not midway.
The connector runbook (RB-009) has been ruled out. This procedure is for database failure, not TLS or connector faults.
Someone other than you is watching the incident channel. Do not drive and communicate alone.

Phase 1

1Confirm the outage and stop the bleeding

Do not restore anything until writes are stopped. A restore over live writes produces a database nobody can reason about.

Confirm the primary is actually unhealthy, not the network path to it.

psql -h db-primary.internal.brightloom.dev -U ops -c 'select 1'

Expect a connection error or a timeout after 5 s. If it returns 1, this is not the right runbook — go to the connector runbook instead.

Put the booking API into read-only mode so no further writes are lost.

blctl api set-mode --service booking --mode read-only --reason "INC restore in progress"

Expect mode=read-only acknowledged by 6/6 instances. If fewer than six acknowledge, repeat before continuing.

Announce on the status page. Do this before you start the restore, not after.

blctl status publish --component booking --state degraded --template restore-in-progress

Expect published: status.brightloom.dev updated.

Verify before continuing

Writes to booking are rejected with a 503 and a retry header.
Status page shows degraded for the booking component.
Incident channel has the runbook link and your name as driver.

Phase 2

2Restore the most recent verified snapshot

Snapshots run hourly and are verified nightly. Always restore into a new host — never over the existing primary.

List the available snapshots and pick the newest verified one.

blctl db snapshots --cluster booking-prod --limit 8 --verified-only

Expect a table of snapshot IDs with timestamps. Note the ID, for example snap-20260724-1400Z.

Provision the restore target. This takes four to six minutes; do not run it twice.

blctl db provision --name db-restore-01 --class primary-equivalent --wait

Expect db-restore-01 ready followed by its internal hostname.

Restore the snapshot into the new host.

pg_restore --host db-restore-01.internal.brightloom.dev --username ops \
  --dbname booking --jobs 4 --clean --if-exists \
  /snapshots/snap-20260724-1400Z.dump

Expect no output on success and an exit code of 0. Warnings about missing roles are safe to ignore.

Run the integrity check bundle before you point anything at it.

blctl db verify --host db-restore-01 --suite booking-integrity

Expect 14/14 checks passed. Any failure stops the procedure — escalate rather than improvise.

Verify before continuing

select count(*) from appointments is within 1% of the pre-incident dashboard value.
select max(created_at) from appointments is no older than the snapshot timestamp.
All 14 integrity checks passed, and the output is pasted into the incident channel.

Phase 3

3Cut over and re-enable writes

Cutover is a DNS-level change and takes up to 90 seconds to propagate internally. Do not skip the soak.

Promote the restored host to primary.

blctl db promote --host db-restore-01 --cluster booking-prod --confirm

Expect db-restore-01 is now primary for booking-prod.

Restart the connector fleet so pooled connections pick up the new primary.

blctl service restart --service pms-connector --rolling

Expect 6/6 instances healthy within 60 s.

Soak for five minutes in read-only, watching the error rate before you re-enable writes.

blctl watch errors --service booking --window 5m

Expect an error rate at or below 0.1%. Anything higher: roll back.

Re-enable writes, then confirm a real booking end to end.

blctl api set-mode --service booking --mode live
blctl smoke booking --clinic test-clinic-001

Expect mode=live and smoke: create, read, cancel all passed.

Update the status page and hand off with a written summary.

blctl status publish --component booking --state operational --template resolved

Expect published. Post the timeline in the incident channel before you log off.

Verify before continuing

Smoke test passes for create, read, and cancel.
Error rate under 0.1% for ten consecutive minutes.
Status page reads operational and the incident has a written timeline.

Rollback

Use this only if the soak in phase 3 shows an error rate above 0.1%, or if any integrity check failed. Rolling back is always safer than continuing with a suspect database.

Demote the restored host immediately.

blctl db demote --host db-restore-01 --cluster booking-prod --confirm

Expect db-restore-01 demoted; no primary assigned.

Promote the previous primary back if it is reachable at all.

blctl db promote --host db-primary --cluster booking-prod --confirm

Expect db-primary is now primary. If this fails, do not retry — escalate.

Hold the API in read-only and escalate. Do not attempt a second restore alone.

blctl api set-mode --service booking --mode read-only --reason "rollback, escalating"

Expect mode=read-only acknowledged by 6/6 instances, then page in order below.

Escalation

Ines Okonkwo — platform on-call, primaryPage first, always. Owns this cluster.@ines · pager: ops-primary
Devon Okafor — CTOPage after 10 minutes with no response from primary, or immediately on any rollback.@devon · pager: ops-secondary
Sam Reyes — engineering, database familiarityPage if a second restore attempt is needed. Do not attempt one without a second engineer.@sam
Mara Chen — CEOInform, do not consult, once downtime passes 30 minutes. She writes to customers, not you.@mara