Incident report · INC-2041

Booking API outage — expired internal TLS certificate

SEV-2

43 minTotal downtime
3,900Failed requests
4 minTime to detection
61Clinics affected

Summary & impact

On Jul 24 2026 the booking API could not reach the PMS connector for 43 minutes because its internal TLS client certificate expired. Every booking, reschedule, and auto-fill offer attempted in that window failed with a 502; 3,900 requests were rejected across 61 clinics. Patients saw a generic booking error, and front desks fell back to phoning patients directly.

No data was lost or corrupted. Confirmation messages queued during the outage were re-sent within 25 minutes of resolution. Two clinics reported that patients gave up rather than retrying; we cannot measure how many did so silently, which is the honest impact statement.

Timeline — Jul 24 2026

  1. Scheduled certificate rotation job runs on the internal PKI host. It renews four of five certificates; the booking API’s client certificate is skipped because its CN does not match the rotation filter.
  2. The booking API’s internal TLS certificate expires. Requests to the PMS connector begin failing TLS handshake. Patient-facing booking returns 502.
  3. detectedSynthetic booking check fails twice in a row and pages the on-call engineer.
  4. On-call acknowledges. First hypothesis is the connector deploy from 13:40; rollback started.
  5. Rollback completes; failures continue. Deploy ruled out.
  6. Handshake error surfaced in connector logs — certificate has expired. Root cause identified.
  7. mitigatedReplacement certificate issued manually and deployed to the connector fleet.
  8. resolvedError rate returns to zero. Booking confirmed working from three clinic accounts.
  9. Backfill job re-sends 412 queued confirmation messages. No patient data lost.

Root cause — five whys

  1. Why did booking fail?

    The API could not complete a TLS handshake with the PMS connector.

  2. Why did the handshake fail?

    The API’s client certificate had expired at 14:02.

  3. Why did it expire?

    The nightly rotation job did not renew it, and no alert fired on the miss.

  4. Why did rotation skip it?

    The job selects certificates by a common-name filter. This certificate was issued in 2024 with a legacy CN that the filter does not match.

  5. Why did nobody notice the gap?

    We alert on rotation job failure, not on certificate expiry. A skipped certificate looks identical to a successful run.

Root cause

Certificate lifecycle was monitored by proxy — we watched the job, not the certificates. Any certificate the job did not select was invisible to us until it failed in production.

What went well

  • Synthetic checks caught it in four minutes, before the first clinic wrote in.
  • The rollback was fast and correctly ruled out the deploy hypothesis in nine minutes.
  • The confirmation backfill worked exactly as designed; no patient message was lost.
  • Status page was updated at 14:12, before customers asked.

Action items

ItemOwnerDueStatus
Alert on certificate expiry directly, at 21 and 7 days remaining, for every internal certificateInes OkonkwoJul 31 2026Done
Replace the CN filter with an inventory-driven rotation listDevon OkaforAug 7 2026In progress
Re-issue the three remaining legacy-CN certificatesInes OkonkwoAug 14 2026In progress
Add a handshake-failure runbook entry with the exact remediation commandsPriya WendtAug 5 2026Open
Surface a clearer patient-facing error and a retry prompt on booking failureMara ChenAug 21 2026Open

Author: Devon OkaforReviewed: Mara Chen, Ines OkonkwoPublished Jul 28 2026Blameless review held Jul 27