Your Backups Are Green. Your Restore Has Never Been Tested.

Every infrastructure review at Tech Hub starts the same way. Someone opens the backup console, points at a wall of green checkmarks, and says the data is covered. The checkmarks mean a job wrote bytes to a target. That is all they mean. They say nothing about whether those bytes reassemble into a working store, and in most of the estates an audit turns up, nobody has tried.

The number that actually matters is not "did the job run last night." It is "how long until a customer can check out again, and how many orders do we lose when they do." Those two answers come from a restore you have performed, timed, and checked against real data. Everything else is an assumption that has never been tested.

A successful backup job proves exactly one thing

A backup job is a copy operation with logging. It confirms the source was readable and the target accepted the write. It does not confirm that the dump can be loaded by the version of the database you will be running next quarter, that the encryption keys still exist, or that the app pointing at the restored instance has the configuration it needs to boot.

Version skew is the most common silent failure. A nightly pg_dump taken with a client that ships with PostgreSQL 16 will fail on restore into a 14 cluster once the dump includes newer syntax. MySQL logical dumps taken without --routines --triggers --events restore cleanly and then quietly lose the logic that keeps your inventory counts honest. Logical dumps also grow restore time roughly linearly with row count, which is why a 2GB development copy restores in four minutes and the 380GB production copy does not.

None of this shows up in a job status. It shows up at 2am, during an incident, in front of people who were told the data was covered.

What breaks before the data does

Key material

Backups store ciphertext. If your database uses transparent encryption through a cloud KMS, or your application encrypts specific columns, then the restore target needs access to the same key material. Restores complete perfectly and produce a database nobody can read, because the recovery role was not in the key policy and the engineer running the drill had no permission to add it. The data was there. It was just opaque.

The same applies to the secrets store. A restored application tier with connection strings pointing at the old hostname will start, fail health checks, and sit in a crash loop that looks like an application bug.

Identity

Service accounts get rotated. Break-glass credentials get disabled, sometimes by the very incident response team that now needs them for recovery. An authentication outage and a data outage have a habit of arriving together, and if your recovery runbook authenticates against the system that is down, you have no runbook.

Keep recovery credentials in a separate trust domain with a documented, tested path to use them. Test that path on a Tuesday afternoon, not under pressure.

Schema drift and extensions

Extensions have versions. A restore that assumes pgvector 0.7 lands in a cluster with 0.5 and every vector index rebuild fails. Collation changes break unique indexes on columns holding customer names, and the failure appears as a duplicate key error on data that looks fine. Custom functions and materialized views are frequently excluded from dumps by teams who never noticed the exclusion flags.

The data itself

A restore returns whatever was in the database at the recovery point, including the parts you did not want. Deleted rows come back. Soft-delete timestamps that were set after the corruption get rolled back. If the incident was an intrusion rather than a hardware failure, the newest backup can contain the attacker's persistence, and restoring it puts you back in the same room with them.

This is why the recovery point is a decision, not a default. "Newest available" is the correct answer for a disk failure and the wrong answer for a compromise.

Attackers now plan for your restore

IBM's X-Force Threat Intelligence Index for 2026 describes a shift toward what it calls recovery denial: attackers deliberately targeting backup infrastructure, identity services, and virtualization management layers so that the victim cannot rebuild without paying. The reasoning is simple: if you can restore in an hour, paying stops making sense.

The recovery numbers back it up. Veeam's 2026 Data Trust and Resilience Report found that among organisations hit by ransomware in the previous year, 56 percent had data encrypted or exfiltrated, and of those whose operations or data were affected, only 28 percent recovered all of it. Another 44 percent recovered less than three quarters. Eon's 2026 Cloud Data Infrastructure Report puts a related number on the table: 60 percent of cloud IT leaders say a full restore takes six hours or more in their environment, and 5 percent can complete one in under an hour. Most of those same teams have an RTO on a slide that says four hours.

Meanwhile, regular restore testing is one of the least deployed recovery controls in the field. A 2026 industry roundup put it at 34.9 percent of organisations, behind cloud backups at 55.4 percent and endpoint protection at 58.5 percent. The control that determines whether recovery works is the one fewest teams run.

Run a drill that produces evidence

A drill is not a screenshot of a restore that worked. It is a timed exercise with a defined scope, a written result, and a list of things that broke. Run it in this order.

1. Pick a customer journey, not a server. "Guest checkout completes and the confirmation email arrives" is a target you can test. "The orders database is restored" is a target you can accidentally satisfy while the site is still broken.
2. Restore into isolated infrastructure with production-equivalent controls. Same encryption, same network policy, same secret management, same database engine version. A restore into a laptop with the schema and no data proves nothing about the production path.
3. Use production data volume. Restore duration is dominated by I/O and index rebuilds, and both scale with size. Drill against a snapshot of the real dataset, or your measured RTO will be wrong by an order of magnitude.
4. Clock from the first failed customer request. Not from when your alert fired, and not from when an engineer opened the ticket. Recovery time is measured from impact, and impact starts before monitoring notices.
5. Verify domain invariants, not row counts. Row counts match trivially and prove little. Check that the number of paid orders equals the number of successful captures at the payment processor for the same window. Check that no line item references a missing order. Check that the ledger sums to zero across double-entry pairs. These are the checks that catch a restore that technically completed.
6. Restore the edges too. The queue, the search index, the object storage prefix, the CDN cache, and the scheduled jobs. A correct database behind a stale search index shows yesterday's prices, and a correct database with a rebuilt queue sends every confirmation email a second time.
7. Write the findings down. The finding list is the deliverable. A drill that ends with "it worked" and no notes has not told you anything you can act on.

E-commerce specifics that bite during a restore

Commerce systems hold state outside the database, and restores rarely account for it.

• Order numbers. Rolling a sequence backwards means the next order reuses an identifier. Fulfilment systems keyed on that number then ship the wrong parcel or reject the order outright.
• Payment tokens. The vault is not in your database. Restore the database to 14:00 and the vault still holds tokens created at 14:40, so you end up with charges that reference orders your restored system cannot see. Reconcile against the processor before you reopen checkout.
• Inventory reservations. Reservations usually live in a cache with a short TTL. Restoring the database to an earlier point while the cache keeps its current state produces exactly the oversell condition your reservation system existed to prevent. Flush and re-derive.
• Outbox and webhooks. Replay is the point of an outbox, and duplicate delivery is its usual failure. Every consumer needs an idempotency key that was persisted before the outage, which means the key store has to be part of the recovery scope.
• Pricing and promotions. Promotion windows and price rules often come from a config service or a CMS that was never in the backup scope. Restoring yesterday's database against today's config gives customers a discount that expired.

What a restore does to your compliance posture

A restored copy is a new copy of personal data, and it inherits obligations the backup job never had. Restoring production customer records into a staging account with weaker access controls can be a reportable incident in its own right under PIPEDA in Canada or GDPR in the EU. Restored copies need the same encryption, the same access logging, and a deletion date.

Two practical rules. First, restored data goes into the same retention schedule as production, which usually means it gets destroyed when the drill ends rather than added to a growing pile of forgotten snapshots. Second, record the drill itself: timestamps, scope, the recovery point used, the verification results, and who signed off. Auditors increasingly ask for restore evidence rather than backup logs, and an untested backup has a way of being the finding.

What to put on a dashboard

Five fields per critical service, reviewed monthly:

• Last drill date, and the age of that drill in days.
• Observed RTO against the target, both in minutes.
• Observed RPO, meaning the gap between the chosen recovery point and the moment of impact.
• Restore dataset size, so you can see duration versus growth.
• Count of open findings from the last drill, and how many are past their due date.

That fourth field is the one people skip and the one that predicts future failures. If a full restore takes four hours at 400GB and your data grows 40 percent a year, you can calculate the date your four-hour RTO target stops being achievable. Put that date next to the dashboard. It turns an abstract risk into a scheduled engineering task.

A realistic first ninety days

Weeks one and two: inventory what is actually protected. Compare the list of systems your team believes are backed up against the list of systems in your architecture diagram, then against the list that actually appear in backup job output. These three lists are rarely the same, and the difference is the gap.

Weeks three and four: drill one tier-one journey end to end. Expect it to fail. Budget the time for that failure, and resist the urge to stop the exercise when the first step breaks, because the second and third failures are usually the expensive ones.

Weeks five to eight: fix the top three findings. In practice those are almost always key access, missing config or secret dependencies, and a runbook that assumes a working identity provider.

Weeks nine to twelve: put the drill on a quarterly calendar with a named owner for each critical service. Tie it to change management, so a migration that changes how the database is stored triggers a re-drill rather than a promise.

The first drill will fail, and that is the useful outcome. Write the findings into the same tracker as your feature work, with owners, and the second one usually passes. A green backup job and a tested restore are two different claims about your business, and only one of them survives contact with an outage.