Why Your DNS Failover Didn’t Actually Fail Over

The failover was declared at 02:14. The runbook was followed. DNS records updated. Health checks passing on secondary. The on-call engineer closed the incident bridge call at 02:31 with a single line in the ticket: failover complete. At 02:32, a monitoring alert fired. Traffic was still hitting the dead primary.

The DNS record had changed in seconds. The traffic moved 18 minutes later. Only one of those numbers mattered.

This is dns failover testing failure in its most common form — not a misconfiguration, not a vendor bug, not a missed step. Every layer in the stack behaved exactly as designed. The system still failed operationally. It belongs to the same class of failure documented in data protection architecture where the protection plane reports success and the recovery plane produces nothing useful.

What the Runbook Said Was Done

The runbook covered the right things. TTL had been pre-reduced to 60 seconds during the maintenance window two weeks prior. The health check interval on the secondary was 30 seconds. The DNS record update propagated to the authoritative nameservers within 90 seconds of execution. By every documented metric in the disaster recovery and failover playbook, the failover was complete.

The team was not wrong to close the bridge. They were wrong about what “complete” meant.

DNS failover is treated as a discrete event — you change the record, propagation happens, traffic moves. The mental model is a switch: off, then on. The operational reality is a drainage problem. Traffic does not move when the record changes. Traffic moves when every active path that was routing to the old record has expired its cached state and re-resolved. Those are different events, separated by an amount of time that no runbook entry captures.

The Declaration Gap is the period between when the failover is declared complete and when traffic has actually moved. It maps directly to RTA — Recovery Time Actual — the measured gap between when recovery is declared and when the system is genuinely operational again. In this case, the Declaration Gap was 18 minutes. In environments with more caching layers, it can be significantly longer.

The Four Layers That Each Did Their Job

This is the part worth understanding precisely. The failure was not caused by any single layer behaving incorrectly. It was caused by four layers each behaving correctly — and nobody having modelled what that looked like in combination.

This is the same structural failure class as the retry storm — every component executes its designed behaviour correctly, and the combination of correct individual behaviours produces a system-level failure nobody modelled. The failure is not in any layer. It is in the assumption that layers coordinate.

What DNS Failover Testing Actually Needs to Measure

Most dns failover testing validates the wrong thing. A test that confirms the DNS record updated and the health check passed has validated the protection plane. It has not validated the recovery plane — whether traffic actually moved, when it moved, and what the distribution looked like during the transition window. This is precisely the gap documented in restore path design applied to a different layer: success on the write side proves nothing about the read side.

This failure mode has a close relative in Azure Private Endpoint DNS resolution — environments where DNS behaves correctly at every individual hop while producing the wrong outcome at the traffic layer. The mechanism differs; the diagnostic principle is identical.

A test that surfaces the Declaration Gap needs to measure traffic distribution, not DNS state. It needs to run active traffic against the production path — including CDN-transited requests and enterprise resolver-cached clients. It needs to timestamp when the DNS record change executes and when traffic distribution on the secondary crosses a defined threshold. The gap between those two timestamps is the real RTO contribution from DNS failover. That number belongs in the RTO model, not a TTL assumption.

Pre-reducing TTL before a planned failover is necessary but not sufficient. The CDN cache TTL needs its own pre-reduction step — most CDN providers allow origin cache TTL configuration independently of DNS TTL. Ignoring this makes the CDN the binding constraint on traffic movement regardless of how aggressively the DNS TTL was tuned.

Monitoring during the failover window needs to watch traffic distribution at the application layer, not DNS propagation at the nameserver layer. The question the on-call engineer needs to answer in real time is not “has the record changed” — it is “what percentage of traffic is hitting the secondary right now.” The incident recovery process depends on being able to answer that question with data, not inference.

The Transferable Principle

DNS failover is not complete when the record changes. It is complete when traffic distribution changes.

That distinction rewrites the RTO model for any architecture that depends on DNS-based failover. The RTO contribution from a DNS failover event is not the TTL value. It is the time required for all active traffic paths to drain their cached state and re-resolve to the new record. These drainage events happen independently, on different timers, with no coordination signal between them.

The TTL is a floor on propagation speed. The actual Declaration Gap is determined by whichever caching layer takes longest to drain. In most environments that layer is either the CDN origin cache or the enterprise recursive resolver — neither of which is controlled by the TTL set on the DNS record.

Testing needs to validate this explicitly — not as a one-time exercise, but on the same cadence as the RTO it is supposed to guarantee. An architecture with a 15-minute RTO commitment that has never measured its Declaration Gap does not have a 15-minute RTO. It has a 15-minute aspiration and an unknown operational reality.

The record changed in seconds. The traffic moved 18 minutes later. Only one of those numbers mattered.

Additional Resources