The night DNSSEC broke .de

- 5 mins read

Last night, large parts of the German internet quietly broke. Not for everyone — which made it more confusing, not less. Bahn.de, Spiegel.de, and thousands of other .de domains returned SERVFAIL to anyone using a security-conscious DNS resolver. The culprit was DENIC, the registry responsible for the .de top-level domain, and a botched key rollover in their DNSSEC setup.

What DNSSEC is supposed to do

DNS — the system that translates domain names like spiegel.de into IP addresses — was designed in an era when the internet was a considerably more trusting place. It has no built-in mechanism to verify that the answers you receive are genuine and haven’t been tampered with in transit. An attacker positioned between you and your DNS resolver can, in principle, return fake records and silently redirect you to a malicious server.

DNSSEC was introduced to fix this. It works by having each zone sign its DNS records with a private key, and publishing the corresponding public key so that resolvers can verify the signatures. When you query spiegel.de, a DNSSEC-validating resolver doesn’t just accept the answer — it checks the cryptographic signature, follows the chain of trust up through .de to the root zone, and only accepts the answer if everything checks out. If validation fails, the resolver returns SERVFAIL rather than an unverifiable answer. This is by design: a resolver that silently falls back to unvalidated answers would defeat the entire point of the system.

What actually went wrong

DENIC operates a Zone Signing Key (ZSK), which they rotate periodically — a standard practice to limit the damage if a key is ever compromised. Last night, something went wrong during that rollover.

DENIC’s signing pipeline signed a set of NSEC3 records using a new ZSK with key tag 33834, but failed to publish that key’s public data in the DNSKEY record set of the .de zone. The signatures were present in the zone, formally valid, and not yet expired — their validity window ran through May 19. But the public key needed to verify them was nowhere to be found. From any resolver’s perspective, the signature existed but could not be checked. Under DNSSEC rules, an unverifiable signature is treated the same as an invalid one: the answer is rejected, and SERVFAIL is returned to the client.

The specific records affected were NSEC3 denial-of-existence records covering a particular hash range within the .de zone. NSEC3 is the mechanism DNSSEC uses to prove that a name does not exist, without leaking which names do exist. Any domain whose name happened to hash into the affected range was caught in the crossfire — not because of anything wrong with those domains themselves, but purely by mathematical coincidence. Bahn.de, Spiegel.de, and Blackfort-tec.de share no infrastructure; they just happen to occupy the same neighbourhood in the NSEC3 hash space.

Who was affected — and who wasn’t

Here is where the story gets interesting.

DNSSEC validation is not universal. Many DNS resolvers — including those operated by major German ISPs like Deutsche Telekom and Vodafone — do not fully validate DNSSEC signatures. Their customers never noticed anything was wrong last night, because their resolvers happily served whatever the authoritative nameservers returned without checking the signatures.

The users who suffered were precisely those using security-conscious resolvers: Cloudflare’s 1.1.1.1, Google’s 8.8.8.8, Quad9, or self-hosted resolvers like Unbound or BIND with DNSSEC validation enabled. In other words, the outage was most severe for the technically aware segment of the internet — the people who had deliberately chosen infrastructure that takes DNS security seriously.

Cloudflare responded by deploying a Negative Trust Anchor (NTA) for the entire .de zone, instructing their resolvers to skip DNSSEC validation for .de until the issue was resolved. This restored access for 1.1.1.1 users but is, in effect, a controlled rollback of security — an emergency measure that works precisely because it makes the resolver behave like the non-validating resolvers it was designed to be better than.

DENIC fixed the underlying problem the same evening by re-signing the affected NSEC3 records with a correctly published key.

The irony is difficult to ignore

A security mechanism caused an outage. The users hit hardest were those who had taken extra steps to use more secure DNS infrastructure. The fix, in the short term, was to turn that security off.

This is not an argument against DNSSEC. The alternative — a DNS system with no integrity checking — is worse. But it is a pointed reminder that security systems have failure modes too, and that the failure modes of DNSSEC tend to be loud and total: SERVFAIL is not a degraded experience, it is a complete one.

ZSK rollovers are a known risk area. The operational procedure of signing zone data with a key before publishing that key — or failing to publish it at all — is exactly the kind of mistake that should be caught by pre-publication validation tooling. Whether DENIC had such checks in place and they failed, or whether they weren’t in place at all, we don’t yet know. A full postmortem hasn’t been published as of this writing.

For now, .de is back. But for a few hours last night, a safety net snapped — and the people who trusted it most were the ones who fell.

Further reading