DNS, OG of High Availability
At May First, we recently received (all within a single week) three different complaints about domain names that previously worked fine suddenly not resolving to our servers.
While that isn’t terribly uncommon, we discovered that in each case, the domain
name’s authoritative name servers were pointing to our mail servers
(a.mx.mayfirst.org, b.mx.mayfirst.org and c.mx.mayfirst.org) instead of
our name servers (a.ns.mayfirst.org, b.ns.mayfirst.org and
c.ns.mayfirst.org). The weird part: this mistaken configuration was happening
at the registrar level, protected by each member’s own credentials that we
don’t have access to.
Each affected member fixed their records to resolve the problem but also made very clear that they had not logged into their registrar in years, sugggesting that the DNS authoritative records in their registrar accounts spontaneously changed on their own. The first time was weird, the second time could possibly be a coincidence? But by the third time this happened, we started to panic. How could registrar records spontaneously change? All three domain names were registered with different companies - so it couldn’t be a single registrar problem? Are we going to get a flood of these complaints? What is going on!?!?
We did an inventory to see if this was happening with other domain names in use by our membership and that’s when we discovered just how hard it is for our mostly non-technical users to set a domain’s authoritative name servers. The error rate was less than 1% but still that was a lot of domain names with typos:
- raise your fist in the air with
a.ns.mayfist.org! - or just plain give up and hit the floor with
a.ns.matfirst.org - Or more commonly people added our name servers, but also left the default name servers in place.
- Also, one person added as their authoritative name servers:
a.ns.mayfirst.org,b.ns.mayfirst.org,c.ns.mayfirst.org,a.mx.mayfirst.org,b.mx.mayfirst.org,c.mx.mayfirst.org, and evena.webproxy.mayfirst.org- in other words, all the domain names we tell you do to anything with. - And lastly, I did find two more domains just pointing to
a.mx.mayfirst.org,b.mx.mayfirst.organdc.mx.mayfirst.org.
That’s when it occurred to me: for years we have maintained an offsite server
that provides both c.ns.mayfirst.org and c.mx.mayfirst.org. It hangs out
in case something terrible happens to our main colo. The week before we started
receiving these complaints, I separated these services, moving
c.mx.mayfirst.org to a dedicated MX server. As a result, these two domain
names stopped pointing to the same IP address. And that’s when the complaints
started rolling in. In other words: the affected members set the incorrect name
servers years ago, but because just one of the name servers resolved to an IP
that happened to provide the correct authoritative lookup services, it went
undeteced all this time.
So… mystery solved. Nobody’s authoritative registrar records “suddenly” changed. They were mis-configured for years but thanks to the amazing resilience of the DNS system, nobody noticed because just one working DNS server is all you need.