DigiCert gives unlucky folks 24 hours to replace doomed certificates after code blunder

DigiCert has given unlucky customers 24 hours to replace their SSL/TLS security certificates it previously issued them – due to a five-year-old blunder in its backend software.

After that period, the digital certs – which are used for providing encrypted HTTPS connections for websites among other things – will be revoked and rendered useless. It appears to us this revocation is being done out of an abundance of caution, and to follow the letter and spirit of the rules on certificate issuance – which is important in the worlds of trust and cryptography – rather than because there's an immediate grave risk to people's privacy and identities.

Essentially, when DigiCert issues a certificate for a domain name, it verifies it's giving that cert to the rightful owner of that domain. The process by which it performed that validation was buggy, and certificates issued by that broken code need to be canceled because they are technically untrustworthy.

This will affect "approximately 0.4 percent of the applicable domain validations we have in effect," the certificate authority said in a July 29 advisory. The Register has asked exactly how many domains this represents, and we'll let you know if DigiCert can come up with a number.

Though this won't be too much fun for those of you who have to replace their certs, the cause of the recall is pretty interesting.

There are various ways in which a domain name owner can demonstrate that ownership to an authority such as DigiCert, though the one relevant to this kerfuffle involves DNS CNAMEs, random digits, and underscores.

Let's say you want to buy an SSL/TLS certificate for a domain you control, example.com, so that you can provide a secure https://example.com/ website to your users. You go to DigiCert and request a cert for example.com. DigiCert generates a random number for you – let's say it's 123456 – so that you can create a DNS CNAME record so that _123456.example.com resolves to DigiCert's dcv.digicert.com. Note the underscore.

When you've set up that DNS entry, and let DigiCert know, it does a DNS lookup of _123456.example.com, sees that it's correctly pointing to dcv.digicert.com, and generates the security certificate for example.com for you. The fact that you can edit the domain name records for example.com indicates you are the owner and controller of that domain, so you get the cert.

Here's the thing: You have to put in an underscore at the start because there was a worry that DigiCert might generate a random number that just so happens, by fluke of nature, to actually exist as a sub-domain for the given domain. Let's say you don't own example.com but you want to try to get a certificate for it from DigiCert so you can do nefarious things that we won't go into here. Let's say DigiCert generates the challenge number 456789, and 456789.example.com actually exists. You end up getting the certificate, if the underscore wasn't required or checked for.

The chance of such a collision is extremely, extremely low because the random number being generated has at least 150 bits of entropy, but surely no one leaves things like that to chance in information security. By putting an _ at the start, the full domain name (eg, _123456.example.com or _456789.example.com) is invalid as it must, by the rules of the internet, start with an alphanumeric character. Thus with this underscore requirement, you not only have the low, low chance of a numeric collision but also the fact that it's extremely unlikely a domain owner would actually have a long random string of digits with a leading underscore as that would be strange and invalid during normal operation.

Unfortunately, DigiCert did not tell its customers in its documentation that they had to put an underscore at the start – and from August 2019 to now, DigiCert's code, after some reorganizing of its software, accidentally no longer added the _ to generated challenge values nor even checked for it. The code previously would at least add the underscore.

Thus customers who validated domains for certificates using the DNS CNAME method described above, and did not use an underscore as the system didn't enforce it, technically broke the rules laid out by the CA/Browser Forum regarding domain validation, and thus the certs can't be trusted. That's why they need to be revoked and reissued, and reinstalled or redeployed.

"Recently, we learned that we did not include the underscore prefix with the random value used in some CNAME-based validation cases," as DigiCert put it. "Under strict CABF rules, certificates with an issue in their domain validation must be revoked within 24 hours, without exception."

If you're wondering how the underscore checks disappeared...

DigiCert said "legacy code in CertCentral (our public TLS certificate issuance portal) automatically added an underscore prefix to random values if a customer selected CNAME-based verification." During that aforementioned modernization effort, this legacy code was not properly carried over to the new system:

This oversight wasn't caught during testing either:

Fast forward to June 11, when DigiCert completed a project to simplify the random value generation process and eliminate the manual addition of the underscore prefix. "As before, we did not compare this UX change against the underscore flow in the legacy system," the cert authority admitted.

And from the sounds of things, the org wouldn't have noticed the bug if it weren't for a customer call it received "several weeks ago" reporting the problem in the validation system:

A couple things about this explanation. First, maybe it's just your humble vulture, but it sounds like DigiCert is low-key dinging — not thanking — the reporter for bringing this bug to its attention, and that gives us pause.

Second, as a loyal Reg reader pointed out to us: "Their root cause analysis makes CrowdStrike's test regime look thorough."

This is in reference to the global outage earlier this month caused by a flawed automatic update that CrowdStrike pushed to millions of Windows machines, also without firmly testing the update.

But back to business: DigiCert offers a series of steps needed to reissue certification.

First, customers need to login to their CertCentral account to see if their certificates are affected. Assuming they are, then go to the Certificates > Orders page and locate the impacted certificates.

Next, generate a new Certificate Signing Request (CSR), and on the certificate's Order # details page, in the certificate actions dropdown menu, select Reissue certificate.

If there's any additional required validation steps, complete those securely, and then install the reissued SSL/TLS certificate.

Affected customers, all of whom have been notified, can also contact their DigiCert account managers or call the support hotline: +1 801-770-1718. ®