{"id":9760,"date":"2019-03-13T13:19:47","date_gmt":"2019-03-13T17:19:47","guid":{"rendered":"https:\/\/www.thesslstore.com\/blog\/?p=9760"},"modified":"2020-08-25T10:17:22","modified_gmt":"2020-08-25T14:17:22","slug":"mass-revocation-millions-of-certificates-revoked-by-apple-google-godaddy","status":"publish","type":"post","link":"https:\/\/www.thesslstore.com\/blog\/mass-revocation-millions-of-certificates-revoked-by-apple-google-godaddy\/","title":{"rendered":"Mass Revocation: Millions of certificates revoked by Apple, Google & GoDaddy"},"content":{"rendered":"\n

The DarkMatter debate is already having industry-wide ramifications<\/h2>\n\n\n\n

Millions of SSL\/TLS certificates \u2013 among other digital\ncertificates \u2013 are being revoked right now as a result of an operational error\nthat caused the generation of non-compliant serial numbers. <\/p>\n\n\n\n

That, in and of itself, is not terribly interesting. While we\u2019ll discuss the generation of serial numbers in a minute, it\u2019s not a topic that most people care, much less think about. No, what\u2019s interesting is how we got here: how we found out that Apple, Google, GoDaddy \u2013 and likely a few other CAs \u2013 mis-issued millions of certificates. <\/p>\n\n\n\n

Last week we went in-depth covering the ongoing debate over DarkMatter CA\u2019s root application<\/a>. During that article we discussed the larger ramifications of the decision and how it’s made. Specifically, that much care and consideration needed to be given to the standards and precedents being set. <\/p>\n\n\n\n

A week later, the ramifications are already being felt. So,\ntoday, we\u2019re going to talk about what happened and how the DarkMatter debate\ncaused three of the biggest companies in the tech industry to realize they had\nbeen mis-issuing certificate themselves. <\/p>\n\n\n\n

Let\u2019s hash it out.<\/span><\/p>\n\n\n\n

How do you mis-issue millions of certificates?<\/h2>\n\n\n\n
\"Mass<\/figure><\/div>\n\n\n\n

Let\u2019s start with the technical side of what happened. If you\ndon\u2019t really want to hear about serial numbers and entropy, go ahead and skip\nto the next section. Otherwise, we\u2019ll give this a cursory look to help provide\nsome context. <\/p>\n\n\n\n

Let\u2019s start with EJBCA, which is an open source certificate\nauthority software package. It can be used on its own to build a complete PKI\ninfrastructure, but many publicly-trusted CAs use it as a Cryptographically\nSecure Pseudorandom Number Generator (CSPRNG) to generate serial numbers.<\/p>\n\n\n\n

Serial numbers are covered in the CA\/B Forum Baseline Requirements<\/a>, section 7.1:<\/p>\n\n\n\n

CAs SHALL generate non-sequential Certificate serial numbers greater than zero (0) containing at least 64 bits of output from a CSPRNG<\/p><\/blockquote>\n\n\n\n

Here\u2019s where things get a bit dicey, and we\u2019ll try to keep this high-level. These serial numbers must be unique, positive integers with 64 bits of entropy. That requires one of the 64 bits to be a fixed value in order to ensure that the serial number is positive. Unfortunately, the default settings for EJBCA didn\u2019t account for that detail and was instead spitting out 63-bit serial numbers.<\/p>\n\n\n\n

<\/span><\/span><\/p>\n\n\n\n

Why is that a big deal? Adam Caudill explained it perfectly on his blog<\/a>:<\/p>\n\n\n\n

When we are talking about numbers this large, it\u2019s easy to think that 1 bit wouldn\u2019t make much difference, but the difference between 2^64<\/code> and 2^63<\/code> is substantial \u2013 to be specific, 2^63<\/code> is off by over 9 quintillion or more specifically 9,223,372,036,854,775,808.<\/p><\/blockquote>\n\n\n\n

That represents an \u201cunacceptable risk\u201d to the entire ecosystem. Thus, every certificate with a 63-bit serial number that was generated using the EJBCA defaults must now be revoked and replaced with a compliant certificate.<\/p>\n\n\n\n

That, by itself, is a big deal. It’s a major business disruption. But that\u2019s not really the whole story, because how we got to this point is equally, if not more important. <\/p>\n\n\n\n

Be careful where you point the finger<\/h2>\n\n\n\n
\"Mass<\/figure><\/div>\n\n\n\n

Let\u2019s go back to the DarkMatter debate that played out last week on the Mozilla root forum<\/a>. As we discussed last week<\/a>, while the answer to the question of whether or not Dark Matter Group should have its root included in the various root programs \u2013 which would give it the ability to issue trusted digital certificates \u2013 seems obvious on its face, the decision will have a great many ramifications.<\/p>\n\n\n\n

That\u2019s already proving true.<\/p>\n\n\n\n

In an effort to avoid making a decision that looked purely political, there were several parties that went looking for a technical reason to deny DarkMatter\u2019s root application. One such reason presented itself in the form of 235 certificates that had been issued with non-compliant serial numbers.<\/p>\n\n\n\n

As Sophos\u2019 Corey Bonnell wrote:<\/p>\n\n\n\n

I would like to bolster my previous assertion that the serial number generation scheme used in the DarkMatter certificate hierarchy likely does not meet the requirements set forth in the Baseline Requirements, section 7.1\u2026

This analysis has revealed that all 235 unique certificates have a serial number of 8 octets (64 bits) and a big-endian most significant bit set to 0. Given that the probability of all 64 bits being output from a CSPRNG with a most significant bit value of 0 for all 235 such certificates is 1 in 2^235, it is extremely likely that these certificates do not contain the minimum number of bits (64) output from a CSPRNG and are therefore mis-issued under the Baseline Requirements.<\/p><\/blockquote>\n\n\n\n

In summation, he\u2019s accusing DarkMatter of mis-issuing certificates with 63-bit serial numbers and providing this as grounds for denying the root application.<\/p>\n\n\n\n

In response, DarkMatter\u2019s Scott Rea outlined its method for generating serial numbers:<\/p>\n\n\n\n

DarkMatter uses an EJBCA platform with the requisite setting for 64-bit random serial numbers and our source of entropy is a FIPS140 certified HSM, so I too was surprised by the findings you reported. However, during our investigation of this potential issue, we have thus far discovered that the platform appears to be compliant with the requisite standard, and the anomaly you are highlighting is potentially due just to the integer representation you are using in your calculations.

RFC5280 (section 4.1.2.2) defines serialNumber to be a positive INTEGER, and X.690 defines INTEGER to consist of one or more octets and (specifically section 8.3.3) says the octets shall be a two\u2019s complement binary number equal to the integer value. Using the two\u2019s complementary representation means that the output of the octet conversion is a signed integer, and it could be positive or negative \u2013 the range of integers from 64-bit numbers being from \u2013(2^63) to [+ (2^63)-1]. But since the RFC requires only positive integers, the 64-bits of output from the CSPRNG function must eventuate only in positive numbers, and negative numbers cannot be used. In two\u2019s complement representation, the leading bit determines whether the number is positive or negative \u2013 for positive numbers, the leading bit will always be zero (if it\u2019s a 1, then that represents a negative number which RFC5280 prohibits).

So our findings are that the platform is indeed using 64-bit output from an appropriate CSPRNG for generating serialNumbers, and that the leading zero is exactly what is required to indicate that it is a positive number in two\u2019s complement representation of the INTEGER, which is the requirement under RFC5280. Therefore our findings indicate that the serial number generation scheme used by DarkMatter in it\u2019s certificate hierarchy does meet the requirements set forth in the Baseline Requirements, section 7.1.<\/p><\/blockquote>\n\n\n\n

This harkens back to what we said earlier about one of the\nbits needing to be a fixed value to ensure that the entire serial number was a\npositive integer. This is also the moment that everyone else realized that the\nproblem wasn\u2019t DarkMatter\u2019s it was EJBCA\u2019s default configuration. One that many\nother CAs were also using.<\/p>\n\n\n\n

\"Mass<\/figure><\/div>\n\n\n\n

This is what we call an \u201coh $#!%\u201d moment.<\/p>\n\n\n\n

So far Apple, Google and GoDaddy have admitted to mis-issuing\ncertificates. GoDaddy initially said it was 1.8 million certificates, though\nthey\u2019ve since revised that number down. Apple took credit for 878,000, though\nabout 300,000 of those had already expired or been revoked as of last week.\nGoogle, meanwhile, estimated that it had issued over 100,000, though only about\n7,100 of them were still valid.<\/p>\n\n\n\n

There\u2019s also a possibility this could affect other CAs, too.\nThough as of right now none have disclosed any mis-issuances stemming from the\nEJBCA misconfiguration.<\/p>\n\n\n\n

We can, at least in part, thank DarkMatter for pointing this out. What we probably shouldn\u2019t do \u2013 as we discussed in last week\u2019s article on this subject<\/a> \u2013 is hold on to this as a reason to deny its root application. <\/p>\n\n\n\n

Not to say our article was particularly prescient, but:<\/p>\n\n\n\n

And what if, by applying that standard to other established CAs, it forces us to re-evaluate their participation, too? Or are we simply condoning applying different standards to different organizations based on their location and other partnerships?<\/p><\/blockquote>\n\n\n\n

What risk does this pose?<\/h2>\n\n\n\n

A business risk, sure. A security risk? No. The 64 bits of entropy required for BR compliant serial numbers was a requirement that was added in 2016 in response to an SSL spoofing proof-of-concept that was able to produce collisions (two matching serial numbers) using the MD5 hashing algorithm that was, at the time, generating certificates. Nowadays certificates are generated using SHA-2 (SHA256 or higher<\/a>), so MD5\u2019s vulnerabilities are no longer an issue.<\/p>\n\n\n\n

So, the need for 64 bits of entropy is really more of a\nsafeguard against attacks that could hypothetically be conceived of in the\nfuture. <\/p>\n\n\n\n

\u201cThis is a big deal for CAs and their customers,\u201d Caudill told Ars Technica<\/a>. \u201cThe impact of replacing large numbers of certificates is substantial. From a threat perspective though, this isn\u2019t exploitable. It would require a major breakthrough in cryptography, and even then, 63 bits of entropy provides a huge safety margin. This is a problem because of impact to people and companies; hackers aren\u2019t going to start forging certificates because of this.\u201d<\/p><\/blockquote>\n\n\n\n

As Caudill alluded to, the bigger issue is the business disruption. <\/p>\n\n\n\n

This will be the second major disruption the digital certificate industry has faced in the past year and a half \u2013 the first being the Symantec distrust<\/a>.<\/p>\n\n\n\n

\"Mass<\/figure><\/div>\n\n\n\n

That\u2019s kind of a black eye for the whole industry. Most businesses and consumers don\u2019t give a lot of thought to digital certificates until they expire or break. And as we\u2019ve discussed lately, for many businesses certificates and certificate management are more a matter of compliance than security<\/a>. And businesses are loathe to risk non-compliance.<\/p>\n\n\n\n

Suffice it to say there are going to be plenty of very angry organizations when they find out the certificates they use to help ensure compliance are being revoked for being non-compliant.<\/p>\n\n\n\n

That sounds like a really fun conversation.<\/p>\n\n\n\n

Distrust Google?<\/h2>\n\n\n\n

This header is facetious. But, one of the other offshoots of this whole situation are some of the similarities it bears to the aforementioned Symantec distrust<\/a>. Only this time it\u2019s Google that\u2019s mis-issued a bunch of certificates that pose no real-world threat. If you\u2019ll recall, Google practically forced Symantec out of the industry after discovering it had mis-issued 33 test certificates in 2016. Symantec claimed that it was a non-issue because they posed no real threat. Google claimed it represented an unacceptable breach of trust.<\/p>\n\n\n\n

Obviously there are some differences, but on some level a\nmis-issuance is a mis-issuance. And the fact that Google didn\u2019t discover this\non its own doesn\u2019t do it any favors. <\/p>\n\n\n\n

The point, obviously, isn\u2019t to vilify Google \u2013 just to, once again, point out the subjectivity of a lot of these decisions. CAs are going to mis-issue, it happens to literally every. Single. One. The more important part is the response.<\/p>\n\n\n\n

GoDaddy is giving itself 30 days to work through its revocations while Apple and Google are operating on much shorter timelines.<\/p>\n\n\n\n

That makes sense, too. Apple and Google issue their certificates internally, within their own organizations. They\u2019re not worrying about a bunch of pissed off customers. GoDaddy is. But, giving itself 30 days also may get GoDaddy into even more trouble from a compliance standpoint. The Baseline Requirements mandate the timely revocation of all non-compliant certificates, per section 4.9.1.1:<\/p>\n\n\n\n

The CA SHOULD revoke a certificate within 24 hours and MUST revoke a Certificate within 5 days if one or more of the following occurs: \u2026 <\/p>

7. The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA\u2019s Certificate Policy or Certification Practice Statement; \u2026<\/p><\/blockquote>\n\n\n\n

I\u2019d say that obviously the root programs aren\u2019t going to distrust GoDaddy, but three years ago I\u2019d have probably said the same thing about Symantec so I\u2019m just going to leave it at that. <\/p>\n\n\n\n

In the meantime, many websites and organizations are going\nto be scrambling to replace their SSL\/TLS certificates \u2013 in addition to many code\n& email signing certificates \u2013 with compliant ones. <\/p>\n\n\n\n

Thanks, DarkMatter.<\/p>\n\n\n\n

As always, leave any comments or questions below\u2026<\/em><\/p>\n\n\n\n

\"Hashed<\/figure>\n","protected":false},"excerpt":{"rendered":"

The DarkMatter debate is already having industry-wide ramifications Millions of SSL\/TLS certificates \u2013 among other digital certificates \u2013 are being revoked right now as a result of an operational error…<\/p>\n","protected":false},"author":6,"featured_media":9767,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":"","tve_updated_post":"","tve_custom_css":"","tve_user_custom_css":"","tve_globals":{},"tcb2_ready":0,"tcb_editor_enabled":0,"tve_landing_page":"","_tve_header":"","_tve_footer":""},"categories":[17],"tags":[134,9981,9980,131,188],"class_list":["post-9760","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-industry-lowdown","tag-apple","tag-darkmatter","tag-godaddy","tag-google","tag-mis-issuance","post-with-tags"],"views":28059,"jetpack_featured_media_url":"https:\/\/www.thesslstore.com\/blog\/wp-content\/uploads\/2019\/03\/Mass-Revocation.jpg","_links":{"self":[{"href":"https:\/\/www.thesslstore.com\/blog\/wp-json\/wp\/v2\/posts\/9760","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.thesslstore.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.thesslstore.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.thesslstore.com\/blog\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/www.thesslstore.com\/blog\/wp-json\/wp\/v2\/comments?post=9760"}],"version-history":[{"count":0,"href":"https:\/\/www.thesslstore.com\/blog\/wp-json\/wp\/v2\/posts\/9760\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.thesslstore.com\/blog\/wp-json\/wp\/v2\/media\/9767"}],"wp:attachment":[{"href":"https:\/\/www.thesslstore.com\/blog\/wp-json\/wp\/v2\/media?parent=9760"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.thesslstore.com\/blog\/wp-json\/wp\/v2\/categories?post=9760"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.thesslstore.com\/blog\/wp-json\/wp\/v2\/tags?post=9760"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}