Ballot SC 22’s failure highlights the dysfunction at the CA/Browser Forum.
For now, at least, SSL/TLS certificates will still have a maximum validity period of two years (or 27 months). The CA/Browser Forum ballot that sought to shorten the maximum lifespan of SSL/TLS certificates to one year failed when the voting ended yesterday afternoon. The final tally was 20 opposed, 18 in favor and two abstentions. The vote wasn’t that close though, it fell well short of what was needed to pass from the Certificate Authorities.
This is now the second time the initiative to shorten certificate validity to a single year has been rejected. The last time shortening validity was discussed, two years was the compromise. This time around the only compromise extended to the CAs was delaying the ballot’s effective date back a month, from March to April 2020.
Citing business disruptions and the pain points of their customers, as well as 4,000 customer survey aggregate results from three CAs showing website owners opposed the change by 83%, the CAs voted down this measure by a count of 20-11. The seven browser vendors joined in supporting the ballot, but ultimately it didn’t matter on account of the CA vote.
But while that might seem like it’s the whole story – it’s really just scraping the surface. This process laid bare the CA/B Forum’s flaws and likely deepened the divide between the browsers and the CAs. So, today we’re going to discuss the ballot, the CA/B Forum and the absolute breakdown in civility that’s unfolding right now in this industry. Then we’ll talk about what needs to change to fix it.
Let’s hash it out.
Max Validity & the CA/B Forum (and a quick word on EV)
For those that aren’t terribly familiar – and admittedly, we do sometimes forget that our singular focus on PKI isn’t shared by the masses – the CA/B Forum is the industry body that collaborates on the “best practices” baseline requirements that govern Certificate Authorities and the issuance of public-facing digital certificates.
Here’s the way the Forum is described in its own bylaws:
The Certification Authority Browser Forum (CA/Browser Forum) is a voluntary gathering of leading Certificate Issuers and vendors of Internet browser software and other applications that use certificates (Certificate Consumers).
Members of the CA/Browser Forum have worked closely together in defining the guidelines and means of implementation for best practices as a way of providing a heightened security for Internet transactions and creating a more intuitive method of displaying secure sites to Internet users.
Now, before we go any further, a quick aside. There’s a real debate over the future of Extended Validation UI. There are some very vocal parties at the Forum that argue it’s not effective so it should be completely eliminated. There’s also a strict adherence to bylaws. They’re regarded as an artifact and are so sacrosanct they can upend a ballot over something as trivial as numbering, or the editorial process.
And yet, eliminating EV UI with no eye towards a viable replacement seems to contradict the CA/B Forum’s own bylaws. Or at least the spirit of them. We haven’t even gotten to the third sentence of the bylaws and already it’s obvious the Forum is only handling half of its stated purpose. Yes, it’s working towards a more secure web. No, it’s not even making an attempt to find a “more intuitive method of displaying secure sites to internet users,” which is one stated purpose of the Forum. The browsers are finding a better way to display “unsecure” sites, but that’s not what the bylaws say, is it?
Anyway, that’s not what we came here to talk about.
We’re here to talk about max validity, and the fact that this is the second time the push to shorten certificate lifespans to a single year has failed. As we stated earlier, the first time there was a compromise. This time? Not so much. Ballot SC 22 was introduced by Google’s Ryan Sleevi in August and came up for a vote from September 3-9.
Why did SC 22 Fail?
If you go back and read the comments from the CAs – and not with a cynical, suspicious predisposition – you see that most of the Certificate Authorities didn’t object to the idea of shorter validity so much as the timing and the changes to how long validation information could be re-used. There were two major timing factors:
- It wasn’t that long ago that SSL/TLS certificate lifespans were shortened to two years. Customers don’t care about deliberations at the CA/B Forum, they care that they’re now being asked to renew certificates twice as often. That drives up their costs, not just financially but in terms of time spent and resources used, and potentially poses security challenges and a greater risk of outages by requiring more frequent replacement.
- The effective date for the ballot was March 1, then it was amended to April 1, which really wasn’t a help so much as it was a sarcastic act of passive aggression. CAs were requesting at least a year before the change became effective. Google’s rep gave them an extra month. Read into that whatever you like.
Beyond that, enterprise customers were concerned that replacing certificates yearly, at that scale, was too large an undertaking to be ready by next Spring. Automating by then just isn’t feasible. According to one CA’s customer survey, when it comes to major enterprises, 75% of the customers use no automation today and 9% only use “1% to 10% automation.” Another CA whose customer base is mostly small businesses, found out that among its 2,732 respondents 22% had never heard of any automation tools, another 36% used no automation, and 17% were “not sure.” That created another debate that quickly devolved into name-calling, but we’ll get to that in a minute.
The less discussed side of SC22
Then there was the less obvious intention of the ballot, which was to reduce the amount of time CAs could re-use validation data. Now, there’s a little bit of subtext here. There are elements of the Forum that feel that validation, specifically business authentication, is broken. That’s part of the whole EV debate.
One of the dynamics at work in the discussion of validation is that the non-CA members (browsers) of the Forum generally have a theoretical knowledge of validation, whereas the CAs are actually performing it and have a different perspective owing to their experience with the process. Neither viewpoint is wrong. In a truly collaborative environment, the two differing perspectives could even be a strength. But as it stands, even validation is a contentious topic at the Forum.
Right now, you can re-use validation data for 27 months (13 months for EV). After the initial validation, a CA can issue any certificate you order with only a domain control check if you ask for a new domain. That means it’s near instant. For large organizations this is a godsend. Reducing the amount of time that validation information stays “fresh” and can be re-used means organizations and CAs must validate more often. That consumes time and resources from both the CA and the organization getting the certificate. It’s also another move that devalues higher-validation certs because the re-validation process is more burdensome.
As we stated earlier, it wasn’t the max validity that was the problem for many CAs so much as it was the validation restrictions. And the timing.
CA/B Forum Ballot SC22 – The Voting
When push came to shove, the ballot failed with CAs 20-11. To pass, this ballot needed two-thirds of the CAs and a majority of the browser voters. It didn’t even come close with the CAs. Here’s the final breakdown of the voting:
The measure was overwhelmingly supported by the browsers. All seven votes were in favor:
Ok, now let’s talk about the ugly fault-lines that this process exposed, and what can be done to fix them. Maybe.
The CA/B Forum – “damned if you do and damned if you don’t”
Coming from the world of sports journalism and having only entered this space in the last few years, I have to admit, the CA/B Forum might be the internet’s best argument against high school bullying. Feelings seem to get hurt easily. Things get petty quickly. And there’s a power dynamic that looms over every discussion.
Here’s how Jeremy Rowley of DigiCert described it.
…any CA voting [on this ballot] is “damned if you do and damned if you don’t”. I suspect almost everyone will wait until the last minute to vote, to see how the ballot is going to turn out, for a couple of reasons.
First, CAs are getting a lot of different input and some CAs believe there are some business advantages to opposing this ballot, regardless of the outcome. Any CA that votes for this ballot will have other CAs use that as marketing material against the voting CA. We saw this with the last change (from 3 years to 2 years) and with the underscore character deprecation. Regardless of outcome, with 85% of the customers answering the survey against the change in validity period, the risk is high that a CA will face some negative reaction if they vote in the affirmative.
On the other side, all of the browsers seem universally aligned with the change and the security reasons for the change are (imo) compelling. To avoid being dragged into the middle, the safest bet for a CA is to not vote. The second safest bet is to wait until the ballot draws out and then vote no if the ballot will pass. I dislike the politics on the voting so I’m hoping calling attention to them will mix things up.
Second, voting “no” gives the CA someone to blame for the change that insulates the CA from ramification of the change. The blame then can be on the ballot voters for the shortened lifecycle. Angry customers can be deflected to the browsers/CAs who vote yes…”
DigiCert ended up not voting. Can you blame them?
Dimitris Zacharopoulos is the current CA/B Forum Chair, he represents the Hellenic Academic & Research Institution’s Certification Authority or HARICA and posted the following:
HARICA does not agree with further reducing the lifetime of TLS Certificates as it creates unnecessary burden to site operators. If the main problem we are trying to solve is Domain Validation and the fact that some domains are “changing owners”, thus putting at risk the new Domain owners as BygoneSSL demonstrated, we should look for alternatives rather than having millions of site operators replace millions of Certificates at a shorter timeframe.
HARICA ended up abstaining.
This can’t happen. DigiCert is one of the largest, most trusted CAs in the industry. HARICA’s representative is the CA/B Forum Chair himself. Both organizations felt it would be disadvantageous to even VOTE on a measure that will have a massive impact on not just the industry but the entire internet.
Other CAs have privately admitted they voted “Yes” for fear of reprisals. They just didn’t want to risk having “another gun” pointed at them.
That’s indicative of the fact that something’s wrong. Something is broken. And again, it’s just the tip of the iceberg.
Browbeating and a general lack of civility
One of the most common refrains that’s used against CAs at the Forum is that there’s a lack of research presented on their part. So, in anticipation of the discussion period and voting on ballot SC22, three CAs surveyed their customers: DigiCert, GoDaddy and Entrust Datacard, as noted above.
The surveys were immediately rejected by some of the browsers. As the ballot’s author, Google, writes:
While I certainly understand that academic rigor is not the objective here, it’s important to consider these facts when evaluating the results DigiCert shared. I also wanted to help DigiCert here; as they’re laboriously working to summarize respondents’ free-form text results, if the survey was spoiled, or if the desired objective was fundamentally unobtainable due to the selection method, perhaps it’s not worth that effort and not worth further discussion?
To be clear, he’s telling DigiCert not to even bother with transcribing the write-in comments from its survey because he faults the methodology and doesn’t view the data as worth the time. This is a CA sharing feedback from its own customers. Again, you can’t win here.
When Entrust Datacard turned in its survey results the discussion turned to enterprise certificate management practices and the hesitation to embrace automation. Eric Mill, a fellow at TechCongress and non-CA/non-browser associate member of the Forum argued this was even more of a compelling reason to vote for the change.
That so many organizations continue to mistakenly believe that doubling their manual renewal rate would cause severe disruption, or that automation of certificate issuance is an unimportant aspect of their own organizational security and agility, is a compelling reason to proceed with this ballot and mandate reduced certificate lifetimes. The survey results make clear that many current enterprise customers are not prioritizing this work on their own, and that a mandate covering all CAs at once is likely the only effective way to drive progress here.
And while that’s a valid point, it also served as a catalyst for the deterioration of the good faith debate.
Dean Coclin is a former CA/B Forum Chair and moved from Symantec to DigiCert when it acquired the CA. When he suggested moving the effective date back, Google’s representative excoriated him.
…If CAs are unable to make configuration changes within 6 months, or if they’re concerned they’re unable to revalidate a fraction of their certificates sooner than expected, then I do fear that those CAs are in dire straights, and it may be time to discuss phasing out trust in them… Considering that the ecosystem needs to be prepared for replacing certificates with five days notice – for example, when it’s discovered that the CA was failing to validate certificates and instead issuing them for “Default City” in “Some-State” – I truly hope that 18 months notice is more than adequate. Certainly, I hope you of all people can appreciate the importance of ensuring customers are able to migrate away, in a timely fashion, from CAs that are or are being distrusted, and the challenges faced by these customers if their certificates become untrusted before they expire, or if they forget how to replace or revalidate.
The effective date was then moved back by a single month to April 1st.
This comment is just dripping with subtext. And frankly, had it come from anyone besides Google it would’ve JUST been in bad taste. But coming from Google? It takes on a much more ominous tone.
It was Google that pushed Symantec – where Coclin worked at the time – out of the CA industry. And here is the same representative that prosecuted that case, implying – no not implying, outright suggesting – that any CA that can’t comply with this ballot could be distrusted by the browsers. And then alluding to Coclin’s own experience navigating the Symantec distrust.
That would be like the government declaring eminent domain on your old farm, then showing up at your new farm and insisting, “this is my land and if you don’t grow sweet corn, or can’t grow sweet corn – by my deadline – we’re going to take your farm.” Then turning to you and adding, “but you already know about losing your farm, don’t you?”
And again, DigiCert didn’t vote. It said it was against the measure on its blog, but was browbeaten into not voting.
And it wasn’t just DigiCert that experienced this glaring lack of collegiality. Doug Beattie, a VP at GlobalSign, noted that the ballot lacked a “comprehensive security analysis” and asked Google to provide some data in support of this ballot so that his organization could communicate it to customers. Not an unreasonable request:
We need a list of issues and attacks that have resulted in, or have a high potential to harm the eco system and exactly how these proposed changes help more than they hurt. Including the reasons across dozens of emails and multiple lists isn’t consumable by the community which will be most impacted by the proposed changes. Describe them without calling out specific CAs or organizations, intimidating the community, or demeaning those that have expressed their opinion in the past.
The opposite happened. In a 1600+ word response Google’s rep detailed four bug reports made against GlobalSign before concluding:
I appreciate that you repeated your call here for the reasons, but you’ve continually skirted engaging on the Substance, and instead presented it as an argument about presentation instead, and so naturally, we haven’t been able to engage.
Google and its browser cartel
The reason I’ve laid out these excerpts from the CA/B Forum discussion period centering around Ballot SC 22 is to give examples of Google intimidating CAs. Google, by virtue of its positioning, exerts considerable influence. Its browser is the most widely-used by a huge margin and its search engine is dominant.
But it’s not just influence over the CAs that Google leverages. It also has a lot of unseen influence over the browser makers. Google is one of Mozilla’s biggest patrons and its economic health is largely contingent upon its search deal with Google, which helped grow its revenue by 8% in 2017. Earlier this year when Firefox was having connectivity issues, it accused Google of damaging it for years to come. Even the criticism had to be measured. After all, Google’s “a partner.” Google said it was a mistake. But regardless of its intentions – a message was received.
Beyond that, Microsoft Edge and Opera both run on Google’s open source Chromium project. Mozilla and Apple both use Google Safe Browsing as their anti-phishing service. And all the browsers need Google’s search and advertising divisions behind them. Pissing off Google is dangerous. And that’s compounded by the fact Google’s rep wields its influence like a cudgel. For example:
The Web PKI is full of stories like this, where users and well-meaning server operators are harmed by the CAs and the recalcitrant customers, such as yourself, and wholly rely on Browsers to do the Right Thing by the user and to protect their interests.
And that all sounds great – what a noble talking point. Is it true though? Ehhh.
- Google threatens to shut down its news service in Europe
- Google, Facebook manipulate users to circumvent GDPR
- Google loses landmark “Right to be Forgotten” case
- Google fined $57 million for GDPR Violations
- Google fined $5 billion for “anti-competitive” Android practices
- Google, Facebook sued for $8.8 billion by EU privacy advocate
- Google’s $80 billion conflict of interest
- Google allegedly developing censored Chinese search engine
- Google’s updated security UI leads to explosion of HTTPS phishing
- Google mis-issues, revokes thousands of digital certificates
And then there’s the recent settlement with the US Federal Trade Commission.
In fact, as 50 US state attorneys general and the US Department of Justice launch an antitrust case against Google, it might be worth pointing out the influence Google flouts, the fact it’s a major supporter of the free CA Let’s Encrypt, that it serves on two policy-making “modules” of Mozilla, as well as the fact its made a multitude of moves to undermine OV and EV SSL certificates and reduce lifespans and validation limits, to align with Let’s Encrypt’s free, 90-day, automated DV-only issuance practices and business model. One might even wonder if all these proposals are intended to drive website owners to move their sites to cloud services that are also CAs, etc., which could more directly favor Google’s own business model.
You COULD make a case it’s consolidated its influence and is exercising excessive market power.
But that would conspiratorial. That’s not what we’re proposing. Our fix is much simpler. The CA/Browser Forum Chair simply needs to hold all parties to account. Evenly. While I’ve seen the chair call out bad behavior from CA representatives before, I’ve never seen it come down on the browser side. The Forum’s Bylaws are explicit about the level of professionalism and consideration that must be exercised by its participating members.
Almost none of the remarks quoted above are made in the spirit of those bylaws and some are, frankly, thinly-veiled threats. Considering the current Chair is from a CA that opposed the ballot and then abstained, you wonder if Google’s influence doesn’t have some impact here, too.
The CA/B Forum is a phenomenal idea and it has the potential to be an example for how other industries can collaborate and regulate themselves. But it has to start with collegiality and respect for diverse opinions. Nobody at the Forum is a “bad guy.” Nobody is looking to cheat or steal. You have to extend that much faith for any debate to work. It’s literally the definition of a good faith discussion.
This article wasn’t fun to write. Nobody wants to spend time talking about why the CA/B Forum is falling short or how the industry is starting to polarize and imperil itself. The fact that a group of CAs – which compete in the same space for customers – get along better with one another than with the browsers is all the indication you need that the CA/B Forum is broken.
So, let’s fix it. Let’s start treating each other professionally. Nobody has to be friends. You don’t have to spend time together outside of the meetings. Nobody’s going to force you all to go get a beer. But treating others with dignity and respect is a kindergarten-level virtue. And one that’s best not forgotten if we want the CA/B Forum to approach what it used to be and still has the potential to be again.
Do you have suggestions on how we can improve the level of discourse at the CA/B Forum? We’d love for you to share them with us.