Unexpected error causes browsers to report certificates as revoked.
Websites secured with GlobalSign had a bad day yesterday when a mishap with the CA’s OCSP service caused browsers to treat their certificates as if they had been revoked.
The Register originally reported on the issue, which was widespread, and affected sites including Wikipedia, Logmein, The Guardian, and Dropbox.
This prevented many users from reaching the affected sites, as their browsers believed the certificates to be revoked and blocked access. Reports of problems came from all across the globe, but outages were inconsistent. Some users did not have any problem at all.
OCSP, which stands for Online Certificate Status Protocol, is a revocation mechanism that clients (like your browser) use to check if an SSL certificate has been revoked by the Certificate Authority who issued it. When a certificate is revoked, browsers prevent connections being made with said certificate, because of possible security risks. OCSP works by querying the CAs server and getting a confirmation that the certificate is “good” or “revoked.”
All CAs operate their own OCSP service. Yesterday GlobalSign’s service started reporting widespread revocations as a result of a planned action to revoke one of their intermediate certificates.
GlobalSign explained the problem in an official statement:
“GlobalSign manages several root certificates and for compatibility and browser ubiquity reasons provides several cross-certificates between those roots to maximize the effectiveness across a variety of platforms. As part of a planned exercise to remove some of those links, a cross-certificate linking two roots together was revoked. CRL responses had been operational for 1 week, however an unexpected consequence of providing OCSP responses became apparent this morning, in that some browsers incorrectly inferred that the cross-signed root had revoked intermediates, which was not the case.”
OCSP is a delicate technology – because it relies on getting a live response from the CA’s server, it needs to have extremely high availability and speed. Otherwise, the OCSP check can add significant time to the SSL handshake which in turn delays the loading of the website.
Many CAs contract with a CDN (Content Delivery Network) to ensure that these conditions are met. In this case, CDNs were the cause of the problem. While GlobalSign quickly fixed the issue at its source – by removing the affected intermediate certificate from its OCSP database – that information was not reaching end users because CDNs were caching the previous information and still reporting that they were revoked.
For some end users, caching will cause this problem to persist for a few more days and into the weekend, so Globalsign has issued new intermediate certificates that won’t be affected. Sites using GlobalSign certificates can replace their current intermediate certs with these to sort out their revocation issues.
If you are a GlobalSign customer who was affected, the company has posted a troubleshooting guide that will help you resolve the issues. GlobalSign’s Status page reports any known problems or upcoming maintenance and it posts regular incident updates on its Twitter account.