PKI Mistake #1: Poorly Managing Your PKI Certificates Leads to Outages & Downtime<\/h2>\n\n\n\n
It doesn\u2019t matter whether you\u2019re a mom-and-pop store or one of the world\u2019s leading governments; you\u2019re not infallible. Important tasks \u2014 such as managing your certificates to ensure they\u2019re replaced before their expiration dates \u2014 will fall through the cracks and get forgotten if you don\u2019t have a management system and documented processes in place to help you manage them.<\/p>\n\n\n\n
All x.509 digital certificates have an expiration date. Industry leaders have agreed that website security certificates (SSL\/TLS certificates)<\/a> must have a maximum validity period of no more than 398 days. This means that by no later than the end of the 398th<\/sup> day, your digital certificate will expire and leave you floundering to find out why your app, website, or other online services associated with that certificate are inaccessible.<\/p>\n\n\n\n If you don\u2019t manage your lifecycle properly (meaning you lack visibility of the PKI assets within your network and IT ecosystem and don\u2019t have the processes and tools in place to manage them), then I\u2019m certain things won\u2019t go the way you hope.<\/p>\n\n\n\n Certificate expirations alone lead to serious issues, including service outages and risks of man-in-the-middle attacks. AppViewX data from a Forrester Consulting study<\/a> shows that nearly 60% of reported data breaches were related to digital certificates. Of those, more than half of survey respondents indicated those incidents cost upwards of $100,000 per outage.<\/p>\n\n\n\n One of the biggest challenges for big organizations and enterprises is that they\u2019re responsible for managing so many<\/em> digital certificates. KeyFactor\u2019s State of Machine Identity Management 2023<\/a> report puts the average number of certificates issued within an organization at 256,000. That\u2019s the equivalent of issuing 701 certificates per day<\/strong> for a year (or one certificate every two or so minutes).<\/p>\n\n\n\n Yup, there\u2019s a reason why we always emphasize the importance of proper certificate lifecycle management and the use of lifecycle automation. We\u2019ve repeatedly seen what happens when public and private organizations lose sight of the certificates within their networks and systems\u2026<\/p>\n\n\n\n We all know the story about Equifax\u2019s headline-making data breach.<\/a> The data breach heard \u2018round the world resulted in the compromise of more than 145.5 million consumers\u2019 personal data. As it turns out, a PKI certificate had expired 10 months before the breach occurred. This expired asset prevented the credit agency from inspecting its traffic, during which time some advantageous cybercriminals found and exploited an Apache Struts-related vulnerability on Equifax\u2019s network.<\/a><\/p>\n\n\n\n The result? They were able to hide in the company\u2019s encrypted network traffic and remained undiscovered for more than two-and-a-half months (76 days). Imagine having some unknown and unwanted intruder in your network. How would you respond to the situation?<\/p>\n\n\n\n We\u2019ve also previously shared about Ericsson\u2019s expired certificate issue<\/a> that knocked out services for tens of millions of cellular users in December 2018. As a result, customers in upwards of 11 countries<\/a> were unable to make or receive phone calls or texts<\/a>. The cause? An expired certificate tied to Ericsson\u2019s management software that was used by several European and Japanese telecommunication companies, including O2 (the United Kingdom\u2019s biggest mobile service provider).<\/p>\n\n\n\n Needless to say, it wasn\u2019t a good look for Ericsson or the other cellular service providers using their software.<\/p>\n\n\n\n Last year, SpaceX\u2019s Starlink satellite internet services system experienced downtime<\/a> for several hours due to an expired digital certificate in an unspecified system.<\/p>\n\n\n\n It goes to show how something seemingly so simple \u2014 a single certificate reaching its end-of-life date \u2014 could impede an application or service for a time. That downtime translates to inaccessibility of services for Starlink\u2019s customers and their customers\u2019 customers.<\/p>\n\n\n\n The music streaming service Spotify experienced a certificate expiration<\/a> that knocked out its streaming services for an hour back in August 2020. The culprit? An expired SSL\/TLS certificate. Thankfully, the downtime was over pretty quick, and listeners could get back to enjoying their music.<\/p>\n\n\n\n Spotify\u2019s online podcast platform, Megaphone, also reported experiencing a service outage<\/a> in May 2022 that lasted nearly 10 hours in total. The downtime that cut podcasters off from their listeners was an expired certificate that someone hadn\u2019t renewed ahead of time. <\/p>\n\n\n\n Even the federal government can be remiss when managing expiring certificates. In early 2019, we reported that 130 U.S. government websites\u2019 SSL\/TLS certificates weren\u2019t renewed in time<\/a>. The expired certificates left websites and their associated services unreachable to U.S. users during the critical period of a federal government shutdown. Furthermore, once the certificates expired, the sites\u2019 infrastructures were vulnerable to threat actors who might have wanted to take advantage of the situation.<\/p>\n\n\n\n Because many of the websites have implemented HTTP strict transport security (HSTS), which forces secure HTTPS connections, it means that browsers using Google Chromium\u2019s HSTS preload list won\u2019t provide users with an option to bypass the security warning. (NOTE: Bypassing insecure website warnings is never a good idea.)<\/p>\n\n\n\n When a PKI digital certificate expires, is revoked, or otherwise invalidated, it means the system will become inaccessible and\/or go down. The result? A really<\/em> bad time for you and your customers \u2014 you\u2019ll experience everything from downtime and service outages to compliance concerns and lost customer relationships.<\/p>\n\n\n\n The best defense is a good offense. What I mean by that is being proactive in your certificate lifecycle management is crucial to enabling you to protect the security of your organization\u2019s most sensitive data.<\/p>\n\n\n\n A good certificate lifecycle management system makes it a lot easier to avoid certificate expirations, with:<\/p>\n\n\n\n Everyone makes mistakes. Sometimes, it\u2019s as simple as not double-checking a setting or accidentally clicking a toggle that changes a setting from \u201cprivate\u201d to \u201cpublic.\u201d But the stakes are pretty high when it comes to encryption keys.<\/p>\n\n\n\n Hopefully, we all know that passwords and cryptographic private keys should never be shared or posted openly. Unfortunately, some companies make mistakes that wind up sharing their cryptographic keys and other secrets on the internet.<\/p>\n\n\n\n Regardless of whether it\u2019s an accident or done intentionally by a malicious insider, the fact is that key exposures due to security misconfigurations and poor key security practices keep happening. Here are a handful of such examples\u2026<\/p>\n\n\n\n SOCRadar researcher Can Yoleri<\/a> discovered that the popular European automaker had a misconfigured server that was accessible on the internet. The Microsoft Azure storage dev bucket, which contained all sorts of sensitive data, including private keys, was set to \u201cpublic access\u201d rather than \u201cprivate.\u201d<\/p>\n\n\n\n But just what did this bucket contain? Some of this sensitive company data included access information for Azure containers, specifics on cloud services, and private keys that are used to access other private buckets. This means that whatever data was in those buckets that the keys were used to secure were also at risk of compromise.<\/p>\n\n\n\n The good news? According to BMW\u2019s statement in a TechCrunch report on the incident<\/a>, \u201cno customer or personal data was impacted as a result.\u201d But that still doesn\u2019t excuse storing keys in an Azure storage bucket instead of a secure key management solution such as a hardware security module (HSM) or a key vault.<\/p>\n\n\n\n When it comes to movement in the realm of artificial intelligence (AI), it\u2019s no secret that leaders within the global tech industry are engaged in a \u201crat race.\u201d Companies are scrambling to build bigger and better mousetraps, meaning better and more effective AI technologies. As part of these efforts, they invest vast amounts of time, people, and financial resources to the cause. <\/p>\n\n\n\n Unfortunately, things go wrong sometimes. In this case, Wiz\u2019s research team<\/a> discovered that 38 terabytes (TB) of Microsoft\u2019s sensitive data, which included private keys, passwords, and private communications, were exposed on GitHub. The data exposure occurred when Microsoft\u2019s AI team published a bucket of open-source training data using the wrong security configuration setting.<\/p>\n\n\n\n But the bad news doesn\u2019t end there for Microsoft\u2026<\/p>\n\n\n\n In April 2021, the company\u2019s consumer signing system crashed<\/a>. It\u2019s thought that this crash, which triggered a data dump that included a signing key that should have been (but was not) redacted, is what gave the threat actor an opportunity to nab the secret. However, Microsoft doesn\u2019t know with certainty<\/a> whether this was the true cause of the key compromise: \u201cDue to log retention policies, we don\u2019t have logs with specific evidence of this exfiltration by this actor, but this was the most probable mechanism by which the actor acquired the key.\u201d \u00a0<\/p>\n\n\n\n If this is the case, Elias Groll at Cyberscoop<\/a> summed up the situation nicely:<\/p>\n\n\n\n \u201cIf a crash dump is the garbage generated by a failing computer system, stealing a signing key via a crash dump is like rifling through a garbage can and discovering the key to the family safe.\u201d<\/em><\/p>\n<\/blockquote>\n\n\n\n While our long-term readers are intimately familiar with this subject, the impact of key compromises might be a new concept for some of our newer readers. So, let\u2019s quickly cover what all of this means.<\/p>\n\n\n\n When private key compromises and exposures occur<\/a>, any data secured using the associated key pairs is at risk of compromise. The exposed data can be stolen, bartered, sold, or used by nefarious individuals to commit any number of crimes. Depending on the types of keys exposed, they also can be used to impersonate whatever entities are associated with them.<\/p>\n\n\n\n Another issue with poorly managing your keys is that when private keys get lost<\/a>, then you can kiss any data that they keys were used to encrypt goodbye.<\/p>\n\n\n\n When it comes to securely managing secrets such as keys and passwords, the best thing to do is adhere to industry best practices. This includes securely storing your keys using a key management system<\/a> or key management service<\/a>. For passwords, you\u2019ll want to adhere to strong password security guidelines<\/a>.<\/p>\n\n\n\n Depending on the use case, there are several types of secure key storage mechanisms:<\/p>\n\n\n\n Yes, cryptographic keys need to be rotated. It\u2019s never a good idea to keep using the same keys. A good practice is to rotate them out for new ones periodically. However, the industry seems to lack consensus regarding key rotation cycles. For example, NIST says key rotation should take place every 12 months<\/a>, whereas Trend Micro recommends approximately every 45 days<\/a>.<\/p>\n\n\n\n5 Real-World Examples of What Happens When Certificates Are Allowed to Expire<\/h3>\n\n\n\n
Equifax<\/h4>\n\n\n\n
Ericsson<\/h4>\n\n\n\n
SpaceX<\/h4>\n\n\n\n
![\"A](\"https:\/\/www.thesslstore.com\/blog\/wp-content\/uploads\/2024\/03\/musk-tweet-starlink-certificate-outage-1024x568.png\")
Spotify<\/h4>\n\n\n\n
![\"A](\"https:\/\/www.thesslstore.com\/blog\/wp-content\/uploads\/2024\/03\/spotify-tweet-outage-status-1024x717.png\")
U.S. Government<\/h4>\n\n\n\n
Consequences: Why It\u2019s a \u201cBig Deal\u201d When Your Certificates Expire<\/h3>\n\n\n\n
What You Can Do to Avoid These Certificate-Related Issues<\/h3>\n\n\n\n
The Solution: Use Certificate Lifecycle Automation<\/h4>\n\n\n\n
\n
PKI Mistake #2: Poor Key Management Lets Bad Guys Steal Your Keys<\/h2>\n\n\n\n
3 Real-World Examples of When Organizations Didn\u2019t Protect Their Keys<\/h3>\n\n\n\n
BMW<\/h4>\n\n\n\n
Microsoft <\/h4>\n\n\n\n
Incident #1<\/h5>\n\n\n\n
Incident #2<\/h5>\n\n\n\n
\n
Consequences: Why It\u2019s a \u201cBig Deal\u201d When These Key Management-Related Exposures Occur<\/h3>\n\n\n\n
What You Can Do to Avoid Suffering a Similar Fate<\/h3>\n\n\n\n
Securely Store Your Keys<\/h4>\n\n\n\n
\n
Rotate Your Keys<\/h4>\n\n\n\n