For the second time in two years certificate expiry bites LinkedIn.
Once again, LinkedIn has underscored the importance of good certificate management. For the second time in two years an SSL/TLS certificate expired and caused downtime for the social media site. This time it was LinkedIn’s link shortener, lnkd.in, that was the culprit. On Tuesday, May 21, desktop users started to see those dreaded SSL connection error messages.
Obviously, LinkedIn scrambled to fix the issue and had a new SSL/TLS certificate installed quickly, but it also provided us with yet another teachable moment in the process, too.
So, today we’re going to talk about what exactly happened, why it happened again and what could have been done to prevent it.
Let’s hash it out.
Certificate Expiration strikes again!
This is what happens when your SSL certificate expires. We write articles about you and use you as a cautionary tale to reaffirm the value of a proper certificate management solution.
And given this is the second time this has happened in two years, you have to wonder whether LinkedIn learned its lesson the first time. Certificate expirations are easily the second worst thing that’s happened to LinkedIn lately. The worst is its addition of video. Now everyone is a motivational speaker and… anyway, certificate expiration can cause a lot more trouble than just downtime, too. It can end up costing money.
How much this actually cost LinkedIn is difficult to quantify. The outage only affected desktop users trying to use the company’s link shortener. LinkedIn is now owned by Microsoft, and Microsoft is unlikely to give specifics about how many users couldn’t connect or exactly how long the outage lasted, but in 2018 LinkedIn’s revenues stood at a whopping $5.3 billion, so it wouldn’t be surprising if the actual cost from downtime alone was in the millions.
Certificate Expiration by the Numbers
Downtime is a much bigger issue for smaller companies, LinkedIn will weather the outage costs easily. The bigger issue for a company the size of LinkedIn is the brand damage it incurs. While, admittedly, some users will never know, or will see and not know, what to make of it – more tech savvy users may start to form critical opinions about how seriously LinkedIn takes security.
Again, this is a very preventable problem provided you’re using the right tools. But this is now twice in just a couple of years. It’s also not the only security-related issue LinkedIn has encountered. It all starts to add up after a while and given some the rather sensitive nature of some of the information people share with LinkedIn – losing trust could prove disastrous.
What exactly happened?
To understand what happened you need to start with a quick explanation of link shortening and the real reason companies actually do it. It’s easy to view it as just a clever way to slip links into character-capped social media posts, like back before Twitter didn’t count URLs – but the real reason it’s done is for analytics. Companies need to be able to track who clicks on their content, whether it’s people or bots, where they came from, etc.
So, when you use a link shortener it’s kind of like putting a proxy in the middle of the connection. The link connects with the link shortening domain, which inspects the traffic and redirects it along to its intended destination. In order for this to work, there actually needs to be two different connections that occur, so there need to be two different SSL certificates. The link shorterning domain needs one and then the actual website itself needs one, too.
In this case, the SSL/TLS certificate securing the link shortening domain, lnkd.in, expired and anyone attempting to click on shortened links was unable to connect.
This seems preventable…
Oftentimes, when an expiration occurs this is how it happens. It’s not unusual, but it’s also not common that companies let the certificate on their flagship website expire. It’s usually a certificate on an application server or even a machine identity that expires and causes the service outage. These can be a lot harder to find and replace.
Shadow IT is any kind of IT product or service that was acquired via non-standard channels. SSL certificates and digital certificates in general are one of the most common Shadow IT items. The cause for this isn’t malicious – it’s typically not someone knowingly circumventing the systems – it just stems from the fact that everything needs to have a certificate nowadays and they are oftentimes requisitioned outside of standard channels.
This creates a considerable risk though, because certificate management is a major compliance and security concern – and when things go sideways it costs companies money and people jobs. And once again LinkedIn has demonstrated that just a single certificate expiration can shut down operations for a great many customers if it happens in a sensitive enough place.
LinkedIn was fortunate that the certificate was visible enough to find quickly, normally your IT or security team has to scramble to find where the certificate was deployed, who issued it, where the keys are stored, etc. And that’s assuming you even notice right away. There’s no shortage of examples of this happening, perhaps the most well-known being the fact Equifax’s breach went undetected for 76 days because of an expired certificate that went unaddressed.
The solution is to have an organized approach to certificate management. You need to be able to scan all of your networks for certificates, you need an interface to manage them all and maintain visibility with. And then you need to establish policies and procedures for managing every stage of the lifecycle: issuance, rotation, renewal, revocation, etc.
If it’s possible – automate everything. Remove as much of the human element as possible.
Too often digital certificates are an afterthought until it’s too late, and when that happens it can shine a light on your company for all the wrong reasons.
As usual leave any comments or questions below…