{"id":12799,"date":"2020-08-21T09:58:50","date_gmt":"2020-08-21T13:58:50","guid":{"rendered":"https:\/\/www.thesslstore.com\/blog\/?p=12799"},"modified":"2023-04-10T18:05:15","modified_gmt":"2023-04-10T22:05:15","slug":"the-day-the-music-died-certificate-expiration-takes-down-spotify","status":"publish","type":"post","link":"https:\/\/www.thesslstore.com\/blog\/the-day-the-music-died-certificate-expiration-takes-down-spotify\/","title":{"rendered":"The Day the Music Died: Certificate Expiration Takes Down Spotify"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\"><em>\u201cSomething expired deep inside<br>The day the music died\u201d<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Earlier this week, the music streaming service Spotify went down for about an hour. (We \u201c<em>heard it from a friend who heard it from another<\/em>\u201d\u2026on Twitter.) All signs point towards a certificate expiration being the root cause of the downtime. So, what happened to take Spotify offline? And what does an hour of downtime add up to for Spotify?<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Let\u2019s hash it out.<span id=\"newline\"><\/span><\/p>\n\n\n<span style=\"--tl-form-height-m:150.25px;--tl-form-height-t:121.4583px;--tl-form-height-d:121.4583px;\" class=\"tl-placeholder-f-type-shortcode_12753 tl-preload-form\"><span><\/span><\/span>\n\n\n<h2 class=\"wp-block-heading\" id=\"h-spotify-fans-cry-a-river-on-twitter\">Spotify Fans Cry A River on Twitter<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">As usually happens when a popular internet services goes down, many people \u201c<em>Heard It Through the Grapevine<\/em>\u201d on Twitter. Starting around 8AM EST on August 19<sup>th<\/sup> Spotify users started posting on Twitter saying they couldn\u2019t access the service:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"589\" height=\"381\" src=\"https:\/\/www.thesslstore.com\/blog\/wp-content\/uploads\/2020\/08\/twitter-spotify-down.png\" alt=\"Twitter: is Spotify down for anyone else\" class=\"wp-image-12800\" srcset=\"https:\/\/www.thesslstore.com\/blog\/wp-content\/uploads\/2020\/08\/twitter-spotify-down.png 589w, https:\/\/www.thesslstore.com\/blog\/wp-content\/uploads\/2020\/08\/twitter-spotify-down-300x194.png 300w\" sizes=\"auto, (max-width: 589px) 100vw, 589px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">As always, many of the Tweets were pretty funny. (Hey, if there\u2019s no music\u2026you may as well enjoy some Twitter comedians!)<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"592\" height=\"380\" src=\"https:\/\/www.thesslstore.com\/blog\/wp-content\/uploads\/2020\/08\/twitter-spotify-down-cds.png\" alt=\"Spotify down, what to play\" class=\"wp-image-12801\" srcset=\"https:\/\/www.thesslstore.com\/blog\/wp-content\/uploads\/2020\/08\/twitter-spotify-down-cds.png 592w, https:\/\/www.thesslstore.com\/blog\/wp-content\/uploads\/2020\/08\/twitter-spotify-down-cds-300x193.png 300w\" sizes=\"auto, (max-width: 592px) 100vw, 592px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"589\" height=\"408\" src=\"https:\/\/www.thesslstore.com\/blog\/wp-content\/uploads\/2020\/08\/twitter-spotify-down-check.png\" alt=\"Checking Twitter to see if Spotify is down\" class=\"wp-image-12802\" srcset=\"https:\/\/www.thesslstore.com\/blog\/wp-content\/uploads\/2020\/08\/twitter-spotify-down-check.png 589w, https:\/\/www.thesslstore.com\/blog\/wp-content\/uploads\/2020\/08\/twitter-spotify-down-check-300x208.png 300w\" sizes=\"auto, (max-width: 589px) 100vw, 589px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Of course, a few fans of competing services took the opportunity to throw some shade at Spotify:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"585\" height=\"430\" src=\"https:\/\/www.thesslstore.com\/blog\/wp-content\/uploads\/2020\/08\/twitter-spotify-down-apple.png\" alt=\"Spotify down, Apple Music\" class=\"wp-image-12803\" srcset=\"https:\/\/www.thesslstore.com\/blog\/wp-content\/uploads\/2020\/08\/twitter-spotify-down-apple.png 585w, https:\/\/www.thesslstore.com\/blog\/wp-content\/uploads\/2020\/08\/twitter-spotify-down-apple-300x221.png 300w\" sizes=\"auto, (max-width: 585px) 100vw, 585px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">In short, Twitter lived up to its reputation as the best downtime monitor in the world \u2013 up to the minute status information and as an added bonus, you might get a laugh from all the salty posts.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">About an hour and a half later, Spotify announced that everything was back up and running normally:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"616\" height=\"425\" src=\"https:\/\/www.thesslstore.com\/blog\/wp-content\/uploads\/2020\/08\/spotify-down-official-tweet.png\" alt=\"Spotify status official tweet\" class=\"wp-image-12804\" srcset=\"https:\/\/www.thesslstore.com\/blog\/wp-content\/uploads\/2020\/08\/spotify-down-official-tweet.png 616w, https:\/\/www.thesslstore.com\/blog\/wp-content\/uploads\/2020\/08\/spotify-down-official-tweet-300x207.png 300w\" sizes=\"auto, (max-width: 616px) 100vw, 616px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-what-caused-spotify-to-go-down\">What Caused Spotify to Go Down?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Spotify doesn\u2019t appear to have made an official announcement explaining the technical details of what happened. But once again, Twitter comes to the rescue with the details. Louis Poinsignon, a Network Engineer at Cloudflare, seems to have sleuthed out the issue:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"592\" height=\"625\" src=\"https:\/\/www.thesslstore.com\/blog\/wp-content\/uploads\/2020\/08\/spotify-down-ssl-certificate.png\" alt=\"SSL certificate expiration that caused Spotify downtime\" class=\"wp-image-12805\" srcset=\"https:\/\/www.thesslstore.com\/blog\/wp-content\/uploads\/2020\/08\/spotify-down-ssl-certificate.png 592w, https:\/\/www.thesslstore.com\/blog\/wp-content\/uploads\/2020\/08\/spotify-down-ssl-certificate-284x300.png 284w\" sizes=\"auto, (max-width: 592px) 100vw, 592px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">The certificate expired at 12 PM GMT, which was 8AM US Eastern Time, just a few minutes before the tweets from music-deprived users started rolling in.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-how-big-of-a-deal-was-this-certificate-expiration\">How Big of a Deal was this Certificate Expiration?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Here at Hashed Out, we\u2019ve <a href=\"https:\/\/www.thesslstore.com\/blog\/what-happens-when-your-ssl-certificate-expires\/\">occasionally highlighted the consequences of letting a certificate expire<\/a>. Here are a few of the incidents we\u2019ve examined before:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>US Government shutdown causes dozens of sites to go down due to SSL certificate expirations<\/li>\n\n\n\n<li>Ericsson lets certificate expire, 32 million people lose cellular service<\/li>\n\n\n\n<li>Equifax misses a breach for 76 days because of an expired certificate<\/li>\n\n\n\n<li>Users experience VPN issues after Cisco lets one of its SSL certificates expire<\/li>\n\n\n\n<li>Pokemon Go goes down after a certificate expires<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Check out our article \u201c<a href=\"https:\/\/www.thesslstore.com\/blog\/what-happens-when-your-ssl-certificate-expires\/\">What happens when your SSL certificate expires?\u201d<\/a> for the full stories.<\/p>\n\n\n<span style=\"--tl-form-height-m:861.156px;--tl-form-height-t:899.625px;--tl-form-height-d:899.625px;\" class=\"tl-placeholder-f-type-shortcode_12653 tl-preload-form\"><span><\/span><\/span>\n\n\n<p class=\"wp-block-paragraph\">Now, let me be clear\u2014we\u2019re not bashing these organizations. Not at all. The reality is that <a href=\"https:\/\/www.thesslstore.com\/blog\/new-study-finds-75-of-cios-are-concerned-about-tls-certificate-related-security-risks\/\">large organizations have 10,000\u2019s of certificates,<\/a> and keeping track of all of those expiration dates is a gargantuan task.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">But here\u2019s the harsh reality: letting even a single certificate expire can have a huge impact.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">How much trouble can a single certificate expiration cause? One easy way to very roughly estimate the cost of a certificate expiration is to look at how much revenue the company would typically make in that time period.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Spotify\u2019s revenue in 2019 was $7.44 billion, which equates to $20,383,561 per day, or <strong>about<\/strong> <strong>$1,273,9726<\/strong> in an hour-and-a-half.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Now, most of Spotify\u2019s revenue comes from subscriptions, so their revenue didn\u2019t literally drop to $0 for that hour and-a-half. But direct revenue losses are only part of the cost\u2014certificate-related downtime can be costly in a variety of ways:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Direct revenue losses<\/li>\n\n\n\n<li>Reduced new customer acquisition<\/li>\n\n\n\n<li>Increased customer support time\/costs<\/li>\n\n\n\n<li>Customer churn (existing customers go elsewhere)<\/li>\n\n\n\n<li>Damage to brand reputation<\/li>\n\n\n\n<li>Potential compliance issues<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Another way to calculate the cost of certificate-related downtime is to ask: how many users were impacted?<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">According to <a href=\"https:\/\/www.businessofapps.com\/data\/spotify-statistics\/\">BusinessofApps<\/a>, Spotify has about 286 million monthly active users, and they listen for an average of 25 hours per month. That means that any given time, there are approximately <strong>9,781,200 users<\/strong> \u201c<em>all in the mood for a melody<\/em>\u201d and listening on Spotify.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It doesn\u2019t appear that Spotify was down for all users in all locations (some users reported they were able to listen with the desktop app but not the mobile app, while others reported the opposite) but it definitely looks like there were millions of Spotify users who were NOT \u201c<em>feelin&#8217; alright<\/em>\u201d while the service was down.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-implementing-the-right-certificate-management-practices-is-key\">Implementing the Right Certificate Management Practices is Key<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">We don&#8217;t know how Spotify\u2019s certificate expired without being caught. Was the certificate not being monitored? Did some wires get crossed? Either way, the bottom line is: <strong>certificate management is challenging<\/strong>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">As organizations require more and more digital certificates for various types of use cases (SSL\/TLS, device, code signing, S\/MIME, etc.) implementing effective certificate management practices has become absolutely critical.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Don\u2019t make the mistake of managing your certificates manually. Certificate management methods such as spreadsheets and calendar reminders might be OK if you just have one or two certificates, but they\u2019re far too error prone for organizations with many certificates.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The first step to minimize the chance of certificate-related downtime is to automate the most critical certificate management functions. Specifically, organizations should implement a certificate management tool with the following features:<\/p>\n\n\n\n<ol class=\"wp-block-list\" type=\"1\">\n<li>Automated certificate discovery (with public and private scanners)<\/li>\n\n\n\n<li>Automated expiration notifications to responsible parties and organizational admins<\/li>\n\n\n\n<li>Automated notification escalations for imminent expirations<\/li>\n\n\n\n<li>Where possible, automated certificate renewal and installation<\/li>\n\n\n\n<li>Automated checks and notifications for security vulnerabilities (e.g. POODLE)<\/li>\n\n\n\n<li>Automated approval flows for staff to request certificates through official channels (to discourage shadow certificates installed without the organization\u2019s knowledge)<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">This screenshot from DigiCert\u2019s CertCentral management platform is a great example showing how thousands of certificates can be summarized in a single screen, making it easy for IT admins to identify and update any expiring certificates:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"723\" height=\"425\" src=\"https:\/\/www.thesslstore.com\/blog\/wp-content\/uploads\/2020\/08\/certcentral-dashboard.png\" alt=\"DigiCert CertCentral certificate management platform screenshot\" class=\"wp-image-12806\" srcset=\"https:\/\/www.thesslstore.com\/blog\/wp-content\/uploads\/2020\/08\/certcentral-dashboard.png 723w, https:\/\/www.thesslstore.com\/blog\/wp-content\/uploads\/2020\/08\/certcentral-dashboard-300x176.png 300w\" sizes=\"auto, (max-width: 723px) 100vw, 723px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Of course, a dashboard like this is only useful if it actually includes all of the certificates in your organization. And that\u2019s why an automated certificate discovery feature is also very important\u2014it finds all your certificates and loads them into the dashboard so you can see and manage them in one convenient place.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-stayin-alive-with-a-little-preparation\"><em>&#8220;Stayin&#8217; Alive&#8221;<\/em>&#8230;With a Little Preparation<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">But these certificate management features only work if your organization has correctly implemented them. The best certificate management tool in the world won\u2019t save you if the discovery feature isn\u2019t setup correctly or the notifications are sent to the wrong person.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The old ways that used to work aren\u2019t good enough anymore. To succeed in the age of \u201calways on\u201d encryption, certificate management needs to be prioritized and \u201cbaked in\u201d to your existing IT and cybersecurity workflows. For many organizations, the certificates themselves aren\u2019t the most important part\u2014the critical part is the management of the certificates. That\u2019s why DigiCert now ships CertCentral with their certificates\u2014but, it\u2019s still up to your organization to implement the certificate management features and processes to avoid situations like this.<\/p>\n\n\n<span style=\"--tl-form-height-m:150.25px;--tl-form-height-t:121.4583px;--tl-form-height-d:121.4583px;\" class=\"tl-placeholder-f-type-shortcode_12753 tl-preload-form\"><span><\/span><\/span>\n\n\n<h2 class=\"wp-block-heading\" id=\"h-summary\">Summary<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Certificate-related downtime, like the incident Spotify experienced earlier this week, has unfortunately become more and more common these days.&nbsp; Organizations are accumulating more certificates than ever, and the task of managing them all while staying ahead of expirations has become a significant challenge.&nbsp; Fortunately, effective certificate management practices can be a huge help in easing this headache, making it easier for your company to avoid costly downtime.&nbsp; The result?&nbsp; Happier customers and more revenue.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u201cSomething expired deep insideThe day the music died\u201d Earlier this week, the music streaming service Spotify went down for about an hour. (We \u201cheard it from a friend who heard&#8230;<\/p>\n","protected":false},"author":23,"featured_media":12808,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":"","tve_updated_post":"","tve_custom_css":"","tve_user_custom_css":"","tve_globals":{},"tcb2_ready":0,"tcb_editor_enabled":0,"tve_landing_page":"","_tve_header":"","_tve_footer":""},"categories":[130,10200],"tags":[180,12691,287],"class_list":["post-12799","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-everything-encryption","category-monthly-digest","tag-expiration","tag-featured","tag-ssl-certificate-expired","post-with-tags"],"views":15284,"jetpack_featured_media_url":"https:\/\/www.thesslstore.com\/blog\/wp-content\/uploads\/2020\/08\/spotify.jpg","_links":{"self":[{"href":"https:\/\/www.thesslstore.com\/blog\/wp-json\/wp\/v2\/posts\/12799","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.thesslstore.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.thesslstore.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.thesslstore.com\/blog\/wp-json\/wp\/v2\/users\/23"}],"replies":[{"embeddable":true,"href":"https:\/\/www.thesslstore.com\/blog\/wp-json\/wp\/v2\/comments?post=12799"}],"version-history":[{"count":0,"href":"https:\/\/www.thesslstore.com\/blog\/wp-json\/wp\/v2\/posts\/12799\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.thesslstore.com\/blog\/wp-json\/wp\/v2\/media\/12808"}],"wp:attachment":[{"href":"https:\/\/www.thesslstore.com\/blog\/wp-json\/wp\/v2\/media?parent=12799"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.thesslstore.com\/blog\/wp-json\/wp\/v2\/categories?post=12799"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.thesslstore.com\/blog\/wp-json\/wp\/v2\/tags?post=12799"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}