The TLS handshake explained: what it is, why it happens and how to fix it when it fails.
Let’s talk about one of the least understood aspects of SSL/TLS: the SSL handshake, or more appropriately the TLS handshake.
As you know, SSL/TLS certificates are requisite for serving your website via HTTPS. And in the last few years, SSL/TLS has gotten much more publicity and attention from the public and the IT industry as part of a larger focus on user security. In 2016, 50% of all web traffic was encrypted with SSL/TLS for the first time. Now, three years later, 79% of the world’s top sites are using HTTPS.
But many people still don’t really know how SSL/TLS works.
One of the most important parts of the SSL/TLS protocol is the SSL/TLS handshake. The handshake is where each connection begins and where the technical underpinnings of SSL/TLS are established.
The ‘SSL/TLS handshake’ is the technical name for the process that establishes an HTTPS connection. Most of the hard work involved in the SSL/TLS protocol is done here. It’s a process that has evolved since the original SSL protocol was first created in 1996, with each new iteration becoming faster, with less overhead. Last year, the IETF finalized TLS 1.3, which features a completely overhauled handshake.
There are now effectively two handshakes.
So, today we’re going to explain the SSL/TLS handshake, for both TLS 1.2 and TLS 1.3, starting high level and then boring down into the specifics:
- Exchanging encryption capabilities
- Authenticating the SSL certificate
- Exchanging/generating a session key.
We’ll also talk about those pesky TLS handshake failed errors and give some suggestions on how to fix them.
Let’s hash it out.
Table of contents
- The TLS handshake explained: what it is, why it happens and how to fix it when it fails.
- A Summary of the TLS Handshake
- The TLS 1.2 Handshake: Step by Step
- The TLS 1.3 Handshake: Step by Step
- The costs of the TLS Handshake
- TLS 1.2 Handshake vs. TLS 1.3 Handshake – Improvements
- The TLS Handshake – Negotiating Cipher Suites
- The TLS Handshake – Authentication
- The TLS Handshake – Key Exchange
- Fixing the SSL/TLS Handshake Failed Error
- Wrapping Up…
Manage Digital Certificates like a Boss
14 Certificate Management Best Practices to keep your organization running, secure and fully-compliant.
A Summary of the TLS Handshake
An HTTPS connection involves two parties: the client (the one who is initiating the connection, usually your web browser), and the server. These two parties are the ones that ‘shake hands.’ The purpose of the SSL/TLS handshake is to perform all the cryptographic work needed to have a secure connection. This includes authenticating the SSL certificate being used, and generating an encryption key.
As we said at the top, the TLS handshake accomplishes 3 main things:
- Exchanging cipher suites and parameters
- Authenticating one or both parties
- Creating/Exchanging symmetric session keys
Let’s start by explaining conceptually what’s happening with each.
Negotiating Cipher Suites
Every piece of software is different. Web browsers are the most common client and features can vary between popular browsers like Google Chrome, Mozilla Firefox and Microsoft Internet Explorer. Likewise, on the server side, popular operating systems like Windows Server, Apache, and NGINX all have very different feature support. And all of this gets even more complicated when you add in custom configurations. So, the first steps of the TLS handshake require the client and server to share their capabilities so they can find the cryptographic features they mutually support.
Once a client and server agree on the exact encryption methods they will use – this is called a cipher suite – the server sends the client its SSL certificate.
Upon its receipt, the client checks to make sure the certificate is “authentic.” This is an extremely important step. To truly have a secure connection, you can’t just encrypt your data, you also need to know it’s being sent to the right website/organization. SSL/TLS certificates provide that authentication. But the way they do it depends on the cipher suite being used.
All trusted SSL certificates are issued by a Certificate Authority (CA), which is a company that has been approved to issue digital certificates. These organizations must follow strict guidelines for issuance and validation so that the certificates they issue continue to be trusted. This is primarily to ensure that you can only get a certificate for a website or company you truly own. You can think of a CA kind of like a notary in this context – its signature signifies the entity on the certificate is who it says it is.
During the authentication portion of the TLS handshake, the client performs several cryptographically secure checks to make sure the certificate provided by the server is authentic. This includes checking the digital signature and making sure the certificate originates from a trusted CA.
During this stage, the client also verifies that the server owns the private key associated with the certificate. All SSL certificates contain a key pair consisting of a public and private key. The public key is used to encrypt data and the private key is used to decrypt. This is known as “asymmetric” public key encryption because the two functions are performed by different keys. The encryption is effectively one-way.
With the most common public key cryptosystem, RSA, the client will encrypt random data with the public key that needs to be used to generate the session key. The server will only be able to decrypt and use that data if it has the private key, which provides proof of possession.
If another type of cryptosystem, like ECC, is used this process changes, but proof of possession is always checked.
The last part of the TLS handshake involves creating the “session key,” which is the key that will actually be used for secure communication.
Session keys are “symmetric,” meaning the same key is used for encryption and decryption. These keys can achieve strong encryption much more efficiently than asymmetric keys, making them appropriate for sending data back and forth in an HTTPS connection.
The exact method for generating the key varies based on the cipher suite that was chosen, with the two most common schemes being RSA and Diffie-Hellman.
To end the handshake, each party lets the other know they have done all the necessary work, then both run check-sums to make sure the handshake occurred without any malicious tampering or corruption.
The entire SSL handshake takes place in a few hundred milliseconds and it’s all behind the scenes. It’s the first thing that must happen in an HTTPS connection, even preceding the webpage loading.
Once the SSL handshake is complete, the encrypted and authenticated HTTPS connection begins and all the data being sent and received between you and the server is protected.
Up until TLS 1.3, every time you revisited a site, the handshake would occur again. The TLS 1.3 handshake supports 0 RTT, or Zero Round Trip Time Resumption, which greatly increases the speed for returning visitors. Previously, many servers implemented a “resumption” process for efficiency and speed, but 0 RTT turns it up to 11.
The TLS 1.2 Handshake: Step by Step
Each TLS handshake involves a series of steps, which accomplish the three main tasks we summarized above: exchanging encryption capabilities, authenticating the SSL certificate, and exchanging/generating a session key.
For those looking to understand the exact process, here are the steps involved with a TLS 1.2 handshake (using RSA – we have an example of a Diffie-Hellman TLS 1.2 handshake farther down the article).
The steps are illustrated to show the client on the left and the server on the right.
- The first message is called the “ClientHello.” This message lists the client’s capabilities so that the server can pick the cipher suite that the two will use to communicate. It also includes a large, randomly picked prime number called a “client random.”
- The server politely responds with a “SeverHello” message. In this message it tells the client what connection parameters it has selected from the provided list and returns its own randomly selected prime number called a “server random.” If the client and server do not share any capabilities in common, the connection terminates unsuccessfully.
- In the “Certificate” message, the Server sends its SSL certificate chain (which includes its leaf certificate and intermediate certificates) to the client. To provide authentication to the connection an SSL certificate is signed by a CA, which allows the client to verify that the certificate is legitimate. Upon receipt, the client performs several checks to authenticate the certificate. This includes checking the certificate’s digital signature, verifying the certificate chain, and checking for any other potential problems with the certificate data (expired certificate, wrong domain name, etc). The client will also make sure the server has possession of the certificate’s private key. This is done during the key exchange/generation process.
- This is an optional message, only needed for certain key exchange methods (Diffie-Hellman) that require the server provides additional data.
- The “Server Hello Done” message tells the client that it has sent over all its messages.
- The client then provides its contribution to the session key. The specifics of this step depend on the key exchange method that was decided on in the initial “Hello” messages. In this example we’re looking at RSA, so the client is going to generate a random string of bytes called a pre-master secret, then encrypt it with the server’s public key and transmit it.
- The “Change Cipher Spec” message lets the other party know that it has generated the session key and is going to switch to encrypted communication.
- The “Finished” message is then sent to indicate that the handshake is complete on the client side. The Finished message is encrypted, and is the first data protected by the session key. The message contains data (MAC) that allows each party to make sure the handshake was not tampered with.
- Now it’s the server’s turn to do the same. It decrypts the pre-master secret and computes the session key. Then it sends its “Change Cipher Spec” message to indicate it is switching to encrypted communication.
- The server sends its “Finished” message using the symmetric session key it just generated, it also performs the same check-sum to verify the integrity of the handshake.
After these steps the SSL handshake is complete. Both parties now have a session key and will begin to communicate with an encrypted and authenticated connection.
The TLS 1.3 Handshake: Step by Step
Let’s start by taking a look at the TLS 1.3 handshake in motion, and then we’ll delve into the improvements that have been made. As you can see, the TLS 1.3 handshake is significantly shorter than its predecessors.
- As with the TLS 1.2 handshake, a Client Hello message starts the handshake off, but this time it comes packaged with a lot more information. TLS 1.3 has reduced the number of supported ciphers from 37 to 5. We’ll get into what that means specifically in a bit, but in the context of the handshake it means that the client can guess what key agreement/exchange protocol will be used in addition to sending its key share from whatever protocol it guessed.
- The server will respond with its own Server Hello message, again, servers are exceedingly polite. Like the 1.2 handshake, it will also send along the certificate at this point. And, provided the client guessed right and the two have agreed on the same AEAD protocol, the server sends its own portion of the key share, calculates the session key and ends with a server finished message.
- Now that it has all of the relevant information, the client will authenticate the SSL certificate and use the two key shares to calculate its own copy of the session key. When this is complete it sends its own Finished message.
Now let’s talk about the TLS 1.3 and its improvements over TLS 1.2.
The costs of the TLS Handshake
Historically, one of the complaints about SSL/TLS was that it strained servers with its additional overhead. This also lent itself to the now-defunct perception that HTTPS was slower than HTTP.
That’s entirely owed to the fact that handshakes, prior to TLS 1.2, were an expensive process and at scale, they can seriously tax a server. Even TLS 1.2 handshakes can slow things down if enough of them are being performed simultaneously. Authentication, encryption and decryption are all expensive processes.
For smaller websites this likely isn’t going to slow things down noticeably, but for enterprise websites that get hundreds of thousands of visitors daily, it can be a big problem.
Each new iteration of the handshake takes substantial steps towards lightening the process. TLS 1.2 makes two round trips while TLS 1.3 has it down to a single round trip with support for 0 RTT.
But part of the problem with the 1.2 handshake actually has very little to do with the SSL/TLS handshake process itself, and more to do with one of the methods for accomplishing it: asymmetric encryption. Specifically the key exchange portion. Asymmetric encryption is 10,000 times slower than symmetric encryption.
We’re going to touch on this a little more in-depth later. But RSA, in particular, is a major offender. So much so that it’s been removed entirely as a key exchange option in TLS 1.3. Its public/private keys are cumbersome: 2048-bit and up, which requires a lot of resources to use, and again, at scale it can really add latency.
That’s not to say RSA isn’t fast, speed and the computing power it takes to run a process are not necessarily the same thing. RSA can still be fast, it just taxes the server more.
It’s not the only culprit, either. ECC/Diffie-Hellman (ECDH) key exchange, by comparison, is lighter – but can still require substantial resources in some configurations (when paired with ECDSA specifically).
All of this is to say that the SSL/TLS handshake has historically been expensive and, if not configured properly, often scaled poorly.
That’s why it’s become a fairly common practice for larger organizations to offload their SSL/TLS functions to load balancers and other edge devices. Offloading all of those handshakes and the encryption functions they facilitate offers two key benefits: it frees up resources on the application servers and provides an opportunity to inspect traffic.
How that method plays with TLS 1.3 and its refined handshake is still being fleshed out.
TLS 1.2 Handshake vs. TLS 1.3 Handshake – Improvements
The best way to understand the improvements TLS 1.3 made to the SSL/TLS handshake is to start by discussing round trips. For the sake of making it easier to understand, we pulled the handshake into ten distinct steps in the explanation above. In reality, a lot of these things are happening side-by-side, simultaneously, so it’s common for them to be grouped together and referred to as round trips.
Optimally, the TLS 1.2 handshake makes two round trips. There are scenarios where additional round trips may be needed, so when we refer to the number of round trips we’re talking about in an optimal scenario. I’ve illustrated it here:
By contrast, TLS 1.3’s handshake makes just a single round trip, though it’s probably important to point out that this can be a little misleading, too. I might even go so far as to say it’s a round trip and a half. Still, it’s significantly quicker than TLS 1.2’s.
Simplified Cipher Suites
The reduction in round trips comes down to the aforementioned reduction in cipher suite support. Nobody ever set out to have 37 acceptable suites, it just sort of evolved into that. Each time a new algorithm was added new combinations had to be added and before long the IANA was administering 37 different cipher suites.
That’s bad for two reasons. A. That amount of variability lends itself to mis-configurations, which make internet users vulnerable to know exploits. B. It made everything about configuring SSL more confusing. Truth be told, SSL/TLS is niche enough that even a lot of security professionals aren’t really well-versed in its specifics. Undue complexity is not appreciated.
So, to cut some of the fat, TLS 1.3 cut down the number of supported ciphers from 37 to 5, it also whittled down the size of its cipher suites by reducing the number of negotiations they cover, cipher suites used to negotiate:
- The types of certificates that are supported
- Hash function (SHA1, SHA2, etc.)
- Message Authentication Code (MAC) function
- Digital Signatures
- Key exchange algorithm
- Symmetric encryption cipher
- Cipher mode
That’s why you typically see four distinct algorithms represented in TLS 1.2 cipher suites:
- Key Exchange
- Bulk Cipher
- Hashing Algorithm
TLS 1.3 cipher suites include just two negotiable ciphers:
- Bulk cipher
- HKDF (KMAC-based Extract-and-Expand Key Derivation Function) Hash
The IETF removed support for all but the most secure, most efficient algorithms eliminating confusion by consolidating choices. Most notably, the entire choice with regard to key exchange has been removed. Diffie-Hellman Ephemeral schemes are now the only game in town, which is what allows the client to send its key share information alongside the Client Hello in the first part of the handshake. RSA encryption has been completely removed along with all other static key exchange schemes. Perfect Forward Secrecy has become a requirement with TLS 1.3.
That being said, there is one potential Achilles heel for PFS in TLS 1.3…
Zero Round Trip Resumption – 0-RTT
Zero Round Trip Resumption or 0-RTT is something that the tech world has been clamoring for, and it’s here with TLS 1.3. As we just covered, the TLS handshake has historically been expensive: taxing servers and adding latency to connections. So the idea of shrinking it is an attractive one. 0-RTT does just that by storing some secret information about the client, typically a Session ID or Session Tickets, for use when the two parties connect in the future.
For all the benefits 0-RTT brings, it actually presents a couple of potential pitfalls, too. It makes clients susceptible to replay attacks, where an attacker that somehow manages to get access to an encrypted session can take the 0-RTT data, which includes the client’s first request, and send it to the server again. That’s a topic for another day though.
The bigger issue is that 0-RTT handshakes could undermine perfect forward secrecy by providing an avenue to decrypt old sessions. Still, it’s not an easy thing to exploit and likely a small price to pay for an extremely useful feature.
Securing more of the TLS 1.3 Handshake
One of the biggest fears, early on, with the handshake was the amount of it that gets sent in plaintext. Obviously, that opens the door to problems, so the more of the handshake that can be secured, the better.
In the TLS 1.2 handshake the negotiation stages of the handshake were not secured, rather a simple MAC is used to ensure nobody tampered with what was transmitted.
- Client Hello
- Cipher suites supported by the client
- Server Hello
- Server-selected cipher suite
- Key Share
- SSL/TLS Certificate
- Digital Signature
That MAC acts as a monitor, but not a safeguard. It can’t prevent it from happening in the first place. You may have heard of a downgrade attack before, this is the attack vector where they’re performed. If a server and a client both support outmoded cipher suites – information that is readily available if you’re listening in on the connection – a Man-in-the-Middle attackercan swap out the server-chosen cipher suite for a weaker one that is mutually supported.
Downgrade attacks aren’t what actually do the harm though, they just open the door for other known exploits that affect whatever version that was downgraded to. That’s why they’re particularly dangerous.
The TLS 1.3 handshake digitally signs the early portions, which makes more of the handshake secure and mitigates downgrade attacks, and by extension many of the exploits they facilitate. This also provides an avenue to more quickly and efficiently authenticate the server by verifying possession of the private key.
Now let’s take a look at how these upgrades to the TLS 1.3 handshake play out across the three main functions of the SSL/TLS handshake itself…
The TLS Handshake – Negotiating Cipher Suites
Let’s start by digging a little deeper into cipher suites. In the past we’ve covered cipher suites in-depth, but if you don’t feel like reading all that we’ll give an abbreviated version here. A cipher suite is a collection of algorithms that determine the parameters of a secure connection.
At the start of any connection, the very first interaction, the Client Hello, is a list of cipher suites that are supported. The server looks for the best, most secure option that is mutually supported and responds with its choice. You can look at a cipher suite and figure out all of the parameters of the handshake and the connection.
Let’s look at a couple examples:
TLS 1.2 Cipher Suites
This is an example of a TLS 1.2 cipher suite, I’ve color-coded it to make it easier to read:
- TLS is the protocol
- ECDHE is the key exchange algorithm
- ECDSA is the authentication algorithm
- AES 128 GCM is the symmetric encryption algorithm
- SHA256 is the hashing algorithm.
In the example above, we’re using Elliptic Curve Diffie-Hellman Ephemeral for key exchange and Elliptic Curve Digital Signature Algorithm for authentication. DH can also be paired with RSA (functioning as a digital signature algorithm) to accomplish authentication.
As we’ll cover in the next section, this entire portion of the cipher suite has been removed from negotiation in the TLS 1.3 handshake.
TLS 1.2 has 37 available ciphers in total, though not all of them are advisable. Here’s a list of the most widely-supported cipher suites:
TLS 1.3 Cipher Suites
TLS 1.3 has made countless improvements over its predecessors, which is good considering it was in development for about a decade. The IETF removed support for older outmoded algorithms and streamlined everything, shortening the entire handshake from two round trips to one and reducing the sizes of cipher suites from four negotiations/algorithms to two.
The number of supported cipher suites has also dropped from 37 to five. Here’s an example of a TLS 1.3 cipher suite:
- TLS is the protocol
- AES 256 GCM is the Authenticated Encryption with Associated Data (AEAD) algorithm
- SHA384 is the Hashed-Key Derivation Function (HKFD) algorithm
We already know we’re going be using some version of Diffie-Hellman Ephemeral key exchange, we just don’t know the parameters, so that means that the first two algorithms in the TLS 1.2 cipher suite are no longer needed. Those functions are still occurring, they just no longer need to be negotiated during the handshake.
From the example above you can see we’re using AES or the Advanced Encryption Standard, as our symmetric bulk cipher. It’s running in Galois Counter Mode using 256-bit keys. We’re using SHA384 (part of the SHA2 family) as our HKFD hash algorithm. If that’s all French to you, we’ll explain it in the next session.
Here are the five supported TLS 1.3 cipher suites:
So, what specifically changed from TLS 1.2 to TLS 1.3?
It’s important to remember that the two biggest priorities in defining 1.3 were to make it safer and to improve performance. To that end, TLS 1.3 has re-thought key generation and phased out static key generation schemes, as well as ones with known vulnerabilities. Reducing the number of key-gen/exchange options is what makes it possible for the client to guess the key agreement scheme and send its key share during the Client Hello – it’s going to be a Diffie-Hellman Ephemeral-based scheme.
The TLS 1.3 handshake also improves certain processes, for instance, message authentication and digital signatures. We just covered how more of the handshake is going to be secured with a digital signature with TLS 1.3. Well, whereas before a MAC function was used as a check-sum to ensure message integrity and a digital signature scheme was used for authentication, with the TLS 1.3 handshake the entire handshake is digitally signed, handling both message and server authentication simultaneously.
Finally, in addition to phasing out old key-gen/exchange algorithms, TLS 1.3 eliminates old symmetric encryption or bulk encryption ciphers. When you get into symmetric encryption there are two types of ciphers: block ciphers and stream ciphers.
A block cipher encrypts data in fix-sized blocks. There’s a bit of a problem with this method though. If your message isn’t the exact length of the key, then you have to do some extra work. Shorter messages need to be padded, larger ones need to be broken up, padded and then reassembled. There are plenty of exploits that target the padding used by block ciphers. If you can successfully guess the padding, it makes it easier to decrypt the message or guess the key.
The other type of cipher is a stream cipher, which encrypts streams of data in pseudo-random sequences of arbitrary length, called key streams. Stream ciphers are a popular choice because they’re easy to implement and they’re fast. In fact, the best way around the aforementioned problem with block ciphers is to set your block cipher to act like a stream cipher, this is called counter mode, with the most popular example being cipher block chaining (CBC).
In the example cipher suite from above, we saw AES_256 in Galois Counter Mode. AES is a block cipher, in this case it’s using 256-bit keys and it’s being run as stream cipher in Galois Counter Mode.
That’s because TLS 1.3 has done away with block ciphers entirely, in addition to deprecating support for stream ciphers with known vulnerabilities like RC4. The only type of symmetric ciphers allowed by TLS 1.3 are called Authenticated Encryption with Additional Data (AEAD) ciphers, which combine both encryption and message authentication (MAC) into a single function.
Now let’s talk about keys.
The TLS Handshake – Authentication
As we just touched on, one of the biggest factors in terms of how taxing an SSL/TLS handshake is on your servers stems from the asymmetric encryption being employed, specifically for authentication and key exchange.
Historically, the two main options for key exchange are RSA and Diffie-Hellman (DH), you often see DH associated with Elliptic Curves nowadays. There are some major similarities, but also some fundamental differences between these two approaches to key exchange.
Or, put another way, an RSA TLS handshake is different from an ECDH one.
RSA vs. Diffie-Hellman/ECC – A Quick History
RSA, as we have covered before, makes use of prime factorization and modular arithmetic. It’s very difficult to factor large prime numbers – this is part of what gobbles up CPU resources. Diffie-Hellman is sometimes called exponential key exchange, indicating its use of exponentiation (in addition to modular arithmetic) – but in truth Diffie-Hellman itself doesn’t encrypt or decrypt anything at all. So calling it an encryption method instead of a mathematical underpinning might be a bit misleading.
A little history lesson might make it easier to understand this portion better. Back in 1976 Whitfield Diffie and Martin Hellman created a key exchange protocol based on the work of Ralph Merkle, whom the two have opined should also have his name on it.
The problem they were trying to solve was how to securely exchange keys over an unsecure channel – even if an attacker is monitoring it. They solved that part, but there was one major problem: DH Key Exchange lacked authentication, so there was no way to verify the party on the other end of a connection.
That was basically the birth of Public Key Cryptographyand PKI. Not long after Diffie and Hellman introduced their key exchange protocol, the earliest iterations of the RSA cryptosystem were being finalized. Diffie and Hellman had the concept of public key encryption in mind, but hadn’t figured out the actual one-way encryption function yet.
It was Ron Rivest (the R in RSA), following a night of imbibing Manischewitz and observing Passover, that created the concept that eventually became the RSA cryptosystem.
In many ways RSA is a spiritual successor to DH. It handles:
- Key Generation
- Key Exchange
So, obviously RSA is more functional, it can handle both key exchange and digital signatures, meaning it authenticates in addition to securely exchanging keys. This is part of why the keys are bigger with RSA: there needs to be ample security for the signature.
While RSA handles authentication AND key exchange, Diffie-Hellman only facilitates key exchange. There are four common variants of the DH family:
- Diffie-Hellman (DH)
- Diffie-Hellman Ephemeral (DHE)
- Elliptic Curve Diffie-Hellman (ECDH)
- Elliptic Curve Diffie-Hellman Ephemeral (ECDHE)
Again, Diffie-Hellman by itself doesn’t authenticate anything. It needs to be paired with a digital signature algorithm. So, for instance, if you were using ECDH or ECDHE, most cipher suites will pair it with Elliptic Curve Digital Signature Algorithm (ECDSA) or RSA.
The TLS 1.2 Handshake – Authentication
As we just touched on, the additional functionality possessed by RSA, to authenticate with digital signatures, requires larger keys. They also have to be sufficiently long to be resistant to brute force attacks. Again, the size of these keys makes the cost of computing them, encrypting and decrypting with them during the handshake, quite expensive.
On the other hand, if Diffie-Hellman doesn’t do authentication, then what does? As we mentioned, DH is often paired with Elliptic Curve-based cryptography to tag team the authentication and key exchange functions.
ECC has much smaller key sizes that correspond with the elliptic curve they’re based on. There are five appropriate curves for this context:
- 192 bits
- 224 bits
- 256 bits
- 384 bits
- 521 bits
But that’s not the only difference between ECC public/private keys and RSA ones. They’re also used for two completely different purposes during the TLS handshake.
In RSA the Public/Private key pair is used to both authenticate the server, as well as exchange the symmetric session key. In fact, successfully using the private key to decrypt the pre-master secret IS what authenticates. the server.
With Diffie-Hellman, the Public/Private key pair is NOT used to exchange the symmetric session key. When Diffie-Hellman is involved, the private key is actually associated with the accompanying signature algorithm (ECDSA or RSA)
The RSA authentication process is entangled with its key exchange process. Or perhaps more appropriately, the key exchange is actually part of the authentication process.
When the client is presented with a server’s SSL certificate, it performs a series of checks to verify that the certificate is valid and trusted.
- It checks the digital signature using the public key.
- It checks the certificate chain, ensuring the certificate descends from one of the root certificates in its trust store.
- It checks the validity period of the certificate to make sure it’s not expired.
- It checks the revocation status of the certificate.
Assuming all of this is satisfactory, the final test comes when the client uses the public to encrypt the pre-master secret and send it to the server.
Any server can try to pass off any SSL/TLS certificate as its own. These are publicly available certificates, after all. So, while the client starts the authentication process by verifying the signature on the certificate, which tests the validity of the certificate itself. It needs to know that the server is the owner of the private key that made the signature in the first place. To do this it needs to see the private key in action.
So, if the server can decrypt the pre-master secret – the one that was originally encrypted with the associated public key – and use it to compute the master secret, it passes. That verifies that the server is the owner of the public/private key pair being used.
When Diffie-Hellman and ECDSA/RSA are in use, the authentication and key exchange unfold side-by-side. And this goes back to the keys and their different uses. RSA public/private keys are used for both key exchange and authentication. With DH + ECDSA/RSA, the asymmetric key pair is only used for the digital signature/authentication portion.
When the client receives the certificate it, still runs through the standard checks:
- Checks the signature on the certificate
- Checks the certificate chain
- Checks validity status
- Checks revocations status
But the proof of possession check for the private key is different. During the Server Key Exchange portion of the TLS handshake (step 4), the server uses its private key to encrypt the client and server randoms, as well as its Diffie-Hellman parameter. This functions as the server’s digital signature and the client can use the associated public key to verify that the server is the rightful owner of the key pair.
The TLS 1.3 Handshake – Authentication
In TLS 1.3, authentication and digital signatures still play a major role, but they’ve been elided from the cipher suites to simplify negotiation. These are implemented server-side, and continue to leverage several already-supported algorithms due to their security and ubiquity. The three main signature algorithms allowed in TLS 1.3 are:
- RSA (signature only)
- Elliptic Curve Digital Signature Algorithm (ECDSA)
- Edwards-curve Digital Signature Algorithm (EdDSA)
EdDSA is an elliptic curve-based algorithm. Unlike in the TLS 1.2 handshake, the authentication portion of the TLS 1.3 handshake isn’t coupled with the actual key exchange itself. Rather, it’s handled alongside the key exchange and message authentication. Remember earlier when we discussed how more of the handshake is now encrypted? This is where that starts paying dividends.
Instead of running a symmetric MAC scheme to verify the integrity of the handshake, the server signs the entire transcript hash when it returns the Server Hello with its part of the key share.
The client receives all of the information contained alongside the Server Hello and runs the standard series of checks to authenticate the SSL/TLS certificate it was just presented with. That includes checking the signature on the certificate, then checking it against the signature that was added to the transcript hash.
A match proves possession of the private key by the server.
The TLS Handshake – Key Exchange
We’re going to get into the actual specifics of the key exchange in just a second, but if you only take one thing away from this section, it should be this:
That may seem a bit abstract at the moment, hopefully it won’t by the end of this discussion. It’s absolutely critical to your understanding of the handshake.
RSA Key Exchange
Calling it RSA key exchange is actually kind of incorrect. It’s really RSA encryption. Remember earlier how I said that no encryption or decryption actually occurs with Diffie-Hellman key exchange?
Well, RSA uses asymmetric encryption to create the session key. Unlike DH, the public/private key pair plays a big role.
Here it is in motion:
- The client and server exchange two prime numbers (x and y) called randoms.
- The client generates a pre-master secret (a), then uses the server’s public key to encrypt it and send it to the server.
- The server decrypts the pre-master secret with the corresponding private key, both parties now have all three inputs, and mix them with some pseudo-random functions (PRFs) to produce a master secret.
- Both parties mix even more PRFs with the master secret and derive matching session keys.
Again, there are a lot of similarities between these two: they accomplish essentially the same thing, both use modular arithmetic and RSA was even inspired by DH.
But, in TLS 1.3 Diffie-Hellman has been declared the unequivocal winner.
Diffie-Hellman Key Exchange
Before we get into the process of Diffie-Hellman key exchange, let’s talk about modular arithmetic. Or skim over it, more appropriately. Modular arithmetic is a form of math where the numbers start over or wrap around after reaching a fixed value, known as the modulus. So, think of a clock. That’s an example of modular arithmetic.
Ok, so here’s how ECDH works:
- The client and server exchange two prime numbers (x and y) called randoms.
- One party picks a secret number called a pre-master secret (a) and computes: xa mod y. Then it sends the result (A) to the other.
- The other party does the same thing, picking its own pre-master secret (b) and computing xb mod y, then sending its value (B) back.
- The two parties finish this portion by taking the given values and repeating the operation. One computes Ba mod y, the other computes Ab mod y.
There’s a property of modulo exponents that dictates that each party is going to arrive at the same value, which is the key that will be used for symmetric encryption during the connection.
The TLS 1.2 handshake – Diffie-Hellman Edition
Earlier we covered the RSA version of the TLS 1.2 handshake. But now that we’ve gone through the way DH differs from RSA, let’s look at what a DH-based TLS 1.2 handshake looks like.
Again, there are plenty of similiarities between these two approaches, but rather than one algorithm handling authentication and key exchange the tasks are being divided up. Instead we’ll be using ECDHE for key exchange and ECDSA for authentication.
- Like with RSA, the client begins with a “ClientHello” message that includes a list of cipher suites, as well as the client random.
- The server responds with its own “ServerHello” message that includes its selected cipher suite and its server random.
- The server sends its SSL certificate, just like with an RSA TLS handshake, the client will run a series of checks to authenticate the certificate is valid, but because DH itself cannot authenticate the server, an additional mechanism is needed.
- To provide authentication, the server takes the client and server randoms, as well as the DH parameter that will be used to compute the session key, and encrypts them with its private key. This functions as the digital signature, the client will use the public key to verify the signature – and that the server is the rightful owner of the keypair – and respond with its own DH parameter.
- The server concludes this roundtrip with a “Server Hello Done” message.
- Unlike RSA, there is no need for the client to send the pre-master secret to the server using asymmetric encryption, instead the client and server use the DH parameters they exchanged earlier to arrive at a pre-master secret. Then each uses the pre-master secret that it just calculated to mutually arrive at the session key.
- The client sends a “Change Cipher Spec” message to inform the other party its switching to encryption.
- The client sends a final “Finished” message to indicate it has completed its part of the handshake.
- Likewise, the server sends a “Change Cipher Spec” message.
- The handshake concludes with a server “Finished” message.
DHE has two advantages over RSA
There are two primary reasons that the cryptography community favors using Diffie-Hellman Ephemeral over RSA: perfect forward secrecy and known vulnerabilities.
Perfect Forward Secrecy
Earlier you may have wondered what the word “ephemeral” means at the end of DHE and ECDHE. Ephemeral literally means “short lived.” And that may help you to understand Perfect Forward Secrecy (PFS), which is a feature of some key exchange (sometimes called key agreement) protocols. PFS ensures that the session keys that are exchanged cannot be compromised, even in the case the private key of the server’s certificate is compromised. In other word, it protects previous sessions from being retrieved and decrypted. PFS became a major priority after the Heartbleed bug. It’s a major component of TLS 1.3
RSA Key Exchange is vulnerable
We wrote about this in depth a couple of months ago, but there are known vulnerabilities that can exploit the padding used during key exchange in older versions of RSA (PKCS #1 1.5). This goes back to the encryption aspect. Remember, with RSA the pre-master secret has to be encrypted with the public key and decrypted by the private key. But when that smaller pre-master secret is put into the larger public key, it has to be padded. Most of the time if you try to guess the padding and send a spoofed request to the server, you’ll get it wrong and an Oracle will recognize the incongruence and filter it out. But there’s a non-negligible chance you could bombard the server with enough requests to guess the proper padding. This will cause the server to send an errant finished message, which in turn narrows down the possible value for the pre-master secret. That would allow the attacker to calculate and compromise the key.
That’s why RSA was removed in favor of Diffie-Hellman Ephemeral in TLS 1.3.
The TLS 1.3 Handshake – Key Exchange
In TLS 1.3’s handshake, because of the limited selection of key exchange schemes, the client can successfully guess the scheme and send their part of the key share during the opening portion (Client Hello) of the handshake.
RSA wasn’t the only key exchange scheme that got tossed out in TLS 1.3. Non-ephemeral Diffie-Hellman schemes were also done away with, as well as a litany of Diffie-Hellman parameters that were insufficiently secure.
What do I mean by insufficient parameters? Well, without going into a mathematics lessons, the hardness of Diffie-Hellman, and most public key cryptosystems, is the difficulty in solving discrete logarithm problems. Earlier when we mentioned that the client and server generate randoms at the outset of the handshake? Those need to be sufficiently difficult for anyone to calculate without knowing them both, lest the entire scheme be rendered useless. Diffie-Hellman schemes that couldn’t provide sufficiently large parameters have been done away with for TLS 1.3.
In all, there are only five acceptable DH schemes for key exchange, just enough to ensure that only secure algorithms are used, but large enough to compensate for any changes that could render one of them insecure in the coming years.
- At the start of the TLS 1.3 handshake, knowing that a DHE-named key agreement scheme will be used, the client includes a key share based on the guessed KA scheme with its Client Hello.
- The server receives this information and, provided the client guessed correctly, returns its own portion of key share with the Server Hello.
- The client and server both calculate the session keys.
It’s extremely similar to what happens with DH in the TLS 1.2 handshake, except with TLS 1.3 the key share happens earlier in the handshake.
Fixing the SSL/TLS Handshake Failed Error
For the vast majority of people, the only time the SSL/TLS handshake crosses their mind is when it fails and they can’t reach a website they were trying to visit. Mozilla Firefox is especially finnicky about that.
Fortunately, we’ve covered this extensively. No matter what causes a TLS handshake error, we know how to fix it. Unfortunately, most of these problems occur server-side, where the average internet user is powerless to fix them.
Still, if you’re having trouble with a TLS handshake error, check out our guide on how to fix it.
In it, we cover:
- Incorrect system time fixes
- Browser configuration mistakes
- Man-in-the-Middle interference
- Protocol mismatches
- Cipher suite mismatches
- Incorrect certificate errors
- SNI-enabled server errors
Now that you’ve read all about the SSL/TLS handshake and what specifically is occurring, you can probably see why some of these errors might occur. For instance, mis-matched cipher support, protocol mismatches and MITM exploits could be affecting the first phase, where encryption parameters are negotiated. Whereas an incorrect certificate or system time error could prevent authentication.
The most important thing we can tell you, regardless of what causes your TLS handshake failed error, do NOT turn off SSL/TLS or disable your security settings. Don’t give websites a pass on bad security. If you stop visiting, they’ll fix it. But don’t put yourself at risk.
The SSL/TLS handshake is a fascinating process that is critical to a secure internet, and yet it happens quickly and quietly behind the scenes where most people never give it a second thought.
At least, until something goes wrong.
As always, leave any comments or questions below…