| |

Internet Backend 06 ๐ŸŒ TLS/SSL Handshake and Encryption

โฑ๏ธ Estimated read: 12 min | |

TLS is not a single encryption algorithm. It is a protocol that negotiates the algorithms, authenticates the server, and establishes a shared session key, and then uses that key to encrypt the application data. The handshake is the negotiation and the key establishment, and the record layer is the encrypted data transfer. The two are distinct phases, and the security of the connection depends on both. The handshake must authenticate the server and establish a key that the attacker cannot derive, and the record layer must encrypt the data with a key that is unique to the session.

The version of the handshake determines the number of round trips and the security properties. TLS 1.2 uses a two-round-trip handshake with the RSA key exchange or the Diffie-Hellman key exchange. TLS 1.3 reduces the handshake to one round trip, removes the static RSA and Diffie-Hellman key exchanges, and requires the forward secrecy for all key exchanges . The forward secrecy is the property that the session key cannot be derived from the server’s long-term private key, so a compromised server key does not decrypt the past sessions. The TLS 1.3 makes the forward secrecy mandatory, and the TLS 1.2 makes it optional.

Key point: The TLS handshake authenticates the server with the certificate, negotiates the cipher suite, and establishes a shared session key with the Diffie-Hellman key exchange. The session key encrypts the application data with the symmetric cipher. TLS 1.3 uses one round trip, requires the forward secrecy, and encrypts the handshake after the ServerHello. TLS 1.2 uses two round trips and allows the static RSA key exchange without the forward secrecy.


Why the handshake is the security foundation

The authentication problem. The client connects to a server, and the server presents a certificate. The certificate contains the server’s public key and the identity information, and it is signed by a certificate authority. The client verifies the signature chain up to a trusted root, and it checks that the certificate is valid for the hostname it is connecting to. The authentication is the mechanism that prevents the man-in-the-middle attack, where an attacker intercepts the connection and presents its own certificate. Without the certificate verification, the Diffie-Hellman key exchange would establish a key with the attacker, not the server .

The key exchange problem. The client and the server need a shared key that the attacker cannot derive. The Diffie-Hellman key exchange is the mechanism: both sides generate an ephemeral private value, exchange the public value, and compute the shared secret. The attacker who observes the exchange cannot compute the shared secret without solving the discrete logarithm problem. The ephemeral keys are the forward secrecy mechanism: the private values are discarded after the session, so a compromised server key does not reveal the past session keys .

The cipher suite problem. The client and the server must agree on the algorithms: the key exchange algorithm, the authentication algorithm, the bulk encryption algorithm, and the MAC algorithm. The cipher suite is the combination, and the negotiation is the mechanism that selects the strongest suite that both sides support. The TLS 1.3 simplifies the cipher suite to the AEAD algorithms and the hash, and it removes the separate MAC and the separate key exchange algorithm from the suite .

The trade-off. The security has a cost. The TLS 1.2 handshake adds two round trips before the application data can be sent. The TLS 1.3 reduces it to one round trip, and the 0-RTT mode reduces it to zero for the resumed connections, but the 0-RTT has the replay risk . The certificate verification adds the latency of the OCSP or the CRL check, and the forward secrecy adds the cost of the ephemeral key generation. The trade-off is the security against the latency, and the protocol version and the cipher suite are the choices that set the balance.


a. The TLS 1.2 handshake and the two round trips

The TLS 1.2 handshake starts with the ClientHello, which contains the client’s random value, the supported cipher suites, and the supported extensions. The server responds with the ServerHello, which contains the server’s random value and the selected cipher suite. The server then sends the Certificate, the ServerKeyExchange (if the key exchange requires it), and the ServerHelloDone .

Client                                Server
ClientHello          -------->
                     <--------      ServerHello
                                    Certificate
                                    ServerKeyExchange
                                    ServerHelloDone
Finished             -------->
                     <--------      Finished
Application Data     <------->      Application Data

The client verifies the certificate, generates the pre-master secret, and encrypts it with the server’s public key (in the RSA key exchange) or performs the Diffie-Hellman exchange. The client sends the ClientKeyExchange, the ChangeCipherSpec, and the Finished. The server responds with the ChangeCipherSpec and the Finished. The application data can then be sent .

The RSA key exchange is the simpler one, but it does not provide the forward secrecy. The pre-master secret is encrypted with the server’s public key, and the server decrypts it with its private key. An attacker who obtains the server’s private key can decrypt the past sessions. The Diffie-Hellman key exchange is the alternative, and the ephemeral Diffie-Hellman provides the forward secrecy because the private values are not stored .


b. The TLS 1.3 handshake and the one round trip

The TLS 1.3 handshake starts with the ClientHello, which contains the client’s random value, the supported cipher suites, the supported groups, and the key share. The key share is the client’s ephemeral Diffie-Hellman public value, and it is sent in the first message. The server responds with the ServerHello, which contains the server’s key share, and the handshake keys are derived immediately. The server then sends the EncryptedExtensions, the Certificate, the CertificateVerify, and the Finished, and the client sends the Finished .

Client                                Server
ClientHello          -------->
  + key_share
                     <--------      ServerHello
                                    + key_share
                                    {EncryptedExtensions}
                                    {Certificate}
                                    {CertificateVerify}
                                    {Finished}
Finished             -------->
Application Data     <------->      Application Data

The key difference is the early key share. The client sends its ephemeral public value in the ClientHello, and the server can compute the shared secret as soon as it receives the ClientHello. The server sends its own key share in the ServerHello, and both sides can derive the handshake keys. The certificate and the other handshake messages are encrypted with the handshake keys, which is the “all handshake messages after ServerHello are encrypted” property . The one round trip is the result of the early key share.

The TLS 1.3 removes the static RSA and the static Diffie-Hellman key exchanges. The only key exchange is the ephemeral Diffie-Hellman (or the ephemeral elliptic curve Diffie-Hellman), which provides the forward secrecy for every connection . The removal is the reason the TLS 1.3 is more secure than the TLS 1.2 with the RSA key exchange.


Complete Example Session

// ============================================
// PART 1: THE TLS 1.2 HANDSHAKE
// ============================================

Client                                Server
ClientHello          -------->
                     <--------      ServerHello
                                    Certificate
                                    ServerKeyExchange
                                    ServerHelloDone
ClientKeyExchange    -------->
ChangeCipherSpec     -------->
Finished             -------->
                     <--------      ChangeCipherSpec
                                    Finished
Application Data     <------->      Application Data

// Two round trips before the application data.
// The RSA key exchange does not provide the forward secrecy.
// The ephemeral Diffie-Hellman provides the forward secrecy.

// ============================================
// PART 2: THE TLS 1.3 HANDSHAKE
// ============================================

Client                                Server
ClientHello          -------->
  + key_share
                     <--------      ServerHello
                                    + key_share
                                    {EncryptedExtensions}
                                    {Certificate}
                                    {CertificateVerify}
                                    {Finished}
Finished             -------->
Application Data     <------->      Application Data

// One round trip before the application data.
// The key share is sent in the ClientHello.
// The handshake messages after ServerHello are encrypted.
// The ephemeral key exchange provides the forward secrecy.

// ============================================
// PART 3: THE CERTIFICATE VERIFICATION
// ============================================

// The server sends the certificate chain.
// The client verifies the signature chain up to the trusted root.
// The client checks the hostname and the validity period.
// The client checks the revocation (OCSP or CRL).

// ============================================
// PART 4: THE KEY EXCHANGE
// ============================================

// The client generates the ephemeral private value a.
// The client sends g^a to the server.
// The server generates the ephemeral private value b.
// The server sends g^b to the client.
// Both compute the shared secret g^(ab).
// The attacker cannot compute g^(ab) from g^a and g^b.

// ============================================
// PART 5: THE SESSION KEY
// ============================================

// The shared secret is the input to the key derivation.
// The HKDF derives the session keys from the shared secret.
// The session key encrypts the application data.
// The symmetric encryption is fast, and the asymmetric is slow.

// ============================================
// PART 6: THE FORWARD SECRECY
// ============================================

// The ephemeral private values are discarded after the handshake.
// The server's long-term private key does not reveal the session key.
// A compromised server key does not decrypt the past sessions.
// The TLS 1.3 requires the forward secrecy for all connections.

// ============================================
// PART 7: THE 0-RTT MODE
// ============================================

// The resumed session sends the application data in the first flight.
// The 0-RTT reduces the latency to zero round trips.
// The 0-RTT has the replay risk.
// The 0-RTT should only be used for the idempotent requests.

// ============================================
// PART 8: THE CIPHER SUITE
// ============================================

// The TLS 1.2 cipher suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256.
// The key exchange: ECDHE.
// The authentication: RSA.
// The bulk cipher: AES-128-GCM.
// The MAC: SHA256 (the AEAD combines the encryption and the MAC).

// The TLS 1.3 cipher suite: TLS_AES_128_GCM_SHA256.
// The key exchange is negotiated separately (the key_share extension).
// The cipher suite only names the AEAD and the hash.

// ============================================
// PART 9: THE COMMON HANDSHAKE FAILURES
// ============================================

// The certificate is expired.
// The certificate is self-signed and not trusted.
// The hostname does not match the certificate.
// The cipher suite is not supported by the server.
// The protocol version is not supported by the server.

// ============================================
// PART 10: THE CHECKLIST
// ============================================

// [ ] The server supports TLS 1.3.
// [ ] The server supports TLS 1.2 for the older clients.
// [ ] The server uses the ephemeral key exchange (ECDHE).
// [ ] The server uses the AEAD cipher suites.
// [ ] The certificate chain is complete.
// [ ] The OCSP stapling is enabled.
// [ ] The HSTS header is set.
// [ ] The 0-RTT is enabled only for the idempotent requests.

The ten parts covered the TLS 1.2 handshake, the TLS 1.3 handshake, the certificate verification, the key exchange, the session key, the forward secrecy, the 0-RTT mode, the cipher suite, the common failures, and the checklist. The through-line is that the handshake is the negotiation and the key establishment, and the record layer is the encrypted data transfer.


Quick Reference

The TLS 1.2 vs 1.3

AspectTLS 1.2TLS 1.3
Round trips21
Key exchangeRSA, DH, ECDHEECDHE only (forward secrecy)
Handshake encryptionAfter ChangeCipherSpecAfter ServerHello
Cipher suiteKey exchange + auth + bulk + MACAEAD + hash
0-RTTNot supportedSupported (with replay risk)
CompressionSupportedRemoved

The cipher suite components

ComponentTLS 1.2TLS 1.3
Key exchangeECDHE, DHE, RSAECDHE (via key_share)
AuthenticationRSA, ECDSARSA, ECDSA
Bulk encryptionAES-GCM, ChaCha20-Poly1305AES-GCM, ChaCha20-Poly1305
MACSHA256, SHA384Included in AEAD

The common handshake failures

FailureCauseFix
Certificate expiredThe validity period has passedRenew the certificate
Certificate untrustedThe self-signed or the missing intermediateComplete the chain
Hostname mismatchThe certificate is for a different nameUse the correct certificate
Cipher mismatchThe server does not support the client’s suiteEnable the modern suites
Version mismatchThe server does not support the client’s versionEnable TLS 1.2 and 1.3

Best Practices

โœ… Do This:

// Enable TLS 1.3 on the server.
// Enable TLS 1.2 for the older clients.
// Use the ECDHE key exchange for the forward secrecy.
// Use the AEAD cipher suites (AES-GCM, ChaCha20-Poly1305).
// Complete the certificate chain with the intermediates.
// Enable the OCSP stapling.

โŒ Don’t Do This:

// Use the RSA key exchange (no forward secrecy).
// Use the CBC-mode cipher suites (the padding oracle risk).
// Use the self-signed certificate in production.
// Enable the 0-RTT for the non-idempotent requests.

Common Pitfalls

PitfallWhy It HappensFix
The forward secrecy is missingThe RSA key exchange is usedUse the ECDHE
The handshake is slowThe two round trips of TLS 1.2Enable TLS 1.3
The certificate is rejectedThe chain is incompleteAdd the intermediate certificates
The 0-RTT replayThe non-idempotent request is sent in 0-RTTDisable the 0-RTT for the non-idempotent requests
The old client cannot connectThe server only supports TLS 1.3Enable TLS 1.2 for the older clients

Summary

ItemValue
TLS 1.2 handshakeTwo round trips, RSA or ECDHE key exchange
TLS 1.3 handshakeOne round trip, ECDHE only, forward secrecy mandatory
CertificateThe server authentication, the chain verification
Key exchangeThe Diffie-Hellman, the ephemeral for forward secrecy
Session keyThe symmetric encryption of the application data
0-RTTThe zero round trip, the replay risk
Cipher suiteThe AEAD + hash in TLS 1.3

Key takeaways:

  • The handshake authenticates the server and establishes the session key. The certificate verification is the authentication, and the Diffie-Hellman key exchange is the key establishment. The session key encrypts the application data with the symmetric cipher, which is fast enough for the bulk data .
  • TLS 1.3 reduces the handshake to one round trip. The client sends its key share in the ClientHello, and the server can derive the handshake keys immediately. The handshake messages after the ServerHello are encrypted, and the forward secrecy is mandatory .
  • The forward secrecy is the protection against the server key compromise. The ephemeral key exchange discards the private values after the session, so a compromised server key does not decrypt the past sessions. The TLS 1.2 with the RSA key exchange does not provide the forward secrecy .
  • The 0-RTT mode trades the replay protection for the latency. The resumed session sends the application data in the first flight, but the attacker can replay the 0-RTT data. The 0-RTT should only be used for the idempotent requests .
  • The certificate chain must be complete. The server sends the leaf certificate and the intermediate certificates, and the client verifies the chain up to the trusted root. The missing intermediate is the most common cause of the handshake failure .

Remember: The TLS handshake is the negotiation and the key establishment, and the record layer is the encrypted data transfer. The certificate authenticates the server, the Diffie-Hellman key exchange establishes the session key, and the symmetric cipher encrypts the data. The TLS 1.3 is one round trip and the forward secrecy is mandatory. The TLS 1.2 is two round trips and the forward secrecy is optional. The 0-RTT is the latency optimization with the replay risk. The handshake is the security foundation, and the record layer is the protection.

Was this helpful?


Stop using slow, ad-bloated tool sites! ๐Ÿคฎ

๐Ÿ”Ž Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
โœ… Finance (Mortgage, Interest, Inflation)
โœ… Tech (Base64, JSON, Dev Suite, IP)
โœ… Health (BMI, BMR, TDEE)
โœ… Productivity (Timer, Workspace, QR)

โšก๏ธ Fast & Private
๐Ÿ”’ No data leaves your device
๐Ÿ’Ž 100% Free

๐Ÿ”— Use it now: https://tools.kandz.me
๐Ÿ”– Bookmark itโ€”youโ€™ll need it later!