When you make a request to a server, you might expect a single response to be sent back in one piece. However, due to network and protocol limitations, responses especially large ones are broken into multiple chunks. In this blog post, we’ll explore how large HTTP responses are handled to help software developers optimize and understand network performance more in their backend applications.

We try to download a file larger than 1GB to monitor how the HTTP response works, step by step by simulating slow internet conditions, capturing HTTP response traces, and monitoring them in realtime.

1. Identifying Your Network Interface

ifconfig

or

ip address

Look for the wireless network interface (wlo1 in most cases, but it may be different on your system).

2. Simulating a Slow Internet Connection
To better understand how data is received in chunks, we can artificially slow down our internet speed. The following command applies a delay of 2000ms (2 seconds) and limits the bandwidth to 500 kbps on the wlo1 interface or change it to your own network Interface:

sudo tc qdisc add dev wlo1 root netem delay 2000ms rate 500kbit

3. Creating a File to Store HTTP Traces
Next, create an empty file that will store the HTTP request and response traces:

touch multi_response_output.txt

4. Monitoring the File in Real Time
To observe incoming data in real-time as it's being written to the file, use the tail -f command:

tail -f multi_response_output.txt

5. Downloading a Large File with Tracing
Now, open a new terminal window to send an HTTPS request to download a large file (Android Studio). This command logs all request and response details into multi_response_output.txt:

curl -v -L -O --trace multi_response_output.txt https://redirector.gvt1.com/edgedl/android/studio/ide-zips/2024.2.2.14/android-studio-2024.2.2.14-linux.tar.gz

Note: Since this script continuously writes trace data as the file downloads, it can generate several GBs of log data. Stop the script when you've gathered enough information by pressing Ctrl + C.

6. Restoring Normal Internet Speed
Once you're done monitoring, remove the artificial network restrictions using:

sudo tc qdisc del dev wlo1 root netem

This restores your internet connection to normal speed.

The following video and multi_response_output.txt file show the output:

multi_response_output.txt

How does a server reply with a large response?

We will dive into our actual example:

Connection Setup and TLS Handshake

IPv4 and IPv6 Connection Attempts:
```
== Info:   Trying 142.250.75.110:443...
== Info:   Trying [2a00:1450:4028:800::200e]:443...
```
The system tries to connect using both IPv4 and IPv6. If IPv6 is unavailable, it falls back to IPv4.

IPv6 Connection Failure, IPv4 Success:

== Info: Immediate connect fail for 2a00:1450:4028:800::200e: Network is unreachable
== Info: Connected to redirector.gvt1.com (142.250.75.110) port 443 (#0)

IPv6 fails, so the client connects via IPv4.

TLS Handshake Begins:
```
== Info: ALPN: offers h2,http/1.1
=> Send SSL data, 5 bytes (0x5)
== Info: TLSv1.3 (OUT), TLS handshake, Client hello (1):
=> Send SSL data, 512 bytes (0x200)
```
== Info: ALPN: offers h2,http/1.1: The client supports the ALPN (Application-Layer Protocol Negotiation) extension. It offers two protocol options: h2 (HTTP/2) and http/1.1 (HTTP/1.1).
== Info: TLSv1.3 (OUT), TLS handshake, Client hello (1): The client sends a Client Hello message offering supported protocols and encryption settings.
=> Send SSL data, 512 bytes (0x200): The system sends a 512-byte "Client Hello" message containing: TLS version (TLS 1.3), Cipher suites, and Supported extensions.
Server Responds with TLS Handshake:
```
== Info: TLSv1.3 (IN), TLS handshake, Server hello (2):
```
The client receives the "Server Hello" response from the server.

Server sends Encrypted Extensions

== Info: TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
<= Recv SSL data, 15 bytes (0xf)
0000: 08 00 00 0b 00 09 00 10 00 05 00 03 02 68 32    .............h2

The server sends Encrypted Extensions, indicating that h2 (HTTP/2) is the selected protocol.
TLS Handshake Completes, HTTP/2 Starts:
```
== Info: TLSv1.3 (OUT), TLS handshake, Finished (20):
== Info: SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
== Info: ALPN: server accepted h2
== Info: using HTTP/2
```
== Info: SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384: The connection is secured using TLS 1.3 and AES-256-GCM-SHA384 is used for encryption.
The secure connection is established, and HTTP/2 is negotiated.

Client Sends HTTP/2 GET Request:

== Info: h2h3 [:method: GET]
== Info: h2h3 [:path: /edgedl/android/studio/ide-zips/2024.2.2.14/android-studio-2024.2.2.14-linux.tar.gz]

The client requests a file using HTTP/2 headers.

Server Response 302:
```
<= Recv data, 548 bytes (0x224)
...
<TITLE>302 Moved</TITLE>
...
```
The 302 Moved response means that the requested file is not available at the original URL, and the server is instructing the client to fetch it from a new location.
SSL Handshake is happening again, the new connection involved another TLS handshake to establish a secure connection.
Server Response Actual Data:
```
<= Recv data, 16384 bytes (0x4000)
```
16384 bytes indicates the maximum number of bytes that a single TLS record can transmit which is exactly the size of the chunk. you can read more about.
Our response is much larger than this. The actual file gets split into multiple records of 16,384 bytes, each of which is transmitted to the client, repeating for the entire response. Due to several reasons, HTTP request responses have many limitations, one of which is the limitation on the number of bytes in TLS.
You can read more about the TLS standard here:
TLS1.2 in RFC 5246 https://datatracker.ietf.org/doc/html/rfc5246
TLS1.3 in RFC 8446 https://datatracker.ietf.org/doc/html/rfc8446.

Conclusion

This real-world example showcases how HTTP, TLS, and TCP practically work together to deliver large responses efficiently. By understanding these mechanisms, developers can debug slow downloads, optimize network performance and their backend application, and ensure reliable data transmission.

HTTP Large Responses Explained: A Practical Approach