In the early era of the network, people send files in single-digit KB size. In 2021, we enjoy hi-res MB-size images and watch 4K (soon 8K) video in several GB.
Even with a good internet connection, it still takes a while to download a 5GB file. If owning an Xbox or PlayStation, you know how it feels.
We have three ways to shorten the time sending extensive data by HTTP:
- compress data
- send chunked data
- request data in a selected range
They are not exclusive. You can use all means together depending on use cases.
Compress data
To compress data, we need compression algorithms.
When sending a request, a browser includes a header Accept-Encoding with a list of supported compression algorithms, including gzip (GZIP), compress, deflate, and br (Brotli).
Next, the server picks the one it supports from the list and sets the algorithm name in the Content-Encoding header.
When the browser receives the response, it knows how to digest the data in the body.
Among these algorithms, the most popular one is GZIP. It is a great option to compress text-based data such as HTML, CSS, and JavaScript.
Brotli is another one worth mentioning. It performs even better than GZIP in compressing HTML.
These efficient algorithms come with limits.
They work perfectly for text but are insufficient for compressing images or videos. After all, media has already been optimized.
Try to compress a video file on your computer. You will not notice much difference before and after the compression.
Moreover, it is nearly impossible to compress a 5GB video to some KB without losing quality.
Compression is good, but we need a better solution — sending the file in chunks and assemble the partial data at the client-side.
Send chunked data
In version 1.1, HTTP introduced chunked data to help with the large-data cases.
When sending a response, the server adds a header Transfer-Encoding: chunked, letting the browser know that data is transmitted in chunks.
Each piece of chunked data has the following components:
- a Length block marks, well, the length of the current part of chunked data
- the chunked data block
- a CRLF separator at the end of each block
Wondering what a CRLF is?
A CR immediately followed by an LF (CRLF, \r\n, or 0x0D0A) moves the cursor down to the next line and then to the beginning of the line.
In the further reading section at the end of this post, you can find more details. Here, you can simply treat it as a separator.
The server continues streaming chunked data to the browser.
When reaching the end of the data stream, it attaches an end mark consisting of the following parts:
- a Length block with the number
0and a CRLF at the end - an additional CRLF
On the browser side, it waits for all data chunks until it reaches the end mark. It then removes the chunked encoding, including the CRLF and the length information.
Next, it combines the chunked data into a whole. Therefore, you can only see the assembled data on Chrome DevTools instead of chunked ones.
Finally, you receive the entire data in one piece.
Chunked data is useful. However, for a 5GB video, it still takes a while for the complete data to arrive.
Can we get a selected chunk of the date and request the others when we need?
HTTP says yes.
Request data in a selected range
Opening a video on YouTube, you see a grey progress bar is moving forward.
What you just saw is YouTube requesting data in a selected range.
This feature enables you to jump anywhere in the timeline. When clicking on the spot on the progress bar, the browser requests a specific range of the video data.
It is optional to implement the range requests on a server. If it does, you can see the Accept-Ranges: bytes in the response header.
Here is an example of a YouTube request. In any “playback” request, you can find the header.
A range request header looks like Range: bytes=0-80, and it is 0-indexed.
This header is a smart design with remarkable flexibility.
Let’s say a data has 100 bytes in total.
Range: bytes=20requests a range starting from 20 to the end, which equalsRange: bytes=20-99.Range: bytes=-20requests the last 20 bytes of data, which equalsRange: bytes=80-99.
If the requested range is valid, the server sends the response with a Content-Range header, verifying the data range and the total length, such as Content-Range: bytes 70-80/100.
Range request is widely used in video streaming and file download services.
Have you ever continued a file download after an internet interruption? That’s a range request.
Furthermore, the range request supports multiple ranges.
For example, you can request two ranges from the file with Range: bytes=20-45, 70-80.
The multi-range body looks similar to chunked data. Each piece of data has the following parts:
- A boundary block indicating, well, the boundary of the data, starting with
--and ending with a CRLF - Two headers,
Content-TypeandContent-Range, showing the property of the corresponding data piece, ending with a CRLF - An additional CRLF telling the client that the real data is coming
- Finally, the data block ending with a CRLF
A boundary is merely a random string that looks like 3d6b6a416f9b5, marking the border of different data pieces.
Eventually, the body ends with a boundary block, starting with a -- and ending with a -- and a CRLF. This piece tells the browser that the multi-parts have ended.
Let’s put it all together. The response body structure looks like the following.
Takeaways
HTTP helps us deliver extensive data with compression, chunked data, and ranged data.
The idea here is to deliver the data we need at the moment and ship others when required. When having trouble designing a similar system, you can try the same idea.
By combining three ways, we can send compressed chunked data in a range.
Join Medium
Purchasing Medium Membership through the above link means that I can get income through the referral link. This does not mean that you have to buy from the link, nor does I deny or oppose other channels. It is your right to know.
Further reading
- To learn more about the CR and LF, MDN is your friend: https://developer.mozilla.org/en-US/docs/Glossary/CRLF.
- Take a look at the range request. It is quite flexible: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Range
