A detailed look at the HTTP protocol
HTTP Basics
Given how fundamental HTTP, the HyperText Transfer Protocol, is for modern communication, it’s surprisingly simple. But maybe that’s exactly part of the reason why it’s still relevant. It relies on concepts like resources and Uniform Resource Identifiers (URIs), simple message structure, and client-server communication flow. Over time, numerous extensions have been developed on top of these basic concepts that add new functionality and semantics.
A typical HTTP session
A simple exchange between a client and server using the HTTP protocol looks like this:
- The client establishes a connection via a transport layer protocol such as TCP.
- The clients sends its request, and waits for the answer.
- The server processes the request, sends back its response that includes a status code and a payload.
In the early versions of HTTP, the server would then close the connection. As of HTTP/1.1, the connection is no longer closed after completing the third phase. Instead, the client can go back to phase 2 as many times as it deems necessary. This is beneficial for transfer speed because it avoids having to repeat the expensive connection setup in phase 1 for each request.
Session tracking
One aspect of the HTTP protocol’s simplicity is particularly relevant in the context of HTTP proxy servers: HTTP is stateless. This means that every request the server receives is independent and does not relate to requests that came prior to it. As long as the client only requests content that has no shared context with other content, a bunch of images for example, this is not a problem.
It becomes an issue, however, when separate requests share a common context. Let’s take as an example a website that allows users to log in. As long as a user is not logged out again, every request they send is likely to require the server to send individual content for this user, and prevent unauthorized access to other resources. HTTP, stateless as it is, does not have any built-in feature that would tell the server “This request is related to that previous successful authentication, and is therefore legitimate.” In order to implement the concept of a user session, a client itself has to submit identifying information with each and every request. Thanks to the GDPR, it has become common knowledge even outside of web developer circles that this identifying information is called a cookie. As Varnish administrators, we have to go deeper, though. What exactly is a cookie?
HTTP as a language
HTTP is a plain text protocol, which means that the payload of HTTP requests and responses are transmitted as plain, human-readable text. It basically the language in which client and server interact. Let’s take a look at the vocabulary of this language!
The HTTP request
Here’s what a simple request for the Monospace Mentor website looks like:
GET / HTTP/2
Host: monospacementor.com
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:123.0) Gecko/20100101 Firefox/123.0
Cookie: «text»
Accept: text/html
Accept-Language: en-US,en
Connection: keep-alive
(I removed a few details that aren’t relevant at this point.)
With the first line of its request, the client tells the server on the other end that it wants to fetch (GET) the home page represented by the root path /, and that the request follows the standard of HTTP version 2 (HTTP/2).
The purpose of the initial line is to declare the so-called method of the request. The GET request is the most common HTTP request method. It asks the server to send the resource identified by the path after the word GET. There are other request methods that are either, like GET, in the form of verbs, for example POST and PUT, or in the form of nouns, for example OPTIONS.
In the following lines, the client tells the server detail information about itself and its expectations. These lines are called HTTP headers. They are simple key-value pairs in the form “key:value”. Whitespace following the colon is ignored but often used for better readability.
The first header is a Host header that defines the domain name of the website this request is targeted at. This host header was introduced into the HTTP standard to make it possible to operate websites with several domain names behind a single IP address.
The User-Agent header tells the server what software is used on the client side.
And then, there’s the Cookie header that my browser sent because I was logged into my website. This header contains as its value the information that allows the web application on the server to associate the request with an existing user session.
The final three headers tell the server what kind of response the client would prefer to receive. For example, the client expects a HTML page in the text/html format, and text content in US English or at least any kind of English.
The final Connection header tells the server to keep the connection open for subsequent requests.
The HTTP response
This is what the website returned as its response to this request:
HTTP/2 200
server: nginx/1.18.0
date: Fri, 16 Feb 2024 14:00:15 GMT
content-type: text/html; charset=UTF-8
content-length: 27683
expires: Wed, 11 Jan 1984 05:00:00 GMT
cache-control: no-cache, must-revalidate, max-age=0, no-store, private
set-cookie: «text»
x-cache-host: c61-varnish-elus
x-cache-backend: c61_web_ecio
x-cache-cacheable: NO (no-cache)
x-varnish: 5931925
age: 0
x-cache-result: MISS
«HTML code of the home page»
I’ve again left out headers that don’t help comprehension in this context.
The first line of the response tells the receiver that the response is going to follow the HTTP/2 standard, just like the request it is answering. It also contains the numerical HTTP status code for this request; it’s 200, indicating success.
The first few HTTP headers tell the client the type of web server it’s talking to, and the time the response was issued. Then, there’s information about the content type of the response’s payload, and its size. Here, it’s a HTML document in UTF-8 encoding with a size of about 28 kilobytes.
The expires and cache-control headers are important instructions how this content may be cached. Since they are relevant for both the client browser and potential cache proxies, we’ll soon discuss them in detail. The remaining headers are custom headers. It’s common to prefix non-standard header names with “X-“. However, this practice has been deprecated since 2012 because it causes at least as much issues as it solves.
Following the HTTP header block, the response also contained the actual page content, separated from the header section by a blank line. This second part of the response is called the response body.
HTTP headers
HTTP headers can roughly be divided into four categories:
- Request headers contain more information about the resource to be fetched, or about the client requesting the resource.
- Response headers hold additional information about the response, for example its location, or about the server providing it.
- Representation headers contain information about the body of the response, for example its MIME type, or encoding/compression applied.
- Payload headers contain representation-independent information about payload data, including content length and the encoding used for transport.
In the following sections, we’ll discuss all the important HTTP headers grouped by their purpose.
Redirect headers
Location
If a resource has moved to a different place, the server can respond to requests using the previous URL with a HTTP status in the 3xx range and a Location header that indicates the URL to redirect to.
Request:
GET / HTTP/1.1
Host: www.opsitive.com
Response:
HTTP/1.1 301 Moved Permanently
Location: https://monospacementor.com
Refresh
The Refresh header directs the browser to reload the page or redirect to another. This header has the same effect as the meta HTML element with the attribute http-equiv="refresh".
Authentication headers
The HTTP protocol has built-in methods for limiting and controlling access to certain resources. The most-used one is the “Basic Authentication” method. It uses specific HTTP status codes and HTTP headers.
When a client tries to access a resource that is protected via Basic Authentication, the server responds with a status 401:
Request:
GET /secret/document.html HTTP/1.1
Response:
HTTP/1.1 401 Unauthorized
Date: Fri, 23 Feb 2024 07:28:00 GMT
WWW-Authenticate: Basic realm="Secret documents"
From the WWW-Authenticate header in the response, the client can tell that Basic Authentication is required. The realm directive can be used to apply the same authentication details to multiple resources.
The client browser will usually go ahead and ask the user to enter a username and password. It’ll store the credentials internally associated with the realm name, and submit them in a new request for the same resource.
Request:
GET /secret/document.html HTTP/1.1
Authorization: Basic YWxhZGRpbjpvcGVuc2VzYW1l
Username and password are not transmitted in clear text but as the base-64 encoded representation of the username and password separated by a colon. Note that this has really no security advantage. Encoding is easily reversible, it’s not encryption. For that reason, many browsers will urgently warn their users not to send Basic Authentication credentials over unencrypted connections.
Caching-related headers
Age
The ‘Age’ header contains the time in seconds that the object has been in a proxy cache.
When an object has been freshly fetched from the origin server, its Age will be 0. Otherwise, its age is usually calculated as the difference between the proxy’s current time and the object’s Date header.
Request:
GET / HTTP/2
Host: monospacementor.com
Response:
HTTP/2 200
cache-control: max-age=900
expires: Sat, 24 Feb 2024 14:49:35 GMT
age: 227
According to its Age header, the web page that was served here has been stored in a proxy cache (spoiler: It’s Varnish!) for 227 seconds. The maximum age this document will remain in cache is 900 because that’s the limit defined by the Cache-control header. After those 900 seconds expire, the proxy will have to fetch the document again from its backend server to refresh its cache.
Cache-control
The Cache-control header is one of the most important headers when it comes to caching. It holds directives that control caching mechanisms in both HTTP requests and responses.
Expires
The date/time after which the response is considered stale.
Conditional headers
Last-modified ETag If-Match If-None-Match If-Modified-Since Vary
Cookie headers
Cookie
Set-Cookie
Proxy headers
Forwarded, X-Forwarded-For
Contains information from the client-facing side of proxy servers that is altered or lost when a proxy is involved in the path of the request.
Via
Added by proxies, both forward and reverse proxies, and can appear in the request headers and the response headers.