HTTP

In the realm of IT development, a deep understanding of the HTTP (Hypertext Transfer Protocol) protocol is of utmost importance.

HTTP forms the foundation of communication on the World Wide Web, facilitating the transfer of information between web servers and clients. It serves as the protocol through which web browsers request resources, such as web pages or multimedia content, and servers respond with the requested data. Proficiency in HTTP enables developers to effectively design and build web applications, ensuring smooth and efficient data exchange.

Understanding the intricacies of HTTP methods, status codes, headers, and cookies allows developers to troubleshoot issues, optimize performance, and implement secure and reliable web solutions. Furthermore, with the rise of API-driven development, knowledge of HTTP is indispensable for working with RESTful APIs, as they often utilize HTTP methods and status codes for interaction.

Key Points

• The Web: A global client/server network using protocols like HTTP for communication.
• HTTP: Application protocol for data transfer on the internet.
• HTTP Communication:
  • Client-server, request-response model.
  • Stateless (each request independent) but cookies enable stateful sessions.
  • Versions: HTTP/1.1, HTTP/2, HTTP/3.
• Browsers: Translate URLs into requests and render responses.
• URLs: Unique resource identifiers with syntax: protocol://hostname:port/path-file-name.
• URIs: More general than URLs, can include parameters and fragments.
• HTTP Messages:
  • Request: Request line (method, URI, version), headers, optional body.
  • Response: Status line (version, code, phrase), headers, optional body.
• HTTP Methods: GET, POST, PUT, DELETE, HEAD, OPTIONS.
• GET vs. POST: GET for retrieving data, POST for sending data.
• Status Codes: Indicate request outcome (e.g., 200 OK, 404 Not Found).
• Request Headers: Provide client information (e.g., Host, Accept, User-Agent).
• Content Negotiation: Selecting the best data format for responses using headers (Accept) or URL patterns.

The Web

Internet (or The Web) is a global distributed client/server system.

In order for proper communication to take place between the client and the server, web applications must agree on a specific protocol.

There are various protocols such as HTTP, FTP, SMTP, POP etc. But perhaps, HTTP is the most popular application protocol used on the Internet.

HTTP Protocol

HTTP (Hypertext Transfer Protocol) is the foundation for data communication for the Global Internet since 1990.

HTTP is based on TCP/IP to deliver data (HTML files, image files, query results etc.) on the Internet. The default port is 80 while other ports are used sometimes.

Http specification specifies how clients’ request data will be constructed and sent to the server, and how the servers respond to these requests.

Request and Response

HTTP is an asymmetric request-response client-server. This means an HTTP client (browser) initializes a request message to an HTTP server and the server, in turn, returns a response message, not the other way around.

Stateless but NOT Session-less

The HTTP protocol is a stateless protocol. That means the server isn’t required to store session information and each request is independent of another. The server and client are aware of each other only during a current request. After that, both sides forget about each other.

While the core of HTTP itself is stateless, HTTP cookies allow stateful sessions. Using header extensibility, HTTP Cookies are added to the workflow, allowing session creation on each HTTP request to share the same context, or the same state.

Version

Two HTTP versions are currently being used widely: HTTP/1.0 and HTTP/1.1. HTTP/2.0 is also underway.

HTTP/1.0 was introduced in 1996 and HTTP/1.1 in 1997. HTTP/1.1 can be considered an upgrade for HTTP/1.0:

• HTTP/1.0 defines 16 status codes
• HTTP/1.1 defines 24 status codes
• HTTP/1.1 has a warning header capable of producing many secondary status alerts
• HTTP/1.0 authentication is somewhat unsafe as it is not encrypted
• HTTP/1.1 is safer as it uses a checksum of username, password and one-time value

Let’s not step into the details for now.

HTTP/2.0 was introduced in 2015. It switches from ASCII to Binary. The general purpose is to make our applications faster, simpler and more robust.

HTTP/3.0 was introduced in 2018. It switches underlying transport from TCP to UDP. It’s still young but Google Chrome has added support for it.

Browser

When you open a browser to visit a URL, you type in something like http://www.myhost.com/index.html, the browser turns the URL into a request message and sends it to the HTTP server. The server interprets the request message, and returns you an appropriate response message, which is either the resource you requested or an error message.

Uniform Resource Locator (URL)

A URL (Uniform Resource Locator) is used to uniquely identify a resource over the web.

URL has this syntax: protocol://hostname:port/path-file-name

• Protocol: The application level protocol used by the client and server, e.g. HTTP, FTP, telnet etc.
• Hostname: The DNS domain name (e.g. www.google.com) or IP address (e.g. 172.217.10.228)
• Port: The TCP port number that the server is listening for incoming request from the clients
• Path-and-file-name: The name and location of the requested resource

Some examples would be:

• ftp://www.ftp.org/docs/doc.txt
• mailto:john.wilson@gmail.com
• telnet://www.somehost.com/

Encoded URL

URLs cannot contain special characters, e.g. space or ‘~’. Special characters are encoded, in the form of %xx, where xx is the ASCII hex code for the character.

For example, ‘~’ is encoded as %7e, a space is encoded as %20.

Uniform Resource Identifier (URI)

URI is more general than URL, which can even locate a fragment within a resource.

URI has this syntax: protocol://hostname:port/path?request-parameters#nameAnchor

• The request parameters are in the form of name=value pairs. Parameters start with a ‘?’ question mark and are separated by ‘&’ ampersand.
• #namerAnchor identifies a fragment within an HTML document

HTTP Request/Response Message

HTTP client and server communicate by sending text messages.

Whenever you enter a URL in the browser address, it translates that URL into a request message based on the protocol specified and sends that request message to the target server.

Here is a sample request message for accessing http://www.myhost.com/index.html in the browser (row numbers are added for reference purpose):

A response will be sent back once the server has processed the request:

When the browser receives the response message, it interprets the message and displays the contents of the message according to the media type of the response (Content-Type: text/html).

Common media type includes:

• “text/plain”
• “text/html”
• “image/gif”
• “image/jpeg”
• “audio/mpeg”
• “video/mpeg”
• “application/msword”
• “application/pdf”

Request/Response Messages

Let’s take a closer look at the message request/response messages.

An HTTP message (request and response) consists of a message header and an optional message body, which are separated by a blank line.

The Request Message

The format of a request message looks like this:

Request Line

The first line in the header is “Request Line”, which has this syntax:

request-method-name<space>request-URI<space>HTTP-version

• Request-method-name: HTTP defines a list of request methods, e.g. GET, POST, HEAD, OPTIONS etc. Clients can use one of these methods to send a request to the server.
• Request-URI: the resource requested
• HTTP-version: two versions are currently in use: HTTP/1.0 or HTTP/1.1

Request Headers

Request headers are in the format of name:value pairs, where value can be multiple ones separated by commas.

Request Example

The Response Message

The format of a response message looks like this:

Status Line

The first line in the header is “Status Line”, which has this syntax:

HTTP-version<space>status-code<space>reason-phrase

• HTTP-version: HTTP/1.0 or HTTP/1
• Status-code: a 3-digit number reflecting the outcome status of the request
• Reason-phrase: a short explanation for the status code returned

Example status code and reasons: “200 OK”, “404 Not Found”, “403 Forbidden”, “500 Internal Server Error”.

Response Headers

Like request headers, response headers are in the format of name:value pairs as well, where values can be multiple ones separated by commas.

Response Example

HTTP Methods

HTTP has defined a list of request methods. Clients can use one of the methods to send a request to HTTP servers.

Here is a list of them:

• GET: get a web resource from the server
• POST: post data to the server
• PUT: update data at the server
• DELETE: delete data at the server
• HEAD: get the header as a GET method would obtain
• OPTIONS: let the server return the list of request methods supported
• Other methods

The GET method is most frequently used and then POST, PUT, DELETE and OPTIONS.

When you type in a URL in your browser and hit enter to access a web page, the HTTP method is GET. In other words, a browser address field can only support the GET method. In order to use other methods, you can use a HTTP REST client. There are online test clients and desktop application clients such as POSTMAN.

GET VS POST

Generally speaking, when you are not sending data to the server but just accessing the server resource, you will use the GET method. Instead, if you are sending data to the server, you probably want to use POST instead of GET.

GET method can also send data through URL query parameter, which is different from POST method, as the data is contained in the POST body instead of URL parameter. This has some advantages:

• The amount of data is unlimited
• The data is not shown on the address box of the browser

A POST Method Example

We use https://apitester.com/, an online API testing tool, to show the example here. You can find plenty of similar tools online.

Besides, we use https://reqres.in/, an online testing server for API testing with fake data.

Test Scenario

The test scenario is fairly straightforward. The testing server has an endpoint /api/users, which accepts POST method for creating a new user. The posted data can be in JSON format and should have Name and Job attributes.

Here is the client setup.

• HTTP method has been set to POST
• Target URL is https://reqres.in/api/users
• Post Data is { "name": "Wilson", "job": "Engineer" }
• A header is added to specify content-type Content-Type : application/json

After clicking the “Test” button, the client sends the request to the target server and receives a response from the server.

Request Information

Response Information

Response Status Code

The status code will be included in the response from the server and it consists of a 3-digit number.

The first digit in the status code defines the category of the status in the response:

• 1xx (Informational): request received, server is continuing the process
• 2xx (Success): the request received successfully, understood, accepted and serviced
• 3xx (Redirection): redirection takes place to continue further action
• 4xx (Client Error): request has some error, e.g. bad syntax
• 5xx (Server Error): server failed to fulfill the valid request

It is important to know the commonly encountered status code.

• 100 Continue: server has received the request and is in the process of giving the response
• 200 OK: the request looks good and has been fulfilled
• 301 Moved Permanently: the requested resource has moved to a new location; the URL of the new location is given in the response header called “Location”
• 302 Found & Redirect (or Move Temporarily): similar to 301, but the new location is temporary
• 400 Bad Request: Server could not interpret or understand the request, probably syntax error in the request message
• 401 Authentication Required: authentication is required (e.g. username/password) to access the resource
• 403 Forbidden: authorization is required to access the resource (usually means client user doesn’t have enough privileges)
• 404 Not Found: the requested resource cannot be found in the server
• 405 Method Not Allowed: the request method used is not allowed from the server side
• 408 Request Timeout: the request to the server took longer than the server was prepared to wait (slow connection or wrong URL)
• 414 Request URI Too Long: the URL requested by client is longer than the server’s expectation
• 500 Internal Server Error: server-side processing error, often caused by server-side programming issue
• 501 Method Not Implemented: invalid request method (e.g “GET” mistakenly spelled as “Get”)
• 502 Bad Gateway: the requested server is acting as a gateway and it received an invalid response from its upstream
• 503 Service Unavailable: server cannot respond due to overloading or maintenance
• 504 Gateway Timeout: the requested server is acting as a proxy or gateway, and it received a timeout from upstream

Request Headers

Let’s see some of the commonly-used request headers.

• Host: <domain-name> -
  • as HTTP/1.1 support virtual hosts, that is to say multiple DNS names (www.myhost1.com, www.myhost2.com) can be attached to the same physical server
  • Host header is required in HTTP/1.1 to select a host
  • client can let server know what MIME types it can handle or prefer by using Accept header
  • if the server has multiple versions of the document requested, it can check this header to decide which version to deliver to the client and this process is called content-type negotiation
• Accept: mime-type1, mime-type2 … -

What is MIME?

MIME stands for Multipurpose Internet Mail Extensions, as this is originally created for emails but nowadays used a lot in other protocols.

It’s a way of identifying files on the internet and its format consist of two parts: “type/subtype”. For example, “text/html” indicates file type as html, “image/png” indicates file type as png image.

The “x-“ prefix in subtype simply means the subtype is non-standard.

• Accept-Language: language-1, language-2 … -
  • client can let server know what language it can handle or prefer by using Accept-Language header; this is called language negotiation
• Accept-Charset: Charset-1, Charset-2 … -
  • which charset is supported by client, e.g. UTF8
• Accept-Encoding: encoding-method1, encoding-method2 … -
  • which encoding method is supported by client; server can choose encode or compress the document before returning to the client, where server must set the response header “Content-Encoding” to inform the client the encoding type
  • e.g. x-gzip, x-compress
• Connection: Close|Keep-Alive –
  • tell server whether to close or keep alive the client-server connection after this request
  • HTTP/1.1 uses persistent connection (keep-alive) by default while HTTP/1.0 close the connection by default
• Referrer: referer-URL –
  • client uses this header to indicated the original URL page
  • for example, you click a link from page1 which opens page2, the page1 URL will be the referer-URL in the request to visit page2
  • all major browsers set this header, which can be used to track where the request is coming from
  • please note this header can be easily spoofed
  • the type of the browser used for the request
  • used by POST method to inform server the length of the request body
  • used by POST method to inform server the media type of the request body
  • used by client to specify how pages are cached
  • “no-cache” (not recognized by HTTP/1.0) means obtaining a fresh copy from server
  • “Pragma: no-cache” is used by HTTP/1.0
  • used by client to supply credentials to access protected resources
  • client uses this header to send cookies back to the server, which was set by the server earlier
  • let the server to send the page only if it has been modified after the specified date
• User-Agent: browser-type –
• Content-Length: number-of-bytes –
• Content-Type: mime-type –
• Cache-Control: no-cache|others –
• Authorization –
• Cookie: cookie-name1=cookie-value1, cookie-name2=cookie-value2… -
• If-Modified-Since: data –

Content Negotiation

Generally, resources can have multiple presentations, mostly because there may be multiple different clients expecting different representations.

Server-driven VS. Agent-driven Content Negotiation

If the selection of the best representation for a response is made by an algorithm at the server, it is called Server-driven negotiation. If that selection is made by agent (client) side, it’s called Agent-driven content negotiation.

In most cases, server-driven negotiation is NOT used, as the server side will have to make a lot of assumptions about client expectations. It would rather be easier to let the client tell the server what it prefers. Hence, agent-driven content negotiation is more popular.

Content Negotiation using HTTP headers

As previously mentioned, a request can have data attached to it. To specify its type, the request can have a “Content-Type” header to tell the server what format the attached data is. Some common examples are “text/plain”, “application/xml”, “text/html”, “application/json”, “image/gif”, and “image/jpeg”.

Content-Type: application/json

Besides the content format being sent, a client can tell the server what format is preferred in the response with the “Accept” header. The values will be similar to the ones mentioned above.

Accept: application/json

Multiple values can be specified in the “Accept” header.

Accept: application/json,application/xml;q=0.9,*/*;q=0.8

If there is no “Accept” header, the server can send the response with the default type.

Content Negotiation using URL patterns

Besides passing the “Accept” header, the client can use file extensions specified in the URL to indicate the preferred format, if the target server supports that. For example,

http://api.myhost.com/v1/users/1281.xml

http://api.myhost.com/v1/users/1281.json

The first request will return an XML response and the second one will return a JSON response.