In this article, I will cover the basics of what is a REST API, share a repository that contains a basic example of a stack of services consuming and exposing REST APIs and also a set of integration tests covering these services, and a set of good principles to have and to apply, as a developer of a team, to design such APIs.

This blog article is based on a course I gave at Ada Tech School, a French inclusive school of developers, as an Engineering Manager at Swan. I will also share the supports I used at this occasion, at the end of this article.

❯  Definitions

Globally, whatever the program you are writing, it has to communicate with its environment:

• From values passed in arguments of a program in command line interface, to specify some configuration keys, the path to a configuration file, …
• From an input prompted by a program (notably used in tech exercises during the interviews: for example, we give a number, and the program has to return the prime numbers strictly smaller than this number);
• in sending emails to notify if its execution, or when an error occurred during the execution;
• via SMS, push notifications, reading data from a file or from a database, … there are multiple examples. On a web context, the need of having different programs that are able to communicate with each other sharply increased, and then solutions appeared.

❯  API

API, for Application Programming Interface, is the term used to designate this kind of communication. To give a more formal definition, this is an interface that allows different services to communicate with other services, enabling them to share data and functionality.

This term is quite generic. I am restricting myself to APIs that are using HTTP (Hyper Text Transfer Protocol): an application layer protocol that is the standard for the communication on the web, that has been initially introduced by Tim Berners-Lee at CERN in 1989. Typically, to retrieve this page and be able to read it, you request this page to the server on which the content of this page is hosted in using HTTP.

Note that there exists some APIs that are actively used, that are not based on HTTP: the Web Sockets, that opens and keep open a communication channel between a server and one or several clients for instance, Message Queuing Telemetry Transport (MQTT), used for example by Home Assistant or Facebook Messenger, Advanced Message Queuing Protocol (AMQP), used for example by the Message Broker solutions like RabbitMQ, …

❯  Major representative of HTTP-based APIs

There are plenty of such APIs, but 4 of them are representing the majority of HTTP APIs that are currently actively used in the industry:

• SOAP: a solution built over XML,
• gRPC: a remote procedure call framework using Protobuf as the interface description language,
• GraphQL: a data query and update language allowing to request data from multiple sources in a unified graph, and
• REST: cf the next part 😁.

❯  REST

REST, for REpresentional State Transfer, is a term initially introduced by Roy Fielding on 2000, on his PhD thesis, as “a solution to create world-wide network-based applications”.

A REST API must respect the following constraints:

• Client-server model. Responsibilities between client and servers are separated, and they are communicating throughout an interface. This allows in particular to have a low coupling between different components, allowing them to evolve independently to the others.
• Stateless. There is no state of session conserved on the server side between several successive calls; if a context has to be kept, it is handled on client side, and transmitted as input of each request. A good example concerns the concept of authentication: typically, in the context of an authenticated API, the client will first make a call to an authentication endpoint that will, in case of success, return the token. Then, the client will use this token to access to the resources that are requiring the authentication.
• Caching capacity. Data returned by the endpoints will give the information of their capacity to be cached, in order to avoid the server to be called when this is not necessary.
• Uniform Interface. This allows the different actors to evolve independently, in simplifying and decoupling the architecture. This is achievable via the four following sub-constraints:
  • Resource identification in requests. In the requests, resources are identified in using URIs. Note that resources and representation of these resources (in JSON, XML, …) are two distinct concepts.
  • Resource manipulation through representations. A resource contains the necessary information to manipulate this resource (for example, updating, or deleting this resource).
  • Self-descriptive messages. Messages contain the necessary information to allow the user to understand the format of the resource, and how to manipulate it. This is typically achieved in using Content-type headers.
  • Hypermedia as the engine of application state (HATEOAS). The server must provide in the message a system of hyperlinks, to browser the other actions and resources available, in a philosophy similar to a home page of a website used by a human user.
• Layered System. There can have different layers between the client and the server, that are transparent for the user on its usage of the API. These layers generally handle the cache of the resources, or allows to balance the requests clients are doing on multiple instances of servers.
• Code on Demand (optional). Servers can transmit code to be executed on the client side. Although this functionality may reduce the functionalities to implement by the clients, and the overall complexity, it also negatively impacts how resources are organized on the server side. This induces that this constraint is optional.

Notes.

REST APIs that we generally find in the industry are taking some liberties with these rules: in particular, the notion of hypermedia / HATEOAS is often ignored. Cf Roy Fielding (source):

[…] if the engine of application state (and hence the API) is not being driven by hypertext, then it cannot be RESTful and cannot be a REST API […]

I sincerely apologize to Roy Fielding, by this is a liberty that I am also personally taking when I am building an API. And also a liberty I will regularly take in this post. Since I will widely neglect this notion, I recommend you to have a look on the following articles if you want to discover in details this concept, and some context on how REST APIs are generally not respecting this fundamental constraint:

• Taking REST APIs to the next level with hypermedia and HATEOAS on theserverside.com
• How Did REST Come To Mean The Opposite of REST? on htmx.org

Also note that we are sometimes talking about REST or RESTful APIs. The difference is fairly simple: REST is an architecture style. An API implementing this architecture is a RESTful API. By abuse of language, we are using the two terms to designate the same thing, both the architectural style and the API implementing it.

❯  Popularity of research

This is quite complex to gather accurate data on the real proportion of APIs using these different technologies. Thus, I have decided to have a look instead of the popularity of research of these different terms, over time, and in comparison to the others.

❯  SOAP

SOAP is the oldest of the four solutions, since it appeared originally under the term XML-RPC, in 1996. The search concerning the term SOAP is quite stable over the past 20 years.

❯  gRPC

In contrary to SOAP, gRPC is way more recent, since it has been released in 2016. Concerning the search, we note a continuous increase of the search on this topic, with a maximum of search reached quite recently (at the right extremity of the chart).

❯  GraphQL

A bit older than gRPC, GraphQL was initially created by Facebook in 2012. Its curve follows the trend of gRPC’s one, at the exception to a tendency to decrease since 2022.

❯  REST API

Way older than gRPC and GraphQL, but more recent than SOAP since the concept was introduced in 2000, the popularity of the search concerning REST continuously increased over time, except since 2022, where the tendency follows GraphQL’s curve tendency.

❯  Elements of a REST API

Typically, a REST API is composed of a collection of endpoints or route, composed of an HTTP verb and a URL, with eventually some data, and also some headers.

❯  URL

URL, for Unified Resource Locator, specifies the location of a resource on an internet context.

Typically, a URL looks like this:

1

https://www.bigseeder.com/trees/tree1/apples?color=red&tasty=true

This URL is composed of several elements.
First, "https" indicates the protocol used (typically, in the context of a REST API, either http or https).
The second part, "www.bigseeder.com", is the domain, or the hostname exposing this API.
"/trees/tree1/apples" indicates the path to access to the resource. We are applying a hierarchical order in this path. In this example, "trees" indicates that we want to retrieve a data concerning the trees, then, with "tree1", that it concerns the tree identified with the id "tree1", and "apples" that we want to access to the resource apples of this tree.
Finally, we generally have at the end of a URL a "?" followed by some extra characters. For example, here: "?color=red&tasty=true". This is not part of the URL. I will detail what is it exactly in the data section.

Note that you may also encounter the concept of URI (Unified Resource Identifier). URLs are a subset of URI specific to the identification of resources in an internet context. Such, in the context of a REST API, we can use the two concepts interchangeably.

❯  HTTP verb

Now that we have designed the resource we want to access or edit via the URL, the HTTP verb indicates which action we want to do on this resource. There exists 9 verbs, with five verbs very common and 4 less common.

The most common are:

• GET To retrieve resources (yes, the name is generally quite straightforward 🙂). Depending on if the URL designate a precise resource or not, this typically returns either a resource or a list of resources.
• POST To create a resource. Data concerning the resource to create are generally transmitted via a payload, and the endpoint will typically return the identifier corresponding to the resource newly created.
• PUT To update a resource identified by its ID. This should replace the integrality of the resource by the new one.
• PATCH To update partially a resource identified by its ID. In other words, in contrary to PUT, it will only replace some specific fields of the resource, and not the integrity of this resource.
• DELETE To delete a resource, given its identifier. Again, no secret in the meaning of this HTTP verb!

The least common are:

• HEAD This is quite close to the GET HTTP verb, at the difference the response does not include the data. This is generally used to verify that an API is reachable, and that a resource exists, without having to retrieve it.
• OPTIONS This is used to return the functionality provided by the API, given the domain or any route in the domain.
• TRACE Generally used out of production environments, this is used to debug API behaviors, in performing a loop-back test of messages, along the path to the resource.
• CONNECT Is generally used to create a tunnel (two-way communications) between the client and the requested resource.

I recommend you to have a look at the MDN web docs to know more about these verbs, and how they are supposed to be used by APIs. Also note that APIs are not always respecting these principles, and this is quite common that PUT endpoints actually behaves like a PATCH endpoint. Another quite common is the use of POST endpoints to fetch data: this is typically due to the fact that you can transmit payload with POST endpoints, but not with GET endpoints. I will give more details on which kind of data is available for each HTTP verb in the next section (only for the endpoints GET, PUT, POST, PATCH and DELETE).

❯  Headers

Headers, in the context of a REST API, are metadata, or extra information, that are included on HTTP requests and responses. They don’t contain resource information, but may contain information on the format of the resource send with the request, the format expected for the response, information concerning the authentication or, as we saw in the section Definitions, elements concerning the caching capacity of the request, the context of the ongoing session, or information added by some layers (for example by CDNs).

❯  Data

Depending on the HTTP verb, the manner of transmitting data is different. I will give the most common type of way to transmit such data, with which HTTP verb they are available, and on which context they are used. Finally, I will give an example of use via a curl request.

See here for more information about curl.

❯  Query Parameters

Available for GET endpoints, this is key value parameters added after the URL identifying the resource, after a question mark "?", and each Query Param is delimited by a "&".

Query params are typically used to filter, sort or paginate results.

Example.

Using TheCatAPI, to request 10 pictures of cats of breed “Chartreux”:

1

curl "https://api.thecatapi.com/v1/images/search\?breed_ids=char\&limit=10"

Note the backslash \ in the URL: this is to tell to Bash to not interpret the characters ? and &.

There are also two alternatives to specify such query parameters via curl.

• With the -G flag and params specified with --data-urlencode:

1
2

curl -G "https://api.thecatapi.com/v1/images/search" --data-urlencode "breed_ids=char"
--data-urlencode "limit=10"

The -G flag tells curl to append the data as query parameters, and the --data-urlencode flag URL-encodes the query parameters.

• Using the -G flag with --data:

1
2
3

curl -G "https://api.thecatapi.com/v1/images/search"
--data "breed_ids=char"
--data "param2=value2"

Similar to the previous example, the -G flag appends the data as query parameters. The --data flag sends the query parameters as is, without URL-encoding.

We have in this request 2 query params: breed_ids=char, to filter the pictures returned by the API to the images of “Chartreux” cats, and limit=10, to request 10 results in a row.

❯  Path parameters

Available for GET, PUT, PATCH, and DELETE endpoints, a path parameter, is a parameter part of the URL that identifies a resource, typically the ID of the resource we want to retrieve.

Example.

Using TheDogAPI, to request the image with ID 3NC7vIjMR:

1

https://api.thedogapi.com/v1/images/3NC7vIjMR

❯  Request body

Available for POST, PUT, PATCH and DELETEendpoints, the request body typically contains the data of the resource concerned by the endpoint. It is also commonly used in case of search requests, for example when doing search request via Algolia or ElasticSearch. In this case, the payload typically contains information concerning the query, the filtering, or the pagination of the results we are requesting.

Also note the presence of DELETE in the list of verbs for which Request bodies are available: actually, even if this is fairly uncommon, some APIs are also accepting such payloads for DELETE endpoints, typically to provide additional information or context for the deletion.

Example.

Using the Todoist API, to create a task of "content"="a new task" in an existing project (with "project_id"="1234"):

1
2
3
4

curl "https://api.todoist.com/rest/v2/tasks"
    -X POST
    --data '{"content": "Buy Milk", "project_id": "2203306141"' 
    -H "Content-Type: application/json"

Note the presence of -H "Content-Type: application/json": this header specifies that the body in input of this request has the JSON format. You have to update this header according to the format of this data (XML, text, HTML, …).

❯  Form parameters

Available for POST and PUT endpoints, form data are majorly used to transmit user’s input fulfilled in forms on a website, to the server. There are two formats to transmit such data: either via the body, with a format relatively similar than for the query parameters (d'{key1=value1&key2=value2&...), or in separated fields (-F "key1=value1" -F "key2=value2" ...). There is also a header to add to specify the format of the data: -H "Content-Type: application/x-www-form-urlencoded". See the following example for the different formats to transmit these form params using curl.

Example.

Inspired from examples listed in ReqBin, an “online API testing tool for REST and SOAP APIs”.

• Example with the body:

1
2
3

curl -X POST https://reqbin.com/echo/post/form
   -H "Content-Type: application/x-www-form-urlencoded" 
   -d "key1=value1&key2=value2"

• Example with -F (or --form):

1
2
3
4

curl -X POST https://reqbin.com/echo/post/form
-H "Content-Type: application/x-www-form-urlencoded"
-F "key1=value1"
-F "key2=value2"

❯  Multipart form data

Available for POST and PUT endpoints, this allows to upload files in including it the request as multipart/form data. Note that there is also a header to add to the request -H "Content-Type: multipart/form-data".

Example.

Use of the Slack API to upload a picture named picture_to_upload.png

1
2
3
4

curl -X POST
--data-binary @/path/to/file.jpg
-H "Authorization: Bearer <your_token>"
-H "Content-Type: application/octet-stream" https://api.example.com/upload

Note. This API requires authentication. To do this, there is a header "Authorization", for with the value contains the authorization token mandatory to perform the upload.

❯  Status code

In addition to some potential data and headers, a response returned by a REST API systematically contains a 3 digits number named “HTTP code”. The first digit indicates the kind of response that has been returned:

• 1XX status code. The informational status codes. Typically, it informs that the request was successfully received, and that the process continues;
• 2XX status code. The successful status codes. The request has been successfully received, understood and accepted;
• 3XX status code. The redirection status codes. Further action needs to be taken in order to complete the request;
• 4XX status code. The client-side error status codes. The request contains bad syntax or cannot be fulfilled;
• 5XX status code. The server-side error status codes. The server failed to fulfill an apparently valid request.

Then, the 2 other digits identifies a specific HTTP code in the corresponding class. This is complex to list the more common errors we are encountering in the context of REST APIs, since it definitively depends on many factors. For instance, we may encounter 3XX responses in fetching websites that contain redirections and aliases on their web pages. In case where there are errors on the path from a client, and a server containing a resource, we can get error codes like 502: Bad Gateway, 503: Service Unavailable or 504: Gateway Timeout. Moreover, intermediaries layers can also return codes like 429: Too Many Requests, if a high number of requests from a given client to a same resource on a given time frame have been detected… in short, the list is almost infinite!

Furthermore, in the case of a (simple, straightforward) REST API, we can generally restrict the HTTP codes returned by such an API to the following ones:

• Successful codes.

  • 200: OK (the generic successful response code);
  • 201: Created (in case of a POST request: the request succeed, and the resource has been successfully created).

• Client-side error codes.

  • 400: Bad Request (the generic client-side error code);
  • 401: Unauthorized (the required authentication credentials are missing);
  • 403: Forbidden (either the provided authentication credentials are invalid, or they didn’t provide the right of executing the request);
  • 404: Not Found (the requested resource does not exist).

• Server-side error codes.

  • 500: Internal Server Error (the generic server-side error code).

Note that this list is not an absolute rule. I generally disagree with myself every time I wrote some notes about the classic HTTP status codes used in a REST API, that lead to a different minimal list of HTTP codes. In this version, for instance, I have added the code 401 ⛔️.

❯  A concrete example

To illustrate the design and use of a REST API, I have created a multi-services and multi-languages GitHub repository, accessible publicly at this link. This contains the following services:

❯  Service “Species”

The service Species is developed in NodeJS and using the framework ExpressJS, that contains endpoints to create, get, update globally or locally, delete and list all the existing species. Cf the README file of this service.

❯  Service “Animals”

This service is developed in PHP, with the framework Slim. It proposes the same kind of endpoint than Species, but concerning Animals. Typically, resources of this service contains IDs concerning other resources (species, pictures and farmers), that should be resolved by a client using this service. And spoiler alert, this is typically what the service “Farm” is doing. More details concerning the service Animals are available on the README file of this service.

❯  Service “Pictures”

The service Pictures is developed in Golang, and is using gorilla/mux to design and expose the REST API. Cf the README file of this service.

❯  Service “Farmers”

This service concerns the resource farmers, and is developed in Python in using FastAPI. Similarly to the service Animals, it also contains references to other resources (pictures). More details are available in the README file of this service.

❯  Service “Farm”

This service is designed to be the entry point of the API. It exposes an API that join resources from the 4 services described before in consuming their REST APIs. This service is developed in Java, using Spring Boot. Similarly to the other services, you can have a look on its README file.

❯  Postman / Newman for the documentation and integration tests

To complement these services, there is also a (public) PostMan project, available here that documents all the endpoints exposed by these different services, and a bunch of integration tests that covers their functional domains. These integration tests can be executed directly from PostMan, or in the GitHub repository, in using Newman. Details on how execute these tests are also available on the README of the different services.

❯  Good principles to respect

My goal here is to provide advices that any developer should apply in a situation where they are developer in a team responsible for implementing and integrating a REST API, typically a situation common in a Squad / Feature team of developers. This is majorly based on my personal experiences as back-end developer, lead dev and also software engineering manager in such teams.

First, as I have shared over the previous examples, some elements of an endpoint can be induced by constraints that are not visible by the other side (constraints on the producer, or on the consumer side). For example, REST APIs exposed by indexation and search engines like ElasticSearch or Algolia are exposing POST endpoints to perform search operations, mainly by the consequent size the search and filtering payload can have. Depending on some choices, like the programming language or some architecture constraints for example, some headers are mandatory or optional, etc… All these arguments lead to the golden rule when a team is collaborating on the design of such a REST API:

Never, ever, underestimate the specification part of a project.

Part of this specification, in addition to the nomenclature of the REST API (URL, HTTP verb, request and response headers, format of the data consumed and produced), we also have to think about the different results (response payload, and response code) the endpoints will return depending on the case, not only the positive outcomes.

For example, if we take the case of the endpoint "get farm" from the previous example that returns information relative to the farm itself, but also the list of farmers and animals: what happens if one of the farmer contains an ID of a picture that does not exist? A 200 with the field picture of the farmer set to null because we want to maximize the positive answers returned by the API? A 404 because one of the resources does not exist? A 500 because this is a problem on the server-side? Even if this last solution is effectively the right choice to adopt in theory, the other possibilities are also open to discussion, depending on the use case and context on which this API will be used.

Another important point: do not try to be original in the design of the API. Except if you have a good reason, use the HTTP verb that fits the action the endpoint will perform on the resource, keep the hierarchy in the manner of building the endpoint URL, from the most generic at the beginning of the URL to the more specific at its right. And do not try to use original response codes and response formats. For example, below is an example of response format I have already encountered in the past:

1
2
3
4
5
6
7

{
 "response_code": 444,
 "response_body": {
   "error_code": 500500,
   "error": "Application Error"
 }
}

This response was returned in case of error on the server-side. Although we expect in this case to have a response_code 500, it was a 444 and, except if there are some hidden magic behind the error_code 500500, it does not add other information than informing that this error is actually a server-side error. This case also induce something more problematic that simple disrespect of good practices in API design: actually, when an API returns a 5XX response code, it implies that an error occurred on the server side, that also encourage the consumer of this API to retry later, in order to receive a positive answer. On the other hand, in a company returns a 4XX response code, it indicates to the client that this is useless to perform retries, since the request is not valid, the response will remain the same for this request over time.

I also recommend, if possible, to avoid the development of different services on a same stack in different programming languages. Even if they are all producing and consuming the same data format, each language has specificities that may induce the difficulties when comes the time of integrating their REST APIs.

Your API should not require and provide optional or useless data. Sometimes, we are tempted to add such extra data, since we already know that some data may be useful in future iterations of the API, or to perform some tracking operations, … this is never a good idea. Your application should only contain the necessary information. And if you need to transmit some extra information, this should be achieved by extra calls done on tracking and analytics solutions.

My last advice, that is also an extension of this notion of analytics: an endpoint should do one and one single task, and not have side effects, like tracking, modifying data that are not the one directly targeted by the endpoint, … To keep it short, respect the Single Responsibility Principle. I recommend you the following article on this topic.

❯  Wrap-up

This blog post aims to introduce the notion of REST API, to make it understandable to people that are not familiar to this notion. It also describes fundamental elements of REST APIs, provides and presents a few examples of implementation in different languages plus a documentation and a set of integration tests, and finally contains a list of good principles and advices, based on my self experience and readings on this topic, that you should apply if you are developing such an API as part of a group of software engineers.

❯  Useful links

• The Mozilla Developer Network documentation, including:
  • documentation on HTTP;
  • documentation on HTTP methods;
  • documentation on headers;
  • documentation on status codes.
• For sake of completeness concerning HTTP: the RFC 9110 - HTTP Semantics
• Concerning the good practices, a blog post on the topic
• HTTP codes… illustrated with cats! http.cat
• Two articles concerning hypermedia/HATEOAS:
  • Taking REST APIs to the next level with hypermedia and HATEOAS; and
  • How Did REST Come To Mean The Opposite of REST?.
• Finally, the presentation I have done on this topic.