API Design: Why It Is Better to Use References rather than Keys to Represent Relationships in an API

Hello, Habr!



We have the highly anticipated second edition of Web Development with Node and Express .



As part of our research on this topic, we found a conceptual article about designing web APIs from a model, where resource links are used instead of database keys and values. Original - from the Google Cloud blog, welcome under cat.





When we are modeling information, the key question is how to define the relationship and relationship between two entities. Describing patterns observed in the real world in terms of entities and their relationships is a fundamental idea that goes back at least to ancient Greece. It also plays a fundamental role in the modern IT industry.



For example, in relational database technology, relationships are described using a foreign key — a value stored in one row of a table that points to a different row, either in another table or in the same table.



It's equally important to express relationships in the API. For example, in a retail API, information entities can correspond to customers, orders, catalog entries, shopping carts, and so on. The bank account API describes which customer a given account belongs to, as well as which account each debt or credit is associated with.



The most common way that API developers use to express relationships is to provide database keys or proxies for them in the fields of those entities associated with those keys. However, in at least one class of API (web oriented) there is a preferable alternative to this approach: using web links.



According to the Internet Engineering Task Force ( IETF), a web link can be thought of as a tool for describing relationships between pages on the web. The most famous web links are those that appear in HTML pages and are enclosed in link or anchor elements or in HTTP headers. But links can also appear in API resources, and using them instead of foreign keys significantly reduces the amount of information that the API provider has to additionally document and the user needs to study.



A link is an element in one web resource that contains a reference to another web resource, as well as the name of the relationship between the two resources. A reference to another entity is written in a special format called "unique resource identifier" (URI), for which there is an IETF standard... In this standard, the word "resource" refers to any entity pointed to by a URI. The name of the relationship type in the link can be considered the same as the name of the database column that contains the foreign keys, and the URI in the link is the same as the foreign key value. The most useful of all URIs are those that provide information about the referenced resource using standard web protocol. These URIs are called "Uniform Resource Locator" (URL), and the most important URL for APIs is the HTTP URL.



While links are not widely used in APIs, some very famous web APIs are still based on HTTP URLs as a means of representing relationships. These are, for example, Google Drive APIand the GitHub API . Why is it so? In this article, I will show you how to use foreign keys API in practice, explain their disadvantages compared to using links, and show you how to convert a design that uses foreign keys to one where links are used.

Representing relationships with foreign keys

Consider the popular educational pet store application. This application stores records for tracking information about pets and their owners. Pets have attributes such as name, species and breed. The owners have names and addresses. Each pet is related to its owner, and the reverse relationship allows you to find all the pets of a particular owner.



In a typical key-based design, the pet store API provides two resources that look like this:







The relationship between Lassie and Joe is expressed as follows: in Lassie's view, Joe is designated as having a name and meaning corresponding to "owner." The opposite relationship is not expressed. Owner value is "98765", which is a foreign key. This is probably the real foreign key of the database - that is, we are dealing with the value of the primary key from some row of some database table. But, even if the API implementation slightly changes the key values, it still approaches the foreign key in its main characteristics.

The value "98765" is not well suited for direct customer use. In the most common cases, the client needs to construct a URL using this value, and the API documentation needs to describe the formula for doing this conversion. Typically, this is done by defining a URI pattern , like this:



/people/{person_id}







The inverse relationship - pets are owned by the owner - can also be exposed to the API by implementing and documenting one of the following URI patterns (the differences between them are stylistic only, not substantive):



/pets?owner={person_id}

/people/{person_id}/pets







APIs designed in this way typically require many URI patterns to be defined and documented. The most popular language for defining such patterns is not the one specified in the IETF specification, but OpenAPI (formerly known as Swagger). Prior to version 3.0, OpenAPI did not have a way to specify which field values ​​could be inserted into which templates, so some of the documentation had to be written in natural language, and some had to be guessed by the client. OpenAPI 3.0 introduces a new syntax called "links" to address this issue, but it takes some work to use this feature consistently.



So, as common as this style is, it requires the vendor to document and the client to learn and use a significant number of URI patterns that are not well documented in the current API specifications. Fortunately, there is a better option.

Representing relationships using links

What if the resources shown above were modified as follows:







The main difference is that in this case the values ​​are expressed using references, not using foreign key values. Here links are written in regular JSON, in the format of name / value pairs (there is a section below that discusses other approaches to writing links in JSON).



Note that the inverse relationship, that is, from pet to owner, is now also explicitly implemented because Joel



a field has been added to the view "pets"



.



The change "id"



to "self"



is essentially not necessary or important, but there is an agreement that using "self"



identifies a resource whose attributes and relationships are specified by other name / value pairs in the same JSON object. "self"



Is the name registered with the IANA for this purpose.



From an implementation point of view, replacing all database keys with links should be quite simple - the server converts all foreign database keys to URLs, so that nothing needs to be done on the client - but the API itself in this case is greatly simplified, and the connectivity between the client and the server goes down. Many URI patterns that were important in the first design are no longer required and can be removed from the API specification and documentation.



Now nothing prevents the server from changing the format of new URLs at any time without affecting clients (of course, the server must continue to comply with all URLs formulated earlier). The URL passed to the client by the server will need to include the primary key of the entity specified in the database plus some routing information. But, since the client simply repeats the URL while responding to the server, and the client never has to parse the URL, clients don't need to know how to format the URL. As a result, there is less connectivity between client and server. The server can even resort to obfuscating its own URLs using base64 or similar encodings if it wants to emphasize to clients that they should not "guess" what the URL format is, or infer the meaning of URLs from their format.



In the previous example, I used the references relative URI notation, for example /people/98765



. Perhaps the client would be a little more comfortable (although the author was not very helpful in formatting this post) if I expressed the URI in an absolute form, eg. pets.org/people/98765... Clients only need to know the standard URI rules defined in the IETF specifications in order to convert such URIs from one form to another, so choosing a specific form for URI is not as important as you might think. Compare this situation to the above conversion from foreign key to URL, which required specific knowledge of the pet store API. Relative URLs are somewhat more convenient for server implementers, as discussed below, but absolute URLs are perhaps more convenient for most clients. This is probably why the Google Drive and GitHub APIs use absolute URLs.



In short, using links rather than foreign keys to express relationships between APIs reduces the amount of information a client needs to know in order to interact with the API, and also reduces the amount of connectivity that can occur between clients and servers.

Underwater rocks

Here are some things to consider before moving on to using links.



Many API implementations have been provided with reverse proxies for security, load balancing, and more. Some proxies like to rewrite URLs. When the API uses foreign keys to represent relationships, the only URL that needs to be rewritten in the proxy is the main request URL. In HTTP, this URL is split between the address bar (the first line of the header) and the host header.



An API that uses links to express relationships will have other URLs in the headers and body of both the request and response, and these URLs will need to be rewritten as well. There are several different ways to deal with this:

1. URL . URL, .
2. , . , , , -, , .
3. . URL; , URL , -. URL, , , , , . - (, URL, «» «»), . , URL, , URL , , , URL .

Relative URLs without leading slashes are also more difficult for clients to use because they have to work with a standardized library rather than just string concatenation to handle those URLs and carefully understand and store the base URL.



Using a standardized library to handle URLs is good practice for clients anyway, but many clients don't.

When using links, you may also need to double-check your API versioning. In many APIs, it is customary to put version numbers in the URL, like this:



/v1/pets/12345

/v2/pets/12345

/v1/people/98765

/v2/people/98765







This is the kind of versioning, where data for a particular resource can be viewed simultaneously in more than one “format” —it is not about versions that replace each other over time as they are subsequently edited.



This situation is very similar to the ability to view the same document in multiple natural languages, for which there is a web standard; what a pity that there is no such standard for versions. By assigning each version its own URL, you upgrade each version to a fully functional web resource. There is nothing wrong with "versioned URLs" of this kind, but they are not suitable for expressing links. If the client requests Lassie in format version 2, this does not mean that he also wants to receive in format 2 information about Joe, the owner of Lassie, so the server cannot choose which version number to include in the link.



Perhaps format 2 for describing owners will not even be provided. There is also no conceptual point in using a specific version of the URL in the links - after all, Lassie does not belong to a specific version of Joe, but Joe as such. Therefore, even if you provide a URL in the format / v1 / people / 98765 and thus identify a specific version of Joe, you must also provide the URL / people / 98765 to identify Joe himself, and it is the second option that you use in links. Another option is to define only the URL / people / 98765 and let clients select a specific version by including the request header for that. There is no standard for this header, but if you call it Accept-Version it works well with naming standard headers.Personally, I prefer to use a header for versioning and avoid using version numbers in the URL. but URLs with version numbers are popular and I often implement the title as well. and “versioned URLs,” since it’s easier to implement both than to argue which is better. You can read more about API versioning in this article .

You may need to document some url patterns anyway

In most web APIs, a new resource URL is allocated by the server when a new resource is created using the POST method. If you use this method to create resources and specify relationships using links, then you do not need to publish a template for the URIs of those resources. However, some APIs allow the client to control the URL of the new resource. By allowing clients to control the URL of new resources, we greatly simplify many of the API scripting patterns for front-end developers, and also support scripts in which the API is used to synchronize the information model with an external source of information. HTTP provides a special method for this purpose: PUT. PUT means "create a resource at this URL if it doesn't already exist, and if it exists, update it."If your API allows clients to create new entities using the PUT method, then you should document the rules for constructing new URLs, perhaps by including the URI pattern in the API specification. You can also give clients partial control over the URL by including a primary key-like value in the body or POST headers. In this case, the POST URI pattern is not required per se, but the client will still have to learn the URI pattern in order to take full advantage of the resulting URI predictability.however, the client will still have to learn the URI pattern to take full advantage of the resulting predictability of the URI.however, the client will still have to learn the URI pattern to take full advantage of the resulting predictability of the URI.



Another context in which it is appropriate to document URL patterns is when the API allows clients to URL-encode requests. Not every API allows you to request your resources, but this can be very useful for clients, and naturally allow clients to URL-encode requests and retrieve the results using a GET method. The following example shows why.



In the above example, we have included the following name / value pair in Joe's view:



"pets": "/pets?owner=/people/98765"







The client does not need to know anything about its structure in order to use this URL, other than that it was written in accordance with standard specifications. Thus, the client can get a list of Joe's pets from this link without having to learn any query language. There is also no need to document the formats of its URL in the API - but only if the client first makes a GET request to /people/98765



... If, in addition, the ability to make requests is documented in the pet store API, then the client can compose the same or an equivalent request URL to retrieve the pets of the owner of interest, without first extracting the owner itself - it will be enough to know the owner's URI. Perhaps even more important, the client can also generate requests like the following, which would otherwise not be possible: The URI specification describes for this purpose a portion of the HTTP URL called the " request component



/pets?owner=/people/98765&species=Dog

/pets?species=Dog&breed=Collie







"Is the portion of the URL after the first"? " to the first “#.” The style of URI request that I prefer to use is to always put client-specific requests in the URI request component, but it is also acceptable to express client requests in the part of the URL called the “path.” Anyway, you need to tell clients how these URLs are constructed - you are actually designing and documenting the request language specific to your API. Of course, you can also allow clients to put requests in the body of the message, not in the URL, and use the POST method rather than GET. practical limit on url size - above 4k bytes you are tempted every time - it is recommended to support POST for requests even if you already support GET.



Because queries are such a useful feature in APIs, and because it is not easy to design and implement query languages, technologies like GraphQL have emerged . I've never used GraphQL, so I can't recommend it, but you can consider it as an alternative for implementing queryability in your API. API request tools, including GraphQL, are best used as an add-on to the standard HTTP API for reading and writing resources, rather than as an alternative to HTTP.

And by the way ... What's the best way to write links in JSON?

JSON, unlike HTML, does not have a built-in mechanism for expressing links. Many people have their own way of understanding how links should be expressed in JSON, and some such opinions have been published in more or less official documents, but at present there are no standards ratified by reputable organizations that would regulate this. In the example above, I expressed links using regular name / value pairs written in JSON - this style is my preferred style and by the way, this style is used in Google Drive and GitHub. Another style you are likely to see is this:

  {"self": "/pets/12345",
 "name": "Lassie",
 "links": [
   {"rel": "owner" ,
    "href": "/people/98765"
   }
 ]
}
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

Personally, I don't see what this style is good for, but some of its variations are quite popular.



There is another style of JSON referencing that I like, and it looks like this:

 {"self": "/pets/12345",
 "name": "Lassie",
 "owner": {"self": "/people/98765"}
}
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

The benefit of this style is that it explicitly gives: "/people/98765"



is a URL, not just a string. I learned this pattern from RDF / JSON . One of the reasons to master this pattern is that you have to use it anyway, whenever you want to display information about one resource nested in another resource, as shown in the following example. If you use this pattern all over the place, the code gets nice uniformity:

{"self": "/pets?owner=/people/98765",
 "type": "Collection",
  "contents": [
   {"self": "/pets/12345",
    "name": "Lassie",
    "owner": {"self": "/people/98765"}
   }
 ]
}
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

For more information on how best to use JSON to represent data, see Terrifically Simple JSON .

Finally, what is the difference between an attribute and a relationship?

I think most readers will agree that JSON lacks a built-in mechanism for expressing links, but there is also a way to interpret JSON that allows you to argue otherwise. Consider the following JSON:

{"self": "/people/98765",
 "shoeSize": 10
}

      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

It is generally accepted that it shoeSize



is an attribute, not a relationship, and 10 is a value, not an entity. True, it is no less logical to assert that the string '10 "is actually a reference written in a special notation intended for referring to numbers, up to the 11th integer, which is itself an entity. If the 11th integer is a perfectly valid entity, and the string '10'



only points to it, then the name / value pair is '"shoeSize": 10'



conceptually a reference, although URIs are not used here.



The same can be said for booleans and strings, so all name / value pairs in JSON can be treated as references. If you think this way of thinking about JSON, then it's natural to use simple name / value pairs in JSON as references to entities that can also be pointed to using a URL.



More generally, this argument is formulated as "there is no fundamental difference between attributes and relationships." Attributes are simply relationships between an entity or another abstract or concrete entity, such as a number or color. But historically, their processing was treated in a special way. Frankly, this is a fairly abstract version of the perception of the world. So, if you show someone a black cat and ask how many objects there are, most people will tell you that there is only one. Few would say that they see two objects - a cat and its black color - and the relationship between them.

Links are simply better

Web APIs that pass database keys rather than simple links are harder to learn and also harder to use for clients. Also APIs of the first kind tie the client and the server more closely, requiring more detailed information as a "common denominator", and all this information needs to be documented and read. The only advantage of APIs of the first kind is that they are so ubiquitous, programmers are comfortable with them, they know how to create and how to consume them. If you are looking to provide customers with high-quality APIs that do not require tons of documentation and maximize client independence from the server, then consider providing links to your web APIs rather than database keys.