Metarpheus: a custom approach to API contracts
How we auto-generate client-side code from our API code, using scala.meta
When it comes to defining an API contract, there are a bunch of things to consider: ease of definition, customization, documentation… Should it be kept in sync with actual code? Should we be able to generate docs and other artifacts automatically out of it? Most of all, it is a matter of how it impacts your development workflow.
In this post I’ll give an overview of how we do this at buildo, building Scala+JS web apps and taking advantage of the great capabilities of scala.meta. I’ll also spend a few words about how our solution compares to other widely adopted ones.
The Problem
If your app is composed of at least a client and an API, you’d typically want to:
- Define and enforce a commonly agreed upon contract for exchanging messages between the two
- Potentially share part of the domain model definition for the scope of this contract: what do the exchanged entities look like? What are the available operations on those entities?
- Provide a clear and painless path for iterations involving both API and client: how do you define new entities and operations? How do you ensure that both client and server implementations can be carried out in parallel, without depending one on the other?
- Have a way to ensure the correctness and completeness of the implementations (both client and server), as automated as possible: is this new PR implementing all the required server-side operations? Is this client code handling correctly all possible entities the API can return?
If the API is intended to be consumed by third party clients, you’d also want to make sure that its specification is built on known and open source standards, to lower the barrier for potential future contributors. I’m listing this point separately since it doesn’t apply to our use case at buildo.
Solutions Out There
The king, when speaking of the HTTP/REST world, is obviously Swagger. It provides a lot of tools aiding in design, documentation and consumption of such APIs. The language agnostic specification to describe REST APIs is at the core of the ecosystem; every tool will either consume or produce it. It is an actual “code” artifact (even though language agnostic), and it should be versioned.
Another widespread solution is Apiary: it offers an hosted platform to sketch your API definition and iterate over it, based on the API Blueprint standard, together with the ability to generate the corresponding documentation and to integrate it with your own GitHub repository. It even serves as a mock server, useful for quickly prototyping and evolving an API in the early stages of development.
Worth mentioning, and recently gaining a lot of traction, is GraphQL. Even though much more revolutionary in the way it supersedes the concept of REST APIs altogether, it embeds again a similar approach for “API” definition: create a GraphQL schema, and use it to build clients and documentation. The specification lives on its own, and even better, there are tools to interactively explore any GraphQL api.
Going Custom
Restricting the problem to buildo, we invested in a custom solution tailored to our needs: metarpheus is the core of such solution, meant to solve the problems discussed above.
First of all, its name comes from scala.meta + Morpheus from The Matrix, for some reason. As Andrea put it when he wrote the first version of metarpheus:
Morpheus is the god of dreams. And it takes our Scala API definitions and transforms it into a type-checked JS API client, or documentation. It morphs Scala into other forms.
Back on track: at buildo, we mainly work on web applications including custom Scala APIs and JavaScript (React) clients. The first step in our journey to produce a client-consumable API contract is to look for it in our Scala source code, and extract it into an intermediate representation.
Parsing and Extracting with scala.meta
Thanks to the awesome scala.meta, metarpheus is able to extract domain models (represented as Scala type definitions) and api routes (spray or akka-http DSL).
Metarpheus uses the scala.meta syntactic API to parse all the relevant scala files and obtain a complete syntax tree out of it. The AST also includes comments, with precise locations: it means we can include them in our intermediate output and, for instance, reproduce them in the final generated JS code.
A thorough review of scala.meta features is out scope for this post but, if you are curious, be sure to take a look at all the things you can do with its powerful api, and all that’s been done already. You can dig into metarpheus code as well (BEWARE: do it at your own risk).
Having obtained the full AST for all the source files of interest, the next step is to extract our API routes, and the “models” (case classes and enums) that are referenced by each route. In other words, we obtain all the API operations and parts of the domain model — the ones exposed to the clients of the API, and thus part of the specification.
Generating JS Code
Once we parsed and extracted all the relevant models and routes, we produce a language-agnostic intermediate representation in JSON.
Given this intermediate representation, other tools can then generate JS code representing type definitions, a JS HTTP client, and reference API documentation.
Speaking of Scala-to-JS, here’s the thing, “full-stack”: this scala code…
case class User(email: String, password: String)class UserRouterImpl extends Router {
override val route = {
// GET /user?id
(path("user") & parameter('id.as[Int]).as(UserId) /**
get a single user
@param id user id
*/)(returns[User].ctrl(userController.user))
}
}
…becomes this JS code
// generated model in JS:export const User = t.struct({
email: t.String,
password: t.String
});// example usage of the generated HTTP client:api.usersController_getById({
params: { id: 42 }
}) // Promise<User>
A few things to note before proceeding:
we are looking at the full-stack beast (from Scala to JS). As already said, the process also involves an intermediate JSON representation, and we can get more than just JS out of it (more about this later).
In the Scala snippet above, the returns[User].ctrl part is custom DSL. The Spray DSL is very powerful; in our code we put some restrictions on it, and we require e.g. userController.user to adhere to a specific interface. This fact also has the nice side-effect of facilitating metarpheus' parsing job.
The JS code uses tcomb for runtime type-checking in JS. We have a fairly comprehensive list of “standard” mapping from Scala types to tcomb types:
- a bunch of “primitive” types, e.g.
String,Boolean, etc. - a case class becomes a
t.struct()(just as in the example above) - case enums are expressed as
t.enums.of() - some generic types are handled, such as
List[T]andSet[T] - in addition to these, project-specific overrides can be provided via a configuration file
tcomb offers runtime type checking for JavaScript. We are currently experimenting with static type checking (flow and Typescript) on different projects, and already have a method in place to produce Typescript type definitions instead of JS/tcomb code. I must say, it was fairly easy to implement.
At a Higher Level
Let’s take a look at what are the differentiating points (and benefits?) of our custom solution, compared to other more popular ones.
First of all, metarpheus can generate an intermediate representation of the API contract, and then JS, straight out of Scala code. The workflow could be described as “generate client code directly from server code”. It simplifies the somewhat more traditional approach of “generate server and client code starting from a versioned definition file”. Note that this can work because all of our API code bases are written in a single language / router framework: we don’t share definitions among services implemented in different languages, server-side.
In our experience, it has worked out pretty well: the API-client development workflow is not impacted negatively, in particular considering that both Scala and JS people can write down (and iterate on) the server “stubs” (models and api routes) that are fairly simple to define in plain Scala.
Sure, there are other means to obtain the same (generation of the API specification from server code) with other solutions. For instance, using Swagger core annotations. With metarpheus though, we can write Scala code without additional annotations or major restrictions on syntax, basically the same code you’d write if it wasn’t there at all. Our specification is plain Scala code.
One last brief point, about versioning. At buildo, Scala APIs are versioned alongside JS clients, in the same git repository. The developer is in charge of regenerating the type definitions and client code before submitting a PR. Our CIs typically run through a step, called “metarpheus-diff”, where server-client parity is checked by re-generating the definitions and diffing against the versioned ones. In other words, when part of the API changes even slightly, the CI will tell you.
Why not Swagger?
You might be asking, and rightfully so, why not just using Swagger and be done with it? As already said above: we like the idea that our Scala code can be the source of truth itself, the specification.
And well, actually, if needed, we do use Swagger. A fairly common case is the one involving generation of API reference docs. In this case we resort to some of the awesome tools available in the Swagger world: it’s just a matter of “translating” our intermediate language into a valid Swagger specification.
Here below, the JSON intermediate representation produced by the example above (User and GET /users/:id)
{
"models": [{
"name": "User",
"members": [{
"name": "username",
"tpe": { "name": "String" }
}, {
"name": "password",
"tpe": { "name": "String" }
}]
}],
"routes": [{
"method": "get",
"route": [{
"str": "users"
}, {
"routeParam": {
"tpe": { "name": "Integer" }
}
}],
"returns": { "name": "User" },
"desc": "get a single user",
"name": ["userController", "user"]
}]
}Using metarpheus-swagger we can easily produce a swagger.yml out of this, and then the docs. The produced Swagger for this example would look something like:
definitions:
User:
type: "object"
properties:
username:
description: "username"
type: "string"
password:
description: "password"
type: "string"
paths:
/users/me:
get:
summary: "gets the currently logged user"
parameters:
-
in: "path"
required: true
type: "integer"
responses:
200:
description: "a User"
schema:
$ref: "#/definitions/User"Should I Build My Own Custom Solution?
It depends.
As any custom solution, it must be evaluated carefully. For us, it has worked well. One reason is that we use a single server-side language, and the same stack is repeated across many different projects, without profound variations.
This article is meant to show that, if your team is ok with trading generality and standardization in exchange for a solution that fits exactly your stack and workflow needs, it’s indeed possible to build and maintain a custom solution like ours. And lots of fun, too!
Summing Up
Hope you enjoyed reading through the article :) To sum it up, these are the main points I tried to convey:
- If you don’t need to support many different server-side languages/frameworks, an intermediate specification language is not strictly needed: you can generate outputs for different client languages, documentation and other artifacts straight out of the API source code.
- By leveraging scala.meta it’s possible to extract and generate an API contract and, in turn, client code, leaving the Scala code intact: no need for ad-hoc annotations.
Overall, being very custom for our stack, metarpheus has been serving us pretty well for a few years now, and required a relatively low effort in maintaining and evolving it.
Happy API specification!
—
If you want to work in a place where we care about the quality of our development workflow, take a look at https://buildo.io/careers
