Clever Geek Handbook

Go-swagger as a framework for microservices interaction





Hello NickName! If you are a programmer and work with a microservice architecture, then imagine that you need to configure the interaction of your service A with some new and still unknown service B. What will you do first?



If you ask this question to 100 programmers from different companies, we will most likely get 100 different answers. Someone describes contracts in swagger, someone in gRPC simply makes clients to their services without describing a contract. And someone even stores JSON in a googleok: D. Most companies develop their own approach to interservice interaction based on some historical factors, competencies, technology stack, and so on. I want to tell you how the services in Delivery Club communicate with each other and why we made such a choice. And most importantly, how we ensure the relevance of the documentation over time. There will be a lot of code!



Hi again! My name is Sergey Popov, I am the team leader of the team responsible for the search results of restaurants in the apps and on the Delivery Club website, and also an active member of our internal development guild for Go (we may talk about this later, but not now).



I'll make a reservation right away, we will mainly talk about services written in Go. We have not yet implemented code generation for PHP services, although we achieve uniformity in approaches there in a different way.



What we wanted to end up with:



  1. Ensure service contracts are up to date. This should speed up the introduction of new services and facilitate communication between teams.
  2. Come to a unified method of interaction over HTTP between services (we will not consider interactions through queues and event streaming for now).
  3. To standardize the approach to working with service contracts.
  4. Use a single repository of contracts so as not to look for docks for all kinds of confluences.
  5. Ideally, generate clients for different platforms.


From all of the above, Protobuf comes to mind as a unified way to describe contracts. It has good tools and can generate clients for different platforms (our clause 5). But there are also obvious drawbacks: for many, gRPC remains something new and unknown, and this would greatly complicate its implementation. Another important factor was that the company had long adopted the “specification first” approach, and the documentation already existed for all services in the form of a swagger or RAML description.



Go-swagger



Coincidentally, at the same time, we started adapting Go in the company. Therefore, our next candidate for consideration was go-swagger - a tool that allows you to generate clients and server code from the swagger specification. The obvious disadvantage is that it only generates code for Go. In fact, it uses gosh code generation, and go-swagger allows flexible working with templates, so in theory it can be used to generate PHP code, but we haven't tried it yet.



Go-swagger is not only about transport layer generation. In fact, it generates the application skeleton, and here I would like to mention a little about the development culture in DC. We have Inner Source, which means that any developer from any team can create a pull request to any service that we have. For such a scheme to work, we try to standardize approaches in development: we use common terminology, a single approach to logging, metrics, working with dependencies and, of course, to the project structure.



Thus, by implementing go-swagger, we are introducing a standard for the development of our services in Go. This is another step towards our goals, which we initially did not expect, but which is important for the development in general.



The first steps



So, go-swagger turned out to be an interesting candidate that seems to be able to cover most of our wanted requirements.

Note: all further code is relevant for version 0.24.0, installation instructions can be viewed in our repository with examples , and the official website has instructions for installing the current version.
Let's see what he can do. Let's take a swagger spec and generate a service: 



> goswagger generate server \
    --with-context -f ./swagger-api/swagger.yml \
    --name example1


We got the following:







Makefile and go.mod I already made myself.



In fact, we ended up with a service that processes the requests described in swagger. 



> go run cmd/example1-server/main.go
2020/02/17 11:04:24 Serving example service at http://127.0.0.1:54586
 
 
 
> curl http://localhost:54586/hello -i
HTTP/1.1 501 Not Implemented
Content-Type: application/json
Date: Sat, 15 Feb 2020 18:14:59 GMT
Content-Length: 58
Connection: close
 
"operation hello HelloWorld has not yet been implemented"


Step two. Understanding templating



Obviously, the code we generated is far from what we want to see in operation.



What we want from the structure of our application:



  • Be able to configure the application: transfer the settings for connecting to the database, specify the port of HTTP connections, and so on.
  • Select an application object that will store the application state, database connection, and so on.
  • Make the handlers functions of our application, this should simplify the work with the code.
  • Initialize dependencies in the main file (in our example this will not happen, but we still want it.


To solve new problems, we can override some templates. To do this, we will describe the following files, as I did ( Github ):







We need to describe the template files ( `*.gotmpl`) and the file for the configuration ( `*.yml`) of generating our service.



Next, in order, we will analyze the templates that I made. I will not dive deeply into working with them, because the go-swagger documentation is quite detailed, for example, here is the description of the configuration file. I will only note that Go-templating is used, and if you already have experience in this or had to describe HELM-configurations, then it will not be difficult to figure it out.



Configuring the Application



config.gotmpl contains a simple structure with one parameter - the port that the application will listen to for incoming HTTP requests. I also made a function InitConfigthat will read the environment variables and fill this structure. I will call it from main.go, so I InitConfigmade it a public function. 



package config
 
import (
    "github.com/pkg/errors"
    "github.com/vrischmann/envconfig"
)
 
// Config struct
type Config struct {
    HTTPBindPort int `envconfig:"default=8001"`
}
 
// InitConfig func
func InitConfig(prefix string) (*Config, error) {
    config := &Config{}
    if err := envconfig.InitWithPrefix(config, prefix); err != nil {
        return nil, errors.Wrap(err, "init config failed")
    }
 
    return config, nil
}


In order for this template to be used when generating code, you need to specify it in the YML config : 



layout:
  application:
    - name: cfgPackage
      source: serverConfig
      target: "./internal/config/"
      file_name: "config.go"
      skip_exists: false


I'll tell you a little about the parameters:



  • name - has a purely informative function and does not affect the generation.
  • source- actually the path to the template file in camelCase, i.e. serverConfig is equivalent to ./server/config.gotmpl .
  • target- directory where the generated code will be saved. Here you can use templating to dynamically generate a path ( example ).
  • file_name - the name of the generated file, here you can also use templating.
  • skip_exists- a sign that the file will be generated only once and will not overwrite the existing one. This is important for us, because the config file will change as the application grows and should not depend on the generated code.


In the code generation config, you need to specify all files, and not just those that we want to override. For files that we do not change, within the meaning of sourcepoint out asset:< >, for example, here : asset:serverConfigureapi. By the way, if you are interested in looking at the original templates, they are here .



Application object and handlers



I will not describe the application object for storing the state, database connections and other things, everything is similar to the just made config. But with handlers, everything is a little more interesting. Our key goal is for us to create a stub function in a separate file when we add a URL to the specification, and most importantly, our server calls this function to process the request.



Let's describe the function template and stubs: 



package app
 
import (
    api{{ pascalize .Package }} "{{.GenCommon.TargetImportPath}}/{{ .RootPackage }}/operations/{{ .Package }}"
    "github.com/go-openapi/runtime/middleware"
)
 
func (srv *Service){{ pascalize .Name }}Handler(params api{{ pascalize .Package }}.{{ pascalize .Name }}Params{{ if .Authorized }}, principal api{{ .Package }}.{{ if not ( eq .Principal "interface{}" ) }}*{{ end }}{{ .Principal }}{{ end }}) middleware.Responder {
    return middleware.NotImplemented("operation {{ .Package }} {{ pascalize .Name }} has not yet been implemented")
}


Let's look at an example a little:



  • pascalize- brings a line with CamelCase (description of other functions here ).
  • .RootPackage - generated web server package.
  • .Package- the name of the package in the generated code, which describes all the necessary structures for HTTP requests and responses, i.e. structures. For example, a structure for the request body or a response structure.
  • .Name- the name of the handler. It is taken from the operationID in the specification, if specified. I recommend always specifying operationIDfor a more obvious result.


The config for the handler is as follows: 



layout:
  operations:
    - name: handlerFns
      source: serverHandler
      target: "./internal/app"
      file_name: "{{ (snakize (pascalize .Name)) }}.go"
      skip_exists: true


As you can see, the handler code will not be overwritten ( skip_exists: true), and the file name will be generated from the handler name.



Okay, there is a stub function, but the web server does not yet know that these functions should be used to process requests. I fixed this in main.go (I will not give the entire code, the full version can be found here ): 



package main
 
{{ $name := .Name }}
{{ $operations := .Operations }}
import (
    "fmt"
    "log"
 
    "github.com/delivery-club/go-swagger-example/{{ dasherize .Name }}/internal/generated/restapi"
    "github.com/delivery-club/go-swagger-example/{{ dasherize .Name }}/internal/generated/restapi/operations"
    {{range $index, $op := .Operations}}
        {{ $found := false }}
        {{ range $i, $sop := $operations }}
            {{ if and (gt $i $index ) (eq $op.Package $sop.Package)}}
                {{ $found = true }}
            {{end}}
        {{end}}
        {{ if not $found }}
        api{{ pascalize $op.Package }} "{{$op.GenCommon.TargetImportPath}}/{{ $op.RootPackage }}/operations/{{ $op.Package }}"
        {{end}}
    {{end}}
 
    "github.com/go-openapi/loads"
    "github.com/vrischmann/envconfig"
 
    "github.com/delivery-club/go-swagger-example/{{ dasherize .Name }}/internal/app"
)
 
func main() {
    ...
    api := operations.New{{ pascalize .Name }}API(swaggerSpec)
 
    {{range .Operations}}
    api.{{ pascalize .Package }}{{ pascalize .Name }}Handler = api{{ pascalize .Package }}.{{ pascalize .Name }}HandlerFunc(srv.{{ pascalize .Name }}Handler)
    {{- end}}
    ...
}


The code in the import looks complicated, although in reality it is just Go-templating and structures from the go-swagger repository. And in a function, mainwe simply assign our generated functions to the handlers.



It remains to generate the code indicating our configuration: 



> goswagger generate server \
        -f ./swagger-api/swagger.yml \
        -t ./internal/generated -C ./swagger-templates/default-server.yml \
        --template-dir ./swagger-templates/templates \
        --name example2


The final result can be viewed in our repository .



What we got:



  • We can use our structures for the application, configs and whatever we want. Most importantly, it is fairly easy to embed into the generated code.
  • We can flexibly manage the structure of the project, down to the names of individual files.
  • Go templating looks complex and takes some getting used to, but overall it's a very powerful tool.


Step three. Generating clients



Go-swagger also allows us to generate a client package for our service that other Go services can use. Here I will not dwell on code generation in detail, the approach is exactly the same as when generating server-side code.



For Go projects, it is customary to put public packages in ./pkg, we will do the same: put the client for our service in pkg, and generate the code itself as follows: 



> goswagger generate client -f ./swagger-api/swagger.yml -t ./pkg/example3


An example of the generated code is here .



Now all consumers of our service can import this client for themselves, for example, by tag (for my example, the tag will be example3/pkg/example3/v0.0.1).



Client templates can be customized to, for example, flow open tracing idfrom context to header.



conclusions



Naturally, our internal implementation differs from the code shown here mainly due to the use of internal packages and approaches to CI (running various tests and linters). In the generated code out of the box, collection of technical metrics, work with configs and logging are configured. We have standardized all common tools. Due to this, we simplified the development in general and the release of new services in particular, ensured a faster passage of the service checklist before deploying to the product.



Let's check if we have achieved our initial goals:



  1. Ensure the relevance of the contracts described for the services, this should accelerate the introduction of new services and simplify communication between teams - Yes .
  2. HTTP ( event streaming) — .
  3. , .. Inner Source — .
  4. , — ( — Bitbucket).
  5. , — ( , , ).
  6. Go — ( ).


The attentive reader has probably already asked the question: how do template files get into our project? We now store them in each of our projects. This simplifies everyday work, allows you to customize something for a specific project. But there is another side of the coin: there is no mechanism for centralized updating of templates and delivery of new features, mainly related to CI.



PS If you like this material, then in the future we will prepare an article about the standard architecture of our services, we will tell you what principles we use when developing services in Go.


More articles:


All Articles