Building A Containerized Microservice In Golang: A Step-by-step Guide

With the evolving architectural design of web applications, microservices have been a successful new trend in architecting the application landscape. Along with the advancements in application architecture, transport method protocols, such as REST and gRPC are getting better in efficiency and speed. Also, containerizing microservice applications help greatly in agile development and high-speed delivery.

In this blog, I will try to showcase how simple it is to build a cloud-native application on the microservices architecture using Go.

We will break the solution into multiple steps. We will learn how to:

1) Build a microservice and set of other containerized services that will have a very specific set of independent tasks and will be related only to the specific logical component.

2) Use go-kit as the framework for developing and structuring the components of each service.

3) Build APIs that will use HTTP (REST) and Protobuf (gRPC) as the transport mechanisms, PostgreSQL for databases, and finally deploy it on Azure stack for API management and CI/CD.

Note: Deployment, setting up the CI-CD, and API-Management on Azure, or any other cloud is not in the scope of the current blog.

Prerequisites:

What are we going to do?

We will develop a simple web application working on the following problem statement:

Example of a document:

{content: “book”, title: “The Dark Code”, author: “Bruce Wayne”, topic: “Science”}

For a detailed understanding of the requirement, please refer to this.

Architecture:

In this project, we will have 3 microservices: Authentication Service, Database Service, and the Watermark Service. We have a PostgreSQL database server and an API-Gateway.

Microservice Architecture
Microservice Architecture

Authentication Service:

The application is supposed to have a role-based and user-based access control mechanism. This service will authenticate the user according to its specific role and return HTTP status codes only. 200 when the user is authorized and 401 for unauthorized users.

APIs:

Database Service:

We will need databases for our application to store the user, their roles and the access privileges to that role. Also, the documents will be stored in the database without the watermark. It is a requirement that any document cannot have a watermark at the time of creation. A document is said to be created successfully only when the data inputs are valid and the database service returns the success status.

We will be using two databases for two different services for them to be consumed. This design is not necessary, but just to follow the “ Single Database per Service “ rule under the microservice architecture.

APIs:

Watermark Service:

This is the main service that will perform the API calls to watermark the passed document. Every time a user needs to watermark a document, it needs to pass the TicketID in the watermark API request along with the appropriate Mark. It will try to call the database Update API internally with the provided request and returns the status of the watermark process which will be initially “ Started”, then in some time the status will be “ InProgress” and if the call was valid, the status will be “ Finished”, or “ Error “, if the request is not valid.

APIs:

Operations and Flow:

Watermark Service APIs are the only ones that will be used by the user/actor to request a watermark or add the document. Authentication and Database service APIs are the private ones that will be called by other services internally. The only URL accessible to the user is the API Gateway URL.

Note:

Each user will have some specific roles, based on which the access controls will be identified for the user. For the sake of simplicity, the roles will be based on the type of document only, not the specific name of the book or journal

Getting Started:

Let’s start by creating a folder for our application in the $GOPATH. This will be the root folder containing our set of services.

Project Layout:

The project will follow the standard Golang project layout. If you want the full working code, please refer here

We are going to use the Go kit framework for developing the set of services. The official Go kit examples of services are very good, though the documentation is not that great.

Watermark Service:

1. Under the Go kit framework, a service should always be represented by an interface.

Create a package named watermark in the pkg folder. Create a new service.go file in that package. This file is the blueprint of our service.

2. As per the functions defined in the interface, we will need five endpoints to handle the requests for the above methods. If you are wondering why we are using a context package, please refer here. Contexts enable the microservices to handle the multiple concurrent requests, but maybe in this blog, we are not using it too much. It’s just the best way to work with it.

3. Implementing our service:

We have defined the new type watermarkService empty struct which will implement the above-defined service interface. This struct implementation will be hidden from the rest of the world.

NewService() is created as the constructor of our “object”. This is the only function available outside this package to instantiate the service.

4. Now we will create the endpoints package which will contain two files. One is where we will store all types of requests and responses. The other file will be endpoints which will have the actual implementation of the requests parsing and calling the appropriate service function.

- Create a file named reqJSONMap.go. We will define all the requests and responses struct with the fields in this file such as GetRequest, GetResponse, StatusRequest, StatusResponse, etc. Add the necessary fields in these structs which we want to have input in a request or we want to pass the output in the response.

In this file, we have a struct Set which is the collection of all the endpoints. We have a constructor for the same. We have the internal constructor functions which will return the objects which implement the generic endpoint. Endpoint interface of Go kit such as MakeGetEndpoint(), MakeStatusEndpoint() etc.

In order to expose the Get, Status, Watermark, ServiceStatus, and AddDocument APIs, we need to create endpoints for all of them. These functions handle the incoming requests and call the specific service methods

5. Adding the Transports method to expose the services. Our services will support HTTP and will be exposed using Rest APIs and protobuf and gRPC.

Create a separate package of transport in the watermark directory. This package will hold all the handlers, decoders, and encoders for a specific type of transport mechanism

6. Create a file http.go: This file will have the transport functions and handlers for HTTP with a separate path as the API routes.

This file is the map of the JSON payload to their requests and responses. It contains the HTTP handler constructor which registers the API routes to the specific handler function (endpoints) and also the decoder-encoder of the requests and responses respectively into a server object for a request. The decoders and encoders are basically defined just to translate the request and responses in the desired form to be processed. In our case, we are just converting the requests/responses using the json encoder and decoder into the appropriate request and response structs.

We have the generic encoder for the response output, which is a simple JSON encoder.

7. Create another file in the same transport package with the name grpc.go. Similar to the above, the name of the file is self-explanatory. It is the map of protobuf payload to their requests and responses. We create a gRPC handler constructor that will create the set of grpcServers and registers the appropriate endpoint to the decoders and encoders of the request and responses.

- Before moving on to the implementation, we have to create a proto file that acts as the definition of all our service interface and the requests response structs, so that the protobuf files (.pb) can be generated to be used as an interface between services to communicate.

- Create package pb in the api/v1 package path. Create a new file watermarksvc.proto. Firstly, we will create our service interface, which represents the remote functions to be called by the client. Refer to this for syntax and deep understanding of the protobuf.

We will convert the service interface to the service interface in the proto file. Also, we have created the request and response structs exactly the same once again in the proto file so that they can be understood by the RPC defined in the service.

Note: Creating the proto files and generating the pb files using protoc is not the scope of this blog. We have assumed that you already know how to create a proto file and generate a pb file from it. If not, please refer protobuf and protoc gen

I have also created a script to generate the pb file, which just needs the path with the name of the proto file.

8. Now, once the pb file is generated in api/v1/pb/watermark package, we will create a new struct grpcserver, grouping all the endpoints for gRPC. This struct should implement pb.WatermarkServer which is the server interface referred by the services.

To implement these services, we are defining the functions such as func (g *grpcServer) Get(ctx context.Context, r *pb.GetRequest) (*pb.GetReply, error). This function should take the request param and run the ServeGRPC() function and then return the response. Similarly, we should implement the ServeGRPC() functions for the rest of the functions.

These functions are the actual Remote Procedures to be called by the service.

We will also need to add the decode and encode functions for the request and response structs from protobuf structs. These functions will map the proto Request/Response struct to the endpoint req/resp structs. For example: func decodeGRPCGetRequest(_ context.Context, grpcReq interface{}) (interface{}, error). This will assert the grpcReq to pb.GetRequest and use its fields to fill the new struct of type endpoints.GetRequest{}. The decoding and encoding functions should be implemented similarly for the other requests and responses.

9. Finally, we just have to create the entry point files (main) in the cmd for each service. As we already have mapped the appropriate routes to the endpoints by calling the service functions, and also we mapped the proto service server to the endpoints by calling ServeGRPC() functions, now we have to call the HTTP and gRPC server constructors here and start them.

Create a package watermark in the cmd directory and create a file watermark.go which will hold the code to start and stop the HTTP and gRPC server for the service

Let’s walk you through the above code. Firstly, we will use the fixed ports to make the server listen to them. 8081 for HTTP Server and 8082 for gRPC Server. Then in these code stubs, we will create the HTTP and gRPC servers, endpoints of the service backend and the service.

Now the next step is interesting. We are creating a variable of oklog.Group. If you are new to this term, please refer here. Group helps you elegantly manage the group of Goroutines. We are creating three Goroutines: One for HTTP server, second for gRPC server and the last one for watching on the cancel interrupts. Just like this:

Similarly, we will start a gRPC server and a cancel interrupt watcher.
Great!! We are done here. Now, let’s run the service.

The server has started locally. Now, just open a Postman or run curl to one of the endpoints. See below:
We ran the HTTP server to check the service status:

We have successfully created a service and ran the endpoints.

Further:

I really like to make a project complete always with all the other maintenance parts revolving around. Just like adding the proper README, have proper .gitignore, .dockerignore, Makefile, Dockerfiles, golang-ci-lint config files, and CI-CD config files etc.

I have created a separate Dockerfile for each of the three services in path /images/.

I have created a multi-staged dockerfile to create the binary of the service and run it. We will just copy the appropriate directories of code in the docker image, build the image all in one and then create a new image in the same file and copy the binary in it from the previous one. Similarly, the dockerfiles are created for other services also.

In the dockerfile, we have given the CMD as go run watermark. This command will be the entry point of the container.
I have also created a Makefile which has two main targets: build-image and build-push. The first one is to build the image and the second is to push it.

Note: I am keeping this blog concise as it is difficult to cover all the things. The code in the repo that I have shared in the beginning covers most of the important concepts around services. I am still working and continue committing improvements and features.

Let’s see how we can deploy:

We will see how to deploy all these services in the containerized orchestration tools (ex: Kubernetes). Assuming you have worked on Kubernetes with at least a beginner’s understanding before.

In deploy dir, create a sample deployment having three containers: auth, watermark, and database. Since for each container, the entry point commands are already defined in the dockerfiles, we don’t need to send any args or cmd in the deployment.

We will also need the service which will be used to route the external traffic of request from another load balancer service or nodeport type service. To make it work, we might have to create a nodeport type of service to expose the watermark-service to make it running for now.

Another important and very interesting part is to deploy the API Gateway. It is required to have at least some knowledge of any cloud provider stack to deploy the API Gateway. I have used Azure stack to deploy an API Gateway using the resource called as “ API-Management” in the Azure plane. Refer to the rules config files for the Azure APIM api-gateway:

Further, only a proper CI/CD setup is remaining which is one of the most essential parts of a project after development.
I would definitely like to discuss all the above deployment-related stuff in more detail but that is not in the scope of my current blog. Maybe I will post another blog for the same.

Wrapping up:

We have learned how to build a complete project with three microservices in Golang using one of the best-distributed system development frameworks: Go kit. We have also used the database PostgreSQL using the GORM used heavily in the Go community.

We did not stop just at the development but also we tried to theoretically cover the development lifecycle of the project by understanding what, how, and where to deploy.

We created one microservice completely from scratch. Go kit makes it very simple to write the relationship between endpoints, service implementations, and the communication/transport mechanisms. Now, go and try to create other services from the problem statement.

Originally published at https://www.velotio.com.

Velotio Technologies is an outsourced software and product development partner for technology startups & enterprises. #Cloud #DevOps #ML #UI #DataEngineering