Developing scalable RESTFull API using GO, POSTGRES, GORM
Summary
The following is the planning what we will build in this blog post.
- Creating RESTFull API in go
- Using Postgress database to store data
- Using GORM as ORM
- Testing the API using CURL
Setting up the environment.
- lnstalling the golang and configure the enviroment. The following documentation shows the necesary information https://golang.org/doc/install.
- Installing the nodejs server for front end application
- Installing docker to speed up the setup of local environment
- Installing the Postgress in your local machine. You can use the following docker compose file to pull the postgress and pgAdmin image and run in your local machine
Create a file and name it docker-compose.yml . Copy the follwoing code.
# Use postgres/example user/password credentials
version: '3.1'
services:
#https://hub.docker.com/_/postgres
pgdb:
container_name: pgdb
image: postgres:latest
restart: always
ports:
- 5432:5432
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
volumes:
- db-data:/var/lib/postgresql/data
networks:
- postgressNetwork
adminer:
image: adminer
restart: always
ports:
- 8080:8080
networks:
- postgressNetwork
#https://hub.docker.com/r/dpage/pgadmin4/
pgadmin:
container_name: pgadmin
image: dpage/pgadmin4
environment:
PGADMIN_DEFAULT_EMAIL: "[email protected]"
PGADMIN_DEFAULT_PASSWORD: "admin"
ports:
- 5050:80
networks:
- postgressNetwork
networks:
postgressNetwork:
driver: bridge
volumes:
db-data:
Save the file and run docker-compose up . It should pull the postgres docker images and run the postgress on port 8080 . Secondly, it will pull the pgadmin docker image and run the project on port 5050 . In a terminal , run docker ps to see the active container. PGADMIN is a admin UI for postgress database. You can interact with postgress database using a nice user interface in your browser. Navigagte to localhost:5050 to login to pgadmin.
Developing the API server
At first we are going to develop the API server in Go. Go provides net/http package for http client and server implementation. Secondly, we will use gorila/mux package which provide fast and easy request router and HTTP dispatcher for incoming HTTP request.
Lets create a go project (demoapi) in src folder. Create a file main.go. Before writing any code, we need to import appropriate package to use in this project.
Ruen the following in the command line to import the package.
got get -u github.com/jinzhu/gorm
go get -u github.com/lib/pq
go get -u github.com/gorilla/mux
go get -u github.com/gorilla/handlers
go get -u github.com/gorilla/context
Deffining the HTTP server
Project summary: In this porject, I am going to develop a simple CURD opertions and expose as RESTFull api endpint.
Create a file main.go in the project folder. This file is the main entry point for the entire API server. It will contain API endpoint defination and routing to appropriate handler and finally start the API server.
Lets examine the minimum code to set up the HTTP server. We first initizlized the gorila/mux router. Then, defined a simple endpoint which should only returns Hello world if the API request match the route / . After that , we passed the route definenation to http.handle method. Finally we called the http.ListenAndServe method to start the application
package main
import (
"log"
"net/http"
"github.com/gorilla/context"
"github.com/gorilla/mux"
)
func main() {
// Initilizing the gorila/mux router
r := mux.NewRouter()
// Test API endpoints
r.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
w.Write([]byte("Hello world"))
})
http.Handle("/", r)
log.Print("Project is serving on port 7000 : http://localhost:7000")
http.ListenAndServe(":7000", context.ClearHandler(http.DefaultServeMux))
}
Run go run command in the project directory. Open a browser window and navigate to http://localhost:7000/ to see the message Hello world. Thats awsome, we managed to setup a HTTP server just writing a few line of code. I personally liked the simplicity of go.
Defining the API endpoints
At this stage, lets define our product endpoint. Here we register routes mapping URL paths to handlers. This is equivalent to how http.HandleFunc() works: if an incoming request URL matches one of the paths, the corresponding handler is called passing (http.ResponseWriter, *http.Request) as parameters.
// Defining API endpoints
// Retrieve the list of products
r.HandleFunc("/api/products", api.ProductHandlerGETALL).Methods("GET")
//Return Individual Products
r.HandleFunc("/api/products/{id}", api.ProductHandlerGETBYID).Methods("GET")
// Creating product
r.HandleFunc("/api/products", api.ProductHandlerPOST).Methods("POST")
//Deleting product
r.HandleFunc("/api/products/{id}", api.ProductHandlerDELETE).Methods("DELETE")
// Update product
r.HandleFunc("/api/products/{id}", api.ProductHandlerUPDATE).Methods("UPDATE")
Create a folder in the project root directory (api). Secondly, create a file ProductHandler which will contain product API endpoint implementation. For the testing purpose, lets just create the boilerplate endpoint implementation.
ProductHandler.go
package api
import (
"net/http"
)
// ProductHandlerGETALL returns all products
func ProductHandlerGETALL(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("product handler GETALL endpoint"))
}
// ProductHandlerGETBYID returns product by id
func ProductHandlerGETBYID(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("product handler GETBYID endpoint"))
}
// ProductHandlerPOST delete product by id
func ProductHandlerPOST(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("product handler POST point"))
}
// ProductHandlerDELETE delete product by id
func ProductHandlerDELETE(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("product handler DELETE endpoint"))
}
// ProductHandlerUPDATE update product
func ProductHandlerUPDATE(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("product handler UPDATE endpoint"))
}
For the testing purpose , I am going to use the CURL command. Lets run the following command and examine the HTTP response
// Test /api/products GET endpoint
curl --request GET http://localhost:7000/api/products
curl --request GET http://localhost:7000/api/products/1
curl --request POST http://localhost:7000/api/products
curl --request DELETE http://localhost:7000/api/products/1
curl --request UPDATE http://localhost:7000/api/products/1
Configuring database and establish connection to save data in database
I like the concept of single responsibiliy. For that reason, I created a new file ProductService.go in service folder and connector.go in db folder.
connector.go
package db
import (
"github.com/jinzhu/gorm"
_ "github.com/lib/pq"
)
var database *gorm.DB
func init() {
db, err := gorm.Open("postgres", "user=postgres password=postgres dbname=postgres sslmode=disable")
if err != nil {
panic(err.Error())
}
database = db
}
/*
Return instance of Db connection
*/
func GetDb() *gorm.DB {
return database
}
It just establish the connection with postgress using jinzhu/gorm package and return an instance of db which will be used in the service file. Note that, I hardcoded all the parameter to keep the example simple. In real world, we store the database configuration in the environment variable. But at the end of this blog post , I will perform test and modify code to make server secure , scalable and elegant. Lets foucs on service layer and develop the basic CURD operation.
package services
import (
"demoapi/db"
"log"
"github.com/jinzhu/gorm"
)
type Product struct {
gorm.Model
Name string
Description string
Category string
Sku string
Price float32
ImageURL string
}
// This funtion is automatically initialized whenever this handler is called
func init() {
db.GetDb().Debug().DropTable(&Product{}) //Droping the Table to postgress. Do do this in production environment.
db.GetDb().Debug().CreateTable(&Product{}) //re-creating the Table
prod := Product{
Name: "Mac book air",
Description: "Mac product",
Category: "laptop",
Sku: "sddf",
Price: 6.6,
ImageURL: "https://imaga.com/png",
}
db.GetDb().Debug().Save(&prod) // Inserting sample row.
}
/*
Method GetAllProduct
Returns all the produce in the product database
*/
func (product Product) GetAllProduct() []Product {
products := []Product{}
res := db.GetDb().Find(&products)
if res.Error != nil {
log.Print(res.Error)
}
return products
}
/*
Method - GetByID(id uint) (*Product, error)
Description - The function takes product ID as input prameter and returns the product with the relevant parameter
Erro - In case of err message, it returns err mesage in the response
*/
func (product Product) GetByID(id uint) (*Product, error) {
prod := Product{}
res := db.GetDb().Where("id = ?", id).First(&prod)
if res.Error != nil {
log.Print(res.Error)
return nil, res.Error
}
return &prod, nil
}
/*
method - CreateProduct
create proudct entry in database
*/
func (product Product) CreateProduct(prod *Product) uint {
res := db.GetDb().Create(prod)
if res.Error != nil {
log.Print(res.Error)
}
return product.ID
}
/*
method - DeleteProduct(id int) error
Description - Delete the product by ID
*/
func (product Product) DeleteProduct(id uint) error {
res := db.GetDb().Where("id = ?", id).Delete(&Product{})
if res.Error != nil {
log.Print(res.Error)
return res.Error
}
return nil
}
init() funciton creates the schema and populate the database with a simple record. gorm Debug() method logs the generated SQL command in the consol. It is really helpful as we can look at the SQL command to see exactly gorm transform our code to SQL syntex. Each method is comendted with it is functionality and I think the code is self explanatory. So, I am not going into the details. But in case, if you want to undestand more, I would recommend to look ato gorm official documentation.
It is time to change our API handler to use service layer and returns the appriate data as HTTP response.
ProductHandler.go
package api
import (
"demoapi/services"
"encoding/json"
"log"
"net/http"
"strconv"
"github.com/gorilla/mux"
)
// ProductHandlerGETALL should handle all the http request to product endpoint
func ProductHandlerGETALL(w http.ResponseWriter, r *http.Request) {
s := services.Product{}
prod := s.GetAllProduct()
res, _ := json.Marshal(prod)
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
w.Write([]byte(res))
}
// ProductHandlerGETBYID should handle all the http request to product endpoint
func ProductHandlerGETBYID(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
id := vars["id"]
s := services.Product{}
uid, _ := strconv.ParseUint(id, 10, 16)
prod, err := s.GetByID(uint(uid))
if err != nil {
w.WriteHeader(http.StatusNotFound)
w.Write([]byte(err.Error()))
}
res, _ := json.Marshal(prod)
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
w.Write([]byte(res))
}
// ProductHandlerPOST should handle all the http request to product endpoint
func ProductHandlerPOST(w http.ResponseWriter, r *http.Request) {
product := &services.Product{}
err := json.NewDecoder(r.Body).Decode(product)
if err != nil {
w.Write([]byte("Invliad Request"))
}
log.Print(&product)
id := product.CreateProduct(product)
w.WriteHeader(http.StatusCreated)
w.Header().Set("Content-Type", "application/json")
data, _ := json.Marshal(id)
w.Write([]byte(data))
}
// ProductHandlerDELETE delete product
func ProductHandlerDELETE(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
id := vars["id"]
s := services.Product{}
uid, _ := strconv.ParseUint(id, 10, 16)
err := s.DeleteProduct(uint(uid))
w.Header().Set("Content-Type", "application/json")
if err != nil {
w.WriteHeader(http.StatusInternalServerError)
w.Write([]byte(err.Error()))
}
w.WriteHeader(http.StatusNoContent)
w.Write([]byte("product deleted"))
}
// ProductHandlerUPDATE update product
func ProductHandlerUPDATE(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("product handler UPDATE endpoint"))
}
Now, we have a fine grained RESTfull API to perfomr CURD operation. Lets do the test using the CURL command.
curl --request GET http://localhost:7000/api/products
curl --request GET http://localhost:7000/api/products/1
curl -X POST \
http://localhost:7000/api/products \
-H 'content-type: application/json' \
-d '{
"Name": "Mac book air 2",
"Description": "Mac product",
"Category": "laptop",
"Sku": "sddSf",
"Price": 7.6,
"ImageURL": "https://imaga.com/png"
}'
curl -X DELETE http://localhost:7000/api/products/2
Improving the project
As of now, we are adding the HTTP header to the outgoing response in the in http request handler. For RESTFull API, It is important to return the response in the JSON format and the HTTP content type response HEADER application/json. Untill now, we are duplicating the code in each handler. Using middleware we can add the HTTP header easily for all outgoing response. Therefore, it should reduce code duplication.
Like other progromming language, go support HTTP middleware. Lets create a folder middlewares and create a file name httpheader.go.
Middlewares are (typically) small pieces of code which take one request, do something with it, and pass it down to another middleware or the final handler. Some common use cases for middleware are request logging, header manipulation, or ResponseWriter hijacking.
The signature of net/http middleware looks like below.
type MiddlewareFunc func(http.Handler) http.Handler
httpheader.go
package middlewares
import "net/http"
func AddingHttpHeaderMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
next.ServeHTTP(w, r)
})
}
Add this middleware in the main.go file , Middlewares can be added to a router using Router.Use()
r.Use(middlewares.AddingHttpHeaderMiddleware)
At this point , we can remove adding HTTP header in the handler methods. As we have seen , middleware is really powerfull. Later we will use middleware to secure API endpoint using JWT.
Updating the project to use go module
Let’s have a look at the new mechanism to maintain the go dependencies (Go Module). Using the go get command, it is not possible to use a specific version of the dependency. Go introduced a module system which completely addresses the issue. Interestingly, the module mechanism removes the go src path requirements. Create a project directory anywhere in the machine and copy paste the project files. Run go mod init api-server. This command generates two files. go.mod and go.sum and . Modify the specific version if you want to use the older version of any package. Run go build which download all the dependencies defined in the go.mod file. Now, run the project go run main.go.
Details on go modules can be found here.
In the upcoming blog post, I will show step by step process on how to
- Analyzing the source code and potentially tweek the code to make it production ready.
- Securing the API using JWT
- Developing the front end application using angular
- Dockerizing the API server
- Deploying the API server
- Scalling the API server using docker swarm