Scalable Architecture DR CoN: Docker, Registrator, Consul, Consul Template and Nginx

Docker is great fun when you start building things by plugging useful containers together. Recently I have been playing with Consul and trying to plug things together to make a truly horizontally scalable web application architecture. Consul is a Service Discovery and Configuration application, made by HashiCorp the people who brought us Vagrant.

Previously I experimented using Consul by using SRV records (described here) to create a scalable architecture, but I found this approach a little complicated, and I am all about simple. Then I found Consul Template which links to Consul to update configurations and restart application when services come up or go down.

In this post I will describe how to use Docker to plug together Consul, Consul Template, Registrator and Nginx into a truly scalable architecture that I am calling DR CoN. Once all plugged together, DR CoN lets you add and remove services from the architecture without having to rewrite any configuration or restart any services, and everything just works!

Docker

Docker is an API wrapper around LXC (Linux containers) so will only run on Linux. Since I am on OSX (as many of you probably are) I have written a post about how to get Docker running in OSX using boot2docker. This is briefly described below:

1. brew install boot2docker
2. boot2docker init  
3. boot2docker up

This will start a virtual machine running a Docker daemon inside an Ubuntu machine. To attach to the daemon you can run:

1. export DOCKER_IP=`boot2docker ip`  
2. export DOCKER_HOST=`boot2docker socket`

You can test Docker is correctly installed using:

1. docker ps

Build a very simple Web Service with Docker

To test the Dr CoN architecture we will need a service. For this, let create the simplest service that I know how (further described here). Create a file called Dockerfile with the contents:

1. FROM  python:3  
2. EXPOSE  80  
3. CMD ["python", "-m", "http.server"]

In the same directory as this file execute:

1. docker build -t python/server .

This will build the docker container and call it python/server, which can be run with:

1. docker run -it \
2. -p 8000:80 python/server

To test that it is running we can call the service with curl:

1. curl $DOCKER_IP:8000

Consul

Consul is best described as a service that has a DNS and a HTTP API. It also has many other features like health checking services, clustering across multiple machines and acting as a key-value store. To run Consul in a Docker container execute:

1. docker run -it -h node \
2.  -p 8500:8500 \
3.  -p 8600:53/udp \
4.  progrium/consul \
5.  -server \
6.  -bootstrap \
7.  -advertise $DOCKER_IP \
8.  -log-level debug

If you browse to $DOCKER_IP:8500 there is a dashboard to see the services that are registered in Consul.
To register a service in Consul's web API we can use curl:

1. curl -XPUT \
2. $DOCKER_IP:8500/v1/agent/service/register \
3. -d '{
4.  "ID": "simple_instance_1",
5.  "Name":"simple",
6.  "Port": 8000, 
7.  "tags": ["tag"]
8. }'

Then we can query Consuls DNS API for the service using dig:

1. dig @$DOCKER_IP -p 8600 simple.service.consul

1. ; <<>> DiG 9.8.3-P1 <<>> simple.service.consul
2. ;; global options: +cmd
3. ;; Got answer:
4. ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39614
5. ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
6. 
   
7. ;; QUESTION SECTION:
8. ;simple.service.consul.        IN    A
9. 
   
10. ;; ANSWER SECTION:
11. simple.service.consul.    0    IN    A    192.168.59.103
12. 
    
13. ;; Query time: 1 msec
14. ;; SERVER: 192.168.59.103#53(192.168.59.103)
15. ;; WHEN: Mon Jan 12 15:35:01 2015
16. ;; MSG SIZE  rcvd: 76

Hold on, there is a problem, where is the port of the service? Unfortunately DNS A records do not return the port of a service, to get that we must check SRV records:

1. dig @$DOCKER_IP -p 8600 SRV simple.service.consul

1. ; <<>> DiG 9.8.3-P1 <<>> SRV simple.service.consul
2. ;; global options: +cmd
3. ;; Got answer:
4. ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3613
5. ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
6. 
   
7. ;; QUESTION SECTION:
8. ;simple.service.consul.        IN    SRV
9. 
   
10. ;; ANSWER SECTION:
11. simple.service.consul.    0    IN    SRV    1 1 8000 node.node.dc1.consul.
12. 
    
13. ;; ADDITIONAL SECTION:
14. node.node.dc1.consul.    0    IN    A    192.168.59.103
15. 
    
16. ;; Query time: 1 msec
17. ;; SERVER: 192.168.59.103#53(192.168.59.103)
18. ;; WHEN: Mon Jan 12 15:36:54 2015
19. ;; MSG SIZE  rcvd: 136

SRV records are difficult to use because they are not supported by many technologies.
The container srv-router can be used with Consul and nginx to route incoming calls to the correct services, as described here. However there is an easier way than that to use nginx to route to services.

Registrator

Registrator takes environment variables defined when a Docker container is started and automatically registers it with Consul. For example:

1. docker run -it \
2. -v /var/run/docker.sock:/tmp/docker.sock \
3. -h $DOCKER_IP progrium/registrator \
4. consul://$DOCKER_IP:8500

Starting a service with:

1. docker run -it \
2. -e "SERVICE_NAME=simple" \
3. -p 8000:80 python/server

Will automatically add the service to Consul, and stopping it will remove it. This is the first part to plugin to DR CoN as it will mean no more having to manually register services with Consul.

##Consul Template

Consul Template uses Consul to update files and execute commands when it detects the services in Consul have changed.

For example, it can rewrite an nginx.conf file to include all the routing information of the services then reload the nginx configuration to load-balance many similar services or provide a single end-point to multiple services.

I modified the Docker container from https://github.com/bellycard/docker-loadbalancer for this example

1. FROM nginx:1.7
2. 
   
3. #Install Curl
4. RUN apt-get update -qq && apt-get -y install curl
5. 
   
6. #Download and Install Consul Template
7. ENV CT_URL http://bit.ly/15uhv24
8. RUN curl -L $CT_URL | \
9. tar -C /usr/local/bin --strip-components 1 -zxf -
10. 
    
11. #Setup Consul Template Files
12. RUN mkdir /etc/consul-templates
13. ENV CT_FILE /etc/consul-templates/nginx.conf
14. 
    
15. #Setup Nginx File
16. ENV NX_FILE /etc/nginx/conf.d/app.conf
17. 
    
18. #Default Variables
19. ENV CONSUL consul:8500
20. ENV SERVICE consul-8500
21. 
    
22. # Command will
23. # 1. Write Consul Template File
24. # 2. Start Nginx
25. # 3. Start Consul Template
26. 
    
27. CMD echo "upstream app {                 \n\
28.   least_conn;                            \n\
29.   {{range service \"$SERVICE\"}}         \n\
30.   server  {{.Address}}:{{.Port}};        \n\
31.   {{else}}server 127.0.0.1:65535;{{end}} \n\
32. }                                        \n\
33. server {                                 \n\
34.   listen 80 default_server;              \n\
35.   location / {                           \n\
36.     proxy_pass http://app;               \n\
37.   }                                      \n\
38. }" > $CT_FILE; \
39. /usr/sbin/nginx -c /etc/nginx/nginx.conf \
40. & CONSUL_TEMPLATE_LOG=debug consul-template \
41.   -consul=$CONSUL \
42.   -template "$CT_FILE:$NX_FILE:/usr/sbin/nginx -s reload";

The repository for this file is here.

NOTE: the \n\ adds a new line and escapes the newline for Docker multiline command
This Docker container will run both Consul Template and nginx, and when the services change it will rewrite the nginx app.conf file, then reload nginx.

This container can be built with:

1. docker build -t drcon .

and run with:

1. docker run -it \
2. -e "CONSUL=$DOCKER_IP:8500" \
3. -e "SERVICE=simple" \
4. -p 80:80 drcon

SERVICE is query used to select which services to include from Consul. So this DR CoN container will now load balance across all services names simple.

##All Together

Lets now plug everything together!

Run Consul

1. docker run -it -h node \
2.  -p 8500:8500 \
3.  -p 53:53/udp \
4.  progrium/consul \
5.  -server \
6.  -bootstrap \
7.  -advertise $DOCKER_IP

Run Registrator

1. docker run -it \
2. -v /var/run/docker.sock:/tmp/docker.sock \
3. -h $DOCKER_IP progrium/registrator \
4. consul://$DOCKER_IP:8500

Run DR CoN

1. docker run -it \
2. -e "CONSUL=$DOCKER_IP:8500" \
3. -e "SERVICE=simple" \
4. -p 80:80 drcon

Running curl $DOCKER_IP:80 will return:

1. curl: (52) Empty reply from server

Now start a service named simple

1. docker run -it \
2. -e "SERVICE_NAME=simple" \
3. -p 8000:80 python/server

This will cause:

• Registrator to register the service with Consul
• Consul Template to rewrite the nginx.conf then reload the configuration

Now curl $DOCKER_IP:80 will be routed successfully to the service.

If we then start another simple service on a different port with:

1. docker run -it \
2. -e "SERVICE_NAME=simple" \
3. -p 8001:80 python/server

Requests will now be load balances across the two services.

A fun thing to do is to run while true; do curl $DOCKER_IP:80; sleep 1; done while killing and starting simple services and see that this all happens so fast no requests get dropped.

Conclusion

Architectures like DR CoN are much easier to describe, distribute and implement using Docker and are impossible without good tools like Consul. Plugging things together and playing with Docker's ever more powerful tools fun and useful. Now I can create a horizontally scalable architecture and have everything just work.
Written by Graham Jenson

If you found this post interesting, follow and support us.
Suggest for you:

Zero to Hero with Python Professional Python Programmer Bundle

The Python Mega Course: Build 10 Python Applications

Advanced Scalable Python Web Development Using Flask

Python 1000: The Python Primer

Data Mining with Python