Vagrant Docker Swarm



As discussed in the previous post I want to create a local environment that utilises many of the tools and architecture needed in production, such as High Availability, Fault Tolerance, Service Discovery, Data Persistence, Centralised Logging and Monitoring

The underlying architecture for experimenting with the different tools will be the latest version of Docker Swarm (1.12) running on a number of VirtualBox virtual machines.

These blog posts will start with Vagrant and the swarm and iteratively add new components all via Ansible.

The status of the project is detailed on the Changelog

Every component in the system will be fully automated. I could have chosen to use Docker Machine to spin up the virtual machines but I chose to use Vagrant due to its clean integration with Ansible. Ansible then provisions each component in the system. In order to keep things modular the tools and tasks are grouped into logical areas

  • Docker Swarm (Docker 1.12)
  • Logging (centralised logging to ELK stack)
  • Service Discovery (1.12 includes discovery but not external to the swarm)
  • Monitoring
  • Applications (simple applications to demo logging, monitoring and the swarm)

Each of these groups can be provisioned separately with Vagrant or directly with Ansible.

The project can be found on github


The Vagrantfile brings up 6 VirtualBox virtual machines.

3 Managers and 3 Worker boxes. There are 3 managers so we can test swarm & consul clustering. It needs to be an odd number so Quorum can be achieved. Its possible to run 1 manager and 1 worker if the resources are not there. Though the Ansible groups will need to be re-balanced to move monitoring and logging

  "managers" => ["manager[1:#{MANAGERS}]"],
  "workers" => ["worker[1:#{WORKERS}]"],
  "elk" => ["manager[2:2]"],
  "influxdb" => ["manager[3:3]"],
  "all_groups:children" => [

The virtual machines are running Centos7. Many of the production environments I have previously worked on used either Redhat or Centos. For machines that are only running Docker I may look to use a more lightweight version of Linux. Many of the small containers use Alpine Linux due to is small footprint. So that could be a good option for the actual virtual machine.

Vagrant plugins

  • vagrant-host-shell
  • vagrant-alpine
  • vagrant-vbguest
> vagrant plugin install vagrant-alpine

Running Vagrant

To provision the virtual machines and install and configure the Swarm and overlay network

> vagrant up --provision

Each area is managed by its own Ansible playbook which can be invoked using

> vagrant provision --provision-with monitoring

If you want the entire thing run consul, monitoring, logging, apps

To log onto the boxes simply run

> vagrant ssh manager1
Last login: Thu Aug 25 11:27:47 2016 from
[vagrant@manager1 ~]$ sudo su
[root@manager1 vagrant]#


Ansible was chosen as it provides a simple, easily readable way to automate the installation and configuration of each component. Vagrant auto-generates the inventory simplifying things further. We can then reference the boxes by the group hostvars information. Each area has its own playbook and the components are all abstracted away in roles.

As mentioned above. Ansible can either by run via vagrant provision or directly. I have created the following alias

alias ansible-vagrant='PYTHONUNBUFFERED=1 ANSIBLE_FORCE_COLOR=true ANSIBLE_HOST_KEY_CHECKING=false ANSIBLE_SSH_ARGS='\''-o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ControlMaster=auto -o ControlPersist=60s'\'' ansible-playbook --connection=ssh --timeout=30 --inventory-file=.vagrant/provisioners/ansible/inventory'

which allows me to run individual playbooks from the root project directory. Its slightly faster and you can add extra verbosity / limits etc.

> ansible-vagrant ansible/monitoring.yml -vvv


One of Ansible’s key features is that it’s modules are (mostly) idempotent. I.e they can be re-run, if the desired effect is already achieved then it skips the task. Use of the shell modules causes a problem as its not idempotent if it changes the system.

Ansible has released many new docker modules with the v2 release though it doesn’t support Docker 1.12 service, networking yet. I have therefore used shell to query the current state and only run shell if the network or service hasn’t been created. This doesn’t allow for restarting but we will have to live with that at the moment.

- name: Get existing services
  shell: >
    docker service ls --filter name={{ }} | tail -n +2
  with_items: "{{ docker_services }}"
  register: services_result

- set_fact:
    docker_current_services: "{{ services_result.results | map(attribute='stdout') | list | join(' ') }}"

The example above formats up the response so we can easily search it later.

The current docker command doesn’t have any json options so we have to filter and strip the header

[root@manager1 vagrant]# docker service ls
ID            NAME      REPLICAS  IMAGE                COMMAND
3k1j6qxshlaq  collectd  global    collectd
63cjh5tbg8gr  influxdb  1/1       tutum/influxdb:0.12

As we are checking multiple apps with with_items the json returned will have a list of dictionaries. The new map filter allows us to select a value from a list of dicts.

I then flatten this and turn it into a string which can be searched. We don’t run the command if the service exists.

shell: >
    docker service create \
      --name {{ }} \
      --env "CONSUL_SERVICE_PORT={{ service_dict.service_port | default(80) }}" \
      --log-driver syslog \
      --log-opt tag={{ docker_syslog_tag }} \
      {{ service_dict.definition }}
when: not in docker_current_services

Docker Swarm

Every virtual machine runs the latest (v1.12) version of Docker. Each host belongs to an Ansible group defined in the Vagrant file. This allows us to target specific types of boxes.

Docker 1.12 is installed via curl as per the docker release page. We only do this if there is no docker systemd service installed on the host.

- name: Check existence of docker
  stat: path=/usr/lib/systemd/system/docker.service
  register: install_result

- name: Install docker 1.12
  shell: >
    curl -fsSL | sh
  when: not install_result.stat.exists

- name: Start docker service
  service: name=docker state=started enabled=true

Before we run the swarm init command we establish whether there is an existing swarm running on this machine. Currently this is done by talking directly to the docker daemon. We know its running as docker as we have previously started it.

- name: Check the docker status
  no_log: true
  shell: >
    echo -e "GET /info HTTP/1.0\r\n" | nc -U /var/run/docker.sock | tail -n +6 | python -m json.tool
  register: docker_result

- set_fact:
    docker_info: "{{ docker_result.stdout | from_json }}"

Swarm manager(s)

Initially we start a swarm manager on the first manager host

- name: Configure primary swarm manager
  hosts: managers[0]
  become: yes
  become_user: root
    - role:
      dds_host: "{{ vagrant_primary_manager_ip }}"
      dds_server_cert_path: /etc/default/docker
      dds_restart_docker: no

    - name: "Starting primary swarm manager"
      shell: >
        docker swarm init --advertise-addr {{ vagrant_primary_manager_ip }}
      register: init_result
      when: docker_info.Swarm.LocalNodeState != "active"

Each other manager initialises using the manager token described below

Swarm tokens

Next we register facts for the manager and worker swarm tokens that are needed when we initialise the other hosts.

- name: "Retrieve manager token"
  shell: >
    docker swarm join-token manager --quiet
  register: manager_token_result

- set_fact:
    manager_token: "{{ manager_token_result.stdout }}"

- name: "Retrieve worker token"
  shell: >
    docker swarm join-token worker --quiet
  register: worker_token_result

- set_fact:
    worker_token: "{{ worker_token_result.stdout }}"

Swarm workers

Each worker host connects to the primary manager instance using the worker token.

- hosts: workers
  become: yes
  become_user: root
    - name: "Starting swarm workers"
      shell: >
        docker swarm join \
          --token {{ hostvars['manager1']['worker_token'] }} \
          {{ vagrant_primary_manager_ip }}:{{ swarm_bind_port }}
      register: init_result
      when: docker_info.Swarm.LocalNodeState != "active"

Swarm node labels

Sometimes we want to restrict where Docker services can run. The cleanest way of doing this with our current setup is to associate the Ansible groups with the Docker nodes.

- hosts: all
  serial: 1
  become: yes
  become_user: root
    - name: "Label nodes"
      shell: >
        docker node update --label-add {{ item }}=true {{ inventory_hostname }}
      when: "item != 'all_groups'"
        - "{{ group_names }}"
      delegate_to: "{{ groups['managers'][0] }}"
        - label

Node updates should be performed on a manager node. In the above example we run a shell command on all hosts. Since docker node update needs to be run against a manager node we delegate to the primary manager. We iterate over all the hosts groups {{ group_names }} and add a boolean label for each host. This means that we can later restrict services to a named group using --constraint 'node.labels.xxxx == true'

TASK [Label nodes] *************************************************************
task path: /Users/jamesdmorgan/Projects/vagrant-ansible-docker-swarm/ansible/swarm.yml:122

skipping: [worker2] => (item=all_groups)  => {"changed": false, "item": "all_groups", "skip_reason": "Conditional check failed", "skipped": true}

changed: [worker2 -> None] => (item=workers) => {"changed": true, "cmd": "docker node update --label-add workers=true worker2", "delta": "0:00:00.014551", "end": "2016-08-30 10:41:57.119466", "item": "workers", "rc": 0, "start": "2016-08-30 10:41:57.104915", "stderr": "", "stdout": "worker2", "stdout_lines": ["worker2"], "warnings": []}

We skip over the group all_groups as it doesn’t add any useful semantics.

Swarm status

After all the nodes have been added and labeled the status of the swarm is output

TASK [debug] *******************************************************************
task path: /Users/jamesdmorgan/Projects/vagrant-ansible-docker-swarm/ansible/swarm.yml:146
ok: [manager1] => {
    "docker_swarm_info.Swarm": {
        "Cluster": {
            "CreatedAt": "2016-08-30T09:41:51.32899677Z",
            "ID": "2m72mdv7c8aslr8vpmis615k4",
            "Spec": {
                "CAConfig": {
                    "NodeCertExpiry": 7776000000000000
                "Dispatcher": {
                    "HeartbeatPeriod": 5000000000
                "Name": "default",
                "Orchestration": {
                    "TaskHistoryRetentionLimit": 5
                "Raft": {
                    "ElectionTick": 3,
                    "HeartbeatTick": 1,
                    "LogEntriesForSlowFollowers": 500,
                    "SnapshotInterval": 10000
                "TaskDefaults": {}
            "UpdatedAt": "2016-08-30T09:41:51.368332853Z",
            "Version": {
                "Index": 11
        "ControlAvailable": true,
        "Error": "",
        "LocalNodeState": "active",
        "Managers": 3,
        "NodeAddr": "",
        "NodeID": "3d5nu66a7nuyjqauz3imkinwz",
        "Nodes": 6,
        "RemoteManagers": [
                "Addr": "",
                "NodeID": "dv4fyiq4mbriyqrace841cybd"
                "Addr": "",
                "NodeID": "0cf1851b78xocoymd3kwvqa1p"
                "Addr": "",
                "NodeID": "3d5nu66a7nuyjqauz3imkinwz"

PLAY [managers[0]] ***********************

This shows that there are 3 manager nodes and 6 nodes in total.

Information about the nodes can be obtained by running the ```docker node ls`` command on a manager host

$ vagrant ssh manager1
Last login: Tue Aug 30 10:41:58 2016 from

[vagrant@manager1 ~]$ sudo su

[root@manager1 vagrant]# docker node ls
0cf1851b78xocoymd3kwvqa1p    manager3  Ready   Active        Reachable
3d5nu66a7nuyjqauz3imkinwz *  manager1  Ready   Active        Leader
50pxd3u3r3uk7e0navs39w3ui    worker3   Ready   Active
67wijncvkd002abmt2c9eekem    worker1   Ready   Active
6j1fzoab9sm1417wjkaudx3w9    worker2   Ready   Active
dv4fyiq4mbriyqrace841cybd    manager2  Ready   Active        Reachable
[root@manager1 vagrant]#
[root@manager1 vagrant]#

[root@manager1 vagrant]# docker node help

Usage:  docker node COMMAND

Manage Docker Swarm nodes

      --help   Print usage

  demote      Demote one or more nodes from manager in the swarm
  inspect     Display detailed information on one or more nodes
  ls          List nodes in the swarm
  promote     Promote one or more nodes to manager in the swarm
  rm          Remove one or more nodes from the swarm
  ps          List tasks running on a node
  update      Update a node

Run 'docker node COMMAND --help' for more information on a command.

[root@manager1 vagrant]# docker node inspect manager1
        "ID": "3d5nu66a7nuyjqauz3imkinwz",
        "Version": {
            "Index": 41
        "CreatedAt": "2016-08-30T09:41:51.329014597Z",
        "UpdatedAt": "2016-08-30T09:41:55.392224112Z",
        "Spec": {
            "Labels": {
                "managers": "true"
            "Role": "manager",
            "Availability": "active"
        "Description": {
            "Hostname": "manager1",
            "Platform": {
                "Architecture": "x86_64",
                "OS": "linux"
            "Resources": {
                "NanoCPUs": 2000000000,
                "MemoryBytes": 1929281536
            "Engine": {
                "EngineVersion": "1.12.1",
                "Plugins": [
                        "Type": "Network",
                        "Name": "bridge"
                        "Type": "Network",
                        "Name": "host"
                        "Type": "Network",
                        "Name": "null"
                        "Type": "Network",
                        "Name": "overlay"
                        "Type": "Volume",
                        "Name": "local"
        "Status": {
            "State": "ready"
        "ManagerStatus": {
            "Leader": true,
            "Reachability": "reachable",
            "Addr": ""
[root@manager1 vagrant]#

Docker networks

In order for our services to be able to communicate with each over across the different nodes we need to add an overlay network. As the system gets more complicated its likely we will want to add more to segment the different layers of the system, split frontend traffic from backend etc.

- name: Create overlay networks
  shell: >
    docker network create -d overlay {{ item }}
  when: item not in docker_current_networks
    - appnet

Docker UI

Docker UI provides visualisation for the docker daemon.



Having the UI run as a service talking to the docker daemon works though doesn’t make a huge amount of sense. Each host has its own UI talking the local daemon. If you hit the swarm endpoint on port 9000 it will round-robin to one of the running UIs.

We could run each UI on a different port and have some means to choose the host we want to look at. We could also just examine the daemon on the primary manager node.

comments powered by Disqus