Ansible and Docker – The Payload Payoff?

I work for a big company.

So, when I endeavor to try to build something that everyone in this “big company” could use, I need to create something pretty darn flexible.

I’ve spent some time with DevOps, evangelizing and implementing, and basically telling people that sliced bread is not neat, but DevOps is. The challenge that we quickly run into is the folks I’m talking to want to get involved sooner than later, even to a limited degree.  They need a kick start, or a simple baseline in which they can extend, or (gasp) use as-is.

Basically, I need an application(s) that can deploy a DevOps environment for my dev teams that provides a good portion of the baseline infrastructure they need.

Each of these teams will need their own environment (at least one), simply due to load, security, and just clean living. Of course, I’m going to use DevOps to deploy DevOps, it only makes meta sense. Möbius would be proud.


Alright, so taking inventory, I need to build an environment that replicates itself, at least to some degree. Something portable, able to be customized on demand, and provides fully automated deployment in a secure fashion. I think I need something like a software RepRap (a 3D printer that can print 3D printers) based, at least partially, on itself.

I’m calling the primary environment (the environment that spawns other environments) Bastion, which is a decent mnemonic for me. So, I need to set up Bastion with many of the same tools to deliver software (other environments) in a repeatable and automated fashion. So, of course, I am using the same base images (that are inherited from) for bastion as I am the environments that are being used by our teams. Again, I’m delivering software in a standard fashion, but the software just happens to be versions of itself (often the team environments will be supersets of the bastion environment).



So some of you might be thinking, “ugh, DevOps in a Box“, which absolutely can be an anti-pattern/idea if my goal was to propagate it as a product. That’s not really what I’m looking at for my organization – basically, in order to promote DevOps, I need a baseline in which we can provide some basics that we can stand up for everything from R&D to fully-delivered projects.

The White Collar Architecture

After looking at what Docker has to offer in terms of the Dockerfile build script, I ran into immediate limitations. Namely, it just wasn’t robust enough – the commands, while they work great, are limited to command line execution via RUN and the like for the most part. Most of the advanced docker instances (e.g. ones that go beyond “look! put your apache html code here!”) wound up running outside shell scripts or other custom scripts.

And while shell is a Good Thing[tm], there’s a few reasons I want more.

  • Capability: I want to be able to run loops, conditionals, templates, and other cool functions that are either outside the capability or “good idea” attributes of other options. Additionally, I want to be able to automate complex scenarios.
  • Flexibility: I need to be able to pull versioned platform configurations, re-use components, and generally be able implement complex configuration steps
  • Accessibility: I’m working with staff that may not be familiar with (bash) shell scripting, and I’m trying to at least keep the number of items to manage and train to a relative minimum.
  • Idempotency: Now someone may call me out on this. Why would a Docker instance need idempotency when we’re working with layers, caching, etc.? Wouldn’t that layer just be thrown out? Well, yes. However, I have a few reasons why I think I need this: inherited containers, and data-only containers. I’ll go into this more below.
  • Standardization: Right now, I’m using Docker containers as much as possible, but what if I wind up deploying this function to a VM or even raw iron? I would have to replicate the same deployment mechanism via other tools, which is at best repetitive and not very DevOpsy. Additionally, I believe I’ve gotten the implementation down to a reasonably “templated” method.

In other words, I need to configure my Docker images with the same powerful software that I use to configure my other containers, regardless of what form they take. I want server immutability regardless of the form my container takes. I decided I wanted Ansible with my Docker. But how do I mix my peanut butter and chocolate? After all, Ansible and Docker have some overlap and I want to make sure I don’t overengineer the task, because bottlenecks are the bane of DevOps.

The Blue Collar Architecture

Ok, let’s get to the gearhead stuff. I’m going to concentrate on configuration of Docker images right now, with some of the greater orchestration aspects in a later post.

First, let’s start with a base image. This is the image that as many of the other images will inherit and incorporates many of the “every image needs this” software or configuration. Some really good examples are universal packages, repo defintions, SSL certs, etc. Do NOT put things like Java in here for a few reasons, include future compatibility and bloating of the image.

Base Image: dockerregistry.local.pagden.ops/pagops.baseline

FROM ubuntu:14.04.3

# Update and upgrade all packages
RUN apt-get update
RUN apt-get dist-upgrade -y

# Install baseline needs for ansible
RUN apt-get install -y software-properties-common
RUN apt-get install -y python-software-properties
RUN apt-get install -y python-apt

# Add Ansible PPA and install
RUN add-apt-repository ppa:ansible/ansible
RUN apt-get update
RUN apt-get install -y ansible

# Set and create the working directory structure
WORKDIR /srv/ansible/workbook

# Copy the necessary security and ansible files
COPY ./ansible/files/hosts /etc/ansible/
COPY ./gitstrap/gitstrap.yml /srv/ansible/gitstrap/
COPY ./ansible /srv/ansible/workbook/

# Run the baseline playbook compendium
RUN ansible-playbook playbook.yml -c local

# Clean up the work directory except for gitstrap, which we will keep around
RUN rm -rf /srv/ansible/workbook/*

######### Delayed Config - this container is never going to be used as stand-alone - always inherited from
# Application Payload
ONBUILD COPY ./ansible /srv/ansible/workbook/

# Set the work directory back for the playbook launch
ONBUILD WORKDIR /srv/ansible/workbook

# Load any packages assigned to this image
ONBUILD RUN ansible-playbook /srv/ansible/gitstrap/gitstrap.yml -c local

# Run the custom playbook
ONBUILD RUN ansible-playbook playbook.yml -c local

# Clean up the work directory
ONBUILD RUN rm -rf /srv/ansible/workbook/*

Ok, what have we done here? Let’s break it down.

FROM ubuntu:14.04.3

I specified a version of Ubuntu which provides me a certain level of expectation of the packages that are coming down, kernel, etc.

# Update and upgrade all packages
RUN apt-get update
RUN apt-get dist-upgrade -y

Go ahead and update the image to the latest packages. Now, of course, you may want to lock in particular package versions.

# Install baseline needs for ansible
RUN apt-get install -y software-properties-common
RUN apt-get install -y python-software-properties
RUN apt-get install -y python-apt

# Add Ansible PPA and install
RUN add-apt-repository ppa:ansible/ansible
RUN apt-get update
RUN apt-get install -y ansible

Ok, this installs Ansible on every Docker image. Yes, I know that provides a little bloat, but it’s essential to this pattern.

# Set and create the working directory structure
WORKDIR /srv/ansible/workbook

# Copy the necessary security and ansible files
COPY ./ansible/files/hosts /etc/ansible/
COPY ./gitstrap/gitstrap.yml /srv/ansible/gitstrap/
COPY ./ansible /srv/ansible/workbook/

Ok, here is where it gets a little interesting. I’ve designated /srv/ansible/ as my work “area” for Ansible.

The hosts file that is copied over includes a singular entry – localhost. We’re running Ansible against the local machine exclusively.

So, what is this gitstrap? Ansible compiles itself before run, so you absolutely need to have the roles that are referenced inside your playbook in place BEFORE you attempt to run your main playbook. You are using roles, right? If not, you probably want to. All the cool kids are doing it. Gitstrap is a short playbook that pulls Ansible roles from git (in my case, Bitbucket) down to the image. This allows me centralized versioning of roles while ensuring a relatively low impact and intuitive (if you know Ansible) implementation for my staff to get what they need, rapidly.

Gitstrap is going to be a “permanent” file in each instance, just so I don’t have to manage versions of the Gitstrap across other implementations (central version control = a Good Theme[tm]). Additionally, gitstrap only runs when there is a gitstrap.yml defined in the vars directory, as provided by the main playbook declaration. In other words, it almost become a baseline function of Ansible.


- hosts: all
  become: yes
  become_method: sudo

    - name: Determine if the vars file exists, and thus to execute the playbook tasks
      stat: path=/srv/ansible/workbook/vars/gitroles.yml
      register: run_gitstrap

    - name: Include the variables for gitstrap dynamically
      include_vars: /srv/ansible/workbook/vars/gitroles.yml
      when: run_gitstrap.stat.exists == True

    - name: Pull the proper roles from Bitbucket
      git: repo={{ item.role_git_location }}
           dest=/srv/ansible/workbook/roles/{{ item.role_name }}
           version={{ item.role_version }}
      with_items: role_list
      when: run_gitstrap.stat.exists == True

Ok, now that we understand what gitstrap does at a base level, let’s finish up our breakdown above.

COPY ./ansible /srv/ansible/workbook/

As part of each of our Docker definitions, we have a subdirectory of Ansible containing our “real” playbook.

# Run the baseline playbook compendium
RUN ansible-playbook playbook.yml -c local

# Clean up the work directory except for gitstrap, which we will keep around
RUN rm -rf /srv/ansible/workbook/*

We now have the execution of our Ansible playbook, the star of the show, a run against the local instance. We’ll dig into this more later. Of course, let’s be tidy and proper and cleanup our working directory.

But wait! We’re not done yet! What about all those ONBUILD statement? What do they do? And why do I have them? Well, again, this instance is a baseline. It won’t be used directly, but rather inherited from when we implement our other Docker containers that are targeted with specific applications. The ONBUILD statements are delayed triggers that execute upon subsequent inherited Docker builds.

Technically, I should create a pagops.baseline-onbuild image, which alerts anyone inheriting downstream that ONBUILD triggers exist and will be executed. I decided against this for simplicity’s sake at this particular tier (as in, I can’t come up with a valid scenario in which I wouldn’t be executing these delayed triggers).

Since we’re not going to instantiate pagops.baseline, what happens when I need pagops.postgres? I’m going to inherit directly (via FROM). The ONBUILD allows me to provide my staff with potentially a very simple Docker file. To that end, I’ll show you exactly what’s in my pagops.postgres Dockerfile.


FROM dockerregistry.local.pagden.ops/pagops.baseline

######### Specific Docker Info
# Set up the entrypoint launch mechanism
COPY ./entrypoint/ / 

# Add environment parameters for proper execution
ENV PATH /usr/lib/postgresql/$PG_MAJOR/bin:$PATH
ENV PGDATA /var/lib/postgresql/$PG_MAJOR/main

# Expose the Postgresql port

# Set the default command to run when starting the container
CMD ["postgres", "-D", "/var/lib/postgresql/9.4/main", "-c", "config_file=/etc/postgresql/9.4/main/postgresql.conf"]

I’ve embedded the execution of where all my REAL commands are – Ansible. Upon building this image, the delayed commands trigger and run my custom playbook. This has the effect of allowing a ton of flexibility, inheritance, central source control, and more narrow definition of the skills that my staff need to know.

If you notice, most of the logic above revolves around this ENTRYPOINT. Proper ENTRYPOINT configuration is pretty key to Docker, so I won’t go into it here. If you have a question around what/why/when, check out the official build Docker entrypoint files for Redis (they do an excellent job and I model mine from theirs).

Note: I could do a significantly better job in making this implementation more flexible by using some variables in the CMD order. The problem is ENV declarations like ${PG_DATA} are processed by Docker, when you need the actual Linux shell to properly substitute them. In other words, you have to implement “echo ${PG_DATA}” in the parameters, because otherwise you get a literal ${PG_DATA} written, and I haven’t quite gotten the syntax down yet.

So, if I have either a very simple image (not very likely, you will probably need at least EXPOSE) or (more likely) an image that inherits from postgres, I will only need the following as my complete Dockerfile.


FROM dockerregistry.local.pagden.ops/pagops.postgres

A famous IT guru weighs in on this implementation here.

So what does my directory look like for these implementations? Excellent question, glad I asked.

drwxrwxr-x  5 admdev admdev 4096 Oct 16 23:42 .
drwxrwxr-x 19 admdev admdev 4096 Oct 26 15:05 ..
drwxrwxr-x  9 admdev admdev 4096 Oct 22 10:16 ansible
-rw-rw-r--  1 admdev admdev 1314 Oct 26 14:32 Dockerfile
drwxrwxr-x  2 admdev admdev 4096 Oct 16 11:30 entrypoint
drwxrwxr-x  8 admdev admdev 4096 Oct 26 14:58 .git

As a good DevOps soldier, I am storing my platform code in git, but you can see how low impact this approach can be by diving into the ansible directory.

drwxrwxr-x 2 admdev admdev 4.0K Oct 16 18:09 defaults
drwxrwxr-x 2 admdev admdev 4.0K Oct 16 15:35 files
drwxrwxr-x 2 admdev admdev 4.0K Oct 16 18:09 handlers
drwxrwxr-x 2 admdev admdev 4.0K Oct 16 18:09 meta
-rw-rw-r-- 1 admdev admdev  485 Oct 23 11:53 playbook.yml
drwxrwxr-x 2 admdev admdev 4.0K Oct 16 18:09 tasks
drwxrwxr-x 2 admdev admdev 4.0K Oct 16 18:09 templates
drwxrwxr-x 2 admdev admdev 4.0K Oct 22 10:16 vars

Looks like a lot, right? Well, it’s not. First, the above directory structure is standard Ansible directory structure, and many of these directories are empty for this particular project since I’m using roles. So, let me delete the empty directories, and see what we get.

├── playbook.yml
└── vars
    ├── gitroles.yml
    └── main.yml

As my IT guru would say, “EXCELLENT!”

Let’s dig into the final couple of files, starting with gitroles:


#### Custom gitstrap packages
# Vars: gitstrap
  - role_name: pagops.postgres
    role_version: master

Boom. Simple. Pull a role, execute it. But, what about variables for my main playbook configuration (you know, the one that pulls and executes the role)? Again, glad I asked. The vars file I specified (main.yml) operates as you would expect, executing the playbook and roles normally. Let’s look at it:


#### Postgresql setup
# Allow access to this instance outside of localhost
  - '*'


postgresql_version: 9.4

Obviously, a simple implementation that more or less opens up my ports the way I need for them to live in a Docker world (since I’m not specifying static IP addresses).


First, I’d like anyone with some deeper knowledge to weigh in here and tell me if this is really necessary to espouse, or if the layer cake of Docker really takes care of this and I stress other points.

So, what’s the deal with me touting Idempotency (especially in the Docker universe)? Well, it’s a little complex, but I plan on using data-only containers. Those containers use images that inherit from each other, so once I get downstream in my inheritance a little, I need to be able to run the same Ansible call against an image that may have already been configured once. Wow, someone needs an example here to properly explain this. That someone is me.

Let’s go back to our postgres implementation, pagops.postgres, and the overall implementation goals (my software RepRap). I will be creating environment postgres instances, or at least potentially configuring the data containers. Let’s pretend I have a project that I’m spinning up an environment for that team – project23.

As part of my base project23 implementation, I’m deploying a Dockered Ansible Tower. As part of that deployment, Ansible Tower will need at least 2 data only containers: 1 for the Ansible Tower config files, and 1 for the postgres data volumes. Since I was having permissions issues with using random images (e.g. busybox) for the data only containers, I located this post. Not only did this make sense from a storage perspective (the VOLUME data, stored separately by the brilliance of Docker, is the only differentiating data – I don’t need to store and configure a separate image), this fixed my permissions issues (and just to be sure, I put logic in the entrypoint).

This now allows me to have an image as a reference image and have another inheriting it (via FROM), and run the same exact Ansible script, changing only the parameters I specify in the vars file for the inherited build. Idempotence to the rescue! I’m running the exact configuration script that built the parent to build the child. Server immutability! Science!

So, as an example, how do I configure a Tower (App) instance for Bastion?

Step 1: Run the Docker build script, writing the configuration to the image. Whoa, you say, what about immutability and docker persistence? Well, that information is going to be written, but overlaid by the volumes we expose in Step 2.

Step 2: Use that image to create a data only container and let’s name it We now instantiate another of the same container (pagops.tower) and bring these two up together via something such as docker-compose. I expose the VOLUMES in that docker compose, and have pagops.tower use the volumes from

(NOTE: Under the penalty of severe time loss, don’t embed a VOLUME command into your dockerfile in these scenarios (inheriting) – when you build, docker says it writes to that layer, but it doesn’t actually write to any directory defined as a VOLUME. This effort took at least 6 weeks off of my life. I hope my grandkids will understand.)

Step 3: Profit.

The run essentially makes the data-only and immutable containers. Rinse and repeat for data only containers for any database you’re using.


I think I’ve made something pretty reasonable here – I can justify the logic behind it, but honestly, I may have also created an anti-pattern that folks point to as as the “To The Pain Pattern” of container deployment/configuration.

Aside from each of the benefits I’ve listed as I went, there are a few others:

  • I’m able to build, run, and initially test my Ansible scripts on another VM, which allows me significantly easier troubleshooting, (some) assurances it will work outside Docker, and quite a bit fewer “moving parts”. I just drop them into the template and I’m typically humming right along.
  • Everything above is stored in git, and additionally, as I do the builds (I’m asking bastion to write out the build scripts, run the builds, and deploy), I will be also storing images in the docker registry as necessary.
  • Of course, many of the concepts can apply to delivering custom software development (and in many circumstances, it will be easier) via LAMP  stacks and the like.
  • Much of this could be adaptable to building out more localized tooling for desktop configuration (for example, a dev VM). I eventually would like the ability to deploy a number of desktop tools to arrive and use of the concepts here.

Of course, I am still working with multiple deployment and management technologies. These are going to be critical as the number of instances proliferate to some pretty significant levels. One idea would be Flocker, which probably will make my data storage container life significantly easier. Additionally Tutum is now part of Docker, so that’s interesting as well.

What are your thoughts? Is embedding of config management into Docker containers with gitstrapping (woo-hoo, I think I created a word) the necessary playbook roles (and potentially full playbooks) a strong way for certain scenarios? I’d like to think so.