Docker — An autopsy
We are currently in a world where the container system became a natural choice for most of the companies who want to keep up with the latest and start-of-the-art technologies. Docker is one of those and it became a natural choice for DevOps and developers to work with. This article will talk about it but not extensively as there are a lot of “small things”, important, but not relevant for this topic.
When you first start with it, the learning curve can be tough, even frustrating but once you get it, you probably don’t want anything else but that. In my previous article, I gave a quick insight on how to build a Django app splitting the settings by environment and integrate with Docker and Nginx. Here I’ll be more specific and use the same boilerplate (you can get it here), I’ll share some of the things that I’ve learned, building a docker image and talking about:
- Dockerfile
- Docker compose
- Docker commands
- Docker images
- Docker tags
- Docker containers
- Docker volumes
Docker
Docker is a software that provides operating-system-level virtualization (containers). Docker provides a layer of abstraction and automation of operating-system-level virtualization on Windows and Linux. You can learn more about it here.
Dockerfile
A Dockerfile is a set of instructions that will allow you to build a docker image. In other words, the only way of building a docker image is through this. Lets have a look at a Dockerfile and try to understand the meaning.
############################################################
# Dockerfile to run a Django-based web application
# Based on an Ubuntu Image
############################################################
FROM ubuntu:16.04
MAINTAINER <your-email-here>
ENV DEBIAN_FRONTEND noninteractive
ENV LANG en_GB.UTF-8
ENV LC_ALL en_GB.UTF-8
#
# Install Apt Packages
#
RUN apt-get update && \
apt-get upgrade -y && \
apt-get install curl -y && \
apt-get clean && apt-get autoclean && \
find /var/lib/apt/lists/ -type f -delete && \
echo "deb http://apt.postgresql.org/pub/repos/apt/ trusty-pgdg main" > /etc/apt/sources.list.d/pgdg.list && \
curl https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -RUN apt-get update && \
apt-get install -y bind9-host \
curl \
geoip-bin \
gettext \
git-core \
gawk \
imagemagick \
iputils-ping \
language-pack-en \
less \
libcurl4-openssl-dev \
libffi-dev \
libgeos-c1v5 \
libmagickwand-dev \
libmemcached-tools \
libxml2-dev \
libxslt-dev \
memcached \
net-tools \
nginx-extras \
perl \
pgbouncer \
postgresql-client-9.4 \
postgresql-server-dev-9.4 \
python-imaging \
python-chardet \
python-colorama \
python-distlib \
python-html5lib \
python-pip \
python-requests \
python-setuptools \
python-six \
python-urllib3 \
python-dev \
rsyslog \
socat \
software-properties-common \
sudo \
supervisor \
gunicorn \
telnet \
unattended-upgrades \
unzip \
vim \
wget && \
apt-get clean && apt-get autoclean && \
find /var/lib/apt/lists/ -type f -deleteRUN apt-get -y upgrade
RUN add-apt-repository -y ppa:jonathonf/python-3.6 && \
apt-get update && apt-get install -y python3.6
RUN apt-get install -y build-essential python3.6 python3.6-dev python3-pip python3.6-venv
# update pip
RUN python3.6 -m pip install pip --upgrade
RUN python3.6 -m pip install wheel
#
# Install Pip Requirements
# upgrade the setuptools from 27.1.2 to 32.3.1.
#
RUN pip3 install -U setuptools
ADD requirements /var/www/requirements
RUN pip3 install -r /var/www/requirements/common.txt
#
# Make Python3 the default python
# Make pip3 the default pip
#
RUN echo 'alias python=python3' >> ~/.bashrc && echo 'alias pip=pip3.6' >> ~/.bashrc
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.5 1
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.6 2
RUN sed -i -e 's/# server_tokens off;/server_tokens off;/g' /etc/nginx/nginx.conf
This is a “big” file and looks extremely confusing, right? Well, no. Docker has a set of directives, like RUN, ADD, COPY… allowing you to give detailed instructions to docker builder during the process. So, The first lines:
FROM ubuntu:16.04
MAINTAINER tiago.arasilva@goutlook.com
ENV DEBIAN_FRONTEND noninteractive
ENV LANG en_GB.UTF-8
ENV LC_ALL en_GB.UTF-8We are telling docker from (FROM) which version of ubuntu we want to build the image as well as some environment variables (ENV), in this case, I’ve chosen ubuntu 16.04 as I needed to be like that but for different uses, you can pick other versions or even use other previous built docker images or, if you want to, build a specific docker python image, the source (FROM) can be the latest python.
You can find some very useful images for different purposes (Postgres, MySQL, python, java…) on docker hub.
#
# Install Apt Packages
#
RUN apt-get update && \
apt-get upgrade -y && \
apt-get install curl -y && \
apt-get clean && apt-get autoclean && \
find /var/lib/apt/lists/ -type f -delete && \
echo "deb http://apt.postgresql.org/pub/repos/apt/ trusty-pgdg main" > /etc/apt/sources.list.d/pgdg.list && \
curl https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
RUN apt-get update && \
apt-get install -y bind9-host \
curl \
geoip-bin \
gettext \
git-core \
gawk \
imagemagick \
iputils-ping \
language-pack-en \
less \
libcurl4-openssl-dev \
libffi-dev \
libgeos-c1v5 \
libmagickwand-dev \
libmemcached-tools \
libxml2-dev \
libxslt-dev \
memcached \
net-tools \
nginx-extras \
perl \
pgbouncer \
postgresql-client-9.4 \
postgresql-server-dev-9.4 \
python-imaging \
python-chardet \
python-colorama \
python-distlib \
python-html5lib \
python-pip \
python-requests \
python-setuptools \
python-six \
python-urllib3 \
python-dev \
rsyslog \
socat \
software-properties-common \
sudo \
supervisor \
gunicorn \
telnet \
unattended-upgrades \
unzip \
vim \
wget && \
apt-get clean && apt-get autoclean && \
find /var/lib/apt/lists/ -type f -delete
RUN apt-get -y upgradeSince we are using an Ubuntu-based source, we can treat him exactly like that, an Ubuntu OS with the same commands as the normal Ubuntu installation. What I’m saying above is something like this, “Hey, image! Run the following commands (RUN), install what I need inside and then compile!”.
What you are seeing is just a bunch of packages that I need for my development environment, so, avoid installing unnecessary packages just because “is nice to have”.
You don’t need to install a text editor inside a database image, right? That’s why!
RUN add-apt-repository -y ppa:jonathonf/python-3.6 && \
apt-get update && apt-get install -y python3.6
RUN apt-get install -y build-essential python3.6 python3.6-dev python3-pip python3.6-venv
# update pip
RUN python3.6 -m pip install pip --upgrade
RUN python3.6 -m pip install wheel
#
# Install Pip Requirements
# upgrade the setuptools from 27.1.2 to 32.3.1.
#
RUN pip3 install -U setuptools
ADD requirements /var/www/requirements
RUN pip3 install -r /var/www/requirements/common.txt
#
# Make Python3 the default python
# Make pip3 the default pip
#
RUN echo 'alias python=python3' >> ~/.bashrc && echo 'alias pip=pip3.6' >> ~/.bashrc
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.5 1
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.6 2Again, more instructions for the docker builder to run.
I need to install python 3 (latest version of 3.6) inside this image as my personal projects are now in python 3 and Ubuntu, brings python 2.7 as the default python.
So, I’m telling to the builder to update to the latest pip and then install all the python requirements that I need to initially start and run the docker container (see the project folder structure to understand the file locations).
After setting up the instructions, you just need to run:
docker build -t <name-you-want-to-give-to-your-image> .By default, docker builder when running this command will look for a Dockerfile (with this name) in the same folder structure level that you have run the command and execute it.
“Can we run the command but instead of using the default Dockerfile, we use a Dockerfile-different-name?”.
Of course, you only need to run:
docker build -t <name-you-want-to-give-to-your-image> -f <other-dockerfile-custom-with-different-name>The builder also looks for a specific name if you want to override the default (Dockerfile), that means, if you want to run a different set of instructions for a specific scenario, docker looks for a file called “Dockerfile.override” and ignores the default.
Using this Dockerfile.override means that you don’t need to pass it through the -t, meaning, you only need to run (again and simply):
docker build -t <name-you-want-to-give-to-your-image> .And docker will use the Dockerfile.override, automatically.
Docker compose
Compose is a tool for defining and running multi-container Docker applications
What does that mean? Means that you also have a file with a set of images and instructions that will tell docker, “Once you start, create these containers based on these images defined by me”.
version: '2'
services:
db:
restart: always
image: postgres:9.6.0
expose:
- "5432"
volumes:
- "tutorial_db_data:/var/lib/postgresql/data"
ports:
- "5432:5432"
redis:
restart: always
image: redis:latest
expose:
- "6379"
tutorial:
restart: always
image: oratio/kale:1.0.0
depends_on:
- redis
environment:
DJANGOENV: development
ENVIRONMENT: development
PYTHON: python
PROJECT_NAME: tutorial
TERM: xterm
ROLE: development
links:
- db:postgres
- redis:redis
ports:
- "80:80"
- "443:443"
- "8000:8000"
expose:
- "80"
- "443"
- "8000"
volumes:
- .:/var/www
- ~/.aws/config:/root/.aws/config
working_dir: /var/www
command: bash -lc "pip install invoke jinja2 && invoke -r roles $${ROLE}"
volumes:
tutorial_db_data:
external: trueThis is a YAML file with the instructions about the containers you need to be created/used once the docker-compose is up and running.
YAML is a human friendly data serialization standard for all programming languages.
As I’ve explained before with more details, above we define what we need for every container to work (Environment variables, volumes, dependencies…).
There is a specific instruction (restart: always) that can be removed. Why? Because you can have problems starting up the compose and with this instruction, you are telling docker to restart every time even if during the process exists constantly. Bad stuff!
So, once we start the compose, docker will create the following containers:
db - The container for PostGres database data pointing for the external volume tutorial_db_data (persisting the data there!)
redis - The container for redis system
tutorial - The container for the application (following the previous article)You can add as many containers as you need for your project, for instance, you can add a Mysql container to your project in parallel with Postgres.
Do you need it? Maybe not, but you can!
Imagine now that you need to add a messaging queuing system, such as RabbitMQ (I’m biased here as I love RabbitMQ for the simplicity of the implementation). You only need to add to your docker-compose.yml something like this (with or without the restart instruction):
rabbitmq:
restart: always
image: rabbitmq:<version-of-your-choice>
expose:
- "5672"And add the link to your project container, like this:
tutorial:
restart: always
image: oratio/kale:1.0.0
depends_on:
- redis
- rabbitmq
links:
...
- rabbitmq:rabbitmqAnd that’s it! Docker will create a rabbitmq container and tell your tutorial container that it also depends on rabbit to start! Amazingly simple.
Can you imagine installing every single application you need in your local system, constantly? Thanks to Docker, that can be “containerized” avoiding those local and sometimes painful installations, unifying everything in containers! Thank you docker!
Docker commands
One of the many problems that I found with Docker was about knowing the commands, those weird names with weird instructions running weird stuff! Just kidding, none of it is weird but you got the idea!
So, let’s talk about some useful commands (not all of then, you can find some documentation here).
docker images
This is the command that will list the docker images in your system. Let’s see how it looks.
REPOSITORY TAG IMAGE ID CREATED SIZE
oratio/kale 1.0.0 43d91ded2308 5 weeks ago 1.92GB
ubuntu 16.04 00fd29ccc6f1 7 weeks ago 111MB
redis latest e5f9a9e182f4 2 months ago 107MB
rabbitmq 3.6.12 7f2c6f8376d9 3 months ago 124MB
postgres 9.6.0 bff88803c2df 15 months ago 265MBAbove, there is a list of all the images downloaded and used in your local system, for instance, the ubuntu image was used to build the titan used in the codebase provided. The others are the one used to create the project containers (which I will talk later about it).
Docker images (not the command)
So, what is this of a docker image? Is it an ISO file? Well, no, but you can actually compare them as the concept is quite similar. So, an image is basically a compilation of all the resources you need (operating system, packages, installers, source code…) “compiled” in one single place and once it’s done, it cannot be changed unless you create a new version. To do that, you need a file called Dockerfile, it is like a Makefile in Linux/Unix but for Docker.
Docker image tags
Building an image it’s quite straightforward, it’s just a simple command that will compile the Dockerfile (mentioned at the beginning of the article) instructions assembling an image to be used by you, or if you like helping other, also for them. Let’s use the code base as an example:
The command:
docker build -t oratio/kale:1.0.0 .The previous command is just telling something like this, “hey docker, compile my instructions from my Dockerfile and call oratio/kale:1.0.0 with a version of 1.0.0. This version is the tag. If I don’t specify any tag, by default, docker will call it, latest and becoming:
docker build -t oratio/kale:latestSome companies like to have the latest tag but I, personally, don’t, and the reason for that it’s because I like to maintain certain version that I might use in the future for different projects and purposes.
Is this enough? Yes and no. Yes if you only want the image locally and don’t want to use in any other place but your local machine and no for the opposite reason, if you want to share with someone, everyone or even making a “security copy” you should upload to a hub that supports docker images. Docker, by default, has something like that and the free account allows you to have multiple public repositories and only one private, but if you are not building something crucial to your business and only want to have the image somewhere where you can access it, this should be more than enough, for instance, I have multiple docker images for many different use cases. Check it here.
So, that means, “I build an image and I want to share with everyone and that’s it?”. No, you need to do just one more step, pushing into docker hub (Docker, AWS ECR…). How? Image this is a git repository, the way you do it, it’s very similar, just like this:
docker push oratio/kale:1.0.0Of course, to do this you need to login via command line, or via docker desktop application. To use it via command line, just run:
λ docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username (tiagoarasilva):Follow the instructions and that’s it, it’s up and running! Pretty cool, hein!?
Docker containers
So, Dockerfile, docker images, tags, what now? Well, now it’s just the containers. And what is that? Well, basically, do you remember building the previous image in order to be used in certain projects, right? Essentially, a docker container is an instance of that image, that means something like this, “Hey you, docker image, I need you, I really would like to use you in my project and include you there”, so what docker does is something like, “oh, ok then, so in order to do that, I will create a container for you then”. Just this, quite simple. After the container is created (I don’t need to show you the log as that varies from project to project) you are more than able to use it.
“If is this simple, how can I use it? Or even, check how many container do I have in my system, just to make sure that I don’t have unnecessary container using system resources”
To answer that question, docker gives us another command to check the existing containers in our system.
docker ps or docker ps -aThis will give us a list of what we have and what’s being used at the moment, something that looks like this:
CONTAINER ID IMAGE
6624a276ce82 oratio/kale:1.0.0
87ac8f25a95f postgres:9.6
0ba42b084774 redis:latestWell, it gives us more information like COMMAND, CREATED, STATUS, PORTS, and NAMES but because we don’t have enough space here, let’s pick up what I consider the most important one, the CONTAINER ID.
Why is this so important? Well, let’s imagine you need to remove the container, you need the ID to do it (and almost everything works with IDs nowadays, really).
docker rm <CONTAINER ID>Or if you need to force it
docker rm <CONTAINER ID> --forceSo, what we are basically saying is, “docker, please remove my container with the ID XXXXXXXXXX”. The ID also helps you with other tools like swarms and monitoring. I’m not saying the rest is not important but for the sake of this article, let’s just focus on the simple ones.
And that’s it, this way not only you know how to list the existing container as well as removing some unnecessary containers if you need too.
Docker volumes
The last but not the least, what is this of a docker volume? Actually, in my opinion, it’s the simplest tool from docker, so think like this, if you want to have a DB and persist some data, if you remove a container that means you will lose your data, right? Yes, so how can we avoid this from happening? Using docker volumes.
So a docker volume is not more than a way of persisting the data from your containers in your local machine. Like docker containers, you have commands to list and remove the volumes from the system, the difference between containers and volumes is that instead of using the ID to remove, you use the VOLUME NAME.
List:
docker volume lsRemove:
docker volume rm <VOLUME NAME>And that’s it. I didn’t want to go in too much detail as docker is a whole wold of things and stuff but I hope, at least, I could help clarifying and make it simpler the way you can use and work with docker.
