Docker

Nginx

Uwsgi

Django

Django — Build an app with Docker and Nginx

Building Django applications can be, sometimes, a little bit hard, not because Django is extremely complex but because some initial configurations might be needed. If you are new using Django, it’s better if you run their tutorial here first and get familiar with it.

This article assumes that you are already familiar with some concepts of the framework and it will only approach some aspects that can be useful while you’re starting a project and allowing you to split the environments as you which. This will cover a specific architecture and design of a Django Web Application using Docker and Nginx. The codebase can be found here for Django ≥ 2.0.

The boilerplates mentioned before come with a lot of stuff that won’t be talked here but it will soon.

Docker

For those who are not familiar with it, Docker is a software that provides operating-system-level virtualization (containers). Docker provides a layer of abstraction and automation of operating-system-level virtualization on Windows and Linux. You can learn more about it here.

There is a more detailed article about docker for people who want to learn a little bit more about this specific technology.

Nginx

Nginx is a web server that can also be used as a reverse proxy, load balancer and HTTP cache used in many different platforms, e.g.: Quora, GoDaddy, Namecheap, UOL, iForce Networks and many others. The reason for these companies to use Nginx comes with its easy implementation and power/performance that comes with it.

In other words, it is awesome! So, let’s start.

When using a Django default start template, the initial structure looks like this:

[projectname]/
├── [projectname]/
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
└── manage.py

This is fine for an initial hello world app but if we want some more complex configurations, we need a little bit more than only this. Using the template provided before and after running django-admin startproject --template=https://github.com/tiagoarasilva/django-boilerplate/archive/master.zip --extension=py,md,html,txt,scss,sass project_name your project will look more like this:

[project_name]/
├── [project_name]/
|   ├── development/
|   ├── ├── __init__.py
|   ├── ├── settings.py
|   ├── staging/
|   ├── ├── __init__.py
|   ├── ├── settings.py
|   ├── testing/
|   ├── ├── __init__.py
|   |── ├── settings.py
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   |── wsgi.py
│   |── databases.py
|   ├── thrid_parties/
|   └── ├── __init__.py
└── manage.py

More files are also cloned from the repository but we will come back to them later.

As you can see above, the Django structure is now more detailed and a little bit more detailed than before. This will allow us to split the settings by the environment for later to be used with Nginx. So let’s break it down a little bit here.

Development

These are all the configurations needed to run in development mode with settings you need in dev (Django extensions, for example) and you don’t want in production, so inside it will be something like this:

from project_name.settings import *
DEBUG = True
DJANGOENV = 'development'
MIDDLEWARE += [
    'debug_toolbar.middleware.DebugToolbarMiddleware'
]
INSTALLED_APPS += [
    'django_nose',
    'django_extensions',
    'debug_toolbar',
    'template_repl',
]

Above is an example of unique development settings that we don’t want to run in production so we load the main base settings and extend some existing properties (MIDDLEWARE and INSTALLED_APPS) allowing us to add specific development only apps.

DJANGOENV is an environment variable passed to Django through docker-compose, something we will talk later on this article.

Staging and Testing

These modules have the same purpose as development but for staging and testing environments, respectively, this will allow us to pass specific settings to every single environment without overloading the main settings.py file and allowing us to work with them individually. Pretty cool, hein? Also, pretty standard nowadays.

Databases

Django by default, inside the settings, provides a database connection to SQLite, but what if we need more than one database running at the same time? Well we need to add it there anyway, so instead of putting all inside the same main settings, let’s also isolate these inside its ow file and it will look something like this:

import os
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
BD_NON_DEFAULT = u'db_non_default'
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
    },
    BD_NON_DEFAULT: {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'postgres',
        'USER': 'postgres',
        'PASSWORD': 'postgres',
        'HOST': 'postgres',
        'PORT': '5432',
    }
}

This way, we can have multiple databases in one file and call it from the main settings, like this:

from project_name.databases import *

Many good settings were provided with this template (Redis is one of them) but if talk about all of them here, it will lose the purpose of this article. That can be explained in a different post in more detail.

So far we talked about how to approach the Django settings allowing us running different environments based on those same settings. Let’s now understand how does this connect with Docker and Nginx.

Docker-compose

The root directory contains a docker-compose.yml file that looks like this:

version: '2'
services:
  db:
    restart: always
    image: postgres:9.6.0
    expose:
      - "5432"
    volumes:
      - "{{ project_name }}_db_data:/var/lib/postgresql/data"
    ports:
      - "5432:5432"
  redis:
    restart: always
    image: redis:latest
    expose:
      - "6379"
  {{ project_name }}:
    restart: always
    image: oratio/kale:1.0.0
    depends_on:
      - redis
    environment:
      DJANGOENV: development
      ENVIRONMENT: development
      PYTHON: python
      PROJECT_NAME: {{ project_name }}
      TERM: xterm
      ROLE: development
    links:
      - db:postgres
      - redis:redis
    ports:
      - "80:80"
      - "443:443"
      - "8000:8000"
    expose:
      - "80"
      - "443"
      - "8000"
    volumes:
      - .:/var/www
    working_dir: /var/www
    command: bash -lc "pip install invoke jinja2 && invoke -r roles $${ROLE}"
volumes:
  {{ project_name }}_db_data:
    external: true

This looks very complex when you have a first look but if not, let’s see what it looks like this.

As you are probably familiar, Docker allows us to run many settings and set up as many containers as we need, the above example is one of those many containers that will be generated once you run the first docker-compose up. So what we are saying here is that the project will have a Postgres database, Redis and finally our main Django project.

Because we need to persist the data of our Postgres database externally, we use docker volumes and to make it happen we just declare:

volumes:
  {{ project_name }}_db_data:
    external: true

And then we pass the setting to the container, like this:

db:
    restart: always
    image: postgres:9.6.0
    expose:
      - "5432"
    volumes:
      - "{{ project_name }}_db_data:/var/lib/postgresql/data"
    ports:
      - "5432:5432"

The main project container

So what is this? We need to tell our docker container what does it need to run.

{{ project_name }}:
    restart: always
    image: oratio/kale:1.0.0
    depends_on:
      - redis
    environment:
      DJANGOENV: development
      ENVIRONMENT: development
      PYTHON: python
      PROJECT_NAME: {{ project_name }}
      TERM: xterm
      ROLE: development
    links:
      - db:postgres
      - redis:redis
    ports:
      - "80:80"
      - "443:443"
      - "8000:8000"
    expose:
      - "80"
      - "443"
      - "8000"
    volumes:
      - .:/var/www
    working_dir: /var/www
    command: bash -lc "pip install invoke jinja2 && invoke -r roles $${ROLE}"

We set the ports we need to expose the container to “the outside world”, the image needed to build the container, the dependencies before starting, the links to other containers and other settings we may or may not use, it’s up to you to decide.

Do you remember before when we declared the DJANGOENV in our settings file (development, staging, testing…)? Well, this is why, here we are saying to docker, “once you start running, set some environment variables for me”. One of those variables is DJANGOENV.

These settings are important for many purposes, so let’s talk about them and why we need them to make the project work with Nginx.

Environment Variables

environment:
      DJANGOENV: development
      ENVIRONMENT: development
      PYTHON: python3
      PROJECT_NAME: {{ project_name }}
      TERM: xterm
      ROLE: development

Makefile — Unix/Linux based file that allows us to pass some instructions and automate some tasks. Instead of running every single command individually, this can be simply automated with the make command. Think about this as an alias, but running on steroids!

Environment — This is used by our, also generated, Makefile. Inside of it, we have some instructions that depend on the value of this variable. Here we pass the value of the environment we want to run (development, testing, staging…) and using the make command, it will automatically catch the value and run specific instructions.

Python — This is to specify which version of python we want to run (in this case, python 3.6+)

Project name — Maybe one of the most important variables here. Why? Because we need the make instruction to know where should it get the settings file, which is the name you gave to your project.

Role — This is something that we use to call some tasks using the invoke instructions when the project is starting, according to the value passed. That role is then used in this command:

command: bash -lc "pip install invoke jinja2 && invoke -r roles $${ROLE}"

There is also a folder called roles, that folder contains a file called tasks and every task name matches the value passed by the environment variable. Quite complex here, right? Yes, but this is needed to make it work. Let’s see for the role “development” what does the task do.

tasks.py

The invoke command requires by default a file called tasks, that file contains a set of instructions that can be executed accordingly.

import os
import sys
from jinja2 import Environment, FileSystemLoader
from invoke import run, task

def _template_file(template, destination, template_dir='/var/www/deploy/'):
    environment = Environment(loader=FileSystemLoader(template_dir))
    template = environment.get_template(template)
    rendered_template = template.render(os.environ)
    if os.path.isfile(destination):
        os.unlink(destination)
    with open(destination, 'w') as f:
        f.write(rendered_template)

@task
def development():
    run('pip3 install -U pip')
    run('pip3 install -r requirements/development.txt')
    _template_file('nginx/nginx.conf', '/etc/nginx/sites-enabled/default')
    _template_file('supervisor.nginx.conf', '/etc/supervisor/conf.d/nginx.conf')
    run('supervisord -n')

So, for development, what we are saying is:
- Upgrade pip3
- Install the requirements needed for development
- Load Nginx configuration file from the deploy folder (the file does not exist in this boilerplate as this allow you to build your configuration file but we are going to create one here as an example) and move it to /etc/nginx/sites-enabled/default
- Load supervisor Nginx configuration and move also to the directory
- Start supervisor

There are more instructions but you can add or remove as you wish, in this case, this will allow us to start Nginx as we wish.

Makefile

We talked about the Makefile and make instructions but without actually applying those concepts, it can be confusing, so the Makefile looks like this:

clean_pyc:
   find . -type f -name "*.pyc" -delete || true
migrate:
   python3 $(PROJECT_NAME)/manage.py migrate --noinput
migrations:
   python3 $(PROJECT_NAME)/manage.py makemigrations
run:
   python3 $(PROJECT_NAME)/manage.py runserver_plus 0.0.0.0:8000 --settings=$(PROJECT_NAME).$(ENVIRONMENT).settings
shell:
   python3 $(PROJECT_NAME)/manage.py shell_plus --settings=$(PROJECT_NAME).$(ENVIRONMENT).settings
show_urls:
   python3 $(PROJECT_NAME)/manage.py show_urls --settings=$(PROJECT_NAME).$(ENVIRONMENT).settings
validate_templates:
   python3 $(PROJECT_NAME)/manage.py validate_templates --settings=$(PROJECT_NAME).$(ENVIRONMENT).settings

The address 0.0.0.0:8000 after setting up Nginx, should be changed to 127.0.0.1:8000 and as we can see, we are using a lot those env variables set by docker.

Nginx configuration

As mentioned before, the boilerplate doesn’t bring any Nginx config file as you can build your own with your settings and I didn’t want to force you using my own. So, let’s build one for this purpose that it’s going to work with the example.

{% if ENVIRONMENT == 'development' -%}
    {% set subdomain = 'dev.' -%}
{% elif ENVIRONMENT == 'testing' -%}
    {% set subdomain = 'testing.' -%}
{% elif ENVIRONMENT == 'staging' -%}
    {% set subdomain = 'staging.' -%}
{% elif ENVIRONMENT == 'live' -%}
    {% set subdomain = '' -%}
{% endif -%}

server {
    server_name {{ subdomain }}project.name.com;
    error_log /dev/stderr;
    {% if ENVIRONMENT not in ('live', 'staging') -%}
    access_log /dev/stdout;
    {% else -%}
    access_log /dev/stdout json_format;
    {% endif -%}
    location /crossdomain.xml {
        return 200;
    }
    location /healthcheck/ {
        include uwsgi_params;
        uwsgi_param HTTP_HOST www.{{ subdomain }}project.name.com;
        uwsgi_param HTTP_X_FORWARDED_PROTO 'https';
        uwsgi_pass unix:/var/www/uwsgi.sock;
    }
    location / {
        return 301 https://www.{{ subdomain }}project.name.com$request_uri;
    }
}

server {
    server_name www.{{ subdomain }}project.name.com ;
    large_client_header_buffers 4 512k;
    error_log /dev/stderr;
    {% if ENVIRONMENT not in ('live', 'staging') -%}
    access_log /dev/stdout;
    {% else -%}
    access_log /dev/stdout json_format;
    {% endif -%}
    client_max_body_size 100M;
    {% if ENVIRONMENT != 'development' -%}
    real_ip_header X-Forwarded-For;
    set_real_ip_from 0.0.0.0/0;
    {% else -%}
    add_header Host $host;
    add_header X-Forwarded-For $http_x_forwarded_for;
    add_header X-Forwarded-Proto $scheme;
    add_header X-CSS-Protection "1; mode=block";
    {% endif -%}
    sendfile off;
    location /favicon.ico {
        alias /var/www/projet_name/static/foundation6/img/favicon.png;
    }
    location /error.html {
        alias /var/www/projet_name/templates/errors/500.html;
    }
    location /fonts {
      alias /var/www/projet_name/static/foundation6/fonts;
    }
    {% if ENVIRONMENT != 'development' -%}
    error_page 500 501 502 503 504 =200 /err.or.html;
    {% endif -%}
    {% if ENVIRONMENT == 'development' -%}
    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_read_timeout 300s;
        proxy_set_header Host $host;
    }
    {% else -%}
    location / {
        include uwsgi_params;
        uwsgi_pass unix:/var/www/uwsgi.sock;
        uwsgi_read_timeout 300s;
        {% if ENVIRONMENT != 'live' -%}
        satisfy any;
        deny all;
        auth_basic "Password Protected Access";
        auth_basic_user_file /var/www/deploy/htpasswd;
        {% endif -%}
    }
    {% endif -%}
}

The codebase uses foundation 6 as the HTML/JS framework but that can be changed by you as you wish.

Well this is a big file and it looks very complicated but again, it’s not. Do you remember the environment variable called “ENVIRONMENT” and it was supposed to be used in multiple places? This is one of them, the file looks huge because this uses Jinja as a template system, which means, we want to run different settings based on some conditions, in this case, different environments.

If you see carefully, I’ve created a bunch of conditions that will allow us to run this one file in multiple environments without building different Nginx configs for every single one of them.

If you want to add uwsgi, we could also set up a setting file for this project like this:

[uwsgi]
# Django-related settings
# Enable threads
enable-threads  = true
single-interpreter=true
wsgi-file       = /var/www/app.wsgi
touch-reload    = /var/www/app.wsgi
touch-logreopen = /var/www/app.wsgi
chown-socket    = www-data:www-data
master          = true
workers         = 4
;processes       = 10
socket          = /var/www/uwsgi.sock
chmod-socket    = 664
harakiri        = 300
harakiri-verbose
max-requests    = 300
log-zero
log-slow
log-500
log-x-forwarded-for
drop-after-apps
buffer-size     = 32768
vacuum          = true

Followed by an app.wsgi like this:

#!/usr/bin/env python
import os
import site
import sys
import _strptime
prev_sys_path = list(sys.path)
site.addsitedir(os.path.abspath('/var/www'))
site.addsitedir(os.path.abspath('/var/www/project_name'))
new_sys_path = []
for item in list(sys.path):
    if item not in prev_sys_path:
        new_sys_path.append(item)
        sys.path.remove(item)
sys.path[:0] = new_sys_path
os.environ["DJANGO_SETTINGS_MODULE"] = "project_name.{{ ENVIRONMENT }}.settings"
from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()

Django by default gives you a wsgi.py file but this way, we make sure that everything is working as we need/want and again, the ENVIRONMENT variable passed, matches the settings module initially configured and starts the corresponding environment accordingly.

How to call all of this? Do you remember the tasks file? Well, we can create a new task for an environment at your choice, let’s use staging as an example:

@task
def staging():
    run('pip3 install -U pip')
    run('pip3 install -r requirements/common.txt')
    _template_file('nginx/nginx.conf', '/etc/nginx/sites-enabled/default')
    _template_file('uwsgi/app.wsgi', '/var/www/app.wsgi')
    _template_file('supervisor.nginx.conf', '/etc/supervisor/conf.d/nginx.conf')
    _template_file('supervisor.uwsgi.conf', '/etc/supervisor/conf.d/uwsgi.conf')
    _template_file('uwsgi/uwsgi.ini', '/var/www/uwsgi.ini')
    run('supervisord -n')

So we have a set of instructions (such as development) but with some additional files to be loaded once the supervisor kicks in. In this particular case, I’ve saved the uwsgi configurations inside the deploy/uwsgi/ and the Nginx inside deploy/nginx/.

Do we need uwsgi in dev? Well, no but you can use it anyway but I don’t recommend it and the reason for that it’s because it’s dev, so why complicate it?

I manually do the make run command (inside the container) and it will trigger what I need (it will start the server). Since everything is set and the project is up and running with Nginx, we can now type in a browser, for example, www.project.name.com (as we set in nginx.conf before) without using the specific 8000 port and Nginx is already mapping all of that for us.

By the way, once you create your “server” in your Nginx configuration, you need to change your hosts file for something like this:

127.0.0.1 localhost
127.0.0.1 dev.project.name.com
127.0.0.1 www.dev.project.name.com

Otherwise, you won’t be able to see anything.

And that’s it. I apologize for the length of this article but I’ve tried to be as shorter as I could without going into much detail. There were many other subjects that I could have talked such as Redis or even more details regarding uWSGI configurations but that it’s for another time.

I hope I could provide some insight on how to build a Django app splitting the settings, with docker and Nginx in a very simple way.

I would like to thanks all the DevOps who taught me all of this and gave me the insight that I need to build the boilerplate and share it with the community.