Docker container are becoming more and more popular over time. It's a really simple way to encapsulate your application code with its dependencies and leave it ready to be deployed into different environments.

One of its key features are that it allows you to test your application together with the operating system that will be used to deploy it. The implication of this is that your tests will not only validate that your app updates don't break your tests, but also make quite simple to upgrade the operating system dependencies and validate it with regular app tests.

Let's say we have a Python project which uses PostgreSQL with this set of dependencies in a requirements.txt file:

psycopg2==2.8.5
requests==2.23.0

A simple Dockerfile for that app would be:

FROM python:3.8-slim

WORKDIR /app

RUN apt-get update && apt-get install -y --allow-unauthenticated \
    build-essential \
    libpq-dev

COPY run_app.sh run_app.sh
COPY my_app.py my_app.py
COPY requirements.txt requirements.txt

# Install app dependencies.
RUN pip install -r requirements.txt

CMD ["/app/run_app.sh"]

Which would produce an image size of 357MB starting from a base Python 3.8 image of 113MB. That means that each deployment needs to download 357MB from your docker registry to your production environment. Imagine how big would it become as your dependencies grow, specially when you start using external libraries or modules that need to be compiled, requiring dev libraries to create the container.

Is there anything we could do to reduce that size?

Yeah, and since the multi stage functionality addition in Docker is quite simple!

The way to approach this optimisation is to use one container only for the build and dependencies installation and then, in a fresh and new container, just copy the files you are interested on.

The way you implement this split depends on the technologies being used to run the service, in our case for this article, we use a Python based application, so we need a way to easily copy the installed dependencies. There are two different ways to achieve it: * Use pip to install all dependencies into an specific directory. * Use a virtualenv so the whole installation is isolated, included bin scripts.

The approach used on this article is the virtualenv one, but it's just a personal preference for simplicity.

A possible implementation would have first, a base stage with the common rules reused in different stages:

FROM python:3.8-slim as base

# PYTHONUNBUFFERED: https://docs.python.org/3.8/using/cmdline.html#envvar-PYTHONUNBUFFERED
ENV PYTHONUNBUFFERED 1
ENV VIRTUAL_ENV /srv/venv
ENV PYTHONDONTWRITEBYTECODE 1

WORKDIR /app

RUN pip install virtualenv

RUN apt-get update && apt-get install -y --allow-unauthenticated \
    libpq5

It basically initialises the environment, sets the final workdir, installs the virtualenv package that will used in the following stages and finally installs library dependencies required by our app, in this case, libpq5 for PostgreSQL.

Next stage would be used only for the build, it will include all development dependencies. It will start from previous base stage:

FROM base as compile-image

RUN apt-get update && apt-get install -y --allow-unauthenticated \
    build-essential \
    libpq-dev && \
    python -m virtualenv ${VIRTUAL_ENV}

ENV PATH "${VIRTUAL_ENV}/bin:${PATH}"

# Install app dependencies.
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt

To install the runtime dependencies for our app, it creates a virtualenv, which is used later to install the requirements.txt dependencies with pip. To simplify the python command execution, the PATH environment variable is updated to use the virtual env's commands by default.

Finally, is time to create the final image with just our application code and the required final libraries:

FROM base as build-image
ENV PATH "${VIRTUAL_ENV}/bin:${PATH}"

COPY --from=compile-image ${VIRTUAL_ENV} ${VIRTUAL_ENV}
COPY run_app.sh run_app.sh
COPY my_app.py my_app.py

CMD ["/app/run_app.sh"]

As you can see, this stage is copying the virtual environment we created in the previous stage which would have the final installed modules required to run our app and the code for our app. The postgres dependency requires libpq library to work, but we already have it installed from the base image, which allows us to share that installation between the dependencies installation stage and this final container stage.

Once this last stage is built, we end having a container of 157MB which is near 2.30 times smaller than the original container.