Artem Udovyk

Search

SearchSearch

Docker Compose

Jun 17, 202414 min read

Docker Compose is a tool for defining and running multi-container applications.

Compose simplifies the control of your entire application stack, making it easy to manage services, networks, and volumes in a single, comprehensible YAML configuration file. Then, with a single command, you create and start all the services from your configuration file.

Compose works in all environments; production, staging, development, testing, as well as CI workflows.

Compose can reduce a multi-page “developer getting started guide” to a single machine-readable Compose file and a few commands.

docker compose up

Why

  • Simplifies orchestrating and coordinating multiple services of your application by defining the entire stack in a single YAML file.
  • Configuration files are easy to share and collaborate on.
  • Docker Compose reuses the containers for services that weren’t changed. It allows to quickly iterate on the application.
  • Multiple environment can have each it own set of input variables.
  • Community.

Use cases

  • Development environments
  • Automated testing environments
  • Single host deployments

Install

https://docs.docker.com/compose/install/

Compose file

The default path for a Compose file is compose.yaml (preferred) or compose.yml that is placed in the working directory. Compose also supports docker-compose.yaml and docker-compose.yml for backwards compatibility of earlier versions. If both files exist, Compose prefers the canonical compose.yaml.

name: myapp
 
services:
  foo:
    image: busybox
    command: echo "I'm running ${COMPOSE_PROJECT_NAME}"

The compose.yaml file follows the rules provided by the Compose Specification in how to define multi-container applications. This is the Docker Compose implementation of the formal Compose Specification.

Compose application model

Computing components of an application are defined as services. A service is an abstract concept implemented on platforms by running the same container image, and configuration, one or more times.

Services communicate with each other through networks. In the Compose Specification, a network is a platform capability abstraction to establish an IP route between containers within services connected together.

Services store and share persistent data into volumes. The Specification describes such a persistent data as a high-level filesystem mount with global options.

Some services require configuration data that is dependent on the runtime or platform. For this, the Specification defines a dedicated configs concept. From a service container point of view, configs are comparable to volumes, in that they are files mounted into the container. But the actual definition involves distinct platform resources and services, which are abstracted by this type.

A secret is a specific flavor of configuration data for sensitive data that should not be exposed without security considerations. Secrets are made available to services as files mounted into their containers.

A project is an individual deployment of an application specification on a platform. A project’s name, set with the top-level name attribute, is used to group resources together and isolate them from other applications or other installation of the same Compose-specified application with distinct parameters.

You can use fragments and extensions to keep your Compose file efficient and easy to maintain.

Multiple Compose files can be merged together to define the application model.

If you want to reuse other Compose files, or factor out parts of your application model into separate Compose files, you can also use include. This is useful if your Compose application is dependent on another application which is managed by a different team, or needs to be shared with others.

services:
  frontend:
    image: example/webapp
    ports:
      - "443:8043"
    networks:
      - front-tier
      - back-tier
    configs:
      - httpd-config
    secrets:
      - server-certificate
 
  backend:
    image: example/database
    volumes:
      - db-data:/etc/data
    networks:
      - back-tier
 
volumes:
  db-data:
    driver: flocker
    driver_opts:
      size: "10GiB"
 
configs:
  httpd-config:
    external: true
 
secrets:
  server-certificate:
    external: true
 
networks:
  # The presence of these objects is sufficient to define them
  front-tier: {}
  back-tier: {}

Network

By default Compose sets up a single network for the entire application.

Control startup and shutdown order

You can control the order of service startup and shutdown with the depends_on attribute. Compose always starts and stops containers in dependency order, where dependencies are determined by depends_on, links, volumes_from, and network_mode: "service:...".

services:
  web:
    build: .
    depends_on:
      - db
      - redis
  redis:
    image: redis
  db:
    image: postgres

On startup, Compose does not wait until a container is “ready”, only until it’s running. This can cause issues if, for example, you have a relational database system that needs to start its own services before being able to handle incoming connections.

The solution for detecting the ready state of a service is to use the condition attribute with one of the following options:

  • service_started
  • service_healthy. This specifies that a dependency is expected to be “healthy”, which is defined with healthcheck, before starting a dependent service.
  • service_completed_successfully. This specifies that a dependency is expected to run to successful completion before starting a dependent service.

https://docs.docker.com/compose/startup-order/

Watch

The watch attribute automatically updates and previews your running Compose services as you edit and save your code. For many projects, this enables a hands-off development workflow once Compose is running, as services automatically update themselves when you save your work.

services:
  web:
    build: .
    ports:
      - "8000:5000"
    develop:
      watch:
        - action: sync
          path: .
          target: /code
  redis:
    image: "redis:alpine"

Whenever a file is changed, Compose syncs the file to the corresponding location under /code inside the container. Once copied, the bundler updates the running application without a restart.

https://docs.docker.com/compose/file-watch/

Most frontend development servers include native live reload support that works with Compose.

watch adheres to the following file path rules:

  • All paths are relative to the project directory
  • Directories are watched recursively
  • Glob patterns aren’t supported
  • Rules from .dockerignore apply
    • Use ignore option to define additional paths to be ignored (same syntax)
    • Temporary/backup files for common IDEs (Vim, Emacs, JetBrains, & more) are ignored automatically
    • .git directories are ignored automatically

Watch vs bind mounts

Compose supports sharing a host directory inside service containers. Watch mode does not replace this functionality but exists as a companion specifically suited to developing in containers.

More importantly, watch allows for greater granularity than is practical with a bind mount. Watch rules let you ignore specific files or entire directories within the watched tree.

For example, in a JavaScript project, ignoring the node_modules/ directory has two benefits:

  • Performance. File trees with many small files can cause high I/O load in some configurations
  • Multi-platform. Compiled artifacts cannot be shared if the host OS or architecture is different to the container

For example, in a Node.js project, it’s not recommended to sync the node_modules/ directory. Even though JavaScript is interpreted, npm packages can contain native code that is not portable across platforms.

Include

# compose.yaml
include:
   - infra.yaml
services:
  web:
    build: .
    ports:
      - "8000:5000"
    develop:
      watch:
        - action: sync
          path: .
          target: /code
# infra.yaml
services:
  redis:
    image: "redis:alpine"

Working with multiple Compose files.

Project name

In Compose, the default project name is derived from the base name of the project directory. However, you have the flexibility to set a custom project name.

Compose uses a project name to isolate environments from each other. There are multiple contexts where a project name is useful:

  • On a development host: Create multiple copies of a single environment, useful for running stable copies for each feature branch of a project.
  • On a CI server: Prevent interference between builds by setting the project name to a unique build number.
  • On a shared or development host: Avoid interference between different projects that might share the same service names.

Project names must contain only lowercase letters, decimal digits, dashes, and underscores, and must begin with a lowercase letter or decimal digit.

The precedence order for each method, from highest to lowest, is as follows:

  1. The -p command line flag.
  2. The COMPOSE_PROJECT_NAME environment variable.
  3. The top-level name: attribute in your Compose file. Or the last name: if you specify multiple Compose files in the command line with the -f flag.
  4. The base name of the project directory containing your Compose file. Or the base name of the first Compose file if you specify multiple Compose files in the command line with the -f flag.
  5. The base name of the current directory if no Compose file is specified.

Environment variables

With Compose, there are two ways you can set environment variables in your containers with your Compose file.

Use the environment attribute

You can set environment variables directly in your container’s environment with the environment attribute in your compose.yml.

It supports both list and mapping syntax:

services:
  webapp:
    environment:
      DEBUG: "true"

is equivalent to:

services:
  webapp:
    environment:
      - DEBUG=true

See environment attribute for more examples on how to use it.

You can choose not to set a value and pass the environment variables from your shell straight through to your containers. It works in the same way as docker run -e VARIABLE ...:

web:
  environment:
    - DEBUG

Note that in this case no warning is issued if the DEBUG variable in the shell environment is not set.

You can also take advantage of interpolation. In the following example, the result is similar to the one above but Compose gives you a warning if the DEBUG variable is not set in the shell environment or in an .env file in the project directory.

web:
  environment:
    - DEBUG=${DEBUG}

Use the env_file attribute

A container’s environment can also be set using .env files along with the env_file attribute.

services:
  webapp:
    env_file: "webapp.env"

The paths to your .env file, specified in the env_file attribute, are relative to the location of your compose.yml file.

Interpolation in .env files is a Docker Compose CLI feature. It is not supported when running docker run --env-file ....

If multiple files are specified, they are evaluated in order and can override values set in previous files.

In addition, as the .env file supports interpolation, it is possible to combine those with values set by environment.

As of Docker Compose version 2.24.0, you can set your .env file, defined by the env_file attribute, to be optional by using the required field. When required is set to false and the .env file is missing, Compose silently ignores the entry.

env_file:
  - path: ./default.env
    required: true # default
  - path: ./override.env
    required: false

Values in your .env file can be overridden from the command line by using docker compose run -e.

Set environment variables with docker compose run --env

Similar to docker run --env, you can set environment variables temporarily with docker compose run --env or its short form docker compose run -e:

 docker compose run -e DEBUG=1 web python console.py

You can also pass a variable from the shell by not giving it a value:

 docker compose run -e DEBUG web python console.py

The value of the DEBUG variable in the container is taken from the value for the same variable in the shell in which Compose is run.

Precedence

When the same environment variable is set in multiple sources, Docker Compose follows a precedence rule to determine the value for that variable in your container’s environment.

The order of precedence (highest to lowest) is as follows:

  1. Set using docker compose run -e in the CLI.
  2. Set with either the environment or env_file attribute but with the value interpolated from your shell or an environment file. (either your default .env file, or with the --env-file argument in the CLI).
  3. Set using just the environment attribute in the Compose file.
  4. Use of the env_file attribute in the Compose file.
  5. Set in a container image in the ENV directive. Having any ARG or ENV setting in a Dockerfile evaluates only if there is no Docker Compose entry for environment, env_file or run --env.

https://docs.docker.com/compose/environment-variables/envvars-precedence/

#docker compose runenvironment attributeenv_file attributeImage ENVHost OS environment.env fileResult
1----VALUE=1.4VALUE=1.3-
2--VALUE=1.6VALUE=1.5VALUE=1.4-VALUE=1.6
3-VALUE=1.7-VALUE=1.5VALUE=1.4-VALUE=1.7
4---VALUE=1.5VALUE=1.4VALUE=1.3VALUE=1.5
5--env VALUE=1.8--VALUE=1.5VALUE=1.4VALUE=1.3VALUE=1.8
6--env VALUE--VALUE=1.5VALUE=1.4VALUE=1.3VALUE=1.4
7--env VALUE--VALUE=1.5-VALUE=1.3VALUE=1.3
8--VALUEVALUE=1.5VALUE=1.4VALUE=1.3VALUE=1.4
9--VALUEVALUE=1.5-VALUE=1.3VALUE=1.3
10-VALUE-VALUE=1.5VALUE=1.4VALUE=1.3VALUE=1.4
11-VALUE-VALUE=1.5-VALUE=1.3VALUE=1.3
12--env VALUEVALUE=1.7-VALUE=1.5VALUE=1.4VALUE=1.3VALUE=1.4
13--env VALUE=1.8VALUE=1.7-VALUE=1.5VALUE=1.4VALUE=1.3VALUE=1.8
14--env VALUE=1.8-VALUE=1.6VALUE=1.5VALUE=1.4VALUE=1.3VALUE=1.8
15--env VALUE=1.8VALUE=1.7VALUE=1.6VALUE=1.5VALUE=1.4VALUE=1.3VALUE=1.8

Pre-defined env variables

Compose already comes with pre-defined environment variables. It also inherits common Docker CLI environment variables, such as DOCKER_HOST and DOCKER_CONTEXT. See Docker CLI environment variable reference for details.

  • COMPOSE_CONVERT_WINDOWS_PATHS
  • COMPOSE_FILE
  • COMPOSE_PROFILES
  • COMPOSE_PROJECT_NAME
  • DOCKER_CERT_PATH
  • COMPOSE_PARALLEL_LIMIT
  • COMPOSE_IGNORE_ORPHANS
  • COMPOSE_REMOVE_ORPHANS
  • COMPOSE_PATH_SEPARATOR
  • COMPOSE_ANSI
  • COMPOSE_STATUS_STDOUT
  • COMPOSE_ENV_FILES
  • COMPOSE_MENU
  • COMPOSE_EXPERIMENTAL

https://docs.docker.com/compose/environment-variables/envvars/

Interpolation

A Compose file can use variables to offer more flexibility. If you want to quickly switch between image tags to test multiple versions, or want to adjust a volume source to your local environment, you don’t need to edit the Compose file each time, you can just set variables that insert values into your Compose file at run time.

Interpolation can also be used to insert values into your Compose file at run time, which is then used to pass variables into your container’s environment

Below is a simple example:

$ cat .env
TAG=v1.5
$ cat compose.yml
services:
  web:
    image: "webapp:${TAG}"

When you run docker compose up, the web service defined in the Compose file interpolates in the image webapp:v1.5 which was set in the .env file.

Interpolation is applied for unquoted and double-quoted values. Both braced (${VAR}) and unbraced ($VAR) expressions are supported.

For braced expressions, the following formats are supported:

  • Direct substitution
    • ${VAR} → value of VAR
  • Default value
    • ${VAR:-default} → value of VAR if set and non-empty, otherwise default
    • ${VAR-default} → value of VAR if set, otherwise default
  • Required value
    • ${VAR:?error} → value of VAR if set and non-empty, otherwise exit with error
    • ${VAR?error} → value of VAR if set, otherwise exit with error
  • Alternative value
    • ${VAR:+replacement} → replacement if VAR is set and non-empty, otherwise empty
    • ${VAR+replacement} → replacement if VAR is set, otherwise empty

Docker Compose can interpolate variables into your Compose file from multiple sources.

Note that when the same variable is declared by multiple sources, precedence applies:

  1. Variables from your shell environment
  2. If --env-file is not set, variables set by an .env file in local working directory (PWD)
  3. Variables from a file set by --env-file or an .env file in project directory

You can check variables and values used by Compose to interpolate the Compose model by running docker compose config --environment.

.env file syntax

.env file syntax The following syntax rules apply to environment files:

  • Lines beginning with # are processed as comments and ignored.
  • Blank lines are ignored.
  • Unquoted and double-quoted (") values have interpolation applied.
  • Each line represents a key-value pair. Values can optionally be quoted.
    • VAR=VAL → VAL
    • VAR="VAL" → VAL
    • VAR='VAL' → VAL
  • Inline comments for unquoted values must be preceded with a space.
    • VAR=VAL # comment → VAL
    • VAR=VAL# not a comment → VAL# not a comment
  • Inline comments for quoted values must follow the closing quote.
    • VAR="VAL # not a comment" → VAL # not a comment
    • VAR="VAL" # comment → VAL
  • Single-quoted (') values are used literally.
    • VAR='$OTHER' → $OTHER
    • VAR='${OTHER}' → ${OTHER}
  • Quotes can be escaped with \.
    • VAR='Let\'s go!' → Let's go!
    • VAR="{\"hello\": \"json\"}" → {"hello": "json"}
  • Common shell escape sequences including \n, \\r, \t, and \\ are supported in double-quoted values.
    • VAR="some\tvalue" → some value
    • VAR='some\tvalue' → some\tvalue
    • VAR=some\tvalue → some\tvalue

Two .env files can be used, one is in the cwd, the other one is where the compose file is placed. Ref: local .env file versus .env file

Be cautious about including sensitive data in environment variables. Consider using Secrets for managing sensitive information.

Profiles

Profiles help you adjust your Compose application for different environments or use cases by selectively activating services. Services can be assigned to one or more profiles; unassigned services start by default, while assigned ones only start when their profile is active. This setup means specific services, like those for debugging or development, to be included in a single compose.yml file and activated only as needed.

services:
  frontend:
    image: frontend
    profiles: [frontend]
 
  phpmyadmin:
    image: phpmyadmin
    depends_on: [db]
    profiles: [debug]
 
  backend:
    image: backend
 
  db:
    image: mysql
docker compose --profile debug up

or

COMPOSE_PROFILES=debug docker compose up

To start multiple profiles:

docker compose --profile frontend --profile debug up

or

COMPOSE_PROFILES=frontend,debug docker compose up

Healthcheck

The healthcheck attribute declares a check that’s run to determine whether or not the service containers are “healthy”.

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost"]
  interval: 1m30s
  timeout: 10s
  retries: 3
  start_period: 40s
  start_interval: 5s

Graph View

Backlinks

  • No backlinks found