Deploy dockerized PHP Apps to production

- using multiple VMS and managed mysql and redis instances from GCP

Posted by Pascal Landau on 2022-09-04 06:00:00
Caution
This post is still in a draft status and likely subject to change!

In this part of the tutorial series on developing PHP on Docker we will deploy our dockerized PHP application to a production environment on GCP using multiple VMs and run it via "plain" docker (without compose).

What will you learn?
We'll modify the deployment process introduced in Deploy dockerized PHP Apps to production on GCP via docker compose as a POC to work with the new production infrastructure that we created in Create a production infrastructure for dockerized PHP Apps on GCP.

You'll learn how to orchestrate the deployment of multiple VMs, ensure their connectivity amongst each other and deploy a dockerized PHP application by building and pushing the images locally to pull and start them from the VMs.

The deployment process will be reflected in a single make target called deploy.

All code samples are publicly available in my Docker PHP Tutorial repository on Github.
You find the branch with the final result of this tutorial at part-11-deploy-dockerized-php-app-production.

All published parts of the Docker PHP Tutorial are collected under a dedicated page at Docker PHP Tutorial. The previous part was Create a production infrastructure for dockerized PHP Apps on GCP. and the following one is Use the gcloud cli docker image instead of installing it locally.

If you want to follow along, please subscribe to the RSS feed or via email to get automatic notifications when the next part comes out :)

Table of contents

Introduction

The general deployment process is essentially the same as outlined in the previous tutorial at Deploy dockerized PHP Apps to production on GCP via docker compose as a POC: Deployment workflow.

For a single VM it looks like shown in the following video:

This process now needs to be done for every Compute Instance VM that runs a dockerized service (i.e. everything labeled "GCP VM" in the following image)

docker based infrastructure on GCP

We are still able to perform all necessary steps via the deploy target defined in .make/05-00-deployment.mk:

.PHONY: deploy
deploy: # Build all images and deploy them to GCP
    @printf "$(GREEN)Cleaning up old 'deployment-settings.env' file$(NO_COLOR)\n"
    @"$(MAKE)" make-remove-deployment-settings
    @printf "$(GREEN)Starting docker setup locally$(NO_COLOR)\n"
    @"$(MAKE)" docker-compose-up
    @printf "$(GREEN)Verifying that there are no changes in the secrets$(NO_COLOR)\n"
    @"$(MAKE)" gpg-init
    @"$(MAKE)" deployment-guard-secret-changes
    @printf "$(GREEN)Verifying that there are no uncommitted changes in the codebase$(NO_COLOR)\n"
    @"$(MAKE)" deployment-guard-uncommitted-changes
    @printf "$(GREEN)Initializing gcloud deployment service account$(NO_COLOR)\n"
    @"$(MAKE)" gcp-init-deployment-account
    @printf "$(GREEN)Switching to 'prod' environment ('deployment-settings.env' file)$(NO_COLOR)\n"
    @"$(MAKE)" make-init-deployment-settings ENVS="ENV=prod TAG=latest"
    @printf "$(GREEN)Creating build information file$(NO_COLOR)\n"
    @"$(MAKE)" deployment-create-build-info-file
    @printf "$(GREEN)Building docker images$(NO_COLOR)\n"
    @"$(MAKE)" docker-compose-build
    @printf "$(GREEN)Pushing images to the registry$(NO_COLOR)\n"
    @"$(MAKE)" docker-compose-push
    @printf "$(GREEN)Creating the service-ips file$(NO_COLOR)\n"
    @"$(MAKE)" deployment-create-service-ip-file
    @printf "$(GREEN)Creating the deployment archive$(NO_COLOR)\n"
    @"$(MAKE)" deployment-create-tar
    @printf "$(GREEN)Copying the deployment archive to the VMs and run the deployment$(NO_COLOR)\n"
    @"$(MAKE)" deployment-run-on-vms
    @printf "$(GREEN)Clearing deployment archive$(NO_COLOR)\n"
    @"$(MAKE)" deployment-clear-tar
    @printf "$(GREEN)Removing 'deployment-settings.env' file$(NO_COLOR)\n"
    @"$(MAKE)" make-remove-deployment-settings

FYI: The individual make targets are explained in detail at the deploy target section of the previous docker compose based approach

There are some changes in this tutorial to accommodate for the switch from docker compose to docker and the usage of individual VMs per service over a single one.

Run the code yourself

CAUTION: The following steps assume, that the necessary infrastructure on GCP was already created and that you have the key files for the two service accounts

  • gcp-master-service-account-key.json (created manually upfront)
  • gcp-service-account-key.json (created automatically as part of the setup script)

If that's not the case, please go through the setup steps outlined in the previous tutorial Create a production infrastructure for dockerized PHP Apps on GCP: Run the code yourself first. If you need to go through the steps, please keep in mind that some targets have been renamed in this part, especially

docker-build => docker-compose-build
docker-up => docker-compose-up


# Prepare the codebase
git clone https://github.com/paslandau/docker-php-tutorial.git && cd docker-php-tutorial
git checkout part-11-deploy-dockerized-php-app-production

# Ensure that the service accounts key files exist
ls -l ./gcp-master-service-account-key.json ./gcp-service-account-key.json 
# Should NOT fail with
#   ls: cannot access './gcp-master-service-account-key.json': No such file or directory
#   ls: cannot access './gcp-service-account-key.json': No such file or directory

# Run the initialization 
make dev-init

# Update the variables `DOCKER_REGISTRY` and `GCP_PROJECT_ID` in `.make/variables.env` 
projectId="SET YOUR GCP_PROJECT_ID HERE"
# CAUTION: Mac users might need to use `sed -i '' -e` instead of `sed -i`! @see https://stackoverflow.com/a/19457213/413531
sed -i "s/DOCKER_REGISTRY=.*/DOCKER_REGISTRY=gcr.io\/${projectId}/" .make/variables.env
sed -i "s/GCP_PROJECT_ID=.*/GCP_PROJECT_ID=${projectId}/" .make/variables.env

# Ensure that the infrastructure is up and running
make gcp-init-master-account
make infrastructure-info

make docker-compose-build
make docker-compose-up
make gpg-init
make secret-decrypt

# Retrieve the AUTH string of the redis instance and set it in the `.secrets/prod/app.env` file
auth_string=$(make -s gcp-get-redis-auth)
# CAUTION: Mac users might need to use `sed -i '' -e` instead of `sed -i`! @see https://stackoverflow.com/a/19457213/413531
sed -i "s/REDIS_PASSWORD=.*/REDIS_PASSWORD=${auth_string}/" .secrets/prod/app.env

# Encrypt the secrets and commit the changes
make secret-encrypt
git add . && git commit -m "Update the REDIS_PASSWORD and re-encrypt the secrets"

# Run the deployment
make deploy

# Migrate the database
make deployment-setup-db-on-vm

# Verify the deployment
make deployment-info

Should show something like this

$ make deployment-info
application:
CONTAINER ID   IMAGE                                                                  COMMAND                  CREATED          STATUS          PORTS     NAMES
f7c8079d07ad   gcr.io/ay-mit-mct-atmo-gcp-temp2-c/dofroscra/application-prod:latest   "/decrypt-secrets.sh…"   23 seconds ago   Up 21 seconds             application
BUILD INFO
==========
User  : Pascal
Date  : 2022-09-13 10:39:12+02:00
Branch: part-11-deploy-dockerized-php-app-production

Commit
------
commit 79438866cc7e699bdb1aab30ec32269d67044366
Author: Pascal Landau <[email protected]>
Date:   Tue Sep 13 10:38:50 2022 +0200

    Update the REDIS_PASSWORD and re-encrypt the secrets


php-fpm:
CONTAINER ID   IMAGE                                                              COMMAND                  CREATED          STATUS          PORTS                                       NAMES
4150fed55c05   gcr.io/ay-mit-mct-atmo-gcp-temp2-c/dofroscra/php-fpm-prod:latest   "/decrypt-secrets.sh…"   43 seconds ago   Up 41 seconds   0.0.0.0:9000->9000/tcp, :::9000->9000/tcp   php-fpm
BUILD INFO
==========
User  : Pascal
Date  : 2022-09-13 10:39:12+02:00
Branch: part-11-deploy-dockerized-php-app-production

Commit
------
commit 79438866cc7e699bdb1aab30ec32269d67044366
Author: Pascal Landau <[email protected]>
Date:   Tue Sep 13 10:38:50 2022 +0200

    Update the REDIS_PASSWORD and re-encrypt the secrets


php-worker:
CONTAINER ID   IMAGE                                                                 COMMAND                  CREATED          STATUS          PORTS      NAMES
4654abb45e1a   gcr.io/ay-mit-mct-atmo-gcp-temp2-c/dofroscra/php-worker-prod:latest   "/decrypt-secrets.sh…"   51 seconds ago   Up 50 seconds   9001/tcp   php-worker
BUILD INFO
==========
User  : Pascal
Date  : 2022-09-13 10:39:12+02:00
Branch: part-11-deploy-dockerized-php-app-production

Commit
------
commit 79438866cc7e699bdb1aab30ec32269d67044366
Author: Pascal Landau <[email protected]>
Date:   Tue Sep 13 10:38:50 2022 +0200

    Update the REDIS_PASSWORD and re-encrypt the secrets


nginx:
CONTAINER ID   IMAGE                                                            COMMAND                  CREATED              STATUS              PORTS                                                                      NAMES
2f815040499a   gcr.io/ay-mit-mct-atmo-gcp-temp2-c/dofroscra/nginx-prod:latest   "/docker-entrypoint.…"   About a minute ago   Up About a minute   0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp   nginx

Visit the UI at: http://34.134.120.87/

The last line prints the public IP address of the VM of the nginx service that can be accessed from the internet:

$ curl -s http://34.134.120.87
<!DOCTYPE html>
<html>
    <head>
        <meta charset="utf-8">
    </head>
    <body>
    <ul>
    <li><a href="?dispatch=foo">Dispatch job 'foo' to the queue.</a></li>
    <li><a href="?queue">Show the queue.</a></li>
    <li><a href="?db">Show the DB.</a></li>
    <li><a href="/info">Show the build info.</a></li>
</ul>
    </body>
</html>

The following video shows the full process (excluding most of the waiting time)

Running containers with docker instead of docker compose

First, we will still use docker compose to build the production images, but we won't use it any longer to run the containers. The good news: There are no changes whatsoever in the Dockerfiles. So the task comes down to modifying the docker compose config files to remove all run settings and provide the equivalent commands via make targets.

Removing docker compose run settings for prod

We can inspect the full compose config for prod via make docker-compose-config ENV=prod:

$ make docker-compose-config ENV=prod
name: dofroscra_prod
services:
  application:
    build:
      context: C:\codebase\test-deploy\docker-php-tutorial\.docker
      dockerfile: ./images/php/application/Dockerfile
      args:
        BASE_IMAGE: gcr.io/dofroscra-part-10/dofroscra/php-base-prod:latest
        ENV: prod
      target: prod
    image: gcr.io/dofroscra-part-10/dofroscra/application-prod:latest
    networks:
      default: null
  nginx:
    build:
      context: C:\codebase\test-deploy\docker-php-tutorial\.docker
      dockerfile: ./images/nginx/Dockerfile
      args:
        APP_CODE_PATH: /var/www/app
        NGINX_VERSION: 1.21.5-alpine
      target: prod
    image: gcr.io/dofroscra-part-10/dofroscra/nginx-prod:latest
    networks:
      default: null
  php-fpm:
    build:
      context: C:\codebase\test-deploy\docker-php-tutorial\.docker
      dockerfile: ./images/php/fpm/Dockerfile
      args:
        BASE_IMAGE: gcr.io/dofroscra-part-10/dofroscra/php-base-prod:latest
        TARGET_PHP_VERSION: "8.1"
      target: prod
    image: gcr.io/dofroscra-part-10/dofroscra/php-fpm-prod:latest
    networks:
      default: null
  php-worker:
    build:
      context: C:\codebase\test-deploy\docker-php-tutorial\.docker
      dockerfile: ./images/php/worker/Dockerfile
      args:
        BASE_IMAGE: gcr.io/dofroscra-part-10/dofroscra/php-base-prod:latest
        PHP_WORKER_PROCESS_NUMBER: "4"
      target: prod
    image: gcr.io/dofroscra-part-10/dofroscra/php-worker-prod:latest
    networks:
      default: null
networks:
  default:
    name: dofroscra_prod_default

It consists of the files

  • .docker/docker-compose/docker-compose.local.ci.prod.yml
  • .docker/docker-compose/docker-compose.local.prod.yml

and the ENV based docker compose config was adjusted accordingly to reflect the file changes in .make/02-00-docker.mk

# We need to "assemble" the correct combination of docker-compose.yml config files
DOCKER_COMPOSE_FILES:=
ifeq ($(ENV),prod)
    DOCKER_COMPOSE_FILES:=-f $(DOCKER_COMPOSE_FILE_LOCAL_CI_PROD) -f $(DOCKER_COMPOSE_FILE_LOCAL_PROD)
else ifeq ($(ENV),ci)
    DOCKER_COMPOSE_FILES:=-f $(DOCKER_COMPOSE_FILE_LOCAL_CI_PROD) -f $(DOCKER_COMPOSE_FILE_LOCAL_CI) -f $(DOCKER_COMPOSE_FILE_CI)
else ifeq ($(ENV),local)
    DOCKER_COMPOSE_FILES:=-f $(DOCKER_COMPOSE_FILE_LOCAL_CI_PROD) -f $(DOCKER_COMPOSE_FILE_LOCAL_CI) -f $(DOCKER_COMPOSE_FILE_LOCAL_PROD) -f $(DOCKER_COMPOSE_FILE_LOCAL)
endif

The changes to the previous tutorial are described subsequently in more detail.

The file .docker/docker-compose/docker-compose.local.ci.prod.yml will be stripped down to only contain the build instructions for the application image:

services:
  application:
    image: ${DOCKER_REGISTRY?}/${DOCKER_NAMESPACE?}/application-${ENV?}:${TAG?}
    build:
      context: ../
      dockerfile: ./images/php/application/Dockerfile
      target: ${ENV?}
      args:
        - BASE_IMAGE=${DOCKER_REGISTRY?}/${DOCKER_NAMESPACE?}/php-base-${ENV?}:${TAG?}
        - ENV=${ENV?}

All the "rest" gets moved to the new file .docker/docker-compose/docker-compose.local.ci.yml because it's still relevant for the local and ci environments.

The same is true for the .docker/docker-compose/docker-compose.local.prod.yml file. It only keeps the build instructions for the services php-fpm php-worker and nginx:

services:
  php-fpm:
    image: ${DOCKER_REGISTRY?}/${DOCKER_NAMESPACE?}/php-fpm-${ENV?}:${TAG?}
    build:
      context: ../
      dockerfile: ./images/php/fpm/Dockerfile
      target: ${ENV?}
      args:
        - BASE_IMAGE=${DOCKER_REGISTRY?}/${DOCKER_NAMESPACE?}/php-base-${ENV?}:${TAG?}
        - TARGET_PHP_VERSION=${PHP_VERSION?}

  php-worker:
    image: ${DOCKER_REGISTRY?}/${DOCKER_NAMESPACE?}/php-worker-${ENV?}:${TAG?}
    build:
      context: ../
      dockerfile: ./images/php/worker/Dockerfile
      target: ${ENV?}
      args:
        - BASE_IMAGE=${DOCKER_REGISTRY?}/${DOCKER_NAMESPACE?}/php-base-${ENV?}:${TAG?}
        - PHP_WORKER_PROCESS_NUMBER=${PHP_WORKER_PROCESS_NUMBER:-4}

  nginx:
    image: ${DOCKER_REGISTRY?}/${DOCKER_NAMESPACE?}/nginx-${ENV?}:${TAG?}
    build:
      context: ../
      dockerfile: ./images/nginx/Dockerfile
      target: ${ENV?}
      args:
        - NGINX_VERSION=${NGINX_VERSION?}
        - APP_CODE_PATH=${APP_CODE_PATH_CONTAINER?}

The "rest" goes into file .docker/docker-compose/docker-compose.local.yml.

File .docker/docker-compose/docker-compose.local.prod.yml file is simply removed completely.

make targets for running containers via "plain" docker

The make targets for docker (and docker compose) are located in file .make/02-00-docker.mk. Since building and pushing the images is still done via compose, we only need to add targets for pulling images and managing containers via docker (i.e. start, stop and remove).

Note: To have clear distinction between targets for docker and those for docker compose, I have renamed the existing docker compose targets to include compose in the target name e.g.

docker-up   => docker-compose-up
docker-push => docker-compose-push

Pulling

Pulling is straight forwardly done via docker pull:

.PHONY: docker-pull
docker-pull: validate-docker-variables ## Pull a single docker images from the remote repository
    @$(if $(DOCKER_SERVICE_NAME),,$(error "DOCKER_SERVICE_NAME is undefined"))
    docker pull $(DOCKER_REGISTRY)/$(DOCKER_NAMESPACE)/$(DOCKER_SERVICE_NAME)-$(ENV):$(TAG)

The image name $(DOCKER_REGISTRY)/$(DOCKER_NAMESPACE)/$(DOCKER_SERVICE_NAME)-$(ENV):$(TAG) follows the convention outlined in Docker from scratch for PHP 8.1 Applications in 2022: Image naming convention. All variables apart from the $(DOCKER_SERVICE_NAME) should be defined as shared variables (i.e. they should be always present) and are checked in the validate-docker-variables target

.PHONY: validate-docker-variables
validate-docker-variables:
    @$(if $(TAG),,$(error TAG is undefined - Did you run 'make make-init'?))
    @$(if $(ENV),,$(error ENV is undefined - Did you run 'make make-init'?))
    @$(if $(DOCKER_REGISTRY),,$(error DOCKER_REGISTRY is undefined))
    @$(if $(DOCKER_NAMESPACE),,$(error DOCKER_NAMESPACE is undefined))
    @$(if $(APP_CODE_PATH_CONTAINER),,$(error APP_CODE_PATH_CONTAINER is undefined))

Running

Running a container is done via docker run

.PHONY: docker-run
docker-run: validate-docker-variables ## Start a single docker container
    @$(if $(DOCKER_SERVICE_NAME),,$(error "DOCKER_SERVICE_NAME is undefined"))
    @$(if $(HOST_STRING),,$(error "HOST_STRING is undefined"))
    docker run --name $(DOCKER_SERVICE_NAME) \
                -d \
                -it \
                --env-file compose-secrets.env \
                --mount type=bind,source="$$(pwd)"/secret.gpg,target=$(APP_CODE_PATH_CONTAINER)/secret.gpg,readonly \
                $(HOST_STRING) \
                $(DOCKER_SERVICE_OPTIONS) \
                $(DOCKER_REGISTRY)/$(DOCKER_NAMESPACE)/$(DOCKER_SERVICE_NAME)-$(ENV):$(TAG)

Though I believe the docker-run target requires some explanation:

--name $(DOCKER_SERVICE_NAME)

When starting a container, we will retain the same service name that we use in the docker compose setup. E.g. the service application will be started with the container name application via --name application. This way, we can streamline the same container management across docker and docker compose, because the $(DOCKER_SERVICE_NAME) works for both cases.

Example: The application container will always have the name application and we will use that information in the Stopping and Removing targets later.

--env-file compose-secrets.env

The environment variables are provided via --env-file compose-secrets.env. This file is created during the deployment process on the VM and contains the secret gpg password (GPG_PASSWORD). It was previously referenced in the (now deleted) docker-compose.local.prod.yml file.

--mount type=bind,source="$$(pwd)"/secret.gpg,target=$(APP_CODE_PATH_CONTAINER)/secret.gpg,readonly

Mounts the secret gpg key. $$(pwd) resolves to the current working directory and $(APP_CODE_PATH_CONTAINER) to the directory of the codebase in the container. This was previously done in the docker-compose.local.prod.yml file.

The variable was previously defined directly in the .docker/.env file but was moved to the shared variables in file .make/variables.env, so that it becomes available for make as well. Its value is

APP_CODE_PATH_CONTAINER=/var/www/app

It was added accordingly to the definition of the DOCKER_COMPOSE_COMMAND in .make/02-00-docker.mk:

DOCKER_COMPOSE_COMMAND:= \
 MSYS_NO_PATHCONV=1 \
 ENV=$(ENV) \
 TAG=$(TAG) \
 DOCKER_REGISTRY=$(DOCKER_REGISTRY) \
 DOCKER_NAMESPACE=$(DOCKER_NAMESPACE) \
 APP_USER_ID=$(APP_USER_ID) \
 APP_GROUP_ID=$(APP_GROUP_ID) \
 APP_USER_NAME=$(APP_USER_NAME) \
 APP_CODE_PATH_CONTAINER=$(APP_CODE_PATH_CONTAINER) \
 docker compose -p $(DOCKER_COMPOSE_PROJECT_NAME) --env-file $(DOCKER_ENV_FILE)

The MSYS_NO_PATHCONV=1 variables was added as well due to the handling of leading slashes of MinGW on Windows.

Note: secret gpg key and password are required for decrypting the encrypted secret files when the container starts.

$(HOST_STRING)

We need to somehow "substitute" the docker compose DNS magic, that allows us to communicate with a service by simply using the service name instead if its IP address. This is done with the $(HOST_STRING) variable that contains a list of --add-host options. See section Poor man's DNS via --add-host for a more in-depth explanation about this topic.

$(DOCKER_SERVICE_OPTIONS)

Each service might require some additional docker run options (e.g. forwarded ports) and those can be passed via the $(DOCKER_SERVICE_OPTIONS) variable. Since these options don't change, I have created a corresponding make target for each service:

.PHONY: docker-run-nginx
docker-run-nginx: ## Start the nginx container
    "$(MAKE)" docker-run DOCKER_SERVICE_NAME="$(VM_NAME_NGINX)" DOCKER_SERVICE_OPTIONS="-p 80:80 -p 443:443"

.PHONY: docker-run-php-fpm
docker-run-php-fpm: ## Start the php-fpm container
    "$(MAKE)" docker-run DOCKER_SERVICE_NAME="$(VM_NAME_PHP_FPM)" DOCKER_SERVICE_OPTIONS="-p 9000:9000"

.PHONY: docker-run-application
docker-run-application: ## Start the application container
    "$(MAKE)" docker-run DOCKER_SERVICE_NAME="$(VM_NAME_APPLICATION)" DOCKER_SERVICE_OPTIONS=""

.PHONY: docker-run-php-worker
docker-run-php-worker: ## Start the php-worker container
    "$(MAKE)" docker-run DOCKER_SERVICE_NAME="$(VM_NAME_PHP_WORKER)" DOCKER_SERVICE_OPTIONS=""

Stopping

Uses docker stop

.PHONY: docker-stop
docker-stop: validate-docker-variables ## Stop a single docker container
    @$(if $(DOCKER_SERVICE_NAME),,$(error "DOCKER_SERVICE_NAME is undefined"))
    docker stop $(DOCKER_SERVICE_NAME)

Removing

Uses docker rm

.PHONY: docker-rm
docker-rm: validate-docker-variables ## Remove a single docker container
    @$(if $(DOCKER_SERVICE_NAME),,$(error "DOCKER_SERVICE_NAME is undefined"))
    docker rm $(DOCKER_SERVICE_NAME)

Changes in the deployment process

Most of the steps in the deploy target of the previous docker compose based approach don't change, because we are still using docker compose to build the images. The main differences are, that

  • we will be using a temporary deployment .env file
  • the deployment archive becomes simpler
  • we need to ensure that the services can still communicate via their service names
  • the deployment script needs to be executed for every service on its VM

Use a temporary deployment .env file

The main Makefile will now include a third file containing variables located at .make/deployment-settings.env. The file is ignored by git and is created only temporary as part of the deployment process via

.PHONY: deploy
deploy: ## Build all images and deploy them to GCP
    # ...
    @printf "$(GREEN)Switching to 'prod' environment (via 'deployment-settings.env' file)$(NO_COLOR)\n"
    @"$(MAKE)" make-init-deployment-settings ENVS="ENV=prod TAG=latest"
    # ...

It is a copy of the .make/variables.env file and takes precedence over .make/variables.env and .make/.env. This is important, because the .make/.env file might hold variables that are specific to the local dev environment, e.g. a custom APP_USER_ID and APP_GROUP_ID (required for Linux users to avoid permission issues in the containers). But when the prod containers are built, we don't want to use any local specific settings.

Thus, the new .make/deployment-settings.env is created before the containers are build and deleted at the end of the deployment process via

.PHONY: deploy
deploy: # Build all images and deploy them to GCP
    # ...
    @printf "$(GREEN)Removing 'deployment-settings.env' file$(NO_COLOR)\n"
    @"$(MAKE)" make-remove-deployment-settings

See also the targets make-init-deployment-settings and make-remove-deployment-settings defined in the main Makefile:

.PHONY: make-init-deployment-settings
make-init-deployment-settings: ## Create a `deployment-settings.env` file to ensure that no local-only variables are affecting the deployment. Use via ENVS="KEY_1=value1 KEY_2=value2"
    @cp .make/variables.env .make/deployment-settings.env
    @for variable in $(ENVS); do \
      echo $$variable | tee -a .make/deployment-settings.env > /dev/null 2>&1; \
    done

.PHONY: make-remove-deployment-settings
make-remove-deployment-settings: ## Remove the `deployment-settings.env` file 
    @rm -f .make/deployment-settings.env

Plus, sometimes stuff "goes wrong" in the deployment, and then it's nice to be able to run make make-remove-deployment-settings and "clean up" the deployment environment settings nicely.

So in total we now have the following three .env files for the make setup:

They are included in the main Makefile in the order

include .make/variables.env
-include .make/.env
-include .make/deployment-settings.env

i.e. .make/deployment-settings.env takes precedence over .make/.env that in turn takes precedence over .make/variables.env.

A simpler deployment archive

This section refers to the previous tutorial (step Create the deployment archive) that uses the make target deployment-create-tar.

Since we don't use docker compose any longer, we don't need any docker compose configuration files in the deployment archive. The .env file is also no longer necessary, so we can get rid of the .secrets/docker.env file entirely.

Deployment archive: docker compose vs docker

The new deployment-create-tar target now looks like this:

.PHONY: deployment-create-tar
deployment-create-tar:
    # create the build directory
    rm -rf .build/deployment
    mkdir -p .build/deployment
    # copy the necessary files
    cp -r .make .build/deployment/
    find .build/deployment -name '*.env' -delete
    cp .make/variables.env .build/deployment/.make/variables.env
    cp Makefile .build/deployment/
    cp .infrastructure/scripts/deploy.sh .build/deployment/
    # move the ip services file
    mv .build/service-ips .build/deployment/service-ips
    # create the archive
    tar -czvf .build/deployment.tar.gz -C .build/deployment/ ./

The make setup is still required, and we are even adding a new file called .build/service-ips, that is created to enable communication via service names and is explained in the next section Poor man's DNS via --add-host.

Poor man's DNS via --add-host

So far, we could make use of the DNS magic provided by docker compose that enabled all our containers to talk to each other via their service name.

DNS for services in the same network works out of the box for docker compose

We use this e.g. to

This will stop working once we drop docker compose and run the containers on individual machines

DNS for docker does not work across different VMs

Since all services will now live on different VMs, we need to identify the private IP address of every VM and make sure that the service name of the container running on that VM resolves to that IP address.

The latter part can be achieved quite easily by using the --add-host option of docker run

Your container will have lines in /etc/hosts which define the hostname of the container itself as well as localhost and a few other common things. The --add-host flag can be used to add additional lines to /etc/hosts.

DNS for docker across different VMs can be "enabled" via --add-host

But how to we find the IP addresses?

Retrieving IP addresses

Fortunately the gcloud cli has us covered here:

I have added corresponding make targets in .make/03-00-gcp.mk

.PHONY: gcp-get-private-ip-vm
gcp-get-private-ip-vm: ## Get the private ip of a VM
    @$(if $(GCP_PROJECT_ID),,$(error "GCP_PROJECT_ID is undefined"))
    @$(if $(VM_NAME),,$(error "VM_NAME is undefined"))
    gcloud compute instances describe $(VM_NAME) --format="get(networkInterfaces[0].networkIP)" --project=$(GCP_PROJECT_ID) --zone=$(GCP_ZONE)

.PHONY: gcp-get-private-ip-mysql
gcp-get-private-ip-mysql: ## Get the private IP address of the SQL service
    gcloud sql instances describe $(VM_NAME_MYSQL) --format="get(ipAddresses[0].ipAddress)" --project=$(GCP_PROJECT_ID)

.PHONY: gcp-get-private-ip-redis
gcp-get-private-ip-redis: ## Get the private IP address of the Redis service
    gcloud redis instances describe $(VM_NAME_REDIS) --format="get(host)" --project=$(GCP_PROJECT_ID) --region=$(GCP_REGION)

The service-ips file: Map service names to IPs

Now that we can "get" the IP addresses of all our services, we can store them in a simple text file as part of the deployment process and transmit this file to all the VMs.

Of course there a is make target for the retrieval (in .make/03-00-gcp.mk)

.PHONY: gcp-get-ips
gcp-get-ips: ## Get the IP addresses for all services
    @printf "$(DOCKER_SERVICE_NAME_MYSQL):"
    @"$(MAKE)" -s gcp-get-private-ip-mysql
    @printf "$(DOCKER_SERVICE_NAME_REDIS):"
    @"$(MAKE)" -s gcp-get-private-ip-redis
    @for vm_name_service_name in $(ALL_VM_SERVICE_NAMES); do \
        vm_name=`echo $$vm_name_service_name | cut -d ":" -f 1`; \
        service_name=`echo $$vm_name_service_name | cut -d ":" -f 2`; \
        printf "$$service_name:"; \
        make -s gcp-get-private-ip-vm VM_NAME=$$vm_name; \
      done;

The gcp-get-ips target uses the targets of the previous section and prints them in the format

$serviceName:$ipAddress

Note: Please refer to section Mapping service names to VM names regarding the variable $(ALL_VM_SERVICE_NAMES) that contains a mapping of the vm names to the service names in the form

$vmName:$serviceName

A full result might look like this:

$ make gcp-get-ips
mysql:10.111.0.5
redis:10.111.0.50
php-fpm:10.128.0.2
application:10.128.0.3
nginx:10.128.0.4
php-worker:10.128.0.5

Storing them in a file (.build/service-ips) is done as part of the deploy target via the target deployment-create-service-ip-file defined in file .make/05-00-deployment.mk.


.PHONY: deployment-create-service-ip-file
deployment-create-service-ip-file: ## Create a file containing the IPs of all services
    @make -s gcp-get-ips > ".build/service-ips"
    @sed -i "s/\r//g" ".build/service-ips"

Note: The sed -i "s/\r//g" ".build/service-ips" part was necessary, because on Windows the line endings in the gcp-get-ips are \r\n instead of just \n and the sed command ensures that no \r (carriage return) are left over.

The application deployment script

On a conceptual level, the deployment "on a VM" in the previous tutorial was done with a script (located at .infrastructure/scripts/deploy.sh) that is transmitted to the VM first and then executed there. This script was slightly updated:

#!/usr/bin/env bash

set -e

usage="Usage: deploy.sh docker_service_name"
[ -z "$1" ] &&  echo "No docker_service_name given! $usage" && exit 1

docker_service_name=$1

echo "Initializing the codebase"
make make-init ENVS="ENV=prod TAG=latest"
echo "Retrieving secrets"
make gcp-secret-get SECRET_NAME=GPG_KEY > secret.gpg
GPG_PASSWORD=$(make gcp-secret-get SECRET_NAME=GPG_PASSWORD)
echo "Creating compose-secrets.env file"
echo "GPG_PASSWORD=$GPG_PASSWORD" > compose-secrets.env
echo "Pulling image for '${docker_service_name}' on the VM from the registry"
make docker-pull DOCKER_SERVICE_NAME="${docker_service_name}"
echo "Stop the '${docker_service_name}' container on the VM"
make docker-stop DOCKER_SERVICE_NAME="${docker_service_name}" || true
make docker-rm DOCKER_SERVICE_NAME="${docker_service_name}" || true

echo "Preparing service IPs as --add-host options"
service_ips=""
while read -r line; 
do 
  service_ips=$service_ips" --add-host $line" 
done < service-ips
echo "Start the container for '${docker_service_name}' on the VM"
make docker-run-"${docker_service_name}" HOST_STRING="$service_ips"

It now expects the service name as a mandatory argument and uses the make targets for running containers via "plain" docker:

make docker-pull DOCKER_SERVICE_NAME="${docker_service_name}"
make docker-stop DOCKER_SERVICE_NAME="${docker_service_name}" || true
make docker-rm DOCKER_SERVICE_NAME="${docker_service_name}" || true

It also utilizes the .build/service-ips file from the previous section to build the HOST_STRING for running the container:

service_ips=""
while read -r line; 
do 
  service_ips=$service_ips" --add-host $line" 
done < service-ips

make docker-run-"${docker_service_name}" HOST_STRING="$service_ips"

Run the deployment script for each service

The application deployment script of the previous section is invoked as part of the deployment-run-on-vm target that we already know from the previous tutorial, see Deployment commands on the VM.

.PHONY: deployment-run-on-vm
deployment-run-on-vm: ## Run the deployment script on the VM specified by VM_NAME for the service specified by DOCKER_SERVICE_NAME
    @$(if $(DOCKER_SERVICE_NAME),,$(error "DOCKER_SERVICE_NAME is undefined"))
    "$(MAKE)" -s gcp-scp-command SOURCE=".build/deployment.tar.gz" DESTINATION="deployment.tar.gz"
    "$(MAKE)" -s gcp-ssh-command COMMAND="sudo rm -rf $(CODEBASE_DIRECTORY) && sudo mkdir -p $(CODEBASE_DIRECTORY) && sudo tar -xzvf deployment.tar.gz -C $(CODEBASE_DIRECTORY) && cd $(CODEBASE_DIRECTORY) && sudo bash deploy.sh $(DOCKER_SERVICE_NAME)"

This target has to be invoked for every VM, and we use the same technique as in section Setup the VMs that run docker containers, i.e.

  • create a make target for every service, e.g.

    .PHONY: deployment-run-on-vm-application
deployment-run-on-vm-application:
"$(MAKE)" --no-print-directory deployment-run-on-vm VM_NAME=$(VM_NAME_APPLICATION) DOCKER_SERVICE_NAME=$(DOCKER_SERVICE_NAME_APPLICATION)

  • create a target to run them all in parallel via make

    .PHONY: deployment-run-on-vms
deployment-run-on-vms: ## Run the deployment script on all VMs
"$(MAKE)" -j --output-sync=target   deployment-run-on-vm-application \
                                  deployment-run-on-vm-php-fpm \
                                  deployment-run-on-vm-php-worker \
                                  deployment-run-on-vm-nginx

Wrapping up

Congratulations, you made it! If some things are not completely clear by now, don't hesitate to leave a comment. You should now be ready to deploy a dockerized PHP application "to production" on GCP using multiple VMs via docker - without compose .

In the next part of this tutorial, we will replace the locally installed gcloud cli with the official docker image to get rid of the dependency.

Please subscribe to the RSS feed or via email to get automatic notifications when this next part comes out :)

Wanna stay in touch?

Since you ended up on this blog, chances are pretty high that you're into Software Development (probably PHP, Laravel, Docker or Google Big Query) and I'm a big fan of feedback and networking.

So - if you'd like to stay in touch, feel free to shoot me an email with a couple of words about yourself and/or connect with me on LinkedIn or Twitter or simply subscribe to my RSS feed or go the crazy route and subscribe via mail and don't forget to leave a comment :)

Subscribe to posts via mail

We use Mailchimp as our newsletter provider. By clicking subscribe, you acknowledge that your information will be transferred to Mailchimp for processing. Learn more about Mailchimp's privacy practices here.
Waving bear

Comments