All code samples are publicly available in my Docker PHP Tutorial repository on Github.
You find the branch with the final result of this tutorial at part-10-create-production-infrastructure-php-app-gcp.

CAUTION: With this codebase it is not possible to deploy any longer! Please refer to the next part Deploy dockerized PHP Apps to production - using multiple VMS and managed mysql and redis instances from GCP for the code to enable deployments again.

All published parts of the Docker PHP Tutorial are collected under a dedicated page at Docker PHP Tutorial. The previous part was Deploy dockerized PHP Apps to production on GCP via docker compose as a POC.

If you want to follow along, please subscribe to the RSS feed or via email to get automatic notifications when the next part comes out :)

Table of contents

• Introduction
  • Run the code yourself
• Additional GCP concepts
  • IPs, Networking and VPCs (Virtual Private Cloud)
  • Routers and NATs
  • IP range allocations, VPC Peering and private service access
  • Additional Service APIs and IAM roles
• Service setup
  • Cloud SQL for MySQL setup
  • Redis Memorystore setup
  • Set up the VMs that run docker containers
    • Provisioning the VMs
  • Mapping service names to VM names
• Appendix: Changes in the codebase
  • Add a make dev-init target
  • Update the gcloud targets
  • Add additional make variables
  • Use make to execute infrastructure and deployment targets in parallel
• Wrapping up

Introduction

In Deploy dockerized PHP Apps to production on GCP via docker compose as a POC we've created a single Compute Instance VM, provisioned it with docker compose and ran our full docker compose setup on it. In other words: All containers ran on the same VM (that had to be reachable from the internet).

In this part we want to split this setup up so that:

• each docker container runs on its own VM
• redis and mysql won't run in docker containers but as the managed GCP products MySQL Cloud SQL for MySQL and Redis Memorystore

In addition, we want to expose only the nginx service to the internet - all other VMs should communicate via private IP addresses.

Run the code yourself

I recommend creating a completely new GCP project to have a "clean slate" that ensures that everything works out of the box as intended.

# Prepare the codebase
git clone https://github.com/paslandau/docker-php-tutorial.git && cd docker-php-tutorial
git checkout part-10-create-production-infrastructure-php-app-gcp

# Run the initialization. 
make dev-init

# Note: 
# You don't have to follow the additional instructions of the `dev-init` target 
# for this part of the tutorial.

# The following steps need to be done manually:
# 
# - Create a new GCP project and "master" service account with Owner permissions.
# - Create a key file for that master service account and place it in the root of the codebase at
#   ./gcp-master-service-account-key.json
#
# @see https://www.pascallandau.com/blog/gcp-compute-instance-vm-docker/#preconditions-project-and-owner-service-account

ls ./gcp-master-service-account-key.json
# Should NOT fail with
#   ls: cannot access './gcp-master-service-account-key.json': No such file or directory

# Update the variables `DOCKER_REGISTRY` and `GCP_PROJECT_ID` in `.make/variables.env` 

projectId="SET YOUR GCP_PROJECT_ID HERE"
# CAUTION: Mac users might need to use `sed -i '' -e` instead of `sed -i`!
# @see https://stackoverflow.com/a/19457213/413531
sed -i "s/DOCKER_REGISTRY=.*/DOCKER_REGISTRY=gcr.io\/${projectId}/" .make/variables.env
sed -i "s/GCP_PROJECT_ID=.*/GCP_PROJECT_ID=${projectId}/" .make/variables.env

make docker-build
make docker-up
make gpg-init
make secret-decrypt

# Then run
make infrastructure-setup ROOT_PASSWORD="production_secret_mysql_root_password"

# FYI: This step can take 15 to 30 minutes to complete.

# Verify the setup
make infrastructure-info

Should show something like this

$ make infrastructure-info
NAME            ZONE           MACHINE_TYPE  PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP    STATUS
application-vm  us-central1-a  e2-micro                   10.128.0.5                  RUNNING
nginx-vm        us-central1-a  e2-micro                   10.128.0.3   34.134.120.87  RUNNING
php-fpm-vm      us-central1-a  e2-micro                   10.128.0.2                  RUNNING
php-worker-vm   us-central1-a  e2-micro                   10.128.0.4                  RUNNING

INSTANCE_NAME  VERSION    REGION       TIER   SIZE_GB  HOST           PORT  NETWORK  RESERVED_IP       STATUS  CREATE_TIME
redis-vm       REDIS_6_X  us-central1  BASIC  1        10.137.150.67  6379  default  10.137.150.64/29  READY   2022-09-12T11:22:14

NAME      DATABASE_VERSION  LOCATION       TIER              PRIMARY_ADDRESS  PRIVATE_ADDRESS  STATUS
mysql-vm  MYSQL_8_0         us-central1-b  db-custom-1-3840  -                10.111.0.3       RUNNABLE

The following video shows the full process (excluding most of the waiting time)

Additional GCP concepts

IPs, Networking and VPCs (Virtual Private Cloud)

In order to restrict access to VMs from the internet, we need to ensure that GCP does not assign them a public IP address. They still need to be able to communicate with each other though and thus need a private IP address within the same network / VPC (Virtual Private Cloud).

So far, we didn't need to think about that at all, because GCP creates a default network (a so-called auto mode VPC network) called default for each project, and we have simply used it as-is. But there are a number of reasons why not using the default network is a good idea, e.g.

• unnecessary subnetworks (one for each region - we'll only need one region for our setup)
• overly permissive firewall rules (e.g. for port 3389 for the Microsoft Remote Desktop Protocol [RDP].)

But for now the default network is "good enough", because it's not per-se insecure (as in "nobody from the outside world has access to it"), and we'll tackle this and some other security hardening measures in a later tutorial.

You can find the default network in the VPC networks UI

Routers and NATs

Using only private IP addresses comes with a non-obvious caveat: Compute Instance VMs cannot only no longer be reached from the internet but also not reach the internet any longer themselves as per GCP docs:

By default, when a Compute Engine VM lacks an external IP address assigned to its network interface, it can only send packets to other internal IP address destinations.

This is a problem for a number of reasons, e.g.

• you can't install new software on the VM (e.g. apt-get will fail)
• it's impossible to retrieve docker images from the registry
• the application itself might need to make HTTP requests to public APIs

Fortunately, there is a simple solution: NAT. NAT is short for Network Address Translation and is basically happening for any private internet access at home. Your router "translates" the internal IP address of your local machine to a public one, so that the response packets can be routed back to the router and from there to your local machine.

GCP offers the same functionality via Cloud NAT.

(Source)

We must first create a Cloud Router that serves as the control plane for Cloud NAT. The creation process is explained in the GCP Cloud Router Guide: Create a Cloud Router. Please note, that for Cloud NAT routers it's not necessary to assign ASN numbers:

Note: Cloud NAT does not use ASN information. Cloud NAT gateways can be connected to Cloud Routers that have any ASN or that have no ASN specified.

Once the Cloud Router exists, we can create a Cloud NAT following the steps outlined in the GCP Cloud NAT Guide: Set up and manage network address translation with Cloud NAT.

The following video explains the full process via the Cloud Console UI by showing that a Compute Instance VM without external IP address cannot reach http://example.com/ unless a Cloud NAT is created for the same network.

Creating a Cloud Router and a Cloud NAT gateway can also be done via gcloud cli. I've added the following commands to .infrastructure/setup-gcp.sh

region=us-central1
router_name=default-router
nat_name=default-nat-gateway
network="default"

gcloud compute routers create "${router_name}" \
      --region="${region}" \
      --network="${network}"

gcloud compute routers nats create "${nat_name}"  \
    --router="${router_name}" \
    --router-region="${region}" \
    --auto-allocate-nat-external-ips \
    --nat-all-subnet-ip-ranges

gcloud cli docs:

• gcloud compute routers create
• gcloud compute routers nats create

IP range allocations, VPC Peering and private service access

Since we will be using managed services for mysql and redis, we need to create a so-called VPC peering with the Google Cloud Platform Service Producer. This is necessary, because GCP will create a dedicated VPCs for MySQL Cloud SQL and Redis Memorystore instances. See also the GCP VPC Guides:

• VPC Network Peering overview
• Private services access
• Configuring private services access

Since we don't want to assign public IPs, we need to "peer" our default VPC with those dedicated service VPCs. I have explained this in more detail in my MySQL Cloud SQL article for connecting via private IP. In short:

• we must allocate a certain range of IPs in the default network
• this range is then assigned to the VPC peering for Google Cloud Platform Services
• in consequence, the redis and mysql instances will receive an IP address from the allocated IP range and are thus "visible" for all other VMs in the default network

See also the Example in the MySQL Guide: Learn about using private IP:

This video shows the full procedure to create an IP range allocation and a VPC peering to enable private access on a MySQL Cloud SQL instance:

Creating an IP range allocation and a VPC peering can also be done via gcloud cli. I've added the following commands to .infrastructure/setup-gcp.sh

network="default"
private_vpc_range_name="google-managed-services-vpc-allocation"
ip_range_network_address="10.111.0.0"

gcloud compute addresses create "${private_vpc_range_name}" \
    --global \
    --purpose=VPC_PEERING \
    --prefix-length=16 \
    --description="Peering range for Google" \
    --network="${network}" \
    --addresses="${ip_range_network_address}"

gcloud services vpc-peerings connect \
    --service=servicenetworking.googleapis.com \
    --ranges="${private_vpc_range_name}" \
    --network="${network}"

gcloud cli docs:

• gcloud compute addresses create
• gcloud services vpc-peerings connect

Additional Service APIs and IAM roles

Since we are using even more GCP services as before, we need to enable the corresponding APIs as well. Those are:

• servicenetworking.googleapis.com
  • required for allocating the IP range and establishing the VPC peering
• sqladmin.googleapis.com
  • required for the Cloud SQL for MySQL setup and management
• redis.googleapis.com
  • required for Redis Memorystore setup and management

As part of the deployment process in the next part of the tutorial Deploy dockerized PHP Apps to production - using multiple VMS and managed mysql and redis instances from GCP, we need to retrieve the IP addresses of all our services (see Poor man's DNS via --add-host) via the gcloud cli. This requires some additional IAM roles / permissions for the deployment service account:

• cloudsql.viewer
  • required to retrieve meta data from MySQL Cloud SQL instances
• redis.viewer
  • required to retrieve meta data from Redis Memorystore instances
• compute.viewer
  • required to retrieve meta data from Compute Instance VMs

I have added the APIs and roles also in the .infrastructure/setup-gcp.sh script:

project_id=$1
deployment_service_account_id=deployment
deployment_service_account_mail="${deployment_service_account_id}@${project_id}.iam.gserviceaccount.com"

gcloud services enable \
  containerregistry.googleapis.com \
  secretmanager.googleapis.com \
  compute.googleapis.com \
  iam.googleapis.com \
  storage.googleapis.com \
  cloudresourcemanager.googleapis.com \
  sqladmin.googleapis.com \
  redis.googleapis.com \
  servicenetworking.googleapis.com

roles="storage.admin secretmanager.admin compute.admin iam.serviceAccountUser iap.tunnelResourceAccessor cloudsql.viewer redis.viewer compute.viewer"
for role in $roles; do
  gcloud projects add-iam-policy-binding "${project_id}" --member=serviceAccount:"${deployment_service_account_mail}" "--role=roles/${role}"
done;

Service setup

Cloud SQL for MySQL setup

Please refer to How to use GCP MySQL Cloud SQL instances - from creation over connection to deletion for a general introduction into Cloud SQL for MySQL.

For this tutorial, we will create a mysql instance with the following settings:

• 1 CPU
• 3840 MB RAM
• Private Service Access / private IP only
• Enabled deletion protection

We'll also set the password of the root user to the password defined in the DB_PASSWORD variable in .secrets/prod/app.env and create an application database named

application_db

I adapted the gcloud setup from How to use GCP MySQL Cloud SQL instances: "Using the gcloud CLI to create the script .infrastructure/setup-mysql.sh that will create a mysql instance and uses the following commands

project_id=$1
mysql_instance_name=$2
root_password=$3
region=us-central1
master_service_account_key_location=./gcp-master-service-account-key.json
memory="3840MB"
cpus=1
version="MYSQL_8_0"
default_database="application_db"
network="default"
private_vpc_range_name="google-managed-services-vpc-allocation"

gcloud beta sql instances create "${mysql_instance_name}" \
        --database-version="${version}" \
        --cpu="${cpus}" \
        --memory="${memory}" \
        --region="${region}" \
        --network="${network}" \
        --deletion-protection \
        --no-assign-ip \
        --allocated-ip-range-name="${private_vpc_range_name}"

gcloud sql users set-password root \
        --host=% \
        --instance "${mysql_instance_name}" \
        --password "${root_password}"

gcloud sql databases create "${default_database}" \
        --instance="${mysql_instance_name}" \
        --charset=utf8mb4 \
        --collation=utf8mb4_unicode_ci

A corresponding make infrastructure-setup-mysql target is defined in .make/04-00-infrastructure.mk

.PHONY: infrastructure-setup-mysql
infrastructure-setup-mysql: ## Set the mysql instance up. The ROOT_PASSWORD variable is required to defined the password for the root user
    @$(if $(ROOT_PASSWORD),,$(error "ROOT_PASSWORD is undefined"))
    bash .infrastructure/setup-mysql.sh $(GCP_PROJECT_ID) $(VM_NAME_MYSQL) $(ROOT_PASSWORD) $(ARGS)

CAUTION: We use MYSQL_8_0 as version for the instance and should ensure to keep this in sync with the value of the MYSQL_VERSION variable in the .docker/.env file (see also Structuring the Docker setup for PHP Projects .env.example and docker-compose.yml), so that we use the same version in the docker compose for our local and ci setup and keep parity between the environments.

Redis Memorystore setup

Please refer to How to use GCP Redis Memorystore instances - from creation over connection to deletion for a general introduction into Redis Memorystore.

For this tutorial, we will create a redis instance with the following settings:

• 1 GiB RAM
• Private Service Access / private IP
• Enabled AUTH

I adapted the gcloud setup from How to use GCP Redis Memorystore instances: "Using the gcloud CLI to create the script .infrastructure/setup-redis.sh that will create a redis instance and uses the following commands

project_id=$1
redis_instance_name=$2
region=us-central1
size=1 # in GiB
network="default"
version="redis_6_x"
private_vpc_range_name="google-managed-services-vpc-allocation"

gcloud redis instances create "${redis_instance_name}" \
      --size="${size}" \
      --region="${region}" \
      --network="${network}" \
      --redis-version="${version}" \
      --connect-mode=private-service-access \
      --reserved-ip-range="${private_vpc_range_name}" \
      --enable-auth \
      -q

A corresponding make infrastructure-setup-redis target is defined in .make/04-00-infrastructure.mk

.PHONY: infrastructure-setup-redis
infrastructure-setup-redis: ## Set the redis instance up
    bash .infrastructure/setup-redis.sh $(GCP_PROJECT_ID) $(VM_NAME_REDIS) $(ARGS)

CAUTION: We use redis_6_x as version for the instance and should ensure to keep this in sync with the value of the REDIS_VERSION variable in the .docker/.env file (see also Structuring the Docker setup for PHP Projects .env.example and docker-compose.yml), so that we use the same version in the docker compose for our local and ci setup and keep parity between the environments.

The AUTH string is created automatically, i.e. we must retrieve it after the creation and update the value of the REDIS_PASSWORD variable in the production .env file of the application at .secrets/prod/app.env. I have created a corresponding make target in .make/03-00-gcp.mk

.PHONY: gcp-get-redis-auth
gcp-get-redis-auth: ## Get the AUTH string of the Redis service
    gcloud redis instances get-auth-string $(VM_NAME_REDIS) --project=$(GCP_PROJECT_ID) --region=$(GCP_REGION)

Please note, that you need to activate the master service account first, because retrieving the AUTH string required the permission redis.instances.getAuthString that is by default only available in role roles/redis.admin.

Set up the VMs that run docker containers

We'll stick mostly with the same VM configuration as outlined in my Run Docker on GCP Compute Instance VMs: Create a Compute Instance VM, though we need to ensure that the VMs for the application containers application, php-fpm and php-workers are not getting a public IP. This is achieved by adding the --no-address flag.

Note: Adding the flag seemed to have no effect at first. This was because I initially used the --network-interface option to define all the network settings "as a single string". When this option is provided, the --no-address has no effect, because --network-interface takes precedence and the default value here is address, i.e. "provide a public IP address".

In other words: This doesn't work as expected

$ gcloud compute instances create test --zone="us-central1-a" --machine-type="f1-micro" \
        --network-interface=network=default \
        --no-address
Created [https://www.googleapis.com/compute/v1/projects/ay-mit-mct-atmo-gcp-mig-temp-c/zones/us-central1-a/instances/test].
NAME  ZONE           MACHINE_TYPE  PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP   STATUS
test  us-central1-a  f1-micro                   10.128.0.6   34.170.88.99  RUNNING

# ==> Note the "EXTERNAL_IP"

This does:

$ gcloud compute instances create test --zone="us-central1-a" --machine-type="f1-micro" \
        --network=default \
        --no-address
Created [https://www.googleapis.com/compute/v1/projects/ay-mit-mct-atmo-gcp-mig-temp-c/zones/us-central1-a/instances/test].
NAME  ZONE           MACHINE_TYPE  PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP  STATUS
test  us-central1-a  f1-micro                   10.128.0.7                RUNNING

# ==> Note that no "EXTERNAL_IP" is given

In addition, the http-server tag responsible for assigning the firewall rule to allow http traffic is also not necessary and must be omitted.

I've separated the script explained under Run Docker on GCP Compute Instance VMs: Putting it all together in the "general" GCP setup at .infrastructure/setup-gcp.sh and a script dedicated for creating a Compute Instance VM at .infrastructure/setup-vm.sh via the following commands

project_id=$1
vm_name=$2
enable_public_access=$3
vm_zone=us-central1-a
master_service_account_key_location=./gcp-master-service-account-key.json
deployment_service_account_id=deployment
deployment_service_account_mail="${deployment_service_account_id}@${project_id}.iam.gserviceaccount.com"
network="default"

# By default, VMs should not get an external IP address
# @see https://cloud.google.com/sdk/gcloud/reference/compute/instances/create#--no-address
args_for_public_access="--no-address"
if [ -n "$enable_public_access" ]
then
  # The only exception is the nginx image - that should also be available via http / port 80
  args_for_public_access="--tags=http-server"
fi

gcloud compute instances create "${vm_name}" \
    --project="${project_id}" \
    --zone="${vm_zone}" \
    --machine-type=e2-micro \
    --network="${network}" \
    --subnet=default \
    --network-tier=PREMIUM \
    --no-restart-on-failure \
    --maintenance-policy=MIGRATE \
    --provisioning-model=STANDARD \
    --service-account="${deployment_service_account_mail}" \
    --scopes=https://www.googleapis.com/auth/cloud-platform \
    --create-disk=auto-delete=yes,boot=yes,device-name="${vm_name}",image=projects/debian-cloud/global/images/debian-11-bullseye-v20220822,mode=rw,size=10,type=projects/"${project_id}"/zones/"${vm_zone}"/diskTypes/pd-balanced \
    --no-shielded-secure-boot \
    --shielded-vtpm \
    --shielded-integrity-monitoring \
    --reservation-affinity=any $args_for_public_access

Note: If the script is invoked with a 3rd argument, we'll assume that it's for the nginx service so that the --no-address flag is omitted and the --tags=http-server is added.

A corresponding "generic" make infrastructure-setup-vm target is defined in .make/04-00-infrastructure.mk

.PHONY: infrastructure-setup-vm
infrastructure-setup-vm: ## Setup the VM specified via VM_NAME. Usage: make infrastructure-setup-vm VM_NAME=php-worker ARGS=""
    @$(if $(VM_NAME),,$(error "VM_NAME is undefined"))
    bash .infrastructure/setup-vm.sh $(GCP_PROJECT_ID) $(VM_NAME) $(ARGS)

In addition, there are dedicated targets for each service, e.g.

.PHONY: infrastructure-setup-vm-php-worker
infrastructure-setup-vm-php-worker:
    "$(MAKE)" --no-print-directory infrastructure-setup-vm VM_NAME=$(VM_NAME_PHP_WORKER)

.PHONY: infrastructure-setup-vm-nginx
infrastructure-setup-vm-nginx:
    "$(MAKE)" --no-print-directory infrastructure-setup-vm VM_NAME=$(VM_NAME_NGINX) ARGS="enable_public_access"

as well as a "combined" target to run the setup in parallel. See also section Use make to execute infrastructure and deployment targets in parallel.

.PHONY: infrastructure-setup-all-vms
infrastructure-setup-all-vms: ## Setup all VMs
    @printf "$(YELLOW)The setup runs in parallel but the output will only be visible once a process is fully finished (can take a couple of minutes)$(NO_COLOR)\n"
    "$(MAKE)" -j --output-sync=target   infrastructure-setup-vm-application \
                                        infrastructure-setup-vm-php-fpm \
                                        infrastructure-setup-vm-php-worker \
                                        infrastructure-setup-vm-nginx \
                                        infrastructure-setup-redis \
                                        infrastructure-setup-mysql

See section Add additional make variables for the definition of variables like $(VM_NAME_PHP_WORKER) and $(VM_NAME_NGINX).

Provisioning the VMs

The original provisioning script explained in Run Docker on GCP Compute Instance VMs: Provisioning located at .infrastructure/scripts/provision.sh stays almost as before, though we were able to remove the docker-compose-plugin dependency as we won't need compose any longer on the production VMs.

In addition, I added some more make targets in .make/04-00-infrastructure.mk to make the provisioning easier

.PHONY: infrastructure-provision-vm
infrastructure-provision-vm: ## Provision the VM specified via VM_NAME. Usage: make infrastructure-provision-vm VM_NAME=php-worker ARGS=""
    @$(if $(VM_NAME),,$(error "VM_NAME is undefined"))
    bash .infrastructure/provision-vm.sh $(GCP_PROJECT_ID) $(VM_NAME) $(ARGS) \
        && printf "$(GREEN)Success at provisioning $(VM_NAME)$(NO_COLOR)\n" \
        || printf "$(RED)Failed provisioning $(VM_NAME)$(NO_COLOR)\n"

.PHONY: infrastructure-provision-vm-nginx
infrastructure-provision-vm-nginx:
    "$(MAKE)" --no-print-directory infrastructure-provision-vm VM_NAME=$(VM_NAME_NGINX)

# ...

.PHONY: infrastructure-provision-all
infrastructure-provision-all: ## Provision all VMs
    @printf "$(YELLOW)The provisioning runs in parallel but the output will only be visible once a process is fully finished (can take a couple of minutes)$(NO_COLOR)\n"
    "$(MAKE)" -j --output-sync=target   infrastructure-provision-vm-application \
                                        infrastructure-provision-vm-php-fpm \
                                        infrastructure-provision-vm-php-worker \
                                        infrastructure-provision-vm-nginx

Mapping service names to VM names

In this tutorial I'm using the terms "service name" and "VM name" quite a lot. To avoid confusion, here is how I think about them:

• service name

  • is used in the sense of a docker compose service

  A Service is an abstract definition of a computing resource within an application which can be scaled/replaced independently from other components. Services are backed by a set of containers, run by the platform according to replication requirements and placement constraints.

  • our application has 6 services (nginx, php-fpm, application, php-worker, mysql and redis) and so far each service was defined as an individual docker image and ran as a single docker container
  • starting from this tutorial, mysql and redis are no longer used via docker images
  • services are a "logical" component - not an infrastructural one
• VM name
  • a "service" can be run on a "VM"
  • is a unique identifier for a Virtual Machine on GCP and is often required in gcloud commands to identify the instance for a command
  • VMs are infrastructure components
  • this includes the "Compute Engine" instances for our application services as well as the MySQL Cloud SQL instance for our mysql service and the Redis Memory Store instance for our redis service

In this tutorial we use "one VM per service" and can thus use a 1-to-1 mapping from service name to VM name. As a convention, we use almost the same name, i.e. the php-fpm service runs on the Compute Engine VM instance with the name php-fpm-vm. Using the -vm suffix avoids confusion and ensures that we don't create an unintended coupling between the service name and the VM name.

Using the exact same name (service name = php-fpm and VM name = php-fpm) has lead to problems for me in the past, because the name of a MySQL Cloud SQL instance used to be blocked even after deletion for one week, so it might not even be possible to use that exact name.

I have defined this mapping as a variable in .make/variables.env as follows:

# must match the names used in the docker-composer.yml files
DOCKER_SERVICE_NAME_NGINX:=nginx
DOCKER_SERVICE_NAME_PHP_BASE:=php-base
DOCKER_SERVICE_NAME_PHP_FPM:=php-fpm
DOCKER_SERVICE_NAME_PHP_WORKER:=php-worker
DOCKER_SERVICE_NAME_APPLICATION:=application
DOCKER_SERVICE_NAME_MYSQL:=mysql
DOCKER_SERVICE_NAME_REDIS:=redis
# VM / instance names
VM_NAME_APPLICATION=$(DOCKER_SERVICE_NAME_APPLICATION)-vm
VM_NAME_PHP_FPM=$(DOCKER_SERVICE_NAME_PHP_FPM)-vm
VM_NAME_PHP_WORKER=$(DOCKER_SERVICE_NAME_PHP_WORKER)-vm
VM_NAME_NGINX=$(DOCKER_SERVICE_NAME_NGINX)-vm
VM_NAME_MYSQL=$(DOCKER_SERVICE_NAME_MYSQL)-vm
VM_NAME_REDIS=$(DOCKER_SERVICE_NAME_REDIS)-vm
# Helpers
ALL_VM_SERVICE_NAMES=$(VM_NAME_APPLICATION):$(DOCKER_SERVICE_NAME_APPLICATION) $(VM_NAME_PHP_FPM):$(DOCKER_SERVICE_NAME_PHP_FPM) $(VM_NAME_PHP_WORKER):$(DOCKER_SERVICE_NAME_PHP_WORKER) $(VM_NAME_NGINX):$(DOCKER_SERVICE_NAME_NGINX)

The ALL_VM_SERVICE_NAMES variables is used for instance for Retrieving all IP addresses if the VMs to create the service-ips file.

Appendix: Changes in the codebase

This section is mostly relevant if you have been following the previous tutorials. It explains some changes that have been introduced for this part.

Add a make dev-init target

A new dev-init target was added in a new sub makefile at .make/00-00-development-setup.mk. It simplifies the setup of the tutorial repository when it has been freshly cloned.

I realized that there are quite some things to keep in mind in this case (e.g. set up various .env files, decrypt the secrets, install composer dependencies, etc.). I believe this can lead to a bad experience if you start with "this" part of the tutorial, as a lot of those setup things are done in previous parts.

Update the gcloud targets

So far, the targets in .make/03-00-gcp.mk could assume a single GCP_VM_NAME variable as there was only one Compute Instance VM. Now, we have one VM per service and thus need to pass the VM_NAME as a required argument to most VM related targets (i.e. GCP_VM_NAME was replaced by VM_NAME).

Example: Logging into a VM via gcp-ssh-login now requires the VM_NAME, because we need to define in which exact VM we want to log in

.PHONY: gcp-ssh-login
gcp-ssh-login: validate-gcp-variables ## Log into a VM via IAP tunnel
    @$(if $(VM_NAME),,$(error "VM_NAME is undefined"))
    gcloud compute ssh $(VM_NAME) --project $(GCP_PROJECT_ID) --zone $(GCP_ZONE) --tunnel-through-iap

# Example to log into the php-fpm VM: 
#   make gcp-ssh-login VM_NAME=php-fpm-vm

In addition, I have added a dedicated target to activate the mater service account, as it is required for some actions (like retrieving the redis AUTH string - see Redis Memorystory Setup):

.PHONY: gcp-init
gcp-init: validate-gcp-variables ## Initialize the `gcloud` cli and authenticate docker with the keyfile defined via SERVICE_ACCOUNT_KEY_FILE.
    @$(if $(SERVICE_ACCOUNT_KEY_FILE),,$(error "SERVICE_ACCOUNT_KEY_FILE is undefined"))
    gcloud auth activate-service-account --key-file="$(SERVICE_ACCOUNT_KEY_FILE)" --project="$(GCP_PROJECT_ID)"

.PHONY: gcp-init-deployment-account
gcp-init-deployment-account: validate-gcp-variables ## Initialize the `gcloud` cli with the deployment service account 
    @$(if $(GCP_DEPLOYMENT_SERVICE_ACCOUNT_KEY_FILE),,$(error "GCP_DEPLOYMENT_SERVICE_ACCOUNT_KEY_FILE is undefined"))
    "$(MAKE)" gcp-init SERVICE_ACCOUNT_KEY_FILE=$(GCP_DEPLOYMENT_SERVICE_ACCOUNT_KEY_FILE)
    cat "$(GCP_DEPLOYMENT_SERVICE_ACCOUNT_KEY_FILE)" | docker login -u _json_key --password-stdin https://gcr.io

.PHONY: gcp-init-master-account
gcp-init-master-account: validate-gcp-variables ## Initialize the `gcloud` cli with the master service account 
    @$(if $(GCP_MASTER_SERVICE_ACCOUNT_KEY_FILE),,$(error "GCP_MASTER_SERVICE_ACCOUNT_KEY_FILE is undefined"))
    "$(MAKE)" gcp-init SERVICE_ACCOUNT_KEY_FILE=$(GCP_MASTER_SERVICE_ACCOUNT_KEY_FILE)

As part of this change, the variable GCP_DEPLOYMENT_SERVICE_ACCOUNT_KEY was renamed to GCP_DEPLOYMENT_SERVICE_ACCOUNT_KEY_FILE and GCP_MASTER_SERVICE_ACCOUNT_KEY_FILE was added.

Add additional make variables

The following variables have been added:

GCP_REGION=us-central1

The region is required to create

• the Cloud Router and Cloud NAT gateway
• the MySQL Cloud SQL instance
• the Redis Memoerystore instance and retrieve its private IP address

GCP_DEPLOYMENT_SERVICE_ACCOUNT_KEY_FILE
GCP_MASTER_SERVICE_ACCOUNT_KEY_FILE

See previous section Update the gcloud targets.

DOCKER_SERVICE_NAME_...
VM_NAME_...
ALL_VM_SERVICE_NAMES

See section Mapping service names to VM names.

Use make to execute infrastructure and deployment targets in parallel

We have already learned about the capability of make to run targets in parallel with the -j flag in Set up PHP QA tools: Parallel execution and a helper target:

.PHONY: foo
foo: 
    "$(MAKE)" -j target-1 target-2

This is particularly helpful when dealing with I/O heavy targets, e.g. when making API calls. This is great, because we are making a lot of those, e.g. during the creation of the infrastructure but also during the deployment of the application. Unfortunately, it's not possible to pass individual arguments per target, i.e. in the following example

.PHONY: infrastructure-setup-all
infrastructure-setup-all: ## Setup all VMs
    "$(MAKE)" -j --output-sync=target infrastructure-setup-vm VM_NAME=application-vm \   
                                      infrastructure-setup-vm VM_NAME=php-fpm-vm

the value of VM_NAME would always be php-fpm-vm. Thus, we must define each target that should run in parallel individually like this:

.PHONY: infrastructure-setup-vm
infrastructure-setup-vm: ## Setup the VM specified via VM_NAME. Usage: make infrastructure-setup-vm VM_NAME=php-worker ARGS=""
    bash .infrastructure/setup-vm.sh $(GCP_PROJECT_ID) $(VM_NAME) $(ARGS)

.PHONY: infrastructure-setup-vm-application
infrastructure-setup-vm-application:
    "$(MAKE)" --no-print-directory infrastructure-setup-vm VM_NAME=$(VM_NAME_APPLICATION)

.PHONY: infrastructure-setup-vm-php-fpm
infrastructure-setup-vm-php-fpm:
    "$(MAKE)" --no-print-directory infrastructure-setup-vm VM_NAME=$(VM_NAME_PHP_FPM)

.PHONY: infrastructure-setup-all
infrastructure-setup-all: ## Setup all VMs
    "$(MAKE)" -j --output-sync=target infrastructure-setup-vm-application \
                                      infrastructure-setup-vm-php-fpm

This introduces some typing overhead, but I didn't find a better way to do this, yet. In theory, we could also use other ways to run the targets in parallel, e.g. by running the process in the background via &

infrastructure-setup-all: ## Provision all VMs
    for vm_name in $(VM_NAME_APPLICATION) $(VM_NAME_PHP_FPM); do \
        $$(make infrastructure-provision-vm VM_NAME="$$vm_name") & \
    done; \
    wait; \
    echo "ALL DONE"

But: This doesn't give us any way to synchronize the output as we can do with make via --output-sync=target so that the output quickly becomes quite messy. Using GNU parallel might be an option, but it doesn't come natively e.g. on Windows and would be a dependency that needed to be installed by all developers (which we try to avoid as much as possible - see Use git-secret to encrypt secrets: Local git-secret and gpg setup).

Wrapping up

Congratulations, you made it! If some things are not completely clear by now, don't hesitate to leave a comment. You should now be ready to create a production ready infrastructure on GCP including multiple VMs and managed services for mysql and redis.

In the next part of this tutorial, we will deploy a dockerized PHP application "to production" via docker (without compose) on multiple VMs.

Please subscribe to the RSS feed or via email to get automatic notifications when this next part comes out :)

Table of contents

• Setup Memorystore
• Create a new redis instance
  • redis AUTH and in-transit encryption
• Connecting to a Redis Memorystore instance
  • Redis Memorystore offers private IP connectivity only
  • Connecting to the redis instance from a Compute Instance VM
  • Connecting to the redis instance via SSH tunnel
• Delete a Redis Memorystore instance
• Using the gcloud CLI
  • Activate the service account
  • Enable the necessary APIs
  • Create an IP range allocation
  • Create the VPC peering with servicenetworking.googleapis.com
  • Create the redis instance
• Wrapping up

Setup Memorystore

The managed solution for in-memory datastores from GCP is called Memorystore and provides multiple datastore technologies - including redis. In the Cloud Console UI it is managed via the Memorystore UI that allows us to create and manage instances.

Create a new redis instance

To get started, we need to enable the following APIs:

• Compute Engine API
• Google Cloud Memorystore for Redis API

Creating a new instance from the Create a redis instance UI is pretty straight forward and well documented in the GCP Redis Guide: Creating and managing Redis instances.

We'll use the following settings:

• Tier Selection: For testing purposes, I recommend choosing the "Basic" option (this will also disable the "Read Replicas")
• Capacity: Enter "1"
• Set up connection > Network: Select the network that the VMs are located in - default in my case
• Additional Configurations > Connections: Select option "Private service access" here as it's the recommended approach.
  • CAUTION: In order to use "Private service access" as connectivity mode, we need to create a reserved IP allocation and a VPC peering with the Google Cloud Platform Service Producer first. The process is exactly the same as I've explained in my MySQL Cloud SQL article for connecting via private IP. Please make sure to pay close attention to section Concrete steps to connect via private IP and especially the video, as it shows the necessary steps to enable the "Private service access" for Memorystore!
• Security > Enable AUTH: Enable the checkbox
  • Note: This will auto-enable the checkbox "Enable in-transit encryption" though I recommend disabling that checkbox (see section redis AUTH and in-transit encryption)
• Configuration > Version: Select "6.x" (which is currently [2023-04-17] the latest version)

FYI: Unfortunately, there is no "EQUIVALENT COMMAND LINE" button as it was the case when creating the Compute Instance - which would have come in handy for creating the instance via gcloud cli.

Once everything is configured, click the "Create Instance" button. The actual creation can take quite some time (I've experienced times from a couple of minutes to ~15 min).

redis AUTH and in-transit encryption

During instance creation we activated the AUTH feature but disabled the in-transit encryption on purpose. Since this goes against GCP's own recommendation

When AUTH is enabled, in-transit encryption is recommended so credentials are confidential when transmitted.

I'd like to provide some thoughts on my reasoning:

According to the GCP Redis Guide: AUTH feature overview > Security and privacy, AUTH is not meant to be used as a security measure:

AUTH helps you ensure that known entities in your organization do not unintentionally access and modify your Redis instance. AUTH does not provide security during data transportation. Also, AUTH does not protect your instance against any malicious entities that have access to your VPC network.

However, it is still certainly "better than not having AUTH at all". Now, in-transit encryption comes at a cost: All communication between redis and the VMs would now be encrypted. Even though this sounds good in theory, it has a negative impact on performance as well as on the maximum number of possible connections.

Thus, I'll go with an in-between solution: Enable AUTH but disable in-transit encryption

Here are some additional things I learned about AUTH:

• you cannot define a custom AUTH string, but it will always be an auto-generated UUID
• the AUTH string will be shown in plain text in the management UI of a redis instance

• you can get the AUTH string via gcloud redis instances get-auth-string

  $ gcloud redis instances get-auth-string redis-instance --region=us-central1
  authString: 568d20ec-b0c2-40a9-908d-a5d6b6717a9c
  

See also the GCP Redis Guides on

• Managing Redis AUTH
• AUTH feature overview > AUTH behavior

Connecting to a Redis Memorystore instance

I'll explain 2 different ways of connecting to a Redis Memorystore instance:

• from a Compute Instance VM on GCP
• "locally" from your laptop via SSH Tunnel

Unfortunately, there is no "One-Click-Solution" like accessing MySQL Cloud SQL instances via Cloud Shell available for Redis.

As always, GCP has also an extensive documentation on the various connection methods available at the GCP Redis Guide: Connecting to a Redis instance.

Redis Memorystore offers private IP connectivity only

Redis Memorystore instances do not have a public IP address!

Regardless of the connection mode, Memorystore for Redis always uses internal IP addresses to provision Redis instances.

(via GCP Redis Guide: Networking)

I.e. it's not possible to connect to an instance without being in the same VPC. This is different from e.g. MySQL Cloud SQL instances, that offer public IPs as an option.

Instead, GCP offers two so-called "Connection modes" for private IP access:

• Direct peering
• Private services access

As Private services access is the newer (and recommended) approach, we'll not cover Direct peering in more detail (even though it's a little easier to set up, because GCP will create the necessary VPC peering automatically).

Connecting to the redis instance from a Compute Instance VM

Once the redis instance is up and running, we can find its private IP address (and port) via its management UI at URL

https://console.cloud.google.com/memorystore/redis/locations/$region/instances/$instanceName/details/overview

# e.g. for an instance named 'redis-1' in region 'us-central1'
# https://console.cloud.google.com/memorystore/redis-1/locations/us-central1/instances/redis/details/overview

Or via gcloud redis instances describe

$ gcloud redis instances describe redis-instance --format="get(host)" --region=us-central1
10.111.1.3

To test the connectivity from a VM, perform the following steps:

• create a Compute Instance VM in the "default" network
• log into the VM via the UI

• install the redis-cli client via

  sudo apt-get install redis-tool -y
  

• connect to the Redis Memorystore instance via

  redis-cli -h $privateIp
  

  where $privateIp is the private IP of the redis instance

• once connected:
  • type the command AUTH followed by the AUTH string of the instance to authenticate
  • type the command PING
  • you should get PONG as response

Connecting to the redis instance via SSH tunnel

The GCP Redis Guide: Connecting from a local machine with port forwarding proposes an interesting approach to connect to a Redis Memorystore instance by using a Compute Instance VM as a jump host, i.e.

• create a VM that lives in the same VPC as the redis instance

• create an SSH tunnel via gcloud comute ssh --ssh-flag

  gcloud compute ssh test-instance --zone=us-central1-a --ssh-flag="-N -L 6379:$privateRedisInstanceIp:6379"
  

  • run this command on your local machine to forward your local port 6379 to port 6379 of the redis instance
  • the traffic flows over the Compute instance VM in this case

• on your local machine connect to the Redis Memorystore instance via

  redis-cli -h localhost
  

Delete a Redis Memorystore instance

To delete a Redis Memorystore instance simply navigate to its management UI and click the "Delete" button. Corresponding docs: GCP Redis Guide: Creating and managing Redis instances > Deleting instances

Using the gcloud CLI

Even though I like using the UI to "explore and understand" how things are working, the goal is always a more "unattended" approach, e.g. via the gcloud cli.

The following commands assume that you have created a master service account with owner permissions and activated it for glcoud with a default project. See also Preconditions: Project and Owner service account

Activate the service account

project_id=pl-dofroscra-p
gcloud auth activate-service-account --key-file=./gcp-master-service-account-key.json --project=${project_id}

Enable the necessary APIs

• gcloud services enable reference

gcloud services enable \
  compute.googleapis.com \
  redis.googleapis.com \
  cloudresourcemanager.googleapis.com \
  servicenetworking.googleapis.com

cloudresourcemanager.googleapis.com is necessary to create the IP range allocation in the next step.

Create an IP range allocation

See "Create an IP range allocation" in the MySQL Cloud SQL article

Create the VPC peering with servicenetworking.googleapis.com

See "Create the VPC peering with servicenetworking.googleapis.com" in the MySQL Cloud SQL article

Create the redis instance

• gcloud redis instances create reference
• GCP Redis Guide: Creating and managing Redis instances > Custom ranges with private services access

region=us-central1
redis_instance_name="redis"
size=1
network="default" # must be the same as for the VMs
version="redis_6_x" # see https://cloud.google.com/sdk/gcloud/reference/redis/instances/create#--redis-version
private_ip_range_name="internal-gcp-services"

gcloud redis instances create "${redis_instance_name}" \
      --size="${size}" \
      --region="${region}" \
      --network="${network}" \
      --redis-version="${version}" \
      --connect-mode=private-service-access \
      --reserved-ip-range="${private_ip_range_name}" \
      --enable-auth \
      -q 

• the --reserved-ip-range must be name of the range created in step Create an IP range allocation

• when using the --enable-auth flag, we need to include the -q flag as well, otherwise we are prompted for an additional confirmation:

  AUTH prevents accidental access to the instance by requiring an AUTH string (automatically generated for you). AUTH credentials are not confidential when transmitted or intended to protect against malicious actors.
  
  Do you want to proceed? (Y/n)?
  

• the --region is required on any subsequent requests to identify the instance, i.e. the instance name is not enough. If the region is not provided, an error is shown:

  $ gcloud redis instances describe redis-instance
  ERROR: (gcloud.redis.instances.describe) Error parsing [instance].
  The [instance] resource is not properly specified.
  Failed to find attribute [region]. The attribute can be set in the following ways:
  - provide the argument `--region` on the command line
  - set the property `redis/region`
  

Retrieve the private IP

• gcloud redis instances create reference

gcloud redis instances describe "${redis_instance_name}" \
    --format="get(host)" \
    --region="${region}"

Retrieve the AUTH string

• gcloud redis instances get-auth-string reference

gcloud redis instances get-auth-string "${redis_instance_name}" \
    --region="${region}"

Wrapping up

Congratulations, you made it! If some things are not completely clear by now, don't hesitate to leave a comment. You are now able to manage redis datastores on GCP via the UI as well as via the gcloud cli.

Table of contents

• Setup Cloud SQL
• Create a new mysql instance
• Connecting to a MySQL Cloud SQL instance
  • Connecting to the mysql instance with Cloud Shell via public IP
  • Connecting to the mysql instance via public IP
  • Connecting to the mysql instance via private IP
    • Concrete steps to connect via private IP
    • Caveats for using a private IP only
• Delete a MySQL Cloud SQL instance
• Using the gcloud CLI
  • Activate the service account
  • Enable the necessary APIs
  • Create an IP range allocation
  • Create the VPC peering with servicenetworking.googleapis.com
  • Create the mysql instance
    • Set the root password
    • Create a default database

Setup Cloud SQL

The managed solution for relational databases from GCP is called Cloud SQL and provides multiple database technologies - including mysql. In the Cloud Console UI it is managed via the SQL UI that allows us to create and manage instances.

Create a new mysql instance

To get started, we need to enable the following APIs:

• Compute Engine API
• Cloud SQL Admin API

Creating a new instance from the Create a MySQL instance UI is pretty straight forward and well documented in the GCP MySQL Guide: Create instances, though there are some configuration options under "Customize your instance" that I want to mention:

• Machine type > Machine type: For testing purposes, I recommend choosing a "Shared core" option here (e.g. 1 vCPU, 0.614 GB) to keep the costs to a minimum
• Connections > Instance IP assignment: For now we'll go with a "Public IP", because setting up private connectivity requires some additional configuration
• Data Protection > Instance deletion protection: When the option Enable deletion protection is enabled, the instance cannot be deleted. The option must first be disabled before this is possible (see also Delete a Cloud SQL instance).

FYI: Unfortunately, there is no "EQUIVALENT COMMAND LINE" button as it was the case when creating the Compute Instance - which would have come in handy for creating the instance via gcloud cli.

Once everything is configured, click the "Create Instance" button. The actual creation can take quite some time (I've experienced times from a couple of minutes to half an hour).

Connecting to a MySQL Cloud SQL instance

I'll explain 3 different ways of connecting to a MySQL Cloud SQL instance:

• via Cloud Shell (requires a public instance IP)
• "locally" from your laptop (requires a public instance IP)
• from a Compute Instance VM on GCP (requires a private instance IP)

Out of scope:

• using the Cloud SQL Auth proxy

As always, GCP has also an extensive documentation on the various connection methods available at GCP MySQL Guide: About connection options.

Connecting to the mysql instance with Cloud Shell via public IP

The easiest way to connect is through Cloud Shell. GCP will spin up a VM "behind the scenes" that we can control via shell session in the browser. Navigate to the management UI of the instance at

https://console.cloud.google.com/sql/instances/$instanceName/overview

# e.g. for an instance named 'mysql-1'
# https://console.cloud.google.com/sql/instances/mysql-1/overview 

and click the "OPEN CLOUD SHELL" button in panel "Connect to this instance".

Please note the

Allowlisting your IP for incoming connection for 5 minutes...working.

step in the video: GCP handles all the "networking stuff" to make the connection possible for us.

Connecting to the mysql instance via public IP

Even though the instance has a public IP address (104.198.240.225 in this case, as shown in the previous video), we can't simply connect to it via

$ mysql -u root -p12345678 -h 104.198.240.225
# ... the request will simply time out

due to firewall rules:

Note: The authorized networks list is implemented on the Cloud SQL instance VM by a local firewall. Learn more about managing connections.

If we want to connect from e.g. our own computer, we need to first create a so-called "Authorized network" as described in the GCP MySQL Guide: Authorize with authorized networks.

For testing purposes I'll retrieve my own public IP address via wget -q -O - ifconfig.me

$ wget -q -O - ifconfig.me
213.21.32.150

and edit the mysql instance to add only this IP address as an Authorized network. The following video shows that the connection isn't possible before I added the Authorized network but will be afterwards.

CAUTION: Please regard this only as a proof of concept. It's not really a viable process to retrieve your own public IP in order to allowlist it "manually". It might be "okay" though, if you are using a VPN with a static IP. However, for the Docker PHP tutorial we'll not use a public IP address at all, because our mysql instance should only be available for our application VMs.

Connecting to the mysql instance via private IP

Having the MySQL Cloud SQL instance exposed via public IP is generally not advised and always carries a certain security risk. Thus, it is desirable to only allow connections via private IP. Be aware that this has some caveats, though (see section Caveats for using a private IP only).

In order to connect to the MySQL Cloud SQL instance via private IP (i.e. from a Compute Instance VM), we must first understand how the network configuration of the "managed" products of GCP is set up. This is explained in detail in the GCP MySQL Guide: Learn about using private IP. In short:

• we use a private network that allows all of our VMs to communicate
• such a network is called VPC (virtual private cloud)
• since "we" (as a user of GCP) create and manage our application VMs ourselves, the VMs live "directly" in this network
• the managed services like Cloud SQL are "doing their own thing", i.e. GCP takes care of all the networking stuff but keeps it isolated from "our" network
• in consequence, our VMs cannot talk to a MySQL Cloud SQL instance, because it lives on a different network
• fortunately, there is a technique called VPC peering that can connect different VPCs and thus enable communication
• to make this work, we need to "reserve" a range of IPs in "our" VPC and make it available to Cloud SQL by peering with the Google Cloud Platform Service Producer

See also the Example in the MySQL Guide: Learn about using private IP

Concrete steps to connect via private IP

In short:

• enable the Service Networking API
• navigate to the "default" network UI
• open tab "PRIVATE SERVICE CONNECTION" and sub tab "ALLOCATED IP RANGES FOR SERVICES" and click the button "ALLOCATE IP RANGE"
  • give it a name, e.g. "google-internal-services"
  • and select option "Automatic" with a prefix length of 16 (this determines the number of possible Cloud SQL instances, see also the docs on Allocated range size)
• open tab "PRIVATE SERVICE CONNECTION" and sub tab "PRIVATE CONNECTIONS TO SERVICES" and click the button "CREATE CONNECTION"
  • choose "Google Cloud Platform" in the "Connected service producer" drop down field and select the "google-internal-services" range that we just created for the "Assigned allocation" drop down field
• once the "peering" is done, you can also observe a corresponding entry in the VPC network peering UI
• create a new MySQL Cloud SQL instance (or modify an existing one)
  • under Connections > Instance IP assignment unselect "Public IP" and select "Private IP" select the network "default"
  • Note: The instance is being shutdown and restarted, i.e. this operation will lead to a service disruption and can take up to several minutes
• once the update is done, navigate to the MySQL Cloud SQL UI of the instance and find the private IP in panel "Private IP address"

To test the connectivity from a VM, perform the following steps:

• create a Compute Instance VM in the "default" network
• log into the VM via the UI

• install the mysql client via

  sudo apt-get install default-mysql-client -y
  

• connect to the MySQL Cloud SQL instance via

  mysql -h $privateIp -u root -p
  

  where $privateIp is the private IP of the mysql instance

See also the GCP MySQL docs:

• Configure private services access
• Configure private IP

Caveats for using a private IP only

• we need to provide some additional networking configuration (explained above)

• it's no longer possible to connect via Cloud Shell:

  Cloud Shell doesn't currently support connecting to a Cloud SQL instance that has only a private IP address.

• once a private IP is assigned, it cannot be un-assigned:

  After you configure an instance to use private IP, you cannot disable private IP connectivity for that instance.

  If you choose to let Cloud SQL allocate your private IP for an instance, the addresses for all instances you later configure in that VPC network are automatically allocated in the same IP address range.

• we can no longer connect from "our" computer, because it doesn't live in the same private network. This can be problematic if you are used to using UI tools like MySQL Workbench to manage your database - please check out the following articles if this is relevant for you:
  • Cloud SQL with private IP only: the Good, the Bad and the Ugly
  • How to Deploy a Cloud SQL DB with a Private IP only, using Terraform
  • GCP | How to access Cloud SQL private IP using Cloud SQL Auth Proxy and Identity-Aware Proxy (IAP)?

But tbh, I would probably rather create an SSH tunnel with the gcloud cli on a small Compute Instance VM via

gcloud compute ssh $computeInstanceName --zone=$computeInstanceZone -- -N -L 3306:$mysqlInstanceIp:3306

Delete a MySQL Cloud SQL instance

Once again, GCP is doing a great job documenting the process to delete a MySQL Cloud SQL instance in the GCP MySQL Guide: Delete instances.

Please give it a read as there is at least once thing that I found quite un-intuitive: Even after deleting an instance, the name of the deleted instance is "blocked" for a week. In my case, this created some problems, because I was making assumptions about the name in other parts of my code (e.g. I was expecting that the name of the instance is always mysql), but when playing around with Cloud SQL I learned too late, that I couldn't re-use the name after deleting the instance for testing purposes.

Note: If "Deletion protection" is enabled, it needs to be disabled first - which is usually quite a fast operation.

See the following video for a visual representation of the process (please ignore the "blocked" instance name issue at the end):

Using the gcloud CLI

Even though I like using the UI to "explore and understand" how things are working, the goal is always a more "unattended" approach, e.g. via the gcloud cli. Since I'm gonna use a MySQL Cloud SQL instance as replacement for a mysql docker container for the next part of the Docker PHP tutorial, I'll focus on creating an instance with a private IP address only, update the root password and set up a default database.

The following commands assume that you have created a master service account with owner permissions and activated it for glcoud with a default project. See also Preconditions: Project and Owner service account

Activate the service account

project_id=pl-dofroscra-p
gcloud auth activate-service-account --key-file=./gcp-master-service-account-key.json --project=${project_id}

Enable the necessary APIs

• gcloud services enable reference

gcloud services enable \
  compute.googleapis.com \
  sqladmin.googleapis.com \
  cloudresourcemanager.googleapis.com \
  servicenetworking.googleapis.com

cloudresourcemanager.googleapis.com is necessary to create the IP range allocation in the next step.

Create an IP range allocation

• gcloud compute addresses create reference

private_ip_range_name="internal-gcp-services"
network="default"
ip_range_network_address="10.111.0.0"

gcloud compute addresses create "${private_ip_range_name}" \
  --global \
  --purpose=VPC_PEERING \
  --prefix-length=16 \
  --addresses="${ip_range_network_address}" \
  --description="Peering range for GCP Services" \
  --network="${network}"

The --addresses flag can be used to define the network address of the defined range. If it is not given, GCP will choose a random one.

Create the VPC peering with servicenetworking.googleapis.com

• gcloud services vpc-peerings connect reference

private_ip_range_name="internal-gcp-services"
network="default"

gcloud services vpc-peerings connect \
  --service=servicenetworking.googleapis.com \
  --ranges="${private_ip_range_name}" \
  --network="${network}"

Create the mysql instance

• gcloud beta sql instances create reference
• GCP MySQL Guide: Create instances
• GCP MySQL Guide: Configure an instance to use private IP

Caution: The --allocated-ip-range-name option is currently (2023-04-13) only available in the beta tools of the gcloud cli. You might need to install them first via gcloud components install beta.

mysql_instance_name="mysql"
version="MYSQL_8_0" # see https://cloud.google.com/sdk/gcloud/reference/sql/instances/create#--database-version
cpus="1"
memory="3840MB"
region="us-central1"
private_ip_range_name="internal-gcp-services"
network="default"

gcloud beta sql instances create "${mysql_instance_name}" \
  --database-version="${version}" \
  --cpu="${cpus}" \
  --memory="${memory}" \
  --region="${region}" \
  --network="${network}" \
  --no-assign-ip \
  --allocated-ip-range-name="${private_ip_range_name}"

The --no-assign-ip flag is responsible for not assigning a public IP.

Set the root password

• gcloud sql users set-password reference
• GCP MySQL Guide: Manage users with built-in authentication: Change a user password

mysql_instance_name="mysql"
root_password=12345678

gcloud sql users set-password root \
  --host=% \
  --instance "${mysql_instance_name}" \
  --password "${root_password}"

Using --host=% will allow access from any remote host, see docu for the --host parameter

Cloud SQL user's host name expressed as a specific IP address or address range. % denotes an unrestricted host name. Applicable flag for MySQL instances; ignored for all other engines.

FYI: Instead of "root", we could also use a different username like "application_user" and MySQL would create that user automatically with the given password (without any privileges).

Create a default database

• gcloud sql databases create reference
• GCP MySQL Guide: Create and manage databases: Create a database on the Cloud SQL instance

mysql_instance_name="mysql"
default_database="application_db"

gcloud sql databases create "${default_database}" \
  --instance="${mysql_instance_name}" \
  --charset=utf8mb4 \
  --collation=utf8mb4_unicode_ci

Wrapping up

Congratulations, you made it! If some things are not completely clear by now, don't hesitate to leave a comment. You are now able to manage MySQL databases on GCP via the UI as well as via the gcloud cli.

Table of contents

• Introduction
• How to install and update Git Bash / MINGW / MSYS2 via Git for Windows
  • Update MINGW
  • How to install make
  • Configuration via .bashrc
• Common issues
  • The role of winpty: Fixing "The input device is not a TTY"
  • The path conversion issue
    • Fixing the path conversion issue for MINGW / MSYS2
    • Fixing the path conversion issue for winpty
• Miscellaneous
  • Change the bash custom prompt to a $

Introduction

When I was learning git I started with the fantastic Git for Windows package, that is maintained in the git-for-windows/git Github repository and comes with Git Bash, a shell that offers a Unix-terminal like experience. It uses MINGW and MSYS2 under the hood and does not only provide git but also a bunch of other common Linux utilities like

bash
sed
awk
ls
cp
rm
...

I believe the main "shell" is actually powered by MINGW64 as that's what will be shown by default:

Thus, I will refer to the tool as MINGW shell or Git Bash throughout this article.

I have been using MINGW for almost 10 years now, and it is still my go-to shell for Windows. I could just never warm up to WSL, because the file sharing performance between WSL and native Windows files was (is?) horrible - but that's a different story.

How to install and update Git Bash / MINGW / MSYS2 via Git for Windows

You can find the latest Git for Windows installation package directly at the homepage of https://gitforwindows.org/. Older releases can be found on Github in the Releases section of the git-for-windows/git repository

Follow the instructions in the How to Install Git Bash on Windows article on git-tower.com to get a guided tour through the setup process.

After the installation is finished, I usually create a desktop icon and assign the shortcut CTRL + ALT + B (for "bash") so that I can open a new shell session conveniently via keyboard.

Update MINGW

To update Git for Windows, you can simply run

git update-git-for-windows

See also the Git for Windows FAQ under "How do I update Git for Windows upon new releases?"

Git for Windows comes with a tool to check for updates and offer to install them. Whether or not you enabled auto-updates during installation, you can manually run git update-git-for-windows.

You can check the current version via git version

$ git --version
git version 2.37.2.windows.2

How to install make

As per How to add more to Git Bash on Windows: make:

• Go to ezwinports
• Download file make-4.3-without-guile-w32-bin.zip (get the version without guile)
• Extract zip

• Copy the contents to your Git/mingw64/ directory, merging the folders, but do NOT overwrite/replace any existing files

  • navigate to the Git/mingw64/ directory via

  $(cd /; explorer .)
  

Test via make version

$ make --version
GNU Make 4.3.1
Built for Windows32
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

PS: There's also an alternative way that I've outlined in Install make on Windows (MinGW), though the one explained here is easier/faster.

Configuration via .bashrc

The MINGW shell is a bash shell and can thus be configured via a .bashrc file located at the home directory of the user. The shell supports the ~ character as an alias for the home directory, i.e. you can use ~/.bashrc to refer to the full path of the file. This means you can also edit it easily via vi ~/.bashrc - though I prefer an actual GUI editor like Notepad++. A common workflow for me to open the file is running the following commands in a MINGW shell session

# navigate to to the home directory
cd ~

# open the file explorer
explorer .

My .bashrc file usually includes the following setup:

# Get bash completion for make targets by parsing make files in the current directory at 
#  the file "Makefile"
#  all files with a ".mk" suffix in the folders ".make" and ".makefile"
# see https://stackoverflow.com/questions/4188324/bash-completion-of-makefile-target
# Notes:
#  -h hides filenames
#  -s hides error messages
complete -W "\`grep -shoE '^[a-zA-Z0-9_.-]+:([^=]|$)' Makefile .make/*.mk .makefile/*.mk | sed 's/[^a-zA-Z0-9_.-]*$//' | grep -v PHONY\`" make

# Docker login helper
# see https://www.pascallandau.com/blog/structuring-the-docker-setup-for-php-projects/#easy-container-access-via-din-bashrc-helper
function din() {
  filter=$1

  user=""
  if [[ -n "$2" ]];
  then
    user="--user $2"
  fi

  shell="bash"
  if [[ -n "$3" ]];
  then
    shell=$3
  fi

  prefix=""
  if [[ "$(expr substr $(uname -s) 1 5)" == "MINGW" ]]; then
    prefix="winpty"
  fi
  ${prefix} docker exec -it ${user} $(docker ps --filter name=${filter} -q | head -1) ${shell}
}

Links:

• SO: bash completion of makefile target
• Easy container access via din .bashrc helper

Common issues

The Git for Windows Known Issues page lists common problems with Git Bash and I want to provide some more context (and solutions) to the things that I have encountered.

The role of winpty: Fixing "The input device is not a TTY"

I encountered the The input device is not a TTY error while using docker. To log into a running docker container or starting a container with a login session, the -i (Keep STDIN open even if not attached) and -t (Allocate a pseudo-tty) options must be given:

For interactive processes (like a shell), you must use -i -t together in order to allocate a tty for the container process. -i -t is often written -it.

But attempting to do so via

docker run --rm -it busybox sh

yields the following error:

$ docker run --rm -it busybox sh
the input device is not a TTY.  If you are using mintty, try prefixing the command with 'winpty'

Fortunately, the fix is included in the message: Prefix the command with winpty. Doing so works as expected:

$ winpty docker run --rm -it busybox sh
/ #

winpty is according to it's readme

[...] a Windows software package providing an interface similar to a Unix pty-master for communicating with Windows console programs. The package consists of a library (libwinpty) and a tool for Cygwin and MSYS for running Windows console programs in a Cygwin/MSYS pty.

So kind of a translator between your "Windows input" and the "command input" to create input that is compatible with a Unix pty (pty=pseudoterminal interface), e.g. for docker.

According to the Git for Windows Known Issues page, there are a number of other cases where winpty is required (though I personally didn't encounter them yet):

Some console programs, most notably non-MSYS2 Python, PHP, Node and OpenSSL, interact correctly with MinTTY only when called through winpty (e.g. the Python console needs to be started as winpty python instead of just python).

CAUTION: I've seen people put an alias in their .bashrc file to always prefix docker commands automatically with winpty like so:

alias docker="winpty docker"

However, winpty seems to break piping and can lead to unexpected results like the error stdout is not a tty. See the following example:

$ docker run --rm busybox echo "foo" | cat
foo

$ winpty docker run --rm busybox echo "foo" | cat
stdout is not a tty

You might work around this by adding the (undocumented) -Xallow-non-tty flag like so

$ winpty -Xallow-non-tty docker run --rm busybox echo "foo" | cat
foo

But this doesn't seem to be a catch-all solution and I would recommend against using it as a default - or if you do, only use it when the -it flag is used as proposed in this answer.

The path conversion issue

Ah. This one has given me lots of headaches over the years. MINGW, MSYS2 and winpty use automatic conversion of Unix paths to Windows paths, e.g. /foo gets translated to something like C:/Program Files/Git/foo where C:/Program Files/Git/ is the installation directory of the Git for Windows installation.

Fixing the path conversion issue for MINGW / MSYS2

First, the behavior is mentioned on the Git for Windows Known Issues page

If you specify command-line options starting with a slash, POSIX-to-Windows path conversion will kick in converting e.g. "/usr/bin/bash.exe" to "C:\Program Files\Git\usr\bin\bash.exe". When that is not desired -- e.g. "--upload-pack=/opt/git/bin/git-upload-pack" or "-L/regex/" -- you need to set the environment variable MSYS_NO_PATHCONV temporarily, like so:

MSYS_NO_PATHCONV=1 git blame -L/pathconv/ msys2_path_conv.cc

Alternatively, you can double the first slash to avoid POSIX-to-Windows path conversion, e.g. "//usr/bin/bash.exe".

and also documented for MINGW at "Posix path conversion", but it's still brought up regularly, see e.g. GH #3619: "/" is replaced with the directory path of Git installation when using MinGW64 Bash. or SO: How to stop MinGW and MSYS from mangling path names given at the command line

Example

$ docker run --rm busybox ls /foo
ls: C:/Program Files/Git/foo: No such file or directory

As quoted above, it can be solved by either

• adding an additional / to the path

  $ docker run --rm busybox ls //foo
  ls: /foo: No such file or directory
  

• prefixing the command with MSYS_NO_PATHCONV=1

  $ MSYS_NO_PATHCONV=1 docker run --rm busybox ls /foo
  ls: /foo: No such file or directory
  

• or exporting the MSYS_NO_PATHCONV=1 variable as an environment variable to disable the behavior completely

  $ export MSYS_NO_PATHCONV=1
  $ docker run --rm busybox ls /foo
  ls: /foo: No such file or directory
  

CAUTION: The value of the MSYS_NO_PATHCONV variable does not matter - we can also set it to 0, false or an empty string. It only matters that the variable is defined!

$ MSYS_NO_PATHCONV=0 docker run --rm busybox ls /foo
ls: /foo: No such file or directory

This is particularly important when using the environment variable approach. In order to selectively enable the path conversion again, you must unset the MSYS_NO_PATHCONV first via env -u MSYS_NO_PATHCONV ..., e.g.

$ env -u MSYS_NO_PATHCONV docker run --rm busybox ls /foo
ls: C:/Program Files/Git/foo: No such file or directory

CAUTION: I've seen people adding MSYS_NO_PATHCONV=1 permanently to their environment in their .bashrc file to always disable path conversion via

export MSYS_NO_PATHCONV=1

However, this can have some unintended side effects. When I tried it out, my local installation of the gcloud cli stopped working with the error

$ MSYS_NO_PATHCONV=1 gcloud version
C:\Users\Pascal\AppData\Local\Programs\Python\Python39\python.exe: can't open file 'C:\c\Users\Pascal\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\gcloud.py': [Errno 2] No such file or directory

So instead I recommend setting MSYS_NO_PATHCONV=1 either selectively per command or scope it to the use case. I do this for example in my Makefiles by only exporting it for the scope of make (and all scripts make invokes) by putting the following code in the beginning of the Makefile:

# OS is a defined variable for WIN systems, so "uname" will not be executed
OS?=$(shell uname)
# Values of OS:
#   Windows => Windows_NT 
#   Mac     => Darwin 
#   Linux   => Linux 
ifeq ($(OS),Windows_NT)
    export MSYS_NO_PATHCONV=1
endif

The path conversion is also documented for MSYS2 at "Filesystem Paths: Automatic Unix ⟶ Windows Path Conversion" and can be disabled via the MSYS2_ARG_CONV_EXCL environment variable:

[...] For these cases you can exclude certain arguments via the MSYS2_ARG_CONV_EXCL environment variable: [...] MSYS2_ARG_CONV_EXCL can either be * to mean exclude everything, or a list of one ore more arguments prefixes separated by ;, like MSYS2_ARG_CONV_EXCL=--dir=;--bla=;/test. It matches the prefix against the whole argument string.

I.e. setting the variable as MSYS2_ARG_CONV_EXCL="*" should disable the path conversion completely. I myself have never had to use this, though. Using MSYS_NO_PATHCONV was always sufficient.

Fixing the path conversion issue for winpty

Unfortunately, winpty suffers from this path conversion issue as well. In the standard installation of Git for Windows we can even see this by simply using echo:

$ winpty echo /
C:/Program Files/Git

The behavior is known and flagged as a bug e.g. in GH issue #411: Path conversion with and without winpty differs.

Remember the example I gave in section The role of winpty e.g. when using docker?

$ winpty docker run --rm -it busybox sh
/ #

Now let's extend this and throw a volume into the mix:

$ winpty docker run --rm -v foo:/foo -it busybox sh
docker: Error response from daemon: create foo;C: "foo;C" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path.

winpty converts /foo to C:/Program Files/Git/foo so that the volume definition becomes -v foo:C:/Program Files/Git/foo - which is of course invalid.

Using an additional / as a prefix does work here as well:

$ winpty docker run --rm -v foo://foo -it busybox sh
/ #

But there is no environment variable that we could use. The only way to "fix" the path conversion is using a newer release of winpty and replace the one that is shipped together with Git for Windows as proposed by the maintainer of winpty.

This comment outlines the full process to replace winpty and is (slightly adapted) as follows:

# create temporary directory
mkdir temp
cd temp
# download a newer release
curl -L https://github.com/rprichard/winpty/releases/download/0.4.3/winpty-0.4.3-msys2-2.7.0-x64.tar.gz --output winpty.tar.gz
# extract the archive
tar -xvf winpty.tar.gz
# copy the content of the bin/ folder to `/usr/bin` 
# (which resolves to e.g `C:/Program Files/Git/usr/bin`; replaces any existing files)
cp winpty-0.4.3-msys2-2.7.0-x64/bin/* /usr/bin
# delete the temporary directory
cd ..
rm -rf temp/

Once the new version is installed, the path conversion does not happen any longer (even without specifying any environment variables).

Related comments:

• https://github.com/docker/for-win/issues/1588#issuecomment-594938988
• https://github.com/docker/for-win/issues/1588#issuecomment-698080757

Caution After updating MinGW, the fix for winpty is "gone"!

I.e. you need to re-run the steps above every time you run an update.

Miscellaneous

Some stuff that I need from time to time - not necessarily only relevant for Git Bash.

Change the bash custom prompt to a $

Via PS1=" $":

Pascal@LAPTOP-0DNL2Q02 MINGW64 ~
$ PS1="$ "
$

See How to Change / Set up bash custom prompt (PS1) in Linux

All code samples are publicly available in my Docker PHP Tutorial repository on Github.
You find the branch with the final result of this tutorial at part-9-deploy-docker-compose-php-gcp-poc.

All published parts of the Docker PHP Tutorial are collected under a dedicated page at Docker PHP Tutorial. The previous part was Create a GCP Compute Instance VM for dockerized PHP Apps.

If you want to follow along, please subscribe to the RSS feed or via email to get automatic notifications when the next part comes out :)

Table of contents

• Introduction
• Deployment workflow
  • The deploy target
  • Avoiding code drift
  • The build-info file
  • Build and push the docker images
  • Create the deployment archive
  • Deployment commands on the VM
    • The deploy.sh script
• Codebase changes
  • Restructure the codebase
    • The .build directory
    • The .secrets directory
    • The .tutorial directory
    • The .infrastructure directory
  • Add a gpg key for production
  • Show the build-info
  • Optimize .gitignore
• Docker changes
  • A .env file for prod
  • Updating the docker-compose.yml configuration files
    • docker-compose.local.ci.prod.yml
    • docker-compose.local.prod.yml
    • docker-compose.prod.yml
  • Adjust the .dockerignore file
  • Build target: prod
    • Build stage prod in the php-base image
      • ENV based branching
      • Avoid composer dev dependencies
      • Remove unnecessary directories
      • Remove secrets for other environments
      • Decrypt the secrets via ENTRYPOINT
      • Copy codebase and build-info file
    • Build stage prod in the remaining images
• Makefile changes
  • Adding GCP values to .make/variables.env
  • ENV based docker compose config
  • Changes to the git-secret recipes
  • Additional docker recipes
  • GCP recipes
  • Infrastructure recipes
  • Deployment recipes
• Wrapping up

Introduction

In the previous tutorial Create a GCP compute instance VM to run dockerized applications we have created a Compute Instance VM on GCP and prepared it to run docker containers. For this tutorial I made a small adjustment and changed the machine type from e2-micro to e2-small because we need a little more memory to run the whole application.

In this tutorial, we will use the VM as a production environment, i.e. we will

• prepare our docker setup for production usage
• build and push the production-ready docker images to the GCP registry from our local system
• pull and start the images on the VM

The whole process will be defined in a single make target called deploy.

To try it yourself:

• create an account on GCP, a project and a master service account
  • create a keyfile for the service account, name it gcp-master-service-account-key.json and move it to the root of the repository
• checkout branch part-9-deploy-docker-compose-php-gcp-poc
• update the .make/variables.env file with your GCP project id and VM name
• initialize local docker setup via
  • copying the secret gpg key to the root of the repository via bash cp .tutorial/secret.gpg.example ./secret.gpg
  • initializing the shared variables via make make-init
  • building the docker setup via make docker-build
  • start the docker setup via make docker-up
  • decrypt the secrets via make gpg-init make secret-decrypt
• run the script located at .infrastructure/setup-gcp.sh to create a GCP VM
• run make deploy IGNORE_UNCOMMITTED_CHANGES=true to deploy the application
• run make deployment-setup-db-on-vm to run the DB migrations
• run make gcp-show-ip to retrieve the public IP of the VM and open it in a browser

project_id=my-new-project100
vm_name=my-vm-name

git checkout part-9-deploy-docker-compose-php-gcp-poc
sed -i "s/pl-dofroscra-p/${project_id}/g" .make/variables.env
sed -i "s/dofroscra-test/${vm_name}/g" .make/variables.env
cp .tutorial/secret.gpg.example ./secret.gpg
make make-init
make docker-build
make docker-up
make gpg-init
make secret-decrypt
bash .infrastructure/setup-gcp.sh $project_id $vm_name
make deploy IGNORE_UNCOMMITTED_CHANGES=true
make deployment-setup-db-on-vm
echo "http://$(make -s gcp-show-ip)/"

Note: It can take a couple of minutes until the infrastructure is up and running.

Note: We will tackle those issues and "remove" the POC status in the next part of the tutorial series.

Deployment workflow

As a precondition we expect that a GCP VM is up and running. The basic idea of the deployment is:

• build the docker images using the prod environment and push them to the remote registry
• log into the VM and pull the images
• use docker compose on the VM to start the docker setup

This shouldn't be too complicated - we already do the same thing locally, don't we? In theory: Yes. In practice, there is one major difference: Locally, we have access to our repository, including the files for running

• the docker setup (=> the .docker directory)
• the make setup to control the application (=> the Makefile and the .make directory)

Fortunately we can solve this issue easily and provide a single make target named deploy that will take care of everything.

The deploy target

The deploy target runs all necessary commands for a deployment:

Run safeguard checks to avoid code drift

    @printf "$(GREEN)Switching to 'local' environment$(NO_COLOR)\n"
    @make --no-print-directory make-init
    @printf "$(GREEN)Starting docker setup locally$(NO_COLOR)\n"
    @make --no-print-directory docker-up
    @printf "$(GREEN)Verifying that there are no changes in the secrets$(NO_COLOR)\n"
    @make --no-print-directory gpg-init
    @make --no-print-directory deployment-guard-secret-changes
    @printf "$(GREEN)Verifying that there are no uncommitted changes in the codebase$(NO_COLOR)\n"
    @make --no-print-directory deployment-guard-uncommitted-changes

Make sure the gcloud cli is initialized with the GCP deployment service account and that this account is also used to authenticate docker. Otherwise, we won't be able to push images to our GCP container registry.

    @printf "$(GREEN)Initializing gcloud$(NO_COLOR)\n"
    @make --no-print-directory gcp-init

Enable the prod environment for the make setup, see section ENV based docker compose config

    @printf "$(GREEN)Switching to 'prod' environment$(NO_COLOR)\n"
    @make --no-print-directory make-init ENVS="ENV=prod TAG=latest"

Create the build-info file

    @printf "$(GREEN)Creating build information file$(NO_COLOR)\n"
    @make --no-print-directory deployment-create-build-info-file

Build and push the docker images

    @printf "$(GREEN)Building docker images$(NO_COLOR)\n"
    @make --no-print-directory docker-build
    @printf "$(GREEN)Pushing images to the registry$(NO_COLOR)\n"
    @make --no-print-directory docker-push

Create the deployment archive

    @printf "$(GREEN)Creating the deployment archive$(NO_COLOR)\n"
    @make deployment-create-tar

Run deployment commands on the VM

    @printf "$(GREEN)Copying the deployment archive to the VM and run the deployment$(NO_COLOR)\n"
    @make --no-print-directory deployment-run-on-vm

Cleanup the deployment by removing the local deployment archive and enabling the default environment (local) for the make setup.

    @printf "$(GREEN)Clearing deployment archive$(NO_COLOR)\n"
    @make --no-print-directory deployment-clear-tar
    @printf "$(GREEN)Switching to 'local' environment$(NO_COLOR)\n"
    @make --no-print-directory make-init

Avoiding code drift

The term "code drift" is derived from configuration drift, which indicates the (subtle) differences in configuration between environments:

If you've ever heard an engineer lamenting (or sometimes arrogantly proclaiming) "well, it works on my machine" then you have been witness to configuration drift.

In our case it refers to differences between our git repository and the code in the docker images as well as changes between the decrypted and encrypted secret files. These problems can occur, because we are currently executing the deployment from our local machine and we might have made some changes in the codebase when we build the docker images that are not yet reflected in git. The build context sent to the docker daemon would then be different from the git repository resp. the encrypted .secret files. This can lead to all sorts of hard-to-debug quirks and should thus be avoided.

When we deploy later from the CI pipelines, those problems simply won't occur, because the whole codebase will be identical with the git repository - but I really do NOT want to lose the ability to deploy code from my local system (devs that went through Gitlab / Github downtimes will understand...)

Corresponding checks are implemented via the deployment-guard-uncommitted-changes and deployment-guard-secret-changes targets that exit with exit 1 (a non-zero status code) which in turn makes the deploy target stop/fail.

IGNORE_SECRET_CHANGES?=

.PHONY: deployment-guard-secret-changes
deployment-guard-secret-changes: ## Check if there are any changes between the decrypted and encrypted secret files
    if ( ! make secret-diff || [ "$$(make secret-diff | grep ^@@)" != "" ] ) && [ "$(IGNORE_SECRET_CHANGES)" == "" ] ; then \
        printf "Found changes in the secret files => $(RED)ABORTING$(NO_COLOR)\n\n"; \
        printf "Use with IGNORE_SECRET_CHANGES=true to ignore this warning\n\n"; \
        make secret-diff; \
        exit 1; \
    fi
    @echo "No changes in the secret files!"

make secret-diff is used to check for differences between decrypted and encrypted secrets.

! make secret-diff checks if the commands exits with a non-zero exit code. This happens for instance, when the secrets have not been decrypted yet. The error is

git-secret: abort: file not found. Consider using 'git secret reveal': <missing-file>

If the command doesn't fail, the changes are displayed in a diff format, e.g.

 --- /dev/fd/63
 +++ /var/www/app/.secrets/shared/passwords.txt
 @@ -1 +1,2 @@
  my_secret_password
 +1
 foo

We use grep ^@@ to check the existence of a "line that starts with @@" to identify a change. If no changes are found, the info "No changes in the secret files!" is printed. Otherwise, a warning is shown. The check an be suppressed by passing IGNORE_SECRET_CHANGES=true.

IGNORE_UNCOMMITTED_CHANGES?=

.PHONY: deployment-guard-uncommitted-changes
deployment-guard-uncommitted-changes: ## Check if there are any git changes and abort if so. The check can be ignore by passing `IGNORE_UNCOMMITTED_CHANGES=true`
    if [ "$$(git status -s)" != "" ] && [ "$(IGNORE_UNCOMMITTED_CHANGES)" == "" ] ; then \
        printf "Found uncommitted changes in git => $(RED)ABORTING$(NO_COLOR)\n\n"; \
        printf "Use with IGNORE_UNCOMMITTED_CHANGES=true to ignore this warning\n\n"; \
        git status -s; \
        exit 1; \
    fi
    @echo "No uncommitted changes found!"

For deployment-guard-uncommitted-changes we use git status -s to check for any uncommitted changes. If no changes are found the info "No uncommitted changes found!" is printed. Otherwise, a warning is shown. The check an be suppressed by passing IGNORE_UNCOMMITTED_CHANGES=true.

The build-info file

When testing the deployments I often needed to identify small bugs in the code. The more complex the whole process gets, the more things can go wrong and the more "stuff needs to be checked". One of them is the code drift mentioned in the previous section, btw.

To make my life a little easier, I added a file called build-info that contains information about the build and will be stored in the docker images - allowing me to inspect the file later, see also section Show the build-info.

The file is created via deployment-create-build-info-file target

.PHONY: deployment-create-build-info-file
deployment-create-build-info-file: ## Create a file containing version information about the codebase
    @echo "BUILD INFO" > ".build/build-info"
    @echo "==========" >> ".build/build-info"
    @echo "User  :" $$(whoami) >> ".build/build-info"
    @echo "Date  :" $$(date --rfc-3339=seconds) >> ".build/build-info"
    @echo "Branch:" $$(git branch --show-current) >> ".build/build-info"
    @echo "" >> ".build/build-info"
    @echo "Commit" >> ".build/build-info"
    @echo "------" >> ".build/build-info"
    @git log -1 --no-color >> ".build/build-info"

The file is created on the host system under .build/build-info and then copied to ./build-info in the Dockerfile of the php-base image. To execute a shell command via $(command), the $ has to be escaped with another $, to not be interpreted by make as a variable. Example:

some-target:
    $$(command)

FYI: I learned that make converts all new lines in spaces when they are echo'd because I initially used

@echo $$(git log -1 --no-color) >> ".build/build-info"

instead of

@git log -1 --no-color >> ".build/build-info"

which would remove all new lines.

A final file build-info file looks like this:

BUILD INFO
==========
User  : Pascal
Date  : 2022-05-22 17:10:21+02:00
Branch: part-9-deploy-docker-compose-php-gcp-poc

Commit
------
commit c47464536613874d192696d93d3c97b138c7a6be
Author: Pascal Landau <pascal.landau@googlemail.com>
Date:   Sun May 22 17:10:15 2022 +0200

    Testing the new `build-info` file

Build and push the docker images

make is initialized with ENV=prod, i.e. calling make docker-build will use the correct docker compose config for building production images. In addition, we have adjusted the DOCKER_REGISTRY to gcr.io/pl-dofroscra-p in the .make/variables.env file, so that the images will immediately be tagged correctly as

gcr.io/pl-dofroscra-p/dofroscra/$service-prod

# e.g. for `php-base`

gcr.io/pl-dofroscra-p/dofroscra/php-base-prod

Example:

$ make docker-build
ENV=prod TAG=latest DOCKER_REGISTRY=gcr.io/pl-dofroscra-p DOCKER_NAMESPACE=dofroscra APP_USER_NAME=application APP_GROUP_NAME=application docker compose -p dofroscra_prod --env-file ./.docker/.env -f ./.docker/docker-compose/docker-compose-php-base.yml build php-base
#1 [internal] load build definition from Dockerfile
# ...

$ docker image ls 
REPOSITORY                                          TAG     IMAGE ID       CREATED          SIZE
gcr.io/pl-dofroscra-p/dofroscra/php-fpm-prod        latest  2be3bec977de   24 seconds ago   147MB
gcr.io/pl-dofroscra-p/dofroscra/php-worker-prod     latest  6dbf14d1b329   25 seconds ago   181MB
gcr.io/pl-dofroscra-p/dofroscra/php-base-prod       latest  9164976a78a6   32 seconds ago   130MB
gcr.io/pl-dofroscra-p/dofroscra/application-prod    latest  377fdee0f12a   32 seconds ago   130MB
gcr.io/pl-dofroscra-p/dofroscra/nginx-prod          latest  42dd1608d126   24 seconds ago   23.5MB

Thanks to the image name, we can also immediately push the images to the remote registry via make docker-push. Note, that we see a lot of Layer already exists infos in the console output for the php-fpm and php-worker images. This is due to the fact that we use a common php-base base image for application, php-fpm and php-worker, i.e. those images have a lot of layers in common and only the layers of application are pushed. docker uses the layer hash to identify which layers already exist.

$ make docker-push
ENV=prod TAG=latest DOCKER_REGISTRY=gcr.io/pl-dofroscra-p DOCKER_NAMESPACE=dofroscra APP_USER_NAME=application APP_GROUP_NAME=application docker compose -p dofroscra_prod --env-file ./.docker/.env -f ./.docker/docker-compose/docker-compose.local.ci.prod.yml -f ./.docker/docker-compose/docker-compose.local.prod.yml -f ./.docker/docker-compose/docker-compose.prod.yml push
mysql Skipped
redis Skipped
Pushing application: c8f4416c4383 Preparing
#...
Pushing application: 6bbfa8829d07 Pushing [==================================================>]  3.584kB
#...
Pushing php-worker: 6bbfa8829d07 Layer already exists
#...
Pushing php-fpm: 6bbfa8829d07 Layer already exists

Create the deployment archive

As described in the introduction of the Deployment workflow, we need to make our make and docker setup somehow available on the VM. We will solve this issue by creating a tar archive with all necessary files locally and transfer it to the VM.

The archive is created via the deployment-create-tar target

.PHONY: deployment-create-tar
deployment-create-tar:
    # create the build directory
    rm -rf .build/deployment
    mkdir -p .build/deployment
    # copy the necessary files
    mkdir -p .build/deployment/.docker/docker-compose/
    cp -r .docker/docker-compose/ .build/deployment/.docker/
    cp -r .make .build/deployment/
    cp Makefile .build/deployment/
    cp .infrastructure/scripts/deploy.sh .build/deployment/
    # make sure we don't have any .env files in the build directory (don't wanna leak any secrets) ...
    find .build/deployment -name '.env' -delete
    # ... apart from the .env file we need to start docker
    cp .secrets/prod/docker.env .build/deployment/.docker/.env
    # create the archive
    tar -czvf .build/deployment.tar.gz -C .build/deployment/ ./

The recipe uses the .build/deployment directory as a temporary location to store all necessary files, i.e.

• the docker compose config files in .docker/docker-compose/
• the Makefile and the .make directory in the root of the codebase for the make setup
• the .infrastructure/scripts/deploy.sh script to run the deployment

In addition, we copy the .secrets/prod/docker.env file to use it as the .env file for docker compose. Caution: This only works, because we have verified previously that there are no changes between the decrypted and encrypted .secret files (which also means that .secrets/prod/docker.env is already decrypted).

Once all files are copied, the whole directory is added to the .build/deployment.tar.gz archive via

tar -czvf .build/deployment.tar.gz -C .build/deployment/ ./

The -C .build/deployment/ option makes sure that the directory structure is retained when extracting the archive. For the remaining options take a look at How to create tar.gz file in Linux using command line.

Deployment commands on the VM

Once the creation of the deployment archive is done, we can transfer the resulting .build/deployment.tar.gz file to the VM, extract it and run the deployment script. All of that is done via the deployment-run-on-vm target

# directory on the VM that will contain the files to start the docker setup
CODEBASE_DIRECTORY=/tmp/codebase

.PHONY: deployment-run-on-vm
deployment-run-on-vm:## Run the deployment script on the VM
    "$(MAKE)" -s gcp-scp-command SOURCE=".build/deployment.tar.gz" DESTINATION="deployment.tar.gz"
    "$(MAKE)" -s gcp-ssh-command COMMAND="sudo rm -rf $(CODEBASE_DIRECTORY) && sudo mkdir -p $(CODEBASE_DIRECTORY) && sudo tar -xzvf deployment.tar.gz -C $(CODEBASE_DIRECTORY) && cd $(CODEBASE_DIRECTORY) && sudo bash deploy.sh"

Under the hood, the target uses the gcp-scp-command and gcp-ssh-command targets. The deployment archive is extracted in /tmp/codebase via

sudo rm -rf /tmp/codebase && sudo mkdir -p /tmp/codebase && sudo tar -xzvf deployment.tar.gz -C /tmp/codebase

and then the deployment script is executed

cd /tmp/codebase && sudo bash deploy.sh

All of those commands are run in a single invocation of gcp-ssh-command, because there's a certain overhead involved when tunneling commands via IAP, i.e. each invocation takes a couple of seconds.

The deploy.sh script

The actual deployment is done "on the VM" via the .infrastructure/scripts/deploy.sh script

#!/usr/bin/env bash

echo "Retrieving secrets"
make gcp-secret-get SECRET_NAME=GPG_KEY > secret.gpg
GPG_PASSWORD=$(make gcp-secret-get SECRET_NAME=GPG_PASSWORD)
echo "Creating compose-secrets.env file"
echo "GPG_PASSWORD=$GPG_PASSWORD" > compose-secrets.env
echo "Initializing the codebase"
make make-init ENVS="ENV=prod TAG=latest"
echo "Pulling images on the VM from the registry"
make docker-pull
echo "Stop the containers on the VM"
make docker-down || true
echo "Start the containers on the VM"
make docker-up

The script is located at the root of the codebase and

• will first retrieve the GPG_KEY and the GPG_PASSWORD values that we created previously in the Secret Manager
  • the GPG_KEY is stored in the file secret.gpg in the root of the codebase so that it is picked up automatically when initializing gpg > [...] > and the private key has to be named secret.gpg and put in the root of the codebase.
  • the GPG_PASSWORD is stored in the bash variable $GPG_PASSWORD and from there stored in a file called compose-secrets.env as dotenv GPG_PASSWORD=$GPG_PASSWORD This file is used in the docker-compose.prod.yml file as env_file for the php docker containers, so that the GPG_PASSWORD becomes available in the decrypt-secrets.sh script used in the ENTRYPOINT of the php-base image
• then the make setup is initialized for the prod environment via bash make-init ENVS="ENV=prod TAG=latest" so that all subsequent docker-* targets use the correct configuration
• and finally, the docker images we pushed in step "Build and push the docker images" are pulled, any running containers are stopped and the whole docker setup is started with the new images
  • FYI: docker compose down would fail (exit with a non-zero status code) if no containers are running. Since this is fine for us (we simply want to ensure that no containers are running), the command is OR'd via make docker-down || true so that the script won't stop if that happens.

Codebase changes

Before we dive into the docker stuff, let's quickly talk about some cleanup work in the codebase that you can get via part-9-deploy-docker-compose-php-gcp-poc.

Restructure the codebase

The .build directory

We already know this directory from the previous tutorial where we used it as a temporary directory to collect build artifacts from the CI pipeline. Now, we will make use of it again as a temporary directory to

• prepare the creation of the deployment archive
• create a build-info file to pass it to the docker daemon in the build context

The files in the directory are ignored via .gitignore as they are only temporarily required

# File: .gitignore

.build/*

However, since the build-info file must be passed to docker, we will have a slight deviation between the .gitignore and the .dockerignore file.

# File: .dockerignore

.build/*

# kept files
!.build/build-info

I'm mentioning this here specifically, because we usually strive for a parity between .gitignore and .dockerignore.

The .secrets directory

Since we will store all secrets for all environments in the codebase, we will organize them by environment as subdirectories in a new .secrets directory:

.secrets/
├── ci
│   └── ci-secret.txt.secret
├── prod
│   ├── app.env.secret
│   └── docker.env.secret
└── shared
    └── passwords.txt.secret

This will also make it easier to pick the correct files per environment when building the docker image and select them for decryption in the ENTRYPOINT.

The .secrets/shared/ directory contains all secret files that are required by all environments, whereas .secrets/ci/ contains only ci secrets and .secrets/prod/ contains only prod secrets, respectively.

In our codebase there are already two files that contain actual secrets: The .env file and the .docker/.env file. Both of them contain the credentials for mysql and redis, and the .env file also contains the APP_KEY that is used by Laravel to encrypt cookies

Laravel uses the [APP_KEY] for all encrypted cookies, including the session cookie, before handing them off to the user's browser, and it uses it to decrypt cookies read from the browser.

# File: .env

APP_KEY=base64:C8X1hLE2bpok8OS+bJ1cTB9wNASJNRLibqUrDq2ls4Q=
DB_PASSWORD=production_secret_mysql_root_password
REDIS_PASSWORD=production_secret_redis_password

# File: .docker/.env

MYSQL_PASSWORD=production_secret
MYSQL_ROOT_PASSWORD=production_secret_mysql_root_password
REDIS_PASSWORD=production_secret_redis_password

I have created the encrypted .secret files by moving the unencrypted files to the .secrets/ directory and made sure to add them to the .gitignore file with the rules

# ...

.secrets/*/*
!**/*.secret

Then I ran

make secret-add FILES=".secrets/*/*"

make secret-encrypt

$ make secret-add FILES=".secrets/*/*"
"C:/Program Files/Git/mingw64/bin/make" -s git-secret ARGS="add .secrets/*/*"
git-secret: 4 item(s) added.

$ make secret-encrypt
"C:/Program Files/Git/mingw64/bin/make" -s git-secret ARGS="hide"
git-secret: done. 4 of 4 files are hidden.

The .tutorial directory

I have mentioned before, that I would normally not store secret gpg keys in the repository, but I'm still doing it in this tutorial so that it's easier to follow along. To make clear which files are affected by this "exception to the rule", I have moved them in a dedicated .tutorial directory:

.tutorial/
├── secret-ci-protected.gpg.example
├── secret-production-protected.gpg.example
└── secret.gpg.example

The .infrastructure directory

The .infrastructure directory contains all files that are used to manage the infrastructure and deployments. The directory was introduced in the previous part and looks as follows

.infrastructure/
├── scripts/
│     ├── deploy.sh
│     └── provision.sh
└── setup-gcp.sh

• the scripts directory contains files that are transferred to and then executed on the VM
  • deploy.sh is a script to perform all necessary deployment steps on the VM and is explained in more detail in section Deployment commands on the VM
  • provision.sh is a script to install docker and docker compose on the VM and contains the commands that are explained in section Installing docker and docker compose
• setup-gcp.sh is run to set up a VM initially and was described in more detail under Putting it all together

Add a gpg key for production

We have created a new gpg key pair for the prod environment when setting up the VM on GCP. The secret key is located at .tutorial/secret-production-protected.gpg.example and the public key at .dev/gpg-keys/production-public.gpg.

Show the build-info

As part of the deployment, we generate a build-info file that allows us to understand "which version of the codebase lives inside a container". This file is located at the root of the repository, and we expose it as the web route /info (via the php-fpm container) and as the command info (via the application container).

routes/web.php

# ...

Route::get('/info', function () {
    $info = file_get_contents(__DIR__."/../build-info");
    return new \Illuminate\Http\Response($info, 200, ["Content-type" => "text/plain"]);
});

Show via curl http://localhost/info

routes/console.php

# ...

Artisan::command('info', function () {
    $info = file_get_contents(__DIR__."/../build-info");
    $this->line($info);
})->purpose('Display build information about the codebase');

Show via php artisan info

Optimize .gitignore

Laravel uses multiple .gitignore files to retain a directory structure, because empty directories cannot be added to git. This is a valid strategy, but it makes understanding "what is actually ignored" more complex.

In addition, it makes it harder to keep .gitignore and .dockerignore in sync, because we can't simply "copy the contents of the ./.gitignore" any longer as it might not contain all rules.

Thus, I have identified all directory-specific .gitignore files via

find . -path ./vendor -prune -o -name .gitignore -print

find . -path ./vendor -prune -o -name .gitignore -print
./.gitignore
./bootstrap/cache/.gitignore
./database/.gitignore
./storage/app/.gitignore
./storage/app/public/.gitignore
./storage/framework/.gitignore
./storage/framework/cache/.gitignore
./storage/framework/cache/data/.gitignore
./storage/framework/sessions/.gitignore
./storage/framework/testing/.gitignore
./storage/framework/views/.gitignore
./storage/logs/.gitignore

CAUTION: Usually, the files simply contain the rules

*
!.gitignore

The only exception is ./database/.gitignore which contains

*.sqlite*

TBH, I would consider this rather a bug, because this rule SHOULD actually live the main .gitignore file. But for us it means, that we need to keep this rule as

database/*.sqlite*

Before we add them to the .gitignore file, it is important to understand how git handles the gitignore file

An optional prefix "!" which negates the pattern; any matching file excluded by a previous pattern will become included again. It is not possible to re-include a file if a parent directory of that file is excluded. Git doesn’t list excluded directories for performance reasons, so any patterns on contained files have no effect, no matter where they are defined.

In other words: Consider the following structure

storage/
└── app
    ├── .gitignore
    └── public
        └── .gitignore

We might be tempted to write the rules as

storage/app/*
!storage/app/.gitignore
!storage/app/public/.gitignore

But that won't work as expected: storage/app/* ignores "everything" in the storage/app/ directory, so that git wouldn't even look into the storage/app/public/ directory and thus wouldn't find the storage/app/public/.gitignore file! In consequence, the rule !storage/app/public/.gitignore doesn't have any affect and the file would not be added to the git repository.

Instead, I need to allow the public directory explicitly by adding the rule !storage/app/public/ and ignore all files in it apart from the .gitignore file via storage/app/public/*. So we are essentially saying:

• ignore all files in the storage/app/ directory
• but NOT the storage/app/public/ directory ITSELF
• though do still ignore all files IN the storage/app/public/ directory
• but NOT the storage/app/public/.gitignore file

# ignore all files in the storage/app/ directory
storage/app/*
!storage/app/.gitignore
# but NOT the storage/app/public/ directory ITSELF
!storage/app/public/
# do still ignore all files IN storage/app/public/
storage/app/public/*
# but NOT the storage/app/public/.gitignore file
!storage/app/public/.gitignore

We can simplify the rules a little more via !**.gitignore, i.e. all gitignore files in any directory should be included

storage/app/*
!storage/app/public/
storage/app/public/*

!**.gitignore

So the final rules for all directory-specific .gitignore files become

bootstrap/cache/*
database/*.sqlite*
storage/app/*
!storage/app/public/
storage/app/public/*
storage/framework/*
!storage/framework/cache/
storage/framework/cache/*
!storage/framework/cache/data/
storage/framework/cache/data/*
!storage/framework/sessions/
storage/framework/sessions/*
!storage/framework/testing/
storage/framework/testing/*
!storage/framework/views/
storage/framework/views/*
!storage/framework/logs/
storage/framework/logs/*

!**.gitignore

Caution: If Laravel changes the rules for their directory-specific .gitignore files, we must adjust our rules as well! Luckily this usually only happens on major version upgrades, though.

The full .gitignore file becomes

**/*.env*
!.env.example
!.make/variables.env
.idea
.phpunit.result.cache
vendor/
secret.gpg
.gitsecret/keys/random_seed
.gitsecret/keys/pubring.kbx~
.secrets/*/*
!**/*.secret
gcp-service-account-key.json
gcp-master-service-account-key.json

# => directory-specific .gitignore files by Laravel
bootstrap/cache/*
database/*.sqlite*
storage/app/*
!storage/app/public/
storage/app/public/*
storage/framework/*
!storage/framework/cache/
storage/framework/cache/*
!storage/framework/cache/data/
storage/framework/cache/data/*
!storage/framework/sessions/
storage/framework/sessions/*
!storage/framework/testing/
storage/framework/testing/*
!storage/framework/views/
storage/framework/views/*
!storage/framework/logs/
storage/framework/logs/*

# => directory-specific .gitignore files from us
.build/*

# => keep ALL .gitignore files
!**.gitignore

It will be "synced" later with the .dockerignore file.

Docker changes

For this tutorial we will use docker compose to build the containers as well as to run them "in production", i.e. on the GCP VM. The docker compose configuration will essentially be a combination of the local and ci config from the previous tutorials:

• We will need all the services that we already know from the local environment, i.e.
  • nginx
  • mysql
  • redis
  • php-worker
  • php-fpm
  • application
• The codebase will be baked into the image and we will bind mount a gpg secret key at runtime to decrypt the secrets - exactly as we did in the previous tutorial with the ci environment

In the previous tutorial we have used the environment (ci) to identify the necessary docker-compose.yml configuration files and also as a new build target in the Dockerfiles. We'll stick to this process and add yet another environment called prod.

A .env file for prod

For our local and ci environments we didn't really care about the .env file that we used for docker compose and have simply added a ready-to-use template at .docker/.env.example that is NOT ignored by git. Even though the file contains credentials, e.g. for mysql and redis, it's okay if those are "exposed" in the repository, because we won't store any production data in those databases in local and ci.

For prod however, the situation is different: We certainly do NOT want to expose the credentials. Luckily we already have git secret set up and thus could simply add an encrypted template file at .secrets/prod/docker.env. We will later decrypt the file and add it to the deployment archive to transfer it to the VM in order to start the docker setup there.

Updating the docker-compose.yml configuration files

We will use the same technique as before to "assemble" our docker compose configuration, i.e. we use multiple compose files with environment specific settings. For prod, we use the files

• docker-compose.local.ci.prod.yml
  • contains config settings for all environments
• docker-compose.local.prod.yml
  • contains config settings only for local and prod
• docker-compose.prod.yml
  • contains config settings only for prod

The assembling is once again performed via Makefile.

docker-compose.local.ci.prod.yml

This file was simply renamed from file docker-compose.local.ci.yml. It contains

• network and volume definitions

  networks:
    network:
      driver: ${NETWORKS_DRIVER?}
  
  volumes:
    mysql:
      name: mysql-${ENV?}
      driver: ${VOLUMES_DRIVER?}
    redis:
      name: redis-${ENV?}
      driver: ${VOLUMES_DRIVER?}
  

• the build instructions for the application service yaml application: image: ${DOCKER_REGISTRY?}/${DOCKER_NAMESPACE?}/application-${ENV?}:${TAG?} build: context: ../ dockerfile: ./images/php/application/Dockerfile target: ${ENV?} args: - BASE_IMAGE=${DOCKER_REGISTRY?}/${DOCKER_NAMESPACE?}/php-base-${ENV?}:${TAG?} - ENV=${ENV?}

• the configuration for the mysql and redis services

  mysql:
    image: mysql:${MYSQL_VERSION?}
    platform: linux/amd64
    environment:
      - MYSQL_DATABASE=${MYSQL_DATABASE:-application_db}
      - MYSQL_USER=${MYSQL_USER:-application_user}
      - MYSQL_PASSWORD=${MYSQL_PASSWORD?}
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD?}
      - TZ=${TIMEZONE:-UTC}
    networks:
      - network
    healthcheck:
      test: mysqladmin ping -h 127.0.0.1 -u $$MYSQL_USER --password=$$MYSQL_PASSWORD
      timeout: 1s
      retries: 30
      interval: 2s
  
  redis:
    image: redis:${REDIS_VERSION?}
    command: >
      --requirepass ${REDIS_PASSWORD?}
    networks:
      - network
  

docker-compose.local.prod.yml

This file is based on the docker-compose.local.yml of the previous tutorial without any settings for local development. It contains

• build instructions for the php-fpm, php-worker and nginx services (because we didn't need those in the configuration for the ci environment)

  php-fpm:
    image: ${DOCKER_REGISTRY?}/${DOCKER_NAMESPACE?}/php-fpm-${ENV?}:${TAG?}
    build:
      context: ../
      dockerfile: ./images/php/fpm/Dockerfile
      target: ${ENV?}
      args:
        - BASE_IMAGE=${DOCKER_REGISTRY?}/${DOCKER_NAMESPACE?}/php-base-${ENV?}:${TAG?}
        - TARGET_PHP_VERSION=${PHP_VERSION?}
  
  php-worker:
    image: ${DOCKER_REGISTRY?}/${DOCKER_NAMESPACE?}/php-worker-${ENV?}:${TAG?}
    build:
      context: ../
      dockerfile: ./images/php/worker/Dockerfile
      target: ${ENV?}
      args:
        - BASE_IMAGE=${DOCKER_REGISTRY?}/${DOCKER_NAMESPACE?}/php-base-${ENV?}:${TAG?}
        - PHP_WORKER_PROCESS_NUMBER=${PHP_WORKER_PROCESS_NUMBER:-4}
  
  nginx:
    image: ${DOCKER_REGISTRY?}/${DOCKER_NAMESPACE?}/nginx-${ENV?}:${TAG?}
    build:
      context: ../
      dockerfile: ./images/nginx/Dockerfile
      target: ${ENV?}
      args:
        - NGINX_VERSION=${NGINX_VERSION?}
        - APP_CODE_PATH=${APP_CODE_PATH_CONTAINER?}
    ports:
      - "${NGINX_HOST_HTTP_PORT:-80}:80"
      - "${NGINX_HOST_HTTPS_PORT:-443}:443"
  

• port forwarding for the nginx service, because we want to forward incoming requests on the VM to the nginx docker container

  nginx:
    ports:
      - "${NGINX_HOST_HTTP_PORT:-80}:80"
      - "${NGINX_HOST_HTTPS_PORT:-443}:443"
  

• volume configuration for the mysql and redis services

  mysql:
    volumes:
      - mysql:/var/lib/mysql
  
  redis:
    volumes:
      - redis:/data
  

The following settings are only retained in docker-compose.local.yml

• bind mount of the codebase

  application|php-fpm|php-worker|nginx:
    volumes:
      - ${APP_CODE_PATH_HOST?}:${APP_CODE_PATH_CONTAINER?}
  

• port sharing with the host system (excluding nginx ports)

  application:
    ports:
      - "${APPLICATION_SSH_HOST_PORT:-2222}:22"
  
  mysql:
    ports:
      - "${MYSQL_HOST_PORT:-3306}:6379"
  
  redis:
    ports:
      - "${REDIS_HOST_PORT:-6379}:6379"
  

• any settings for local dev tools for all php images (application, php-fpm, php-worker)

  application|php-fpm|php-worker:
    environment:
      - PHP_IDE_CONFIG=${PHP_IDE_CONFIG?}
    cap_add:
      - "SYS_PTRACE"
    security_opt:
      - "seccomp=unconfined"
    extra_hosts:
      host.docker.internal:host-gateway  
  

docker-compose.prod.yml

In this file,

• we bind-mount the secret gpg key into all php services (as we did in the previous tutorial for the docker-compose.ci.yml file)

    volumes:
      - ${APP_CODE_PATH_HOST?}/secret.gpg:${APP_CODE_PATH_CONTAINER?}/secret.gpg:ro
  

• we provide an env file for all php services

    env_file:
      - ../../compose-secrets.env
  

  The env file is used to pass the GPG_PASSWORD as environment variable to the containers. This is required to decrypt the secrets on container start. See section Decrypt the secrets via ENTRYPOINT for a more in-depth explanation of the process and The deploy.sh script for the origin of the compose-secrets.env file.

Adjust the .dockerignore file

Since we have modified the .gitignore file , we need to adjust the .dockerignore file as well. In addition to the gitignore rules, we need three additional rules:

1. .git: Known from the previous tutorial. We don't need to transfer the .git directory to the build context
2. !.build/build-info: We don't need the content of the .build/ directory, but we do need the the build-info file
3. vendor/**: Has to be added as a workaround for the docker compose bug Inconsistent ".dockerignore" behavior between "docker build" and "docker compose build" that causes docker compose build to include .gitignore files in the vendor/ directory. This would mess with the build of the php-base image as we expect that no vendor/ folder exists

So the full .dockerignore file becomes

# ---
# Rules from .gitignore
# ---
**/*.env*
!.env.example
!.make/variables.env
.idea
.phpunit.result.cache
vendor/
secret.gpg
.gitsecret/keys/random_seed
.gitsecret/keys/pubring.kbx~
.secrets/*/*
!**/*.secret
.build/*
!.gitkeep
gcp-service-account-key.json

# => directory-specific .gitignore files by Laravel
bootstrap/cache/*
database/*.sqlite*
storage/app/*
!storage/app/public/
storage/app/public/*
storage/framework/*
!storage/framework/cache/
storage/framework/cache/*
!storage/framework/cache/data/
storage/framework/cache/data/*
!storage/framework/sessions/
storage/framework/sessions/*
!storage/framework/testing/
storage/framework/testing/*
!storage/framework/views/
storage/framework/views/*
!storage/framework/logs/
storage/framework/logs/*

# => directory-specific .gitignore files from us
.build/*

# => keep ALL .gitignore files
!**.gitignore

# ---
# Rules specifically for .dockerignore
# ---

# => don't transfer the git directory
.git

# => keep the build-info file
!.build/build-info

# [WORKAROUND]
# temporary fix for https://github.com/docker/compose/issues/9508
# Otherwise, `docker compose build` would transfer the `.gitignore` files 
# in the vendor/ directory to the build context
vendor/**

Build target: prod

For building the images for the prod environment, we stick to the technique of the previous tutorial once again. In short:

• initialize the make setup for ENV=prod via bash make make-init ENVS="ENV=prod" so that all subsequent make invocations use ENV=prod by default, see also section Initialize the shared variables of the previous tutorial
• the ENV is passed as environment variable to the docker compose command as defined in the .make/02-00-docker.mk Makefile include AND the config files for prod are selected bash ENV=prod docker compose -p -f ./.docker/docker-compose/docker-compose.local.ci.prod.yml -f ./.docker/docker-compose/docker-compose.local.prod.yml -f ./.docker/docker-compose/docker-compose.prod.yml
• the ENV environment variable is used to define the target property as well as the image name in the docker compose config files. This can be verified via make docker-config, for instance text $ make docker-config services: application: build: target: prod image: gcr.io/pl-dofroscra-p/dofroscra/application-prod:latest # ...

For ci we only needed the php-base images as well as the application image. For prod however, we need all images.

Build stage prod in the php-base image

We will re-use a lot of the code that we used in the previous tutorial to build the php-base image - so I'd recommend giving the corresponding section Build stage ci in the php-base image a look if anything is not clear.

The relevant parts of the .docker/images/php/base/Dockerfile will be explained in the following sections but are also shown here for the sake of a better overview:

ARG ALPINE_VERSION
ARG COMPOSER_VERSION
FROM composer:${COMPOSER_VERSION} as composer
FROM alpine:${ALPINE_VERSION} as base

ARG APP_USER_NAME
ARG APP_GROUP_NAME
ARG APP_CODE_PATH
ARG TARGET_PHP_VERSION
ARG ENV
ENV APP_USER_NAME=${APP_USER_NAME}
ENV APP_GROUP_NAME=${APP_GROUP_NAME}
ENV APP_CODE_PATH=${APP_CODE_PATH}
ENV TARGET_PHP_VERSION=${TARGET_PHP_VERSION}
ENV ENV=${ENV}

RUN apk add --no-cache php8~=${TARGET_PHP_VERSION}

COPY --from=composer /usr/bin/composer /usr/local/bin/composer

WORKDIR $APP_CODE_PATH

FROM base as codebase

# By only copying the composer files required to run composer install
# the layer will be cached and only invalidated when the composer dependencies are changed
COPY ./composer.json /dependencies/
COPY ./composer.lock /dependencies/

# use a cache mount to cache the composer dependencies
# this is essentially a cache that lives in Docker BuildKit (i.e. has nothing to do with the host system) 
RUN --mount=type=cache,target=/tmp/.composer \
    cd /dependencies && \
    # COMPOSER_HOME=/tmp/.composer sets the home directory of composer that
    # also controls where composer looks for the cache 
    # so we don't have to download dependencies again (if they are cached)
    # @see https://stackoverflow.com/a/60518444 for the correct if-then-else syntax:
    # - end all commands with ; \
    # - except THEN and ELSE
    if [ "$ENV" == "prod" ] ; \
    then \
      # on production, we don't want test dependencies
      COMPOSER_HOME=/tmp/.composer composer install --no-scripts --no-plugins --no-progress -o --no-dev; \
    else  \
      COMPOSER_HOME=/tmp/.composer composer install --no-scripts --no-plugins --no-progress -o; \
    fi

# copy the full codebase
COPY ../_blog /codebase

# move the dependencies
RUN mv /dependencies/vendor /codebase/vendor

# remove files we don't require in the image to keep the image size small
RUN cd /codebase && \
    rm -rf .docker/ .build/ .infrastructure/ && \
    if [ "$ENV" == "prod" ] ; \
    then \
      # on production, we don't want tests
      rm -rf tests/; \
    fi

# Remove all secrets that are NOT required for the given ENV:
#  `find /codebase/.secrets -type f -print` lists all files in the .secrets directory
#  `grep -v "/\(shared\|$ENV\)/"` matches only the files that are NOT in the shared/ or $ENV/ (e.g. prod/) directories
#  `grep -v ".secret\$"` ensures that we remove all files that are NOT ending in .secret
#    FYI: 
#     the "$" has to be escaped with a "\" 
#     "Escaping is possible by adding a \ before the variable"
#     @see https://docs.docker.com/engine/reference/builder/#environment-replacement
#  `xargs rm -f` retrieves the remaining file and deletes them
#    FYI: 
#     `xargs` is necessary to convert the stdin to args for `rm`
#     @see https://stackoverflow.com/a/20307392/413531
#     the `-f` flag is required so that `rm` doesn't fail if no files are matched
RUN find /codebase/.secrets -type f -print | grep -v "/\(shared\|$ENV\)/" | xargs rm -f && \
    find /codebase/.secrets -type f -print | grep -v ".secret\$" | xargs rm -f && \
    # list the remaining files for debugging purposes
    find /codebase/.secrets -type f -print

# We need a git repository for git-secret to work (can be an empty one)
RUN cd /codebase && \
    git init

FROM base as prod

# We will use a custom ENTRYPOINT to decrypt the secrets when the container starts.
# This way, we can store the secrets in their encrypted form directly in the image.
# Note: Because we defined a custom ENTRYPOINT, the default CMD of the base image
#       will be overriden. Thus, we must explicitly re-define it here via `CMD ["/bin/sh"]`.
#       This behavior is described in the docs as:
#       "If CMD is defined from the base image, setting ENTRYPOINT will reset CMD to an empty value. In this scenario, CMD must be defined in the current image to have a value."
#       @see https://docs.docker.com/engine/reference/builder/#understand-how-cmd-and-entrypoint-interact
COPY ./.docker/images/php/base/decrypt-secrets.sh /decrypt-secrets.sh
RUN chmod +x /decrypt-secrets.sh
CMD ["/bin/sh"]
ENTRYPOINT ["/decrypt-secrets.sh"]

COPY --from=codebase --chown=$APP_USER_NAME:$APP_GROUP_NAME /codebase $APP_CODE_PATH

COPY --chown=$APP_USER_NAME:$APP_GROUP_NAME ./.build/build-info $APP_CODE_PATH/build-info

ENV based branching

So far, we have used the ENV to determine the final build stage of the docker image, by using it as value for the target property in the docker compose config files. This introduces a certain level of flexibility, but we would be forced to duplicate code if the same logic is required for multiple ENV that use different build stages.

For RUN statements we can achieve branching on a more granular level by using if...else conditions. The syntax is described in
this SO answer to "Dockerfile if else condition with external arguments":

• place a \ at the end of each line
• end each command with ;

Example:

RUN if [ "$ENV" == "prod" ] ; \
    then \
      echo "ENV is prod"; \
    else \
      echo "ENV is NOT prod"; \
    fi

This works, because we don't just use the ENV as the build target, but we also pass it as a build argument in the .docker/docker-compose/docker-compose-php-base.yml file:

services:
  php-base:
    image: ${DOCKER_REGISTRY?}/${DOCKER_NAMESPACE?}/php-base-${ENV?}:${TAG?}
    build:
      args:
        - ENV=${ENV?}
      target: ${ENV?}

Avoid composer dev dependencies

A production image should only contain the dependencies that are necessary to run the code in production. This explicitly excludes dependencies that are only required for development or testing.

This isn't only about image size, but also about security. E.g. take a look at CVE-2017-9841 - a remote code execution vulnerability in phpunit. In a blog post (german) Sebastian Bergmann (creator of PHPUnit) mentions that

A dependency like PHPUnit, that is only required for the developing the software but not running it, is not supposed to be deployed to the production system

Thus, we will add the --no-dev flag of the composer install command for ENV=prod:

FROM base as codebase

# ...

RUN --mount=type=cache,target=/tmp/.composer \
    cd /dependencies && \
    if [ "$ENV" == "prod" ] ; \
    then \
      COMPOSER_HOME=/tmp/.composer composer install --no-scripts --no-plugins --no-progress -o --no-dev; \
    else  \
      COMPOSER_HOME=/tmp/.composer composer install --no-scripts --no-plugins --no-progress -o; \
    fi

Note: See also section Build the dependencies of the previous tutorial for an explanation of the --mount=type=cache,target=/tmp/.composer part.

Remove unnecessary directories

We already learned that no unnecessary stuff should end up in the image. This doesn't stop at composer dependencies but does in our case also include some other directories:

• .docker (the docker setup files)
• .build (used to pass the build-info file)
• .infrastructure (see The .infrastructure directory)
• tests (the test files)

FROM base as codebase

# ...

RUN cd /codebase && \
    rm -rf .docker/ .build/ .infrastructure/ && \
    if [ "$ENV" == "prod" ] ; \
    then \
      rm -rf tests/; \
    fi

Remove secrets for other environments

We have re-organized the secrets previously so that all secrets for an environment are located in the corresponding .secrets/$ENV/ subdirectory. We can now make use of that separation to keep only the secret files that we actually need. This adds an additional layer of security, because everybody with a correct secret gpg key file can decrypt all the secrets. But: If our "prod secrets" don't even exist in the "ci images" they also cannot be leaked if ci is compromised.

FROM base as codebase

# ...

RUN find /codebase/.secrets -type f -print | grep -v "/\(shared\|$ENV\)/" | xargs rm -f && \
    find /codebase/.secrets -type f -print | grep -v ".secret\$" | xargs rm -f && \
    # list the remaining files for debugging purposes
    find /codebase/.secrets -type f -print

Notes:

• find /codebase/.secrets -type f -print lists all files in the .secrets directory
• grep -v "/\(shared\|$ENV\)/" matches only the files that are NOT in the shared/ or $ENV/ (e.g. prod/) directories
• grep -v ".secret\$" ensures that we remove all files that are NOT ending in .secret
  • as per documentation, the $ has to be escaped with a \ > "Escaping is possible by adding a \ before the variable"
• xargs rm -f retrieves the remaining file and deletes them
  • xargs is necessary to convert the stdin to arguments for rm
  • the -f flag is required so that rm doesn't fail if no files are matched

Decrypt the secrets via ENTRYPOINT

In the CI pipeline setup we decrypt the secret files in a manual step after the containers have been started, see Run details:

[...] then, the docker setup is started [...] and gpg is initialized so that the secrets can be decrypted

   make gpg-init
   make secret-decrypt-with-password

It "works" as long as no secrets are required when the container starts. This is unfortunately no longer the case, because the php-worker container will start its workers immediately, and they require a valid .env file - but the .env file for prod (app.env) is only stored encrypted in the image. Thus, we must ensure that the secrets are decrypted as soon as the container starts - preferably before any other command is run. This sounds like the perfect job for ENTRYPOINT:

the ENTRYPOINT is executed every time we run a container. Some things can't be done during build but only at runtime [...] - ENTRYPOINT is a good solution for that problem

In our case, the ENTRYPOINT should

• initialize gpg (i.e. "read the secret gpg key") via bash make gpg-init
• decrypt the secrets via bash make secret-decrypt
• move/copy the decrypted files if necessary
  bash cp .secrets/prod/app.env .env

and I have created a corresponding script at .docker/images/php/base/decrypt-secrets.sh:

#!/usr/bin/env bash

# exit immediately on error
set -e

# initialize make
make make-init ENVS="ENV=prod GPG_PASSWORD=$GPG_PASSWORD"

# read the secret gpg key
make gpg-init

# Only decrypt files required for production
files=$(make secret-list | grep "/\(shared\|prod\)/" | tr '\n' ' ')
make secret-decrypt-with-password FILES="$files"

cp .secrets/prod/app.env .env

# treat this script as a "decorator" and execute any other command after running it
# @see https://www.pascallandau.com/blog/structuring-the-docker-setup-for-php-projects/#using-entrypoint-for-pre-run-configuration
exec "$@"

Notes:

• make make-init ENVS="ENV=prod GPG_PASSWORD=$GPG_PASSWORD" initializes make and requires that an environment variable named $GPG_PASSWORD exists
  • this is ensured via the env_file property in docker-compose.prod.yml
• make gpg-init requires that a secret gpg key exists at ./secret.gpg (see Local git-secret and gpg setup)
  • this is ensured via the bind mount in docker-compose.prod.yml
• files=$(make secret-list | grep "/\(shared\|prod\)/" | tr '\n' ' ') retrieves all secret files that are relevant for production
  • make secret-list will list all secret files
  • grep "/\(shared\|prod\)/" reduces the list to only the ones in the .secrets/shared/ and .secrets/prod/ directories
  • tr '\n' ' ' replaces new lines with spaces in the result so that the $files variable can be passed savely as an argument to the next command

• via make secret-decrypt-with-password FILES="$files"

  • we make sure to only decrypt files that are relevant for production. This is important, because the prod image only contains production secrets. Secrets for any other environment will not be part of the image. If we wouldn't provide a dedicated list of files, git-secret would attempt to decrypt all secrets that it knows and would fail if an encrypted file is missing with the error

  gpg: can't open '/missing-file.secret': No such file or directory
  gpg: decrypt_message failed: No such file or directory
  git-secret: abort: problem decrypting file with gpg: exit code 2: /missing-file
  

  • in addition, the secret-decrypt-with-password target expects that the GPG_PASSWORD variable is populated (see first point).
• the last line exec "$@" ensures that everything "works as before", i.e. the same ENTRYPOINT / CMD is used in the containers (e.g. php-fpm will still invoke the php-fpm process once the secrets have been decrypted)

The final code in the Dockerfile looks like this:

FROM base as prod

COPY ./.docker/images/php/base/decrypt-secrets.sh /decrypt-secrets.sh
RUN chmod +x /decrypt-secrets.sh
CMD ["/bin/sh"]
ENTRYPOINT ["/decrypt-secrets.sh"]

Note: Because we defined a custom ENTRYPOINT, the default CMD of the base image will be overridden. Thus, we must explicitly re-define it here via CMD ["/bin/sh"]. This behavior is also described in the docs:

If CMD is defined from the base image, setting ENTRYPOINT will reset CMD to an empty value. In this scenario, CMD must be defined in the current image to have a value.

Copy codebase and build-info file

As before, we will copy the final "build artifact" from the codebase build stage to only retain a single layer in the image. In addition, we also copy the build-info file from ./.build/build-info to the root of the codebase.

FROM base as prod

COPY --from=codebase --chown=$APP_USER_NAME:$APP_GROUP_NAME /codebase $APP_CODE_PATH

COPY --chown=$APP_USER_NAME:$APP_GROUP_NAME ./.build/build-info $APP_CODE_PATH/build-info

Build stage prod in the remaining images

In the remaining images for nginx, php-fpm, php-worker and application, there are no dedicated instructions for the prod target. We must still define the build stage, though, via

FROM base as prod

Otherwise, the build would fail with the error

failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to create LLB definition: target stage prod could not be found

Makefile changes

I have updated the default help target that prints all available commands to also include some information about the current environment (usually set e.g. via make make-init ENVS="ENV=prod"). The full recipe is

help:
    @printf '%-43s \033[1mDefault values: \033[0m     \n'
    @printf '%-43s ===================================\n'
    @printf '%-43s ENV: \033[31m "$(ENV)" \033[0m     \n'
    @printf '%-43s TAG: \033[31m "$(TAG)" \033[0m     \n'
    @printf '%-43s ===================================\n'
    @printf '%-43s \033[3mRun the following command to set them:\033[0m\n'
    @printf '%-43s \033[1mmake make-init ENVS="ENV=prod TAG=latest"\033[0m\n'
    @awk 'BEGIN {FS = ":.*##"; printf "\n\033[1mUsage:\033[0m\n  make \033[36m<target>\033[0m\n"} /^[a-zA-Z0-9_-]+:.*?##/ { printf "  \033[36m%-40s\033[0m %s\n", $$1, $$2 } /^##@/ { printf "\n\033[1m%s\033[0m\n", substr($$0, 5) } ' Makefile .make/*.mk

and will now print a header showing the values of the $(ENV) and $(TAG) variables:

$ make
                                            Default values:
                                            ===================================
                                            ENV:  "local"
                                            TAG:  "latest"
                                            ===================================
                                            Run the following command to set them:
                                            make make-init ENVS="ENV=prod TAG=latest"

Usage:
  make <target>

[Make]
  make-init                                 Initializes the local .makefile/.env file with ENV variables for make. Use via ENVS="KEY_1=value1 KEY_2=value2"

[Application: Setup]

Adding GCP values to .make/variables.env

The .make/variables.env file contains the "default" shared variables, that are neither "secret" nor likely to be changed (see Initialize the shared variables).

Those variables include the "ingredients" for the image naming convention

$(DOCKER_REGISTRY)/$(DOCKER_NAMESPACE)/$(DOCKER_SERVICE_NAME)-$(ENV)

Since we will now use our own registry, we need to change the DOCKER_REGISTRY value from docker.io to gcr.io/pl-dofroscra-p (see section Pushing images to the registry).

In addition, we will need three more GCP specific variables that are required for the new gcloud cli make targets:

• GCP_PROJECT_ID: The GCP project id
• GCP_ZONE: The availability zone of the GCP VM
• GCP_VM_NAME: The name of the GCP VM

So the full content of .make/variables.env becomes

DOCKER_REGISTRY=gcr.io/pl-dofroscra-p
DOCKER_NAMESPACE=dofroscra
APP_USER_NAME=application
APP_GROUP_NAME=application
GCP_PROJECT_ID=pl-dofroscra-p
GCP_ZONE=us-central1-a
GCP_VM_NAME=dofroscra-test

ENV based docker compose config

We use the same technique as described in the previous tutorial to assemble the docker compose config files by adding the config files for ENV=prod and the corresponding DOCKER_COMPOSE_FILES definition in .make/02-00-docker.mk:

# File .make/02-00-docker.mk

# ...

DOCKER_COMPOSE_DIR:=...
DOCKER_COMPOSE_COMMAND:=...

DOCKER_COMPOSE_FILE_LOCAL_CI_PROD:=$(DOCKER_COMPOSE_DIR)/docker-compose.local.ci.prod.yml
DOCKER_COMPOSE_FILE_LOCAL_PROD:=$(DOCKER_COMPOSE_DIR)/docker-compose.local.prod.yml
DOCKER_COMPOSE_FILE_LOCAL:=$(DOCKER_COMPOSE_DIR)/docker-compose.local.yml
DOCKER_COMPOSE_FILE_CI:=$(DOCKER_COMPOSE_DIR)/docker-compose.ci.yml
DOCKER_COMPOSE_FILE_PROD:=$(DOCKER_COMPOSE_DIR)/docker-compose.prod.yml

ifeq ($(ENV),prod)
    DOCKER_COMPOSE_FILES:=-f $(DOCKER_COMPOSE_FILE_LOCAL_CI_PROD) -f $(DOCKER_COMPOSE_FILE_LOCAL_PROD) -f $(DOCKER_COMPOSE_FILE_PROD)
else ifeq ($(ENV),ci)
    DOCKER_COMPOSE_FILES:=-f $(DOCKER_COMPOSE_FILE_LOCAL_CI_PROD) -f $(DOCKER_COMPOSE_FILE_CI)
else ifeq ($(ENV),local)
    DOCKER_COMPOSE_FILES:=-f $(DOCKER_COMPOSE_FILE_LOCAL_CI_PROD) -f $(DOCKER_COMPOSE_FILE_LOCAL_PROD) -f $(DOCKER_COMPOSE_FILE_LOCAL)
endif

DOCKER_COMPOSE:=$(DOCKER_COMPOSE_COMMAND) $(DOCKER_COMPOSE_FILES)

FYI: There is no dedicated docker compose config file for settings that only affect ci and prod (i.e. docker-compose.ci.prod.yml).

The "final" $(DOCKER_COMPOSE_FILES) variable will look like this:

-f ./.docker/docker-compose/docker-compose.local.ci.prod.yml -f ./.docker/docker-compose/docker-compose.local.prod.yml -f ./.docker/docker-compose/docker-compose.prod.yml

and the "full" $(DOCKER_COMPOSE) variable like this:

ENV=prod TAG=latest DOCKER_REGISTRY=gcr.io/pl-dofroscra-p DOCKER_NAMESPACE=dofroscra APP_USER_NAME=application APP_GROUP_NAME=application docker compose -p dofroscra_prod --env-file ./.docker/.env -f ./.docker/docker-compose/docker-compose.local.ci.prod.yml -f ./.docker/docker-compose/docker-compose.local.prod.yml -f ./.docker/docker-compose/docker-compose.prod.yml

Changes to the git-secret recipes

The git-secret recipes are defined in 01-00-application-setup.mk and have been added originally in Use git secret to encrypt secrets in the repository: Makefile adjustments.

I have modified the targets secret-decrypt and secret-decrypt-with-password to accept an optional list of files to decrypt via the FILES variable. If the variable is empty, all files are decrypted. This is required for the decrypt-secrets.sh script, because we will only store the secrets that are relevant for the curently built environment in the image and the decryption would fail if we attempted to decrypt files that don't exist.

# ...

.PHONY: secret-decrypt
secret-decrypt: ## Decrypt secret files via `git-secret reveal -f`. Use FILES=file1 to decrypt only file1 instead of all files
    "$(MAKE)" -s git-secret ARGS="reveal -f $(FILES)"

.PHONY: secret-decrypt-with-password
secret-decrypt-with-password: ## Decrypt secret files using a password for gpg. Use FILES=file1 to decrypt only file1 instead of all files
    @$(if $(GPG_PASSWORD),,$(error GPG_PASSWORD is undefined))
    "$(MAKE)" -s git-secret ARGS="reveal -f -p $(GPG_PASSWORD) $(FILES)" 

In addition, I made a minor adjustment to the targets secret-add, secret-cat and secret-remove to use the variable name FILES (plural) instead of FILE, because all of them can also work with a list of files instead of just a single one.

Additional docker recipes

The following targets have been added to .make/02-00-docker.mk

compose-secrets.env:
    @echo "# This file only exists because docker compose cannot run `build` otherwise," > ./compose-secrets.env
    @echo "# because it is referenced as an `env_file` in the docker compose config file" > ./compose-secrets.env
    @echo "# for the `prod` environment. On `prod` it will contain the necessary ENV variables," > ./compose-secrets.env
    @echo "# but on all other environments this 'placeholder' file is created." > ./compose-secrets.env
    @echo "# The file is generated automatically via `make` if a docker compose target is executed" > ./compose-secrets.env
    @echo "# @see https://github.com/docker/compose/issues/1973#issuecomment-1148257736" > ./compose-secrets.env

.PHONY: docker-push
docker-push: validate-docker-variables ## Push all docker images to the remote repository
    $(DOCKER_COMPOSE) push $(ARGS)

.PHONY: docker-pull
docker-pull: validate-docker-variables ## Pull all docker images from the remote repository
    $(DOCKER_COMPOSE) pull $(ARGS)

.PHONY: docker-exec
docker-exec: validate-docker-variables ## Execute a command in a docker container. Usage: `make docker-exec DOCKER_SERVICE_NAME="application" DOCKER_COMMAND="echo 'Hello world!'"`
    @$(if $(DOCKER_SERVICE_NAME),,$(error "DOCKER_SERVICE_NAME is undefined"))
    @$(if $(DOCKER_COMMAND),,$(error "DOCKER_COMMAND is undefined"))
    $(DOCKER_COMPOSE) exec -T $(DOCKER_SERVICE_NAME) $(DOCKER_COMMAND)

# @see https://www.linuxfixes.com/2022/01/solved-how-to-test-dockerignore-file.html
# helpful to debug a .dockerignore file
.PHONY: docker-show-build-context
docker-show-build-context: ## Show all files that are in the docker build context for `docker build`
    @echo -e "FROM busybox\nCOPY . /codebase\nCMD find /codebase -print" | docker image build --no-cache -t build-context -f - .
    @docker run --rm build-context | sort

# `docker build` and `docker compose build` are behaving differently
# @see https://github.com/docker/compose/issues/9508
.PHONY: docker-show-compose-build-context
docker-show-compose-build-context: ## Show all files that are in the docker build context for `docker compose build`
    @.dev/scripts/docker-compose-build-context/show-build-context.sh

# Note: This is only a temporary target until https://github.com/docker/for-win/issues/12742 is fixed
.PHONY: docker-fix-mount-permissions
docker-fix-mount-permissions: ## Fix the permissions of the bind-mounted folder, @see https://github.com/docker/for-win/issues/12742
    $(EXECUTE_IN_APPLICATION_CONTAINER) ls -al

• compose-secrets.env ensures that the file ./compose-secrets.env exists in the root of the repository. The target is added as precondition to the validate-docker-variables target so that it is checked whenever another docker compose target is run

  .PHONY: validate-docker-variables
  validate-docker-variables: .docker/.env compose-secrets.env
  

  Without this file, docker compose build would fail for the prod environment, because the file only exists "on the VM" and is defined as env_file setting in docker-compose.prod.yml. This is one of those annoying cases when the dual-build-and-run-usage of docker compose gets in the way, see also GH issue "Allow some commands to run without full config validation"

• docker-push uses docker compose push to push all defined services to the registry
• docker-pull uses docker compose pull to pull all defined services from the registry
• docker-exec uses docker compose exec to run arbitrary commands in the docker container that is specified via the $(DOCKER_SERVICE_NAME) variable. The command itself has to be passed via the $(DOCKER_COMMAND) variable
• docker-show-build-context is a small helper script that builds a temporary image and lists all files in the build context - this is very helpful to debug entries in the .dockerignore file. See also: [SOLVED] How to test dockerignore file?
• docker-show-compose-build-context does the same but for docker compose build - which currently (in docker compose v2.5.1) seems to behave differently than docker build
• docker-fix-mount-permissions is a only a temporary target that provides a workaround for a bug I filed in the Docker Desktop for Windows repository: Ownership of files set via bind mount is set to user who accesses the file first

GCP recipes

For the deployment, we need to communicate with the VM and will use the gcloud cli to run SSH commands via IAP tunneling. The cli requires a couple of default parameters like the VM name, the project id, the availability zone and the location of the key file for the service account, that we conveniently defined in the .make/variables.env file.

The GCP targets are defined in .make/03-00-gcp.mk:

##@ [GCP]

.PHONY: gcp-init
gcp-init: validate-gcp-variables ## Initialize the `gcloud` cli and authenticate docker with the keyfile defined via GCP_DEPLOYMENT_SERVICE_ACCOUNT_KEY.
    @$(if $(GCP_DEPLOYMENT_SERVICE_ACCOUNT_KEY),,$(error "GCP_DEPLOYMENT_SERVICE_ACCOUNT_KEY is undefined"))
    @$(if $(GCP_PROJECT_ID),,$(error "GCP_PROJECT_ID is undefined"))
    gcloud auth activate-service-account --key-file="$(GCP_DEPLOYMENT_SERVICE_ACCOUNT_KEY)" --project="$(GCP_PROJECT_ID)"
    cat "$(GCP_DEPLOYMENT_SERVICE_ACCOUNT_KEY)" | docker login -u _json_key --password-stdin https://gcr.io

.PHONY: validate-gcp-variables
validate-gcp-variables:
    @$(if $(GCP_PROJECT_ID),,$(error "GCP_PROJECT_ID is undefined"))
    @$(if $(GCP_ZONE),,$(error "GCP_ZONE is undefined"))
    @$(if $(GCP_VM_NAME),,$(error "GCP_VM_NAME is undefined"))

# @see https://cloud.google.com/sdk/gcloud/reference/compute/ssh
.PHONY: gcp-ssh-command
gcp-ssh-command: validate-gcp-variables ## Run an arbitrary SSH command on the VM via IAP tunnel. Usage: `make gcp-ssh-command COMMAND="whoami"`
    @$(if $(COMMAND),,$(error "COMMAND is undefined"))
    gcloud compute ssh $(GCP_VM_NAME) --project $(GCP_PROJECT_ID) --zone $(GCP_ZONE) --tunnel-through-iap --command="$(COMMAND)"

.PHONY: gcp-ssh-login
gcp-ssh-login: validate-gcp-variables ## Log into a VM via IAP tunnel
    gcloud compute ssh $(GCP_VM_NAME) --project $(GCP_PROJECT_ID) --zone $(GCP_ZONE) --tunnel-through-iap

# @see https://cloud.google.com/sdk/gcloud/reference/compute/scp
.PHONY: gcp-scp-command
gcp-scp-command: validate-gcp-variables ## Copy a file via scp to the VM via IAP tunnel. Usage: `make gcp-scp-command SOURCE="foo" DESTINATION="bar"`
    @$(if $(SOURCE),,$(error "SOURCE is undefined"))
    @$(if $(DESTINATION),,$(error "DESTINATION is undefined"))
    gcloud compute scp $(SOURCE) $(GCP_VM_NAME):$(DESTINATION) --project $(GCP_PROJECT_ID) --zone $(GCP_ZONE) --tunnel-through-iap

# Defines the default secret version to retrieve from the Secret Manager
SECRET_VERSION?=latest

# @see https://cloud.google.com/sdk/gcloud/reference/secrets/versions/access
.PHONY: gcp-secret-get
gcp-secret-get: ## Retrieve and print the secret $(SECRET_NAME) in version $(SECRET_VERSION) from the Secret Manager
    @$(if $(SECRET_NAME),,$(error "SECRET_NAME is undefined"))
    @$(if $(SECRET_VERSION),,$(error "SECRET_VERSION is undefined"))
    @gcloud secrets versions access $(SECRET_VERSION) --secret=$(SECRET_NAME)

.PHONY: gcp-docker-exec
gcp-docker-exec: ## Run a command in a docker container on the VM. Usage: `make gcp-docker-exec DOCKER_SERVICE_NAME="application" DOCKER_COMMAND="echo 'Hello world!'"`
    @$(if $(DOCKER_SERVICE_NAME),,$(error "DOCKER_SERVICE_NAME is undefined"))
    @$(if $(DOCKER_COMMAND),,$(error "DOCKER_COMMAND is undefined"))
    "$(MAKE)" -s gcp-ssh-command COMMAND="cd $(CODEBASE_DIRECTORY) && make docker-exec DOCKER_SERVICE_NAME='$(DOCKER_SERVICE_NAME)' DOCKER_COMMAND='$(DOCKER_COMMAND)'"

# @see https://cloud.google.com/compute/docs/instances/view-ip-address
.PHONY: gcp-show-ip
gcp-show-ip: ## Show the IP address of the VM specified by GCP_VM_NAME.
    gcloud compute instances describe $(GCP_VM_NAME) --zone $(GCP_ZONE) --project=$(GCP_PROJECT_ID) --format='get(networkInterfaces[0].accessConfigs[0].natIP)'

• gcp-init ensures that the correct service account is activated for the gcloud cli and is also used for docker authentication
• validate-gcp-variables is the pendant to the validate-docker-variables in the .make/02-00-docker.mk file and checks if the default variables GCP_PROJECT_ID, GCP_ZONE and GCP_VM_NAME exist. See also section Enforce required parameters
• gcp-ssh-command is used to run arbitrary SSH commands on the VM using gcloud compute ssh The command is defined via the COMMAND variable
• gcp-ssh-login is a convenience target to log into the GCP VM via SSH using IAP tunneling
• gcp-scp-command copies files from the local system to the VM via scp using gcloud compute scp The source must be defined via the SOURCE variable and the destination via the DESTINATION variable
• gcp-secret-get retrieves a secret from the Secret Manager. The secret has to specified via the SECRET_NAME variable and an optional version can be given via the VERSION variable (that defaults to latest if omitted)
• gcp-docker-exec runs the gcp-ssh-command target to execute the docker-exec target on the VM
• gcp-show-ip retrieves the ip address of the VM on GCP as explained in

Infrastructure recipes

For the infrastructure, we currently only have a single target defined in the .make/04-00-infrastructure.mk file

##@ [Infrastructure]

.PHONY: infrastructure-setup-vm
infrastructure-setup-vm: ## Set the GCP VM up
    bash .infrastructure/setup-gcp.sh   

• infrastructure-setup-vm runs the script defined in the .infrastructure/setup-gcp.sh

Deployment recipes

The Deployment workflow is described in more detail in the following section, thus I'll only show the corresponding targets defined in .make/05-00-deployment.mk here

##@ [Deployment]

.PHONY: deploy
deploy: # Build all images and deploy them to GCP
    @printf "$(GREEN)Switching to 'local' environment$(NO_COLOR)\n"
    @make --no-print-directory make-init
    @printf "$(GREEN)Starting docker setup locally$(NO_COLOR)\n"
    @make --no-print-directory docker-up
    @printf "$(GREEN)Verifying that there are no changes in the secrets$(NO_COLOR)\n"
    @make --no-print-directory gpg-init
    @make --no-print-directory deployment-guard-secret-changes
    @printf "$(GREEN)Verifying that there are no uncommitted changes in the codebase$(NO_COLOR)\n"
    @make --no-print-directory deployment-guard-uncommitted-changes
    @printf "$(GREEN)Initializing gcloud$(NO_COLOR)\n"
    @make --no-print-directory gcp-init
    @printf "$(GREEN)Switching to 'prod' environment$(NO_COLOR)\n"
    @make --no-print-directory make-init ENVS="ENV=prod TAG=latest"
    @printf "$(GREEN)Creating build information file$(NO_COLOR)\n"
    @make --no-print-directory deployment-create-build-info-file
    @printf "$(GREEN)Building docker images$(NO_COLOR)\n"
    @make --no-print-directory docker-build
    @printf "$(GREEN)Pushing images to the registry$(NO_COLOR)\n"
    @make --no-print-directory docker-push
    @printf "$(GREEN)Creating the deployment archive$(NO_COLOR)\n"
    @make deployment-create-tar
    @printf "$(GREEN)Copying the deployment archive to the VM and run the deployment$(NO_COLOR)\n"
    @make --no-print-directory deployment-run-on-vm
    @printf "$(GREEN)Clearing deployment archive$(NO_COLOR)\n"
    @make --no-print-directory deployment-clear-tar
    @printf "$(GREEN)Switching to 'local' environment$(NO_COLOR)\n"
    @make --no-print-directory make-init

# directory on the VM that will contain the files to start the docker setup
CODEBASE_DIRECTORY=/tmp/codebase

IGNORE_SECRET_CHANGES?=

.PHONY: deployment-guard-secret-changes
deployment-guard-secret-changes: ## Check if there are any changes between the decrypted and encrypted secret files
    if ( ! make secret-diff || [ "$$(make secret-diff | grep ^@@)" != "" ] ) && [ "$(IGNORE_SECRET_CHANGES)" == "" ] ; then \
        printf "Found changes in the secret files => $(RED)ABORTING$(NO_COLOR)\n\n"; \
        printf "Use with IGNORE_SECRET_CHANGES=true to ignore this warning\n\n"; \
        make secret-diff; \
        exit 1; \
    fi
    @echo "No changes in the secret files!"

IGNORE_UNCOMMITTED_CHANGES?=

.PHONY: deployment-guard-uncommitted-changes
deployment-guard-uncommitted-changes: ## Check if there are any git changes and abort if so. The check can be ignore by passing `IGNORE_UNCOMMITTED_CHANGES=true`
    if [ "$$(git status -s)" != "" ] && [ "$(IGNORE_UNCOMMITTED_CHANGES)" == "" ] ; then \
        printf "Found uncommitted changes in git => $(RED)ABORTING$(NO_COLOR)\n\n"; \
        printf "Use with IGNORE_UNCOMMITTED_CHANGES=true to ignore this warning\n\n"; \
        git status -s; \
        exit 1; \
    fi
    @echo "No uncommitted changes found!"

# FYI: make converts all new lines in spaces when they are echo'd 
# @see https://stackoverflow.com/a/54068252/413531
# To execute a shell command via $(command), the $ has to be escaped with another $
#  ==> $$(command)
# @see https://stackoverflow.com/a/26564874/413531
.PHONY: deployment-create-build-info-file
deployment-create-build-info-file: ## Create a file containing version information about the codebase
    @echo "BUILD INFO" > ".build/build-info"
    @echo "==========" >> ".build/build-info"
    @echo "User  :" $$(whoami) >> ".build/build-info"
    @echo "Date  :" $$(date --rfc-3339=seconds) >> ".build/build-info"
    @echo "Branch:" $$(git branch --show-current) >> ".build/build-info"
    @echo "" >> ".build/build-info"
    @echo "Commit" >> ".build/build-info"
    @echo "------" >> ".build/build-info"
    @git log -1 --no-color >> ".build/build-info"

# create tar archive
#  tar -czvf archive.tar.gz ./source
#
# extract tar archive
#  tar -xzvf archive.tar.gz -C ./target
#
# @see https://www.cyberciti.biz/faq/how-to-create-tar-gz-file-in-linux-using-command-line/
# @see https://serverfault.com/a/330133
.PHONY: deployment-create-tar
deployment-create-tar:
    # create the build directory
    rm -rf .build/deployment
    mkdir -p .build/deployment
    # copy the necessary files
    mkdir -p .build/deployment/.docker/docker-compose/
    cp -r .docker/docker-compose/ .build/deployment/.docker/
    cp -r .make .build/deployment/
    cp Makefile .build/deployment/
    cp .infrastructure/scripts/deploy.sh .build/deployment/
    # make sure we don't have any .env files in the build directory (don't wanna leak any secrets) ...
    find .build/deployment -name '.env' -delete
    # ... apart from the .env file we need to start docker
    cp .secrets/prod/docker.env .build/deployment/.docker/.env
    # create the archive
    tar -czvf .build/deployment.tar.gz -C .build/deployment/ ./

.PHONY: deployment-clear-tar
deployment-clear-tar:
    # clear the build directory
    rm -rf .build/deployment
    # remove the archive
    rm -rf .build/deployment.tar.gz

.PHONY: deployment-run-on-vm
deployment-run-on-vm:## Run the deployment script on the VM
    "$(MAKE)" -s gcp-scp-command SOURCE=".build/deployment.tar.gz" DESTINATION="deployment.tar.gz"
    "$(MAKE)" -s gcp-ssh-command COMMAND="sudo rm -rf $(CODEBASE_DIRECTORY) && sudo mkdir -p $(CODEBASE_DIRECTORY) && sudo tar -xzvf deployment.tar.gz -C $(CODEBASE_DIRECTORY) && cd $(CODEBASE_DIRECTORY) && sudo bash deploy.sh"

.PHONY: deployment-setup-db-on-vm
deployment-setup-db-on-vm: ## Setup the application on the VM. CAUTION: The docker setup must be running!
    "$(MAKE)" -s gcp-docker-exec DOCKER_SERVICE_NAME="application" DOCKER_COMMAND="make setup-db"

• deploy is the main target to trigger a full deployment
• deployment-guard-uncommitted-changes checks if there are changes in the codebase that have not been committed to git, yet. We want to avoid that, because that might cause a "drift" between the codebase and the deployed code
• deployment-guard-secret-changes does the same but for secrets
• deployment-create-build-info-file creates the build-info file in the .build/ directory
• deployment-create-tar creates a deployment tar archive that contains all necessary files to start the docker setup via docker compose on the VM
• deployment-clear-tar removes the archive including any temporary files that have been created via deployment-create-tar
• deployment-run-on-vm copies the deployment tar archive to the VM, extracts it and runs the deployment script defined in .infrastructure/scripts/deploy.sh, see section Deployment commands on the VM
• deployment-setup-db-on-vm executes the setup-db command in the application container on the VM

Wrapping up

Congratulations, you made it! If some things are not completely clear by now, don't hesitate to leave a comment. You should now be ready to deploy a dockerized PHP application "to production" on GCP via docker compose.

In the next part of this tutorial, we will use terraform to create the infrastructure for production deployments on GCP and deploy the docker containers on individual VMs without docker compose.

Please subscribe to the RSS feed or via email to get automatic notifications when this next part comes out :)

• creating VMs
• using a container registry
• using a secret manager

All code samples are publicly available in my Docker PHP Tutorial repository on Github.
You find the branch with the final result of this tutorial at part-8-gcp-compute-instance-vm-docker.

All published parts of the Docker PHP Tutorial are collected under a dedicated page at Docker PHP Tutorial. The previous part was Create a CI pipeline for dockerized PHP Apps. and the following one is Deploy dockerized PHP Apps to production on GCP via docker compose.

If you want to follow along, please subscribe to the RSS feed or via email to get automatic notifications when the next part comes out :)

Table of contents

• Introduction
• Set up a GCP project
• Create a service account
  • Create service account key file
  • Configure IAM permissions
• Set up the gcloud CLI tool
• Set up the Container Registry
  • Authenticate docker
  • Pushing images to the registry
  • Images are stored in Google Cloud Storage buckets
  • Pulling images from the registry
• Set up the Secret Manager
  • Create a secret via the UI
  • View a secret via the UI
  • Retrieve a secret via the gcloud cli
  • Add the secret gpg key and password
• Compute Instances: The GCP VMs
  • Create a VM
    • General VM settings
    • Firewall and networks tags
    • The role of the service account
    • Adding a public SSH key
    • Define Availability Policies
    • The actual VM creation
  • Log into a VM
    • Login via SSH from the GCP UI
    • Login via SSH with your own key from your host machine
    • Login using the Identity-Aware Proxy (IAP) concept
      • Additional notes on IAP
    • Get root permissions
  • ssh and scp commands
    • gcloud ssh --command=""
    • gcloud scp
• Provision the VM
  • Get the secret gpg key and password from the Secret Manager
  • Installing docker and docker compose
  • Authenticate docker via gcloud
  • Pulling the nginx image
  • Start the nginx container
• Automate via gcloud commands
  • Preconditions: Project and Owner service account
  • Configure gcloud to use the master service account
  • Enable APIs
  • Create and configure a "deployment" service account
  • Create secrets
  • Create firewall rule for HTTP traffic
  • Create a Compute Instance VM
  • Provisioning
  • Deployment
• Putting it all together
• Cleanup
• Wrapping up

Introduction

In the next tutorial we will deploy our dockerized PHP Apps to "production" via docker compose and will create this "production" environment on GCP (Google Cloud Platform). This tutorial serves as a primer on GCP to build up some fundamental knowledge, because we will use the platform to provide all the infrastructure required to run our dockerized PHP application.

In the process, we'll learn about GCP projects as our own "space" in GCP and service accounts as a way to communicate programmatically. We'll start by doing everything manually via the UI, but will also explain how to do it programmatically via the gcloud cli and end with a fully automated script.

The following video shows the overall flow

The API keys (see service account key files) that I use are not in the repository, because I would be billed for any usage. I.e. you must create you own project and keys to follow along.

Set up a GCP project

On GCP, resources are organized under so-called projects. We can create a project via the Create Project UI:

The project ID must be a globally unique string, and I have chosen pl-dofroscra-p for this tutorial (pl => Pascal Landau; dofroscra => Docker From Scratch; p => production).

Create a service account

As a next step, we need a service account that we can use to make API requests, because we don't want to use our "personal GCP account". Service accounts are created via the IAM & Admin > Service Accounts UI:

Create service account key file

In order to use the account programmatically, we also need to create a key file by choosing the "Manage Keys" option of the corresponding service account.

This will open up a UI at

https://console.cloud.google.com/iam-admin/serviceaccounts/details/$serviceAccountId/keys

where $serviceAccountId is the numeric id of the service account, e.g. 109548647107864470967. To create a key:

• click "ADD KEY" and select "Create new key" from the drop down menu
  • This will bring up a modal window to choose the key type.
• select the recommended JSON type and click "Create".
  • GCP will then generate a new key pair, store the public key and offer the private key file as download.
• download the file and make sure to treat it like any other private key (ssh, gpg, ...) i.e. never share it publicly!

We will store this file in the root of the codebase at gcp-service-account-key.json and add it to the .gitignore file.

Each service account has also a unique email address that consists of its (non-numeric) id and the project id. You can also find it directly in the key file:

$ grep "email" ./gcp-service-account-key.json
  "client_email": "docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com",

This email address is usually used to reference the service account, e.g. when assigning IAM permissions.

Configure IAM permissions

IAM stands for Identity and Access Management (IAM) and is used for managing permissions on GCP. The two core concepts are "permissions" and "roles":

• permissions are fine-grained for particular actions, e.g. storage.buckets.create to "Create Cloud Storage buckets"
• roles combine a selection of permissions, e.g. the Cloud Storage Admin role has permissions like
  • storage.buckets.create
  • storage.buckets.get
  • etc.
• roles are assigned to users (or service accounts)

You can find a full overview of all permissions in the Permissions Reference and all roles under Understanding roles > Predefined roles.

For this tutorial, we'll assign the following roles to the service account "user" docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com:

• Storage Admin
  • required to create the GCP bucket for the registry and to pull the images on the VM
• Secret Manager Admin
  • required to retrieve secrets from the Secret Manager
• Compute Admin, Service Account User and IAP-secured Tunnel User
  • are necessary for logging into a VM via IAP.

Roles can be assigned through the Cloud Console IAM UI by editing the corresponding user.

Caution: It might take some time (usually a couple of seconds) until changes in IAM permissions take effect.

Set up the gcloud CLI tool

The CLI tool for GCP is called gcloud and is available for all operating systems.

In this tutorial we are using version 380.0.0 installed natively on Windows via the GoogleCloudSDKInstaller.exe using the "Bundled Python" option.

FYI: As described under Uninstalling the Google Cloud CLI you can find the installation and config directories via

# installation directory
$ gcloud info --format='value(installation.sdk_root)'
C:\Users\Pascal\AppData\Local\Google\Cloud SDK\google-cloud-sdk

# config directory
$ gcloud info --format='value(config.paths.global_config_dir)'
C:\Users\Pascal\AppData\Roaming\gcloud

I will not use my personal Google account to run gcloud commands, thus I'm not using the "usual" initialization process by running gcloud init. Instead, I will use the service account that we created previously and activate it as described under gcloud auth activate-service-account via

gcloud auth activate-service-account docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com --key-file=./gcp-service-account-key.json --project=pl-dofroscra-p

Output

$ gcloud auth activate-service-account docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com --key-file=./gcp-service-account-key.json --project=pl-dofroscra-p
Activated service account credentials for: [docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com]

FYI: Because we are using a json key file that includes the service account ID, we can also omit the id in the command, i.e.

$ gcloud auth activate-service-account --key-file=./gcp-service-account-key.json --project=pl-dofroscra-p
Activated service account credentials for: [docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com]

Set up the Container Registry

We will use docker compose to run our PHP application in the next tutorial part and need to make our docker images available in a container registry. Luckily, GCP offers a Container Registry product that gives us a ready-to-use private registry as part of a GCP project. Before we can use it, the corresponding Google Container Registry API must be enabled:

You find the Container Registry in the Cloud Console UI under Container Registry.

Authenticate docker

Since the Container Registry is private, we need to authenticate before we can push our docker images. The available authentication methods are described in the GCP docu "Container Registry Authentication methods". For pushing images from our local host system, we will use the service account key file that we created previously and run the command shown in the "JSON key file" section of the the docu.

key=./gcp-service-account-key.json
cat "$key" | docker login -u _json_key --password-stdin https://gcr.io

A successful authentication looks as follows:

$ cat "$key" | docker login -u _json_key --password-stdin https://gcr.io
Login Succeeded

Logging in with your password grants your terminal complete access to your account.
For better security, log in with a limited-privilege personal access token. Learn more at https://docs.docker.com/go/access-tokens/

So what exactly "happens" when we run this command? According to the docker logindocumentation

When you log in, the command stores credentials in $HOME/.docker/config.json on Linux or %USERPROFILE%/.docker/config.json on Windows [...]

The Docker Engine can keep user credentials in an external credentials store, such as the native keychain of the operating system. [...]

You need to specify the credentials store in $HOME/.docker/config.json to tell the docker engine to use it. [...]

By default, Docker looks for the native binary on each of the platforms, i.e. “osxkeychain” on macOS, “wincred” on windows, and “pass” on Linux.

In other words: I won't be able to see the content of the service account key file in "plain text" anywhere but docker will utilize the OS specific tools to store them securely. After I ran the docker login command on Windows, I found the following content in ~/.docker/config.json:

$ cat ~/.docker/config.json
{
        "auths": {
                "gcr.io": {}
        },
        "credsStore": "desktop"
}

FYI: "desktop" seems to be a wrapper for the Wincred executable.

Pushing images to the registry

For this tutorial, we will create a super simple nginx alpine image that provides a "Hello world" hello.html file via

docker build -t my-nginx -f - . <<EOF
FROM nginx:1.21.5-alpine

RUN echo "Hello world" >> /usr/share/nginx/html/hello.html

EOF

The name of the image is my-nginx

$ docker image ls | grep my-nginx
my-nginx         latest             42dd1608d126   50 seconds ago    23.5MB

In order to push an image to a registry, the image name must be prefixed with the corresponding registry. This was quite confusing to me, because I would have expected to be able to run something like this:

$ docker push my-nginx --registry=gcr.io

unknown flag: --registry
See 'docker push --help'.

But nope, there is no such --registry option. Even worse: Omitting it would cause a push to docker.io, the "default" registry:

$ docker push my-nginx
Using default tag: latest
The push refers to repository [docker.io/my-nginx]

According to the GCP docs on Pushing and pulling images, the following steps are necessary to push an image to a GCP registry:

• Tag the image with its target path in Container Registry, including the gcr.io registry host and the project ID my-project
• Push the image to the registry

In our case the target path to our Container Registry is

gcr.io/pl-dofroscra-p

because pl-dofroscra-p is the id of the GCP project we created previously.

The full image name becomes

gcr.io/pl-dofroscra-p/my-nginx

To push the my-nginx image, we must first "add another name" to it via docker tag

$ docker tag my-nginx gcr.io/pl-dofroscra-p/my-nginx

$ docker image ls
REPOSITORY                       TAG                IMAGE ID       CREATED          SIZE
my-nginx                         latest             ba7a2c5faf0d   15 minutes ago   23.5MB
gcr.io/pl-dofroscra-p/my-nginx   latest             ba7a2c5faf0d   15 minutes ago   23.5MB

and push that name afterwards

$ docker push gcr.io/pl-dofroscra-p/my-nginx
Using default tag: latest
The push refers to repository [gcr.io/pl-dofroscra-p/my-nginx]
134174afa9ad: Preparing
cb7b4430c52d: Preparing
419df8b60032: Preparing
0e835d02c1b5: Preparing
5ee3266a70bd: Preparing
3f87f0a06073: Preparing
1c9c1e42aafa: Preparing
8d3ac3489996: Preparing
8d3ac3489996: Waiting
3f87f0a06073: Waiting
1c9c1e42aafa: Waiting
cb7b4430c52d: Pushed
134174afa9ad: Pushed
419df8b60032: Pushed
5ee3266a70bd: Pushed
0e835d02c1b5: Pushed
8d3ac3489996: Layer already exists
3f87f0a06073: Pushed
1c9c1e42aafa: Pushed
latest: digest: sha256:0740591fb686227d8cdf4e42b784f634cbaf9f5caa6ee478e3bcc24aeef75d7f size: 1982

You can then find the image in the UI of the Container Registry:

Don't worry: We won't have to do the tagging every time before a push, because we will set up make to use the correct name automatically when building the images in the next part.

Images are stored in Google Cloud Storage buckets

We assigned the Storage Admin role to the service account previously that contains the storage.buckets.create permission. If we wouldn't have done that, the following error would have occurred:

denied: Token exchange failed for project 'pl-dofroscra-p'. Caller does not have permission 'storage.buckets.create'. To configure permissions, follow instructions at: https://cloud.google.com/container-registry/docs/access-control

The Container Registry tries to store the docker images in a Google Cloud Storage bucket that is created on the fly when the very first image is pushed, see the GCP docs on "Adding a registry":

The first image push to a hostname triggers creation of the registry in a project and the corresponding Cloud Storage storage bucket. This initial push requires project-wide permissions to create storage buckets.

You can find the bucket, that in my case is named artifacts.pl-dofroscra-p.appspot.com in the Cloud Storage UI:

CAUTION: Make sure to delete this bucket once you are done with the tutorial - otherwise storage costs will incur.

Pulling images from the registry

According to the GCP docs to pull an image from the Container Registry we need to be authenticated with a user that has the permissions of the Storage Object Viewer role to access the "raw" images. FYI: The Storage Admin role that we assigned previously has all the permissions of the Storage Object Viewer role.

Then use the fully qualified image name as before:

docker pull gcr.io/pl-dofroscra-p/my-nginx

Output if the image is cached

$ docker pull gcr.io/pl-dofroscra-p/my-nginx
Using default tag: latest
latest: Pulling from pl-dofroscra-p/my-nginx
Digest: sha256:0740591fb686227d8cdf4e42b784f634cbaf9f5caa6ee478e3bcc24aeef75d7f
Status: Image is up to date for gcr.io/pl-dofroscra-p/my-nginx:latest
gcr.io/pl-dofroscra-p/my-nginx:latest

or if doesn't exist

docker pull gcr.io/pl-dofroscra-p/my-nginx
Using default tag: latest
latest: Pulling from pl-dofroscra-p/my-nginx
59bf1c3509f3: Pull complete 
f3322597df46: Pull complete 
d09cf91cabdc: Pull complete 
3a97535ac2ef: Pull complete 
919ade35f869: Pull complete 
40e5d2fe5bcd: Pull complete 
c72acb0c83a5: Pull complete 
d6baa2bee4a5: Pull complete 
Digest: sha256:0740591fb686227d8cdf4e42b784f634cbaf9f5caa6ee478e3bcc24aeef75d7f
Status: Downloaded newer image for gcr.io/pl-dofroscra-p/my-nginx:latest
gcr.io/pl-dofroscra-p/my-nginx:latest

Set up the Secret Manager

Even though we use git secret to manage our secrets, we still need the gpg secret key for decryption. This key is "a secret in itself" and we will use the GCP Secret Manager to store it and retrieve it later from a VM.

It can be managed from the Security > Secret Manager UI once we have enabled the Secret Manager API

Create a secret via the UI

To create a secret:

• navigate to the Secret Manager UI and click the "+ CREATE SECRET" button
• enter the secret name and secret value in the form (we can ignore the other advanced settings like "Replication policy" and "Encryption" for now)
  • FYI: the secret name can only contain English letters (A-Z), numbers (0-9), dashes (-), and underscores (_)
• click the "CREATE SECRET" button

The following gif shows the creation of a secret named my_secret_key with the value my_secret_value.

View a secret via the UI

The Security > Secret Manager UI lists all existing secrets. Clicking on a secret will lead you to the Secret Detail UI at the URL

https://console.cloud.google.com/security/secret-manager/secret/$secretName/versions

e.g. for secret name my_secret_key:

https://console.cloud.google.com/security/secret-manager/secret/my_secret_key/versions

The UI shows all versions of the secret, though we currently only have one. To view the actual secret value, click on the three dots in the "Actions" column and select "View secret value" (see gif in the previous section).

Retrieve a secret via the gcloud cli

To retrieve a secret with a service account via the gcloud cli, it needs the permission secretmanager.versions.access, that is part of the Secret Manager Secret Accessor role (as well as the Secret Manager Admin role). Let's first show all available secrets via

gcloud secrets list

$ gcloud secrets list
NAME           CREATED              REPLICATION_POLICY  LOCATIONS
my_secret_key  2022-05-15T05:38:11  automatic           -

To "see" the value of my_secret_key, we must also define the corresponding version. All versions can be listed via

gcloud secrets versions list my_secret_key

$ gcloud secrets versions list my_secret_key
NAME  STATE    CREATED              DESTROYED
1     enabled  2022-05-15T05:38:13  -

The actual secret value for version 1 of my_secret_key is accessed via

gcloud secrets versions access 1 --secret=my_secret_key

$ gcloud secrets versions access 1 --secret=my_secret_key
my_secret_value

You can also use the string latest as version to retrieve the latest enabled version

Version resource - Numeric secret version to access or a configured alias (including 'latest' to use the latest version).

gcloud secrets versions access latest --secret=my_secret_key

$ gcloud secrets versions access latest --secret=my_secret_key
my_secret_value

Add the secret gpg key and password

We already know how to create another gpg key pair and add the corresponding email to git secret from when we have set up the CI pipelines (see section Add a password-protected secret gpg key) . We will do the same once more for the production environment via

name="Production Deployment"
email="production@example.com"
passphrase=87654321
secret=secret-production-protected.gpg.example
public=.dev/gpg-keys/production-public.gpg
# export key pair
gpg --batch --gen-key <<EOF
Key-Type: 1
Key-Length: 2048
Subkey-Type: 1
Subkey-Length: 2048
Name-Real: $name
Name-Email: $email
Expire-Date: 0
Passphrase: $passphrase
EOF

# export the private key
gpg --output $secret --pinentry-mode=loopback --passphrase  "$passphrase" --armor --export-secret-key $email

# export the public key
gpg --armor --export $email > $public

You can find the secret key in the codebase at secret-production-protected.gpg.example and the public key at .dev/gpg-keys/production-public.gpg.

I have also added the secret gpg key and password as secrets to the secret manager

Compute Instances: The GCP VMs

Compute Instances are the equivalent of AWS EC2 instances.

To run our application, we need a VM with a public IP address so that it can be reached from the internet. VMs on GCP are called Compute Instances and can be managed from the Compute Instance UI - though we must first activate the Compute Instance API.

Create a VM

We can simply create a new instance from the Create an instance UI.

General VM settings

We'll use the following settings:

• Name: dofroscra-test
• Region us-central1 (Iowa) and Zone us-central1-a
• Machine family: General Purpose > E2 > e2-micro (2 vCPU, 1 GB memory)
• Boot Disk: Debian GNU/Linux 11 (bullseye); 10 GB
• Identity and API access: Choose the "Docker PHP Tutorial deployment account" service account that we created previously
• Firewall: Allow HTTP traffic

Firewall and networks tags

Ticking the Allow HTTP traffic checkbox will cause two things when the instance is created:

• a new firewall rule named default-allow-http is created that allows incoming traffic from port 80 and is only applied to instances with the network tag http-server. You can see the new rule in the Firewall UI
• the network tag http-server is added to the instance, effectively enabling the aforementioned firewall rule for the instance

The role of the service account

The service account that we have chosen in the previous step will be attached to the Compute Instance. I.e. it will be available on the instance and we will have access to the account when we log into the instance. Conveniently, thegcloud cli is pre-installed on every Compute Instance:

If you're using an instance on Compute Engine, the gcloud CLI is installed by default.

The exact rules for what the service account can do are described in the Compute Engine docs for "Service accounts". In short: The possible actions will be constrained by the IAM permissions of the service account and the Access scopes of the Compute Instance which are set as outlined in the "Default scopes" section. Just take this as an aside, we won't have to modify anything here.

We will use the service account later, when we log into the Compute Instance and run docker pull from there to pull images from the registry.

Adding a public SSH key

In addition, I added my own public ssh key

ssh-rsa AAAAB3NzaC1yc2....6row== pascal.landau@MY_LAPTOP

CAUTION: The username for this key will be defined at the end of the key! E.g. in the example above, the username would be pascal.landau. This is important for logging in later via SSH from your local machine.

Define Availability Policies

We will make the instance preemptible, by choosing the VM provisioning model Spot. This makes it much cheaper but GCP "might" terminate the instance randomly if the capacity is needed somewhere else (and definitely after 24 hours). This is completely fine for our test use case. The final costs are ~2$ per month for this instance.

The actual VM creation

Finally, click the "Create" button at the bottom of the page

Once the instance is created, you can see it in the Compute Instances > VM instances UI

Next to the instance name dofroscra-test we can see the external IP address 35.192.212.130 Opening it in a browser via http://35.192.212.130/ won't show anything though, because we didn't deploy the application yet.

CAUTION: Make sure to shutdown and remove this instance by the end of the tutorial - otherwise compute costs will incur.

Log into a VM

There are multiple ways to log into the VM / Compute Instance outlined in Connecting to Linux VMs using advanced methods.

I'm going to describe three of them:

• Login via SSH from the GCP UI
• Login via SSH with your own key from your host machine
• Login using the Identity-Aware Proxy (IAP) concept

PS: It's worth keeping the Troubleshooting SSH guide as a bookmark.

Login via SSH from the GCP UI

Probably the easiest way to log in: Simply click the "SSH" button in the Compute Instances > VM instances UI next to the instance you want to log in. This will create a web shell that uses an ephemeral SSH key according to the GCP documentation: Connect to Linux VMs > Connect to VMs

When you connect to VMs using the Cloud Console, Compute Engine creates an ephemeral SSH key for you.

Login via SSH with your own key from your host machine

This method is probably closest to what you are used to from working with "other" VMs. In this case, the instance has to be publicly available (i.e. reachable "from the internet") and expose a port for SSH connections (usually 22). In addition, your public SSH key needs to be deployed.

All of those requirements are true for the Compute Instance that we just created:

• the public ip address is 35.192.212.130
• port 22 is open by default via the default-allow-ssh firewall rule, see How to Configure Firewall Rules in Google Cloud Platform and the official documentation on Troubleshooting SSH > By default, Compute Engine VMs allow SSH access on port 22.
• we added our public SSH key using pascal.landau as the username

So we can now simply login via

ssh pascal.landau@35.192.212.130

or if you need to specify the location of the private key via the -i option

ssh -i ~/.ssh/id_rsa pascal.landau@35.192.212.130

$ ssh pascal.landau@35.192.212.130
Linux dofroscra-test 4.19.0-20-cloud-amd64 #1 SMP Debian 4.19.235-1 (2022-03-17) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sun Apr 10 16:21:00 2022 from 54.74.228.207
pascal.landau@dofroscra-test:~$

Login using the Identity-Aware Proxy (IAP) concept

Note: This is the preferred way of logging into a GCP VM

To login via IAP we need the gcloud CLI that will use API requests (via HTTPS) under the hood to authenticate the Google user (or in our case: the service account) and then proxy the requests to the VM via GCP's Identity-Aware Proxy (IAP) as described in Connecting through Identity-Aware Proxy (IAP) for TCP and in more detail under Using IAP for TCP forwarding > Tunneling SSH connections.

The corresponding command is gcloud compute ssh, e.g.:

gcloud compute ssh dofroscra-test --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p

Note the --tunnel-through-iap flag: Without it, gcloud would instead attempt to use a "normal" ssh connection if the VM is publicly reachable as per docu:

If the instance doesn't have an external IP address, the connection automatically uses IAP TCP tunneling. If the instance does have an external IP address, the connection uses the external IP address instead of IAP TCP tunneling.

Output

$ gcloud compute ssh dofroscra-test --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p
WARNING: The private SSH key file for gcloud does not exist.
WARNING: The public SSH key file for gcloud does not exist.
WARNING: The PuTTY PPK SSH key file for gcloud does not exist.
WARNING: You do not have an SSH key for gcloud.
WARNING: SSH keygen will be executed to generate a key.
Updating project ssh metadata...
..............................................Updated [https://www.googleapis.com/compute/v1/projects/pl-dofroscra-p].
.done.
Waiting for SSH key to propagate.

Under the hood, this command will automatically create an SSH key pair on your local machine under ~/.ssh/ named google_compute_engine (unless they already exist) and upload the public key to the instance.

$ ls -l ~/.ssh/ | grep google_compute_engine
-rw-r--r-- 1 Pascal 197121 1675 Apr 11 09:25 google_compute_engine
-rw-r--r-- 1 Pascal 197121 1456 Apr 11 09:25 google_compute_engine.ppk
-rw-r--r-- 1 Pascal 197121  420 Apr 11 09:25 google_compute_engine.pub

It also opens a Putty session and logs you into in the instance:

Using username "Pascal".
Authenticating with public key "LAPTOP-0DNL2Q02\Pascal@LAPTOP-0DNL2Q02"
Linux dofroscra-test 4.19.0-20-cloud-amd64 #1 SMP Debian 4.19.235-1 (2022-03-17) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Mon Apr 11 07:22:32 2022 from 54.74.228.207
Pascal@dofroscra-test:~$

Caution: The Putty session will be closed automatically when you abort the original gcloud compute ssh command!

Additional notes on IAP

In order to enable SSH connections via IAP, our service account needs the following IAM roles:

• roles/compute.instanceAdmin.v1, role: Compute Admin
• roles/iam.serviceAccountUser, role: Service Account User
• roles/iap.tunnelResourceAccessor, role: IAP-secured Tunnel User

Otherwise, you might run into a couple of errors like:

ERROR: (gcloud.compute.ssh) Could not fetch resource:
 - Required 'compute.instances.get' permission for 'projects/pl-dofroscra-p/zones/us-central1-a/instances/dofroscra-test'

(roles/compute.instanceAdmin.v1 missing)

ERROR: (gcloud.compute.ssh) Could not add SSH key to instance metadata:
 - The user does not have access to service account 'docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com'.  User: 'docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com'.  Ask a project owner to grant you the iam.serviceAccountUser role on the service account

(roles/iam.serviceAccountUser missing)

Remote side unexpectedly closed connection

(roles/iap.tunnelResourceAccessor missing)

Note: This error can also occur if you try to use IAP directly after starting a VM. In this case wait a couple of seconds and try again.

The general "flow" looks like this

Using IAP might look overly complicated at first, but it offers a number of benefits:

• we can leverage Google's authentication system and the powerful IAM permission management to manage permissions - no more need to deploy custom SSH keys to the VMs

• we don't need a public IP address any longer, see Overview of TCP forwarding > How IAP's TCP forwarding works

  A special case, establishing an SSH connection using gcloud compute ssh wraps the SSH connection inside HTTPS and forwards it to the remote instance without the need of a listening port on local host.

  [...]

  TCP forwarding with IAP doesn't require a public, routable IP address assigned to your resource. Instead, it uses internal IPs.

  This is nice, because in our final PHP application only the nginx container should be accessible publicly - php-fpm and the php-workers shouldn't. Usually we would use a
  so-called Bastion Host or Jump Box to deal with this problem, but thanks to IAP we don't have to

Additional resources

• Accessing Secure Servers Using IAP
• Connecting Securely to Google Compute Engine VMs without a Public IP or VPN

Get root permissions

The group google-sudoers exists on each GCP Compute Instance. It is configured for passwordless sudo via /etc/sudoers.d/google_sudoers

$ sudo cat /etc/sudoers.d/google_sudoers 
%google-sudoers ALL=(ALL:ALL) NOPASSWD:ALL

I.e. each user added to this group can use sudo without password or simply run sudo -i to become root. Your user should be added automatically to that group. This can be verified with the id command (run on the VM)

Pascal@dofroscra-test:~$ id
uid=1002(Pascal) gid=1003(Pascal) groups=1003(Pascal),4(adm),30(dip),44(video),46(plugdev),1000(google-sudoers)

# => 1000(google-sudoers)

ssh and scp commands

It is quite common to copy files from / to a VM and to run commands on it. This is usually done via ssh and scp. The gcloud cli offers equivalent commands that can also make use of the Identity-Aware Proxy (IAP) concept:

gcloud ssh --command=""

Documentation: gcloud ssh --command

Example:

gcloud compute ssh dofroscra-test --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p --command="whoami"

Output:

$ gcloud compute ssh dofroscra-test --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p --command="whoami"
Pascal

gcloud scp

Documentation: gcloud scp

Example:

gcloud compute scp --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p ./local-file1.txt ./local-file2.txt dofroscra-test:tmp/

# ---
echo "1" >> ./local-file1.txt
echo "2" >> ./local-file2.txt
gcloud compute ssh dofroscra-test --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p --command="rm -rf ~/test; mkdir -p ~/test"
gcloud compute ssh dofroscra-test --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p --command="ls -l ~/test"
gcloud compute scp --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p ./local-file1.txt ./local-file2.txt dofroscra-test:test/
gcloud compute ssh dofroscra-test --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p --command="ls -l ~/test"

Output:

$ echo "1" >> ./local-file1.txt
$ echo "2" >> ./local-file2.txt
$ gcloud compute ssh dofroscra-test --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p --command="ls -l ~/test"
total 0

$ gcloud compute scp --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p ./local-file1.txt ./local-file2.txt dofroscra-test:test/
local-file1.txt           | 0 kB |   0.0 kB/s | ETA: 00:00:00 | 100%
local-file2.txt           | 0 kB |   0.0 kB/s | ETA: 00:00:00 | 100%

$ gcloud compute ssh dofroscra-test --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p --command="ls -l ~/test"
total 8
-rw-r--r-- 1 Pascal Pascal 4 Jun  2 13:06 local-file1.txt
-rw-r--r-- 1 Pascal Pascal 4 Jun  2 13:06 local-file2.txt

Note: According to the examples in the documentation, it should also be possible to use the ~ to define that the destination on the remote VM is relative to the home directory. However, this did not work for me, as I kept getting the error

$ gcloud compute scp --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p ./local-file1.txt dofroscra-test:~/test
pscp: remote filespec ~/test: not a directory

even though the diretory existed. But removing the ~/ part worked

$ gcloud compute scp --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p ./local-file1.txt  dofroscra-test:test/
local-file1.txt           | 0 kB |   0.0 kB/s | ETA: 00:00:00 | 100%

Provision the VM

Get the secret gpg key and password from the Secret Manager

Before we shift our attention to docker, let's quickly deal with the secrets. We won't need them for this part of the tutorial. but will need the secret gpg key and its password to decrypt the secrets in the php containers in the next part. To recap: We have added the them previously and can now retrieve them as explained under section Retrieve a secret via the gcloud cli via

gcloud secrets versions access 1 --secret=GPG_KEY > secret.gpg

GPG_PASSWORD=$(gcloud secrets versions access 1 --secret=GPG_PASSWORD)

$ gcloud secrets versions access 1 --secret=GPG_KEY > secret.gpg
$ head secret.gpg
-----BEGIN PGP PRIVATE KEY BLOCK-----

lQPGBGKA1psBCACq5zYDT587CVZEIWXbUplfAGQZOQJALmzErYpTp0jt+rp4vJhR
U5xahy3pqCq81Cnny5YME50ybB3pW/WcHxWLBDo+he8PKeLbp6wFFjJns+3u4opH
9gFMElyHpzTGiDQYfx/CgY2hKz7GSqpjmnOaKxYvGv0EsbZczyHY1WIN/YFzb0tI
tY7J4zTSH05I+aazRdHyn28QcCRcIT9+4q+5Vk8gz8mmgoqVpyeNgQcqJjcd03iP
WUZd1vZCumOvdG5PZNlc/wPFhqLDmYyLmJ7pt5bWIgty9BjYK8Z2NOdUaekqVEJ+
r29HbzwgFLLE2gd52f07h2y2YgMdWdz4FDxVABEBAAH+BwMC9veBYT2oigXxExLl
7fZKVjw02lEr1NpYd5X1ge9WPU/1qumATJWounzciiETpsYGsbPd9zFRJP4E3JZl
sFSh4p0/kXYTuenYD8wgGkeYyN4lm53IHfqSn2z9JMW5Kz9XEODtKJl8fjcn9Zeb

$ GPG_PASSWORD=$(gcloud secrets versions access 1 --secret=GPG_PASSWORD)
$ echo $GPG_PASSWORD
87654321

Installing docker and docker compose

Since we created the VM with a Debian OS, we'll follow the official Debian installation instructions for Docker Engine and run the following commands while we are logged into the VM:

# install required tools
sudo apt-get update -yq && apt-get install -yq \
     ca-certificates \
     curl \
     gnupg \
     lsb-release

# add Docker’s official GPG key
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

# set up the stable repository
echo \
 "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
 $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# install Docker Engine
sudo apt-get update -yq && sudo apt-get install -yq \
     docker-ce \
     docker-ce-cli \
     containerd.io \
     docker-compose-plugin

I have also added these instructions to the script .infrastructure/scripts/provision.sh.

Afterwards we check via

docker --version
docker compose version

if docker and docker compose are available

$ docker --version
Docker version 20.10.15, build fd82621

$ docker compose version
Docker Compose version v2.5.0

Authenticate docker via gcloud

I recommend running the commands as root via

sudo -i

pascal_landau@dofroscra-test:~$ sudo -i
root@dofroscra-test:~# 

Otherwise, we might run into docker permission errors like

Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/images/create?fromImage=...": dial unix /var/run/docker.sock: connect: permission denied

FYI: As an alternative we could also add the non-root user to the docker group via

user=$(whoami)
sudo usermod -a -G docker $user

$ user=$(whoami)
$ sudo usermod -a -G docker $user
$ id
uid=1001(pascal_landau) gid=1002(pascal_landau) groups=1002(pascal_landau),4(adm),30(dip),44(video),46(plugdev),997(docker),1000(google-sudoers)

Locally, we could use the JSON key file of the service account for authentication, but we don't have access to that key file on the VM. Instead, we will authenticate docker via the pre-installed gcloud cli and the attached service account as described in the GCP docs for the Container Registry Authentication methods under section "gcloud credential helper" via

gcloud auth configure-docker --quiet

Adding credentials for all GCR repositories.
WARNING: A long list of credential helpers may cause delays running 'docker build'. We recommend passing the registry name to configure only the registry you are using.
Docker configuration file updated.

This creates the file /root/.docker/config.json that we already encountered when we authenticated docker locally to push images. In this case it has the following content

$ cat /root/.docker/config.json
{
  "credHelpers": {
    "gcr.io": "gcloud",
    "us.gcr.io": "gcloud",
    "eu.gcr.io": "gcloud",
    "asia.gcr.io": "gcloud",
    "staging-k8s.gcr.io": "gcloud",
    "marketplace.gcr.io": "gcloud"
  }
}

The creadHelpers are described in the docker login docs

Credential helpers are similar to the credential store above, but act as the designated programs to handle credentials for specific registries. The default credential store (credsStore or the config file itself) will not be used for operations concerning credentials of the specified registries.

In other words:

• we are using gcr.io as registry
• this registry is defined to use the credential helper gcloud
• i.e. it will use the pre-initialized gcloud cli for authentication

Pulling the nginx image

Once we are authenticated, we can simply run docker pull with the full image name to retrieve the nginx image that we pushed previously

docker pull gcr.io/pl-dofroscra-p/my-nginx

$ docker pull gcr.io/pl-dofroscra-p/my-nginx
Using default tag: latest
latest: Pulling from pl-dofroscra-p/my-nginx
59bf1c3509f3: Pull complete 
f3322597df46: Pull complete 
d09cf91cabdc: Pull complete 
3a97535ac2ef: Pull complete 
919ade35f869: Pull complete 
40e5d2fe5bcd: Pull complete 
c72acb0c83a5: Pull complete 
d6baa2bee4a5: Pull complete 
Digest: sha256:0740591fb686227d8cdf4e42b784f634cbaf9f5caa6ee478e3bcc24aeef75d7f
Status: Downloaded newer image for gcr.io/pl-dofroscra-p/my-nginx:latest
gcr.io/pl-dofroscra-p/my-nginx:latest

If we wouldn't be authenticated, we would run into the following error

$ docker pull gcr.io/pl-dofroscra-p/my-nginx
Using default tag: latest
Error response from daemon: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication

Start the nginx container

For now, we will simply run the nginx container with docker run via

docker run --name nginx -p 80:80 --rm -d gcr.io/pl-dofroscra-p/my-nginx:latest

We

• give it the name nginx so we can easily reference it later via --name nginx
• map port 80 from the host to port 80 of the container so that HTTP requests to the VM are handled via the container via -p 80:80, see Docker docs on "Publish or expose port (-p, --expose)"
• make it run in the background via -d (--detach) and remove it after shutdown automatically via -rm (--remove)

Output

root@dofroscra-test:~# docker run -p 80:80 -d --name nginx gcr.io/pl-dofroscra-p/my-nginx:latest
dd49bedad97c06f698d06a140c5091c04ad81b2f75632e222927e7f71cf28c18

root@dofroscra-test:~# docker logs nginx
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf differs from the packaged version
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2022/05/08 12:01:43 [notice] 1#1: using the "epoll" event method
2022/05/08 12:01:43 [notice] 1#1: nginx/1.21.5
2022/05/08 12:01:43 [notice] 1#1: built by gcc 10.3.1 20211027 (Alpine 10.3.1_git20211027) 
2022/05/08 12:01:43 [notice] 1#1: OS: Linux 4.19.0-20-cloud-amd64
2022/05/08 12:01:43 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2022/05/08 12:01:43 [notice] 1#1: start worker processes
2022/05/08 12:01:43 [notice] 1#1: start worker process 30
2022/05/08 12:01:43 [notice] 1#1: start worker process 31

We're all set to serve HTTP requests: The requests will be passed to the nginx container due to the mapped port 80, i.e. we can simply use the public IP address 35.192.212.130 in a curl request and check the "Hello world" file we've added to the image:

$ curl -s http://35.192.212.130/hello.html
Hello world

Automate via gcloud commands

So far we have mostly used the UI to configure everything. This is great for understanding how things work, but it's not so great for maintainability:

• UIs change
• "Clicking" stuff takes quite some time
• we can't automate anything

Fortunately, we can also use the gcloud cli for most of the tasks.

Preconditions: Project and Owner service account

I'll assume that

• gcloud is installed
• a GCP project exists

In addition, I'll manually create a new "master" service account and assign it the role Owner. We'll use that service account in the gcloud cli to enable all the APIs and manage the resources.

FYI: We could also do this with our personal GCP user, but I plan to use terraform in a later tutorial which will require a service account anyway.

We will use docker-php-tutorial-master as service account ID and store the key file of the account at the root of the codebase under gcp-master-service-account-key.json (which is also added to the .gitignore file).

Configure gcloud to use the master service account

Run

gcloud auth activate-service-account --key-file=./gcp-master-service-account-key.json --project=pl-dofroscra-p

$ gcloud auth activate-service-account --key-file=./gcp-master-service-account-key.json --project=pl-dofroscra-p
Activated service account credentials for: [docker-php-tutorial-master@pl-dofroscra-p.iam.gserviceaccount.com]

Enable APIs

APIs can be enabled via gcloud services enable $serviceName, (see also the Docu on "Enabling an API in your Google Cloud project").

The $serviceName is shown on the API overview page of a service, see the following example for the Container Registry

We need the APIs for

• Container Registry: containerregistry.googleapis.com
• Secret Manager: secretmanager.googleapis.com
• Compute Engine: compute.googleapis.com
• IAM: iam.googleapis.com
• Cloud Storage: storage.googleapis.com
• Cloud Resource Manager: cloudresourcemanager.googleapis.com

gcloud services enable containerregistry.googleapis.com secretmanager.googleapis.com compute.googleapis.com iam.googleapis.com storage.googleapis.com cloudresourcemanager.googleapis.com

$ gcloud services enable containerregistry.googleapis.com secretmanager.googleapis.com compute.googleapis.com iam.googleapis.com storage.googleapis.com cloudresourcemanager.googleapis.com
Operation "operations/acat.p2-386551299607-87333b29-c7eb-40b8-b951-8c86185bbf49" finished successfully.

Create and configure a "deployment" service account

We won't use the master Owner service account for the deployment but create a custom one with only the necessary permissions.

Service accounts are created via gcloud iam service-accounts create $serviceAccountId, (see also the Docu on "Creating and managing service accounts").

The $serviceAccountId is the id of the service account, e.g. docker-php-tutorial-deployment in our previous example. In addition, we can define a --description and a --display-name.

gcloud iam service-accounts create docker-php-tutorial-deployment \
  --description="Used for the deployment of the Docker PHP Tutorial application" \
  --display-name="Docker PHP Tutorial Deployment Account"

$ gcloud iam service-accounts create docker-php-tutorial-deployment \
>   --description="Used for the deployment of the Docker PHP Tutorial application" \
>   --display-name="Docker PHP Tutorial Deployment Account"
Created service account [docker-php-tutorial-deployment].

Then, we need a key file for authentication. It can be created via gcloud iam service-accounts keys create $localPathToKeyFile --iam-account=$serviceAccountEmail (see also the Docu on "Create and manage service account keys").

The $localPathToKeyFile is used to store the key file locally, e.g. gcp-service-account-key.json in our previous example and $serviceAccountEmail is the email address of the service account. It has the form

$serviceAccountId@$projectId.iam.gserviceaccount.com

e.g.

docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com

Example:

gcloud iam service-accounts keys create ./gcp-service-account-key.json \
  --iam-account=docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com

$ gcloud iam service-accounts keys create ./gcp-service-account-key.json \
>   --iam-account=docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com
created key [810df0c6df21de44dc5e431d2b569d74555ba3f9] of type [json] as [./gcp-service-account-key.json] for [docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com]

Finally, we must assign the required IAM roles via gcloud projects add-iam-policy-binding $projectName --member=serviceAccount:$serviceAccountEmail --role=$roleId, (see also the Docu on "Manage access to projects, folders, and organizations").

The $projectName is the name of the GCP project, $serviceAccountEmail the email address from before and the $roleId the name of the role as listed under Understanding roles. In our case that's:

• Storage Admin: roles/storage.admin
• Secret Manager Admin: roles/secretmanager.admin
• Compute Admin: roles/compute.admin
• Service Account User: roles/iam.serviceAccountUser
• IAP-secured Tunnel User: roles/iap.tunnelResourceAccessor

projectName="pl-dofroscra-p"
serviceAccountEmail="docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com"

gcloud projects add-iam-policy-binding $projectName --member=serviceAccount:$serviceAccountEmail --role=roles/storage.admin
gcloud projects add-iam-policy-binding $projectName --member=serviceAccount:$serviceAccountEmail --role=roles/secretmanager.admin
gcloud projects add-iam-policy-binding $projectName --member=serviceAccount:$serviceAccountEmail --role=roles/compute.admin
gcloud projects add-iam-policy-binding $projectName --member=serviceAccount:$serviceAccountEmail --role=roles/iam.serviceAccountUser
gcloud projects add-iam-policy-binding $projectName --member=serviceAccount:$serviceAccountEmail --role=roles/iap.tunnelResourceAccessor

$ projectName="pl-dofroscra-p"
$ serviceAccountEmail="docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com"
$ gcloud projects add-iam-policy-binding $projectName --member=serviceAccount:$serviceAccountEmail --role=roles/storage.admin
Updated IAM policy for project [pl-dofroscra-p].
bindings:
- members:
  - serviceAccount:service-386551299607@compute-system.iam.gserviceaccount.com
  role: roles/compute.serviceAgent
- members:
  - serviceAccount:service-386551299607@containerregistry.iam.gserviceaccount.com
  role: roles/containerregistry.ServiceAgent
- members:
  - serviceAccount:386551299607-compute@developer.gserviceaccount.com
  - serviceAccount:386551299607@cloudservices.gserviceaccount.com
  role: roles/editor
- members:
  - serviceAccount:docker-php-tutorial-master@pl-dofroscra-p.iam.gserviceaccount.com
  - user:pascal.landau@gmail.com
  role: roles/owner
- members:
  - serviceAccount:service-386551299607@gcp-sa-pubsub.iam.gserviceaccount.com
  role: roles/pubsub.serviceAgent
- members:
  - serviceAccount:docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com
  role: roles/storage.admin
etag: BwXgKxHg7gA=
version: 1

# ...

Updated IAM policy for project [pl-dofroscra-p].
bindings:
- members:
  - serviceAccount:docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com
  role: roles/compute.admin
- members:
  - serviceAccount:service-386551299607@compute-system.iam.gserviceaccount.com
  role: roles/compute.serviceAgent
- members:
  - serviceAccount:service-386551299607@containerregistry.iam.gserviceaccount.com
  role: roles/containerregistry.ServiceAgent
- members:
  - serviceAccount:386551299607-compute@developer.gserviceaccount.com
  - serviceAccount:386551299607@cloudservices.gserviceaccount.com
  role: roles/editor
- members:
  - serviceAccount:docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com
  role: roles/iam.serviceAccountUser
- members:
  - serviceAccount:docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com
  role: roles/iap.tunnelResourceAccessor
- members:
  - serviceAccount:docker-php-tutorial-master@pl-dofroscra-p.iam.gserviceaccount.com
  - user:pascal.landau@gmail.com
  role: roles/owner
- members:
  - serviceAccount:service-386551299607@gcp-sa-pubsub.iam.gserviceaccount.com
  role: roles/pubsub.serviceAgent
- members:
  - serviceAccount:docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com
  role: roles/secretmanager.admin
- members:
  - serviceAccount:docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com
  role: roles/storage.admin
etag: BwXgKxm0Vtk=
version: 1

Create secrets

Secrets are created via gcloud secrets create $secretId", (see also the Docu on "Creating and accessing secrets").

The $secretId is the name of the secret, e.g. my_secret_key in our previous example, and we must also add a new version with the value of the secret via gcloud secrets versions add $secretId --data-file="/path/to/file.txt".

gcloud secrets create my_secret_key

echo -n "my_secret_value" | gcloud secrets versions add my_secret_key --data-file=-

$ gcloud secrets create my_secret_key
Created secret [my_secret_key].

$ echo -n "my_secret_value" | gcloud secrets versions add my_secret_key --data-file=-
Created version [1] of the secret [my_secret_key].

We also need to do the same for the gpg secret key and its password:

gcloud secrets create GPG_KEY
echo gcloud secrets versions add GPG_KEY --data-file=secret-production-protected.gpg.example

gcloud secrets create GPG_PASSWORD
echo -n "87654321" | gcloud secrets versions add GPG_PASSWORD --data-file=-

$ gcloud secrets create GPG_KEY
Created secret [GPG_KEY].
$ gcloud secrets versions add GPG_KEY --data-file=secret-production-protected.gpg.example
Created version [1] of the secret [GPG_KEY].

$ gcloud secrets create GPG_PASSWORD
Created secret [GPG_PASSWORD].
$ echo -n "87654321" | gcloud secrets versions add GPG_PASSWORD --data-file=-
Created version [1] of the secret [GPG_PASSWORD].

Create firewall rule for HTTP traffic

Firewall rules can be created via gcloud compute firewall-rules create $ruleName (see also the Docu on "Using firewall rules").

We'll stick to the same conventions that have been used when creating the VM via the UI by using default-allow-http as the $ruleName and http-server as the network tag (via the --target-tags option)

 gcloud compute firewall-rules create default-allow-http --allow tcp:80 --target-tags=http-server

$ gcloud compute firewall-rules create default-allow-http --allow tcp:80 --target-tags=http-server
Creating firewall...
..Created [https://www.googleapis.com/compute/v1/projects/pl-dofroscra-p/global/firewalls/default-allow-http].
done.
NAME                NETWORK  DIRECTION  PRIORITY  ALLOW   DENY  DISABLED
default-allow-http  default  INGRESS    1000      tcp:80        False

Create a Compute Instance VM

Compute Instances can be created via gcloud compute instances create $vmName (see also the Docu on "Creating and starting a VM instance").

The $vmName defines the name of the VM, e.g. dofroscra-test in the previous example. In addition, there are a lot of options to customize the VM, e.g.

• --image-family and --image-project
  • define the operating system, e.g. --image-family="debian-11" and --image-project=debian-cloud
  • See Docu on "Operating system details" for a list of available values
• --machine-type
  • defines the machine type (i.e. the "specs" like CPUs and memory), e.g. --machine-type=e2-micro
  • See Docu on "About machine families" that contains links to the machine type categories with the concrete values, e.g. the General-purpose machine family
• etc.

The docu is doing a great job at describing all available configuration options. Luckily, we can make our lives a little easier, configure the VM via UI and then click the "EQUIVALENT COMMAND LINE" button at the end of the page to get a copy-paste-ready gcloud compute instances create command.

Note that we are using dofroscra-test as the instance name and us-central1-a as the zone.

gcloud compute instances create dofroscra-test \
    --project=pl-dofroscra-p \
    --zone=us-central1-a \
    --machine-type=e2-micro \
    --network-interface=network-tier=PREMIUM,subnet=default \
    --no-restart-on-failure \
    --maintenance-policy=TERMINATE \
    --provisioning-model=SPOT \
    --instance-termination-action=STOP \
    --service-account=docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com \
    --scopes=https://www.googleapis.com/auth/cloud-platform \
    --tags=http-server \
    --create-disk=auto-delete=yes,boot=yes,device-name=dofroscra-test,image=projects/debian-cloud/global/images/debian-11-bullseye-v20220519,mode=rw,size=10,type=projects/pl-dofroscra-p/zones/us-central1-a/diskTypes/pd-balanced \
    --no-shielded-secure-boot \
    --shielded-vtpm \
    --shielded-integrity-monitoring \
    --reservation-affinity=any

Created [https://www.googleapis.com/compute/v1/projects/pl-dofroscra-p/zones/us-central1-a/instances/dofroscra-test-1].

NAME            ZONE           MACHINE_TYPE  PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP     STATUS
dofroscra-test  us-central1-a  e2-small      true         10.128.0.2   34.122.227.169  RUNNING

Provisioning

For provisioning, we need to install docker and authenticate the root user to pull images from our registry. We've already created an installation script at .infrastructure/scripts/provision.sh and the easiest way to run it on the VM is to transmit it via scp

gcloud compute scp --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p ./.infrastructure/scripts/provision.sh dofroscra-test:provision.sh

$ gcloud compute scp --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p ./.infrastructure/scripts/provision.sh dofroscra-test:provision.sh
provision.sh              | 0 kB |   0.7 kB/s | ETA: 00:00:00 | 100%

The previous command transmitted the script in the home directory of the user, and we can now execute it via

gcloud compute ssh dofroscra-test --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p --command="bash provision.sh"

$ gcloud compute ssh dofroscra-test --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p --command="bash provision.sh"
Hit:1 https://download.docker.com/linux/debian bullseye InRelease
Hit:2 http://packages.cloud.google.com/apt cloud-sdk-bullseye InRelease
# ...
The following additional packages will be installed:
  dbus-user-session docker-ce-rootless-extras docker-scan-plugin git git-man
  libcurl3-gnutls liberror-perl libgdbm-compat4 libltdl7 libperl5.32 libslirp0
  patch perl perl-modules-5.32 pigz slirp4netns
Suggested packages:
  aufs-tools cgroupfs-mount | cgroup-lite git-daemon-run | git-daemon-sysvinit
  git-doc git-el git-email git-gui gitk gitweb git-cvs git-mediawiki git-svn
  ed diffutils-doc perl-doc libterm-readline-gnu-perl
  | libterm-readline-perl-perl make libtap-harness-archive-perl
The following NEW packages will be installed:
  containerd.io dbus-user-session docker-ce docker-ce-cli
  docker-ce-rootless-extras docker-compose-plugin docker-scan-plugin git
  git-man libcurl3-gnutls liberror-perl libgdbm-compat4 libltdl7 libperl5.32
  libslirp0 patch perl perl-modules-5.32 pigz slirp4netns
0 upgraded, 20 newly installed, 0 to remove and 6 not upgraded.
Need to get 124 MB of archives.
After this operation, 535 MB of additional disk space will be used.
# ...
Setting up docker-ce (5:20.10.16~3-0~debian-bullseye) ...
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /lib/systemd/system/docker.service.
Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket.
Setting up liberror-perl (0.17029-1) ...
Setting up git (1:2.30.2-1) ...
Processing triggers for man-db (2.9.4-2) ...
Processing triggers for libc-bin (2.31-13+deb11u3) ...

Once docker is installed, we can run the authentication of the root user using sudo su root -c via

gcloud compute ssh dofroscra-test --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p --command="sudo su root -c 'gcloud auth configure-docker --quiet'"

$ gcloud compute ssh dofroscra-test --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p --command="sudo su root -c 'gcloud auth configure-docker --quiet'"
Adding credentials for all GCR repositories.
WARNING: A long list of credential helpers may cause delays running 'docker build'. We recommend passing the registry name to configure only the registry you are using.
gcloud credential helpers already registered correctly.

Deployment

We're almost done - the last step in the process consists of

• building the docker image locally using the correct tag gcr.io/pl-dofroscra-p/my-nginx

  docker build -t "gcr.io/pl-dofroscra-p/my-nginx" -f - . <<EOF
  FROM nginx:1.21.5-alpine
  
  RUN echo "Hello world" >> /usr/share/nginx/html/hello.html
  
  EOF
  

  $ docker build -t "gcr.io/pl-dofroscra-p/my-nginx" --no-cache -f - . <<EOF
  FROM nginx:1.21.5-alpine
  RUN echo "Hello world" >> /usr/share/nginx/html/hello.html
  EOF
  
  #1 [internal] load build definition from Dockerfile
  #1 sha256:21ee68236cc00e4d1638480de32fd6304d796e6a72306082d3164e9164073843
  
  #...
  
  #5 [2/2] RUN echo "Hello world" >> /usr/share/nginx/html/hello.html
  #5 sha256:72dee3f3637a6b78ff7e50591e3e9108e2b93eee09443e19890f9cb36b35bdc6
  #5 DONE 0.5s
  
  #6 exporting to image
  #6 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
  #6 exporting layers 0.1s done
  #6 writing image sha256:d0b23a80ebd7311953eeff3818b58007e8747e531e0128eef820f39105f9f1fe done
  #6 naming to gcr.io/pl-dofroscra-p/my-nginx done
  #6 DONE 0.1s
  

• local authentication

  cat ./gcp-service-account-key.json | docker login -u _json_key --password-stdin https://gcr.io
  

  $ cat ./gcp-service-account-key.json | docker login -u _json_key --password-stdin https://gcr.io
  Login Succeeded
  
  Logging in with your password grants your terminal complete access to your account.
  For better security, log in with a limited-privilege personal access token. Learn more at https://docs.docker.com/go/access-tokens/
  

• pushing it to the registry

  docker push gcr.io/pl-dofroscra-p/my-nginx
  

  
  $ docker push gcr.io/pl-dofroscra-p/my-nginx
  Using default tag: latest
  The push refers to repository [gcr.io/pl-dofroscra-1/my-nginx]
  ad4683501621: Preparing
  # ...
  ad4683501621: Pushed
  latest: digest: sha256:680086b3c77e4b895099c0a5f6e713ff79ba0d78e1e1df1bc2546d6f979126e4 size: 1775
  

• and "deploying it on the VM"

For the last step we will once again use a script that we transmit to the VM: .infrastructure/scripts/deploy.sh

#!/usr/bin/env bash

usage="Usage: deploy.sh image_name"
[ -z "$1" ] &&  echo "No image_name given! $usage" && exit 1

image_name=$1
container_name=nginx

echo "Pulling '${image_name}' from registry"
sudo docker pull "${image_name}"

echo "Starting container"
sudo docker kill "${container_name}"; sudo docker run --name "${container_name}" -p 80:80 --rm -d "${image_name}"

echo "Getting secret GPG_KEY"
gcloud secrets versions access latest --secret=GPG_KEY >> ./secret.gpg
head ./secret.gpg

echo "Getting secret GPG_PASSWORD"
GPG_PASSWORD=$(gcloud secrets versions access latest --secret=GPG_PASSWORD)
echo $GPG_PASSWORD

The script will pull the image on the VM from the registry

$ sudo docker pull 'gcr.io/pl-dofroscra-p/my-nginx'
Using default tag: latest
latest: Pulling from pl-dofroscra-p/my-nginx
# ...
Digest: sha256:680086b3c77e4b895099c0a5f6e713ff79ba0d78e1e1df1bc2546d6f979126e4
Status: Downloaded newer image for gcr.io/pl-dofroscra-p/my-nginx:latest
gcr.io/pl-dofroscra-p/my-nginx:latest

start a container with the image

$ docker kill "${container_name}"; sudo docker run --name "${container_name}" -p 80:80 --rm -d "${image_name}"
Error response from daemon: Cannot kill container: nginx: No such container: nginx
8ac0cc055041e18d3ce244ded49c82be985e580c2c9f316c6d6ef7c7bf3bc0b5

and finally retrieve the secrets (just to demonstrate that it works)

$ gcloud secrets versions access latest --secret=GPG_KEY >> ./secret.gpg
$ head ./secret.gpg
-----BEGIN PGP PRIVATE KEY BLOCK-----

lQPGBGKA1psBCACq5zYDT587CVZEIWXbUplfAGQZOQJALmzErYpTp0jt+rp4vJhR
U5xahy3pqCq81Cnny5YME50ybB3pW/WcHxWLBDo+he8PKeLbp6wFFjJns+3u4opH
9gFMElyHpzTGiDQYfx/CgY2hKz7GSqpjmnOaKxYvGv0EsbZczyHY1WIN/YFzb0tI
tY7J4zTSH05I+aazRdHyn28QcCRcIT9+4q+5Vk8gz8mmgoqVpyeNgQcqJjcd03iP
WUZd1vZCumOvdG5PZNlc/wPFhqLDmYyLmJ7pt5bWIgty9BjYK8Z2NOdUaekqVEJ+
r29HbzwgFLLE2gd52f07h2y2YgMdWdz4FDxVABEBAAH+BwMC9veBYT2oigXxExLl
7fZKVjw02lEr1NpYd5X1ge9WPU/1qumATJWounzciiETpsYGsbPd9zFRJP4E3JZl
sFSh4p0/kXYTuenYD8wgGkeYyN4lm53IHfqSn2z9JMW5Kz9XEODtKJl8fjcn9Zeb

$ GPG_PASSWORD=$(gcloud secrets versions access latest --secret=GPG_PASSWORD)
$ echo $GPG_PASSWORD
87654321

Bonus: For retrieving the puplic IP address of the Compute Instance VM we can use the gcloud compute instances describe $instanceName command:

gcloud compute instances describe dofroscra-test --zone us-central1-a --project=pl-dofroscra-p --format='get(networkInterfaces[0].accessConfigs[0].natIP)'

$ ip=$(gcloud compute instances describe dofroscra-test --zone us-central1-a --project=pl-dofroscra-p --format='get(networkInterfaces[0].accessConfigs[0].natIP)')
$ echo $ip
35.224.250.208
$ curl -s "http://${ip}/hello.html"
Hello world

Putting it all together

Since none of the previous steps requires manual intervention any longer, I have created a script at .infrastructure/setup-gcp.sh to run everything from Configure gcloud to use the master service account to Provisioning:

#!/usr/bin/env bash

usage="Usage: deploy.sh project_id vm_name"
[ -z "$1" ] &&  echo "No project_id given! $usage" && exit 1
[ -z "$2" ] &&  echo "No vm_name given! $usage" && exit 1

GREEN="\033[0;32m"
NO_COLOR="\033[0m"

project_id=$1
vm_name=$2
vm_zone=us-central1-a
master_service_account_key_location=./gcp-master-service-account-key.json
deployment_service_account_id=deployment
deployment_service_account_key_location=./gcp-service-account-key.json
deployment_service_account_mail="${deployment_service_account_id}@${project_id}.iam.gserviceaccount.com"
gpg_secret_key_location=secret-production-protected.gpg.example
gpg_secret_key_password=87654321

printf "${GREEN}Setting up GCP project for${NO_COLOR}\n"
echo "==="
echo "project_id: ${project_id}"
echo "vm_name:    ${vm_name}"

printf "${GREEN}Activating master service account${NO_COLOR}\n"
gcloud auth activate-service-account --key-file="${master_service_account_key_location}" --project="${project_id}"

printf "${GREEN}Enabling APIs${NO_COLOR}\n"
gcloud services enable \
  containerregistry.googleapis.com \
  secretmanager.googleapis.com \
  compute.googleapis.com \
  iam.googleapis.com \
  storage.googleapis.com \
  cloudresourcemanager.googleapis.com

printf "${GREEN}Creating deployment service account with id '${deployment_service_account_id}'${NO_COLOR}\n"
gcloud iam service-accounts create "${deployment_service_account_id}" \
  --description="Used for the deployment application" \
  --display-name="Deployment Account"

printf "${GREEN}Creating JSON key file for deployment service account at ${deployment_service_account_key_location}${NO_COLOR}\n"
gcloud iam service-accounts keys create "${deployment_service_account_key_location}" \
  --iam-account="${deployment_service_account_mail}"

printf "${GREEN}Adding roles for service account${NO_COLOR}\n"  
roles="storage.admin secretmanager.admin compute.admin iam.serviceAccountUser iap.tunnelResourceAccessor"

for role in $roles; do
  gcloud projects add-iam-policy-binding "${project_id}" --member=serviceAccount:"${deployment_service_account_mail}" "--role=roles/${role}"
done;

printf "${GREEN}Creating secrets${NO_COLOR}\n"
gcloud secrets create GPG_KEY
echo gcloud secrets versions add GPG_KEY --data-file="${gpg_secret_key_location}"

gcloud secrets create GPG_PASSWORD
echo -n "${gpg_secret_key_password}" | gcloud secrets versions add GPG_PASSWORD --data-file=-

printf "${GREEN}Creating firewall rule to allow HTTP traffic${NO_COLOR}\n"
gcloud compute firewall-rules create default-allow-http --allow tcp:80 --target-tags=http-server

printf "${GREEN}Creating a Compute Instance VM${NO_COLOR}\n"
gcloud compute instances create "${vm_name}" \
    --project="${project_id}" \
    --zone="${vm_zone}" \
    --machine-type=e2-micro \
    --network-interface=network-tier=PREMIUM,subnet=default \
    --no-restart-on-failure \
    --maintenance-policy=TERMINATE \
    --provisioning-model=SPOT \
    --instance-termination-action=STOP \
    --service-account="${deployment_service_account_mail}" \
    --scopes=https://www.googleapis.com/auth/cloud-platform \
    --tags=http-server \
    --create-disk=auto-delete=yes,boot=yes,device-name="${vm_name}",image=projects/debian-cloud/global/images/debian-11-bullseye-v20220519,mode=rw,size=10,type=projects/"${project_id}"/zones/"${vm_zone}"/diskTypes/pd-balanced \
    --no-shielded-secure-boot \
    --shielded-vtpm \
    --shielded-integrity-monitoring \
    --reservation-affinity=any

printf "${GREEN}Activating deployment service account${NO_COLOR}\n"
gcloud auth activate-service-account --key-file="${deployment_service_account_key_location}" --project="${project_id}"

printf "${GREEN}Transferring provisioning script${NO_COLOR}\n"
echo "Waiting 60s for the instance to be fully ready to receive IAP connections"
sleep 60
gcloud compute scp --zone ${vm_zone} --tunnel-through-iap --project=${project_id} ./.infrastructure/scripts/provision.sh ${vm_name}:provision.sh

printf "${GREEN}Executing provisioning script${NO_COLOR}\n"
gcloud compute ssh ${vm_name} --zone ${vm_zone} --tunnel-through-iap --project=${project_id} --command="bash provision.sh"

printf "${GREEN}Authenticating docker via gcloud in the VM${NO_COLOR}\n"
gcloud compute ssh ${vm_name} --zone ${vm_zone} --tunnel-through-iap --project=${project_id} --command="sudo su root -c 'gcloud auth configure-docker --quiet'"

printf "\n\n${GREEN}Provisioning done!${NO_COLOR}\n"

$ bash .infrastructure/setup-gcp.sh pl-dofroscra-p dofroscra-test
Setting up GCP project for
===
project_id: pl-dofroscra-p
vm_name:    dofroscra-test
Activating master service account
Activated service account credentials for: [master@pl-dofroscra-p.iam.gserviceaccount.com]
Enabling APIs
Operation "operations/acf.p2-305055072470-625a52a9-96a7-4e2f-831e-a76491c8882c" finished successfully.
Creating deployment service account with id 'deployment'
Created service account [deployment].
Creating JSON key file for deployment service account at ./gcp-service-account-key.json
created key [fa74d605f5891a6f875a2663b6beeea425730b5b] of type [json] as [./gcp-service-account-key.json] for [deployment@pl-dofroscra-p.iam.gserviceaccount.com]
Adding roles for service account
Updated IAM policy for project [pl-dofroscra-p].
bindings:
# ...
- members:
  - serviceAccount:deployment@pl-dofroscra-p.iam.gserviceaccount.com
  role: roles/storage.admin
etag: BwXgw51MPTM=
version: 1
Creating secrets
Created secret [GPG_KEY].
Created version [1] of the secret [GPG_KEY].
Created secret [GPG_PASSWORD].
Created version [1] of the secret [GPG_PASSWORD].
Creating firewall rule to allow HTTP traffic
Creating firewall...
..Created [https://www.googleapis.com/compute/v1/projects/pl-dofroscra-p/global/firewalls/default-allow-http].
done.
Creating a Compute Instance VM
Created [https://www.googleapis.com/compute/v1/projects/pl-dofroscra-p/zones/us-central1-a/instances/dofroscra-test].
NAME        ZONE           MACHINE_TYPE  PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP    STATUS
dofroscra-test  us-central1-a  e2-micro      true         10.128.0.2   34.70.213.101  RUNNING
Activating deployment service account
Activated service account credentials for: [deployment@pl-dofroscra-p.iam.gserviceaccount.com]
Transferring provisioning script
Updating project ssh metadata...
..........................................................................Updated [https://www.googleapis.com/compute/v1/projects/pl-dofroscra-p].
.done.
Waiting for SSH key to propagate.
# ...
provision.sh              | 0 kB |   0.7 kB/s | ETA: 00:00:00 | 100%)
Executing provisioning script
Get:1 http://security.debian.org/debian-security bullseye-security InRelease [44.1 kB]
# ...
Processing triggers for man-db (2.9.4-2) ...
Processing triggers for libc-bin (2.31-13+deb11u3) ...
Authenticating docker via gcloud in the VM
# ...
Adding credentials for all GCR repositories.
WARNING: A long list of credential helpers may cause delays running 'docker build'. We recommend passing the registry name to configure only the registry you are using.
Docker configuration file updated.


Provisioning done!

In addition, I also created a script for the Deployment at .infrastructure/deploy.sh:

#!/usr/bin/env bash

usage="Usage: deploy.sh project_id vm_name"
[ -z "$1" ] &&  echo "No project_id given! $usage" && exit 1
[ -z "$2" ] &&  echo "No vm_name given! $usage" && exit 1

GREEN="\033[0;32m"
NO_COLOR="\033[0m"

project_id=$1
vm_name=$2
vm_zone=us-central1-a
image_name="gcr.io/${project_id}/nginx:latest"  
container_name=nginx
deployment_service_account_key_location=./gcp-service-account-key.json

printf "${GREEN}Building nginx docker image with name '${image_name}'${NO_COLOR}\n"
docker build -t "${image_name}" -f - . <<EOF
FROM nginx:1.21.5-alpine

RUN echo "Hello world" >> /usr/share/nginx/html/hello.html

EOF

printf "${GREEN}Authenticating docker${NO_COLOR}\n"
cat "${deployment_service_account_key_location}" | docker login -u _json_key --password-stdin https://gcr.io

printf "${GREEN}Pushing '${image_name}' to registry${NO_COLOR}\n"
docker push "${image_name}"

printf "${GREEN}Transferring deployment script${NO_COLOR}\n"
gcloud compute scp --zone ${vm_zone} --tunnel-through-iap --project=${project_id} ./.infrastructure/scripts/deploy.sh ${vm_name}:deploy.sh

printf "${GREEN}Executing deployment script${NO_COLOR}\n"
gcloud compute ssh ${vm_name} --zone ${vm_zone} --tunnel-through-iap --project=${project_id} --command="bash deploy.sh '${image_name}'"

printf "${GREEN}Retrieving external IP of the VM${NO_COLOR}\n"
ip_address=$(gcloud compute instances describe ${vm_name} --zone ${vm_zone} --project=${project_id} --format='get(networkInterfaces[0].accessConfigs[0].natIP)')
printf "http://${ip_address}/hello.html\n"

printf "\n\n${GREEN}Deployment done!${NO_COLOR}\n"

$ bash .infrastructure/deploy.sh pl-dofroscra-p dofroscra-test
Building nginx docker image with name 'gcr.io/pl-dofroscra-p/nginx:latest'
#...
#7 writing image sha256:65ff457111599e3f4dd439138b052cff74f10e3332c593178d8288aebb88bb1c done
#7 naming to gcr.io/pl-dofroscra-p/nginx:latest done
#7 DONE 0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Authenticating docker
Login Succeeded

Logging in with your password grants your terminal complete access to your account.
For better security, log in with a limited-privilege personal access token. Learn more at https://docs.docker.com/go/access-tokens/
Pushing 'gcr.io/pl-dofroscra-p/nginx:latest' to registry
The push refers to repository [gcr.io/pl-dofroscra-p/nginx]
ad4683501621: Preparing
# ...
1c9c1e42aafa: Pushed
latest: digest: sha256:680086b3c77e4b895099c0a5f6e713ff79ba0d78e1e1df1bc2546d6f979126e4 size: 1775
Transferring deployment script
deploy.sh                 | 0 kB |   0.6 kB/s | ETA: 00:00:00 | 100%
Executing deployment script
Pulling 'gcr.io/pl-dofroscra-p/nginx:latest' from registry
latest: Pulling from pl-dofroscra-p/nginx
Digest: sha256:f17a9092051c389abf76d254e4d564dbd7a814eb21e3cc47b667db301aa9b497
# ...
gcr.io/pl-dofroscra-p/nginx:latest
Starting container
nginx
58c0a34ca44c9bec97c991bdc69d2353ed75f4214f221e444da36e195a215c75
Getting secret GPG_KEY
-----BEGIN PGP PRIVATE KEY BLOCK-----

lQPGBGKA1psBCACq5zYDT587CVZEIWXbUplfAGQZOQJALmzErYpTp0jt+rp4vJhR
U5xahy3pqCq81Cnny5YME50ybB3pW/WcHxWLBDo+he8PKeLbp6wFFjJns+3u4opH
9gFMElyHpzTGiDQYfx/CgY2hKz7GSqpjmnOaKxYvGv0EsbZczyHY1WIN/YFzb0tI
tY7J4zTSH05I+aazRdHyn28QcCRcIT9+4q+5Vk8gz8mmgoqVpyeNgQcqJjcd03iP
WUZd1vZCumOvdG5PZNlc/wPFhqLDmYyLmJ7pt5bWIgty9BjYK8Z2NOdUaekqVEJ+
r29HbzwgFLLE2gd52f07h2y2YgMdWdz4FDxVABEBAAH+BwMC9veBYT2oigXxExLl
7fZKVjw02lEr1NpYd5X1ge9WPU/1qumATJWounzciiETpsYGsbPd9zFRJP4E3JZl
sFSh4p0/kXYTuenYD8wgGkeYyN4lm53IHfqSn2z9JMW5Kz9XEODtKJl8fjcn9Zeb
Getting secret GPG_PASSWORD
87654321
Retrieving external IP of the VM
http://34.70.213.101/hello.html


Deployment done!

Cleanup

The easiest way to "cleanup everything" that might create any costs, e.g.

• the docker images stored on Cloud Storage
• the secrets in the Secret Manager
• the VM itself

is to delete the whole project, e.g. via the Settings UI

FYI: GCP projects have 30-day-grace-period during that you can "revert" the deletion.

Wrapping up

Congratulations, you made it! If some things are not completely clear by now, don't hesitate to leave a comment. You should now be familiar enough with GCP to create a Compute Instance VM and configure it to run dockerized applications on it.

In the next part of this tutorial, we will deploy our dockerized PHP application "to production" on GCP via docker compose.

Please subscribe to the RSS feed or via email to get automatic notifications when this next part comes out :)

All code samples are publicly available in my Docker PHP Tutorial repository on Github.
You find the branch for this tutorial at part-7-ci-pipeline-docker-php-gitlab-github.

All published parts of the Docker PHP Tutorial are collected under a dedicated page at Docker PHP Tutorial. The previous part was Use git-secret to encrypt secrets in the repository and the following one is A primer on GCP Compute Instance VMs for dockerized Apps.

If you want to follow along, please subscribe to the RSS feed or via email to get automatic notifications when the next part comes out :)

Table of contents

• Introduction
  • Recommended reading
  • Approach
  • Try it yourself
• CI setup
  • General CI notes
    • Initialize make for CI
    • wait-for-service.sh
  • Setup for a "local" CI run
    • Run details
    • Execution example
  • Setup for Github Actions
    • The Workflow file
  • Setup for Gitlab Pipelines
    • The .gitlab-ci.yml pipeline file
  • Performance
    • The caching problem on CI
• Docker changes
  • Compose file updates
    • docker-compose.local.yml
    • docker-compose.ci.yml
    • Adding a health check for mysql
  • Build target: ci
    • Build stage ci in the php-base image
      • Use the whole codebase as build context
      • Build the dependencies
      • Create the final image
    • Build stage ci in the application image
  • .dockerignore
• Makefile changes
  • Initialize the shared variables
  • ENV based docker compose config
• Codebase changes
  • Add a test for encrypted files
  • Add a password-protected secret gpg key
  • Create a JUnit report from PhpUnit
• Wrapping up

Introduction

CI is short for Continuous Integration and to me mostly means running the code quality tools and tests of a codebase in an isolated environment (preferably automatically). This is
particularly important when working in a team, because the CI system acts as the final gatekeeper before features or bugfixes are merged into the main branch.

I initially learned about CI systems when I stubbed my toes into the open source water. Back in the day I used Travis CI for my own projects and replaced it with Github Actions at some point. At ABOUT YOU we started out with a self-hosted Jenkins server and then moved on to Gitlab CI as a fully managed solution (though we use custom runners).

Recommended reading

This tutorial builds on top of the previous parts. I'll do my best to cross-reference the corresponding articles when necessary, but I would still recommend to do some upfront reading on:

• the general folder structure, the update of the .docker/ directory and the introduction of a .make/ directory
• the general usage of make and it's evolution as well as the connection to docker compose commands
• the concepts of the docker containers and the docker compose setup

And as a nice-to-know: - the setup of PhpUnit for the test make target as well as the qa make target - the usage of git-secret to handle secret values

Approach

In this tutorial I'm going to explain how to make our existing docker setup work with Github Actions and Gitlab CI/CD Pipelines. As I'm a big fan of a "progressive enhancement" approach, we will ensure that all necessary steps can be performed locally through make. This has the additional benefit of keeping a single source of truth (the Makefile) which will come in handy when we set up the CI system on two different providers (Github and Gitlab).

The general process will look very similar to the one for local development:

• build the docker setup
• start the docker setup
• run the qa tools
• run the tests

You can see the final results in the CI setup section, including the concrete yml files and links to the repositories, see

• Setup for a "local" CI run
• Setup for Github Actions
• Setup for Gitlab Pipelines

On a code level, we will treat CI as an environment, configured through the env variable ENV. So far we only used ENV=local and we will extend that to also use ENV=ci. The necessary changes are explained after the concrete CI setup instructions in the sections

• Docker changes
• Makefile changes
• Codebase changes

Try it yourself

To get a feeling for what's going on, you can start by executing the local CI run:

• checkout branch part-7-ci-pipeline-docker-php-gitlab-github
• initialize make
• run the .local-ci.sh script

This should give you a similar output as presented in the Execution example.

git checkout part-7-ci-pipeline-docker-php-gitlab-github

# Initialize make
make make-init

# Execute the local CI run
bash .local-ci.sh

CI setup

General CI notes

Initialize make for CI

As a very first step we need to "configure" the codebase to operate for the ci environment. This is done through the make-init target as explained later in more detail in the Makefile changes section via

make make-init ENVS="ENV=ci TAG=latest EXECUTE_IN_CONTAINER=true GPG_PASSWORD=12345678"

$ make make-init ENVS="ENV=ci TAG=latest EXECUTE_IN_CONTAINER=true GPG_PASSWORD=12345678"
Created a local .make/.env file

ENV=ci ensures that we

• use the correct docker compose config files
• use the ci build target

TAG=latest is just a simplification for now because we don't do anything with the images yet. In an upcoming tutorial we will push them to a container registry for later usage in production deployments and then set the TAG to something more meaningful (like the build number).

EXECUTE_IN_CONTAINER=true forces every make command that uses a RUN_IN_*_CONTAINER setup to run in a container. This is important, because the Gitlab runner will actually run in a docker container itself. However, this would cause any affected target to omit the $(DOCKER_COMPOSER) exec prefix.

GPG_PASSWORD=12345678 is the password for the secret gpg key as mentioned in Add a password-protected secret gpg key.

wait-for-service.sh

I'll explain the "container is up and running but the underlying service is not" problem for the mysql service and how we can solve it with a health check later in this article at Adding a health check for mysql. On purpose, we don't want docker compose to take care of the waiting because we can make "better use of the waiting time" and will instead implement it ourselves with a simple bash script located at .docker/scripts/wait-for-service.sh:

#!/bin/bash

name=$1
max=$2
interval=$3

[ -z "$1" ] && echo "Usage example: bash wait-for-service.sh mysql 5 1"
[ -z "$2" ] && max=30
[ -z "$3" ] && interval=1

echo "Waiting for service '$name' to become healthy, checking every $interval second(s) for max. $max times"

while true; do 
  ((i++))
  echo "[$i/$max] ..."; 
  status=$(docker inspect --format "{{json .State.Health.Status }}" "$(docker ps --filter name="$name" -q)")
  if echo "$status" | grep -q '"healthy"'; then 
   echo "SUCCESS";
   break
  fi
  if [ $i == $max ]; then 
    echo "FAIL"; 
    exit 1
  fi 
  sleep $interval; 
done

This script waits for a docker $service to become "healthy" by checking the .State.Health.Status info of the docker inspect command.

CAUTION: The script uses $(docker ps --filter name="$name" -q) to determine the id of the container, i.e. it will "match" all running containers against the $name - this would fail if there is more than one matching container! I.e. you must ensure that $name is specific enough to identify one single container uniquely.

The script will check up to $max times in a interval of $interval seconds. See these answers on the "How do I write a retry logic in script to keep retrying to run it up to 5 times?" question for the implementation of the retry logic. To check the health of the mysql service for 5 times with 1 seconds between each try, it can be called via

bash wait-for-service.sh mysql 5 1

Output

$ bash wait-for-service.sh mysql 5 1
Waiting for service 'mysql' to become healthy, checking every 1 second(s) for max. 5 times
[1/5] ...
[2/5] ...
[3/5] ...
[4/5] ...
[5/5] ...
FAIL

# OR

$ bash wait-for-service.sh mysql 5 1
Waiting for service 'mysql' to become healthy, checking every 1 second(s) for max. 5 times
[1/5] ...
[2/5] ...
SUCCESS

The problem of "container dependencies" isn't new and there are already some existing solutions out there, e.g.

• wait-for
• wait-for-it
• dockerize
• docker-compose-wait

But unfortunately all of them operate by checking the availability of a host:port combination and in the case of mysql that didn't help, because the container was up, the port was reachable but the mysql service in the container was not.

Setup for a "local" CI run

As mentioned under Approach, we want to be able to perform all necessary steps locally and I created a corresponding script at .local-ci.sh:

#!/bin/bash
# fail on any error 
# @see https://stackoverflow.com/a/3474556/413531
set -e

make docker-down ENV=ci || true

start_total=$(date +%s)

# STORE GPG KEY
cp secret-protected.gpg.example secret.gpg

# DEBUG
docker version
docker compose version
cat /etc/*-release || true

# SETUP DOCKER
make make-init ENVS="ENV=ci TAG=latest EXECUTE_IN_CONTAINER=true GPG_PASSWORD=12345678"
start_docker_build=$(date +%s)
make docker-build
end_docker_build=$(date +%s)
mkdir -p .build && chmod 777 .build

# START DOCKER
start_docker_up=$(date +%s)
make docker-up
end_docker_up=$(date +%s)
make gpg-init
make secret-decrypt-with-password

# QA
start_qa=$(date +%s)
make qa || FAILED=true
end_qa=$(date +%s)

# WAIT FOR CONTAINERS
start_wait_for_containers=$(date +%s)
bash .docker/scripts/wait-for-service.sh mysql 30 1
end_wait_for_containers=$(date +%s)

# TEST
start_test=$(date +%s)
make test || FAILED=true
end_test=$(date +%s)

end_total=$(date +%s)

# RUNTIMES
echo "Build docker:        " `expr $end_docker_build - $start_docker_build`
echo "Start docker:        " `expr $end_docker_up - $start_docker_up  `
echo "QA:                  " `expr $end_qa - $start_qa`
echo "Wait for containers: " `expr $end_wait_for_containers - $start_wait_for_containers`
echo "Tests:               " `expr $end_test - $start_test`
echo "---------------------"
echo "Total:               " `expr $end_total - $start_total`

# CLEANUP
# reset the default make variables
make make-init
make docker-down ENV=ci || true

# EVALUATE RESULTS
if [ "$FAILED" == "true" ]; then echo "FAILED"; exit 1; fi

echo "SUCCESS"

Run details

• as a preparation step, we first ensure that no outdated ci containers are running (this is only necessary locally, because runners on a remote CI system will start "from scratch")

  make docker-down ENV=ci || true
  

• we take some time measurements to understand how long certain parts take via

  start_total=$(date +%s)
  

  to store the current timestamp

• we need the secret gpg key in order to decrypt the secrets and simply copy the password-protected example key (in the actual CI systems the key will be configured as a secret value that is injected in the run)

  # STORE GPG KEY
  cp secret-protected.gpg.example secret.gpg
  

• I like printing some debugging info in order to understand which exact circumstances we're dealing with (tbh, this is mostly relevant when setting the CI system up or making modifications to it)

  # DEBUG
  docker version
  docker compose version
  cat /etc/*-release || true
  

• for the docker setup, we start with initializing the environment for ci

  # SETUP DOCKER
  make make-init ENVS="ENV=ci TAG=latest EXECUTE_IN_CONTAINER=true GPG_PASSWORD=12345678"
  

  then build the docker setup

  make docker-build
  

  and finally add a .build/ directory to collect the build artifacts

  mkdir -p .build && chmod 777 .build
  

• then, the docker setup is started

  # START DOCKER
  make docker-up
  

  and gpg is initialized so that the secrets can be decrypted

  make gpg-init
  make secret-decrypt-with-password
  

  We don't need to pass a GPG_PASSWORD to secret-decrypt-with-password because we have set it up in the previous step as a default value via make-init

• once the application container is running, the qa tools are run by invoking the qa make target

  # QA
  make qa || FAILED=true
  

  The || FAILED=true part makes sure that the script will not be terminated if the checks fail. Instead, the fact that a failure happened is "recorded" in the FAILED variable so that we can evaluate it at the end. We don't want the script to stop here because we want the following steps to be executed as well (e.g. the tests).

• to mitigate the "mysql is not ready" problem, we will now apply the wait-for-service.sh script

  # WAIT FOR CONTAINERS
  bash .docker/scripts/wait-for-service.sh mysql 30 1
  

• once mysql is ready, we can execute the tests via the test make target and apply the same || FAILED=true workaround as for the qa tools

  # TEST
  make test || FAILED=true
  

• finally, all the timers are printed

  # RUNTIMES
  echo "Build docker:        " `expr $end_docker_build - $start_docker_build`
  echo "Start docker:        " `expr $end_docker_up - $start_docker_up  `
  echo "QA:                  " `expr $end_qa - $start_qa`
  echo "Wait for containers: " `expr $end_wait_for_containers - $start_wait_for_containers`
  echo "Tests:               " `expr $end_test - $start_test`
  echo "---------------------"
  echo "Total:               " `expr $end_total - $start_total`
  

• we clean up the resources (this is only necessary when running locally, because the runner of a CI system would be shut down anyway)

  # CLEANUP
  make make-init
  make docker-down ENV=ci || true
  

• and finally evaluate if any error occurred when running the qa tools or the tests

  # EVALUATE RESULTS
  if [ "$FAILED" == "true" ]; then echo "FAILED"; exit 1; fi
  
  echo "SUCCESS"
  

Execution example

Executing the script via

bash .local-ci.sh

yields the following (shortened) output:

$ bash .local-ci.sh
Container dofroscra_ci-redis-1  Stopping
# Stopping all other `ci` containers ...
# ...

Client:
 Cloud integration: v1.0.22
 Version:           20.10.13
# Print more debugging info ...
# ...

Created a local .make/.env file
ENV=ci TAG=latest DOCKER_REGISTRY=docker.io DOCKER_NAMESPACE=dofroscra APP_USER_NAME=application APP_GROUP_NAME=application docker compose -p dofroscra_ci --env-file ./.docker/.env -f ./.docker/docker-compose/docker-compose-php-base.yml build php-base
#1 [internal] load build definition from Dockerfile
# Output from building the docker containers 
# ...

ENV=ci TAG=latest DOCKER_REGISTRY=docker.io DOCKER_NAMESPACE=dofroscra APP_USER_NAME=application APP_GROUP_NAME=application docker compose -p dofroscra_ci --env-file ./.docker/.env -f ./.docker/docker-compose/docker-compose.local.ci.yml -f ./.docker/docker-compose/docker-compose.ci.yml up -d
Network dofroscra_ci_network  Creating
# Starting all `ci` containers ...
# ...

"C:/Program Files/Git/mingw64/bin/make" -s gpg-import GPG_KEY_FILES="secret.gpg"
gpg: directory '/home/application/.gnupg' created
gpg: keybox '/home/application/.gnupg/pubring.kbx' created
gpg: /home/application/.gnupg/trustdb.gpg: trustdb created
gpg: key D7A860BBB91B60C7: public key "Alice Doe protected <alice.protected@example.com>" imported
# Output of importing the secret and public gpg keys
# ...

"C:/Program Files/Git/mingw64/bin/make" -s git-secret ARGS="reveal -f -p 12345678"
git-secret: done. 1 of 1 files are revealed.
"C:/Program Files/Git/mingw64/bin/make" -j 8 -k --no-print-directory --output-sync=target qa-exec NO_PROGRESS=true
phplint                             done   took 4s
phpcs                               done   took 4s
phpstan                             done   took 8s
composer-require-checker            done   took 8s
Waiting for service 'mysql' to become healthy, checking every 1 second(s) for max. 30 times
[1/30] ...
SUCCESS
PHPUnit 9.5.19 #StandWithUkraine

........                                                            8 / 8 (100%)

Time: 00:03.077, Memory: 28.00 MB

OK (8 tests, 15 assertions)
Build docker:         12
Start docker:         2
QA:                   9
Wait for containers:  3
Tests:                5
---------------------
Total:                46
Created a local .make/.env file

Container dofroscra_ci-application-1  Stopping
Container dofroscra_ci-mysql-1  Stopping
# Stopping all other `ci` containers ...
# ...

SUCCESS

Setup for Github Actions

• Repository (branch part-7-ci-pipeline-docker-php-gitlab-github)
• CI/CD overview (Actions)
• Example of a successful job
• Example of a failed job

If you are completely new to Github Actions, I recommend to start with the official Quickstart Guide for GitHub Actions and the Understanding GitHub Actions article. In short:

• Github Actions are based on so called Workflows
  • Workflows are yaml files that live in the special .github/workflows directory in the repository
• a Workflow can contain multiple Jobs
• each Job consists of a series of Steps

• each Step needs a run: element that represents a command that is executed by a new shell

  • multi-line commands that should use the same shell are written as

    - run : |
          echo "line 1"
          echo "line 2"
  

  See also difference between "run |" and multiple runs in github actions

The Workflow file

Github Actions are triggered automatically based on the files in the .github/workflows directory. I have added the file .github/workflows/ci.yml with the following content:

name: CI build and test

on:
  # automatically run for pull request and for pushes to branch "part-7-ci-pipeline-docker-php-gitlab-github"
  # @see https://stackoverflow.com/a/58142412/413531
  push:
    branches:
      - part-7-ci-pipeline-docker-php-gitlab-github
  pull_request: {}
  # enable to trigger the action manually
  # @see https://github.blog/changelog/2020-07-06-github-actions-manual-triggers-with-workflow_dispatch/
  # CAUTION: there is a known bug that makes the "button to trigger the run" not show up
  # @see https://github.community/t/workflow-dispatch-workflow-not-showing-in-actions-tab/130088/29
  workflow_dispatch: {}

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v1

      - name: start timer
        run: |
          echo "START_TOTAL=$(date +%s)" > $GITHUB_ENV

      - name: STORE GPG KEY
        run: |
          # Note: make sure to wrap the secret in double quotes (")
          echo "${{ secrets.GPG_KEY }}" > ./secret.gpg

      - name: SETUP TOOLS
        run : |
          DOCKER_CONFIG=${DOCKER_CONFIG:-$HOME/.docker}
          # install docker compose
          # @see https://docs.docker.com/compose/cli-command/#install-on-linux
          # @see https://github.com/docker/compose/issues/8630#issuecomment-1073166114
          mkdir -p $DOCKER_CONFIG/cli-plugins 
          curl -sSL https://github.com/docker/compose/releases/download/v2.2.3/docker-compose-linux-$(uname -m) -o $DOCKER_CONFIG/cli-plugins/docker-compose
          chmod +x $DOCKER_CONFIG/cli-plugins/docker-compose

      - name: DEBUG
        run: |
          docker compose version
          docker --version
          cat /etc/*-release

      - name: SETUP DOCKER
        run: |
          make make-init ENVS="ENV=ci TAG=latest EXECUTE_IN_CONTAINER=true GPG_PASSWORD=${{ secrets.GPG_PASSWORD }}"
          make docker-build
          mkdir .build && chmod 777 .build

      - name: START DOCKER
        run: |
          make docker-up
          make gpg-init
          make secret-decrypt-with-password

      - name: QA
        run: |
          # Run the tests and qa tools but only store the error instead of failing immediately
          # @see https://stackoverflow.com/a/59200738/413531
          make qa || echo "FAILED=qa" >> $GITHUB_ENV

      - name: WAIT FOR CONTAINERS
        run: |
          # We need to wait until mysql is available.
          bash .docker/scripts/wait-for-service.sh mysql 30 1 

      - name: TEST
        run: |
          make test || echo "FAILED=test $FAILED" >> $GITHUB_ENV

      - name: RUNTIMES
        run: |
          echo `expr $(date +%s) - $START_TOTAL`

      - name: EVALUATE
        run: |
          # Check if $FAILED is NOT empty
          if [ ! -z "$FAILED" ]; then echo "Failed at $FAILED" && exit 1; fi

      - name: upload build artifacts
        uses: actions/upload-artifact@v3
        with:
          name: build-artifacts
          path: ./.build

The steps are essentially the same as explained before at Run details for the local run. Some additional notes:

• I want the Action to be triggered automatically only when I push to branch part-7-ci-pipeline-docker-php-gitlab-github OR when a pull request is created (via pull_request). In addition, I want to be able to trigger the Action manually on any branch (via workflow_dispatch).

  on:
  push:
    branches:
      - part-7-ci-pipeline-docker-php-gitlab-github
  pull_request: {}
  workflow_dispatch: {}
  

  For a real project, I would let the action only run automatically on long-living branches like main or develop. The manual trigger is helpful if you just want to test your current work without putting it up for review. CAUTION: There is a known issue that "hides" the "Trigger workflow" button to trigger the action manually.

• a new shell is started for each run: instruction, thus we must store our timer in the "global" environment variable $GITHUB_ENV

    - name: start timer
    run: |
      echo "START_TOTAL=$(date +%s)" > $GITHUB_ENV 
  

  This will be the only timer we use, because the job uses multiple steps that are timed automatically - so we don't need to take timestamps manually:

• the gpg key is configured as an encrypted secret named GPG_KEY and is stored in ./secret.gpg. The value is the content of the secret-protected.gpg.example file

    - name: STORE GPG KEY
      run: |
        echo "${{ secrets.GPG_KEY }}" > ./secret.gpg
  

  Secrets are configured in the Github repository under Settings > Secrets > Actions at

  https://github.com/$user/$repository/settings/secrets/actions
  
  e.g.
  
  https://github.com/paslandau/docker-php-tutorial/settings/secrets/actions
  

  

• the ubuntu-latest image doesn't contain the docker compose plugin, thus we need to install it manually

    - name: SETUP TOOLS
    run : |
      DOCKER_CONFIG=${DOCKER_CONFIG:-$HOME/.docker}
      mkdir -p $DOCKER_CONFIG/cli-plugins 
      curl -sSL https://github.com/docker/compose/releases/download/v2.2.3/docker-compose-linux-$(uname -m) -o $DOCKER_CONFIG/cli-plugins/docker-compose
      chmod +x $DOCKER_CONFIG/cli-plugins/docker-compose
  

• for the make initialization we need the second secret named GPG_PASSWORD - which is configured as 12345678 in our case, see Add a password-protected secret gpg key

    - name: SETUP DOCKER
      run: |
        make make-init ENVS="ENV=ci TAG=latest EXECUTE_IN_CONTAINER=true GPG_PASSWORD=${{ secrets.GPG_PASSWORD }}"
  

• because the runner will be shutdown after the run, we need to move the build artifacts to a permanent location, using the actions/upload-artifact@v3 action

    - name: upload build artifacts
      uses: actions/upload-artifact@v3
      with:
        name: build-artifacts
        path: ./.build
  

  You can download the artifacts in the Run overview UI

Setup for Gitlab Pipelines

• Repository (branch part-7-ci-pipeline-docker-php-gitlab-github)
• CI/CD overview (Pipelines)
• Example of a successful job
• Example of a failed job

If you are completely new to Gitlab Pipelines, I recommend to start with the official Get started with GitLab CI/CD guide. In short:

• the core concept of Gitlab Pipelines is the Pipeline
  • it is defined in the yaml file .gitlab-ci.yml that lives in the root of the repository
• a Pipeline can contain multiple Stages
• each Stage consists of a series of Jobs
• each Job contains a script section
• the script section consists of a series of shell commands

The .gitlab-ci.yml pipeline file

Gitlab Pipelines are triggered automatically based on a .gitlab-ci.yml file located at the root of the repository. It has the following content:

stages:
  - build_test

QA and Tests:
  stage: build_test

  rules:
    # automatically run for pull request and for pushes to branch "part-7-ci-pipeline-docker-php-gitlab-github"
    - if: '($CI_PIPELINE_SOURCE == "merge_request_event" || $CI_COMMIT_BRANCH == "part-7-ci-pipeline-docker-php-gitlab-github")'

  # see https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#use-docker-in-docker
  image: docker:20.10.12

  services:
    - name: docker:dind

  script:
    - start_total=$(date +%s)

    ## STORE GPG KEY
    - cp $GPG_KEY_FILE ./secret.gpg

    ## SETUP TOOLS
    - start_install_tools=$(date +%s)
    # "curl" is required to download docker compose
    - apk add --no-cache make bash curl
    # install docker compose
    # @see https://docs.docker.com/compose/cli-command/#install-on-linux
    - mkdir -p ~/.docker/cli-plugins/
    - curl -sSL https://github.com/docker/compose/releases/download/v2.2.3/docker-compose-linux-x86_64 -o ~/.docker/cli-plugins/docker-compose
    - chmod +x ~/.docker/cli-plugins/docker-compose
    - end_install_tools=$(date +%s)

    ## DEBUG
    - docker version
    - docker compose version
    # show linux distro info
    - cat /etc/*-release

    ## SETUP DOCKER
    # Pass default values to the make-init command - otherwise we would have to pass those as arguments to every make call
    - make make-init ENVS="ENV=ci TAG=latest EXECUTE_IN_CONTAINER=true GPG_PASSWORD=$GPG_PASSWORD"
    - start_docker_build=$(date +%s)
    - make docker-build
    - end_docker_build=$(date +%s)
    - mkdir .build && chmod 777 .build

    ## START DOCKER
    - start_docker_up=$(date +%s)
    - make docker-up
    - end_docker_up=$(date +%s)
    - make gpg-init
    - make secret-decrypt-with-password

    ## QA
    # Run the tests and qa tools but only store the error instead of failing immediately
    # @see https://stackoverflow.com/a/59200738/413531
    - start_qa=$(date +%s)
    - make qa ENV=ci || FAILED=true
    - end_qa=$(date +%s)

    ## WAIT FOR CONTAINERS
    # We need to wait until mysql is available.
    - start_wait_for_containers=$(date +%s)
    - bash .docker/scripts/wait-for-service.sh mysql 30 1
    - end_wait_for_containers=$(date +%s)

    ## TEST
    - start_test=$(date +%s)
    - make test ENV=ci || FAILED=true
    - end_test=$(date +%s)

    - end_total=$(date +%s)

    # RUNTIMES
    - echo "Tools:" `expr $end_install_tools - $start_install_tools`
    - echo "Build docker:" `expr $end_docker_build - $start_docker_build`
    - echo "Start docker:" `expr $end_docker_up - $start_docker_up  `
    - echo "QA:" `expr $end_qa - $start_qa`
    - echo "Wait for containers:" `expr $end_wait_for_containers - $start_wait_for_containers`
    - echo "Tests:" `expr $end_test - $start_test`
    - echo "Total:" `expr $end_total - $start_total`

    # EVALUATE RESULTS
    # Use if-else constructs in Gitlab pipelines
    # @see https://stackoverflow.com/a/55464100/413531
    - if [ "$FAILED" == "true" ]; then exit 1; fi

  # Save the build artifact, e.g. the JUNIT report.xml file, so we can download it later
  # @see https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html
  artifacts:
    when: always
    paths:
      # the quotes are required
      # @see https://stackoverflow.com/questions/38009869/how-to-specify-wildcard-artifacts-subdirectories-in-gitlab-ci-yml#comment101411265_38055730
      - ".build/*"
    expire_in: 1 week

The steps are essentially the same as explained before under Run details for the local run. Some additional notes:

• we start by defining the stages of the pipeline - though that's currently just one (build_test)

  stages:
  - build_test
  

• then we define the job QA and Tests and assign it to the build_test stage

  QA and Tests:
  stage: build_test
  

• I want the Pipeline to be triggered automatically only when I push to branch part-7-ci-pipeline-docker-php-gitlab-github OR when a pull request is created Triggering the Pipeline manually on any branch is possible by default.

  rules:
  - if: '($CI_PIPELINE_SOURCE == "merge_request_event" || $CI_COMMIT_BRANCH == "part-7-ci-pipeline-docker-php-gitlab-github")'
  

• since we want to build and run docker images, we need to use a docker base image and activate the docker:dind service. See Use Docker to build Docker images: Use Docker-in-Docker

  image: docker:20.10.12
  
  services:
  - name: docker:dind
  

• we store the secret gpg key as a secret file (using the "file" type) in the CI/CD variables configuration of the Gitlab repository and move it to ./secret.gpg in order to decrypt the secrets later

  ## STORE GPG KEY
  - cp $GPG_KEY_FILE ./secret.gpg
  

  Secrets can be configured under Settings > CI/CD > Variables at

  https://gitlab.com/$project/$repository/-/settings/ci_cd
  
  e.g.
  
  https://gitlab.com/docker-php-tutorial/docker-php-tutorial/-/settings/ci_cd
  

  

• the docker base image doesn't come with all required tools, thus we need to install the missing ones (make, bash, curl and docker compose)

    ## SETUP TOOLS
    - apk add --no-cache make bash curl
    - mkdir -p ~/.docker/cli-plugins/
    - curl -sSL https://github.com/docker/compose/releases/download/v2.2.3/docker-compose-linux-x86_64 -o ~/.docker/cli-plugins/docker-compose
    - chmod +x ~/.docker/cli-plugins/docker-compose
  

• for the initialization of make we use the $GPG_PASSWORD variable that we defined in the CI/CD settings

  ## SETUP DOCKER
  - make make-init ENVS="ENV=ci TAG=latest EXECUTE_IN_CONTAINER=true GPG_PASSWORD=$GPG_PASSWORD"
  

  Note: I have marked the variable as "masked" so it won't show up in any logs

• finally, we store the job artifacts

  artifacts:
  when: always
  paths:
    - ".build/*"
  expire_in: 1 week 
  

  They can be accessed in the Pipeline overview UI

Performance

Performance isn't an issue right now, because the CI runs take only about ~1 min (Github Actions) and ~2 min (Gitlab Pipelines), but that's mostly because we only ship a super minimal application and those times will go up when things get more complex. For the local setup I used all 8 cores of my laptop. The time breakdown is roughly as follows:

Times taken from - "CI build and test #83" Github Action run - "Pipeline #511355192" Gitlab Pipeline run - "local without cache" via bash .local-ci.sh with no local images at all - "local with cached images" via bash .local-ci.sh with cached images for mysql and redis - "local with cached images + layers" via bash .local-ci.sh with cached images for mysql and redis and a "warm" layer cache for the application image

Optimizing the performance is out of scope for this tutorial, but I'll at least document my current findings.

The caching problem on CI

A good chunk of time is usually spent on building the docker images. We did our best to optimize the process by leveraging the layer cache and using cache mounts (see section Build stage ci in the php-base image). But those steps are futile on CI systems, because the corresponding runners will start "from scratch" for every CI run - i.e. there is no local cache that they could use. In consequence, the full docker setup is also built "from scratch" on every run.

There are ways to mitigate that e.g.

• pushing images to a container registry and pulling them before building the images to leverage the layer cache via the cache_from option of docker compose
• exporting and importing the images as tar archives via docker save and
  docker load, storing them either in the built-in cache of Github or Gitlab
  • see also the satackey/action-docker-layer-caching@v0.0.11 Github Action and the official actions/cache@v3 Github Action
• using the --cache-from and --cache-to options of buildx
  • see also the "cache" docu of the docker/build-push-action@v2 Github Action

But: None of that worked for me out-of-the-box :( We will take a closer look in an upcoming tutorial. Some reading material that I found valuable so far:

• Caching Docker builds in GitHub Actions: Which approach is the fastest? 🤔 A research.
• Caching strategies for CI systems
• Build images on GitHub Actions with Docker layer caching
• Faster CI Builds with Docker Layer Caching and BuildKit
• Image rebase and improved remote cache support in new BuildKit

Docker changes

As a first step we need to decide which containers are required and how to provide the codebase.

Since our goal is running the qa tools and tests, we only need the application php container. The tests also need a database and a queue, i.e. the mysql and redis containers are required as well - whereas nginx, php-fpm and php-worker are not required. We'll handle that through dedicated docker compose configuration files that only contain the necessary services. This is explained in more detail in section Compose file updates.

In our local setup, we have sheen the host system and docker - mainly because we wanted our changes to be reflected immediately in docker.

This isn't necessary for the CI use case. In fact we want our CI images as close as possible to our production images - and those should "contain everything to run independently". I.e. the codebase should live in the image - not on the host system. This will be explained in section Use the whole codebase as build context.

Compose file updates

We will not only have some differences between the CI docker setup and the local docker setup (=different containers), but also in the configuration of the individual services. To accommodate for that, we will use the following docker compose config files in the .docker/docker-compose/ directory:

• docker-compose.local.ci.yml:
  • holds configuration that is valid for local and ci, trying to keep the config files DRY
• docker-compose.ci.yml:
  • holds configuration that is only valid for ci
• docker-compose.local.yml:
  • holds configuration that is only valid for local

When using docker compose we then need to make sure to include only the required files, e.g. for ci:

docker compose -f docker-compose.local.ci.yml -f docker-compose.ci.yml

I'll explain the logic for that later in section ENV based docker compose config. In short:

docker-compose.local.yml

When comparing ci with local, for ci

• we don't need to share the codebase with the host system

  application:
    volumes:
    - ${APP_CODE_PATH_HOST?}:${APP_CODE_PATH_CONTAINER?}
  

• we don't need persistent volumes for the redis and mysql data

  mysql:
    volumes:
      - mysql:/var/lib/mysql
  
  redis:
    volumes:
      - redis:/data
  

• we don't need to share ports with the host system

  application:
    ports:
      - "${APPLICATION_SSH_HOST_PORT:-2222}:22"
  
  redis:
    ports:
      - "${REDIS_HOST_PORT:-6379}:6379"
  

• we don't need any settings for local dev tools like xdebug or strace

  application:
    environment:
      - PHP_IDE_CONFIG=${PHP_IDE_CONFIG?}
    cap_add:
      - "SYS_PTRACE"
    security_opt:
      - "seccomp=unconfined"
    extra_hosts:
      - host.docker.internal:host-gateway  
  

So all of those config values will only live in the docker-compose.local.yml file.

docker-compose.ci.yml

In fact, there are only two things that ci needs that local doesn't:

• a bind mount to share only the secret gpg key from the host with the application container

  application:
    volumes:
      - ${APP_CODE_PATH_HOST?}/secret.gpg:${APP_CODE_PATH_CONTAINER?}/secret.gpg:ro
  

  This is required to decrypt the secrets:

  [...] the private key has to be named secret.gpg and put in the root of the codebase, so that the import can be simplified with make targets

  The secret files themselves are baked into the image, but the key to decrypt them will be provided only during runtime and

• a bind mount to share a .build folder for build artifacts with the application container

  application:
    volumes:
      - ${APP_CODE_PATH_HOST?}/.build:${APP_CODE_PATH_CONTAINER?}/.build
  

  This will be used to collect any files we want to retain from a build (e.g. code coverage information, log files, etc.)

Adding a health check for mysql

When running the tests for the first time on a CI system, I noticed some weird errors related to the database:

1) Tests\Feature\App\Http\Controllers\HomeControllerTest::test___invoke with data set "default" (array(), '    <li><a href="?dispatch=fo...></li>')
PDOException: SQLSTATE[HY000] [2002] Connection refused

As it turned out, the mysql container itself was up and running - but the mysql process within the container was not yet ready to accept connections. Locally, this hasn't been a problem, because we usually would not run the tests "immediately" after starting the containers - but on CI this is the case.

Fortunately, docker compose has us covered here and provides a healtcheck configuration option:

healthcheck declares a check that’s run to determine whether or not containers for this service are "healthy".

Since this healthcheck is also "valid" for local, I defined it in the combined docker-compose.local.ci.yml file:

  mysql:
    healthcheck:
      # Only mark the service as healthy if mysql is ready to accept connections
      # Check every 2 seconds for 30 times, each check has a timeout of 1s
      test: mysqladmin ping -h 127.0.0.1 -u $$MYSQL_USER --password=$$MYSQL_PASSWORD
      timeout: 1s
      retries: 30
      interval: 2s

The script in test was taken from SO: Docker-compose check if mysql connection is ready.

When starting the docker setup, docker ps will now add a health info to the STATUS:

$ make docker-up

$ docker ps
CONTAINER ID   IMAGE                            STATUS                           NAMES
b509eb2f99c0   dofroscra/application-ci:latest  Up 1 seconds                     dofroscra_ci-application-1
503e52fd9e68   mysql:8.0.28                     Up 1 seconds (health: starting)  dofroscra_ci-mysql-1

# a couple of seconds later

$ docker ps
CONTAINER ID   IMAGE                            STATUS                   NAMES
b509eb2f99c0   dofroscra/application-ci:latest  Up 13 seconds            dofroscra_ci-application-1
503e52fd9e68   mysql:8.0.28                     Up 13 seconds (healthy)  dofroscra_ci-mysql-1

Note the (health: starting) and (healthy) infos for the mysql service.

We can also get this info from docker inspect (used by our wait-for-service.sh script) via:

$ docker inspect --format "{{json .State.Health.Status }}" dofroscra_ci-mysql-1
"healthy"

FYI: We could also use the depends_on property with a condition: service_healthy on the application container so that docker compose would only start the container once the mysql service is healthy:

application:
  depends_on:
    mysql: 
      condition: service_healthy

However, this would "block" the make docker-up until mysql is actually up and running. In our case this is not desirable, because we can do "other stuff" in the meantime (namely: run the qa checks, because they don't require a database) and thus save a couple of seconds on each CI run.

Build target: ci

We've already introduced build targets in Environments and build targets and how to "choose" them through make with the ENV variable defined in a shared .make/.env file. Short recap:

• create a .make/.env file via make make-init that contains the ENV, e.g. makefile ENV=ci
• the .make/.env file is included in the main Makefile, making the ENV variables available to make
• configure a $DOCKER_COMPOSE variable that passes the ENV as an environment variable, i.e. via bash ENV=$(ENV) docker-compose

• use the ENV variable in the docker compose configuration file to determine the build.target property. E.g. in .docker/docker-compose/docker-compose-php-base.yml

  php-base:
    build:
      target: ${ENV?}
  

• in the Dockerfile of a service, define the ENV as a build stage. E.g. in .docker/images/php/base/Dockerfile Dockerfile FROM base as ci # ...

So to enable the new ci environment, we need to modify the Dockerfiles for the php-base and the application image.

Build stage ci in the php-base image

Use the whole codebase as build context

As mentioned in section Docker changes we want to "bake" the codebase into the ci image of the php-base container.

Thus, we must change the context property in .docker/docker-compose/docker-compose-php-base.yml to not only use the .docker/ directory but instead the whole codebase. I.e. "dont use ../ but ../../":

# File: .docker/docker-compose/docker-compose-php-base.yml

  php-base:
    build:
      # pass the full codebase to docker for building the image
      context: ../../

Build the dependencies

The composer dependencies must be set up in the image as well, so we introduce a new stage stage in .docker/images/php/base/Dockerfile. The most trivial solution would look like this:

• copy the whole codebase
• run composer install

FROM base as ci

COPY . /codebase

RUN composer install --no-scripts --no-plugins --no-progress -o

However, this approach has some downsides:

• if any file in the codebase changes, the COPY . /codebase layer will be invalidated. I.e. docker could not use the layer cache which also means that every layer afterwards cannot use the cache as well. In consequence the composer install would run every time - even when the composer.json file doesn't change.
• composer itself uses a cache for storing dependencies locally so it doesn't have to download dependencies that haven't changed. But since we run composer install in Docker, this cache would be "thrown away" every time a build finishes. To mitigate that, we can use --mount=type=cache to define a directory that docker will re-use between builds: > Contents of the cache directories persists between builder invocations without invalidating > the instruction cache.

Keeping those points in mind, we end up with the following instructions:

# File: .docker/images/php/base/Dockerfile
# ...

FROM base as ci

# By only copying the composer files required to run composer install
# the layer will be cached and only invalidated when the composer dependencies are changed
COPY ./composer.json /dependencies/
COPY ./composer.lock /dependencies/

# use a cache mount to cache the composer dependencies
# this is essentially a cache that lives in Docker BuildKit (i.e. has nothing to do with the host system) 
RUN --mount=type=cache,target=/tmp/.composer \
    cd /dependencies && \
    # COMPOSER_HOME=/tmp/.composer sets the home directory of composer that
    # also controls where composer looks for the cache 
    # so we don't have to download dependencies again (if they are cached)
    COMPOSER_HOME=/tmp/.composer composer install --no-scripts --no-plugins --no-progress -o 

# copy the full codebase
COPY . /codebase

RUN mv /dependencies/vendor /codebase/vendor && \
    cd /codebase && \
    # remove files we don't require in the image to keep the image size small
    rm -rf .docker/ && \
    # we need a git repository for git-secret to work (can be an empty one)
    git init

FYI: The COPY . /codebase step doesn't actually copy "everything in the repository", because we have also introduced a .dockerignore file to exclude some files from being included in the build context - see section .dockerignore.

Some notes on the final RUN step:

• rm -rf .docker/ doesn't really save "that much" in the current setup - please take it more as an example to remove any files that shouldn't end up in the final image (e.g. "tests in a production image")
• the git init part is required because we need to decrypt the secrets later - and git-secret requires a git repository (which can be empty). We can't decrypt the secrets during the build, because we do not want decrypted secret files to end up in the image.

When tested locally, the difference between the trivial solution and the one that makes use of layer caching is ~35 seconds, see the results in the Performance section.

Create the final image

As a final step, we will rename the current stage to codebase and copy the "build artifact" from that stage into our final ci build stage:

FROM base as codebase

# build the composer dependencies and clean up the copied files
# ...

FROM base as ci

COPY --from=codebase --chown=$APP_USER_NAME:$APP_GROUP_NAME /codebase $APP_CODE_PATH

Why are we not just using the previous stage directly as ci?

Because using multistage-builds is a good practice to keep the final layers of an image to a minimum: Everything that "happened" in the previous codebase stage will be "forgotten", i.e. not exported as layers.

That does not only save us some layers, but also allows us to get rid of files like the .docker/ directory. We needed that directory in the build context because some files where required in other parts of the Dockerfile (e.g. the php ini files), so we can't exclude it via .dockerignore. But we can remove it in the codebase stage - so it will NOT be copied over and thus not end up in the final image. If we wouldn't have the codebase stage, the folder would be part of the layer created when COPYing all the files from the build context and removing it via rm -rf .docker/ would have no effect on the image size.

Currently, that doesn't really matter, because the building step is super simple (just a composer install) - but in a growing and more complex codebase you can easily save a couple MB.

To be concrete, the multistage build has 31 layers and the final layer containing the codebase has a size of 65.1MB.

$ docker image history -H dofroscra/application-ci
IMAGE          CREATED          CREATED BY                                      SIZE      COMMENT
d778c2ee8d5e   17 minutes ago   COPY /codebase /var/www/app # buildkit          65.1MB    buildkit.dockerfile.v0
                                                                                ^^^^^^
<missing>      17 minutes ago   WORKDIR /var/www/app                            0B        buildkit.dockerfile.v0
<missing>      17 minutes ago   COPY /usr/bin/composer /usr/local/bin/compos…   2.36MB    buildkit.dockerfile.v0
<missing>      17 minutes ago   COPY ./.docker/images/php/base/.bashrc /root…   395B      buildkit.dockerfile.v0
<missing>      17 minutes ago   COPY ./.docker/images/php/base/.bashrc /home…   395B      buildkit.dockerfile.v0
<missing>      17 minutes ago   COPY ./.docker/images/php/base/conf.d/zz-app…   196B      buildkit.dockerfile.v0
<missing>      17 minutes ago   COPY ./.docker/images/php/base/conf.d/zz-app…   378B      buildkit.dockerfile.v0
<missing>      17 minutes ago   RUN |8 APP_USER_ID=10000 APP_GROUP_ID=10001 …   1.28kB    buildkit.dockerfile.v0
<missing>      17 minutes ago   RUN |8 APP_USER_ID=10000 APP_GROUP_ID=10001 …   41MB      buildkit.dockerfile.v0
<missing>      18 minutes ago   ADD https://php.hernandev.com/key/php-alpine…   451B      buildkit.dockerfile.v0
<missing>      18 minutes ago   RUN |8 APP_USER_ID=10000 APP_GROUP_ID=10001 …   62.1MB    buildkit.dockerfile.v0
<missing>      18 minutes ago   ADD https://gitsecret.jfrog.io/artifactory/a…   450B      buildkit.dockerfile.v0
<missing>      18 minutes ago   RUN |8 APP_USER_ID=10000 APP_GROUP_ID=10001 …   4.74kB    buildkit.dockerfile.v0
<missing>      18 minutes ago   ENV ENV=ci                                      0B        buildkit.dockerfile.v0
<missing>      18 minutes ago   ENV ALPINE_VERSION=3.15                         0B        buildkit.dockerfile.v0
<missing>      18 minutes ago   ENV TARGET_PHP_VERSION=8.1                      0B        buildkit.dockerfile.v0
<missing>      18 minutes ago   ENV APP_CODE_PATH=/var/www/app                  0B        buildkit.dockerfile.v0
<missing>      18 minutes ago   ENV APP_GROUP_NAME=application                  0B        buildkit.dockerfile.v0
<missing>      18 minutes ago   ENV APP_USER_NAME=application                   0B        buildkit.dockerfile.v0
<missing>      18 minutes ago   ENV APP_GROUP_ID=10001                          0B        buildkit.dockerfile.v0
<missing>      18 minutes ago   ENV APP_USER_ID=10000                           0B        buildkit.dockerfile.v0
<missing>      18 minutes ago   ARG ENV                                         0B        buildkit.dockerfile.v0
<missing>      18 minutes ago   ARG ALPINE_VERSION                              0B        buildkit.dockerfile.v0
<missing>      18 minutes ago   ARG TARGET_PHP_VERSION                          0B        buildkit.dockerfile.v0
<missing>      18 minutes ago   ARG APP_CODE_PATH                               0B        buildkit.dockerfile.v0
<missing>      18 minutes ago   ARG APP_GROUP_NAME                              0B        buildkit.dockerfile.v0
<missing>      18 minutes ago   ARG APP_USER_NAME                               0B        buildkit.dockerfile.v0
<missing>      18 minutes ago   ARG APP_GROUP_ID                                0B        buildkit.dockerfile.v0
<missing>      18 minutes ago   ARG APP_USER_ID                                 0B        buildkit.dockerfile.v0
<missing>      2 days ago       /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B
<missing>      2 days ago       /bin/sh -c #(nop) ADD file:5d673d25da3a14ce1…   5.57MB

The non-multistage build has 32 layers and the final layer(s) containing the codebase have a combined size of 65.15MB (60.3MB + 4.85MB).

$ docker image history -H dofroscra/application-ci
IMAGE          CREATED          CREATED BY                                      SIZE      COMMENT
94ba50438c9a   2 minutes ago    RUN /bin/sh -c COMPOSER_HOME=/tmp/.composer …   60.3MB    buildkit.dockerfile.v0
<missing>      2 minutes ago    COPY . /var/www/app # buildkit                  4.85MB    buildkit.dockerfile.v0
                                                                                ^^^^^^
<missing>      31 minutes ago   WORKDIR /var/www/app                            0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   COPY /usr/bin/composer /usr/local/bin/compos…   2.36MB    buildkit.dockerfile.v0
<missing>      31 minutes ago   COPY ./.docker/images/php/base/.bashrc /root…   395B      buildkit.dockerfile.v0
<missing>      31 minutes ago   COPY ./.docker/images/php/base/.bashrc /home…   395B      buildkit.dockerfile.v0
<missing>      31 minutes ago   COPY ./.docker/images/php/base/conf.d/zz-app…   196B      buildkit.dockerfile.v0
<missing>      31 minutes ago   COPY ./.docker/images/php/base/conf.d/zz-app…   378B      buildkit.dockerfile.v0
<missing>      31 minutes ago   RUN |8 APP_USER_ID=10000 APP_GROUP_ID=10001 …   1.28kB    buildkit.dockerfile.v0
<missing>      31 minutes ago   RUN |8 APP_USER_ID=10000 APP_GROUP_ID=10001 …   41MB      buildkit.dockerfile.v0
<missing>      31 minutes ago   ADD https://php.hernandev.com/key/php-alpine…   451B      buildkit.dockerfile.v0
<missing>      31 minutes ago   RUN |8 APP_USER_ID=10000 APP_GROUP_ID=10001 …   62.1MB    buildkit.dockerfile.v0
<missing>      31 minutes ago   ADD https://gitsecret.jfrog.io/artifactory/a…   450B      buildkit.dockerfile.v0
<missing>      31 minutes ago   RUN |8 APP_USER_ID=10000 APP_GROUP_ID=10001 …   4.74kB    buildkit.dockerfile.v0
<missing>      31 minutes ago   ENV ENV=ci                                      0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   ENV ALPINE_VERSION=3.15                         0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   ENV TARGET_PHP_VERSION=8.1                      0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   ENV APP_CODE_PATH=/var/www/app                  0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   ENV APP_GROUP_NAME=application                  0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   ENV APP_USER_NAME=application                   0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   ENV APP_GROUP_ID=10001                          0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   ENV APP_USER_ID=10000                           0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   ARG ENV                                         0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   ARG ALPINE_VERSION                              0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   ARG TARGET_PHP_VERSION                          0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   ARG APP_CODE_PATH                               0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   ARG APP_GROUP_NAME                              0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   ARG APP_USER_NAME                               0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   ARG APP_GROUP_ID                                0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   ARG APP_USER_ID                                 0B        buildkit.dockerfile.v0
<missing>      2 days ago       /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B
<missing>      2 days ago       /bin/sh -c #(nop) ADD file:5d673d25da3a14ce1…   5.57MB

Again: It is expected that the differences aren't big, because the only size savings come from the .docker/ directory with a size of ~70kb.

$ du -hd 0 .docker
73K     .docker

Finally, we are also using the --chown option of the RUN instruction to ensure that the files have the correct permissions.

Build stage ci in the application image

There is actually "nothing" to be done here. We don't need SSH any longer because it is only required for the SSH Configuration of PhpStorm. So the build stage is simply "empty":

ARG BASE_IMAGE
FROM ${BASE_IMAGE} as base

FROM base as ci

FROM base as local
# ...

Though there is one thing to keep in mind: In the local image we used sshd as the entrypoint, i.e. we had a long running process that would keep the container running. To keep the ci application container running, we must

• start it via the -d flag of docker compose (already done in the make docker-up target) makefile .PHONY: docker-up docker-up: validate-docker-variables $(DOCKER_COMPOSE) up -d $(DOCKER_SERVICE_NAME)

• allocate a tty via tty: true in the docker-compose.local.ci.yml file

  application:
    tty: true
  

.dockerignore

The .dockerignore file is located in the root of the repository and ensures that certain files are kept out of the Docker build context. This will

• speed up the build (because less files need to be transmitted to the docker daemon)
• keep images smaller (because irrelevant files are kept out of the image)

The syntax is quite similar to the .gitignore file - in fact I've found it to be quite often the case that the contents of the .gitignore file are a subset of the .dockerignore file. This makes kinda sense, because you typically wouldn't want files that are excluded from the repository to end up in a docker image (e.g. unencrypted secret files). This has also been noticed by others, see e.g.

• Reddit: Any way to copy .gitignore contents to .dockerignore
• SO: Should .dockerignore typically be a superset of .gitignore?

but to my knowledge there is currently (2022-04-24) no way to "keep the two files in sync".

CAUTION: The behavior between the two files is NOT identical! The documentation says

Matching is done using Go’s filepath.Match rules. A preprocessing step removes leading and trailing whitespace and eliminates . and .. elements using Go’s filepath.Clean. Lines that are blank after preprocessing are ignored.

Beyond Go’s filepath.Match rules, Docker also supports a special wildcard string ** that matches any number of directories (including zero). For example, **/*.go will exclude all files that end with .go that are found in all directories, including the root of the build context.

Lines starting with ! (exclamation mark) can be used to make exceptions to exclusions.

Please note the part regarding **\*.go: In .gitignore it would be sufficient to write .go to match any file that contains .go, regardless of the directory. In .dockerignore you must specify it as **/*.go!

In our case, the content of the .dockerignore file looks like this:

# gitignore
!.env.example
**/*.env
.idea
.phpunit.result.cache
vendor/
secret.gpg
.gitsecret/keys/random_seed
.gitsecret/keys/pubring.kbx~
!*.secret
passwords.txt
.build

# additionally ignored files
.git

Makefile changes

Initialize the shared variables

We have introduced the concept of shared variables via .make/.env previously. It allows us to define variables in one place (=single source of truth) that are then used as "defaults" so we don't have to define them explicitly when invoking certain make targets (like make docker-build). We'll make use of this concept by setting the environment to civiaENV=ci and thus making sure that all docker commands use ci "automatically" as well.

In addition, I made a small modification by introducing a second file at .make/variables.env that is also included in the main Makefile and holds the "default" shared variables. Those are neither "secret" nor are they likely to be changed for environment adjustments. The file is NOT ignored by .gitignore and is basically just the previous .make/.env.example file without the environment specific variables:

# File .make/variables.env

DOCKER_REGISTRY=docker.io
DOCKER_NAMESPACE=dofroscra
APP_USER_NAME=application
APP_GROUP_NAME=application

The .make/.env file is still .gitignored and can be initialized with the make-init target using the ENVS variable:

make make-init ENVS="ENV=ci SOME_OTHER_DEFAULT_VARIABLE=foo"

which would create a .make/.env file with the content

ENV=ci
SOME_OTHER_DEFAULT_VARIABLE=foo

If necessary, we could also override variables defined in the .make/variables.env file, because the .make/.env is included last in the Makefile:

# File: Makefile
# ...

# include the default variables
include .make/variables.env
# include the local variables
-include .make/.env

The default value for ENVS is ENV=local TAG=latest to retain the same default behavior as before when ENVS is omitted. The corresponding make-init target is defined in the main Makefile and now looks like this:

ENVS?=ENV=local TAG=latest
.PHONY: make-init
make-init: ## Initializes the local .makefile/.env file with ENV variables for make. Use via ENVS="KEY_1=value1 KEY_2=value2"
    @$(if $(ENVS),,$(error ENVS is undefined))
    @rm -f .make/.env
    for variable in $(ENVS); do \
      echo $$variable | tee -a .make/.env > /dev/null 2>&1; \
    done
    @echo "Created a local .make/.env file" 

ENV based docker compose config

As mentioned in section Compose file updates we need to select the "correct" docker compose configuration files based on the ENV value. This is done in .make/02-00-docker.mk:

# File .make/02-00-docker.mk

# ...

DOCKER_COMPOSE_DIR:=...
DOCKER_COMPOSE_COMMAND:=...

DOCKER_COMPOSE_FILE_LOCAL_CI:=$(DOCKER_COMPOSE_DIR)/docker-compose.local.ci.yml
DOCKER_COMPOSE_FILE_CI:=$(DOCKER_COMPOSE_DIR)/docker-compose.ci.yml
DOCKER_COMPOSE_FILE_LOCAL:=$(DOCKER_COMPOSE_DIR)/docker-compose.local.yml

# we need to "assemble" the correct combination of docker-compose.yml config files
ifeq ($(ENV),ci)
    DOCKER_COMPOSE_FILES:=-f $(DOCKER_COMPOSE_FILE_LOCAL_CI) -f $(DOCKER_COMPOSE_FILE_CI)
else ifeq ($(ENV),local)
    DOCKER_COMPOSE_FILES:=-f $(DOCKER_COMPOSE_FILE_LOCAL_CI) -f $(DOCKER_COMPOSE_FILE_LOCAL)
endif

DOCKER_COMPOSE:=$(DOCKER_COMPOSE_COMMAND) $(DOCKER_COMPOSE_FILES)

When we now take a look at a full recipe when using ENV=ci with a docker target (e.g. docker-up), we can see that the correct files are chosen, e.g.

$ make docker-up ENV=ci -n
ENV=ci TAG=latest DOCKER_REGISTRY=docker.io DOCKER_NAMESPACE=dofroscra APP_USER_NAME=application APP_GROUP_NAME=application docker compose -p dofroscra_ci --env-file ./.docker/.env -f ./.docker/docker-compose/docker-compose.local.ci.yml -f ./.docker/docker-compose/docker-compose.ci.yml up -d

# =>
# -f ./.docker/docker-compose/docker-compose.local.ci.yml 
# -f ./.docker/docker-compose/docker-compose.ci.yml

Codebase changes

Add a test for encrypted files

We've introduced git-secret in the previous tutorial Use git-secret to encrypt secrets in the repository and used it to store the file passwords.txt encrypted in the codebase. To make sure that the decryption works as expected on the CI systems, I've added a test at tests/Feature/EncryptionTest.php to check if the file exists and if the content is correct.

class EncryptionTest extends TestCase
{
    public function test_ensure_that_the_secret_passwords_file_was_decrypted()
    {
        $pathToSecretFile = __DIR__."/../../passwords.txt";

        $this->assertFileExists($pathToSecretFile);

        $expected = "my_secret_password\n";
        $actual   = file_get_contents($pathToSecretFile);

        $this->assertEquals($expected, $actual);
    }
}

Of course this doesn't make sense in a "real world scenario", because the secret value would now be exposed in a test - but it suffices for now as proof of a working secret decryption.

Add a password-protected secret gpg key

I've mentioned in Scenario: Decrypt file that it is also possible to use a password-protected secret gpg key for an additional layer of security. I have created such a key and stored it in the repository at secret-protected.gpg.example (in a "real world scenario" I wouldn't do that - but since this is a public tutorial I want you to be able to follow along completely). The password for that key is 12345678.

The corresponding public key is located at .dev/gpg-keys/alice-protected-public.gpg and belongs to the email address alice.protected@example.com. I've added this email address and re-encrypted the secrets afterwards via

make gpg-init
make secret-add-user EMAIL="alice.protected@example.com"
make secret-encrypt

When I now import the secret-protected.gpg.example key, I can decrypt the secrets, though I cannot use the usual secret-decrypt target but must instead use secret-decrypt-with-password

make secret-decrypt-with-password GPG_PASSWORD=12345678

or store the GPG_PASSWORD in the .make/.env file when it is initialized for CI

make make-init ENVS="ENV=ci TAG=latest EXECUTE_IN_CONTAINER=true GPG_PASSWORD=12345678"
make secret-decrypt-with-password

Create a JUnit report from PhpUnit

I've added the --log-junit option to the phpunit configuration of the test make target in order to create an XML report in the .build/ directory in the .make/01-02-application-qa.mk file:

# File: .make/01-02-application-qa.mk
# ...

PHPUNIT_CMD=php vendor/bin/phpunit
PHPUNIT_ARGS= -c phpunit.xml --log-junit .build/report.xml

I.e. each run of the tests will now create a Junit XML report at .build/report.xml. The file is used as an example of a build artifact, i.e. "something that we would like to keep" from a CI run.

Wrapping up

Congratulations, you made it! If some things are not completely clear by now, don't hesitate to leave a comment. You should now have a working CI pipeline for Github (via Github Actions) and/or Gitlab (via Gitlab pipelines) that runs automatically on each push.

In the next part of this tutorial, we will create a VM on GCP and provision it to run dockerized applications.

Please subscribe to the RSS feed or via email to get automatic notifications when this next part comes out :)

FYI: This tutorial is a precursor to the next a part Create a CI pipeline for dockerized PHP Apps because dealing with secrets is an important aspect when setting up a CI system (and later when deploying to production) - but I feel it's complex enough to warrant its own article.

All code samples are publicly available in my Docker PHP Tutorial repository on Github.
You find the branch with the final result of this tutorial at part-6-git-secret-encrypt-repository-docker.

All published parts of the Docker PHP Tutorial are collected under a dedicated page at Docker PHP Tutorial. The previous part was Set up PHP QA tools and control them via make and the following one is Create a CI pipeline for dockerized PHP Apps.

If you want to follow along, please subscribe to the RSS feed or via email to get automatic notifications when the next part comes out :)

Table of contents

• Introduction
• Tooling
  • gpg
    • gpg installation
    • gpg usage
      • Create GPG key pair
      • Export, list and import private GPG keys
      • Export, list and import public GPG keys
  • git-secret
    • git-secret installation
      • The git permission issue
    • git-secret usage
      • Initialize git-secret
        • The git-secret directory and the gpg-agent socket
      • Adding, listing and removing users
        • Reminder: Rotate the encrypted secrets
      • Adding, listing and removing files for encryption
      • Encrypt files
      • Decrypting files
      • Show changes between encrypted and decrypted files
• Makefile adjustments
• Workflow
  • Process challenges
    • Updating secrets
    • Code reviews and merge conflicts
    • Local git-secret and gpg setup
  • Scenarios
    • Initial setup of gpg keys
    • Initial setup of git-secret
    • Initialize gpg after container startup
    • Adding (new) team members
    • Adding and encrypting files
    • Decrypt files
    • Removing files
    • Removing team members
• Pros and cons
  • Pro
  • Cons
• Wrapping up

Introduction

Dealing with secrets (passwords, tokens, key files, etc.) is close to "naming things" when it comes to hard problems in software engineering. Some things to consider:

• security is paramount - but high security often goes hand in hand with high inconvenience
  • and if things get too complicated, people look for shortcuts...
• in a team, sharing certain secret values is often mandatory
  • so now we need to think about secure ways to distribute and update secrets across multiple people
• concrete secret values often depend on the environment
  • inherently tricky to "test" or even "review", because those values are "by definition" different on "your machine" than on "production"

In fact, entire products have been build around dealing with secrets, e.g. HashiCorp Vault, AWS Secrets Manager or the GCP Secret Manager. Introducing those in a project comes with a certain overhead as it's yet another service that needs to be integrated and
maintained. Maybe it is the exactly right decision for your use-case - maybe it's overkill. By the end of this article you'll at least be aware of an alternative with a lower barrier to entry. See also the Pros and cons section in the end for an overview.

Even though it's generally not advised to store secrets in a repository, I'll propose exactly that in this tutorial:

• identify files that contain secret values
• make sure they are added to .gitignore
• encrypt them via git-secret
• commit the encrypted files to the repository

In the end, we will be able to call

make secret-decrypt

to reveal secrets in the codebase, make modifications to them if necessary and then run

make secret-encrypt

to encrypt them again so that they can be committed (and pushed to the remote repository). To see it in action, check out branch part-6-git-secret-encrypt-repository-docker and run the following commands:

# checkout the branch
git checkout part-6-git-secret-encrypt-repository-docker

# build and start the docker setup
make make-init
make docker-build
make docker-up

# "create" the secret key - the file "secret.gpg.example" would usually NOT live in the repo!
cp secret.gpg.example secret.gpg

# initialize gpg
make gpg-init

# ensure that the decrypted secret file does not exist
ls passwords.txt

# decrypt the secret file
make secret-decrypt

# show the content of the secret file
cat passwords.txt

Tooling

We will set up gpg and git-secret in the php base image, so that the tools become available in all other containers. Please refer to Docker from scratch for PHP 8.1 Applications in 2022 for an in-depth explanation of the docker images.

Please note, that there is a caveat when using git-secret in a folder that is shared between the host system and a docker container. I'll explain that in more detail (including a workaround) in section The git-secret directory and the gpg-agent socket.

gpg

gpg is short for The GNU Privacy Guard and is an open source implementation of the OpenPGP standard. In short, it allows us to create a personal key file pair (similar to SSH keys) with a private secret key and a public key that can be shared with other parties whose messages you want to decrypt.

gpg installation

To install it, we can simply run apk add gnupg and thus update .docker/images/php/base/Dockerfile accordingly

# File: .docker/images/php/base/Dockerfile

RUN apk add --update --no-cache \
        bash \
        gnupg \
        make \
#...

gpg usage

I'll only cover the strictly necessary gpg commands here. Please refer to the "Using GPG" section in the git-secret docu for further information.

Create GPG key pair

We need gpg to create the gpg key pair via

name="Pascal Landau"
email="pascal.landau@example.com"
gpg --batch --gen-key <<EOF
Key-Type: 1
Key-Length: 2048
Subkey-Type: 1
Subkey-Length: 2048
Name-Real: $name
Name-Email: $email
Expire-Date: 0
%no-protection
EOF

The %no-protection will create a key without password, see also this gist to "Creating gpg keys non-interactively". To use a password (e.g. 12345678, we could have replace the %no-protection line with

Passphrase: 12345678

All options for the unattended creation are defined in the official docs at "Unattended key generation".

Output:

$ name="Pascal Landau"
$ email="pascal.landau@example.com"
$ gpg --batch --gen-key <<EOF
> Key-Type: 1
> Key-Length: 2048
> Subkey-Type: 1
> Subkey-Length: 2048
> Name-Real: $name
> Name-Email: $email
> Expire-Date: 0
> %no-protection
> EOF
gpg: key E1E734E00B611C26 marked as ultimately trusted
gpg: revocation certificate stored as '/root/.gnupg/opengpg-revocs.d/74082D81525723F5BF5B2099E1E734E00B611C26.rev'

You could also run gpg --gen-key without the --batch flag to be guided interactively through the process.

Export, list and import private GPG keys

The private key can be exported via

email="pascal.landau@example.com"
path="secret.gpg"
gpg --output "$path" --armor --export-secret-key "$email"

This secret key must never be shared!

It looks like this:

-----BEGIN PGP PRIVATE KEY BLOCK-----

lQOYBF7VVBwBCADo9un+SySu/InHSkPDpFVKuZXg/s4BbZmqFtYjvUUSoRAeSejv
G21nwttQGut+F+GdpDJL6W4pmLS31Kxpt6LCAxhID+PRYiJQ4k3inJfeUx7Ws339
XDPO3Rys+CmnZchcEgnbOfQlEqo51DMj6mRF2Ra/6svh7lqhrixGx1BaKn6VlHkC
...
ncIcHxNZt7eK644nWDn7j52HsRi+wcWsZ9mjkUgZLtyMPJNB5qlKQ18QgVdEAhuZ
xT3SieoBPd+tZikhu3BqyIifmLnxOJOjOIhbQrgFiblvzU1iOUOTOcSIB+7A
=YmRm
-----END PGP PRIVATE KEY BLOCK-----

All secret keys can be listed via

gpg --list-secret-keys

Output:

$ gpg --list-secret-keys
/root/.gnupg/pubring.kbx
------------------------
sec   rsa2048 2022-03-27 [SCEA]
      74082D81525723F5BF5B2099E1E734E00B611C26
uid           [ultimate] Pascal Landau <pascal.landau@example.com>
ssb   rsa2048 2022-03-27 [SEA]

You can import the private key via

path="secret.gpg"
gpg --import "$path"

and get the following output:

$ path="secret.gpg"
$ gpg --import "$path"
gpg: key E1E734E00B611C26: "Pascal Landau <pascal.landau@example.com>" not changed
gpg: key E1E734E00B611C26: secret key imported
gpg: Total number processed: 1
gpg:              unchanged: 1
gpg:       secret keys read: 1
gpg:  secret keys unchanged: 1

Caution: If the secret key requires a password, you would now be prompted for it. We can circumvent the prompt by using --batch --yes --pinentry-mode loopback:

path="secret.gpg"
gpg --import --batch --yes --pinentry-mode loopback "$path"

See also Using Command-Line Passphrase Input for GPG. In doing so, we don't need to provide the password just yet - but we must pass it later when we attempt to decrypt files.

Export, list and import public GPG keys

The public key can be exported to public.gpg via

email="pascal.landau@example.com"
path="public.gpg"
gpg --armor --export "$email" > "$path"

It looks like this:

-----BEGIN PGP PUBLIC KEY BLOCK-----

mQENBF7VVBwBCADo9un+SySu/InHSkPDpFVKuZXg/s4BbZmqFtYjvUUSoRAeSejv
G21nwttQGut+F+GdpDJL6W4pmLS31Kxpt6LCAxhID+PRYiJQ4k3inJfeUx7Ws339
...
3LLbK7Qxz0cV12K7B+n2ei466QAYXo03a7WlsPWn0JTFCsHoCOphjaVsncIcHxNZ
t7eK644nWDn7j52HsRi+wcWsZ9mjkUgZLtyMPJNB5qlKQ18QgVdEAhuZxT3SieoB
Pd+tZikhu3BqyIifmLnxOJOjOIhbQrgFiblvzU1iOUOTOcSIB+7A
=g0hF
-----END PGP PUBLIC KEY BLOCK-----

List all public keys via

gpg --list-keys

Output:

$ gpg --list-keys
/root/.gnupg/pubring.kbx
------------------------
pub   rsa2048 2022-03-27 [SCEA]
      74082D81525723F5BF5B2099E1E734E00B611C26
uid           [ultimate] Pascal Landau <pascal.landau@example.com>
sub   rsa2048 2022-03-27 [SEA]

The public key can be imported in the same way as private keys via

path="public.gpg"
gpg --import "$path"

Example:

$ gpg --import /var/www/app/public.gpg
gpg: key E1E734E00B611C26: "Pascal Landau <pascal.landau@example.com>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1

git-secret

The official website of git-secret is already doing a great job of introducing the tool. In short, it allows us to declare certain files as "secrets" and encrypt them via gpg - using the keys of all trusted parties. The encrypted file can then by stored safely directly in the git repository and decrypted if required.

In this tutorial I'm using git-secret v0.4.0

$ git secret --version
0.4.0

git-secret installation

The installation instructions for Alpine read as follows:

sh -c "echo 'https://gitsecret.jfrog.io/artifactory/git-secret-apk/all/main'" >> /etc/apk/repositories
wget -O /etc/apk/keys/git-secret-apk.rsa.pub 'https://gitsecret.jfrog.io/artifactory/api/security/keypair/public/repositories/git-secret-apk'
apk add --update --no-cache git-secret

Plus, we need to account for a recent change in git that requires that the parent directory is owned by the user executing the git command. See also the more detailed explanation in section The git permission issue.

We update the .docker/images/php/base/Dockerfile accordingly:

# File: .docker/images/php/base/Dockerfile

# install git-secret
# @see https://git-secret.io/installation#alpine
ADD https://gitsecret.jfrog.io/artifactory/api/security/keypair/public/repositories/git-secret-apk /etc/apk/keys/git-secret-apk.rsa.pub

RUN echo "https://gitsecret.jfrog.io/artifactory/git-secret-apk/all/main" >> /etc/apk/repositories  && \
    apk add --update --no-cache \
        bash \
        git-secret \
        gawk \
        gnupg \
        make \
#...

# Fix the git permission issue
RUN git config --system --add safe.directory "$APP_CODE_PATH"

The git permission issue

In April 2022, Github accounced the security vulnerability CVE-2022-24765, that was fixed in git v2.35.2

This version changes Git’s behavior when looking for a top-level .git directory to stop when its directory traversal changes ownership from the current user.

In practice, the following error occurs if the parent directory is not owned by the user that executes the git command

Error: fatal: unsafe repository ('/parent/dir/of/.git-folder' is owned by someone else)
To add an exception for this directory, call:

    git config --global --add safe.directory /parent/dir/of/.git-folder

When using git secret, we would get the slightly misleading error message

git-secret: abort: not in dir with git repo. Use 'git init' or 'git clone', then in repo use 'git secret init'

We can "fix" the issue by using the new multi-valued safe.directory configuration via

git config --system --add safe.directory /parent/dir/of/.git-folder

Note, that we didn't use the suggested --global option but --system instead, so that the configuration is set for any user.

Wait - why not just ensure that the parent directory of the .git folder has the correct permissions?

Well... there's currently (2022-05-28) a bug in Docker Desktop that makes the permissions of bind mounts kinda unpredictable, see Ownership of files set via bind mount is set to user who accesses the file first and by applying the fix directly in the Dockerfile we can solve the issue reliably.

git-secret usage

Initialize git-secret

git-secret is initialized via the following command run in the root of the git repository

git secret init

$ git secret init
git-secret: init created: '/var/www/app/.gitsecret/'

We only need to do this once, because we'll commit the folder to git later. It contains the following files:

$ git status | grep ".gitsecret"
        new file:   .gitsecret/keys/pubring.kbx
        new file:   .gitsecret/keys/pubring.kbx~
        new file:   .gitsecret/keys/trustdb.gpg
        new file:   .gitsecret/paths/mapping.cfg

The pubring.kbx~ file (with the trailing tilde ~) is only a temporary file and can safely be git-ignored. See also Can't find any docs about keyring.kbx~ file.

The git-secret directory and the gpg-agent socket

To use git-secret in a directory that is shared between the host system and docker, we need to also run the following commands:

tee .gitsecret/keys/S.gpg-agent <<EOF
%Assuan%
socket=/tmp/S.gpg-agent
EOF

tee .gitsecret/keys/S.gpg-agent.ssh <<EOF
%Assuan%
socket=/tmp/S.gpg-agent.ssh
EOF

tee .gitsecret/keys/gpg-agent.conf <<EOF
extra-socket /tmp/S.gpg-agent.extra
browser-socket /tmp/S.gpg-agent.browser
EOF

This is necessary because there is an issue when git-secret is used in a setup where the codebase is shared between the host system and a docker container. I've explained the details in the Github issue "gpg: can't connect to the agent: IPC connect call failed" error in docker alpine on shared volume.

In short:

• gpg uses a gpg-agent to perform its tasks and the two tools communicate through sockets that are created in the --home-directory of the gpg-agent
• the agent is started implicitly through a gpg command used by git-secret, using the .gitsecret/keys directories as a --home-directory
• because the location of the --home-directory is shared with the host system, the socket creation fails (potentially only an issue for Docker Desktop, see the related discussion in Github issue Support for sharing unix sockets)

The corresponding error messages are

gpg: can't connect to the agent: IPC connect call failed

gpg-agent: error binding socket to '/var/www/app/.gitsecret/keys/S.gpg-agent': I/O error

The workaround for this problem can be found in this thread: Configure gpg to use different locations for the sockets by placing additional gpg configuration files in the .gitsecret/keys directory:

S.gpg-agent

%Assuan%
socket=/tmp/S.gpg-agent

S.gpg-agent.ssh

%Assuan%
socket=/tmp/S.gpg-agent.ssh

gpg-agent.conf

extra-socket /tmp/S.gpg-agent.extra
browser-socket /tmp/S.gpg-agent.browser

Adding, listing and removing users

To add a new user, you must first import its public gpg key. Then run:

email="pascal.landau@example.com"
git secret tell "$email"

In this case, the user pascal.landau@example.com will now be able to decrypt the secrets.

To show the users run

git secret whoknows

$ git secret whoknows
pascal.landau@example.com

To remove a user, run

email="pascal.landau@example.com"
git secret killperson "$email"

FYI: This command was renamed to removeperson in git-secret >= 0.5.0

$ git secret killperson pascal.landau@example.com
git-secret: removed keys.
git-secret: now [pascal.landau@example.com] do not have an access to the repository.
git-secret: make sure to hide the existing secrets again.

User pascal.landau@example.com will no longer be able to decrypt the secrets.

Caution: The secrets need to be re-encrypted after removing a user!

Reminder: Rotate the encrypted secrets

Please be aware that not only your secrets are stored in git, but who had access as well. I.e. even if you remove a user and re-encrypt the secrets, that user would still be able to decrypt the secrets of a previous commit (when the user was still added). In consequence, you need to rotate the encrypted secrets themselves as well after removing a user.

But isn't that a great flaw in the system, making it a bad idea to use git-secret in general?

In my opinion: No.

If the removed user had access to the secrets at any point in time (no matter where they have been stored), he could very well have just created a local copy or simply "written them down". In terms of security there is really no "added downside" due to git-secret. It just makes it very clear that you must rotate the secrets ¯\_(ツ)_/¯

See also this lengthy discussion on git-secret on Hacker News.

Adding, listing and removing files for encryption

Run git secret add [filenames...] for files you want to encrypt. Example:

git secret add .env

If .env is not added in .gitignore, git-secret will display a warning and add it automatically.

git-secret: these files are not in .gitignore: .env
git-secret: auto adding them to .env
git-secret: 1 item(s) added.

Otherwise, the file is added with no warning.

$ git secret add .env
git-secret: 1 item(s) added.

You only need to add files once. They are then stored in .gitsecret/paths/mapping.cfg:

$ cat .gitsecret/paths/mapping.cfg
.env:505070fc20233cb426eac6a3414399d0f466710c993198b1088e897fdfbbb2d5

You can also show the added files via

git secret list

$ git secret list
.env

Caution: The files are not yet encrypted!

If you want to remove a file from being encrypted, run

git secret remove .env

Output

$ git secret remove .env
git-secret: removed from index.
git-secret: ensure that files: [.env] are now not ignored.

Encrypt files

To actually encrypt the files, run:

git secret hide

Output:

$ git secret hide
git-secret: done. 1 of 1 files are hidden.

The encrypted (binary) file is stored at $filename.secret, i.e. .env.secret in this case:

$ cat .env.secret
�☺♀♥�H~�B�Ӯ☺�"��▼♂F�►���l�Cs��S�@MHWs��e������{♣♫↕↓�L� ↕s�1�J$◄♥�;���dž֕�Za�����\u�ٲ& ¶��V�► ���6��
;<�d:��}ҨD%.�;��&��G����vWW�]>���߶��▲;D�+Rs�S→�Y!&J��۪8���ٔF��→f����*��$♠���&RC�8▼♂�☻z h��Z0M�T>

The encrypted files are de-cryptable for all users that have been added via git secret tell. That also means that you need to run this command again whenever a new user is added.

Decrypting files

You can decrypt files via

git secret reveal

Output:

$ git secret reveal
File '/var/www/app/.env' exists. Overwrite? (y/N) y
git-secret: done. 1 of 1 files are revealed.

• the files are decrypted and will overwrite the current, unencrypted files (if they already exist)
  • use the -f option to force the overwrite and run non-interactively
• if you only want to check the content of an encrypted file, you can use git secret cat $filename (e.g. git secret cat .env)

In case the secret gpg key is password protected, you must pass the password via the -p option. E.g. for password 123456

git secret reveal -p 123456

Show changes between encrypted and decrypted files

One problem that comes with encrypted files: You can't review them during a code review in a remote tool. So in order to understand what changes have been made, it is helpful to show the changes between the encrypted and the decrypted files. This can be done via

git secret changes

Output:

$ echo "foo" >> .env
$ git secret changes
git-secret: changes in /var/www/app/.env:
--- /dev/fd/63
+++ /var/www/app/.env
@@ -34,3 +34,4 @@
 MAIL_ENCRYPTION=null
 MAIL_FROM_ADDRESS=null
 MAIL_FROM_NAME="${APP_NAME}"
+foo

Note the +foo at the bottom of the output. It was added in the first line via echo "foo"> >> .env.

Makefile adjustments

Since I won't be able to remember all the commands for git-secret and gpg, I've added them to the Makefile at .make/01-00-application-setup.mk:

# File: .make/01-00-application-setup.mk

#...

# gpg

DEFAULT_SECRET_GPG_KEY?=secret.gpg
DEFAULT_PUBLIC_GPG_KEYS?=.dev/gpg-keys/*

.PHONY: gpg
gpg: ## Run gpg commands. Specify the command e.g. via ARGS="--list-keys"
    $(EXECUTE_IN_APPLICATION_CONTAINER) gpg $(ARGS)

.PHONY: gpg-export-public-key
gpg-export-public-key: ## Export a gpg public key e.g. via EMAIL="john.doe@example.com" PATH=".dev/gpg-keys/john-public.gpg"
    @$(if $(PATH),,$(error PATH is undefined))
    @$(if $(EMAIL),,$(error EMAIL is undefined))
    "$(MAKE)" -s gpg ARGS="gpg --armor --export $(EMAIL) > $(PATH)"

.PHONY: gpg-export-private-key
gpg-export-private-key: ## Export a gpg private key e.g. via EMAIL="john.doe@example.com" PATH="secret.gpg"
    @$(if $(PATH),,$(error PATH is undefined))
    @$(if $(EMAIL),,$(error EMAIL is undefined))
    "$(MAKE)" -s gpg ARGS="--output $(PATH) --armor --export-secret-key $(EMAIL)"

.PHONY: gpg-import
gpg-import: ## Import a gpg key file e.g. via GPG_KEY_FILES="/path/to/file /path/to/file2"
    @$(if $(GPG_KEY_FILES),,$(error GPG_KEY_FILES is undefined))
    "$(MAKE)" -s gpg ARGS="--import --batch --yes --pinentry-mode loopback $(GPG_KEY_FILES)"

.PHONY: gpg-import-default-secret-key
gpg-import-default-secret-key: ## Import the default secret key
    "$(MAKE)" -s gpg-import GPG_KEY_FILES="$(DEFAULT_SECRET_GPG_KEY)"

.PHONY: gpg-import-default-public-keys
gpg-import-default-public-keys: ## Import the default public keys
    "$(MAKE)" -s gpg-import GPG_KEY_FILES="$(DEFAULT_PUBLIC_GPG_KEYS)" 

.PHONY: gpg-init
gpg-init: gpg-import-default-secret-key gpg-import-default-public-keys ## Initialize gpg in the container, i.e. import all public and private keys

# git-secret

.PHONY: git-secret
git-secret: ## Run git-secret commands. Specify the command e.g. via ARGS="hide"
    $(EXECUTE_IN_APPLICATION_CONTAINER) git-secret $(ARGS)

.PHONY: secret-init
secret-init: ## Initialize git-secret in the repository via `git-secret init`
    "$(MAKE)" -s git-secret ARGS="init"

.PHONY: secret-init-gpg-socket-config
secret-init-gpg-socket-config: ## Initialize the config files to change the gpg socket locations
    echo "%Assuan%" > .gitsecret/keys/S.gpg-agent
    echo "socket=/tmp/S.gpg-agent" >> .gitsecret/keys/S.gpg-agent
    echo "%Assuan%" > .gitsecret/keys/S.gpg-agent.ssh
    echo "socket=/tmp/S.gpg-agent.ssh" >> .gitsecret/keys/S.gpg-agent.ssh
    echo "extra-socket /tmp/S.gpg-agent.extra" > .gitsecret/keys/gpg-agent.conf
    echo "browser-socket /tmp/S.gpg-agent.browser" >> .gitsecret/keys/gpg-agent.conf

.PHONY: secret-encrypt
secret-encrypt: ## Decrypt secret files via `git-secret hide`
    "$(MAKE)" -s git-secret ARGS="hide"

.PHONY: secret-decrypt
secret-decrypt: ## Decrypt secret files via `git-secret reveal -f`
    "$(MAKE)" -s git-secret ARGS="reveal -f" 

.PHONY: secret-decrypt-with-password
secret-decrypt-with-password: ## Decrypt secret files using a password for gpg via `git-secret reveal -f -p $(GPG_PASSWORD)`
    @$(if $(GPG_PASSWORD),,$(error GPG_PASSWORD is undefined))
    "$(MAKE)" -s git-secret ARGS="reveal -f -p $(GPG_PASSWORD)" 

.PHONY: secret-add
secret-add: ## Add a file to git secret via `git-secret add $FILE`
    @$(if $(FILE),,$(error FILE is undefined))
    "$(MAKE)" -s git-secret ARGS="add $(FILE)"

.PHONY: secret-cat
secret-cat: ## Show the contents of file to git secret via `git-secret cat $FILE`
    @$(if $(FILE),,$(error FILE is undefined))
    "$(MAKE)" -s git-secret ARGS="cat $(FILE)"

.PHONY: secret-list
secret-list: ## List all files added to git secret `git-secret list`
    "$(MAKE)" -s git-secret ARGS="list"

.PHONY: secret-remove
secret-remove: ## Remove a file from git secret via `git-secret remove $FILE`
    @$(if $(FILE),,$(error FILE is undefined))
    "$(MAKE)" -s git-secret ARGS="remove $(FILE)"

.PHONY: secret-add-user
secret-add-user: ## Remove a user from git secret via `git-secret tell $EMAIL`
    @$(if $(EMAIL),,$(error EMAIL is undefined))
    "$(MAKE)" -s git-secret ARGS="tell $(EMAIL)"

.PHONY: secret-show-users
secret-show-users: ## Show all users that have access to git secret via `git-secret whoknows`
    "$(MAKE)" -s git-secret ARGS="whoknows"

.PHONY: secret-remove-user
secret-remove-user: ## Remove a user from git secret via `git-secret killperson $EMAIL`
    @$(if $(EMAIL),,$(error EMAIL is undefined))
    "$(MAKE)" -s git-secret ARGS="killperson $(EMAIL)"

.PHONY: secret-diff
secret-diff: ## Show the diff between the content of encrypted and decrypted files via `git-secret changes`
    "$(MAKE)" -s git-secret ARGS="changes"

Workflow

Working with git-secret is pretty straight forward:

• initialize git-secret
• add all users
• add all secret files and make sure they are ignored via .gitignore
• encrypt the files
• commit the encrypted files like "any other file"
• if any changes were made by other team members to the files:
  • => decrypt to get the most up-to-date ones
• if any modifications are required from your side:
  • => make the changes to the decrypted files and then re-encrypt them again

But: The devil is in the details. The Process challenges section explains some of the pitfalls that we have encountered and the Scenarios section gives some concrete examples for common scenarios.

Process challenges

From a process perspective we've encountered some challenges that I'd like to mention - including how we deal with them.

Updating secrets

When updating secrets you must ensure to always decrypt the files first in order to avoid using "stale" files that you might still have locally. I usually check out the latest main branch and run git secret reveal to have the most up-to-date versions of the secret files. You could also use a post-merge git hook to do this automatically, but I personally don't want to risk overwriting my local secret files by accident.

Code reviews and merge conflicts

Since the encrypted files cannot be diffed meaningfully, the code reviews become more difficult when secrets are involved. We use Gitlab for reviews and I usually first check the diff of the .gitsecret/paths/mapping.cfg file to see "which files have changed" directly in the UI.

In addition, I will

• checkout the main branch
• decrypt the secrets via git secret reveal -f
• checkout the feature-branch
• run git secret changes to see the differences between the decrypted files from main and the encrypted files from feature-branch

Things get even more complicated when multiple team members need to modify secret files at the same time on different branches, as the encrypted files cannot be compared - i.e. git cannot be smart about delta updates. The only way around this is coordinating the pull requests, i.e. merge the first, update the secrets of the second and then merge the second.

Fortunately, this has only happened very rarely so far.

Local git-secret and gpg setup

Currently, all developers in our team have git-secret installed locally (instead of using it through docker) and use their own gpg keys.

This means more onboarding overhead, because

• a new dev must
  • install git-secret locally (*)
  • install and setup gpg locally (*)
  • create a gpg key pair
• the public key must be added by every other team member (*)
• the user of the key must be added via git secret tell
• the secrets must be re-encrypted

And for offboarding

• the public key must be removed by every other team member (*)
• the user of the key must be removed via git secret killperson
• the secrets must be re-encrypted

Plus, we need to ensure that the git-secret and gpg versions are kept up-to-date for everyone to not run into any compatibility issues.

As an alternative, I'm currently leaning more towards handling everything through docker (as presented in this tutorial). All steps marked with (*) are then obsolete, i.e. there is no need to setup git-secret and gpg locally.

But the approach also comes with some downsides, because - the secret key and all public keys have to be imported every time the container is started - each dev needs to put his private gpg key "in the codebase" (ignored by .gitignore) so it can be shared with docker and imported by gpg (in docker). The alternative would be using a single secret key that is shared within the team - which feels very wrong :P

To make this a little more convenient, we put the public gpg keys of every dev in the repository under .dev/gpg-keys/ and the private key has to be named secret.gpg and put in the root of the codebase.

In this setup, secret.gpg must also be added to the.gitignore file.

# File: .gitignore
#...
vendor/
secret.gpg

The import can now be simplified with make targets:

# gpg

DEFAULT_SECRET_GPG_KEY?=secret.gpg
DEFAULT_PUBLIC_GPG_KEYS?=.dev/gpg-keys/*

.PHONY: gpg
gpg: ## Run gpg commands. Specify the command e.g. via ARGS="--list-keys"
    $(EXECUTE_IN_APPLICATION_CONTAINER) gpg $(ARGS)

.PHONY: gpg-import
gpg-import: ## Import a gpg key file e.g. via GPG_KEY_FILES="/path/to/file /path/to/file2"
    @$(if $(GPG_KEY_FILES),,$(error GPG_KEY_FILES is undefined))
    "$(MAKE)" -s gpg ARGS="--import --batch --yes --pinentry-mode loopback $(GPG_KEY_FILES)"

.PHONY: gpg-import-default-secret-key
gpg-import-default-secret-key: ## Import the default secret key
    "$(MAKE)" -s gpg-import GPG_KEY_FILES="$(DEFAULT_SECRET_GPG_KEY)"

.PHONY: gpg-import-default-public-keys
gpg-import-default-public-keys: ## Import the default public keys
    "$(MAKE)" -s gpg-import GPG_KEY_FILES="$(DEFAULT_PUBLIC_GPG_KEYS)" 

.PHONY: gpg-init
gpg-init: gpg-import-default-secret-key gpg-import-default-public-keys ## Initialize gpg in the container, i.e. import all public and private keys

"Everything" can now be handled via

make gpg-init

that needs to be run one single time after a container has been started.

Scenarios

The scenarios assume the following preconditions:

• You have checked out branch part-6-git-secret-encrypt-repository-docker bash git checkout part-6-git-secret-encrypt-repository-docker and no running docker containers bash make docker-down
• You have deleted the existing git-secret folder, the keys in .dev/gpg-keys, the secret.gpg key and the passwords.* files bash rm -rf .gitsecret/ .dev/gpg-keys/* secret.gpg passwords.*

Initial setup of gpg keys

Unfortunately, I didn't find a way to create and export gpg keys through make and docker. You need to either run the commands interactively OR pass a string with newlines to it. Both things are horribly complicated with make and docker. Thus, you need to log into the application container and run the commands in there directly. Not great - but this needs to be done only once when a new developer is onboarded anyways.

FYI: I usually log into containers via Easy container access via din .bashrc helper.

The secret key is exported to secret.gpg and the public key to .dev/gpg-keys/alice-public.gpg.

# start the docker setup
make docker-up

# log into the container ('winpty' is only required on Windows)
winpty docker exec -ti dofroscra_local-application-1 bash

# export key pair
name="Alice Doe"
email="alice@example.com"
gpg --batch --gen-key <<EOF
Key-Type: 1
Key-Length: 2048
Subkey-Type: 1
Subkey-Length: 2048
Name-Real: $name
Name-Email: $email
Expire-Date: 0
%no-protection
EOF

# export the private key
gpg --output secret.gpg --armor --export-secret-key $email

# export the public key
gpg --armor --export $email > .dev/gpg-keys/alice-public.gpg

$ make docker-up
ENV=local TAG=latest DOCKER_REGISTRY=docker.io DOCKER_NAMESPACE=dofroscra APP_USER_NAME=application APP_GROUP_NAME=application docker compose -p dofroscra_local --env-file ./.docker/.env -f ./.docker/docker-compose/docker-compose.yml -f ./.docker/docker-compose/docker-compose.local.yml up -d
Container dofroscra_local-application-1  Created
...
Container dofroscra_local-application-1  Started
$ docker ps
CONTAINER ID   IMAGE                                COMMAND                  CREATED          STATUS          PORTS                NAMES
...
95f740607586   dofroscra/application-local:latest   "/usr/sbin/sshd -D"      21 minutes ago   Up 21 minutes   0.0.0.0:2222->22/tcp dofroscra_local-application-1

$ winpty docker exec -ti dofroscra_local-application-1 bash
root:/var/www/app# name="Alice Doe"
root:/var/www/app# email="alice@example.com"
gpg --batch --gen-key <<EOF
Key-Type: 1
Key-Length: 2048
Subkey-Type: 1
Subkey-Length: 2048
Name-Real: $name
Name-Email: $email
Expire-Date: 0
%no-protection
EOF
root:/var/www/app# gpg --batch --gen-key <<EOF
> Key-Type: 1
> Key-Length: 2048
> Subkey-Type: 1
> Subkey-Length: 2048
> Name-Real: $name
> Name-Email: $email
> Expire-Date: 0
> %no-protection
> EOF
gpg: directory '/root/.gnupg' created
gpg: keybox '/root/.gnupg/pubring.kbx' created
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key BBBE654440E720C1 marked as ultimately trusted
gpg: directory '/root/.gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/root/.gnupg/openpgp-revocs.d/225C736E0E70AC222C072B70BBBE654440E720C1.rev'

root:/var/www/app# gpg --output secret.gpg --armor --export-secret-key $email
root:/var/www/app# head secret.gpg
-----BEGIN PGP PRIVATE KEY BLOCK-----

lQOYBGJD+bwBCADBGKySV5PINc5MmQB3PNvCG7Oa1VMBO8XJdivIOSw7ykv55PRP
3g3R+ERd1Ss5gd5KAxLc1tt6PHGSPTypUJjCng2plwD8Jy5A/cC6o2x8yubOslLa
x1EC9fpcxUYUNXZavtEr+ylOaTaRz6qwSabsAgkg2NZ0ey/QKmFOZvhL8NlK9lTI
GgZPTiqPCsr7hiNg0WRbT5h8nTmfpl/DdTgwfPsDn5Hn0TEMa79WsrPnnq16jsq0
Uusuw3tOmdSdYnT8j7m1cpgcSj0hRF1eh4GVE0o62GqeLTWW9mfpcuv7n6mWaCB8
DCH6H238gwUriq/aboegcuBktlvSY21q/MIXABEBAAEAB/wK/M2buX+vavRgDRgR
hjUrsJTXO3VGLYcIetYXRhLmHLxBriKtcBa8OxLKKL5AFEuNourOBdcmTPiEwuxH
5s39IQOTrK6B1UmUqXvFLasXghorv8o8KGRL4ABM4Bgn6o+KBAVLVIwvVIhQ4rlf

root:/var/www/app# gpg --armor --export $email > .dev/gpg-keys/alice-public.gpg
root:/var/www/app# head .dev/gpg-keys/alice-public.gpg
-----BEGIN PGP PUBLIC KEY BLOCK-----

mQENBGJD+bwBCADBGKySV5PINc5MmQB3PNvCG7Oa1VMBO8XJdivIOSw7ykv55PRP
3g3R+ERd1Ss5gd5KAxLc1tt6PHGSPTypUJjCng2plwD8Jy5A/cC6o2x8yubOslLa
x1EC9fpcxUYUNXZavtEr+ylOaTaRz6qwSabsAgkg2NZ0ey/QKmFOZvhL8NlK9lTI
GgZPTiqPCsr7hiNg0WRbT5h8nTmfpl/DdTgwfPsDn5Hn0TEMa79WsrPnnq16jsq0
Uusuw3tOmdSdYnT8j7m1cpgcSj0hRF1eh4GVE0o62GqeLTWW9mfpcuv7n6mWaCB8
DCH6H238gwUriq/aboegcuBktlvSY21q/MIXABEBAAG0HUFsaWNlIERvZSA8YWxp
Y2VAZXhhbXBsZS5jb20+iQFOBBMBCgA4FiEEIlxzbg5wrCIsBytwu75lREDnIMEF
AmJD+bwCGy8FCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQu75lREDnIMEN4Af+

That's it. We now have a new secret and private key for alice@example.com and have exported it to secret.gpg resp. .dev/gpg-keys/alice-public.gpg (and thus shared it with the host system). The remaining commands can now be run outside of the application container directly on the host system.

Initial setup of git-secret

Let's say we want to introduce git-secret "from scratch" to a new codebase. Then you would run the following commands:

Initialize git-secret

make secret-init

$ make secret-init
"C:/Program Files/Git/mingw64/bin/make" -s git-secret ARGS="init";
git-secret: init created: '/var/www/app/.gitsecret/'

Apply the gpg fix for shared directories

See The git-secret directory and the gpg-agent socket.

$ make secret-init-gpg-socket-config

$ make secret-init-gpg-socket-config
echo "%Assuan%" > .gitsecret/keys/S.gpg-agent
echo "socket=/tmp/S.gpg-agent" >> .gitsecret/keys/S.gpg-agent
echo "%Assuan%" > .gitsecret/keys/S.gpg-agent.ssh
echo "socket=/tmp/S.gpg-agent.ssh" >> .gitsecret/keys/S.gpg-agent.ssh
echo "extra-socket /tmp/S.gpg-agent.extra" > .gitsecret/keys/gpg-agent.conf
echo "browser-socket /tmp/S.gpg-agent.browser" >> .gitsecret/keys/gpg-agent.conf

Initialize gpg after container startup

After restarting the containers, we need to initialize gpg, i.e. import all public keys from .dev/gpg-keys/* and the private key from secret.gpg. Otherwise we will not be able to en- and decrypt the files.

make gpg-init

$ make gpg-init
"C:/Program Files/Git/mingw64/bin/make" -s gpg-import GPG_KEY_FILES="secret.gpg"
gpg: directory '/home/application/.gnupg' created
gpg: keybox '/home/application/.gnupg/pubring.kbx' created
gpg: /home/application/.gnupg/trustdb.gpg: trustdb created
gpg: key BBBE654440E720C1: public key "Alice Doe <alice@example.com>" imported
gpg: key BBBE654440E720C1: secret key imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg:       secret keys read: 1
gpg:   secret keys imported: 1
"C:/Program Files/Git/mingw64/bin/make" -s gpg-import GPG_KEY_FILES=".dev/gpg-keys/*"
gpg: key BBBE654440E720C1: "Alice Doe <alice@example.com>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1

Adding (new) team members

Let's start by adding our own user to git-secret

make secret-add-user EMAIL="alice@example.com"

$ make secret-add-user EMAIL="alice@example.com"
"C:/Program Files/Git/mingw64/bin/make" -s git-secret ARGS="tell alice@example.com"
git-secret: done. alice@example.com added as user(s) who know the secret.

And verify that it worked via

make secret-show-users

$ make secret-show-users
"C:/Program Files/Git/mingw64/bin/make" -s git-secret ARGS="whoknows"
alice@example.com

Adding and encrypting files

Let's add a new encrypted file secret_password.txt.

Create the file

echo "my_new_secret_password" > secret_password.txt

Add it to .gitignore

echo "secret_password.txt" >> .gitignore

Add it to git-secret

make secret-add FILE="secret_password.txt"

$ make secret-add FILE="secret_password.txt"
"C:/Program Files/Git/mingw64/bin/make" -s git-secret ARGS="add secret_password.txt"
git-secret: 1 item(s) added.

Encrypt all files

make secret-encrypt

$ make secret-encrypt
"C:/Program Files/Git/mingw64/bin/make" -s git-secret ARGS="hide"
git-secret: done. 1 of 1 files are hidden.

$ ls secret_password.txt.secret
secret_password.txt.secret

Decrypt files

Let's first remove the "plain" secret_password.txt file

rm secret_password.txt

$ rm secret_password.txt

$ ls secret_password.txt
ls: cannot access 'secret_password.txt': No such file or directory

and then decrypt the encrypted one.

make secret-decrypt

$ make secret-decrypt
"C:/Program Files/Git/mingw64/bin/make" -s git-secret ARGS="reveal -f"
git-secret: done. 1 of 1 files are revealed.

$ cat secret_password.txt
my_new_secret_password

Caution: If the secret gpg key is password protected (e.g. 123456), run

make secret-decrypt-with-password GPG_PASSWORD=123456

You could also add the GPG_PASSWORD variable to the .make/.env file as a local default value so that you wouldn't have to specify the value every time and could then simply run

make secret-decrypt-with-password

without passing GPG_PASSWORD

Removing files

Remove the secret_password.txt file we added previously:

make secret-remove FILE="secret_password.txt"

$ make secret-remove FILE="secret_password.txt"
"C:/Program Files/Git/mingw64/bin/make" -s git-secret ARGS="remove secret_password.txt"
git-secret: removed from index.
git-secret: ensure that files: [secret_password.txt] are now not ignored.

Caution: this will neither remove the secret_password.txt file nor the secret_password.txt.secret file automatically"

$ ls -l | grep secret_password.txt
-rw-r--r-- 1 Pascal 197121     19 Mar 31 14:03 secret_password.txt
-rw-r--r-- 1 Pascal 197121    358 Mar 31 14:02 secret_password.txt.secret

But even though the encrypted secret_password.txt.secret file still exists, it will not be decrypted:

$ make secret-decrypt
"C:/Program Files/Git/mingw64/bin/make" -s git-secret ARGS="reveal -f"
git-secret: done. 0 of 0 files are revealed.

Removing team members

Removing a team member can be done via

make secret-remove-user EMAIL="alice@example.com"

$ make secret-remove-user EMAIL="alice@example.com"
"C:/Program Files/Git/mingw64/bin/make" -s git-secret ARGS="killperson alice@example.com"
git-secret: removed keys.
git-secret: now [alice@example.com] do not have an access to the repository.
git-secret: make sure to hide the existing secrets again.

If there are any users left, we must make sure to re-encrypt the secrets via

make secret-encrypt

Otherwise (if no more users are left) git-secret would simply error out

$ make secret-decrypt
"C:/Program Files/Git/mingw64/bin/make" -s git-secret ARGS="reveal -f"
git-secret: abort: no public keys for users found. run 'git secret tell email@address'.
make[1]: *** [.make/01-00-application-setup.mk:57: git-secret] Error 1
make: *** [.make/01-00-application-setup.mk:69: secret-decrypt] Error 2

Caution: Please keep in mind to rotate the secrets themselves as well!

Pros and cons

Pro

• very low barrier to entry:
  • no third party service required
  • easy to integrate in existing codebases, because the secrets are located directly in the codebase
  • everything can be handled through docker (no additional local software necessary)
• once set up, it is very easy/convenient to use and can be integrated in a team workflow
• changes to secrets can be reviewed before they are merged
  • this leads to less fuck-ups on deployments
• "everything" is in the repository, which brings a lot of familiar benefits like
  • version control
  • a single git pull is the only thing you need to get everything (=> good dev experience)

Cons

• some overhead during onboarding and offboarding
• the secret key must be put in the root of the repository at ./secret.gpg
• no fine grained permissions for different secrets, e.g. the mysql password on production and staging can not be treated differently
  • if somebody can decrypt secrets, ALL of them are exposed
• if the a secret key ever gets leaked, all secrets are compromised
  • => can be mitigated (to a degree) by using a passphrase on the secret key
  • => this is kinda true for any other system that stores secrets as well BUT third parties could probably implement additional measures like multi factor authentication
• secrets are versioned alongside the users that have access, i.e. even if a user is removed at some point, he can still decrypt a previous version of the encrypted secrets
  • => must be mitigated by rotating the secrets themselves as well

Wrapping up

Congratulations, you made it! If some things are not completely clear by now, don't hesitate to leave a comment. You are now able to encrypt and decrypt secret files so that they can be stored directly in the git repository.

In the next part of this tutorial, we will set up a CI pipeline for dockerized PHP Apps on Github and Gitlab that decrypts all necessary secrets and then runs our tests and qa tools.

Please subscribe to the RSS feed or via email to get automatic notifications when this next part comes out :)

FYI: Originally I wanted this tutorial to be a part of Create a CI pipeline for dockerized PHP Apps because QA checks are imho vital part of a CI setup - but it kinda grew "too big" and took a way too much space from, well, actually setting up the CI pipelines :)

All code samples are publicly available in my Docker PHP Tutorial repository on Github.
You find the branch with the final result of this tutorial at part-5-php-qa-tools-make-docker.

All published parts of the Docker PHP Tutorial are collected under a dedicated page at Docker PHP Tutorial. The previous part was Run Laravel 9 on Docker in 2022 and the following one is Use git-secret to encrypt secrets in the repository.

If you want to follow along, please subscribe to the RSS feed or via email to get automatic notifications when the next part comes out :)

Table of contents

• Introduction
  • The QA tools
    • phpcs and phpcbf
    • phpstan
    • php-parallel-lint
    • composer-require-checker
    • Additional tools (out of scope)
  • QA make targets
    • The qa target
    • The execute "function"
    • Parallel execution and a helper target
    • Sprinkle some color on top
  • Further updates in the codebase
  • Wrapping up

Introduction

Code quality tools ensure a baseline of code quality by automatically checking certain rules, e.g. code style definitions, proper usage of types, proper declaration of dependencies, etc. When run regularly they are a great way to enforce better code and are thus a perfect fit for a CI pipeline. For this tutorial, I'm going to setup the following tools:

• Style Checker: phpcs
• Static Analyzer: phpstan
• Code Linter: php-parallel-lint
• Dependency Checker: composer-require-checker

and provide convenient access through a qa make target. The end result will look like this:

FYI: When we started out with using code quality tools in general, we have used GrumPHP - and I would still recommend it. We have only transitioned away from it because make gives us a little more flexibility and control.

You can find the "final" makefile at .make/01-02-application-qa.mk.

CAUTION: The Makefile is build on top of the setup that I introduced in Docker from scratch for PHP 8.1 Applications in 2022, so please refer to that tutorial if anything is not clear.

##@ [Application: QA]

# variables
CORES?=$(shell (nproc  || sysctl -n hw.ncpu) 2> /dev/null)

# constants
 ## files
ALL_FILES=./
APP_FILES=app/
TEST_FILES=tests/

 ## bash colors
RED:=\033[0;31m
GREEN:=\033[0;32m
YELLOW:=\033[0;33m
NO_COLOR:=\033[0m

# Tool CLI config
PHPUNIT_CMD=php vendor/bin/phpunit
PHPUNIT_ARGS= -c phpunit.xml
PHPUNIT_FILES=
PHPSTAN_CMD=php vendor/bin/phpstan analyse
PHPSTAN_ARGS=--level=9
PHPSTAN_FILES=$(APP_FILES) $(TEST_FILES)
PHPCS_CMD=php vendor/bin/phpcs
PHPCS_ARGS=--parallel=$(CORES) --standard=psr12
PHPCS_FILES=$(APP_FILES)
PHPCBF_CMD=php vendor/bin/phpcbf
PHPCBF_ARGS=$(PHPCS_ARGS)
PHPCBF_FILES=$(PHPCS_FILES)
PARALLEL_LINT_CMD=php vendor/bin/parallel-lint
PARALLEL_LINT_ARGS=-j 4 --exclude vendor/ --exclude .docker --exclude .git
PARALLEL_LINT_FILES=$(ALL_FILES)
COMPOSER_REQUIRE_CHECKER_CMD=php vendor/bin/composer-require-checker
COMPOSER_REQUIRE_CHECKER_ARGS=--ignore-parse-errors

# call with NO_PROGRESS=true to hide tool progress (makes sense when invoking multiple tools together)
NO_PROGRESS?=false
ifeq ($(NO_PROGRESS),true)
    PHPSTAN_ARGS+= --no-progress
    PARALLEL_LINT_ARGS+= --no-progress
else
    PHPCS_ARGS+= -p
    PHPCBF_ARGS+= -p
endif

# Use NO_PROGRESS=false when running individual tools.
# On  NO_PROGRESS=true  the corresponding tool has no output on success
#                       apart from its runtime but it will still print 
#                       any errors that occured. 
define execute
    if [ "$(NO_PROGRESS)" = "false" ]; then \
        eval "$(EXECUTE_IN_APPLICATION_CONTAINER) $(1) $(2) $(3) $(4)"; \
    else \
        START=$$(date +%s); \
        printf "%-35s" "$@"; \
        if OUTPUT=$$(eval "$(EXECUTE_IN_APPLICATION_CONTAINER) $(1) $(2) $(3) $(4)" 2>&1); then \
            printf " $(GREEN)%-6s$(NO_COLOR)" "done"; \
            END=$$(date +%s); \
            RUNTIME=$$((END-START)) ;\
            printf " took $(YELLOW)$${RUNTIME}s$(NO_COLOR)\n"; \
        else \
            printf " $(RED)%-6s$(NO_COLOR)" "fail"; \
            END=$$(date +%s); \
            RUNTIME=$$((END-START)) ;\
            printf " took $(YELLOW)$${RUNTIME}s$(NO_COLOR)\n"; \
            echo "$$OUTPUT"; \
            printf "\n"; \
            exit 1; \
        fi; \
    fi
endef

.PHONY: test
test: ## Run all tests
    @$(EXECUTE_IN_APPLICATION_CONTAINER) $(PHPUNIT_CMD) $(PHPUNIT_ARGS) $(ARGS)

.PHONY: phplint
phplint: ## Run phplint on all files
    @$(call execute,$(PARALLEL_LINT_CMD),$(PARALLEL_LINT_ARGS),$(PARALLEL_LINT_FILES), $(ARGS))

.PHONY: phpcs
phpcs: ## Run style check on all application files
    @$(call execute,$(PHPCS_CMD),$(PHPCS_ARGS),$(PHPCS_FILES), $(ARGS))

.PHONY: phpcbf
phpcbf: ## Run style fixer on all application files
    @$(call execute,$(PHPCBF_CMD),$(PHPCBF_ARGS),$(PHPCBF_FILES), $(ARGS))

.PHONY: phpstan
phpstan:  ## Run static analyzer on all application and test files 
    @$(call execute,$(PHPSTAN_CMD),$(PHPSTAN_ARGS),$(PHPSTAN_FILES), $(ARGS))

.PHONY: composer-require-checker
composer-require-checker: ## Run dependency checker
    @$(call execute,$(COMPOSER_REQUIRE_CHECKER_CMD),$(COMPOSER_REQUIRE_CHECKER_ARGS),"", $(ARGS))

.PHONY: qa
qa: ## Run code quality tools on all files
    @"$(MAKE)" -j $(CORES) -k --no-print-directory --output-sync=target qa-exec NO_PROGRESS=true

.PHONY: qa-exec
qa-exec: phpstan \
    phplint \
    composer-require-checker \
    phpcs

The QA tools

phpcs and phpcbf

phpcs is the CLI tool of the style checker squizlabs/PHP_CodeSniffer. It also comes with phpcbf - a tool to automatically fix style errors.

Installation via composer:

make composer ARGS="require --dev squizlabs/php_codesniffer"

For now we will simply use the pre-configured ruleset for PSR-12: Extended Coding Style. When run in the application container for the first time on the app directory via

vendor/bin/phpcs --standard=PSR12 --parallel=4 -p app

i.e.

--standard=PSR12 => use the PSR12 ruleset
--parallel=4     => run with 4 parallel processes
-p               => show the progress

we get the following result:

root:/var/www/app# vendor/bin/phpcs --standard=PSR12 --parallel=4 -p app

FILE: /var/www/app/app/Console/Kernel.php
----------------------------------------------------------------------
FOUND 2 ERRORS AFFECTING 1 LINE
----------------------------------------------------------------------
 28 | ERROR | [x] Expected at least 1 space before "."; 0 found
 28 | ERROR | [x] Expected at least 1 space after "."; 0 found
----------------------------------------------------------------------
PHPCBF CAN FIX THE 2 MARKED SNIFF VIOLATIONS AUTOMATICALLY
----------------------------------------------------------------------


FILE: /var/www/app/app/Http/Controllers/HomeController.php
----------------------------------------------------------------------
FOUND 4 ERRORS AFFECTING 2 LINES
----------------------------------------------------------------------
 37 | ERROR | [x] Expected at least 1 space before "."; 0 found
 37 | ERROR | [x] Expected at least 1 space after "."; 0 found
 45 | ERROR | [x] Expected at least 1 space before "."; 0 found
 45 | ERROR | [x] Expected at least 1 space after "."; 0 found
----------------------------------------------------------------------
PHPCBF CAN FIX THE 4 MARKED SNIFF VIOLATIONS AUTOMATICALLY
----------------------------------------------------------------------


FILE: /var/www/app/app/Jobs/InsertInDbJob.php
-------------------------------------------------------------------------------
FOUND 1 ERROR AFFECTING 1 LINE
-------------------------------------------------------------------------------
 13 | ERROR | [x] Each imported trait must have its own "use" import statement
-------------------------------------------------------------------------------
PHPCBF CAN FIX THE 1 MARKED SNIFF VIOLATIONS AUTOMATICALLY
-------------------------------------------------------------------------------


FILE: /var/www/app/app/Models/User.php
-------------------------------------------------------------------------------
FOUND 1 ERROR AFFECTING 1 LINE
-------------------------------------------------------------------------------
 13 | ERROR | [x] Each imported trait must have its own "use" import statement
-------------------------------------------------------------------------------
PHPCBF CAN FIX THE 1 MARKED SNIFF VIOLATIONS AUTOMATICALLY
-------------------------------------------------------------------------------

All errors can be fixed automatically with phpcbf:

root:/var/www/app# vendor/bin/phpcbf --standard=PSR12 --parallel=4 -p app

PHPCBF RESULT SUMMARY
-------------------------------------------------------------------------
FILE                                                     FIXED  REMAINING
-------------------------------------------------------------------------
/var/www/app/app/Console/Kernel.php                      2      0
/var/www/app/app/Http/Controllers/HomeController.php     4      0
/var/www/app/app/Jobs/InsertInDbJob.php                  1      0
/var/www/app/app/Models/User.php                         1      0
-------------------------------------------------------------------------
A TOTAL OF 8 ERRORS WERE FIXED IN 4 FILES
-------------------------------------------------------------------------

Time: 411ms; Memory: 8MB

and a follow-up run of phpcs doesn't show any more errors:

root:/var/www/app# vendor/bin/phpcs --standard=PSR12 --parallel=4 -p app
.................... 20 / 20 (100%)


Time: 289ms; Memory: 8MB

phpstan

phpstan is the CLI tool of the static code analyzer phpstan/phpstan (see also the full PHPStan documentation). It provides some default "levels" of increasing strictness to report potential bugs based on the AST of the analyzed PHP code.

Installation via composer:

make composer ARGS="require --dev phpstan/phpstan"

Since this is a "fresh" codebase with very little code let's go for the highest level 9 (as of 2022-04-24) and run it in the application container on the app and tests directories via:

vendor/bin/phpstan analyse app tests --level=9

--level=9        => use level 9

root:/var/www/app# vendor/bin/phpstan analyse app tests --level=9
 25/25 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100%

 ------ ------------------------------------------------------------------------------------------------------------------
  Line   app/Commands/SetupDbCommand.php
 ------ ------------------------------------------------------------------------------------------------------------------
  22     Method App\Commands\SetupDbCommand::getOptions() return type has no value type specified in iterable type array.
         � See: https://phpstan.org/blog/solving-phpstan-no-value-type-specified-in-iterable-type
  34     Method App\Commands\SetupDbCommand::handle() has no return type specified.
 ------ ------------------------------------------------------------------------------------------------------------------

 ------ -------------------------------------------------------------------------------------------------------
  Line   app/Http/Controllers/HomeController.php
 ------ -------------------------------------------------------------------------------------------------------
  22     Parameter #1 $jobId of class App\Jobs\InsertInDbJob constructor expects string, mixed given.
  25     Part $jobId (mixed) of encapsed string cannot be cast to string.
  35     Call to an undefined method Illuminate\Redis\Connections\Connection::lRange().
  62     Call to an undefined method Illuminate\Contracts\View\Factory|Illuminate\Contracts\View\View::with().
 ------ -------------------------------------------------------------------------------------------------------

 ------ ------------------------------------------------------------------------------------------------------------------
  Line   app/Http/Middleware/Authenticate.php
 ------ ------------------------------------------------------------------------------------------------------------------
  17     Method App\Http\Middleware\Authenticate::redirectTo() should return string|null but return statement is missing.
 ------ ------------------------------------------------------------------------------------------------------------------

 ------ ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  Line   app/Http/Middleware/RedirectIfAuthenticated.php
 ------ ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  26     Method App\Http\Middleware\RedirectIfAuthenticated::handle() should return Illuminate\Http\RedirectResponse|Illuminate\Http\Response but returns Illuminate\Http\RedirectResponse|Illuminate\Routing\Redirector.
 ------ ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

 ------ -----------------------------------------------------------------------
  Line   app/Jobs/InsertInDbJob.php
 ------ -----------------------------------------------------------------------
  22     Method App\Jobs\InsertInDbJob::handle() has no return type specified.
 ------ -----------------------------------------------------------------------

 ------ -------------------------------------------------
  Line   app/Providers/RouteServiceProvider.php
 ------ -------------------------------------------------
  36     PHPDoc tag @var above a method has no effect.
  36     PHPDoc tag @var does not specify variable name.
  60     Cannot access property $id on mixed.
 ------ -------------------------------------------------

 ------ ----------------------------------------------------------------------------------------------------------------------------------------------------------
  Line   tests/Feature/App/Http/Controllers/HomeControllerTest.php
 ------ ----------------------------------------------------------------------------------------------------------------------------------------------------------
  24     Method Tests\Feature\App\Http\Controllers\HomeControllerTest::test___invoke() has parameter $params with no value type specified in iterable type array.
         � See: https://phpstan.org/blog/solving-phpstan-no-value-type-specified-in-iterable-type
  38     Method Tests\Feature\App\Http\Controllers\HomeControllerTest::__invoke_dataProvider() return type has no value type specified in iterable type array.
         � See: https://phpstan.org/blog/solving-phpstan-no-value-type-specified-in-iterable-type
 ------ ----------------------------------------------------------------------------------------------------------------------------------------------------------

 ------ ---------------------------------------------------------------------------------------------------------------------
  Line   tests/TestCase.php
 ------ ---------------------------------------------------------------------------------------------------------------------
  68     Cannot access offset 'database' on mixed.
  71     Parameter #1 $config of method Illuminate\Database\Connectors\MySqlConnector::connect() expects array, mixed given.
 ------ ---------------------------------------------------------------------------------------------------------------------


 [ERROR] Found 16 errors

After fixing (or ignoring :P) all errors, we now get

root:/var/www/app# vendor/bin/phpstan analyse app tests --level=9
25/25 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100%

[OK] No errors

php-parallel-lint

php-parallel-lint is the CLI tool of the PHP code linter php-parallel-lint/PHP-Parallel-Lint. It ensures that all PHP files are syntactically correct.

Installation via composer:

make composer ARGS="require --dev php-parallel-lint/php-parallel-lint"

"Parallel" is already in the name, so we run it on the full codebase ./ with 4 parallel processes and exclude the .git and vendor directories to speed up the execution via

vendor/bin/parallel-lint -j 4 --exclude .git --exclude vendor ./

i.e.

-j 4                              => use 4 parallel processes
--exclude .git --exclude vendor   => ignore the .git/ and vendor/ directories

we get

root:/var/www/app# vendor/bin/parallel-lint -j 4 --exclude .git --exclude vendor ./
PHP 8.1.1 | 4 parallel jobs
............................................................ 60/61 (98 %)
.                                                            61/61 (100 %)


Checked 61 files in 0.2 seconds
No syntax error found

No further TODOs here.

composer-require-checker

composer-require-checker is the CLI tool of the dependency checker maglnet/ComposerRequireChecker. The tool ensures that the composer.json file contains all dependencies that are used in the codebase.

Installation via composer:

make composer ARGS="require --dev maglnet/composer-require-checker"

Run it via

vendor/bin/composer-require-checker check

root:/var/www/app# vendor/bin/composer-require-checker check
ComposerRequireChecker 4.0.0@baa11a4e9e5072117e3d180ef16c07036cafa4a2
The following 1 unknown symbols were found:
+---------------------------------------------+--------------------+
| Unknown Symbol                              | Guessed Dependency |
+---------------------------------------------+--------------------+
| Symfony\Component\Console\Input\InputOption |                    |
+---------------------------------------------+--------------------+

What's going on here?

We use Symfony\Component\Console\Input\InputOption in our \App\Commands\SetupDbCommand and the code doesn't "fail" because InputOption is defined in thesymfony/console package that is a transitive dependency of laravel/framework, see the laravel/framework composer.json file.

I.e. the symfony/console package does actually exist in our vendor directory - but since we also use it as a first-party-dependency directly in our code, we must declare the dependency explicitly. Otherwise, Laravel might at some point decide to drop symfony/console and we would be left with broken code.

To fix this, I run

make composer ARGS="require symfony/console"

which will update the composer.json file and add the dependency. Running composer-require-checker again will now yield no further errors.

root:/var/www/app# vendor/bin/composer-require-checker check
ComposerRequireChecker 4.0.0@baa11a4e9e5072117e3d180ef16c07036cafa4a2
There were no unknown symbols found.

Additional tools (out of scope)

In general, I'm a huge fan of code quality tools and we use them quite extensively. At some point I'll probably dedicate a whole article to go over them in detail - but for now I'm just gonna leave a list for inspiration:

• brianium/paratest
  • Running PhpUnit tests in parallel
• malukenho/mcbumpface
  • Update the versions in the composer.json file after an update
• qossmic/deptrac-shim
  • A shim for qossmic/deptrac: A tool to define dependency layers based on e.g. namespaces
• icanhazstring/composer-unused
  • Show dependencies in the composer.json that are not used in the codebase
• roave/security-advisories
  • Gives a warning when packages with known vulnerabilities are used
  • Alternative: local-php-security-checker

QA make targets

You might have noticed that all tools have their own configuration options. Instead of remembering each of them, I'll define corresponding make targets in .make/01-02-application-qa.mk. The easiest way to do so would be to "hard-code" the exact commands that I ran previously, e.g.

.PHONY: phpstan
phpstan:  ## Run static analyzer on all application and test files 
    @$(EXECUTE_IN_APPLICATION_CONTAINER) vendor/bin/phpstan analyse app tests --level=9

(Please refer to the Run commands in the docker containers section in the previous tutorial for an explanation of the EXECUTE_IN_APPLICATION_CONTAINER variable).

However, this implementation is quite inflexible: What if we want to check a single file or try out other options? So let's create some variables instead:

PHPSTAN_CMD=php vendor/bin/phpstan analyse
PHPSTAN_ARGS=--level=9
PHPSTAN_FILES=$(APP_FILES) $(TEST_FILES)

.PHONY: phpstan
phpstan: ## Run static analyzer on all application and test files 
    @$(EXECUTE_IN_APPLICATION_CONTAINER) $(PHPSTAN_CMD) $(PHPSTAN_ARGS) $(PHPSTAN_FILES) 

This target allows me to override the defaults and e.g. check only the file app/Commands/SetupDbCommand.php with --level=1

make phpstan PHPSTAN_FILES=app/Commands/SetupDbCommand.php PHPSTAN_ARGS="--level=1" 

$ make phpstan PHPSTAN_FILES=app/Commands/SetupDbCommand.php PHPSTAN_ARGS="--level=1" 
 1/1 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100%


 [OK] No errors

The remaining tool variables can be configured in the exact same way:

# constants
 ## files
ALL_FILES=./
APP_FILES=app/
TEST_FILES=tests/

# Tool CLI config
PHPUNIT_CMD=php vendor/bin/phpunit
PHPUNIT_ARGS= -c phpunit.xml
PHPUNIT_FILES=
PHPSTAN_CMD=php vendor/bin/phpstan analyse
PHPSTAN_ARGS=--level=9
PHPSTAN_FILES=$(APP_FILES) $(TEST_FILES)
PHPCS_CMD=php vendor/bin/phpcs
PHPCS_ARGS=--parallel=$(CORES) --standard=psr12
PHPCS_FILES=$(APP_FILES)
PHPCBF_CMD=php vendor/bin/phpcbf
PHPCBF_ARGS=$(PHPCS_ARGS)
PHPCBF_FILES=$(PHPCS_FILES)
PARALLEL_LINT_CMD=php vendor/bin/parallel-lint
PARALLEL_LINT_ARGS=-j 4 --exclude vendor/ --exclude .docker --exclude .git
PARALLEL_LINT_FILES=$(ALL_FILES)
COMPOSER_REQUIRE_CHECKER_CMD=php vendor/bin/composer-require-checker
COMPOSER_REQUIRE_CHECKER_ARGS=--ignore-parse-errors

The qa target

From a workflow perspective I usually want to run all configured qa tools instead of each one individually (being able to run individually is still great if a tool fails, though).

A trivial approach would be a new target that uses all individual tool targets as preconditions:

.PHONY: qa
qa: phpstan \
    phplint \
    composer-require-checker \
    phpcs

But we can do better, because this target produces quite a noisy output:

$ make qa
 25/25 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100%


 [OK] No errors

PHP 8.1.1 | 4 parallel jobs
............................................................ 60/61 (98 %)
.                                                            61/61 (100 %)


Checked 61 files in 0.3 seconds
No syntax error found
ComposerRequireChecker 4.0.0@baa11a4e9e5072117e3d180ef16c07036cafa4a2
There were no unknown symbols found.
.................... 20 / 20 (100%)


Time: 576ms; Memory: 8MB

I'd rather have something like this:

$ make qa
phplint                             done   took 1s
phpcs                               done   took 1s
phpstan                             done   took 3s
composer-require-checker            done   took 6s

The execute "function"

We'll make this work by suppressing the tool output and using a user-defined execute make function to format all targets nicely.

Though "function" isn't quite correct here, because it's rather a multiline variable defined via define ... endef that is then "invoked" via the call function.

# File: 01-02-application-qa.mk

# call with NO_PROGRESS=true to hide tool progress (makes sense when invoking multiple tools together)
NO_PROGRESS?=false
ifeq ($(NO_PROGRESS),true)
    PHPSTAN_ARGS+= --no-progress
endif


# Use NO_PROGRESS=false when running individual tools.
# On  NO_PROGRESS=true  the corresponding tool has no output on success
#                       apart from its runtime but it will still print 
#                       any errors that occured. 
define execute
    if [ "$(NO_PROGRESS)" = "false" ]; then \
        eval "$(EXECUTE_IN_APPLICATION_CONTAINER) $(1) $(2) $(3) $(4)"; \
    else \
        START=$$(date +%s); \
        printf "%-35s" "$@"; \
        if OUTPUT=$$(eval "$(EXECUTE_IN_APPLICATION_CONTAINER) $(1) $(2) $(3) $(4)" 2>&1); then \
            printf " %-6s" "done"; \
            END=$$(date +%s); \
            RUNTIME=$$((END-START)) ;\
            printf " took $${RUNTIME}s\n"; \
        else \
            printf " %-6s" "fail"; \
            END=$$(date +%s); \
            RUNTIME=$$((END-START)) ;\
            printf " took $${RUNTIME}s\n"; \
            echo "$$OUTPUT"; \
            printf "\n"; \
            exit 1; \
        fi; \
    fi
endef

• the NO_PROGRESS variable is set to false by default and will cause a target to be invoked as before, showing all its output immediately
  • if the variable is set to true, the target is instead invoked via eval and the output is captured in the OUTPUT bash variable that will only be printed if the result of the invocation is faulty

The tool targets are then adjusted to use the new function.

.PHONY: phpstan
phpstan: ## Run static analyzer on all application and test files
    @$(call execute,$(PHPSTAN_CMD),$(PHPSTAN_ARGS),$(PHPSTAN_FILES),$(ARGS))

We can now call the phpstan target with NO_PROGRESS=true like so:

$ make phpstan NO_PROGRESS=true
phpstan                             done   took 4s

An "error" would look likes this:

$ make phpstan NO_PROGRESS=true
phpstan                             fail   took 9s
 ------ ----------------------------------------
  Line   app/Providers/RouteServiceProvider.php
 ------ ----------------------------------------
  49     Cannot access property $id on mixed.
 ------ ----------------------------------------

Parallel execution and a helper target

Technically, this also already works with the qa target and we can even speed up the process by running the tools in parallel with the -j flag for "Parallel Execution"

$ make -j 4 qa NO_PROGRESS=true
phpstan                            phplint                            composer-require-checker           phpcs                               done   took 5s
done   took 5s
done   took 7s
done   took 10s

Well... not quite what we wanted. We also need to use --output-sync=target to controll the "Output During Parallel Execution"

$ make -j 4 --output-sync=target qa NO_PROGRESS=true
phpstan                             done   took 3s
phplint                             done   took 1s
composer-require-checker            done   took 5s
phpcs                               done   took 1s

Since this is quite a mouthful to type, we'll use a helper target qa-exec for running the tools and put all the inconvenient-to-type options in the final qa target.

# File: 01-02-application-qa.mk
#...

# variables
CORES?=$(shell (nproc  || sysctl -n hw.ncpu) 2> /dev/null)

.PHONY: qa
qa: ## Run code quality tools on all files
    @"$(MAKE)" -j $(CORES) -k --no-print-directory --output-sync=target qa-exec NO_PROGRESS=true

.PHONY: qa-exec
qa-exec: phpstan \
    phplint \
    composer-require-checker \
    phpcs

For the number of parallel processes I use nproc (works on Linux and Windows) resp. sysctl -n hw.ncpu (works on Mac) to determine the number of available cores. If you dedicate less resources to docker you might want to hard-code this setting to a lower value (e.g. by adding a CORES variable in the .make/.env file).

Sprinkle some color on top

The final piece for getting to the output mentioned in the Introduction is the bash-coloring:

To make this work, we need to understand first how colors work in bash:

This [coloring] can be accomplished by adding a \e [or \033] at the beginning to form an escape sequence. The escape sequence for specifying color codes is \e[COLORm (COLOR represents our (numeric) color code in this case).

(via Adding colors to Bash scripts)

E.g. the following script will print a green text:

printf "\033[0;32mThis text is green\033[0m"

So we define the required colors as variables and use them in the corresponding places in the execute function:

 ## bash colors
RED:=\033[0;31m
GREEN:=\033[0;32m
YELLOW:=\033[0;33m
NO_COLOR:=\033[0m

# ...

# Use NO_PROGRESS=false when running individual tools.
# On  NO_PROGRESS=true  the corresponding tool has no output on success
#                       apart from its runtime but it will still print 
#                       any errors that occured. 
define execute
    if [ "$(NO_PROGRESS)" = "false" ]; then \
        eval "$(EXECUTE_IN_APPLICATION_CONTAINER) $(1) $(2) $(3) $(4)"; \
    else \
        START=$$(date +%s); \
        printf "%-35s" "$@"; \
        if OUTPUT=$$(eval "$(EXECUTE_IN_APPLICATION_CONTAINER) $(1) $(2) $(3) $(4)" 2>&1); then \
            printf " $(GREEN)%-6s$(NO_COLOR)" "done"; \
            END=$$(date +%s); \
            RUNTIME=$$((END-START)) ;\
            printf " took $(YELLOW)$${RUNTIME}s$(NO_COLOR)\n"; \
        else \
            printf " $(RED)%-6s$(NO_COLOR)" "fail"; \
            END=$$(date +%s); \
            RUNTIME=$$((END-START)) ;\
            printf " took $(YELLOW)$${RUNTIME}s$(NO_COLOR)\n"; \
            echo "$$OUTPUT"; \
            printf "\n"; \
            exit 1; \
        fi; \
    fi
endef

Please note, that i did not include the tests in the qa target. I like to run those separately, because our tests usually take a lot longer to execute. So in my day-to-day work I would run make qa and make test to ensure that code quality and tests are passing:

$ make qa
phplint                             done   took 1s
phpcs                               done   took 1s
composer-require-checker            done   took 14s
phpstan                             done   took 16s

$ make test
PHPUnit 9.5.19 #StandWithUkraine

.......                                                             7 / 7 (100%)

Time: 00:03.772, Memory: 28.00 MB

OK (7 tests, 13 assertions)

Further updates in the codebase

I've also cleaned up the codebase a little in branch part-5-php-qa-tools-make-docker and even though those changes have nothing to todo with "QA tools" I didn't want to leave them unnoticed:

• removing unnecessary files (.styleci.yml, package.json, webpack.mix.js)

  • removing unused values from the .env.example file
  • run a composer update to get the latest Laravel version
  • add a show-help script to the scripts section of the composer.json file that references the Makefile (see also this discussion on Twitter)

  
  {
      "scripts": {
          "show-help": [
              "make"
          ]
      },
      "scripts-descriptions": {
          "show-help": "Display available 'make' commands (we use make instead of composer scripts)."
      }
  }
  

  

  • replace docker-compose with docker compose to use compose v2

For some reason, the last point caused some trouble because Linux and Docker Desktop for Windows require a -T flag for the exec command to disable a TTY allocation in some cases. Whereas on Docker Desktop for Mac the missing TTY lead to a cluttered output ("staircase effect").

Thus I modified the Makefile to populate a DOCKER_COMPOSE_EXEC_OPTIONS variable based on the OS

# Add the -T options to "docker compose exec" to avoid the 
# "panic: the handle is invalid"
# error on Windows and Linux 
# @see https://stackoverflow.com/a/70856332/413531
DOCKER_COMPOSE_EXEC_OPTIONS=-T

# OS is defined for WIN systems, so "uname" will not be executed
OS?=$(shell uname)
ifeq ($(OS),Windows_NT)
    # Windows requires the .exe extension, otherwise the entry is ignored
    # @see https://stackoverflow.com/a/60318554/413531
    SHELL := bash.exe
else ifeq ($(OS),Darwin)
    # On Mac, the -T must be omitted to avoid cluttered output
    # @see https://github.com/moby/moby/issues/37366#issuecomment-401157643
    DOCKER_COMPOSE_EXEC_OPTIONS=
endif

And use the variable when defining EXECUTE_IN_*_CONTAINER in .make/02-00-docker.mk

ifeq ($(EXECUTE_IN_CONTAINER),true)
    EXECUTE_IN_ANY_CONTAINER:=$(DOCKER_COMPOSE) exec $(DOCKER_COMPOSE_EXEC_OPTIONS) --user $(APP_USER_NAME) $(DOCKER_SERVICE_NAME)
    EXECUTE_IN_APPLICATION_CONTAINER:=$(DOCKER_COMPOSE) exec $(DOCKER_COMPOSE_EXEC_OPTIONS) --user $(APP_USER_NAME) $(DOCKER_SERVICE_NAME_APPLICATION)
    EXECUTE_IN_WORKER_CONTAINER:=$(DOCKER_COMPOSE) exec $(DOCKER_COMPOSE_EXEC_OPTIONS) --user $(APP_USER_NAME) $(DOCKER_SERVICE_NAME_PHP_WORKER)
endif

Wrapping up

Congratulations, you made it! If some things are not completely clear by now, don't hesitate to leave a comment. You should now have a blueprint for adding code quality tools for your dockerized application and way to conveniently control them through a Makefile.

In the next part of this tutorial, we will set up git secret to encrypt secret values and store them directly in the git repository.

Please subscribe to the RSS feed or via email to get automatic notifications when this next part comes out :)

All code samples are publicly available in my Docker PHP Tutorial repository on Github.
You find the branch for this tutorial at part-4-3-run-laravel-9-docker-in-2022.

All published parts of the Docker PHP Tutorial are collected under a dedicated page at Docker PHP Tutorial. The previous part was PhpStorm, Docker and Xdebug 3 on PHP 8.1 in 2022 and the following one is Set up PHP QA tools and control them via make.

If you want to follow along, please subscribe to the RSS feed or via email to get automatic notifications when the next part comes out :)

Table of contents

• Introduction
• Install extensions
• Install Laravel
• Update the PHP POC
  • config
    • database connection
    • queue connection
  • Controllers
  • Commands
  • Jobs and workers
  • Tests
• Makefile updates
  • Clearing the queue
• Running the POC
• Wrapping up

Introduction

The goal of this tutorial is to run the PHP POC from part 4.1 using Laravel as a framework instead of "plain PHP". We'll use the newest version of Laravel (Laravel 9) that was released at the beginning of February 2022.

If you want to follow along, please use the branch of the previous tutorial, i.e.

git checkout git checkout part-4-2-phpstorm-docker-xdebug-3-php-8-1-in-2022

The branch of this tutorials contains the "end result", i.e. Laravel will already exist in the repository.

Install extensions

Before Laravel can be installed, we need to add the necessary extensions of the framework (and all its dependencies) to the php-base image:

# File: .docker/images/php/base/Dockerfile

# ...

RUN apk add --update --no-cache  \
        php-curl~=${TARGET_PHP_VERSION} \

Install Laravel

We'll start by creating a new Laravel project with composer

composer create-project --prefer-dist laravel/laravel /tmp/laravel "9.0.*" --no-install --no-scripts

The files are added to /tmp/laravel because composer projects cannot be created in non-empty folders , so we need to create the project in a temporary location first and move it afterwards.

Since I don't have PHP 8 installed on my laptop, I'll execute the command in the application docker container via

make execute-in-container DOCKER_SERVICE_NAME="application" COMMAND='composer create-project --prefer-dist laravel/laravel /tmp/laravel "9.0.*" --no-install --no-scripts'

and then move the files into the application directory via

rm -rf public/ tests/ composer.* phpunit.xml
make execute-in-container DOCKER_SERVICE_NAME="application" COMMAND="bash -c 'mv -n /tmp/laravel/{.*,*} .' && rm -f /tmp/laravel"
cp .env.example .env

Notes

• composer install is skipped via --no-install to avoid having to copy over the vendor/ folder (which is super slow on Docker Desktop)
• existing directories cannot be overwritten by mv thus I remove public/ and tests/ upfront (as well as the composer and phpunit config files)
• mv uses the -n flag so that existing files like our .editorconfig are not overwritten
• I need to use bash -c to run the command in the container because otherwise the * wildcard would have no effect in the container

To finalize the installation I need to install the composer dependencies and execute the create-project scripts defined in composer.json:

make composer ARGS=install
make composer ARGS="run-script post-create-project-cmd"

Since our nginx configuration was already pointing to the public/ directory, we can immediately open http://127.0.0.1 in the browser and should see the frontpage of a fresh Laravel installation.

Update the PHP POC

config

We need to update the connection information for the database and the queue (previously configured via dependencies.php) in the .env file

database connection

DB_CONNECTION=mysql
DB_HOST=mysql
DB_PORT=3306
DB_DATABASE=application_db
DB_USERNAME=root
DB_PASSWORD=secret_mysql_root_password

queue connection

QUEUE_CONNECTION=redis

REDIS_HOST=redis
REDIS_PASSWORD=secret_redis_password

Controllers

The functionality of the previous public/index.php file now lives in the HomeController at app/Http/Controllers/HomeController.php

class HomeController extends Controller
{
    use DispatchesJobs;

    public function __invoke(Request $request, QueueManager $queueManager, DatabaseManager $databaseManager): View
    {
        $jobId = $request->input("dispatch") ?? null;
        if ($jobId !== null) {
            $job = new InsertInDbJob($jobId);
            $this->dispatch($job);

            return $this->getView("Adding item '$jobId' to queue");
        }

        if ($request->has("queue")) {

            /**
             * @var RedisQueue $redisQueue
             */
            $redisQueue = $queueManager->connection();
            $redis =  $redisQueue->getRedis()->connection();
            $queueItems = $redis->lRange("queues:default", 0, 99999);

            $content = "Items in queue\n".var_export($queueItems, true);

            return $this->getView($content);
        }

        if ($request->has("db")) {
            $items = $databaseManager->select($databaseManager->raw("SELECT * FROM jobs"));

            $content = "Items in db\n".var_export($items, true);

            return $this->getView($content);
        }
        $content = <<<HTML
            <ul>
                <li><a href="?dispatch=foo">Dispatch job 'foo' to the queue.</a></li>
                <li><a href="?queue">Show the queue.</a></li>
                <li><a href="?db">Show the DB.</a></li>
            </ul>
            HTML;

        return $this->getView($content);
    }

    private function getView(string $content): View
    {
        return view('home')->with(["content" => $content]);
    }
}

Its content is displayed via the home view located at resources/views/home.blade.php:

<!DOCTYPE html>
<html>
    <head>
        <meta charset="utf-8">
    </head>
    <body>
    {!! $content !!}
    </body>
</html>

The controller is added as a route in routes/web.php:

Route::get('/', \App\Http\Controllers\HomeController::class)->name("home");

Commands

We will replace the setup.php script with a SetupDbCommand that is located at app/Commands/SetupDbCommand.php

class SetupDbCommand extends Command
{
    /**
     * @var string
     */
    protected $name = "app:setup-db";

    /**
     * @var string
     */
    protected $description = "Run the application database setup";

    protected function getOptions(): array
    {
        return [
            [
                "drop",
                null,
                InputOption::VALUE_NONE,
                "If given, the existing database tables are dropped and recreated.",
            ],
        ];
    }

    public function handle()
    {
        $drop = $this->option("drop");
        if ($drop) {
            $this->info("Dropping all database tables...");

            $this->call(WipeCommand::class);

            $this->info("Done.");
        }

        $this->info("Running database migrations...");

        $this->call(MigrateCommand::class);

        $this->info("Done.");
    }
}

Register it the AppServiceProvider in app/Providers/AppServiceProvider.php

    public function register()
    {
        $this->commands([
            \App\Commands\SetupDbCommand::class
        ]);
    }

and update the setup-db target in .make/01-00-application-setup.mk to run the artisan Command

.PHONY: setup-db
setup-db: ## Setup the DB tables
    $(EXECUTE_IN_APPLICATION_CONTAINER) php artisan app:setup-db $(ARGS);

We will also create a migration for the jobs table in database/migrations/2022_02_10_000000_create_jobs_table.php:

return new class extends Migration
{
    public function up(): void
    {
        Schema::create('jobs', function (Blueprint $table) {
            $table->id();
            $table->string('value');
        });
    }
};

Jobs and workers

We will replace the worker.php script with InsertInDbJob located at app/Jobs/InsertInDbJob.php

class InsertInDbJob implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable;

    public function __construct(
        public readonly string $jobId
    ) {
    }

    public function handle(DatabaseManager $databaseManager)
    {
        $databaseManager->insert("INSERT INTO `jobs`(value) VALUES(?)", [$this->jobId]);
    }
}

though this will "only" handle the insertion part. For the worker itself we will use the native \Illuminate\Queue\Console\WorkCommand via

php artisan queue:work

We need to adjust the .docker/images/php/worker/Dockerfile and change

ARG PHP_WORKER_COMMAND="php $APP_CODE_PATH/worker.php"

to

ARG PHP_WORKER_COMMAND="php $APP_CODE_PATH/artisan queue:work"

Since this change takes place directly in the Dockerfile, we must now rebuild the image

$ make docker-build-image DOCKER_SERVICE_NAME=php-worker

and restart it

$ make docker-up

Tests

I'd also like to take this opportunity to add a Feature test for the HomeController at tests/Feature/App/Http/Controllers/HomeControllerTest.php:

class HomeControllerTest extends TestCase
{
    public function setUp(): void
    {
        parent::setUp();

        $this->setupDatabase();
        $this->setupQueue();
    }

    /**
     * @dataProvider __invoke_dataProvider
     */
    public function test___invoke(array $params, string $expected): void
    {
        $urlGenerator = $this->getDependency(UrlGenerator::class);

        $url = $urlGenerator->route("home", $params);

        $response = $this->get($url);

        $response
            ->assertStatus(200)
            ->assertSee($expected, false)
        ;
    }

    public function __invoke_dataProvider(): array
    {
        return [
            "default"           => [
                "params"   => [],
                "expected" => <<<TEXT
                        <li><a href="?dispatch=foo">Dispatch job 'foo' to the queue.</a></li>
                        <li><a href="?queue">Show the queue.</a></li>
                        <li><a href="?db">Show the DB.</a></li>
                    TEXT
                ,
            ],
            "database is empty" => [
                "params"   => ["db"],
                "expected" => <<<TEXT
                        Items in db
                    array (
                    )
                    TEXT
                ,
            ],
            "queue is empty"    => [
                "params"   => ["queue"],
                "expected" => <<<TEXT
                        Items in queue
                    array (
                    )
                    TEXT
                ,
            ],
        ];
    }

    public function test_shows_existing_items_in_database(): void
    {
        $databaseManager = $this->getDependency(DatabaseManager::class);

        $databaseManager->insert("INSERT INTO `jobs` (id, value) VALUES(1, 'foo');");

        $urlGenerator = $this->getDependency(UrlGenerator::class);

        $params = ["db"];
        $url    = $urlGenerator->route("home", $params);

        $response = $this->get($url);

        $expected = <<<TEXT
                Items in db
            array (
              0 => 
              (object) array(
                 'id' => 1,
                 'value' => 'foo',
              ),
            )
            TEXT;

        $response
            ->assertStatus(200)
            ->assertSee($expected, false)
        ;
    }

    public function test_shows_existing_items_in_queue(): void
    {
        $queueManager = $this->getDependency(QueueManager::class);

        $job = new InsertInDbJob("foo");
        $queueManager->push($job);

        $urlGenerator = $this->getDependency(UrlGenerator::class);

        $params = ["queue"];
        $url    = $urlGenerator->route("home", $params);

        $response = $this->get($url);

        $expectedJobsCount = <<<TEXT
                Items in queue
            array (
              0 => '{
            TEXT;

        $expected = <<<TEXT
            \\\\"jobId\\\\";s:3:\\\\"foo\\\\";
            TEXT;

        $response
            ->assertStatus(200)
            ->assertSee($expectedJobsCount, false)
            ->assertSee($expected, false)
        ;
    }
}

The test checks the database as well as the queue and uses the helper methods $this->setupDatabase() and $this->setupQueue() that I defined in the base test case at tests/TestCase.php as follows

   /**
     * @template T
     * @param class-string<T> $className
     * @return T
     */
    protected function getDependency(string $className)
    {
        return $this->app->get($className);
    }

    protected function setupDatabase(): void
    {
        $databaseManager = $this->getDependency(DatabaseManager::class);

        $actualConnection  = $databaseManager->getDefaultConnection();
        $testingConnection = "testing";
        if ($actualConnection !== $testingConnection) {
            throw new RuntimeException("Database tests are only allowed to run on default connection '$testingConnection'. The current default connection is '$actualConnection'.");
        }

        $this->ensureDatabaseExists($databaseManager);

        $this->artisan(SetupDbCommand::class, ["--drop" => true]);
    }

    protected function setupQueue(): void
    {
        $queueManager = $this->getDependency(QueueManager::class);

        $actualDriver  = $queueManager->getDefaultDriver();
        $testingDriver = "testing";
        if ($actualDriver !== $testingDriver) {
            throw new RuntimeException("Queue tests are only allowed to run on default driver '$testingDriver'. The current default driver is '$actualDriver'.");
        }

        $this->artisan(ClearCommand::class);
    }

    protected function ensureDatabaseExists(DatabaseManager $databaseManager): void
    {
        $connection = $databaseManager->connection();

        try {
            $connection->getPdo();
        } catch (PDOException $e) {
            // e.g. SQLSTATE[HY000] [1049] Unknown database 'testing'
            if ($e->getCode() !== 1049) {
                throw $e;
            }
            $config             = $connection->getConfig();
            $config["database"] = "";

            $connector = new MySqlConnector();
            $pdo       = $connector->connect($config);
            $database  = $connection->getDatabaseName();
            $pdo->exec("CREATE DATABASE IF NOT EXISTS `{$database}`;");
        }
    }

The methods ensure that the tests are only executed if the proper database connection and queue driver is configured. This is done through environment variables and I like using a dedicated .env file located at .env.testing
for all testing ENV values instead of defining them in the phpunit.xml config file via <env> elements:

# File: .env.testing

DB_CONNECTION=testing
DB_DATABASE=testing
QUEUE_CONNECTION=testing
REDIS_DB=1000

The corresponding connections have to be configured in the config files

# File: config/database.php

return [
// ...
    'connections' => [
// ...
        'testing' => [
            'driver' => 'mysql',
            'url' => env('DATABASE_URL'),
            'host' => env('DB_HOST'),
            'port' => env('DB_PORT', '3306'),
            'database' => env('DB_DATABASE', 'testing'),
            'username' => env('DB_USERNAME'),
            'password' => env('DB_PASSWORD', ''),
            'unix_socket' => env('DB_SOCKET', ''),
            'charset' => 'utf8mb4',
            'collation' => 'utf8mb4_unicode_ci',
            'prefix' => '',
            'prefix_indexes' => true,
            'strict' => true,
            'engine' => null,
            'options' => extension_loaded('pdo_mysql') ? array_filter([
                PDO::MYSQL_ATTR_SSL_CA => env('MYSQL_ATTR_SSL_CA'),
            ]) : [],
        ],
    ],
// ...
    'redis' => [
// ...
        'testing' => [
            'url' => env('REDIS_URL'),
            'host' => env('REDIS_HOST', '127.0.0.1'),
            'password' => env('REDIS_PASSWORD'),
            'port' => env('REDIS_PORT', '6379'),
            'database' => env('REDIS_DB', '1000'),
        ],
    ],
];

# File: config/queue.php

return [
// ...

    'connections' => [
// ...
        'testing' => [
            'driver' => 'redis',
            'connection' => 'testing', // => refers to the "database.redis.testing" config entry
            'queue' => env('REDIS_QUEUE', 'default'),
            'retry_after' => 90,
            'block_for' => null,
            'after_commit' => false,
        ],
    ],
];

The tests can be executed via make test

$ make test
ENV=local TAG=latest DOCKER_REGISTRY=docker.io DOCKER_NAMESPACE=dofroscra APP_USER_NAME=application APP_GROUP_NAME=application docker-compose -p dofroscra_local --env-file ./.docker/.env -f ./.docker/docker-compose/docker-compose.yml -f ./.docker/docker-compose/docker-compose.local.yml exec -T --user application php-worker vendor/bin/phpunit -c phpunit.xml
PHPUnit 9.5.13 by Sebastian Bergmann and contributors.

.......                                                             7 / 7 (100%)

Time: 00:02.709, Memory: 28.00 MB

OK (7 tests, 13 assertions)

Makefile updates

Clearing the queue

For convenience while testing I added a make target to clear all items from the queue in .make/01-01-application-commands.mk

.PHONY: clear-queue
clear-queue: ## Clear the job queue
    $(EXECUTE_IN_APPLICATION_CONTAINER) php artisan queue:clear $(ARGS)

Running the POC

Since the POC only uses make targets and we basically just "refactored" them, there is no modification necessary to make the existing test.sh work:

$ bash test.sh


  Building the docker setup


//...


  Starting the docker setup


//...


  Clearing DB


ENV=local TAG=latest DOCKER_REGISTRY=docker.io DOCKER_NAMESPACE=dofroscra APP_USER_NAME=application APP_GROUP_NAME=application docker-compose -p dofroscra_local --env-file ./.docker/.env -f ./.docker/docker-compose/docker-compose.yml -f ./.docker/docker-compose/docker-compose.local.yml exec -T --user application application php artisan app:setup-db --drop;
Dropping all database tables...
Dropped all tables successfully.
Done.
Running database migrations...
Migration table created successfully.
Migrating: 2014_10_12_000000_create_users_table
Migrated:  2014_10_12_000000_create_users_table (64.04ms)
Migrating: 2014_10_12_100000_create_password_resets_table
Migrated:  2014_10_12_100000_create_password_resets_table (50.06ms)
Migrating: 2019_08_19_000000_create_failed_jobs_table
Migrated:  2019_08_19_000000_create_failed_jobs_table (58.61ms)
Migrating: 2019_12_14_000001_create_personal_access_tokens_table
Migrated:  2019_12_14_000001_create_personal_access_tokens_table (94.03ms)
Migrating: 2022_02_10_000000_create_jobs_table
Migrated:  2022_02_10_000000_create_jobs_table (31.85ms)
Done.


  Stopping workers


ENV=local TAG=latest DOCKER_REGISTRY=docker.io DOCKER_NAMESPACE=dofroscra APP_USER_NAME=application APP_GROUP_NAME=application docker-compose -p dofroscra_local --env-file ./.docker/.env -f ./.docker/docker-compose/docker-compose.yml -f ./.docker/docker-compose/docker-compose.local.yml exec -T --user application php-worker supervisorctl stop worker:*;
worker:worker_00: stopped
worker:worker_01: stopped
worker:worker_02: stopped
worker:worker_03: stopped


  Ensuring that queue and db are empty


<!DOCTYPE html>
<html>
    <head>
        <meta charset="utf-8">
    </head>
    <body>
    Items in queue
array (
)
    </body>
</html>
<!DOCTYPE html>
<html>
    <head>
        <meta charset="utf-8">
    </head>
    <body>
    Items in db
array (
)
    </body>
</html>


  Dispatching a job 'foo'


<!DOCTYPE html>
<html>
    <head>
        <meta charset="utf-8">
    </head>
    <body>
    Adding item 'foo' to queue
    </body>
</html>


  Asserting the job 'foo' is on the queue


<!DOCTYPE html>
<html>
    <head>
        <meta charset="utf-8">
    </head>
    <body>
    Items in queue
array (
  0 => '{"uuid":"7ea63590-2a86-4739-abf8-8a059d41bd60","displayName":"App\\\\Jobs\\\\InsertInDbJob","job":"Illuminate\\\\Queue\\\\CallQueuedHandler@call","maxTries":null,"maxExceptions":null,"failOnTimeout":false,"backoff":null,"timeout":null,"retryUntil":null,"data":{"commandName":"App\\\\Jobs\\\\InsertInDbJob","command":"O:22:\\"App\\\\Jobs\\\\InsertInDbJob\\":11:{s:5:\\"jobId\\";s:3:\\"foo\\";s:3:\\"job\\";N;s:10:\\"connection\\";N;s:5:\\"queue\\";N;s:15:\\"chainConnection\\";N;s:10:\\"chainQueue\\";N;s:19:\\"chainCatchCallbacks\\";N;s:5:\\"delay\\";N;s:11:\\"afterCommit\\";N;s:10:\\"middleware\\";a:0:{}s:7:\\"chained\\";a:0:{}}"},"id":"I3k5PNyGZc6Z5XWCC4gt0qtSdqUZ84FU","attempts":0}',
)
    </body>
</html>


  Starting the workers


ENV=local TAG=latest DOCKER_REGISTRY=docker.io DOCKER_NAMESPACE=dofroscra APP_USER_NAME=application APP_GROUP_NAME=application docker-compose -p dofroscra_local --env-file ./.docker/.env -f ./.docker/docker-compose/docker-compose.yml -f ./.docker/docker-compose/docker-compose.local.yml exec -T --user application php-worker supervisorctl start worker:*;
worker:worker_00: started
worker:worker_01: started
worker:worker_02: started
worker:worker_03: started


  Asserting the queue is now empty


<!DOCTYPE html>
<html>
    <head>
        <meta charset="utf-8">
    </head>
    <body>
    Items in queue
array (
)
    </body>
</html>


  Asserting the db now contains the job 'foo'


<!DOCTYPE html>
<html>
    <head>
        <meta charset="utf-8">
    </head>
    <body>
    Items in db
array (
  0 =>
  (object) array(
     'id' => 1,
     'value' => 'foo',
  ),
)
    </body>
</html>

Wrapping up

Congratulations, you made it! If some things are not completely clear by now, don't hesitate to leave a comment. Laravel 9 should now be up and running on the previously set up docker infrastructure.

In the next part of this tutorial, we will Set up PHP QA tools and control them via make.

Please subscribe to the RSS feed or via email to get automatic notifications when this next part comes out :)