How to use GCP Redis Memorystore instances

In this blog post I'll summarize my experience with GCP Redis Memorystore instances. Memorystore is the managed in-memory datastore solution from Google Cloud Platform and was mentioned in Deploy dockerized PHP Apps to production on GCP via docker compose as a POC as the "better" way to deal with in-memory datastores in a dockerized application (compared to running an in-memory datastore via docker).

Table of contents

• Setup Memorystore
• Create a new redis instance
  • redis AUTH and in-transit encryption
• Connecting to a Redis Memorystore instance
  • Redis Memorystore offers private IP connectivity only
  • Connecting to the redis instance from a Compute Instance VM
  • Connecting to the redis instance via SSH tunnel
• Delete a Redis Memorystore instance
• Using the gcloud CLI
  • Activate the service account
  • Enable the necessary APIs
  • Create an IP range allocation
  • Create the VPC peering with servicenetworking.googleapis.com
  • Create the redis instance
• Wrapping up

Setup Memorystore

The managed solution for in-memory datastores from GCP is called Memorystore and provides multiple datastore technologies - including redis. In the Cloud Console UI it is managed via the Memorystore UI that allows us to create and manage instances.

Create a new redis instance

To get started, we need to enable the following APIs:

• Compute Engine API
• Google Cloud Memorystore for Redis API

Creating a new instance from the Create a redis instance UI is pretty straight forward and well documented in the GCP Redis Guide: Creating and managing Redis instances.

We'll use the following settings:

• Tier Selection: For testing purposes, I recommend choosing the "Basic" option (this will also disable the "Read Replicas")
• Capacity: Enter "1"
• Set up connection > Network: Select the network that the VMs are located in - default in my case
• Additional Configurations > Connections: Select option "Private service access" here as it's the recommended approach.
  • CAUTION: In order to use "Private service access" as connectivity mode, we need to create a reserved IP allocation and a VPC peering with the Google Cloud Platform Service Producer first. The process is exactly the same as I've explained in my MySQL Cloud SQL article for connecting via private IP. Please make sure to pay close attention to section Concrete steps to connect via private IP and especially the video, as it shows the necessary steps to enable the "Private service access" for Memorystore!
• Security > Enable AUTH: Enable the checkbox
  • Note: This will auto-enable the checkbox "Enable in-transit encryption" though I recommend disabling that checkbox (see section redis AUTH and in-transit encryption)
• Configuration > Version: Select "6.x" (which is currently [2023-04-17] the latest version)

FYI: Unfortunately, there is no "EQUIVALENT COMMAND LINE" button as it was the case when creating the Compute Instance - which would have come in handy for creating the instance via gcloud cli.

Once everything is configured, click the "Create Instance" button. The actual creation can take quite some time (I've experienced times from a couple of minutes to ~15 min).

redis AUTH and in-transit encryption

During instance creation we activated the AUTH feature but disabled the in-transit encryption on purpose. Since this goes against GCP's own recommendation

When AUTH is enabled, in-transit encryption is recommended so credentials are confidential when transmitted.

I'd like to provide some thoughts on my reasoning:

According to the GCP Redis Guide: AUTH feature overview > Security and privacy, AUTH is not meant to be used as a security measure:

AUTH helps you ensure that known entities in your organization do not unintentionally access and modify your Redis instance. AUTH does not provide security during data transportation. Also, AUTH does not protect your instance against any malicious entities that have access to your VPC network.

However, it is still certainly "better than not having AUTH at all". Now, in-transit encryption comes at a cost: All communication between redis and the VMs would now be encrypted. Even though this sounds good in theory, it has a negative impact on performance as well as on the maximum number of possible connections.

Thus, I'll go with an in-between solution: Enable AUTH but disable in-transit encryption

Here are some additional things I learned about AUTH:

• you cannot define a custom AUTH string, but it will always be an auto-generated UUID
• the AUTH string will be shown in plain text in the management UI of a redis instance

• you can get the AUTH string via gcloud redis instances get-auth-string

  $ gcloud redis instances get-auth-string redis-instance --region=us-central1
  authString: 568d20ec-b0c2-40a9-908d-a5d6b6717a9c
  

See also the GCP Redis Guides on

• Managing Redis AUTH
• AUTH feature overview > AUTH behavior

Connecting to a Redis Memorystore instance

I'll explain 2 different ways of connecting to a Redis Memorystore instance:

• from a Compute Instance VM on GCP
• "locally" from your laptop via SSH Tunnel

Unfortunately, there is no "One-Click-Solution" like accessing MySQL Cloud SQL instances via Cloud Shell available for Redis.

As always, GCP has also an extensive documentation on the various connection methods available at the GCP Redis Guide: Connecting to a Redis instance.

Redis Memorystore offers private IP connectivity only

Redis Memorystore instances do not have a public IP address!

Regardless of the connection mode, Memorystore for Redis always uses internal IP addresses to provision Redis instances.

(via GCP Redis Guide: Networking)

I.e. it's not possible to connect to an instance without being in the same VPC. This is different from e.g. MySQL Cloud SQL instances, that offer public IPs as an option.

Instead, GCP offers two so-called "Connection modes" for private IP access:

• Direct peering
• Private services access

As Private services access is the newer (and recommended) approach, we'll not cover Direct peering in more detail (even though it's a little easier to set up, because GCP will create the necessary VPC peering automatically).

Connecting to the redis instance from a Compute Instance VM

Once the redis instance is up and running, we can find its private IP address (and port) via its management UI at URL

https://console.cloud.google.com/memorystore/redis/locations/$region/instances/$instanceName/details/overview

# e.g. for an instance named 'redis-1' in region 'us-central1'
# https://console.cloud.google.com/memorystore/redis-1/locations/us-central1/instances/redis/details/overview

Or via gcloud redis instances describe

$ gcloud redis instances describe redis-instance --format="get(host)" --region=us-central1
10.111.1.3

To test the connectivity from a VM, perform the following steps:

• create a Compute Instance VM in the "default" network
• log into the VM via the UI

• install the redis-cli client via

  sudo apt-get install redis-tool -y
  

• connect to the Redis Memorystore instance via

  redis-cli -h $privateIp
  

  where $privateIp is the private IP of the redis instance

• once connected:
  • type the command AUTH followed by the AUTH string of the instance to authenticate
  • type the command PING
  • you should get PONG as response

Connecting to the redis instance via SSH tunnel

The GCP Redis Guide: Connecting from a local machine with port forwarding proposes an interesting approach to connect to a Redis Memorystore instance by using a Compute Instance VM as a jump host, i.e.

• create a VM that lives in the same VPC as the redis instance

• create an SSH tunnel via gcloud comute ssh --ssh-flag

  gcloud compute ssh test-instance --zone=us-central1-a --ssh-flag="-N -L 6379:$privateRedisInstanceIp:6379"
  

  • run this command on your local machine to forward your local port 6379 to port 6379 of the redis instance
  • the traffic flows over the Compute instance VM in this case

• on your local machine connect to the Redis Memorystore instance via

  redis-cli -h localhost
  

Delete a Redis Memorystore instance

To delete a Redis Memorystore instance simply navigate to its management UI and click the "Delete" button. Corresponding docs: GCP Redis Guide: Creating and managing Redis instances > Deleting instances

Using the gcloud CLI

Even though I like using the UI to "explore and understand" how things are working, the goal is always a more "unattended" approach, e.g. via the gcloud cli.

The following commands assume that you have created a master service account with owner permissions and activated it for glcoud with a default project. See also Preconditions: Project and Owner service account

Activate the service account

project_id=pl-dofroscra-p
gcloud auth activate-service-account --key-file=./gcp-master-service-account-key.json --project=${project_id}

Enable the necessary APIs

• gcloud services enable reference

gcloud services enable \
  compute.googleapis.com \
  redis.googleapis.com \
  cloudresourcemanager.googleapis.com \
  servicenetworking.googleapis.com

cloudresourcemanager.googleapis.com is necessary to create the IP range allocation in the next step.

Create an IP range allocation

See "Create an IP range allocation" in the MySQL Cloud SQL article

Create the VPC peering with servicenetworking.googleapis.com

See "Create the VPC peering with servicenetworking.googleapis.com" in the MySQL Cloud SQL article

Create the redis instance

• gcloud redis instances create reference
• GCP Redis Guide: Creating and managing Redis instances > Custom ranges with private services access

region=us-central1
redis_instance_name="redis"
size=1
network="default" # must be the same as for the VMs
version="redis_6_x" # see https://cloud.google.com/sdk/gcloud/reference/redis/instances/create#--redis-version
private_ip_range_name="internal-gcp-services"

gcloud redis instances create "${redis_instance_name}" \
      --size="${size}" \
      --region="${region}" \
      --network="${network}" \
      --redis-version="${version}" \
      --connect-mode=private-service-access \
      --reserved-ip-range="${private_ip_range_name}" \
      --enable-auth \
      -q 

• the --reserved-ip-range must be name of the range created in step Create an IP range allocation

• when using the --enable-auth flag, we need to include the -q flag as well, otherwise we are prompted for an additional confirmation:

  AUTH prevents accidental access to the instance by requiring an AUTH string (automatically generated for you). AUTH credentials are not confidential when transmitted or intended to protect against malicious actors.
  
  Do you want to proceed? (Y/n)?
  

• the --region is required on any subsequent requests to identify the instance, i.e. the instance name is not enough. If the region is not provided, an error is shown:

  $ gcloud redis instances describe redis-instance
  ERROR: (gcloud.redis.instances.describe) Error parsing [instance].
  The [instance] resource is not properly specified.
  Failed to find attribute [region]. The attribute can be set in the following ways:
  - provide the argument `--region` on the command line
  - set the property `redis/region`
  

Retrieve the private IP

• gcloud redis instances create reference

gcloud redis instances describe "${redis_instance_name}" \
    --format="get(host)" \
    --region="${region}"

Retrieve the AUTH string

• gcloud redis instances get-auth-string reference

gcloud redis instances get-auth-string "${redis_instance_name}" \
    --region="${region}"

Wrapping up

Congratulations, you made it! If some things are not completely clear by now, don't hesitate to leave a comment. You are now able to manage redis datastores on GCP via the UI as well as via the gcloud cli.

---

Wanna stay in touch?

Since you ended up on this blog, chances are pretty high that you're into Software Development (probably PHP, Laravel, Docker or Google Big Query) and I'm a big fan of feedback and networking.

So - if you'd like to stay in touch, feel free to shoot me an email with a couple of words about yourself and/or connect with me on LinkedIn or Twitter or simply subscribe to my RSS feed or go the crazy route and subscribe via mail and don't forget to leave a comment :)

Comments