Enable GKE Cluster Node Encryption with Customer-Managed Keys

Trend Micro Cloud One™ – Conformity is a continuous assurance tool that provides peace of mind for your cloud infrastructure, delivering over 750 automated best practice checks.

Risk level: High (not acceptable risk)

Ensure that your Google Kubernetes Engine (GKE) cluster nodes are encrypted with Customer-Managed Keys (CMKs) in order to have a fine control over your GKE data encryption/decryption process. You can create and manage your own Customer-Managed Keys (CMKs) with Cloud Key Management Service (Cloud KMS). Cloud KMS provides secure and efficient encryption key management, controlled key rotation, and revocation mechanisms.

Security

By default, GKE service encrypts all data at rest using Google-managed encryption keys and manages the encryption/decryption process without any action on your end. However, if you need to achieve strict compliance or regulatory requirements, you can choose to fully control and manage GKE cluster data encryption yourself, using your own Customer-Managed Keys (CMKs). If you have to encrypt business-critical, sensitive, or confidential GKE data, it is strongly recommended to encrypt your GKE cluster nodes using Customer-Managed Keys.


Audit

To determine if your Google Kubernetes Engine (GKE) cluster nodes are encrypted with Customer-Managed Keys (CMKs), perform the following actions:

Using GCP Console

01 Sign in to Google Cloud Management Console.

02 Select the Google Cloud Platform (GCP) project that you want to access from the console top navigation bar.

03 Navigate to Google Kubernetes Engine (GKE) console at https://console.cloud.google.com/kubernetes.

04 In the navigation panel, select Clusters to access the list of the GKE clusters deployed within the selected project.

05 Click on the name of the GKE cluster that you want to examine and select the Details tab to access the cluster configuration information.

06 Under Node pools, click on the name of the cluster node pool that you want to examine.

07 In the Nodes section, check the Boot disk encryption attribute value. If the configuration attribute value is set to Google-managed key, the nodes provisioned for the selected GKE cluster node pool are not encrypted with a KMS Customer-Managed Key (CMK).

08 Repeat step no. 6 and 7 for each node pool provisioned for the selected GKE cluster.

09 Repeat step no. 5 – 8 for each GKE cluster created for the selected GCP project.

10 Repeat steps no. 2 – 9 for each project deployed within your Google Cloud account.

Using GCP CLI

01 Run projects list command (Windows/macOS/Linux) using custom query filters to list the IDs of all the Google Cloud Platform (GCP) projects available in your cloud account:

gcloud projects list
	--format="table(projectId)"

02 The command output should return the requested GCP project identifiers:

PROJECT_ID
cc-bigdata-project-123123
cc-iot-app-project-112233

03 Run container clusters list command (Windows/macOS/Linux) using custom query filters to describe the name and the region of each GKE cluster provisioned for the selected Google Cloud project:

gcloud container clusters list
	--project cc-bigdata-project-123123
	--format="(NAME,LOCATION)"

04 The command output should return the requested GKE cluster name(s) and their region(s):

NAME                     LOCATION
cc-gke-frontend-cluster  us-central1
cc-gke-backend-cluster   us-central1

05 Run rcontainer node-pools list command (Windows/macOS/Linux) using the name of the Google Cloud GKE cluster that you want to examine as identifier parameter and custom query filters to describe the name of each node pool provisioned for the selected cluster:

gcloud container node-pools list
	--cluster=cc-gke-frontend-cluster
	--region=us-central1
	--format="(NAME)"

06 The command output should return the requested cluster node pool name(s):

NAME
cc-gke-frontend-pool-001
cc-gke-frontend-pool-002

07 Run container node-pools describe command (Windows/macOS/Linux) using the name of the cluster node pool that you want to examine as identifier parameter and custom output filtering to describe the resource ID of the Customer-Managed Key used to encrypt the node pool data:

gcloud container node-pools describe cc-gke-frontend-pool-001
	--cluster=cc-gke-frontend-cluster
	--region=us-central1
	--format="json(config.bootDiskKmsKey)"

08 The command output should return the full resource ID of the CMK used to encrypt the cluster node pool data:

null

If the container node-pools describe command output returns null, the nodes provisioned for the selected GKE cluster node pool are not encrypted with a KMS Customer-Managed Key (CMK).

09 Repeat step no. 7 and 8 for each node pool provisioned for the selected GKE cluster.

10 Repeat step no. 5 – 9 for each GKE cluster created for the selected GCP project.

11 Repeat steps no. 3 – 10 for each GCP project deployed in your Google Cloud account.

Remediation / Resolution

To enable encryption with Customer-Managed Keys (CMKs) for your Google Kubernetes Engine (GKE) cluster nodes, you have to re-create the existing GKE cluster node pools with the appropriate encryption configuration by performing the following operations:

Using GCP Console

01 Sign in to Google Cloud Management Console.

02 Select the GCP project that you want to access from the console top navigation bar.

03 To create and configure your new Customer-Managed Key (CMK), perform the following:

  1. Navigate to Cloud Key Management Service (Cloud KMS) dashboard at https://console.cloud.google.com/security/kms.
  2. Before you can set up and manage any Customer-Managed Keys (CMKs), you must create a key ring. A KMS key ring is a grouping of cryptographic keys made available for organizational purposes in a specific location. In the navigation panel, select Cryptographic Keys, and click on the CREATE KEY RING button to set up the required key ring and the new Customer-Managed Key (CMK).
  3. A key ring requires a name and location. On the Create key ring page, provide a unique name in the Key ring name box, then choose the appropriate location from the Key ring location dropdown list. The location can be either global or associated with a particular region. If the CMKs created later within the key ring will be used to encrypt/decrypt resources in a given region, select that region as the key ring location. Click CREATE to deploy the new key ring.
  4. On the Create key page, select Generated key as the type of the CMK that you want to create. Provide a name for your new key in the Key name box, choose the protection level (software or Hardware Security Module) that you want to use, select Symmetric encrypt/decrypt from the Purpose dropdown list to define the types of operations that your cryptographic key can perform, and configure the key rotation parameters. Click CREATE to deploy your new Cloud KMS Customer-Managed Key (CMK).

04 Navigate to Google Kubernetes Engine (GKE) console at https://console.cloud.google.com/kubernetes.

05 In the navigation panel, select Clusters to access the list of the GKE clusters available within the selected project.

06 Click on the name of the GKE cluster that you want to reconfigure and select the Details tab to access the cluster configuration information.

07 Under Node pools, click on the name of the cluster node pool that you want to re-create and collect all the configuration information available for the selected resource.

08 Go back to the GKE cluster configuration page and click on the ADD NODE POOL button from the dashboard top menu to initiate the node pool setup process.

09 On the Node pool details panel, provide a unique name for the new node pool in the Name box, choose the GKE version from the Node version dropdown list, and select the number of nodes for the new pool from the Size dropdown list. Configure the rest of the node pool settings based on the configuration information taken at step no. 7.

10 On the Nodes panel, perform the following actions:

  • Check the Enable customer-managed encryption for boot disk checkbox and select the CMK created at step no. 3 from the Select a customer-managed key dropdown list. If the newly created CMK does not appear in the dropdown list, select Don't see your key? Enter key resource ID and provide the full resource ID of your Customer-Managed Key (CMK).
  • Inside "The service-<project-number>@compute-system.iam.gserviceaccount.com service account does not have permissions to encrypt/decrypt with the selected key." configuration box, click Grant to grant the specified service account the required IAM role on the selected CMK.
  • Configure the hardware and network configuration for the new node pool based on the configuration information taken from the pool at step no. 7. Ensure that the new node pool has the same network, compute and storage configuration as the source pool.

11 On the Node security panel, select the appropriate service account from the Service account dropdown list and configure other security settings based on the configuration details taken at step no. 7.

12 On the Node metadata panel, configure the metadata settings such as GCE instance metadata based on the configuration information taken from the source node pool at step no. 7.

13 Click CREATE to launch your new GKE cluster node pool.

14 Once the new cluster node pool is operating successfully, you can remove the source node pool in order to stop adding charges to your Google Cloud monthly bill. Click on the name of the node pool that you want to delete (see Audit section part I to identify the source pool).

15 Click on the DELETE button from the dashboard top menu to initiate the removal process.

16 Within Are you sure you want to delete <pool-name>? dialog box, click DELETE to confirm the node pool deletion.

17 Repeat steps no. 7 – 16 to enable encryption at rest with Customer-Managed Keys (CMKs) for other node pools provisioned for the selected GKE cluster.

18 Repeat step no. 6 – 17 to reconfigure other GKE clusters created for the selected GCP project.

19 Repeat steps no. 2 – 18 for each GCP project available in your Google Cloud account.

Using GCP CLI

01 Before you can set up and manage your Customer-Managed Keys (CMKs), you must create a key ring to store the CMKs. Run kms keyrings create command (Windows/macOS/Linux) to create a new Cloud KMS key ring in the specified location. If the CMKs created later within this key ring will be used to encrypt/decrypt resources in a given region, select that region as the key ring location:

gcloud kms keyrings create cc-cloud-gke-key-ring
	--location=us
	--project=cc-bigdata-project-123123
	--format="table(name)"

02 The command output should return the identifier (name) of the newly created key ring:

NAME
projects/cc-bigdata-project-123123/locations/us/keyRings/cc-cloud-gke-key-ring

03 Run kms keys create command (Windows/macOS/Linux) to create a new Cloud KMS Customer-Managed Key (CMK) within the KMS key ring created at the previous steps:

gcloud kms keys create cc-cloud-gke-cmk
	--location=us-central1
	--keyring=cc-cloud-gke-key-ring
	--purpose=encryption
	--protection-level=software
	--rotation-period=90d
	--next-rotation-time=2020-7-15T15:00:00.0000Z
	--format="table(name)"

04 The command output should return the name of the new Customer-Managed Key (CMK):

NAME
projects/cc-bigdata-project-123123/locations/us-central1/keyRings/cc-cloud-gke-key-ring/cryptoKeys/cc-cloud-gke-cmk

05 Run projects add-iam-policy-binding command (Windows/macOS/Linux) to assign the Cloud KMS "CryptoKey Encrypter/Decrypter" role to the appropriate service account. Replace <kms-project-id> with the ID of the Google Cloud project where the Customer-Managed Keys are provisioned, and replace <project-number> with the project number of the Google Cloud project that is running your GKE clusters:

gcloud projects add-iam-policy-binding <kms-project-id>
	--member serviceAccount:service-<project-number>@compute-system.iam.gserviceaccount.com
	--role roles/cloudkms.cryptoKeyEncrypterDecrypter

06 The command output should return the updated IAM policy (YAML format):

Updated IAM policy for project <kms-project-id>.
bindings:
- members:
  - serviceAccount:service-<project-number>@compute-system.iam.gserviceaccount.com
  role: roles/cloudkms.cryptoKeyEncrypterDecrypter
- members:
  - user:admin@cloudconformity.com
  role: roles/owner
etag: abcdabcdabcd
version: 1

07 Run container node-pools describe command (Windows/macOS/Linux) using the name of the cluster node pool that you want to re-create as identifier parameter and custom output filtering to describe the configuration information available for the selected node pool:

gcloud container node-pools describe cc-gke-frontend-pool-001
	--cluster=cc-gke-frontend-cluster
	--region=us-central1
	--format=json

08 The command output should return the return the requested configuration metadata:

{
  "config": {
    "diskSizeGb": 150,
    "diskType": "pd-standard",
    "imageType": "COS",
    "metadata": {
      "disable-legacy-endpoints": "true"
    },
    "serviceAccount": "default",
    "shieldedInstanceConfig": {
      "enableIntegrityMonitoring": true
    }
  },
  "locations": [
    "us-central1-a",
    "us-central1-b"
  ],

  ...

  "management": {
    "autoRepair": true,
    "autoUpgrade": true
  },
  "maxPodsConstraint": {
    "maxPodsPerNode": "110"
  },
  "name": "cc-gke-frontend-pool-001",
  "podIpv4CidrSize": 24,
  "status": "RUNNING",
  "upgradeSettings": {
    "maxSurge": 1
  },
  "version": "1.14.10-gke.36"
}

09 Run container node-pools create command (Windows/macOS/Linux) using the information returned at the previous step as configuration data for the command parameters, to create a new Google Cloud GKE cluster node pool, encrypted with the Customer-Managed Key (CMK) created at step no. 3:

gcloud beta container node-pools create cc-gke-encrypted-pool-001
	--cluster=cc-gke-frontend-cluster
	--region=us-central1
	--disk-type=pd-standard
	--disk-size=150
	--boot-disk-kms-key=projects/cc-bigdata-project-123123/locations/us-central1/keyRings/cc-cloud-gke-key-ring/cryptoKeys/cc-cloud-gke-cmk

10 The command output should return the URL of the newly created GKE cluster node pool:

Created [https://dataproc.googleapis.com/v1/projects/cc-bigdata-project-123123/regions/us-central1/clusters/cc-encrypted-dataproc-cluster]

11 Once the new cluster node pool is operating successfully, you can remove the source node pool in order to stop adding charges to your Google Cloud bill. Run container node-pools delete command (Windows/macOS/Linux) using the name of the resource that you want to remove as identifier parameter (see Audit section part II to identify the right node pool), to delete the specified GKE cluster node pool:

gcloud container node-pools delete cc-gke-frontend-pool-001
	--cluster=cc-gke-frontend-cluster
	--region=us-central1

12 Type Y to confirm the Google Cloud GKE resource removal:

The following node pool will be deleted.
[cc-gke-frontend-pool-001] in cluster [cc-gke-frontend-cluster] in [us-central1]
Do you want to continue (Y/n)?  Y

13 The output should return the container node-pools delete command request status:

Deleting node pool cc-gke-frontend-pool-001...done.
Deleted [https://container.googleapis.com/v1/projects/cc-bigdata-project-123123/zones/us-central1/clusters/cc-gke-frontend-cluster/nodePools/cc-gke-frontend-pool-001].

14 Repeat steps no. 7 – 13 to enable encryption at rest with Customer-Managed Keys (CMKs) for other node pools provisioned for the selected GKE cluster.

15 Repeat step no. 7 – 14 to reconfigure other GKE clusters created for the selected GCP project.

16 Repeat steps no. 1 – 15 for each GCP project deployed in your Google Cloud account.

References

Publication date May 4, 2021

Unlock the Remediation Steps


Gain free unlimited access
to our full Knowledge Base


Over 750 rules & best practices
for AWS and Azure

You are auditing:

Enable GKE Cluster Node Encryption with Customer-Managed Keys

Risk level: High