|   Trend Micro™ Cloud One
Open menu

Use KMS Customer Master Keys for AWS MSK Clusters

Cloud Conformity allows you to automate the auditing process of this resolution page. Register for a 14 day evaluation and check your compliance level for free!

Start a Free Trial Product features
Security
Risk level: Medium (should be achieved)

Ensure that Amazon Managed Streaming for Kafka (MSK) clusters are using AWS KMS Customer Master Keys (CMKs) instead of AWS managed-keys (i.e. default keys) for data encryption, in order to have a fine-grained control over data-at-rest encryption/decryption process and meet compliance requirements. MSK is a fully managed AWS service that enables you to migrate, build and run real-time streaming applications on Apache Kafka.

To determine the encryption configuration for your AWS MSK clusters, perform the following actions:

Note: Verifying encryption configuration for Amazon MSK clusters using AWS Management Console is not currently supported, the feature can to be configured only through AWS Command Line Interface (CLI).

Audit

To determine if encryption at rest is enabled for your Amazon DynamoDB Accelerator (DAX) clusters, perform the following actions:

Using AWS CLI

01 Run list-clusters command (OSX/Linux/UNIX) using custom query filters to list the Amazon Resource Names (ARNs) of the Amazon MSK clusters available in the selected region:

aws kafka list-clusters
	--region us-east-1
	--query 'ClusterInfoList[*].ClusterArn'

02 The command output should return an array with the requested ARNs:

[
"arn:aws:kafka:us-east-1:123456789012:cluster/cc-kafka-cluster/abcd1234-abcd-1234-abcd-1234abcd1234-a",
"arn:aws:kafka:us-east-1:123456789012:cluster/cc-msk-app-cluster/aabbccdd-1234-aabb-1234-aabbccddaabb-c"
]

03 Run describe-cluster command (OSX/Linux/UNIX) using the Amazon Resource Name of the cluster that you want to examine as identifier and custom query filters to return the ARN of the AWS KMS key used for MSK data encryption:

aws kafka describe-cluster
	--region us-east-1
	--cluster-arn arn:aws:kafka:us-east-1:123456789012:cluster/cc-kafka-cluster/abcd1234-abcd-1234-abcd-1234abcd1234-a
	--query 'ClusterInfo.EncryptionInfo.EncryptionAtRest.DataVolumeKMSKeyId'

04 The command output should return the requested KMS key ARN:

"arn:aws:kms:us-east-1:123456789012:key/abcdabcd-abcd-abcd-abcd-abcdabcdabcd"

05 Run describe-key command (OSX/Linux/UNIX) using the AWS KMS key ARN returned at the previous step as identifier to expose the name of the manager (either "AWS" or "CUSTOMER") for the specified encryption key:

aws aws kms describe-key
	--region us-east-1
	--key-id "arn:aws:kms:us-east-1:123456789012:key/abcdabcd-abcd-abcd-abcd-abcdabcdabcd"
	--query 'KeyMetadata.KeyManager'

06 The command output should return the selected AWS KMS key manager name:

"AWS"

If the value returned by the describe-key command output is "AWS", the encryption key manager is Amazon Web Services, therefore the selected AWS Managed Streaming for Kafka (MSK) cluster is encrypting its data at rest using the default master key (i.e. aws/kafka key) instead of a customer-managed Customer Master Key (CMK).

07 Repeat step no. 3 and 4 to determine the encryption configuration for other Amazon MSK clusters available in the current region.

08 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 – 5 to perform the entire process for other regions.

Remediation / Resolution

Encryption at rest using Customer Master Keys cannot be configured for existing AWS MSK clusters. To encrypt Amazon Managed Streaming for Kafka (MSK) cluster data using your own AWS KMS Customer Master Keys (CMKs), you have to re-create the specified cluster. To create the required AWS KMS CMK and relaunch the required cluster, perform the following actions:

Note: Creating and configuring Amazon MSK clusters using the AWS Management Console is not currently supported.

Using AWS CLI

01 Define the access policy that enables the specified IAM users and/or roles to manage the new KMS Customer Master Key and to encrypt/decrypt MSK cluster data using the AWS KMS API. Create a new policy document, name it kafka-cmk-policy.json, and paste the following content (replace the highlighted details, i.e. the ARNs for the IAM users and/or roles, with your own details):

{
  "Version": "2012-10-17",
  "Id": "aws-kafka-custom-key-policy",
  "Statement": [
    {
      "Sid": "Enable IAM User Permissions",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:root"
      },
      "Action": "kms:*",
      "Resource": "*"
    },
    {
      "Sid": "Grant access to CMK manager",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:role/KafkaManager"
      },
      "Action": [
        "kms:Create*",
        "kms:Describe*",
        "kms:Enable*",
        "kms:List*",
        "kms:Put*",
        "kms:Update*",
        "kms:Revoke*",
        "kms:Disable*",
        "kms:Get*",
        "kms:Delete*",
        "kms:ScheduleKeyDeletion",
        "kms:CancelKeyDeletion"
      ],
      "Resource": "*"
    },
    {
      "Sid": "Allow the use of the CMK",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:user/KafkaAdmin"
      },
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:DescribeKey"
      ],
      "Resource": "*"
    },
    {
      "Sid": "Allow attachment of persistent resources",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:user/KafkaAdmin"
      },
      "Action": [
        "kms:CreateGrant",
        "kms:ListGrants",
        "kms:RevokeGrant"
      ],
      "Resource": "*",
      "Condition": {
        "Bool": {
          "kms:GrantIsForAWSResource": "true"
        }
      }
    }
  ]
}

02 Run create-key command (OSX/Linux/UNIX) using the file name of the policy document created at the previous step (i.e. kafka-cmk-policy.json) as command parameter to create the new AWS KMS CMK:

aws kms create-key
	--region us-east-1
	--description 'KMS CMK for encrypting AWS MKS data.'
	--policy file://kafka-cmk-policy.json

03 The command output should return the new KMS CMK metadata. Copy the key Amazon Resource Name (Arn parameter value - highlighted) as this information will be required later when you need to specify the key required for MSK data encryption:

{
    "KeyMetadata": {
        "Origin": "AWS_KMS",
        "KeyId": "abcdabcd-1234-1234-1234-abcdabcdabcd",
        "Description": "KMS CMK for encrypting AWS MSK data.",
        "Enabled": true,
        "KeyUsage": "ENCRYPT_DECRYPT",
        "KeyState": "Enabled",
        "CreationDate": 1517237844.370,
        "Arn": "arn:aws:kms:us-east-1:123456789012:key/abcdabcd-1234-1234-1234-abcdabcdabcd",
        "AWSAccountId": "123456789012"
    }
}

04 Run create-alias command (OSX/Linux/UNIX) using the key ARN returned at the previous step to attach an alias to the new CMK. The alias must start with the prefix "alias/" (the command does not return an output):

aws kms create-alias
	--region us-east-1
	--alias-name alias/KafkaCustomCMK
	--target-key-id arn:aws:kms:us-east-1:123456789012:key/abcdabcd-1234-1234-1234-abcdabcdabcd

05 Run describe-cluster command (OSX/Linux/UNIX) using the ARN of the cluster that you want to re-create as identifier (see Audit section to identify the right resources) and custom query filters to get the cluster configuration metadata:

aws kafka describe-cluster
	--region us-east-1
	--cluster-arn arn:aws:kafka:us-east-1:123456789012:cluster/cc-kafka-cluster/abcd1234-abcd-1234-abcd-1234abcd1234-a

06 The command output should return the requested resource configuration metadata:

{
    "ClusterInfo": {,
        "BrokerNodeGroupInfo": {
            "BrokerAZDistribution": "DEFAULT",
            "ClientSubnets": [
                "subnet-abcd1234",
                "subnet-1234abcd",
                "subnet-12341234"
            ],
            "StorageInfo": {
                "EbsStorageInfo": {
                    "VolumeSize": 450
                }
            },

            ...

            "SecurityGroups": [
                "sg-aabbccdd"
            ],
            "InstanceType": "kafka.m5.large"
        },
        "ClusterName": "cc-kafka-cluster",
        "CurrentBrokerSoftwareInfo": {
            "KafkaVersion": "2.1.0"
        },
        "CreationTime": "2019-02-15T10:35:31.353Z",
        "NumberOfBrokerNodes": 3,
        "EnhancedMonitoring": "DEFAULT"
    }
}

07 Define the necessary parameters for the create-cluster command using the configuration metadata returned at the previous step and save the JSON document to a file named msk-cluster-config.json:

{
  "BrokerAZDistribution": "DEFAULT",
  "ClientSubnets": [
    "subnet-abcd1234",
    "subnet-1234abcd",
    "subnet-12341234"
  ],
  "InstanceType": "kafka.m5.large",
  "SecurityGroups": [
    "sg-aabbccdd"
  ],
  "StorageInfo": {
    "EbsStorageInfo": {
      "VolumeSize": 450
    }
  }
}

08 Run create-cluster command (OSX/Linux/UNIX) to launch your Amazon Managed Streaming for Kafka (MSK) cluster using the configuration parameters defined at the previous step (i.e. msk-cluster-config.json) and the ARN of the AWS KMS Customer Master Key created earlier in the process:

aws kafka create-cluster
	--region us-east-1
	--cluster-name cc-msk-production-cluster
	--kafka-version 2.1.0
	--number-of-broker-nodes 3
	--broker-node-group-info file://msk-cluster-config.json
	--enhanced-monitoring DEFAULT
	--encryption-info EncryptionAtRest={DataVolumeKMSKeyId=arn:aws:kms:us-east-1:123456789012:key/abcdabcd-1234-1234-1234-abcdabcdabcd}

09 The command output should return the command request metadata:

{
    "ClusterName": "cc-msk-production-cluster",
    "State": "CREATING",
    "ClusterArn": "arn:aws:kafka:us-east-1:123456789012:cluster/cc-msk-production-cluster/12341234-abcd-abcd-abcd-1234abcd1234-c"
}

10 Repeat step no. 5 – 9 to configure encryption at rest using KMS Customer Master Keys for other Amazon MSK clusters that you need to re-create within the selected region.

11 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 – 10 to perform the entire process for other regions.

References

Publication date Mar 4, 2019

Thanks!

A verification email has been sent to

Thanks!

A verification email has been sent to

Thanks!

A verification email has been sent to