Open menu
-->

ElasticSearch Domain Encrypted with KMS CMKs

Cloud Conformity allows you to automate the auditing process of this resolution page. Register for a 14 day evaluation and check your compliance level for free!

Start a Free Trial Product features
Security

Risk level: High (not acceptable risk)

Ensure that your Amazon ElasticSearch (ES) domains are encrypted with KMS Customer Master Keys (CMKs) instead of AWS managed-keys (default keys used by the ES service when there are no customer keys defined) in order to have more granular control over the data-at-rest encryption/decryption process and to meet compliance requirements.

When you use your own KMS Customer Master Keys to protect your ElasticSearch domains (clusters) and their storage systems, you have full control over who can use these keys to access the clusters data. The AWS KMS service allows you to easily create, rotate, disable and audit CMK encryption keys for your ES domains. Note: At-rest encryption using KMS CMKs can be enabled only for AWS ES domains with ElasticSearch version 5.1 and above.

Audit

To determine the encryption status and configuration for your AWS ElasticSearch domains, perform the following actions:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to ElasticSearch (ES) dashboard at https://console.aws.amazon.com/es/.

03 Choose the ES domain that you want to examine and click on the domain name (link) to access its configuration page.

04 On the selected domain configuration page, select the Overview tab and make sure that the Encryption at rest attribute value is set to Enabled (otherwise, if the attribute value is set to Disabled, see this rule to enable domain encryption), then check the encryption key name set for the KMS master key attribute. If the key alias (name) is "(Default) aws/es", the selected Amazon ElasticSearch domain is encrypted using the default master key (AWS-managed key) instead of the AWS KMS Customer Master Key.

05 Repeat step no. 3 and 4 to verify the encryption status and configuration for other ES domains (clusters) provisioned in the current region.

06 Change the AWS region from the navigation bar and repeat the process for the other regions.

Using AWS CLI

01 Run list-domain-names command (OSX/Linux/UNIX) to list the names of all Amazon ElasticSearch domains currently available within the selected region:

aws es list-domain-names
	--region us-east-1

02 The command output should return the requested ES domain names:

{
    "DomainNames": [
        {
            "DomainName": "cc-es-main-domain"
        },
        {
            "DomainName": "cc-es-project5-domain"
        }
    ]
}

03 Run describe-elasticsearch-domain command (OSX/Linux/UNIX) using the ElasticSearch domain name returned at the previous step as identifier and custom query filters to describe the name of the encryption key used to encrypt the data for the selected ES domain:

aws es describe-elasticsearch-domain
	--region us-east-1
	--domain-name cc-es-main-domain
	--query 'DomainStatus.EncryptionAtRestOptions.KmsKeyId'

04 The command output should return the requested key name (alias) or an empty array such as [ ], if there is no key used to encrypt the selected ES domain data, i.e. the encryption is not enabled (see this rule to enable encryption):

[
    "(Default) aws/es"
]

If the key name returned by the describe-elasticsearch-domain command output is "(Default) aws/es", the selected Amazon ElasticSearch domain is encrypted using the default master key (AWS-managed key) instead of the AWS KMS Customer Master Key, therefore the data stored on the domain file systems, primary and replica indices, log files, memory swap files and so on, is not encrypted with a custom KMS CMK.

05 Repeat step no. 3 and 4 to verify the encryption status and the encryption key for other ES domains (clusters) available in the current region.

06 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 - 5 to perform the audit process for other regions.

Remediation / Resolution

To encrypt an existing AWS ElasticSearch domain with your own KMS Customer Master Key, you must re-create the domain with the necessary encryption configuration. To create the necessary KMS CMK and set up the new ES domain, enable custom encryption and copy your existing data to it, perform the following actions:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to IAM dashboard at https://console.aws.amazon.com/iam/.

03 In the left navigation panel click Encryption Keys.

04 Select the appropriate AWS region from the Filter menu (must match the region where your ES domain is provisioned).

05 Click Create Key button from the dashboard top menu.

06 In the Alias (required) and Description fields, enter a unique name (alias) and a description for the new CMK, then click the Next Step button.

07 Under Key Administrators section, select which IAM users and/or roles can administer the new CMK, then click Next Step.

08 Under This Account section, select which IAM users and/or roles can use the new CMK to encrypt/decrypt the domain data with the AWS KMS API.

09 (Optional) Under External Accounts section, click Add an External Account and enter an external account ID in order to add another AWS account that can use this CMK to encrypt/decrypt the ES domain (cluster) data. The owners of the external AWS accounts must also provide access to this CMK by creating appropriate policies for their IAM users.

10 Click Next Step to continue.

11 Under Preview Key Policy section, review the key policy generated by AWS then click Finish to create your new CMK. Once the key is created, the KMS dashboard will display a confirmation message: “Your master key was created successfully. Alias: <the CMK display name>”.

12 Now that the KMS key has been created, navigate to ElasticSearch (ES) dashboard at https://console.aws.amazon.com/es/.

13 Choose the ES domain that you want to relaunch and click on the domain name (link) to access its configuration page.

14 On the domain configuration page, perform the following actions:

  1. Select the Overview tab and copy the domain configuration information such as Instance count, Instance type, Dedicated master instance type, Dedicated master instance count, Storage Type, EBS volume type, EBS volume size and so on.
  2. Select the VPC tab and copy the network configuration information such as VPC ID, Security groups ID(s), IAM role name and AZs and Subnets IDs.
  3. Click on Modify access policy button from the dashboard top menu and copy the policy document available in the Add or edit the access policy textbox.

15 Go back to the AWS ElasticSearch service dashboard and click Create new domain to launch a new ES domain.

16 On the Define domain page, perform the following actions:

  1. Provide a unique name for the new ES domain in the Elasticsearch domain name box.
  2. Select the right version of the Elasticsearch engine from the Elasticsearch version dropdown list.
  3. Click Next to continue the setup process.

17 On the Configure cluster page, perform the following:

  1. Set the new domain parameters using the configuration details copied at step no. 14 a.
  2. Select Enable encryption at rest checkbox to enable data-at-rest encryption feature for the new ES domain. From the KMS master key dropdown list choose your newly created AWS KMS Customer Master Key or select Enter a key ARN and paste the new CMK ARN into the ARN / ID box.
  3. Click Next to continue.

18 On the Set up access page, configure the network access to the new ES domain by using the same parameters copied at step no. 14 b. Also, paste the access policy copied at step no. 14 c. into the Add or edit the access policy textbox to use the same access policy as the source domain, then click the Next button to continue.

19 On the Review page, verify the domain configuration details then click Confirm to launch your new AWS ElasticSearch domain, encrypted with the custom KMS key.

20 Once the new AWS ES domain is created, upload the data from the source domain to the new (destination) ES cluster.

21 Now it’s safe to remove the source ElasticSearch domain from your AWS account to avoid further charges. To delete the required domain, perform the following:

  1. Click on the name of the domain that you want to remove (see Audit section part I to identify the right ElasticSearch resource).
  2. On the selected domain description page, click Delete domain to start the removal process.
  3. Within Delete domain dialog box, check Delete the domain <domain_name> then click the Delete button to confirm the action.

22 Repeat steps no. 2 - 21 to enable data-at-rest encryption for other Amazon ElasticSearch domains available in the current region, using AWS KMS CMKs.

23 Change the AWS region from the navigation bar and repeat the entire process for other regions.

Using AWS CLI

01 Before creating your KMS CMK key, you must define a policy that enables your selected IAM users and/or roles to administer the new KMS Customer Master Key and to encrypt/decrypt ES domains data using the AWS KMS API. Create a new policy document called es-kms-cmk-policy.json and paste the following content (replace the highlighted details, i.e. the ARNs for the IAM users and/or roles, with your own details):

{
  "Version": "2012-10-17",
  "Id": "elasticsearch-custom-key-policy",
  "Statement": [
    {
      "Sid": "Enable IAM User Permissions",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:root"
      },
      "Action": "kms:*",
      "Resource": "*"
    },
    {
      "Sid": "Grant access to CMK manager",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:role/AmazonESManager"
      },
      "Action": [
        "kms:Create*",
        "kms:Describe*",
        "kms:Enable*",
        "kms:List*",
        "kms:Put*",
        "kms:Update*",
        "kms:Revoke*",
        "kms:Disable*",
        "kms:Get*",
        "kms:Delete*",
        "kms:ScheduleKeyDeletion",
        "kms:CancelKeyDeletion"
      ],
      "Resource": "*"
    },
    {
      "Sid": "Allow the use of the CMK",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:user/ESAdmin"
      },
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:DescribeKey"
      ],
      "Resource": "*"
    },
    {
      "Sid": "Allow attachment of persistent resources",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:user/ESAdmin"
      },
      "Action": [
        "kms:CreateGrant",
        "kms:ListGrants",
        "kms:RevokeGrant"
      ],
      "Resource": "*",
      "Condition": {
        "Bool": {
          "kms:GrantIsForAWSResource": "true"
        }
      }
    }
  ]
}

02 Run create-key command (OSX/Linux/UNIX) using the file name of the policy document created at the previous step (i.e. es-kms-cmk-policy.json) as required command parameter to create the new KMS CMK:

aws kms create-key
	--region us-east-1
	--description 'KMS CMK for encrypting ES domain data'
	--policy file://es-kms-cmk-policy.json

03 The command output should return the new KMS CMK metadata. Copy the CMK unique ID (KeyID parameter value - highlighted) as this ID will be required later when you need to specify the key required for ES domain data encryption:

{
    "KeyMetadata": {
        "Origin": "AWS_KMS",
        "KeyId": "aaaabbbb-cccc-dddd-eeee-aaaabbbbcccc",
        "Description": "KMS CMK for encrypting ES domain data",
        "Enabled": true,
        "KeyUsage": "ENCRYPT_DECRYPT",
        "KeyState": "Enabled",
        "CreationDate": 1517234459.314,
        "Arn": "arn:aws:kms:us-east-1:123456789012:key/aaaabbbb-cccc-dddd-eeee-aaaabbbbcccc",
        "AWSAccountId": "123456789012"
    }
}

04 Run create-alias command (OSX/Linux/UNIX) using the key ARN returned at the previous step to attach an alias to the new CMK. The alias must start with the prefix "alias/" (the command does not produce an output):

aws kms create-alias
	--region us-east-1
	--alias-name alias/ESCustomCMK
	--target-key-id arn:aws:kms:us-east-1:123456789012:key/aaaabbbb-cccc-dddd-eeee-aaaabbbbcccc

05 Run describe-elasticsearch-domain command (OSX/Linux/UNIX) using the name of the ES domain that you want to relaunch (see Audit section part II to identify the right resource) to describe the configuration information for the selected domain:

aaws es describe-elasticsearch-domain
	--region us-east-1
	--domain-name cc-es-main-domain

06 The command output should return the configuration details for the selected AWS ES domain:

{
    "DomainStatus": {
        "ElasticsearchClusterConfig": {
            "DedicatedMasterEnabled": false,
            "InstanceCount": 2,
            "ZoneAwarenessEnabled": false,
            "InstanceType": "m4.large.elasticsearch"
        },
        "DomainName": "cc-es-main-domain",

        ...

        "ElasticsearchVersion": "5.5",
        "Processing": false,
        "Endpoints": {
            "vpc": "vpc-cc-es-main-domain-aaaabbbbccccddddeeeeffff.us-east-1.es.amazonaws.com"
        },
        "ARN": "arn:aws:es:us-east-1:123456789012:domain/cc-es-main-domain"
    }
}

07 Run create-elasticsearch-domain command (OSX/Linux/UNIX) using the configuration metadata returned at the previous step to relaunch the selected AWS ElasticSearch domain with the necessary encryption configuration (i.e. using the KMS CMK with the ID "aaaabbbb-cccc-dddd-eeee-aaaabbbbcccc"):

aws es create-elasticsearch-domain
	--region us-east-1
	--domain-name cc-es-main-new-domain
	--elasticsearch-version 5.5
	--elasticsearch-cluster-config InstanceType=m4.large.elasticsearch,InstanceCount=2
	--ebs-options EBSEnabled=true,VolumeType=standard,VolumeSize=150
	--access-policies file://es-source-domain-access-policy.json
	--vpc-options SubnetIds=subnet-aaaa1234,SecurityGroupIds=sg-1234bbbb
	--encryption-at-rest-options Enabled=true,KmsKeyId="aaaabbbb-cccc-dddd-eeee-aaaabbbbcccc"

08 The command output should return the metadata for the new Amazon ElasticSearch domain:

{
    "DomainStatus": {
        "ElasticsearchClusterConfig": {
            "DedicatedMasterEnabled": false,
            "InstanceCount": 2,
            "ZoneAwarenessEnabled": false,
            "InstanceType": "m4.large.elasticsearch"
        },
        "DomainId": "123456789012/cc-es-main-new-domain",

        ...

        "ElasticsearchVersion": "5.5",
        "Processing": true,
        "EncryptionAtRestOptions": {
            "Enabled": true,
            "KmsKeyId": "aaaabbbb-cccc-dddd-eeee-aaaabbbbcccc"
        },
        "ARN": "arn:aws:es:us-east-1:123456789012:domain/cc-es-main-new-domain"
    }
}

09 Once the new AWS ES cluster is provisioned, upload the existing data (exported from the source ES cluster) to the newly created cluster.

10 After all the data is uploaded, it is safe to remove the source ElasticSearch domain in order to stop incurring charges for the resource. To shut it down, run delete-elasticsearch-domain command (OSX/Linux/UNIX) using the name of the domain that you want to delete as command parameter:

aws es delete-elasticsearch-domain
	--region us-east-1
	--domain-name cc-es-main-domain

11 The command output should return the source AWS ElasticSearch domain metadata:

{
    "DomainStatus": {
        "ElasticsearchClusterConfig": {
            "DedicatedMasterEnabled": false,
            "InstanceCount": 2,
            "ZoneAwarenessEnabled": false,
            "InstanceType": "m4.large.elasticsearch"
        },

	  ...

        "AdvancedOptions": {
            "rest.action.multi.allow_explicit_index": "true",
            "indices.fielddata.cache.size": ""
        },
        "ElasticsearchVersion": "5.5",
        "ARN": "arn:aws:es:us-east-1:123456789012:domain/cc-es-main-domain"
    }
}

12 Repeat steps no. 1 - 11 to enable data-at-rest encryption for other Amazon ElasticSearch domains available in the current region, using AWS KMS Customer Master Keys.

13 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 – 12 to perform the entire process for other regions.

References

Publication date Feb 2, 2018