acquires Cloud Conformity
Open menu

Elasticsearch Domain In VPC

Cloud Conformity allows you to automate the auditing process of this resolution page. Register for a 14 day evaluation and check your compliance level for free!

Start a Free Trial Product features
Knowledge Base Best practice rules for Amazon Web Services AWS Elasticsearch Best Practices Elasticsearch Domain In VPC
Security
Risk level: Medium (should be achieved)
Rule ID: ES-010

Ensure that your Amazon Elasticsearch (ES) domains (clusters) are accessible only from AWS VPCs for better flexibility and control over the clusters access and security as this feature lets you keep all traffic between your VPC and Elasticsearch domains within the AWS network instead of going over the public Internet.

This rule can help you with the following compliance standards:

This rule can help you work with the AWS Well-Architected Framework

This rule resolution is part of the Cloud Conformity Security & Compliance tool for AWS

AWS Elasticsearch domains that reside within a VPC have an extra layer of security when compared to ES domains that use public endpoints. Launching an Amazon ES cluster within an AWS VPC enables secure communication between the ES cluster (domain) and other AWS services without the need for an Internet Gateway, a NAT device or a VPN connection and all traffic remains secure within the AWS Cloud.

Audit

To determine the access endpoint configuration for your existing Elasticsearch domains, perform the following:

Using AWS Console

01 Login to the AWS Management Console.

02 Navigate to Elasticsearch (ES) dashboard at https://console.aws.amazon.com/es/.

03 Choose the ES domain that you want to examine and check its endpoint attribute value available within Endpoint column. If the endpoint value is set to Internet, the domain is publicly accessible, therefore the selected Elasticsearch cluster does not reside within an AWS VPC.

04 Repeat step no. 3 to verify the endpoint configuration for other AWS ES domains (clusters) available in the current region.

05 Change the AWS region from the navigation bar and repeat the process for the other regions.

Using AWS CLI

01 Run list-domain-names command (OSX/Linux/UNIX) to list the names of all Amazon Elasticsearch domains currently available within the selected region: 

aws es list-domain-names
	--region us-east-1

02 The command output should return the requested ES domain names: 

{
    "DomainNames": [
        {
            "DomainName": "cc-es-main-domain"
        },
        {
            "DomainName": "cc-project5-domain"
        }
    ]
}

03 Run describe-elasticsearch-domain command (OSX/Linux/UNIX) using the Elasticsearch domain name returned at the previous step as identifier and custom query filters to return the access endpoint URL assigned to the selected domain: 

aws es describe-elasticsearch-domain
	--domain-name cc-es-main-domain
	--region us-east-1
	--query 'DomainStatus.Endpoint'

04 The command output should return the requested endpoint URL or null if the selected ES domain is currently associated with a VPC: 

search-cc-es-main-domain-aaabbbcccdddeee.us-east-1.es.amazonaws.com

If the describe-elasticsearch-domain command output returns a public endpoint URL, as shown in the output example above, the domain is publicly accessible, therefore the selected Elasticsearch cluster does not reside within an AWS VPC.

05 Repeat step no. 3 and 4 to verify the endpoint configuration for other AWS ES domains available in the current region.

06 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 - 5 to perform the entire audit process for other regions.

Remediation / Resolution

To migrate your AWS Elasticsearch domains from public access to VPC access (recommended), you must unload the existing data from the domain (cluster) to Amazon S3 then upload this data in a new ES cluster, launched within a Virtual Private Cloud. To relaunch and configure your Elasticsearch cluster(s) within an AWS VPC, perform the following actions:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Elasticsearch (ES) dashboard at https://console.aws.amazon.com/es/..

03 Click on the ES domain that you want to relaunch (see Audit section part I to identify the right resource).

04 On the selected ES domain description page, click the Configure cluster button from the dashboard top menu to open the cluster configuration page.

05 On the Configure cluster page, copy the selected cluster configuration information such as Instance count, Instance type, Dedicated master instance type, Dedicated master instance count, Storage Type, EBS volume size, etc.

06 On the Set up access policy page, copy the access policy available in the Add or edit the access policy textbox.

07 Go back to the AWS ES service dashboard and click the Create new domain button from the dashboard top menu to launch a new Elasticsearch domain.

08 On the Define domain page, perform the following actions:

  1. Provide a unique name for the new ES domain in the Elasticsearch domain name box.
  2. Select the right version of the Elasticsearch engine from the Elasticsearch version dropdown list.
  3. Click Next to continue the setup process.

09 On the Configure cluster page, set the new domain parameters using the configuration details copied at step no. 5 then click Next to continue.

10 On the Set up access page of the new domain, perform the following actions:

  1. Inside Network configuration section, choose VPC access option to launch the domain within a VPC, then select the VPC identifier from the VPC dropdown list, an available subnet from the Subnet list and one or multiple security groups from Security Groups dropdown list.
  2. Within Access policy section, paste the access policy copied at step no. 6 (if required) into the Add or edit the access policy box or simply select a pre-configured policy from the Set the domain access policy to dropdown list and edit it to meet the needs of your ES domain.
  3. Click Next to continue the process.

11 On the Review page, verify the domain configuration and its access policy then click Confirm and create to launch the new AWS Elasticsearch domain within the specified VPC.

12 Once the new AWS ES domain is created, upload the data from the source cluster (domain) to the new ES cluster.

13 Now it’s safe to remove the source (publicly accessible) Elasticsearch domain in order to stop incurring charges for it. To delete the source domain, perform the following:

  1. Click on the name of the domain that you want to remove (see Audit section part I to identify the right resource).
  2. On the selected domain description page, click Delete Elasticsearch domain to expand the section panel then click Delete domain button to start the removal process.
  3. Within Delete domain dialog box, check Delete the domain <domain_name> then click the Delete button to confirm the action.

14 Repeat steps no. 3 - 13 to relaunch other AWS ES domains, available in the current region, into an AWS VPC.

15 Change the AWS region from the navigation bar and repeat the entire process for other regions.

Using AWS CLI

01 Run describe-elasticsearch-domain command (OSX/Linux/UNIX) using the name of the ES domain that you want to relaunch (see Audit section part II to identify the right resource) to list the selected domain (cluster) configuration information: 

aws es describe-elasticsearch-domain
	--region us-east-1
	--domain-name cc-es-main-domain

02 The command output should return the configuration details (metadata) for the selected AWS ES domain: 

{
    "DomainStatus": {
        "ElasticsearchClusterConfig": {
            "DedicatedMasterEnabled": false,
            "InstanceCount": 2,
            "ZoneAwarenessEnabled": false,
            "InstanceType": "m4.xlarge.elasticsearch"
        },
        "Endpoint": "search-cc-es-main-domain-aaabbbcccdddeee.us-east-1.es.amazonaws.com",
        "Created": true,
        "Deleted": false,
        "DomainName": "cc-es-main-domain",
        "EBSOptions": {
            "VolumeSize": 100,
            "VolumeType": "gp2",
            "EBSEnabled": true
        },
        "SnapshotOptions": {
            "AutomatedSnapshotStartHour": 0
        },
        "DomainId": "123456789012/cc-es-main-domain",
        "AccessPolicies": "{...}",
        "Processing": false,
        "AdvancedOptions": {
            "rest.action.multi.allow_explicit_index": "true"
        },
        "ElasticsearchVersion": "2.3",
        "ARN": "arn:aws:es:us-east-1:123456789012:domain/cc-es-main-domain"
    }
}

03 Run create-elasticsearch-domain command (OSX/Linux/UNIX) using the configuration metadata returned at the previous step to relaunch the selected Amazon Elasticsearch domain into an AWS Virtual Private Cloud (VPC): 

aws es create-elasticsearch-domain
	--region us-east-1
	--domain-name cc-vpc-main-domain
	--elasticsearch-version 2.3
	--elasticsearch-cluster-config InstanceType=m4.xlarge.elasticsearch,InstanceCount=2
	--ebs-options EBSEnabled=true,VolumeType=standard,VolumeSize=100
	--vpc-options SubnetIds=subnet-aaaabbbb,SecurityGroupIds=sg-ccccdddd

04 The command output should return the metadata for the new AWS Elasticsearch domain: 

{
    "DomainStatus": {
        "ElasticsearchClusterConfig": {
            "DedicatedMasterEnabled": false,
            "InstanceCount": 2,
            "ZoneAwarenessEnabled": false,
            "InstanceType": "m4.xlarge.elasticsearch"
        },
        "DomainId": "123456789012/cc-vpc-main-domain",
        "VPCOptions": {
            "SubnetIds": [
                "subnet-aaaabbbb"
            ],
            "VPCId": "vpc-aabbccdd",
            "SecurityGroupIds": [
                "sg-ccccdddd"
            ],
            "AvailabilityZones": [
                "us-east-1a"
            ]
        },
        "Created": true,
        "Deleted": false,
        "EBSOptions": {
            "VolumeSize": 100,
            "VolumeType": "standard",
            "EBSEnabled": true
        },
        "Processing": true,
        "DomainName": "cc-vpc-main-domain",
        "SnapshotOptions": {
            "AutomatedSnapshotStartHour": 0
        },
        "ElasticsearchVersion": "2.3",
        "AccessPolicies": "",
        "AdvancedOptions": {
            "rest.action.multi.allow_explicit_index": "true"
        },
        "ARN": "arn:aws:es:us-east-1:123456789012:domain/cc-vpc-main-domain"
    }
}

05 Once the new AWS ES cluster is provisioned, upload the existing data (exported from the source ES cluster) to the newly created cluster.

06 After all the data is uploaded, it is safe to remove the source (publicly accessible) Elasticsearch domain in order to stop incurring charges for the resource. To shut it down run delete-elasticsearch-domain command (OSX/Linux/UNIX) using the name of the domain that you want to delete as command parameter: 

aws es delete-elasticsearch-domain
	--region us-east-1
	--domain-name cc-es-main-domain

07 The command output should return the source AWS Elasticsearch domain metadata: 

{
    "DomainStatus": {
        "ElasticsearchClusterConfig": {
            "DedicatedMasterEnabled": false,
            "InstanceCount": 2,
            "ZoneAwarenessEnabled": false,
            "InstanceType": "m4.xlarge.elasticsearch"
        },

	  ...

        "AdvancedOptions": {
            "rest.action.multi.allow_explicit_index": "true",
            "indices.fielddata.cache.size": ""
        },
        "ElasticsearchVersion": "2.3",
        "ARN": "arn:aws:es:us-east-1:123456789012:domain/cc-es-main-domain"
    }
}

08 Repeat steps no. 1 - 7 to relaunch other AWS ES domains available in the current region.

09 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 – 8 to perform the process for other regions.

References

Publication date Oct 27, 2017

Related Elasticsearch rules

Thanks!

A verification email has been sent to

Thanks!

A verification email has been sent to

Thanks!

A verification email has been sent to