Open menu
-->

Elasticsearch Version

Cloud Conformity allows you to automate the auditing process of this resolution page. Register for a 14 day evaluation and check your compliance level for free!

Start a Free Trial Product features
Last updated: 03 October 2018
Performance
efficiency, Reliability, Security

Risk level: Medium (should be achieved)

Ensure that your Amazon Elasticsearch (ES) clusters are using the latest version of Elasticsearch engine in order to adhere to AWS best practices and receive the newest Elasticsearch features, benefit from better performance and security and get the latest bug fixes. Elasticsearch is a full-text search engine based on Lucene. Amazon Elasticsearch (ES) is a managed service designed to help you deploy, operate, and scale Elasticsearch clusters within the AWS Cloud.

When running your AWS ES clusters with the latest version of Elasticsearch engine you will benefit from new features and enhancements, better performance, better memory management and resource utilization, bug fixes and security patches for the engine. For example, upgrading your AWS Elasticsearch clusters (domains) version to 6.x will get you all the improvements that come with Elasticsearch 6 (better indexing performance, new data structures, instant aggregations, automatic parallel tasking of reindex, etc) plus the new ones added by AWS such as: support for newer instance types, higher number of supported APIs that can give you finer control over your clusters, and an improved visualization engine (powered by Kibana 5).

Audit

To determine the current version of your Elasticsearch (ES) domains, perform the following:

Using AWS Console

01 Login to the AWS Management Console.

02 Navigate to Elasticsearch (ES) dashboard at https://console.aws.amazon.com/es/.

03 Click on the name (link) of the ES domain that you want to examine. A domain is a collection of resources required to run an AWS Elasticsearch cluster.

04 On the selected domain description page, verify the Elasticsearch version installed, listed as value for the Elasticsearch version attribute:

Elasticsearch version

05 Now check the latest version of the Elasticsearch engine supported by the Amazon ES service. If there is a newer engine version released and supported by AWS Elasticsearch service, your ES clusters should be upgraded to benefit from all the improvements that come with the latest version of Elasticsearch.

06 Repeat steps no. 3 - 5 to verify the Elasticsearch engine version of other AWS ES domains (clusters) available within the current region.

07 Change the AWS region from the navigation bar and repeat the process for the other regions.

Using AWS CLI

01 Run list-domain-names command (OSX/Linux/UNIX) to list the names of all AWS Elasticsearch (ES) domains currently available within the selected region:

aws es list-domain-names
	--region us-east-1

02 The command output should return the requested ES domain names:

{
    "DomainNames": [
        {
            "DomainName": "cc-es-cluster-v2"
        },
        {
            "DomainName": "cc-prod-us-cluster"
        }
    ]
}

03 Run describe-elasticsearch-domain command (OSX/Linux/UNIX) using the ES domain name returned at the previous step and custom query filters to expose the Elasticsearch engine version used by the selected AWS ES domain:

aws es describe-elasticsearch-domain
	--domain-name cc-es-cluster-v2
	--region us-east-1
	--query 'DomainStatus.ElasticsearchVersion'

04 The command output should return the Elasticsearch version currently used by the selected ES domain (cluster):

"2.3"

05 Check the latest version of the Elasticsearch engine supported by the AWS ES service. If there is a newer Elasticsearch engine version released and supported by the ES service, your Amazon Elasticsearch clusters should be upgraded to benefit from all the improvements delivered with the latest version of the engine.

06 Repeat steps no. 3 – 5 to verify the Elasticsearch engine version for other AWS ES domains (clusters) available in the current region.

07 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 - 6 to perform the audit process for other regions.

Remediation / Resolution

To upgrade the Elasticsearch engine version for your AWS ES domain, you must unload the existing data from the cluster to Amazon S3 then upload this data in a new AWS ES cluster, created using the latest version of the Elasticsearch engine. To launch and configure a new Amazon Elasticsearch cluster (domain) with the latest search engine version, perform the following:

Using AWS Console

01 Login to the AWS Management Console.

02 Navigate to Elasticsearch (ES) dashboard at https://console.aws.amazon.com/es/.

03 Click on the ES domain that you want to upgrade (see Audit section part I to identify the right resource).

04 On the selected ES domain description page, click the Configure cluster button from the dashboard top menu to open the cluster configuration page.

05 On the Configure cluster page, copy the selected cluster configuration information such as Instance count, Instance type, Storage Type, EBS volume size, etc.

06 On the Set up access policy page, copy the access policy available in the Add or edit the access policy textbox.

07 Go back to the AWS ES service dashboard and click the Create new domain button from the dashboard top menu to launch a new Elasticsearch domain.

08 On the Define domain page, perform the following actions:

  1. Provide a unique name for the new ES domain in the Elasticsearch domain name box.
  2. Select the latest version of the Elasticsearch engine from the Elasticsearch version dropdown list.
  3. Click Next to continue the setup process.

09 On the Configure cluster page, set the new domain parameters using the configuration details copied at step no. 5 then click Next to define the ES domain access policy.

10 On the Set up access policy page of the new domain, paste the access policy copied at step no. 6 into the Add or edit the access policy box or simply select a pre-configured policy from the Set the domain access policy to dropdown list and edit it to meet the needs of your ES domain. Click Next to continue.

11 On the Review page, verify one more time the domain configuration and its access policy then click Confirm and create to launch your new AWS Elasticsearch domain.

12 Once you have made backups of your existing Elasticsearch data, it’s safe to remove the old Elasticsearch domain in order to stop incurring charges for it. To shut down the cluster, perform the following:

  1. Click on the old ES domain name/link (see Audit section part I to identify the right resource).
  2. On the selected domain description page, click Delete Elasticsearch domain to expand the section panel then click Delete domain button to start the removal process.
  3. Within Delete domain dialog box, check Delete the domain <domain_name> then click the Delete button to confirm the action.

13 Repeat steps no. 3 - 12 to upgrade the Elasticsearch engine version for other AWS ES domains available in the current region.

14 Change the AWS region from the navigation bar and repeat the entire process for other regions.

Using AWS CLI

01 Run describe-elasticsearch-domain command (OSX/Linux/UNIX) using the name of the ES domain that you want to upgrade (see Audit section part I to identify the right resource) to describe the selected domain (cluster) configuration information:

aws es describe-elasticsearch-domain
	--region us-east-1
	--domain-name cc-es-cluster-v2

02 The command output should return the configuration details (metadata) for the selected AWS ES domain:

{
    "DomainStatus": {
        "ElasticsearchClusterConfig": {
            "DedicatedMasterEnabled": false,
            "InstanceCount": 2,
            "ZoneAwarenessEnabled": false,
            "InstanceType": "m4.large.elasticsearch"
        },
        "Endpoint": "search-cc-es-cluster-v2-ljfdatey65mlbaw4d.us-east-1.es.amazonaws.com",
        "Created": true,
        "Deleted": false,
        "DomainName": "cc-es-cluster-v2",
        "EBSOptions": {
            "Iops": 0,
            "VolumeSize": 250,
            "VolumeType": "gp2",
            "EBSEnabled": true
        },
        "SnapshotOptions": {
            "AutomatedSnapshotStartHour": 0
        },
        "DomainId": "123456789012/cc-es-cluster-v2",
        "AccessPolicies": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":\"*\",\"Action\":\"es:*\",\"Resource\":\"arn:aws:es:us-east-1:575392585563:domain/cc-es-cluster-v5/*\",\"Condition\":{\"IpAddress\":{\"aws:SourceIp\":\"172.31.115.28/32\"}}}]}",
        "Processing": false,
        "AdvancedOptions": {
            "rest.action.multi.allow_explicit_index": "true",
            "indices.fielddata.cache.size": ""
        },
        "ElasticsearchVersion": "2.3",
        "ARN": "arn:aws:es:us-east-1:123456789012:domain/cc-es-cluster-v2"
    }
}

03 Run create-elasticsearch-domain command (OSX/Linux/UNIX) using the configuration metadata returned at the previous step to launch a new Amazon Elasticsearch domain with the latest version of Elasticsearch engine:

aws es create-elasticsearch-domain
	--region us-east-1
	--domain-name cc-es-cluster-v5
	--elasticsearch-version 6.2
	--elasticsearch-cluster-config InstanceType=m4.large.elasticsearch,InstanceCount=2
	--ebs-options EBSEnabled=true,VolumeType=standard,VolumeSize=250
	--access-policies '{"Version": "2012-10-17", "Statement": [{"Action": "es:*", "Principal":"*","Effect": "Allow", "Condition": {"IpAddress":{"aws:SourceIp":["172.31.115.28/32"]}}}]}'

04 The command output should return the metadata for the new AWS Elasticsearch domain:

{
    "DomainStatus": {
        "ElasticsearchClusterConfig": {
            "DedicatedMasterEnabled": false,
            "InstanceCount": 2,
            "ZoneAwarenessEnabled": false,
            "InstanceType": "m4.large.elasticsearch"
        },
        "DomainId": "123456789012/cc-es-cluster-v5",
        "Created": true,
        "Deleted": false,
        "EBSOptions": {
            "VolumeSize": 250,
            "VolumeType": "standard",
            "EBSEnabled": true
        },
        "Processing": true,
        "DomainName": "cc-es-cluster-v5",
        "SnapshotOptions": {
            "AutomatedSnapshotStartHour": 0
        },
        "ElasticsearchVersion": "6.2",
        "AccessPolicies": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":\"*\",\"Action\":\"es:*\",\"Resource\":\"arn:aws:es:us-east-1:123456789012:domain/cc-es-cluster-v5/*\",\"Condition\":{\"IpAddress\":{\"aws:SourceIp\":\"172.31.115.28/32\"}}}]}",
        "AdvancedOptions": {
            "rest.action.multi.allow_explicit_index": "true"
        },
        "ARN": "arn:aws:es:us-east-1:123456789012:domain/cc-es-cluster-v5"
    }
}

05 Once you have made backups of your existing data, it is safe to remove the old Elasticsearch domain in order to stop incurring charges for the resource. To shut down it down run delete-elasticsearch-domain command (OSX/Linux/UNIX) using the name of the domain that you want to delete as command parameter:

aws es delete-elasticsearch-domain
	--region us-east-1
	--domain-name cc-es-cluster-v2

06 The command output should return the old AWS Elasticsearch domain metadata:

{
    "DomainStatus": {
        "ElasticsearchClusterConfig": {
            "DedicatedMasterEnabled": false,
            "InstanceCount": 2,
            "ZoneAwarenessEnabled": false,
            "InstanceType": "m4.large.elasticsearch"
        },

	  ...

        "AdvancedOptions": {
            "rest.action.multi.allow_explicit_index": "true",
            "indices.fielddata.cache.size": ""
        },
        "ElasticsearchVersion": "2.3",
        "ARN": "arn:aws:es:us-east-1:123456789012:domain/cc-es-cluster-v2"
    }
}

07 Repeat steps no. 1 - 6 to upgrade the Elasticsearch engine version for other AWS ES domains available in the current region.

08 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 - 7 to perform the process for other regions.

References

Publication date Jun 12, 2017