Idle Elasticsearch Clusters

Cloud Conformity allows you to automate the auditing process of this resolution page. Register for a 14 day evaluation and check your compliance level for free!

Start a Free Trial Product features

Knowledge Base Amazon Web Services Amazon Elasticsearch Service Idle Elasticsearch Clusters

Last updated: 13 May 2020

Risk level: High (not acceptable risk)

Rule ID: ES-022

Identify any Amazon Elasticsearch (ES) clusters that appear to be idle and remove them from your account to help lower the cost of your monthly AWS bill. By default, an AWS ES cluster (domain) is considered "idle" when meets the following criteria:

The average CPU Utilization has been less than 2% for the last 7 days.

- The AWS CloudWatch metrics used to detect idle Elasticsearch clusters are:

CPUUtilization - the percentage of CPU resources used for data nodes within the ES cluster (Units: Percentage).

Note 1: You can easily change the default threshold for this rule on Cloud Conformity console and set your own value for the CPU usage in order to configure the ES clusters idleness based on your requirements.
Note 2: For this rule Cloud Conformity assumes that your Elasticsearch domains (clusters) are tagged with "Role" and "Owner" tags which provide visibility into their usage profile and help you decide whether it`s safe or not to terminate these resources. Knowing the role and the owner of these resources before you take the decision to terminate them is very important because, for example, a minimal CPU utilization recorded within a 48 hour period may mean that the cluster is being idle or not being used at all.

This rule resolution is part of the Cloud Conformity Security & Compliance tool for AWS

Cost
optimisation

Idle AWS ES clusters represent a good candidate to reduce your monthly AWS costs and avoid accumulating unnecessary usage charges.

Audit

To identify any idle ES clusters currently available in your AWS account, perform the following:

Using AWS Console

01 Login to the AWS Management Console.

02 Navigate to Elasticsearch (ES) dashboard at https://console.aws.amazon.com/es/.

03 Click on the ES domain that you want to examine, e.g.

A domain is a collection of resources required to run an AWS Elasticsearch cluster.

04 Select the Cluster health tab from the dashboard bottom panel.

05 Within the monitoring metrics section, click on the Maximum CPU utilization (Percent) usage graph thumbnail to open the ES cluster CPU usage details box. Inside the Metrics for <ES-cluster-name> dialog box, set the following parameters:

From the Statistic dropdown list, select Average.
From the Period dropdown list, select 1 hour.
From the Time Range list, select Last 1 week.

Once the monitoring data is loaded, verify the cluster CPU usage for the last 7 days. If the average usage (percent) has been less than 2%, e.g.

the selected Elasticsearch cluster qualifies as candidates for idle ES cluster. Click X (close) to return to the ES cluster details page.

06 Now determine the selected cluster role within the stack and its owner by checking the Role and Owner tags values assigned to the ES domain in order to decide whether it's safe or not to terminate the resource. To check for the necessary tags, perform the following:

Click the Manage tags button from the dashboard top menu to access the domain tags.
On the Manage tags page, verify the requested tags and their values:
- Check the Role tag value, available in the Value column, or any Role-like tag value that can provide information about the usage profile of the ES cluster in order to decide if the resource can be terminated or not.
- Check the Owner tag value, available in the Value column, or any Owner-like tag value that can provide the contact information (name, email, phone number) of the resource owner in order to get the confirmation to delete or not the selected ES cluster.

If all conditions outlined at step no. 5 and 6 are met, the selected AWS Elasticsearch cluster is considered "idle" and the ES resource can be terminated in order to stop incurring charges for it.

07 Repeat steps no. 3 – 6 to verify the CPU usage, the role and the owner tags for the rest of the ES clusters provisioned in the current region.

08 Change the AWS region from the navigation bar and repeat the audit process for the other regions.

Using AWS CLI

01 Run list-domain-names command (OSX/Linux/UNIX) to list the names of all AWS Elasticsearch (ES) domains currently available in the selected region:

aws es list-domain-names
	--region us-east-1

02 The command output should return the requested AWS ES domain names:

{
    "DomainNames": [
        {
            "DomainName": "cloudconformity"
        },
        {
            "DomainName": "elasticsearch-cmp"
        },
        {
            "DomainName": "elasticsearch-cvs"
        }

    ]
}

03 Run get-metric-statistics command (OSX/Linux/UNIX) to get the statistics recorded by AWS CloudWatch for the CPUUtilization metric representing the CPU usage of the selected Elasticsearch cluster (domain). The following command example returns the average CPU utilization of an Elasticsearch domain identified by the name "cloudconformity" in AWS account ID (e.g. 123456789012), usage data captured during a 7 days period, using a 1 hour period

aws cloudwatch get-metric-statistics
	--region us-east-1
	--metric-name CPUUtilization
	--start-time 2017-04-05T08:46:05
	--end-time 2017-04-12T08:46:05
	--period 3600
	--namespace AWS/ES
	--statistics Average
	--dimensions Name=DomainName,Value=cloudconformity Name=ClientId,Value=123456789012

04 The command output should return the CPU usage details requested:

{
    "Datapoints": [
        {
            "Timestamp": "2017-04-05T08:46:05Z",
            "Average": 0.031533333333333333,
            "Unit": "Percent"
        },
        {
            "Timestamp": "2017-04-05T09:46:05Z",
            "Average": 0.033499999999999995,
            "Unit": "Percent"
        },
        {
            "Timestamp": "2017-04-05T10:46:05Z",
            "Average": 0.10425,
            "Unit": "Percent"
        },

        ...

        {
            "Timestamp": "2017-04-12T05:46:05Z",
            "Average": 0.030999999999999993,
            "Unit": "Percent"
        },
        {
            "Timestamp": "2017-04-12T06:46:05Z",
            "Average": 0.02833333333333333,
            "Unit": "Percent"
        },
        {
            "Timestamp": "2017-04-12T07:46:05Z",
            "Average": 0.02783333333333333,
            "Unit": "Percent"
        }
    ],
    "Label": "CPUUtilization"
}

05 Run describe-elasticsearch-domain command (OSX/Linux/UNIX) using the name of the selected Elasticsearch domain as identifier and custom query filters to describe the resource Amazon Resource Name (ARN):

aws es describe-elasticsearch-domain
	--domain-name cloudconformity
	--region us-east-1
	--query 'DomainStatus.ARN'

06 The command output should return the requested AWS ES domain ARN:

"arn:aws:es:us-east-1:123456789012:domain/cloudconformity"

07 Now run describe-tags command (OSX/Linux/UNIX) using the ES ARN returned at the previous step as identifier to describe the tags applied to the selected ES domain (cluster). These tags are used to determine the cluster role within your application stack in order to decide whether it's safe or not to terminate the resource:

aws es list-tags
	--region us-east-1
	--arn arn:aws:es:us-east-1:123456789012:domain/cloudconformity

08 The command output should return the requested cluster tags:

{
    "TagList": [
        {
            "Value": "test-es-cluster",
            "Key": "Role"
        },
        {
            "Value": "staging",
            "Key": "Environment"
        }
    ]
}

If the data returned for the step no. 4 and 10 satisfy the conditions set by the conformity rule, the selected Elasticsearch cluster (domain) is considered "idle" and can be safely removed in order to reduce the AWS monthly costs.

09 Repeat steps no. 3 – 8 to verify the CPU usage and the role/owner tags for the rest of the ES clusters provisioned in the current region.

10 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 - 9 to perform the entire audit process for other regions.

Remediation / Resolution

Option 1: Delete any AWS Elasticsearch clusters that are currently running in idle mode. To remove the idle ES clusters (domains), perform the following actions:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Elasticsearch (ES) dashboard at .

03 Click on the ES domain that you want to remove, e.g.

to access its details page (see Audit section part I to identify the right resource)..

04 On the selected domain details page, click Actions to expand the section panel, then click the Delete domain button to initiate the removal process.

05 Within the Delete domain dialog box, select Delete the domain <domain_name> checkbox then click Delete to confirm the action.

06 Repeat steps no. 3 – 5 to remove any other idle AWS Elasticsearch domains available within the current region.

07 Change the AWS region from the navigation bar and repeat the remediation/resolution process for other regions.

Using AWS CLI

01 Run delete-elasticsearch-domain command (OSX/Linux/UNIX) to remove the selected idle Elasticsearch domain (cluster) from your AWS account. The following command example deletes an AWS ES domain identified by the name "cloudconformity", available within the US East (N. Virginia) region:

aws es delete-elasticsearch-domain
	--domain-name cloudconformity
	--region us-east-1

02 The command output should return the deleted AWS ES domain metadata:

{
    "DomainStatus": {
        "ElasticsearchClusterConfig": {
            "DedicatedMasterEnabled": false,
            "InstanceCount": 2,
            "ZoneAwarenessEnabled": false,
            "InstanceType": "m4.large.elasticsearch"
        },
        "Endpoint": "search-cloud...us-east-1.es.amazonaws.com",
        "Created": true,
        "Deleted": true,
        "DomainName": "cloudconformity",

        ...

        "EBSOptions": {
            "Iops": 0,
            "VolumeSize": 75,
            "VolumeType": "gp2",
            "EBSEnabled": true
        },
        "SnapshotOptions": {
            "AutomatedSnapshotStartHour": 0
        },
        "DomainId": "123456789012/cloudconformity",
        "Processing": true,
        "AdvancedOptions": {
            "rest.action.multi.allow_explicit_index": "true",
            "indices.fielddata.cache.size": ""
        },
        "ARN": "arn:aws:es:us-east-1:123456789012:domain/cloudconformity"
    }
}

03 Repeat step no. 1 and 2 to remove other idle AWS Elasticsearch domains provisioned within the current region.

04 Change the AWS region by updating the --region command parameter value and repeat the entire process for other regions.

Option 2: Disable the rule check. If the selected idle Elasticsearch domain (cluster) is needed (its role within your application stack/environment is important), you may want turn off the conformity rule check for the specified AWS ES cluster from the Cloud Conformity console.

References

AWS Command Line Interface (CLI) Documentation
es
list-domain-names
describe-elasticsearch-domain
list-tags
delete-elasticsearch-domain
cloudwatch
get-metric-statistics

Publication date May 2, 2016

Idle Elasticsearch Clusters

Audit

Using AWS Console

Using AWS CLI

Remediation / Resolution

Using AWS Console

Using AWS CLI

References

Related Elasticsearch rules