Idle Elasticsearch Clusters

01 Login to the AWS Management Console.

02 Navigate to Elasticsearch (ES) dashboard at https://console.aws.amazon.com/es/.

03 Click on the ES domain that you want to examine, e.g.

A domain is a collection of resources required to run an AWS Elasticsearch cluster.

04 Select the Cluster health tab from the dashboard bottom panel.

05 Within the monitoring metrics section, click on the Maximum CPU utilization (Percent) usage graph thumbnail to open the ES cluster CPU usage details box. Inside the Metrics for <ES-cluster-name> dialog box, set the following parameters:

1. From the Statistic dropdown list, select Average.
2. From the Period dropdown list, select 1 day.
3. From the Time Range list, select Last 1 week.

Once the monitoring data is loaded, verify the cluster CPU usage for the last 7 days. If the average usage (percent) has been less than 2%, e.g.

the selected Elasticsearch cluster qualifies as candidates for idle ES cluster. Click X (close) to return to the ES cluster details page.

06 Now determine the selected cluster role within the stack and its owner by checking the Role and Owner tags values assigned to the ES domain in order to decide whether it's safe or not to terminate the resource. To check for the necessary tags, perform the following:

1. Click the Manage tags button from the dashboard top menu to access the domain tags.
2. On the Manage tags page, verify the requested tags and their values:
   • Check the Role tag value, available in the Value column, or any Role-like tag value that can provide information about the usage profile of the ES cluster in order to decide if the resource can be terminated or not.
   • Check the Owner tag value, available in the Value column, or any Owner-like tag value that can provide the contact information (name, email, phone number) of the resource owner in order to get the confirmation to delete or not the selected ES cluster.

If all conditions outlined at step no. 5 and 6 are met, the selected AWS Elasticsearch cluster is considered "idle" and the ES resource can be terminated in order to stop incurring charges for it.

07 Repeat steps no. 3 – 6 to verify the CPU usage, the role and the owner tags for the rest of the ES clusters provisioned in the current region.

08 Change the AWS region from the navigation bar and repeat the audit process for the other regions.

01 Run list-domain-names command (OSX/Linux/UNIX) to list the names of all AWS Elasticsearch (ES) domains currently available in the selected region:

aws es list-domain-names
	--region us-east-1

02 The command output should return the requested AWS ES domain names:

{
    "DomainNames": [
        {
            "DomainName": "cloudconformity"
        },
        {
            "DomainName": "elasticsearch-cmp"
        },
        {
            "DomainName": "elasticsearch-cvs"
        }

    ]
}

03 Run get-metric-statistics command (OSX/Linux/UNIX) to get the statistics recorded by AWS CloudWatch for the CPUUtilization metric representing the CPU usage of the selected Elasticsearch cluster (domain). The following command example returns the average CPU utilization of an Elasticsearch domain identified by the name "cloudconformity" in AWS account ID (e.g. 123456789012), usage data captured during a 7 days period, using a 1 day period

aws cloudwatch get-metric-statistics
	--region us-east-1
	--metric-name CPUUtilization
	--start-time 2017-04-05T08:46:05
	--end-time 2017-04-12T08:46:05
	--period 86400
	--namespace AWS/ES
	--statistics Average
	--dimensions Name=DomainName,Value=cloudconformity Name=ClientId,Value=123456789012

04 The command output should return the CPU usage details requested:

{
    "Datapoints": [
        {
            "Timestamp": "2017-04-05T08:46:05Z",
            "Average": 0.031533333333333333,
            "Unit": "Percent"
        },
        {
            "Timestamp": "2017-04-06T08:46:05Z",
            "Average": 0.033499999999999995,
            "Unit": "Percent"
        },
        {
            "Timestamp": "2017-04-07T08:46:05Z",
            "Average": 0.10425,
            "Unit": "Percent"
        },

        ...

        {
            "Timestamp": "2017-04-11T08:46:05Z",
            "Average": 0.02833333333333333,
            "Unit": "Percent"
        },
        {
            "Timestamp": "2017-04-12T08:46:05Z",
            "Average": 0.02783333333333333,
            "Unit": "Percent"
        }
    ],
    "Label": "CPUUtilization"
}

05 Run describe-elasticsearch-domain command (OSX/Linux/UNIX) using the name of the selected Elasticsearch domain as identifier and custom query filters to describe the resource Amazon Resource Name (ARN):

aws es describe-elasticsearch-domain
	--domain-name cloudconformity
	--region us-east-1
	--query 'DomainStatus.ARN'

06 The command output should return the requested AWS ES domain ARN:

"arn:aws:es:us-east-1:123456789012:domain/cloudconformity"

07 Now run describe-tags command (OSX/Linux/UNIX) using the ES ARN returned at the previous step as identifier to describe the tags applied to the selected ES domain (cluster). These tags are used to determine the cluster role within your application stack in order to decide whether it's safe or not to terminate the resource:

aws es list-tags
	--region us-east-1
	--arn arn:aws:es:us-east-1:123456789012:domain/cloudconformity

08 The command output should return the requested cluster tags:

{
    "TagList": [
        {
            "Value": "test-es-cluster",
            "Key": "Role"
        },
        {
            "Value": "staging",
            "Key": "Environment"
        }
    ]
}

If the data returned for the step no. 4 and 10 satisfy the conditions set by the conformity rule, the selected Elasticsearch cluster (domain) is considered "idle" and can be safely removed in order to reduce the AWS monthly costs.

09 Repeat steps no. 3 – 8 to verify the CPU usage and the role/owner tags for the rest of the ES clusters provisioned in the current region.

10 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 - 9 to perform the entire audit process for other regions.

01 Sign in to the AWS Management Console.

02 Navigate to Elasticsearch (ES) dashboard at .

03 Click on the ES domain that you want to remove, e.g.

to access its details page (see Audit section part I to identify the right resource)..

04 On the selected domain details page, click Actions to expand the section panel, then click the Delete domain button to initiate the removal process.

05 Within the Delete domain dialog box, select Delete the domain <domain_name> checkbox then click Delete to confirm the action.

06 Repeat steps no. 3 – 5 to remove any other idle AWS Elasticsearch domains available within the current region.

07 Change the AWS region from the navigation bar and repeat the remediation/resolution process for other regions.

01 Run delete-elasticsearch-domain command (OSX/Linux/UNIX) to remove the selected idle Elasticsearch domain (cluster) from your AWS account. The following command example deletes an AWS ES domain identified by the name "cloudconformity", available within the US East (N. Virginia) region:

aws es delete-elasticsearch-domain
	--domain-name cloudconformity
	--region us-east-1

02 The command output should return the deleted AWS ES domain metadata:

{
    "DomainStatus": {
        "ElasticsearchClusterConfig": {
            "DedicatedMasterEnabled": false,
            "InstanceCount": 2,
            "ZoneAwarenessEnabled": false,
            "InstanceType": "m4.large.elasticsearch"
        },
        "Endpoint": "search-cloud...us-east-1.es.amazonaws.com",
        "Created": true,
        "Deleted": true,
        "DomainName": "cloudconformity",

        ...

        "EBSOptions": {
            "Iops": 0,
            "VolumeSize": 75,
            "VolumeType": "gp2",
            "EBSEnabled": true
        },
        "SnapshotOptions": {
            "AutomatedSnapshotStartHour": 0
        },
        "DomainId": "123456789012/cloudconformity",
        "Processing": true,
        "AdvancedOptions": {
            "rest.action.multi.allow_explicit_index": "true",
            "indices.fielddata.cache.size": ""
        },
        "ARN": "arn:aws:es:us-east-1:123456789012:domain/cloudconformity"
    }
}

03 Repeat step no. 1 and 2 to remove other idle AWS Elasticsearch domains provisioned within the current region.

04 Change the AWS region by updating the --region command parameter value and repeat the entire process for other regions.