Logo of Trend Micro

ElasticSearch Free Storage Space

Trend Micro Cloud One™ – Conformity is a continuous assurance tool that provides peace of mind for your cloud infrastructure, delivering over 750 automated best practice checks.

Risk level: High (not acceptable risk)
Rule ID: ES-014

Identify any Amazon ElasticSearch (ES) clusters that appear to run low on disk space and scale them up (add EBS-based storage) to help mitigate any issues triggered by insufficient disk space and improve their I/O performance. The default threshold value set for the amount of free storage space is 10% as any value below this could have a serious impact on your ES clusters performance. For example, if the free storage space becomes dangerously low, your clusters can start blocking incoming write requests.
The AWS CloudWatch metric used to detect ElasticSearch clusters with low free storage space is:
FreeStorageSpace – the amount of available storage space for all data nodes in the cluster (Units: Megabytes). AWS ES service throws an "ClusterBlockException" error when this metric reaches 0.

This rule can help you work with the AWS Well-Architected Framework

This rule resolution is part of the Cloud Conformity Security & Compliance tool for AWS

Performance
efficiency

Low disk space leads to instability and slowdowns. Detecting ES clusters that run low on disk space is crucial, especially when these AWS resources are used in production because when ES clusters run out of free storage space, basic write operations like adding documents and creating indices begin to fail.

Note: You can change the default threshold value (10%) for this rule on Cloud Conformity console and set your own value for the amount of available storage space to configure the storage limits for your ElasticSearch clusters.

Audit

To identify AWS ElasticSearch clusters that run low on disk space, perform the following actions:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to ElasticSearch (ES) dashboard at https://console.aws.amazon.com/es/.

03 Choose the ES domain (cluster) that you want to examine and click on the domain name (link) to access its configuration page.

04 Select Overview tab from the dashboard top panel and check the Instance count and EBS volume size attribute values in order to compute (volume size x instance count) the amount of storage allocated for the selected cluster data nodes, in gigabytes.

05 Select the Monitoring tab from the dashboard top panel.

06 On the monitoring panel displayed for the selected cluster, click on the FreeStorageSpace (Megabytes) Statistic: Sum usage graph thumbnail to open the dialog box that contains details about the amount of free disk space available. Inside Metrics for <domain_name> dialog box, set the following parameters:

  1. From the Statistic dropdown list, select Sum.
  2. From the Time Range list, select Last 24 Hours.
  3. From the Period dropdown list, select 1 Hour.

Once the monitoring data is loaded, verify the total free storage space available (in megabytes) for the selected ES cluster.

07 Based on storage information verified at step no. 4 and 6, if the amount of free storage space is less than 10%, the selected Amazon Elasticsearch cluster has insufficient disk space to perform optimally, therefore adding EBS-based storage to the existing data nodes is highly recommended.

08 Repeat steps no. 3 – 7 to determine if other AWS ES clusters, provisioned in the current region, run low on disk space.

09 Change the AWS region from the navigation bar and repeat the process for the other regions.

Using AWS CLI

01 Run list-domain-names command (OSX/Linux/UNIX) to list the names of all Amazon ElasticSearch domains (clusters) currently available within the selected region: 

aws es list-domain-names 
	--region us-east-1

02 The command output should return the requested ES domain names: 

{
    "DomainNames": [
        {
            "DomainName": "cc-production-cluster"
        },
        {
            "DomainName": "cc-es-sandbox-cluster"
        }
    ]
}

03 Run describe-elasticsearch-domain command (OSX/Linux/UNIX) using the ElasticSearch domain name returned at the previous step as identifier and custom query filters to get details about the cluster data nodes and their allocated storage: 

aws es describe-elasticsearch-domain
	--region us-east-1
	--domain-name cc-production-cluster
	--query 'DomainStatus.[ElasticsearchClusterConfig.InstanceCount,EBSOptions.VolumeSize]'

04 The command output should return an array that contains two values. First value represents the number of data nodes provisioned within the selected cluster and the second value represents the volume size of each data node available (gigabytes). This information is required to compute (volume size x instance count) the amount of storage allocated for the selected ES cluster: 

[
    2,
    10
]

05 Run get-metric-statistics command (OSX/Linux/UNIX) to get the statistics recorded by AWS CloudWatch for the FreeStorageSpace metric. The following command example returns the total amount of free storage space available for data nodes within an AWS Elasticsearch cluster, identified by the name "cc-production-cluster", storage usage data collected over a period of 24 hours, using 1 hour period as the granularity of the returned datapoints: 

aws cloudwatch get-metric-statistics
	--region us-east-1
	--metric-name FreeStorageSpace
	--start-time 2018-02-01T12:21:05
	--end-time 2018-02-02T12:21:05
	--period 3600
	--namespace AWS/ES
	--statistics Sum
	--dimensions Name=DomainName,Value=cc-production-cluster

06 The command output should return the storage details requested (in megabytes): 

{
    "Datapoints": [
        {
            "Timestamp": "2018-02-01T12:21:05Z",
            "Sum": 1,410.4770,
            "Unit": "Megabytes"
        },
        {
            "Timestamp": "2018-02-01T13:21:05Z",
            "Sum": 1,410.4765,
            "Unit": "Megabytes"
        },
        {
            "Timestamp": "2018-02-01T14:21:05Z",
            "Sum": 1,410.4760,
            "Unit": "Megabytes"
        },
 
        ...
 
        {
            "Timestamp": "2018-02-02T10:21:05Z",
            "Sum": 1,410.4700,
            "Unit": "Megabytes"
        },
        {
            "Timestamp": "2018-02-02T11:21:05Z",
            "Sum": 1,410.4695,
            "Unit": "Megabytes"
        },
        {
            "Timestamp": "2018-02-02T12:21:05Z",
            "Sum": 1,410.4690,
            "Unit": "Megabytes"
        }
    ],
    "Label": "FreeStorageSpace"
}

07 Based on storage information verified at step no. 4 and 6, if the amount of free storage space available is less than 10%, the selected Amazon Elasticsearch cluster has insufficient disk space to perform optimally.

08 Repeat steps no. 3 – 7 to determine if other AWS ES clusters, created in the current region, run low on disk space.

09 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 - 8 to perform the audit process for other regions.

Remediation / Resolution

To expand the storage space for AWS Elasticsearch clusters that run low on disk space, you can scale them up by adding storage to the existing data nodes volumes. To scale up and recover from the lack of free disk space, perform the following actions:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Elasticsearch (ES) dashboard at https://console.aws.amazon.com/es/.

03 Click on the ES cluster that you want to reconfigure (see Audit section part I to identify the right AWS resource).

04 On the selected cluster description page, click Configure cluster button from the dashboard top menu to open the resource configuration page.

05 On the Configure cluster page, within Storage configuration section, inside EBS volume size box, set a new size value (in gigabytes) to upgrade the storage size for each data node volume provisioned within the selected cluster.

06 Click Submit to apply the storage size configuration changes. The selected AWS ES cluster status should change from Active to Processing and back to Active, once the service finishes processing your changes.

07 Repeat steps no. 3 - 6 to expand the storage space for other AWS Elasticsearch clusters that run low on disk space, available in the current region.

08 Change the AWS region from the navigation bar and repeat the entire process for other regions.

Using AWS CLI

01 Run update-elasticsearch-domain-config command (OSX/Linux/UNIX) using the name of the Elasticsearch domain that you want to reconfigure (see Audit section part II to identify the right resource) to add storage to the existing data nodes volumes provisioned within the selected AWS ES cluster: 

aws es update-elasticsearch-domain-config
	--region us-east-1
	--domain-name cc-production-cluster
	--ebs-options EBSEnabled=true,VolumeType="gp2",VolumeSize=50

02 The command output should return the new configuration metadata for the upgraded AWS Elasticsearch cluster: 

{
    "DomainConfig": {
        "ElasticsearchClusterConfig": {
            "Status": {
                "PendingDeletion": false,
                "State": "Active",
                "CreationDate": 1517569277.153,
                "UpdateVersion": 6,
                "UpdateDate": 1517570021.757
            },
            "Options": {
                "DedicatedMasterEnabled": false,
                "InstanceCount": 2,
                "ZoneAwarenessEnabled": false,
                "InstanceType": "c4.large.elasticsearch"
            }
        },
 
        ...
 
        "EBSOptions": {
            "Status": {
                "PendingDeletion": false,
                "State": "Processing",
                "CreationDate": 1517569277.153,
                "UpdateVersion": 16,
                "UpdateDate": 1517578473.523
            },
            "Options": {
                "VolumeSize": 50,
                "VolumeType": "gp2",
                "EBSEnabled": true
            }
        },
        "EncryptionAtRestOptions": {
            "Status": {
                "PendingDeletion": false,
                "State": "Active",
                "CreationDate": 1517569277.153,
                "UpdateVersion": 6,
                "UpdateDate": 1517570021.757
            }
        }
    }
}

03 Repeat step no. 1 and 2 to expand the storage space for other AWS Elasticsearch clusters that run low on disk space, available in the current region.

04 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 – 3 to perform the process for other regions.

References

Publication date Feb 3, 2018

Related Elasticsearch rules

Unlock the Remediation Steps

Gain free unlimited access
to our full Knowledge Base

Over 750 rules & best practices
for AWS and Azure

You are auditing:

ElasticSearch Free Storage Space

Risk level: High