Logo of Trend Micro

Elasticsearch General Purpose SSD

Trend Micro Cloud One™ – Conformity is a continuous assurance tool that provides peace of mind for your cloud infrastructure, delivering over 750 automated best practice checks.

Risk level: Medium (should be achieved)
Rule ID: ES-001

Ensure that your Amazon Elasticsearch (ES) clusters are using General Purpose SSD (gp2) data nodes instead of Provisioned IOPS SSD (io1) nodes for cost-effective storage that fits a broad range of workloads. Cloud Conformity recommends converting your io1 ES EBS nodes to gp2-type nodes in order to lower the cost of your monthly AWS bill while maintaining the I/O performance at the same level. For example, if your Elasticsearch application requirements are 150GB storage and 1000 IOPS (input/output operations per second) you can save roughly $68 per month using gp2 nodes instead of io1 nodes as an io1-type EBS data node with 1000 IOPS and 150GB storage costs $113.35/month while a gp2 node with 1000 IOPS and 335GB storage costs only $45.25/month (as of November 2016).

This rule can help you work with the AWS Well-Architected Framework

This rule resolution is part of the Cloud Conformity Security & Compliance tool for AWS

Cost
optimisation

Using General Purpose SSD (gp2) data nodes instead of Provisioned IOPS SSD (io1) nodes represents a good strategy to cut down on Elasticsearch service storage costs because for gp2-type nodes you only pay for the storage compared to io1 nodes where you pay for both storage and I/O operations. Converting existing io1 resources to gp2 is often possible by configuring larger storage which gives higher baseline performance of IOPS for a lower cost.

Note: The storage settings outlined in this conformity rule do not apply to dedicated master nodes provisioned within your AWS Elasticsearch clusters.

Audit

To determine the type of your existing Elasticsearch (ES) data nodes, perform the following:

Using AWS Console

01 Login to the AWS Management Console.

02 Navigate to Elasticsearch (ES) dashboard at https://console.aws.amazon.com/es/.

03 Click on the ES domain that you want to examine, e.g. https://goo.gl/Y3uq59. A domain is a collection of resources required to run an AWS Elasticsearch cluster.

04 On the selected domain description page, click the Configure cluster button from the dashboard top menu to access the ES cluster configuration page.

05 On the Configure cluster page, in the Storage configuration section, verify the storage type used by the cluster nodes by checking the EBS volume type attribute value currently selected. If the current value is set to Provisioned IOPS (SSD):

verify the storage type used by the cluster nodes by checking the EBS volume type attribute value currently selected. If the current value is set to Provisioned IOPS (SSD)

the node type used is Provisioned IOPS SSD, therefore the selected Elasticsearch cluster nodes are not optimized with respect to cost. To convert an ES Provisioned IOPS SSD (io1) EBS node to an ES General Purpose SSD (gp2) node, follow the steps listed in the Remediation/Resolution section of the conformity rule.

06 Repeat steps no. 3 - 5 to verify the node type of other Elasticsearch domains (clusters) available within the current region.

07 Change the AWS region from the navigation bar and repeat the process for the other regions.

Using AWS CLI

01 Run list-domain-names command (OSX/Linux/UNIX) to list the names of all AWS Elasticsearch (ES) domains currently available in the selected region: 

aws es list-domain-names
	--region us-east-1

02 The command output should return the requested ES domain names: 

{
    "DomainNames": [
        {
            "DomainName": "cloudconformity"
        },
        {
            "DomainName": "cc-es-cluster-v1"
        }
    ]
}

03 Run describe-elasticsearch-domain command (OSX/Linux/UNIX) using the ES domain name returned at the previous step and custom query filters to expose the EBS node(s) type for the selected domain (cluster): 

aws es describe-elasticsearch-domain
	--domain-name cloudconformity
	--region us-east-1
	--query 'DomainStatus.EBSOptions.VolumeType'

04 The command output should return the node(s) type used by the selected ES cluster: 

"io1"

If the value returned by the command output is "io1", the node type used is Provisioned IOPS SSD, therefore the selected Elasticsearch cluster nodes are not optimized to run as cost-effective EBS storage.

05 Repeat steps no. 3 and 4 to check the node type of other Elasticsearch domains (clusters) created in the current region

06 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 - 5 to perform the audit process for other regions.

Remediation / Resolution

To convert your AWS Elasticsearch Provisioned IOPS SSD (io1) nodes to General Purpose SSD (gp2) nodes, perform the following:

Using AWS Console

01 Login to the AWS Management Console.

02 Navigate to Elasticsearch (ES) dashboard at https://console.aws.amazon.com/es/.

03 Click on the ES domain that you want to reconfigure (see Audit section part I to identify the right AWS resource).

04 On the selected ES domain description page, click the Configure cluster button from the dashboard top menu to open the cluster configuration page.

05 On the Configure cluster page, inside the Storage configuration section, select General Purpose (SSD) from the EBS volume type dropdown list:

select General Purpose (SSD) from the EBS volume type dropdown list

06 In the EBS volume size box, increase the cluster node(s) size to get the necessary baseline performance (i.e. the number of IOPS required).

07 Click Submit to apply the storage configuration changes.

08 In the Change cluster configuration dialog box, click OK to confirm the change action. The ES domain status should change from Active to Processing and back to Active once the service finishes processing your changes.

09 Repeat steps no. 3 - 8 to change the storage type for other Elasticsearch domains (clusters) available within the current region.

10 Change the AWS region from the navigation bar and repeat the process for other regions.

Using AWS CLI

01 Run update-elasticsearch-domain-config command (OSX/Linux/UNIX) using the name of the Elasticsearch domain that you want to reconfigure (see Audit section part II to identify the right ES resource) to convert the selected cluster storage type to General Purpose SSD (gp2): 

aws es update-elasticsearch-domain-config
	--domain-name cloudconformity
	--region us-east-1
	--ebs-options EBSEnabled=true,VolumeType="gp2",VolumeSize=335

02 The command output should return the new configuration metadata for the modified Elasticsearch domain (cluster): 

{
    "DomainConfig": {
        "ElasticsearchClusterConfig": {
            "Status": {
                "PendingDeletion": false,
                "State": "Active",
                "CreationDate": 1480506921.905,
                "UpdateVersion": 5,
                "UpdateDate": 1480507483.027
            },
            "Options": { ... }
        },

        ...


        "EBSOptions": {
            "Status": {
                "PendingDeletion": false,
                "State": "Processing",
                "CreationDate": 1480506921.905,
                "UpdateVersion": 16,
                "UpdateDate": 1480512056.099
            },
            "Options": {
                "VolumeSize": 335,
                "VolumeType": "gp2",
                "EBSEnabled": true
            }
        },


        ...


        "AccessPolicies": {
            "Status": {
                "PendingDeletion": false,
                "State": "Active",
                "CreationDate": 1480506921.905,
                "UpdateVersion": 5,
                "UpdateDate": 1480507483.027
            },
            "Options": "{ ... }"
        }
    }
}

03 Repeat step no. 1 and 2 to change the storage type for other Elasticsearch domains (clusters) available within the current region.

04 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 - 3 to perform the entire process for other regions.

References

Publication date Dec 3, 2016

Related Elasticsearch rules

Unlock the Remediation Steps

Gain free unlimited access
to our full Knowledge Base

Over 750 rules & best practices
for AWS and Azure

You are auditing:

Elasticsearch General Purpose SSD

Risk level: Medium