Elasticsearch Instance Counts

Cloud Conformity allows you to automate the auditing process of this resolution page. Register for a 14 day evaluation and check your compliance level for free!

Start a Free Trial Product features

Knowledge Base Amazon Web Services Amazon Elasticsearch Service Elasticsearch Instance Counts

Last updated: 13 April 2018

Risk level: Medium (should be achieved)

Rule ID: ES-008

Ensure that the number of Amazon Elasticsearch cluster instances (including dedicated master instances) provisioned in your AWS account has not reached the limit quota established by your organization for the Elasticsearch workload deployed. By default, Cloud Conformity sets a threshold value of 10 for the maximum number of provisioned Elasticsearch instances, however, you have the ability to adjust this threshold based on your organization resources policy upon enabling this rule. Once you define your own threshold for the maximum number of Elasticsearch instances that you need to run across all AWS regions, Cloud Conformity engine will start to continuously scan your account for Elasticsearch cluster instances and when the number of instances reach the specified threshold you will get notified via communication channels configured within your Cloud Conformity account. If the Elasticsearch compute capacity limit quota defined for your AWS account is reached, you can raise an AWS support case where you can request to limit the number of provisioned Elasticsearch cluster instances.

This rule can help you with the following compliance standards:

APRA
MAS

This rule can help you work with the AWS Well-Architected Framework

This rule resolution is part of the Cloud Conformity Security & Compliance tool for AWS

Security

Monitoring and configuring limits for the maximum number of Elasticsearch (ES) instances provisioned within your AWS account will help you to manage better your Elasticsearch compute resources, prevent unexpected charges on your AWS bill and act fast to mitigate attacks that can use Elasticsearch resources. For example, users within your organization can create more Elasticsearch instances than the number established in the company resources policy, exceeding the monthly budget allocated for cloud computing resources. Another example could be a misconfiguration in your CloudFormation templates that can lead to launching more cluster instances than required. Also, if your AWS account security is compromised and the attackers gain the capability to provision a large number of Elasticsearch instances in order to run their malicious data analytics tools, you risk to accrue a lot of AWS charges in a short period of time.

Note: The threshold for the maximum number of Elasticsearch cluster instances per AWS account set for this conformity rule is 10 (default value).

Audit

To determine the number of Amazon Elasticsearch instances (both data instances and dedicated master instances) currently available within your AWS account, perform the following:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Elasticsearch (ES) dashboard at https://console.aws.amazon.com/es/.

03 Click on the ES domain (link) that you want to examine. A domain is a collection of resources required to run an AWS Elasticsearch cluster.

04 Click Configure cluster button from the ES dashboard top menu to access the configuration information for the selected cluster.

05 Check the total number of Elasticsearch instances provisioned for the selected cluster, listed in the Instance count and Dedicated master instance count fields, e.g.:

Instance count and Dedicated master instance count

06 Go back to the AWS ES dashboard and repeat steps no. 3 – 5 to determine the number of instances provisioned by all other Elasticsearch clusters available in the current region.

07 Change the AWS region from the navigation bar and repeat steps no. 3 – 6 for all other regions. If the total number of Elasticsearch cluster instances provisioned in your AWS account is greater than 10, the defined threshold was exceeded, therefore you must take action and build an AWS support case to limit the number of AWS Elasticsearch instances based on your requirements.

Using AWS CLI

01 Run list-domain-names command (OSX/Linux/UNIX) to list the names of all AWS Elasticsearch (ES) domains currently available in the selected region:

aws es list-domain-names
	--region us-east-1

02 The command output should return the requested AWS ES domain names:

{
    "DomainNames": [
        {
            "DomainName": "cc-es-cluster"
        },
        {
            "DomainName": "cc-es-cmpa-cluster"
        },
        {
            "DomainName": "cc-es-cvsa-cluster"
        }

    ]
}

03 Run describe-elasticsearch-domain command (OSX/Linux/UNIX) using the name of the Elasticsearch domain that you want to examine as identifier and custom query filters to return the number of data instances and dedicated master instances provisioned by the current ES cluster:

aws es describe-elasticsearch-domain
	--domain-name cc-es-cluster
	--region us-east-1
	--query 'DomainStatus.ElasticsearchClusterConfig.[InstanceCount,DedicatedMasterCount]'

04 The command output should return the number of instances for the selected AWS ES cluster. The first value returned represents the number of data instances and second value the number of dedicated master instances:

05 Repeat step no. 3 and 4 to determine the number of instances provisioned by all other AWS Elasticsearch clusters, available in the current region.

06 Repeat steps no. 1 – 5 to perform the process for all other AWS regions. The describe-elasticsearch-domain command output should return an array with the current number of ES cluster instances (data instances and dedicated master instances), available in the selected region. If the total number of ES instances within all the arrays returned is greater than 10 (combined), the recommended limit threshold was exceeded, therefore you must take action and raise an AWS support case to limit the number of Elasticsearch cluster instances that can be provisioned in your account.

Remediation / Resolution

To build an AWS support case to limit the number of provisioned Elasticsearch instances based on your requirements, perform the following actions:

Note: Requesting a limit for the number of AWS Elasticsearch instances per region using the AWS API via Command Line Interface (CLI) is not currently supported.

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to AWS Support Center dashboard at https://console.aws.amazon.com/support/.

03 In the left navigation panel, choose Create Case to create a new AWS support case.

04 On the Create Case page, perform the following:

Under Regarding, select Service Limit Increase.

Choose Elasticsearch Service from the Limit Type dropdown list.

In the Request 1 section, perform the following actions:

From the Region dropdown list, select the AWS region where you need to limit the launch of Elasticsearch cluster instances.
Select Max Instance Count per Domain from the Limit dropdown list.
In the New limit value box, enter the limit value to request for the number of provisioned instances.

In the Use Case Description textbox, enter a brief description where you explain the limit request so that AWS support can evaluate your case promptly.

From Supported Language, choose your preferred correspondence language for the current case.

Under Contact method, select a preferred contact method that AWS support team can use to respond to your request.

Click Submit to send the limit request to Amazon Web Services. A customer support representative will contact you shortly.

References

AWS Documentation
Amazon Elasticsearch Service FAQs
Amazon Elasticsearch Service Limits
Managing Amazon Elasticsearch Service Domains

AWS Command Line Interface (CLI) Documentation
es
list-domain-names

Publication date Sep 13, 2017

Elasticsearch Instance Counts

Audit

Using AWS Console

Using AWS CLI

Remediation / Resolution

Using AWS Console

References

Related Elasticsearch rules