Cloud Conformity allows you to automate the auditing process of this resolution page. Register for a 14 day evaluation and check your compliance level for free!
Start a Free Trial Product featuresEnsure that all active Amazon Elasticsearch (ES) Reserved Instance purchases are reviewed every 7 days to make sure that no unwanted RI purchase has been placed recently.
This rule can help you work with the AWS Well-Architected Framework
This rule resolution is part of the Cloud Conformity Cost Management tool for AWS
By verifying your Elasticsearch Reserved Instance purchases on a regular basis you can detect and cancel any unwanted purchases placed accidentally or intentionally within your AWS account in order to avoid unexpected charges on your AWS bill.
Note: You can change the default threshold value (i.e. 7 days) set for the review time frame within the conformity rule settings, on your Cloud Conformity account console.
To identify the active Elasticsearch Reserved Instance purchases placed recently within your AWS account for review purposes, perform the following:
01 Sign in to AWS Management Console.
02 Navigate to Elasticsearch (ES) dashboard at https://console.aws.amazon.com/es/.
03 In the left navigation panel, choose Reserved Instances.
04 Select the Reserved Instance (RI) that you want to examine and check the values listed within Term (year) and Remaining Days columns. Based on these values calculate if the ES resource has been purchased recently (i.e. within the last 7 days). If the selected AWS Elasticsearch Reserved Instance has been purchased recently and you are unaware of this purchase, check your AWS CloudTrail logs or contact Amazon Web Services using the Support Center console to resolve the unwanted Elasticsearch RI purchase issue (see Remediation/Resolution section for more information).
05 Repeat step no. 4 to determine the purchase date for the rest of AWS Elasticsearch RIs available in the current region.
06 Change the AWS region from the navigation bar and repeat the process for other regions.
01 Run describe-reserved-elasticsearch-instances command (OSX/Linux/UNIX) using custom query filters to list the IDs of all Elasticsearch Reserved Instances available in the selected AWS region:
aws es describe-reserved-elasticsearch-instances --region us-east-1 --output table --query 'ReservedElasticsearchInstances[*].ReservedElasticsearchInstanceId'
02 The command output should return a table with the requested ES RI IDs:
---------------------------------------- | ReservedElasticsearchInstanceIds | +--------------------------------------+ | abcdabcd-abcd-1234-abcd-abcdabcdabcd | | aaaaaaaa-bbbb-cccc-dddd-bbbbbbcccccc | +--------------------------------------+
03 Run describe-reserved-elasticsearch-instances command (OSX/Linux/UNIX) using the ID of the Elasticsearch Reserved Instance that you want to examine as identifier and custom filtering to obtain the date when the selected RI was purchased:
aws es describe-reserved-elasticsearch-instances --region us-east-1 --reserved-elasticsearch-instance-id abcdabcd-abcd-1234-abcd-abcdabcdabcd --query 'ReservedElasticsearchInstances[*].StartTime'
04 The command output should return the timestamp (date) at which the reservation started:
[
"StartTime": 1539274423.430
]
05 The value returned for "StartTime" attribute at the previous step is using the Unix time format, which represents the number of seconds that have passed since midnight UTC of 1 January 1970. To convert the returned value into a human-readable format, run the following command (replace the Unix timestamp with your own timestamp returned at the previous step):
date -d @1507369500.430
06 The command output should return the RI purchase date in a human-readable format:
Thu Oct 11 16:13:43 UTC 2018
07 Repeat steps no. 3 – 6 to determine the purchase date for the rest of Amazon Elasticsearch RIs available in the current region.
08 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 – 7 to perform the entire audit process for other regions.
Case A: Verify Amazon CloudTrail logs from the date when the Elasticsearch RI purchase request was placed to determine the request origin and context. To find and analyze the necessary API logging data, perform the following actions:
01 Sign in to AWS Management Console.
02 Navigate to CloudTrail dashboard at https://console.aws.amazon.com/cloudtrail/.
03 In the left navigation panel, select Trails.
04 Under Name column, select the trail name that you need to examine, available in the same AWS region with the identified Elasticsearch RI purchase (reservation).
05 Within Storage location section, check the name of the S3 bucket used to store the trail log data.
06 Navigate to S3 dashboard at https://console.aws.amazon.com/s3/.
07 Select the S3 bucket used for CloudTrail logging and use the date/time bucket name format (e.g. cloudtrail-logging-bucket/AWSLogs/123456789012/CloudTrail/us-east-1/2018/10/11) to open the right log file for analysis.
08 Based on the log file name (i.e. 123456789012_CloudTrail_us-east-1_20181011T1613Z_aaaabbbbccccdddd.json.gz), identify the CloudTrail log file that contains the API activity recorded on the same date as the unwanted Elasticsearch RI purchase request, click the Actions dropdown button from the dashboard top menu and select Open to download and open the log file in your web browser.
09 Once the right CloudTrail log file is opened, search for the following attributes in order to identify the necessary log record:
10 Identify the right CloudTrail log record based on the attributes listed at the previous step and verify the "userIdentity" attribute value to determine the unwanted Elasticsearch Reserved Instance purchase request origin and context.
11 Repeat steps no. 7 – 10 to verify the request origin and context for other unwanted Elasticsearch Reserved Node (RI) purchases placed in the selected region.
12 Change the AWS region from the navigation bar and repeat the process for other regions.
01 Run describe-trails command (OSX/Linux/UNIX) to list the names of all CloudTrail trails currently available within the selected AWS region:
aws cloudtrail describe-trails --region us-east-1 --output table --query 'trailList[*].Name'
02 The command output should return a table with the requested trail name(s):
--------------------------- | DescribeTrails | +-------------------------+ | cc-prod-env-trail | +-------------------------+
03 Run again describe-trails command (OSX/Linux/UNIX) using the name of the trail returned at the previous step and custom query filters to get the name of the S3 bucket used to store the log files for the selected AWS CloudTrail trail:
aws cloudtrail describe-trails --region us-east-1 --trail-name-list cc-prod-env-trail --query 'trailList[*].S3BucketName'
04 The command output should return the name of the requested S3 bucket:
[
"cc-prod-env-trail-logs"
]
05 Now run list-objects command (OSX/Linux/UNIX) to list the names of all S3 objects available in the selected S3 bucket:
aws s3api list-objects --region us-east-1 --bucket cc-prod-env-trail-logs --query 'Contents[].Key'
06 The command output should return the name of each S3 object (i.e. CloudTrail log file) currently available within the selected S3 bucket:
[
"AWSLogs/123456789012/CloudTrail/us-east-1/2018/10/11/123456789012_CloudTrail_us-east-1_20181011T1613Z_aaaabbbbccccdddd.json.gz",
...
"AWSLogs/123456789012/CloudTrail/us-east-1/2018/10/17/123456789012_CloudTrail_us-east-1_20181017T1235Z_abcdabcdabcdabcd.json.gz"
]
07 Run get-object command (OSX/Linux/UNIX) to get the appropriate CloudTrail log file (e.g. 20181011T1613Z_aaaabbbbccccdddd.json.gz) from the specified S3 bucket and download it to your machine:
aws s3api get-object --region us-east-1 --bucket cc-prod-env-trail-logs --key AWSLogs/123456789012/CloudTrail/us-east-1/2018/10/11/123456789012_CloudTrail_us-east-1_20181011T1613Z_aaaabbbbccccdddd.json.gz 20181011T1613Z_aaaabbbbccccdddd.json.gz
08 The command output should return the GET request metadata:
{
"AcceptRanges": "bytes",
"ContentType": "application/json",
"ContentLength": 5980,
...
"ContentEncoding": "gzip",
"ServerSideEncryption": "AES256",
"Metadata": {}
}
09 Extract and open the required CloudTrail log file, downloaded at the previous step (e.g. 20181011T1613Z_aaaabbbbccccdddd.json.gz), in your favorite text editor.
10 Once the log file is opened, search for the following attributes in order to identify the necessary log record:
11 Identify the right CloudTrail log record based on the attributes listed at the previous step and verify the "userIdentity" attribute value to determine the unwanted Elasticsearch Reserved Instance purchase request origin and context.
12 Repeat steps no. 7 – 11 to verify the request origin and context for other unwanted Elasticsearch RI purchases available in the selected region.
13 Change the AWS region by updating the --region command parameter value and perform the entire remediation process for other regions.
Case B: To mitigate unwanted AWS Elasticsearch Reserved Instance purchase requests you can contact Amazon Web Services and request the RI purchase cancellation. To create the necessary case using the AWS Support Center console, perform the following:
Note: Requesting Amazon to cancel unwanted Elasticsearch Reserved Instance purchase requests using AWS Management Console or AWS API via Command Line Interface (CLI) is not currently supported.01 Sign in to AWS Management Console.
02 Navigate to AWS Support Center page at https://console.aws.amazon.com/support/.
03 On Support Center page, in the My support cases panel, click Create case to open the support case form.
04 On the Create Case page, perform the following actions:
Whether your AWS exploration is just starting to take shape, you’re mid-way through a migration or you’re already running complex workloads in the cloud, Cloud Conformity offers full visibility of your infrastructure and provides continuous assurance it’s secure, optimized and compliant.
Chat with us to set up your onboarding session and start a free trial.
Let’s Chat