Open menu
-->

Enable Shard-Level Metrics for Kinesis Streams

Cloud Conformity allows you to automate the auditing process of this resolution page. Register for a 14 day evaluation and check your compliance level for free!

Start a Free Trial Product features
Operational
excellence
Cost
optimization
Reliability

Risk level: Low (generally tolerable level of risk)

Ensure enhanced Amazon Kinesis stream monitoring for shard-level metrics is enabled in order to increase visibility into your streams performance. Kinesis is a platform for streaming data on Amazon Web Services that provides you with the ability to build and manage your own custom streaming data applications for specialized needs. A Kinesis stream is an ordered sequence of data records collected within a dedicated storage layer. The data records within a Kinesis stream is distributed into unique identified groups called shards.

This rule resolution is part of the Cloud Conformity Base Auditing Package

You can gain better insight into your AWS Kinesis streams usage (i.e. distribution of data throughput) by enabling shard-level metrics such as IncomingBytes and IncomingRecords – helpful metrics that identify which shard is receiving more data within a stream, WriteProvisionedThroughputExceeded – metric that determines if the writes are throttled within a stream shard over a specified period of time, ReadProvisionedThroughputExceeded – metric that returns the number of GetRecords calls throttled within a shard over a specified time frame, etc.

Audit

To determine if shard-level metrics are enabled for your Amazon Kinesis streams, perform the following:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Kinesis dashboard at https://console.aws.amazon.com/kinesis/.

03 In the navigation panel, under Amazon Kinesis, choose Streams.

04 Select the Kinesis stream that you want to examine, click the Actions dropdown button and select Details to access the stream configuration details.

05 Choose the Details tab from the top panel and verify the Shard level metrics feature status available within the Shard level metrics section. If the status is set to "No shard level metrics enabled", the selected AWS Kinesis streams does not have any shard-level metrics currently enabled.

06 Repeat step no. 4 and 5 for each Amazon Kinesis stream available in the current AWS region.

07 Change the AWS region from the navigation bar to repeat the audit process for other regions.

Using AWS CLI

01 Run list-streams command (OSX/Linux/UNIX) to list the names of all Kinesis streams available within the selected AWS region - US East (N. Virginia):

aws kinesis list-streams
	--region us-east-1
	--query 'StreamNames'

02 The command output should return the requested Kinesis stream names:

[
    "iot-kinesis-stream",
    "customers-data-stream"
]

03 Run describe-stream command (OSX/Linux/UNIX) using the stream name returned at the previous step as identifier and custom query filters to return the list of the shard-level metrics enabled for the selected Kinesis stream:

aws kinesis describe-stream
	--region us-east-1
	--stream-name iot-kinesis-stream
	--query 'StreamDescription.EnhancedMonitoring[*].ShardLevelMetrics[]'

04 The describe-stream command output should return an array that contains the name(s) of the shard-level metric(s) enabled for the Kinesis stream, the value "ALL" if the stream enhances every metric available and an empty array if there are no shard-level metrics activated. If the output returned is an empty array, i.e. [ ], the selected AWS Kinesis streams does not have any shard-level metrics enabled.

05 Repeat step no. 3 and 4 for each AWS Kinesis stream provisioned in the current AWS region.

06 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 - 5 to perform the entire process for other regions.

Remediation / Resolution

To enable enhanced monitoring for your Amazon Kinesis streams using shard-level metrics, perform the following:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Kinesis dashboard at https://console.aws.amazon.com/kinesis/.

03 In the navigation panel, under Amazon Kinesis, choose Streams.

04 Select the Kinesis stream that you want to examine, click the Actions dropdown button and select Details to open the stream configuration page.

05 Select the Details tab from the top panel, locate the Shard level metrics section and click the Edit button next to it:

Edit Button

06 Check "All shard level metrics" checkbox to enable every shard-level metric available for the selected stream or select individual metrics based on your requirements. Full description for each shard-level metric is available at this URL.

07 Click Save to apply the configuration changes. Once the request is made, a pop-up message will be displayed: "Updating shard level metrics for the stream. This will take up to 70 seconds.". The selected stream transitions through a "pending" state. Once the Kinesis stream returns to the "active" state, the shard-level metrics that you selected become active.

08 Repeat steps no. 4 – 7 to enable enhanced monitoring for other Amazon Kinesis streams available in the current AWS region.

09 Change the AWS region from the navigation bar and repeat the process for other regions.

Using AWS CLI

01 Run enable-enhanced-monitoring command (OSX/Linux/UNIX) using the name of the stream that you want to reconfigure (see Audit section part II to identify the right Kinesis resource) to enable enhanced monitoring for the selected Amazon Kinesis stream using valid shard-level metrics. The following command example enables all shard-level metrics made available by Amazon Kinesis for a stream named "iot-kinesis-stream", provisioned within the US East (N. Virginia) region:

aws kinesis enable-enhanced-monitoring
	--region us-east-1
	--stream-name iot-kinesis-stream
	--shard-level-metrics ALL

02 The command output should return the enhanced monitoring metadata which contains the name of each shard-level metric enabled for the selected AWS Kinesis stream:

{
    "StreamName": "iot-kinesis-stream",
    "CurrentShardLevelMetrics": [],
    "DesiredShardLevelMetrics": [
        "IncomingBytes",
        "OutgoingRecords",
        "IteratorAgeMilliseconds",
        "IncomingRecords",
        "ReadProvisionedThroughputExceeded",
        "WriteProvisionedThroughputExceeded",
        "OutgoingBytes"
    ]
}

03 Repeat step no. 1 and 2 to enable enhanced stream monitoring using shard-level metrics for other Amazon Kinesis streams created in the current AWS region.

04 Change the AWS region by updating the --region command parameter value and repeat the entire process for other regions.

References

Publication date Jul 19, 2017