Open menu

SQS Dead Letter Queue

Cloud Conformity allows you to automate the auditing process of this resolution page. Register for a 14 day evaluation and check your compliance level for free!

Start a Free Trial Product features
Knowledge Base Best practice rules for Amazon Web Services AWS SQS Best Practices SQS Dead Letter Queue
Last updated: 02 October 2018
Operational
excellence
Reliability
Risk level: Low (generally tolerable level of risk)
Rule ID: SQS-006

Ensure that each AWS Simple Queue Service (SQS) queue is configured to use a Dead Letter Queue (DLQ) in order to help maintain the queue flow and avoid losing data by detecting and mitigating failures and service disruptions on time. A Dead Letter Queue is an SQS queue useful for debugging your application or your messaging system, that can isolate messages that can't be processed successfully for later analysis.

This rule can help you with the following compliance standards:

This rule can help you work with the AWS Well-Architected Framework

This rule resolution is part of the Cloud Conformity Security & Compliance tool for AWS

Enabling Dead Letter Queues (DLQs) for your SQS queues can help you troubleshoot incorrect message transmission operations that can lead to data loss. Use DLQs to decrease the number of unprocessed messages and reduce the possibility of exposing your queues to poison pill messages (i.e. messages that are received but can't be processed for some reason).

Audit

To determine if Dead Letter Queues are enabled for your AWS SQS queues, perform the following:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to SQS dashboard at https://console.aws.amazon.com/sqs/.

03 Select the SQS queue that you want to examine.

04 Select the Redrive Policy tab from the bottom panel and check for any redrive policies configured for the selected queue. A redrive policy sends messages into a DLQ after the source queue fails to process a message after a specified number of times. If there is no redrive policy configured and the following message is displayed: "The queue does not have a redrive policy. Learn how you can set a redrive policy for this queue.", the selected AWS SQS queue does not have a Dead Letter Queue configured to help maintain the queue flow.

05 Repeat step no. 3 and 4 for each Amazon SQS queue available in the selected AWS region.

06 Change the AWS region from the navigation bar to repeat the audit process for other regions.

Using AWS CLI

01 Run list-queues command (OSX/Linux/UNIX) to list the URLs of all SQS queues available in the selected AWS region: 

aws sqs list-queues
	--region us-east-1

02 The command output should return the requested SQS URL(s): 

{
    "QueueUrls": [
        "https://queue.amazonaws.com/123456789012/cc-web-app-worker",
        "https://queue.amazonaws.com/123456789012/cc-mobile-app-queue",
        "https://queue.amazonaws.com/123456789012/cc-project5-queue"
    ]
}

03 Run get-queue-attributes command (OSX/Linux/UNIX) using the URL of the queue that you want to examine as identifier and custom query filters to return the redrive policy configuration metadata, which is basically a string that includes the parameters for Dead Letter Queue functionality of the source SQS queue: 

aws sqs get-queue-attributes
	--region us-east-1
	--queue-url https://queue.amazonaws.com/123456789012/cc-web-app-worker
	--attribute-names RedrivePolicy

04 The command output should return the requested metadata if the queue has a redrive policy configured. A redrive policy redirects messages to a DLQ after the source queue fails to process a message after a specified number of times. If there is no redrive policy configured, the get-queue-attributes command does not return an output, therefore there is no Dead Letter Queue configured for the selected Amazon SQS queue.

05 Repeat step no. 3 and 4 for each AWS SQS queue provisioned in the current AWS region.

06 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 - 5 to perform the entire audit process for other regions.

Remediation / Resolution

To create and configure a Dead Letter Queue in order to prevent endless processing of invalid messages for your AWS SQS queues, perform the following actions:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to SQS dashboard at https://console.aws.amazon.com/sqs/.

03 To set up the necessary Dead Letter Queue, click Create New Queue button from the dashboard top menu, select Standard Queue and click Configure Queue.

04 On the Create New Queue page, enter a unique name for the queue in the Queue Name box and leave the queue default parameters unchanged, unless you need a custom configuration.

05 Click Create Queue to launch your new SQS Dead Letter Queue.

06 Select the SQS queue that you want to reconfigure in order to implement the redrive policy, (see Audit section part I to identify the right SQS resource).

07 Click the Queue Actions button from the dashboard top menu and select Configure Queue option.

08 Within Configure <queue_name> dialog box, locate the Dead Letter Queue Settings section and perform the following actions:

  1. Check Use Redrive Policy checkbox to set up the policy that sends SQS messages into the Dead Letter Queue after exceeding the Maximum Receives value.
  2. In the Dead Letter Queue box, enter the name of the queue created at steps no. 3 – 5.
  3. In the Maximum Receives box, enter the maximum number of times an SQS message can be received before it is sent to the Dead Letter Queue (DLQ). The value must be between 1 and 1000.
  4. Click Save Changes to apply the new configuration changes. The newly created DLQ can now receive unprocessed messages from the source SQS queue.

09 Repeat step no. 6 – 8 for each SQS queue that you want to configure a DLQ, available in the current AWS region.

10 Change the AWS region from the navigation bar and repeat the process for other regions.

Using AWS CLI

01 Run create-queue command (OSX/Linux/UNIX) to set up the necessary SQS Dead Letter Queue. The following command example creates an AWS SQS queue that will serve as DLQ, named "cc-dead-letter-queue": 

aws sqs create-queue
	--region us-east-1
	--queue-name cc-dead-letter-queue

02 The command output should return the complete URL of the new queue: 

{
 "QueueUrl":  "https://queue.amazonaws.com/123456789012/cc-dead-letter-queue"
}

03 Define the required redrive policy, as shown in the example below, policy that will enable your SQS queues to send unprocessed messages to the newly created Dead Letter Queue, and save the policy to a JSON file named set-redrive-policy.json: 

{
  "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:us-east-1:123456789012:cc-dead-letter-queue\",\"maxReceiveCount\":\"5\"}"
}

04 Run set-queue-attributes command (OSX/Linux/UNIX) using the URL of the SQS queue that you want to reconfigure (see Audit section part II to identify the right queue) and the policy document defined at the previous step to implement the redrive policy and enable the Dead Letter Queue for the selected AWS SQS queue (the command does not produce an output): 

aws sqs set-queue-attributes
	--queue-url https://queue.amazonaws.com/123456789012/cc-web-app-worker
	--attributes file://set-redrive-policy.json

05 Repeat step no. 4 to assign the new Dead Letter Queue to each SQS queue available in the current AWS region.

06 Change the AWS region by updating the --region command parameter value and repeat the entire remediation/resolution process for other regions.

References

Publication date Sep 10, 2018

Related SQS rules

Thanks!

A verification email has been sent to

Thanks!

A verification email has been sent to

Thanks!

A verification email has been sent to