Open menu
-->

AWS CloudFormation Stack Failed Status

Cloud Conformity allows you to automate the auditing process of this resolution page. Register for a 14 day evaluation and check your compliance level for free!

Start a Free Trial Product features
Operational
excellence

Risk level: Medium (should be achieved)

Ensure that none of your Amazon CloudFormation stacks are in 'Failed' mode for more than 6 hours. A stack is on 'Failed' mode when its status is set to 'CREATE_FAILED' - unsuccessful creation of the stack, 'DELETE_FAILED' - unsuccessful deletion of the stack, 'ROLLBACK_FAILED' - unsuccessful removal of the stack after the creation process failed or 'UPDATE_ROLLBACK_FAILED' - unsuccessful return of the stack to a previous working state after a failed update.

This rule resolution is part of the Cloud Conformity Base Auditing Package

Any failed CloudFormation stacks that are not fixed on time can lead to application downtime, security issues or unexpected costs on your AWS bill. For example, the unsuccessful deletion ("DELETE_FAILED") of one or more stacks can accrue charges for the unused AWS resources provisioned by the stack.

Audit

To determine if there are any failed Amazon CloudFormation stacks available in your account, perform the following:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to CloudFormation dashboard at https://console.aws.amazon.com/cloudformation/.

03 Select the Failed option from the Filter dropdown menu to list all failed CloudFormation stacks available in the current AWS region.

04 Choose a failed stack that you want to examine then select the Events tab from the bottom panel.

05 On the Events panel, verify the timestamp when the event occurred, e.g.

verify the timestamp when the event occurred

If the "Failed" mode was triggered more than 6 hours ago, the selected CloudFormation stack is rendered as unsuccessful and can be removed from your AWS account.

06 Repeat step no. 4 and 5 to verify other failed CloudFormation stacks available in the selected region.

07 Change the AWS region from the navigation bar and repeat the audit process for other regions.

Using AWS CLI

01 Run lists-stacks command (OSX/Linux/UNIX) using status filters to retrieve a list with CloudFormation stacks currently available in "Failed" mode, created in the selected AWS region:

aws cloudformation list-stacks
    --region us-east-1
    --stack-status-filter "CREATE_FAILED" "DELETE_FAILED" "ROLLBACK_FAILED"
    --query 'StackSummaries[*].StackName'

02 The command output should return the name(s) of the failed CloudFormation stack(s), e.g.:

[
    "WebAppCFNStack",
    "CFNWordpressStack"
]

03 Run describe-stack-events command (OSX/Linux/UNIX) using the name of the stack returned at the previous step to list the related events for the selected stack in reverse chronological order:

aws cloudformation describe-stack-events
    --region us-east-1
    --stack-name WebAppCFNStack

04 The command output should return the event metadata for the selected CloudFormation stack:

{
    "StackEvents": [
        {
            "EventId": "533e08c0-e569-11e6-83b4-50d5ca632656",
            "ResourceStatus": "DELETE_FAILED",
            "ResourceType": "AWS::CloudFormation::Stack",
            "Timestamp": "2017-01-28T19:45:36.192Z",
            "ResourceStatusReason": "The following resource(s) failed to delete: [EC2Instance].",
            "StackName": "WebAppCFNStack",
            "LogicalResourceId": "WebAppCFNStack"
        },

        ...
      
        {
            "EventId": "42cd5ae0-e569-11e6-b46b-50fae988d8d2",
            "ResourceStatus": "CREATE_IN_PROGRESS",
            "ResourceType": "AWS::CloudFormation::Stack",
            "Timestamp": "2017-01-27T14:41:33.772Z",
            "ResourceStatusReason": "User Initiated",
            "StackName": "WebAppCFNStack",
            "LogicalResourceId": "WebAppCFNStack"
        }
    ]
}

Verify the Timestamp element value to determine when the event occurred and enabled the "Failed" status (in this example the status is "DELETE_FAILED"). Based on the timestamp returned, if the "Failed" mode was triggered more than 6 hours ago, the selected CloudFormation stack is declared unsuccessful and can be removed from your AWS account.

05 Repeat step no. 3 and 4 to verify other failed CloudFormation stacks available in the selected region.

06 Change the AWS region by updating the --region command parameter value and repeat steps 1 – 5 to perform the entire process for other regions.

Remediation / Resolution

To remove any Amazon CloudFormation stacks available in "Failed" mode for more than 6 hours, perform the following:

Using AWS Console

01 Login to the AWS Management Console.

02 Navigate to CloudFormation dashboard at https://console.aws.amazon.com/cloudformation/.

03 Select the CloudFormation stack that you want to delete (see Audit section part I to identify the right resource).

04 Click the Actions button from the dashboard top menu and select Delete Stack:

Delete Stack

05 Inside the Delete Stack dialog box, review the stack details (i.e. the stack name) then click Yes, Delete to confirm the action. The resource status should change now to DELETE_IN_PROGRESS. Once the stack and all its resources have been successfully deleted, Amazon will remove the entry from the CloudFormation service dashboard.

06 Repeat steps no. 3 – 5 to remove other CloudFormation stacks available in the selected region.

07 Change the AWS region from the navigation bar and repeat the entire process for other regions.

Using AWS CLI

01 Run delete-stack command (OSX/Linux/UNIX) using the stack name as identifier (see Audit section part II to identify the right resource) to delete the selected (failed) CloudFormation stack (the command does not produce an output):

aws cloudformation delete-stack
    --region us-east-1
    --stack-name WebAppCFNStack

02 Repeat step no. 1 to remove other failed CloudFormation stacks available in "Failed" mode for more than 6 hours, created in the selected AWS region.

03 Change the AWS region by updating the --region command parameter value and repeat step no. 1 and 2 to perform the entire process for other regions.

References

Publication date Feb 6, 2017