Open menu
-->

AWS CloudTrail Log Files Delivery Failing

Cloud Conformity allows you to automate the auditing process of this resolution page. Register for a 14 day evaluation and check your compliance level for free!

Start a Free Trial Product features
Security
Operational
excellence

Risk level: Medium (should be achieved)

Ensure that the log files generated by your AWS CloudTrail trails are delivered without any failures to designated recipients in order to keep CloudTrail logging data for security and compliance audits.

This rule resolution is part of the Cloud Conformity Security Package

When your Amazon CloudTrail trails are not able to deliver log files to their recipients due to delivery errors or misconfigurations (usually involving the access policies that you have in place), the logging data recorded by these trails cannot be saved and used for future security audits.

Audit

Case A: to identify CloudTrail trails that are not able to deliver log files to the designated S3 bucket(s), perform the following actions:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to CloudTrail dashboard at https://console.aws.amazon.com/cloudtrail/.

03 In the left navigation panel, select Trails.

04 Choose the trail that you want to examine then click on its identifier (link) to access the configuration page.

05 Within Storage location section, verify the Last log file delivered attribute value (date). If the value has not been updated recently (log files are generally published every 5 minutes) and a warning sign is displayed next to the attribute, i.e.

Last Log File Delivered

the selected AWS CloudTrail trail failed to deliver the last log file to the designated S3 bucket.

06 Repeat step no. 4 and 5 to identify other trails that failed to deliver their log files, available in the selected region.

07 Change the AWS region from the navigation bar and repeat the audit process for other regions.

Using AWS CLI

01 Run describe-trails command (OSX/Linux/UNIX) to list the names of all CloudTrail trails currently available within the selected AWS region:

aws cloudtrail describe-trails
	--region us-east-1
	--output table
	--query 'trailList[*].Name'

02 The command output should return a table with the requested trail names:

--------------------------
|     DescribeTrails     |
+------------------------+
|  cc-prod-cloud-trail   |
|  cc-internal-trail     |
+------------------------+

03 Run get-trail-status command (OSX/Linux/UNIX) using the name of the trail returned at the previous step and custom query filters the expose any log file delivery errors (if any) triggered for the selected trail:

aws cloudtrail get-trail-status
	--region us-east-1
	--name cc-prod-cloud-trail
	--query 'LatestDeliveryError'

04 The command output should return null if the selected trail delivered its latest log file successfully or an error code if the trail failed to deliver the log file:

"AccessDenied"

If the CLI command respond with an error code (as shown in the example above) instead of null, the selected Amazon CloudTrail trail failed to deliver the last log file to the specified S3 bucket.

05 Repeat step no. 3 and 4 to identify other trails that failed to deliver their log files, available in the selected region.

06 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 - 5 to perform the entire audit process for other regions.

Case B: to identify CloudTrail trails that are not able to send SNS notifications, perform the following actions:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to CloudTrail dashboard at https://console.aws.amazon.com/cloudtrail/.

03 In the left navigation panel, select Trails.

04 Choose the trail that you want to examine then click on its identifier (link) to access the configuration page.

05 Within Storage location section, verify the Last SNS notification attribute status. If a warning sign is displayed next to the attribute, i.e.

Last SNS notification

the selected AWS CloudTrail trail failed to send a notification for each log file delivery.

06 Repeat step no. 4 and 5 to identify other trails that failed to send SNS notifications, available in the selected region.

07 Change the AWS region from the navigation bar and repeat the audit process for other regions.

Using AWS CLI

01 Run describe-trails command (OSX/Linux/UNIX) to list the names of all CloudTrail trails currently available within the selected AWS region:

aws cloudtrail describe-trails
	--region us-east-1
	--output table
	--query 'trailList[*].Name'

02 The command output should return a table with the requested trail names:

--------------------------
|     DescribeTrails     |
+------------------------+
|  cc-prod-cloud-trail   |
|  cc-internal-trail     |
+------------------------+

03 Run get-trail-status command (OSX/Linux/UNIX) using the name of the trail returned at the previous step and custom query filters to find any SNS notification errors (if any) triggered for the selected trail:

aws cloudtrail get-trail-status
	--region us-east-1
	--name cc-prod-cloud-trail
	--query 'LatestNotificationError'

04 The command output should return null if the selected trail is sending an SNS notification for each log file delivery or an error code if the trail failed to send notifications:

"AuthorizationError"

If the command output returns an error code, as shown in the example above, instead of null, the selected Amazon CloudTrail trail failed to send a notification for each log file delivery.

05 Repeat step no. 3 and 4 to identify other trails that failed to send SNS notifications, available in the selected region.

06 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 - 5 to perform the audit process for other regions.

Remediation / Resolution

Case A: Usually, the CloudTrail trails fail to deliver their log files when there is a problem with the destination S3 bucket and will not occur for timeouts. To remediate the issue, create a new S3 bucket and update the trail configuration to reference the new bucket so that CloudTrail can again write log files to S3. To update CloudTrail trails configuration, perform the following:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to CloudTrail dashboard at https://console.aws.amazon.com/cloudtrail/.

03 In the navigation panel, select Trails.

04 Choose the CloudTrail trail that you want to reconfigure (see Audit section part I to identify the right resource) then click on its identifier to access the configuration page.

05 On the selected trail configuration page, click the Edit icon available next to the Storage location section to create a new S3 bucket and associate the trail with this bucket.

06 Select Yes next to Create a new S3 bucket and enter a unique name for the new bucket inside the S3 bucket box. (Optional) You can also specify a prefix for the log files within Log file prefix box.

07 Click Save to apply the changes. Once the bucket is created and configured, AWS CloudTrail will begin to deliver log files to this new S3 bucket and the Last log file delivered attribute value set for the selected trail will be updated.

08 Repeat steps no. 4 – 7 to reconfigure other CloudTrail trails that failed to deliver the necessary log files, available in the current region.

09 Change the AWS region from the navigation bar and repeat the process for other regions.

Using AWS CLI

01 Run update-subscription command (OSX/Linux/UNIX) to update the selected trail configuration settings (see Audit section part II to identify the right resource), create and configure a new S3 bucket as recipient for the trail log files:

aws cloudtrail
	--region us-east-1 update-subscription
	--name cc-prod-cloud-trail
	--s3-new-bucket cc-new-cloudtrail-bucket

02 The command output should return the new configuration information for the updated trail:

Setting up new S3 bucket cc-new-cloudtrail-bucket...
Creating/updating CloudTrail configuration...
CloudTrail configuration:
{
  "trailList": [
    {
      "IncludeGlobalServiceEvents": true,
      "Name": "cc-prod-cloud-trail",
      "TrailARN": "arn:aws:cloudtrail:us-east-1:123456789012:trail/cc-prod-cloud-trail",
      "LogFileValidationEnabled": true,
      "IsMultiRegionTrail": false,
      "HasCustomEventSelectors": false,
      "S3BucketName": "cc-new-cloudtrail-bucket",
      "HomeRegion": "us-east-1"
    }
  ],
  "ResponseMetadata": {
    "RetryAttempts": 0,
    "HTTPStatusCode": 200,
    "RequestId": "aaabbccc-e0c2-42e1-3ae5-aaaabbbbcccc",
    "HTTPHeaders": {
      "date": "Sun, 15 Oct 2017 12:13:04 GMT",
      "content-length": "317",
      "content-type": "application/x-amz-json-1.1"
    }
  }
}

03 Repeat step no. 1 and 2 to reconfigure other trails that failed to deliver the necessary log files, available in the current region.

04 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 - 3 to perform the entire process for other regions.

Case B: When a CloudTrail trail fails to send SNS notifications for log files publishing, it's usually because there is a problem with the designated SNS topic. To resolve the issue, create a new SNS topic and update the trail configuration to point to the new topic so that CloudTrail can send notifications again. To update CloudTrail trails configuration, perform the following actions:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to CloudTrail dashboard at https://console.aws.amazon.com/cloudtrail/.

03 In the navigation panel, select Trails.

04 Choose the CloudTrail trail that you want to reconfigure (see Audit section part I to identify the right resource) then click on its identifier to access the configuration page.

05 Click the Edit icon available next to the Storage location section, to update the existing SNS topic reference.

06 Select Yes next to Create a new SNS topic and enter a name for your new topic inside the SNS topic box.

07 Click Save to create the new SNS topic and apply the appropriate permissions to receive notifications whenever trail log files are delivered to the designated S3 bucket.

08 Now go to SNS dashboard at https://console.aws.amazon.com/sns/v2/.

09 Choose Topics from the left navigation panel.

10 Select the SNS topic created at step no. 6, click the Actions dropdown menu from the dashboard top menu and select Subscribe to topic option.

11 In the Create subscription dialog box, select Email from the Protocol dropdown list and provide the email address where you can receive notifications within the Endpoint box.

12 Click Create Subscription to apply the new subscription to the selected SNS topic.

13 Use your email client application and open the message from AWS Notifications, then click on the appropriate link to confirm your subscription.

14 Repeat steps no. 4 – 13 to reconfigure other CloudTrail trails that failed to send SNS notifications, available in the current region.

15 Change the AWS region from the navigation bar and repeat the process for other regions.

Using AWS CLI

01 Run update-subscription command (OSX/Linux/UNIX) to update the selected trail configuration settings (see Audit section part II to identify the right trail) and create a new SNS topic as recipient for trail notifications:

aws cloudtrail
	--region us-east-1 update-subscription
	--name cc-prod-cloud-trail
	--sns-new-topic cc-new-sns-topic

02 The command output should return the new configuration information for the updated trail:

Setting up new SNS topic cc-new-sns-topic...
Creating/updating CloudTrail configuration...
CloudTrail configuration:
{
  "trailList": [
    {
      "IncludeGlobalServiceEvents": true,
      "Name": "cc-prod-cloud-trail",
      "TrailARN": "arn:aws:cloudtrail:us-east-1:123456789012:trail/cc-prod-cloud-trail",
      "LogFileValidationEnabled": true,
      "SnsTopicARN": "arn:aws:sns:us-east-1:123456789012:cc-new-sns-topic",
      "IsMultiRegionTrail": false,
      "HasCustomEventSelectors": false,
      "S3BucketName": "cc-cloudtrail-bucket",
      "SnsTopicName": "cc-new-sns-topic",
      "HomeRegion": "us-east-1"
    }
  ],
  "ResponseMetadata": {
    "RetryAttempts": 0,
    "HTTPStatusCode": 200,
    "RequestId": "aaabbccc-42d1-474e-a6a0-aaaabbbbbcccc",
    "HTTPHeaders": {
      "date": "Sun, 15 Oct 2017 13:23:57 GMT",
      "content-length": "419",
      "content-type": "application/x-amz-json-1.1"
    }
  }
}

03 Run subscribe command (OSX/Linux/UNIX) to subscribe to the SNS topic created at the previous step using email as subscription protocol:

aws sns subscribe
	--region us-east-1
	--topic-arn arn:aws:sns:us-east-1:123456789012:cc-new-sns-topic
	--protocol email
	--notification-endpoint noreply@cloudconformity.com

04 The command output should return the "pending confirmation" status:

{
    "SubscriptionArn": "pending confirmation"
}

05 Run confirm-subscription command (OSX/Linux/UNIX) to confirm the subscription by validating the token sent to the notification endpoint (email address) selected at the previous step (the command does not produce an output):

aws sns confirm-subscription
	--region us-east-1
	--topic-arn arn:aws:sns:us-east-1:123456789012:cc-new-sns-topic
	--token 5294392f37fb687f5d51e6e241d7700ae02f7124d8268910b858cb4db727ceeb2474bb937929d3bdd7ce5d0cce19325d036bc855d3c217426bcafa9c501a2cace93b83f1dd3797627467553dc438a8c974119496fc3eff026eaa5d14472ded6f9a5c43aec62d83ef5f49109da713379

06 Repeat steps no. 1 – 5 to reconfigure other trails that failed to send SNS notifications, available in the current region.

07 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 - 6 to perform the entire remediation process for other regions.

References

Publication date Oct 17, 2017