Open menu

CloudWatch Logs Agent for App-Tier Auto Scaling Group In Use

Cloud Conformity allows you to automate the auditing process of this resolution page. Register for a 14 day evaluation and check your compliance level for free!

Start a Free Trial Product features
Knowledge Base AutoScaling CloudWatch Logs Agent for App-Tier Auto Scaling Group In Use
Operational
excellence

Risk level: Medium (should be achieved)

Ensure that the EC2 instances launched within your app-tier Auto Scaling Group (ASG) are using CloudWatch log agents to monitor, store and access log files (application or system data logs) from these instances. A CloudWatch Logs agent needs to be installed on the guest Operating System of the app-tier EC2 instance that you want to get logs from. This conformity rule assumes that all AWS resources provisioned for your app tier are tagged with <app_tier_tag>:<app_tier_tag_value>, where <app_tier_tag> is the tag name and <app_tier_tag_value> is the tag value. Prior to running this rule by the Cloud Conformity engine, the app-tier tags must be configured in the rule settings, on your Cloud Conformity account dashboard.

An Amazon CloudWatch Logs agent helps providing centralized logging, monitoring and incident reporting of both system-level and application-level events available on the EC2 instances provisioned within the app-tier Auto Scaling Group. Note: Make sure that you replace all <app_tier_tag>:<app_tier_tag_value> tag placeholders found in the conformity rule content with your own tag name and value created for the app tier.

Audit

To determine if each app-tier ASG instance is being launched with an AWS CloudWatch Logs agent, perform the following:

Using AWS Console

01 Sign in to your Cloud Conformity console, access Cloudwatch Logs Agent for App-Tier Auto Scaling Group In Use conformity rule settings and identify the tags defined for all AWS resources available in your app tier (e.g. <app_tier_tag>:<app_tier_tag_value>).

02 Sign in to the AWS Management Console.

03 Navigate to EC2 dashboard at https://console.aws.amazon.com/ec2/.

04 In the navigation panel, under AUTO SCALING, click Auto Scaling Groups.

05 Select the Auto Scaling Group that you want to examine.

06 Select the Tags tab from the dashboard bottom panel.

07 On the Tags panel, search for the tag set identified at step no. 1 (i.e. <app_tier_tag> : <app_tier_tag_value>). If the two tag sets do not match, or the verified resource is not tagged at all, the selected ASG is not a component of your app tier and the audit process ends here. If the tag sets match, the selected resource is a app-tier Auto Scaling Group and the audit process continues with the next step.

08 Select the Details tab from the dashboard bottom panel and copy the name of the attached launch configuration set as value for the Launch Configuration attribute.

09 In the navigation panel, under AUTO SCALING, click Launch Configurations.

10 Paste the value copied at step no. 8 in the Filter launch configurations box and press Enter to return the launch configuration used by the selected app-tier ASG.

11 Select the launch configuration returned as search result and check the value (script) of the User Data attribute available on the Details panel. If the User Data attribute has no script assigned, or the script defined as user data does not contain the following block: #!/bin/bash curl https://s3.amazonaws.com//aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O chmod +x ./awslogs-agent-setup.py ./awslogs-agent-setup.py -n -r <AWS_REGION> -c s3://<S3_BUCKET_NAME>/<CLOUDWATCH_AGENT_CONFIG_FILE>, the Cloudwatch Logs agent was not installed through UserData feature at launch, therefore the EC2 instances available within the selected app-tier ASG are not using a CloudWatch Logs agent to monitor, store and access their log files.

12 Repeat steps no. 5 – 11 to check other app-tier ASG launch configurations, created in the selected region, for Cloudwatch Logs agent installation information.

13 Change the AWS region from the navigation bar and repeat steps no. 5 – 12 for other regions.

Using AWS CLI

01 Sign in to your Cloud Conformity console, access Cloudwatch Logs Agent for App-Tier Auto Scaling Group In Use conformity rule settings and identify the tags defined for AWS resources available in your app tier (e.g. <app_tier_tag>:<app_tier_tag_value>).

02 Run describe-auto-scaling-groups command (OSX/Linux/UNIX) to list the names of all Auto Scaling Groups available in the selected AWS region: 

aws autoscaling describe-auto-scaling-groups
	--region us-east-1
	--output table
	--query 'AutoScalingGroups[*].AutoScalingGroupName'

03 The command output should return a table with the requested ASG names: 

---------------------------
|DescribeAutoScalingGroups|
+-------------------------+
|  cc-mobile-app-asg      |
|  cc-test-scaling-group  |
+-------------------------+

04 Run describe-tags command (OSX/Linux/UNIX) using the name of the ASG that you want to examine as identifier and custom query filters to describe the tags defined for the selected Auto Scaling Group: 

aws autoscaling describe-tags
	--region us-east-1
	--filters "Name=auto-scaling-group,Values=cc-mobile-app-asg"
	--query 'Tags[*].{Value:Value, Key:Key}'

05 The command request should return one of the following outputs:

  1. If the command output returns an empty array (i.e. []), as shown in the example below, the verified ASG is not tagged, therefore the audit process for the selected resource stops here: 
    []
  2. If the describe-tags command output returns a set of tags that is different than the one identified at step no. 1, as shown in the example below, the verified AWS Auto Scaling Group does not belong to your app tier, therefore the audit process for the selected resource ends here: 
    [
    {
        "Value": "Owner",
        "Key": "MobileDevTeam"
    }
]
  3. If the command output returns a set of tags that match the one identified at step no. 1 (e.g. <app_tier_tag>:<app_tier_tag_value>), as shown in the example below, the verified AWS ASG is tagged as a app-tier resource, therefore the audit process continues with the next step: 
    [
    {
        "Key": "<app_tier_tag>",
        "Value": "<app_tier_tag_value>"
    }
]

06 Run describe-auto-scaling-groups command (OSX/Linux/UNIX) with custom query filters to return the name of the launch configuration associated with the selected app-tier ASG: 

aws autoscaling describe-auto-scaling-groups
	--region us-east-1
	--query 'AutoScalingGroups[*].LaunchConfigurationName'

07 The command output should return the name of the associated launch configuration: 

[
    "cc-app-launch-config"
]

08 Run describe-launch-configurations command (OSX/Linux/UNIX) using the name of the launch configuration returned at the previous step as identifier and custom query filters to describe the user data defined for the EC2 instances launched inside the specified app-tier ASG with the selected launch configuration: 

aws autoscaling describe-launch-configurations
	--region us-east-1
	--launch-configuration-names cc-app-launch-config
	--query "LaunchConfigurations[*].UserData"

09 The command request should return one of the following outputs:

  1. If the describe-launch-configurations command output returns an empty string (i.e. “”), as shown in the example below, there is no user data defined for the EC2 instances created by the selected launch configuration: 
    [
    ""
]
  2. If the command output returns a configuration script that that does not contain the following block: "#!/bin/bash curl https://s3.amazonaws.com//aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O chmod +x ./awslogs-agent-setup.py ./awslogs-agent-setup.py -n -r <AWS_REGION> -c s3://<S3_BUCKET_NAME>/<CLOUDWATCH_AGENT_CONFIG_FILE>", the EC2 instances launched within the specified app-tier ASG, by the selected launch configuration, are not using an agent to publish their log files to AWS CloudWatch Logs: 
    [
    "#!/bin/bash
    yum update –y
	 yum install nginx
    service nginx start"
]

10 Repeat steps no. 4 – 9 to verify other app-tier ASG launch configurations, available in the selected region, for Cloudwatch Logs agent installation details.

11 Change the AWS region by updating the --region command parameter value and repeat steps no. 2 – 10 to perform the entire audit process for other regions.

Remediation / Resolution

- To install the Amazon Cloudwatch Logs agent on the EC2 instances within your app-tier ASG, you must re-create the ASG launch configuration and set it up with the necessary user data (i.e. agent installation script). To create a new launch configuration and replace the existing one, perform the following actions:

Note: The guest OS used in this remediation/resolution section is Amazon Linux.

Using AWS Console

01 Create the required agent configuration file for Amazon Linux and save it as a text file named "app-instance-logs.cfg" then upload the file to AWS S3. This is a text file that defines the log file(s) to monitor and the log groups and log streams to upload them to. The AWS Cloudwatch Logs agent utilize this configuration file and starts monitoring and uploading the log files defined in it. The following is an agent configuration file for the general system activity log on Amazon Linux: 

[general]
state_file = /var/awslogs/state/agent-state  
 
[/var/log/messages]
file = /var/log/messages
log_group_name = /var/log/messages
log_stream_name = {instance_id}
datetime_format = %b %d %H:%M:%S

02 Sign in to the AWS Management Console.

03 Navigate to EC2 dashboard at https://console.aws.amazon.com/ec2/.

04 In the navigation panel, under AUTO SCALING, click Launch Configurations.

05 Select the app-tier ASG launch configuration that you want to re-create (see Audit section part I to identify the right ASG resource).

06 Click the Actions dropdown button from the dashboard top menu and select Copy launch configuration option.

07 On Copy Launch Configuration from <LAUNCH_CONFIGURATION_NAME> page, in the Launch configuration details section, click Edit details to edit the template.

08 Click Advanced Details to display the panel with the advanced settings and paste the following script inside the User data box (make sure that you replace the <AWS_REGION> with the appropriate AWS region and the <S3_BUCKET_NAME> with the name of the S3 bucket that store the agent configuration file defined at step no. 1). Once the script is added, click Skip to review to finish the configuration process: 

#!/bin/bash 
curl https://s3.amazonaws.com//aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O
chmod +x ./awslogs-agent-setup.py 
./awslogs-agent-setup.py -n -r <AWS_REGION> -c s3://<S3_BUCKET_NAME>/instance-logs.cfg

09 Click Create launch configuration. In the Select an existing key pair or create a new key pair dialog box, select Choose an existing key pair and use the same key pair as the source launch configuration. Click Create launch configuration to make a copy of the specified web-tier ASG launch configuration. Click Close to return to the EC2 dashboard.

10 In the navigation panel, under AUTO SCALING, click Auto Scaling Groups.

11 Select the app-tier Auto Scaling Group that you want to reconfigure (see Audit section part I to identify the right ASG).

12 Select the Details tab from the dashboard bottom panel and click Edit to change the group configuration.

13 Select the name of the launch configuration created at steps no. 6 – 9 from the Launch configuration dropdown list, then click Save to apply the changes. Once a new EC2 instance is launched within the selected app-tier ASG, you should be able to see and access the newly created AWS CloudWatch log stream and log group, supplied with the instance logging data.

14 Repeat steps no. 5 – 13 to reconfigure other app-tier ASGs, available in the selected region, to install the AWS Cloudwatch Logs agent on the app-tier ASG instances.

15 Change the AWS region from the navigation bar and repeat steps no. 5 – 14 for other regions.

Using AWS CLI

01 Create the necessary agent configuration file for Amazon Linux and save it as a text file named "app-instance-logs.cfg" then upload the file to AWS S3. This represents a text file that describes the log files to monitor and the log groups and log streams to upload them to. The AWS Cloudwatch Logs agent utilize this configuration file and starts monitoring and uploading all the log files defined in it. The following is an agent configuration file for the general system activity log on Amazon Linux: 

[general]
state_file = /var/awslogs/state/agent-state  
 
[/var/log/messages]
file = /var/log/messages
log_group_name = /var/log/messages
log_stream_name = {instance_id}
datetime_format = %b %d %H:%M:%S

02 Now define the installation script for the AWS Cloudwatch Logs agent. Paste the following script into a text file named "awslogs-agent-setup.txt" (replace the <AWS_REGION> with the necessary AWS region and the <S3_BUCKET_NAME> with the name of the S3 bucket that has the agent configuration file defined at the previous step): 

#!/bin/bash 
curl https://s3.amazonaws.com//aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O
chmod +x ./awslogs-agent-setup.py 
./awslogs-agent-setup.py -n -r <AWS_REGION> -c s3://<S3_BUCKET_NAME>/instance-logs.cfg

03 Run describe-launch-configurations command (OSX/Linux/UNIX) using the name of the app-tier ASG launch configuration that you want to re-create as identifier (see Audit section part II to identify the right resource), to describe its configuration details, information required later when the new launch configuration will be created: 

aws autoscaling describe-launch-configurations
	--region us-east-1
	--launch-configuration-names cc-app-launch-config

04 The command output should return the requested details: 

{
    "LaunchConfigurations": [
        {
            "BlockDeviceMappings": [
                {
                    "DeviceName": "/dev/xvda",
                    "Ebs": {
                        "DeleteOnTermination": true,
                        "VolumeSize": 80,
                        "VolumeType": "gp2"
                    }
                }
            ],
 
	      ... 
 
            "KeyName": "cc-private-ssh-key",
            "SecurityGroups": [
                "sg-12345678"
            ],
            "LaunchConfigurationName": "cc-app-launch-config",
            "KernelId": "",
            "RamdiskId": "",
            "ImageId": "ami-abcd1234",
            "InstanceType": "c3.large"
        }
    ]
}

05 Run create-launch-configuration command (OSX/Linux/UNIX) using the configuration metadata returned at the previous step to create a new launch configuration that runs the installation script for the AWS Cloudwatch Logs agent, defined at step no. 2. The following CLI command example creates an ASG launch configuration named "cc-new-app-launch-config", based on an AWS AMI identified by the ID "ami-abcd1234", with an IAM instance profile identified by the name "cc-app-instance-profile" and the required user data to install the Cloudwatch Logs agent, available within "awslogs-agent-setup.txt" file (the command does not produce an output): 

aws autoscaling create-launch-configuration
	--region us-east-1
	--launch-configuration-name cc-new-app-launch-config
	--image-id ami-abcd1234
	--instance-type c3.large
	--key-name cc-private-ssh-key
	--security-groups sg-12345678
	--iam-instance-profile cc-app-instance-profile
	--user-data file://awslogs-agent-setup.txt
	--block-device-mappings "[{\"DeviceName\": \"/dev/xvda\",\"Ebs\":{\"VolumeSize\":80}}]"

06 Run update-auto-scaling-group command (OSX/Linux/UNIX) to update the configuration for your app-tier Auto Scaling Group, in order to replace the existing launch configuration with the new one created at the previous step (the command does not return an output): 

aws autoscaling update-auto-scaling-group
	--region us-east-1
	--auto-scaling-group-name cc-mobile-app-asg
	--launch-configuration-name cc-new-app-launch-config

07 Repeat steps no. 3 – 6 to reconfigure other app-tier ASGs, available in the selected region, to install the AWS Cloudwatch Logs agent on the app-tier ASG instances.

08 Change the AWS region by updating the --region command parameter value and repeat steps no. 2 – 7 to perform the process for other regions.

References

Publication date Apr 2, 2018

Related AutoScaling rules