CloudWatch Logs Agent for Web-Tier Auto Scaling Group In Use

01 Sign in to your Cloud Conformity console, access Cloudwatch Logs Agent for Web-Tier Auto Scaling Group In Use conformity rule settings and identify the tag set defined for all AWS resources available in your web tier (e.g. <web_tier_tag>:<web_tier_tag_value>).

02 Sign in to the AWS Management Console.

03 Navigate to EC2 dashboard at https://console.aws.amazon.com/ec2/.

04 In the navigation panel, under AUTO SCALING, click Auto Scaling Groups.

05 Select the Auto Scaling Group that you want to examine.

06 Select the Tags tab from the dashboard bottom panel.

07 On the Tags panel, search for the tag set identified at step no. 1 (i.e. <web_tier_tag> : <web_tier_tag_value>). If the two tag sets do not match, or the verified resource is not tagged at all, the selected ASG is not a component of your web tier and the audit process ends here. If the tag sets match, the selected resource is a web-tier Auto Scaling Group and the audit process continues with the next step.

08 Select the Details tab from the dashboard bottom panel and copy the name of the associated launch configuration set as value for the Launch Configuration attribute.

09 In the navigation panel, under AUTO SCALING, click Launch Configurations.

10 Paste the value copied at step no. 8 in the Filter launch configurations box and press Enter to return the launch configuration template used by the selected web-tier ASG.

11 Select the launch configuration returned as search result and check the value (script) of the User Data attribute available on the Details tab panel. If the User Data attribute has no script assigned, or the script configured as user data does not contain the following block: x#!/bin/bash curl https://s3.amazonaws.com//aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O chmod +x ./awslogs-agent-setup.py ./awslogs-agent-setup.py -n -r <AWS_REGION> -c s3://<S3_BUCKET_NAME>/<CLOUDWATCH_AGENT_CONFIG_FILE>, the Cloudwatch Logs agent was not installed through UserData feature at launch, therefore the EC2 instances available within the selected web-tier ASG are not using a CloudWatch Logs agent to monitor, store and access their log files.

12 Repeat steps no. 5 – 11 to check other web-tier ASG launch configurations, created in the selected region, for Cloudwatch Logs agent installation information.

13 Change the AWS region from the navigation bar and repeat steps no. 5 – 12 for other regions.

01 Sign in to your Cloud Conformity console, access Cloudwatch Logs Agent for Web-Tier Auto Scaling Group In Use conformity rule settings and identify the tag set defined for AWS resources available in your web tier (e.g. <web_tier_tag>:<web_tier_tag_value>).

02 Run describe-auto-scaling-groups command (OSX/Linux/UNIX) to list the names of all Auto Scaling Groups available in the selected AWS region:

aws autoscaling describe-auto-scaling-groups
	--region us-east-1
	--output table
	--query 'AutoScalingGroups[*].AutoScalingGroupName'

03 The command output should return a table with the requested ASG name(s):

---------------------------
|DescribeAutoScalingGroups|
+-------------------------+
|  cc-auto-scaling-group  |
|  cc-web-scaling-group   |
+-------------------------+

04 Run describe-tags command (OSX/Linux/UNIX) using the name of the ASG that you want to examine as identifier and custom query filters to describe the tags defined for the selected Auto Scaling Group:

aws autoscaling describe-tags
	--region us-east-1
	--filters "Name=auto-scaling-group,Values=cc-auto-scaling-group"
	--query 'Tags[*].{Value:Value, Key:Key}'

05 The command request should return one of the following outputs:

1. If the describe-tags command output returns an empty array (i.e. []), as shown in the example below, the verified ASG is not tagged, therefore the audit process for the selected resource ends here:
   []
   

2. If the command output returns a set of tags that is different than the one identified at step no. 1, as shown in the example below, the verified AWS Auto Scaling Group does not belong to your web tier, therefore the audit process for the selected resource ends here:
   [
       {
           "Value": "Admin",
           "Key": "AccountManager"
       }
   ]
   

3. If the describe-tags command output returns a set of tags that match the one identified at step no. 1 (e.g. <web_tier_tag>:<web_tier_tag_value>), as shown in the example below, the verified AWS ASG is tagged as a web-tier resource, therefore the audit process continues with the next step:
   [
       {
           "Key": "<web_tier_tag>",
           "Value": "<web_tier_tag_value>"
       }
   ]
   

06 Run describe-auto-scaling-groups command (OSX/Linux/UNIX) with custom query filters to return the name of the launch configuration associated with the selected web-tier ASG:

aws autoscaling describe-auto-scaling-groups
	--region us-east-1
	--query 'AutoScalingGroups[*].LaunchConfigurationName'

07 The command output should return the name of the associated launch configuration:

[
    "cc-launch-configuration"
]

08 Run describe-launch-configurations command (OSX/Linux/UNIX) using the name of the launch configuration returned at the previous step as identifier and custom query filters to describe the user data (usually a configuration script) defined for the EC2 instances launched within your web-tier ASG with the selected launch configuration:

aws autoscaling describe-launch-configurations
	--region us-east-1
	--launch-configuration-names cc-launch-configuration
	--query "LaunchConfigurations[*].UserData"

09 The command request should return one of the following outputs:

1. If the describe-launch-configurations command output returns an empty string (i.e. “”), as shown in the example below, there is no user data defined for the EC2 instances provisioned by the selected launch configuration within the specified web-tier ASG:
   [
       ""
   ]
   

2. If the command output returns a configuration script that that doesn't contain the following block: "#!/bin/bash curl https://s3.amazonaws.com//aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O chmod +x ./awslogs-agent-setup.py ./awslogs-agent-setup.py -n -r <AWS_REGION> -c s3://<S3_BUCKET_NAME>/<CLOUDWATCH_AGENT_CONFIG_FILE>", the EC2 instances launched within the specified web-tier ASG, by the selected launch configuration, are not using an agent to publish their log files to AWS CloudWatch:
   [
       "#!/bin/bash
       yum update -y
       service httpd start
       chkconfig httpd on"
   ]
   

10 Repeat steps no. 4 – 9 to verify other web-tier ASG launch configurations, available in the selected region, for Cloudwatch Logs agent installation details.

11 Change the AWS region by updating the --region command parameter value and repeat steps no. 2 – 10 to perform the audit process for other regions.

01 Create a sample agent configuration file for Amazon Linux and save it as a text file named "instance-logs.cfg" then upload the file to AWS S3. This is basically a text file that describes the log file(s) to monitor and the log groups and log streams to upload them to. The AWS Cloudwatch Logs agent consumes this configuration file and starts monitoring and uploading all the log files defined in it. The following is a sample agent configuration file for the general system activity log on Amazon Linux (i.e. "/var/log/messages"):

[general]
state_file = /var/awslogs/state/agent-state

[/var/log/messages]
file = /var/log/messages
log_group_name = /var/log/messages
log_stream_name = {instance_id}
datetime_format = %b %d %H:%M:%S

02 Sign in to the AWS Management Console.

03 Navigate to EC2 dashboard at https://console.aws.amazon.com/ec2/.

04 In the navigation panel, under AUTO SCALING, click Launch Configurations.

05 Select the web-tier ASG launch configuration that you want to re-create (see Audit section part I to identify the right launch configuration).

06 Click the Actions dropdown button from the dashboard top menu and select Copy launch configuration option.

07 On Copy Launch Configuration from <LAUNCH_CONFIGURATION_NAME> page, within Launch configuration details section, click Edit details to edit the template.

08 Click Advanced Details to display the panel with the advanced settings and paste the following script inside the User data box (ensure that you replace the <AWS_REGION> with the appropriate AWS region and the <S3_BUCKET_NAME> with the name of the S3 bucket that holds the agent configuration file defined at step no. 1). Once the script is pasted, click Skip to review to finish the configuration process:

#!/bin/bash
curl https://s3.amazonaws.com//aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O
chmod +x ./awslogs-agent-setup.py
./awslogs-agent-setup.py -n -r <AWS_REGION> -c s3://<S3_BUCKET_NAME>/instance-logs.cfg

09 Click Create launch configuration. In the Select an existing key pair or create a new key pair dialog box, select Choose an existing key pair and use the same key pair as the source launch configuration. Click Create launch configuration to make a copy of the specified web-tier ASG launch configuration. Click Close to return to the EC2 dashboard.

10 In the navigation panel, under AUTO SCALING, click Auto Scaling Groups.

11 Select the web-tier Auto Scaling Group that you want to reconfigure (see Audit section part I to identify the right ASG resource).

12 Select the Details tab from the dashboard bottom panel and click Edit to change the group configuration.

13 Select the name of the launch configuration created at steps no. 6 – 9 from the Launch configuration dropdown list, then click Save to apply the changes. Once a new EC2 instance is launched within the selected web-tier ASG, you should be able to see and access the newly created AWS CloudWatch log stream and log group, populated with the instance logging data.

14 Repeat steps no. 5 – 13 to reconfigure other web-tier ASGs, available in the selected region, to automatically install the AWS Cloudwatch Logs agent on the EC2 instances launched inside the group.

15 Change the AWS region from the navigation bar and repeat steps no. 5 – 14 for other regions.

01 Create a sample agent configuration file for Amazon Linux and save it as a text file named "instance-logs.cfg" then upload the file to AWS S3. This represents a text file that describes the log files to monitor and the log groups and log streams to upload them to. The AWS Cloudwatch Logs agent consumes this configuration file and starts monitoring and uploading all the log files defined in it. The following is a sample agent configuration file for the general system activity log on Amazon Linux (i.e. "/var/log/messages"):

[general]
state_file = /var/awslogs/state/agent-state

[/var/log/messages]
file = /var/log/messages
log_group_name = /var/log/messages
log_stream_name = {instance_id}
datetime_format = %b %d %H:%M:%S

02 Now define the installation script for the AWS Cloudwatch Logs agent. Paste the following script into a text file named "awslogs-agent-setup.txt" (make sure that you replace the <AWS_REGION> with the necessary AWS region and the <S3_BUCKET_NAME> with the name of the S3 bucket that has the agent configuration file defined at the previous step):

#!/bin/bash
curl https://s3.amazonaws.com//aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O
chmod +x ./awslogs-agent-setup.py
./awslogs-agent-setup.py -n -r <AWS_REGION> -c s3://<S3_BUCKET_NAME>/instance-logs.cfg

03 Run describe-launch-configurations command (OSX/Linux/UNIX) using the name of the web-tier ASG launch configuration that you want to re-create as identifier (see Audit section part II to identify the right resource), to describe its configuration details, information required later when the new launch configuration will be created:

aws autoscaling describe-launch-configurations
	--region us-east-1
	--launch-configuration-names cc-launch-configuration

04 The command output should return the requested details:

{
    "LaunchConfigurations": [
        {
            "BlockDeviceMappings": [
                {
                    "DeviceName": "/dev/xvda",
                    "Ebs": {
                        "DeleteOnTermination": true,
                        "VolumeSize": 100,
                        "VolumeType": "gp2"
                    }
                }
            ],

	      ...

            "KeyName": "cc-linux-ssh-key",
            "SecurityGroups": [
                "sg-12345678"
            ],
            "LaunchConfigurationName": "cc-launch-configuration",
            "KernelId": "",
            "RamdiskId": "",
            "ImageId": "ami-abcd1234",
            "InstanceType": "c4.xlarge"
        }
    ]
}

05 Run create-launch-configuration command (OSX/Linux/UNIX) using the configuration metadata returned at the previous step to create a new launch configuration that runs the installation script (user data) for the AWS Cloudwatch Logs agent, defined at step no. 2. The following CLI command example creates an ASG launch configuration named "cc-new-launch-configuration", based on an AWS AMI identified by the ID "ami-abcd1234", with an IAM instance profile identified by the name "cc-web-instance-profile" and the required user data to install the Cloudwatch Logs agent, available within "awslogs-agent-setup.txt" file (the command does not produce an output):

aws autoscaling create-launch-configuration
	--region us-east-1
	--launch-configuration-name cc-new-launch-configuration
	--image-id ami-abcd1234
	--instance-type c4.xlarge
	--key-name cc-linux-ssh-key
	--security-groups sg-12345678
	--iam-instance-profile cc-web-instance-profile
	--user-data file://awslogs-agent-setup.txt
	--block-device-mappings "[{\"DeviceName\": \"/dev/xvda\",\"Ebs\":{\"VolumeSize\":100}}]"

06 Run update-auto-scaling-group command (OSX/Linux/UNIX) to update the configuration for your web-tier Auto Scaling Group, in order to replace the existing launch configuration with the new one created at the previous step (the command does not return an output):

aws autoscaling update-auto-scaling-group
	--region us-east-1
	--auto-scaling-group-name cc-auto-scaling-group
	--launch-configuration-name cc-new-launch-configuration

07 Repeat steps no. 3 – 6 to reconfigure other web-tier ASGs, available in the selected region, to automatically install the AWS Cloudwatch Logs agent on the EC2 instances provisioned inside the group.

08 Change the AWS region by updating the --region command parameter value and repeat steps no. 2 – 7 to perform the process for other regions.