Use the Conformity Knowledge Base AI to help improve your Cloud Posture

ELBv2 Instances Distribution Across AZs

Trend Micro Cloud One™ – Conformity is a continuous assurance tool that provides peace of mind for your cloud infrastructure, delivering over 750 automated best practice checks.

Risk Level: Medium (should be achieved)

Ensure that the EC2 instances (targets) registered to your Amazon Application Load Balancers (ALBs) and Network Load Balancers (NLBs) are evenly distributed across all Availability Zones (AZs) in order to improve the reliability of your load balancers configuration.

Reliability

Having a balanced distribution of target instances across all Availability Zones will improve the availability and reliability of the application(s) behind your load balancers (ELBv2). The more Availability Zones assigned and greater the spread, the better redundancy and availability within your load balancing environment.


Audit

To determine if the EC2 target instances registered to your load balancers are distributed evenly across all assigned Availability Zones, perform the following actions:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to EC2 dashboard at https://console.aws.amazon.com/ec2/.

03 In the left navigation panel, under LOAD BALANCING section, choose Target Groups.

04 Select the target group associated with the AWS ELBv2 load balancer that you want to examine. To check the resources association, verify the Load balancer attribute value available on the Description tab.

05 Select Targets tab from the dashboard bottom panel to view the registered targets.

06 Under Availability Zones section, check the value available within the Target count column for each Availability Zone assigned. If the target instance count value (number) is not even, e.g.

ELBv2 Target Count

the target instances registered to the selected Amazon ELBv2 load balancer are not evenly distributed across Availability Zones.

07 Repeat steps no. 4 – 6 to verify the distribution of EC2 target instances for other AWS load balancers available within the current region.

08 Change the AWS region from the navigation bar and repeat the audit process for other regions.

Using AWS CLI

01 Run describe-load-balancers command (OSX/Linux/UNIX) using custom query filters to list the ARNs of all existing AWS ELBv2 load balancers available in the selected region:

aws elbv2 describe-load-balancers
	--region us-east-1
	--query 'LoadBalancers[*].LoadBalancerArn'

02 The command output should return a table with the requested Amazon Resource Names (ARNs):

[
    "arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/app/cc-application-elb/aaaabbbbccccdddd",
"arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/net/cc-network-elb/aaaabbbbccccdddd"

]

03 Run describe-load-balancers command (OSX/Linux/UNIX) using the ARN of the load balancer that you want to examine as identifier and custom filtering to list the Availability Zones assigned to the selected AWS ELBv2 resource (ALB or NLB):

aws elbv2 describe-load-balancers
	--region us-east-1
	--load-balancer-arns arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/app/cc-application-elb/aaaabbbbccccdddd
	--query 'LoadBalancers[*].AvailabilityZones[*].ZoneName[]'

04 The command output should return the names of the AZs currently assigned to the selected load balancer:

[
    "us-east-1a",
    "us-east-1b"
]

05 Run describe-target-groups command (OSX/Linux/UNIX) using the ARN of the load balancer that you want to examine as identifier and custom query filters to expose the ARN of the target group associated with the selected ELBv2 resource:

aws elbv2 describe-target-groups
	--region us-east-1
	--load-balancer-arn arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/app/cc-application-elb/aaaabbbbccccdddd
	--query 'TargetGroups[*].TargetGroupArn'

06 The command output should return the ARN of the associated target group:

[

"arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/cc-alb-target-group/aaaabbbbccccdddd"

]

07 Run describe-target-health command (OSX/Linux/UNIX) using the ARN of the target group returned at the previous step as identifier and custom query filters to list the ID(s) of the target instances registered to the selected AWS ELBv2 load balancer:

aws elbv2 describe-target-health
	--region us-east-1
	--target-group-arn arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/cc-alb-target-group/aaaabbbbccccdddd
	--query 'TargetHealthDescriptions[*].Target.Id'

08 The command output should return the IDs of the registered EC2 target instances:

[
    "i-0e9649c48fe61c2f5",
    "i-0cadda433bc075d46",
    "i-048dce5c515375bc2"
]

09 Based on the metadata returned at step no. 4 and 8, if the selected AWS ELBv2 load balancer has more target instances than Availability Zones, the EC2 instances registered to the load balancer are not evenly distributed across AZs.

10 Repeat steps no. 3 – 9 to verify the distribution of target instances for other AWS ELBv2 load balancer provisioned in the current region.

11 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 - 10 to perform the entire audit process for other regions.

Remediation / Resolution

To equally distribute your existing EC2 target instances across all Availability Zones within the selected AWS region, you need to add new Availability Zones to the ELBv2 load balancer configuration and migrate the registered instances between these AZs. To implement this method, perform the following actions:

Note: As example, this section demonstrates how to add a new AZ named us-east-1c to an existing AWS Application Load Balancer configuration and migrate a Linux EC2 instance from us-east-1b to the newly added AZ, i.e. us-east-1c, within the US East (N. Virginia) region:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to EC2 dashboard at https://console.aws.amazon.com/ec2/.

03 In the left navigation panel, under LOAD BALANCING section, choose Load Balancers.

04 Select the AWS ELBv2 load balancer that you want to reconfigure (see Audit section part I to identify the right resource).

05 Select Description tab from the dashboard bottom panel to view the ELBv2 resource configuration.

06 In the Basic Configuration section, click Edit availability zones to change the AZs current configuration.

07 Within Edit availability zones dialog box, add a new AZ (e.g. us-east-1c) by selecting the subnet of the wanted AZ, available in the subnets list. Once added, click Save to apply the configuration changes.

08 Now migrate one of the target instances to the newly added AZ by relaunching it within the new AZ (in this case us-east-1c). To start with the migration process, you need to create first an Amazon Machine Image (AMI) from the target instance. The image is required to re-create the backend instance in the new Availability Zone, within the same AWS region. To create the AMI, perform the following actions:

  1. In the navigation panel, under INSTANCES section, select Instances.
  2. Select the EC2 target instance that you want to migrate to another Availability Zone (see Audit section part I to identify the right AZ).
  3. Click the Actions dropdown button from the dashboard top menu, select Image and click Create Image.
  4. Inside Create Image dialog box, provide the following information:
    • Enter a name for the new AMI in the Image Name box.
    • In the Image description box, provide a description that reflects the instance usage (e.g. AWS ALB target instance).
    • Leave No reboot option unchecked so that Amazon Web Services can guarantee the file system integrity for the new image.
  5. Click Create Image to submit the request to create the image. Click Close to return to the AWS EC2 dashboard. The AMI creation process may take few minutes. Once the process is completed, the image status should change from pending to available.

09 Once your AWS AMI is ready, use it to re-create the target instance into the new Availability Zone. To launch the EC2 instance, perform the following:

  1. In the left navigation panel, under IMAGES section, select AMIs.
  2. Select the Amazon Machine Image (AMI) created at step no. 8.
  3. Click the Launch button from the EC2 dashboard top menu to initiate the deployment.
  4. On the Choose Instance Type page, select the same EC2 instance type used by the source resource, then click Next: Configure Instance Details.
  5. On the Choose an Amazon Machine Image (AMI) page, choose My AMIs tab then select the newly created image.
  6. On the Configure Instance Details page, select the Availability Zone where the EC2 target instance will be re-created from the Subnet dropdown list and configure any other options such as IAM role, Monitoring and Shutdown Behavior based on the source instance configuration. Click Next: Add Storage and go through the next pages until you reach the Configure Security Group page, without changing any configuration settings.
  7. On the Configure Security Groups, choose Select an existing security group and select the security group(s) currently assigned to the source EC2 instance. Click the Review and Launch button, review your new target instance configuration details and click Launch.
  8. In the Select an existing key pair or create a new key pair dialog box, select Choose an existing key pair and use the same key pair as the source EC2 resource. Check I acknowledge that I have access to the selected private key file (<key_name>.pem) option then click Launch Instances.
  9. Click View Instances to return to the Instances page. The new EC2 target instance will have the same data and system configuration as the source instance but it will be located in a different AZ (i.e. us-east-1c), within the same region.
10 Transfer the Elastic IP (if any allocated) from the source instance to the new target instance in order to migrate the public IP reference. To transfer the Elastic IP (EIP), perform the following actions:
  1. In the navigation panel, under NETWORK & SECURITY section, select Elastic IPs.
  2. Select the EIP address attached to the source instance, click the Actions dropdown button then select Disassociate Address.
  3. In the Disassociate Address dialog box, review the details then click Yes, Disassociate.
  4. Select the same address, disassociated in the previous step, click the Actions dropdown button then select Associate Address.
  5. In the Associate Address dialog box, select the new target instance from Instance dropdown list then click Associate to attach the EIP.
11 Once you have verified that your new EC2 target instance is working 100% within the new AZ, register the new instance to the load balancer, then remove the source target instance from the ELBv2 resource configuration by performing the following actions:
  1. In the left navigation panel, under LOAD BALANCING section, choose Target Groups.
  2. Select the target group associated with the AWS ELBv2 load balancer that you want to reconfigure.
  3. Select Targets tab from the dashboard bottom panel, then click Edit to edit the registered targets configuration.
  4. Within Register and deregister targets dialog box, perform the following:
    • Select the new instance from the Instances list and click Add to registered button to add the new target instance to the Registered targets list.
    • Select the source (already registered) instance from the Registered targets list then click Remove button to remove it from the selected target group.
    • Click Save to apply the new configuration changes. Once all target instances are successfully registered with the ELBv2 load balancer, the target count between the assigned AZs should be equal, e.g. Target Count Equal.
12 Repeat steps no. 4 – 11 to reconfigure the distribution of target instances for other ELBv2 load balancer available within the current region. 13 Change the AWS region from the navigation bar and repeat the audit process for other regions.

Using AWS CLI

01 Run set-subnets command (OSX/Linux/UNIX) using the ARN of the load balancer that you want to reconfigure (see Audit section part II to identify the right ELBv2 resource) to add a new Availability Zone (i.e. us-east-1c) to the selected Amazon ELBv2 load balancer configuration:

aws elbv2 set-subnets
	--region us-east-1
	--load-balancer-arn arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/app/cc-application-elb/aaaabbbbccccdddd
	--subnets subnet-aaaabbbb subnet-bbbbcccc subnet-ccccdddd

02 The command output should list all Availability Zones currently assigned to the load balancer:

{
    "AvailabilityZones": [
        {
            "SubnetId": "subnet-aaaabbbb",
            "ZoneName": "us-east-1a"
        },
        {
            "SubnetId": "subnet-bbbbcccc",
            "ZoneName": "us-east-1b"
        },
        {
            "SubnetId": "subnet-ccccdddd",
            "ZoneName": "us-east-1c"
        }
    ]
}

03 Run create-image command (OSX/Linux/UNIX) to create an Amazon Machine Image (AMI) from the instance that you want to migrate (i.e. source target instance). The image is required to re-create the instance within the newly added Availability Zone, in the same AWS region. Include the –no-reboot command parameter to guarantee the file system integrity for the new AMI:

aws ec2 create-image
	--region us-east-1
	--instance-id i-0e9649c48fe61c2f5
	--name "US-EAST-1A Target Instance Image"
	--description "ELBv2 Target Instance AMI for AZ migration."
	--no-reboot

04 The command output should return the new image ID:

{
    "ImageId": "ami-12345678"
}

05 Get the configuration details from the source target instance, required for the next step. Run describe-instances command (OSX/Linux/UNIX) using the ID of the instance that you want to re-create (see Audit section part II to identify the right EC2 resource) to describe its configuration details:

aws ec2 describe-instances
	--region us-east-1
	--instance-ids i-0e9649c48fe61c2f5

06 The command output should return the source target instance configuration metadata:

{
    "Reservations": [
        {
            "OwnerId": "123456789012",
            "ReservationId": "r-00b34341832e7c58a",
            "Instances": [
                {
                    "Monitoring": {
                        "State": "disabled"
                    },
                    "EbsOptimized": false,
                    "LaunchTime": "2018-01-11T16:14:00.000Z",
                    "StateTransitionReason": "",
                    "InstanceId": "i-0e9649c48fe61c2f5",

                    ...

                    "EnaSupport": true,
                    "KeyName": "cc-web-key",
                    "Architecture": "x86_64",
                    "RootDeviceType": "ebs",
                    "RootDeviceName": "/dev/xvda",
                    "VirtualizationType": "hvm",
                    "AmiLaunchIndex": 0
                }
            ]
        }
    ]
}

07 Execute run-instances command (OSX/Linux/UNIX) using the configuration information returned at the previous step to launch an instance from the image created at step no. 3. The following command example re-creates a target instance inside the "us-east-1c" Availability Zone (identified by the subnet ID subnet-ccccdddd), within the US East region, using an AMI with the ID ami-12345678:

aws ec2 run-instances
	--region us-east-1
	--image-id ami-12345678
	--count 1
	--instance-type c4.xlarge
	--key-name cc-web-key
	--security-group-ids sg-aaaabbbb
	--subnet-id subnet-ccccdddd
	--no-ebs-optimized

08 The command output should return the new EC2 target instance metadata:

{
    "Reservations": [
        {
            "OwnerId": "123456789012",
            "ReservationId": "r-0e463c80bcf85d7d1",
            "Instances": [
                {
                    "EbsOptimized": false,
                    "LaunchTime": "2018-01-21T20:56:20.000Z",
                    "ProductCodes": [],
                    "StateTransitionReason": "",

                    ...

                    "KeyName": "cc-web-key",
                    "Architecture": "x86_64",
                    "RootDeviceType": "ebs",
                    "RootDeviceName": "/dev/xvda",
                    "VirtualizationType": "hvm",
                    "AmiLaunchIndex": 0
                }
            ]
        }
    ]
}

09 Transfer the Elastic IP (if any) from the source instance to the new target instance in order to migrate the public IP reference. To transfer the necessary EIP, perform the following commands:

  1. Run disassociate-address command (OSX/Linux/UNIX) to detach the Elastic IP address from the source instance (the command does not produce an output):
    aws ec2 disassociate-address
    	--region us-east-1
    	--public-ip 59.180.141.23
    
  2. Run associate-address command (OSX/Linux/UNIX) to associate the EIP address detached at the previous step with the new target instance, identified by the ID i-0bf85d7d1bc07545d:
    aws ec2 associate-address
    	--instance-id i-0bf85d7d1bc07545d
    	--allocation-id eipalloc-abcd1234
    

10 Once you have verified that your new EC2 target instance is working 100% within the new AZ, register the new target instance to the load balancer, then deregister the source instance from the ELBv2 resource configuration by performing the following commands:

  1. Run register-targets command (OSX/Linux/UNIX) to register the new target instance, identified by the ID i-0bf85d7d1bc07545d, to the target group associated with the selected ELBv2 load balancer (the command does not produce an output):
    aws elbv2 register-targets
    	--region us-east-1
    	--target-group-arn arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/cc-alb-target-group/aaaabbbbccccdddd
    	--targets Id=i-0bf85d7d1bc07545d
    
  2. Run deregister-targets command (OSX/Linux/UNIX) to remove the source target instance, identified by the ID i-0e9649c48fe61c2f5, from the target group associated with the selected AWS ELBv2 load balancer (the command does not produce an output):
    aws elbv2 deregister-targets
    	--region us-east-1
    	--target-group-arn arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/cc-alb-target-group/aaaabbbbccccdddd
    	--targets Id=i-0e9649c48fe61c2f5
    

11 Repeat steps no. 1 – 10 to reconfigure the distribution of target instances for other ELBv2 load balancer available in the current region.

12 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 – 11 to perform the entire process for other regions.

References

Publication date Feb 5, 2018

Unlock the Remediation Steps


Free 30-day Trial

Automatically audit your configurations with Conformity
and gain access to our cloud security platform.

Confirmity Cloud Platform

No thanks, back to article

You are auditing:

ELBv2 Instances Distribution Across AZs

Risk Level: Medium