Open menu
-->

AWS ELB Instances Distribution Across AZs

Cloud Conformity allows you to automate the auditing process of this resolution page. Register for a 14 day evaluation and check your compliance level for free!

Start a Free Trial Product features
Reliability

Risk level: Medium (should be achieved)

Ensure that the EC2 instances registered to your Amazon Elastic Load Balancing (ELB) are evenly distributed across all Availability Zones (AZs) in order to improve the ELBs configuration reliability. To route incoming requests evenly across the registered EC2 instances within the Availability Zones, the cross-zone load balancing feature must be enabled.

This rule resolution is part of the Cloud Conformity Security Package

Having a balanced distribution of EC2 instances across all AZs will improve the availability and reliability of your load balanced applications. The more Availability Zones assigned and better the spread, the more redundancy and availability within your load balancing environment.

Audit

To determine if your ELB backend instances are distributed evenly across all assigned AZs, perform the following:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to EC2 dashboard at https://console.aws.amazon.com/ec2/.

03 In the left navigation panel, under LOAD BALANCING section, choose Load Balancers.

04 Select the AWS ELB that you want to examine.

05 Select Description tab from the dashboard bottom panel and check the Cross-Zone Load Balancing feature status. If the feature status is set to Disabled, follow the instructions outlined in the Enable AWS ELB Cross-Zone Load Balancing conformity rule to enable the feature, otherwise skip this step and continue the audit process with the next step.

06 Select Instances tab from the dashboard bottom panel.

07 Under Edit Availability Zones section, check the value available within the Instance Count column for each Availability Zone assigned. If the instance count value (number) is not even, e.g.

Distance Count

the backend instances registered to the selected ELB are not evenly distributed across AZs.

08 Repeat steps no. 4 – 7 to verify the distribution of backend instances for other ELBs available within the current region.

09 Change the AWS region from the navigation bar and repeat the audit process for other regions.

Using AWS CLI

01 Run describe-load-balancers command (OSX/Linux/UNIX) with custom query filters to list the names of all your AWS ELBs provisioned within the selected region:

aws elb describe-load-balancers
	--region us-east-1
	--output table
	--query 'LoadBalancerDescriptions[*].LoadBalancerName'

02 The command output should return a table with the requested ELB name(s):

---------------------------
|  DescribeLoadBalancers  |
+-------------------------+
|  MyCCProductionELB      |
|  CCFrontendWebELB       |
+-------------------------+

03 Now run describe-load-balancer-attributes command (OSX/Linux/UNIX) to verify if the Cross-Zone Load Balancing feature is enabled for the selected AWS ELB:

aws elb describe-load-balancer-attributes
	--region us-east-1
	--load-balancer-name MyCCProductionELB
	--query 'LoadBalancerAttributes.CrossZoneLoadBalancing.Enabled'

04 The command output should expose the Cross-Zone Load Balancing status (true for enabled, false for disabled). If the value returned is false, the Cross-Zone Load Balancing feature is not currently enabled, therefore you must follow the instructions outlined in the Enable AWS ELB Cross-Zone Load Balancing conformity rule to enable the feature, otherwise, if the value returned is true, skip this step and continue the audit process with the next step:

true

05 Run describe-load-balancers command (OSX/Linux/UNIX) using the name of the ELB that you want to examine as identifier and custom filtering to list the Availability Zones assigned to the selected AWS ELB:

aws elb describe-load-balancers
	--region us-east-1
	--load-balancer-name MyCCProductionELB
	--query 'LoadBalancerDescriptions[*].AvailabilityZones[]'

06 The command output should return the names of the AZs assigned to the selected ELB:

[
    "us-east-1a",
    "us-east-1b"
]

07 Run again describe-load-balancers command (OSX/Linux/UNIX) using the name of the ELB that you want to examine as identifier and custom query filters to list the EC2 instances currently registered to the selected ELB:

aws elb describe-load-balancers
	--region us-east-1
	--load-balancer-name MyCCProductionELB
	--query 'LoadBalancerDescriptions[*].Instances[*].InstanceId[]'

08 The command output should return the IDs of the backend instances registered to the selected ELB:

[
    "i-0325887e44fff2d15",
    "i-08e5da433bc07545d",
    "i-0976ce5c515375cb3"
]

09 Based on the metadata returned at step no. 6 and 8, if the selected load balancer has more backend EC2 instances than AZs, the instances registered to the AWS ELB are not evenly distributed across AZs.

10 Repeat steps no. 4 – 9 to verify the distribution of backend instances for other ELBs available within the current region.

11 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 - 10 to perform the audit process for other regions.

Remediation / Resolution

To equally distribute your existing ELB backend instances across all Availability Zones within the selected AWS region, you need to add new Availability Zones to the ELB configuration and migrate the registered instances between these AZs. To implement this strategy, perform the following actions:

Note: As example, this section will explain how to add a new AZ named us-east-1c to the existing ELB configuration and migrate a Linux EC2 instance from us-east-1a to the newly added AZ, i.e. us-east-1c, within the US East (N. Virginia) region:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to EC2 dashboard at https://console.aws.amazon.com/ec2/.

03 In the left navigation panel, under LOAD BALANCING section, choose Load Balancers.

04 Select the AWS ELB that you want to reconfigure (see Audit section part I to identify the right resource).

05 Select Instances tab from the dashboard bottom panel and click the Edit Availability Zones button to add the necessary AZ.

06 Within Add and Remove Subnets dialog box, click the + button next to a subnet available within us-east-1c Availability Zone to add the selected AZ to the ELB configuration. Click Save to apply the changes.

07 Now you need migrate one of the backend instances to the newly added AZ by relaunching it within the new AZ (in this case us-east-1c). To start with the migration process, you need to create first an Amazon Machine Image (AMI) from the instance. The image is required to re-create the instance in the new Availability Zone, in the same AWS region. To instantiate the AMI, perform the following actions:

  1. In the navigation panel, under INSTANCES section, select Instances.
  2. Select the EC2 instance that you want to migrate to another AZ (see Audit section part I to identify the right resource).
  3. Click the Actions dropdown button from the dashboard top menu, select Image and click Create Image.
  4. Inside Create Image dialog box, provide the following information:
    • Enter a name for the new AMI in the Image Name box.
    • In the Image description box, provide a description that reflects the instance usage.
    • Leave No reboot option unchecked so that Amazon can guarantee the file system integrity for the new image.
  5. Click Create Image to submit the request to create the image. Click Close to return to the EC2 dashboard. The AMI creation process may take few minutes. Once the process is complete the image status should change from pending to available.

08 Once the AMI is ready, use it to re-create the EC2 instance into the new Availability Zone. To launch the instance, perform the following:

  1. In the left navigation panel, under IMAGES section, select AMIs.
  2. Select the Amazon Machine Image (AMI) created at step no. 7.
  3. Click the Launch button from the EC2 dashboard top menu to initiate the deployment.
  4. On the Choose Instance Type page, select the same EC2 instance type used by the existing resource then click Next: Configure Instance Details.
  5. On the Choose an Amazon Machine Image (AMI) page, choose My AMIs tab then select the image created at step no. 3.
  6. On the Configure Instance Details page, select the Availability Zone where the EC2 instance will be re-created from the Subnet dropdown list and configure any other options such as IAM role, Monitoring and Shutdown Behavior based on the source instance configuration. Click Next: Add Storage and go through the next pages until you reach the Configure Security Group page, without changing any configuration.
  7. On the Configure Security Groups, choose Select an existing security group and select the security group(s) currently assigned to the running EC2 instance. Click the Review and Launch button, review your new instance configuration details and click Launch.
  8. In the Select an existing key pair or create a new key pair dialog box, select Choose an existing key pair and use the same key pair as the running resource. Check I acknowledge that I have access to the selected private key file (<key_name.pem) option then click Launch Instances.
  9. Click View Instances to return to the Instances page. The new instance will have the same data and system configuration as the source instance but will be located in a different AZ (i.e. us-east-1c), within the same region.

09 Transfer the Elastic IP (EIP) from the source EC2 instance to the new instance in order to migrate the public IP reference as well (if required). To transfer the Elastic IP, perform the following actions:

  1. In the navigation panel, under NETWORK & SECURITY section, select Elastic IPs.
  2. Select the EIP address attached to the source instance, click the Actions dropdown button then select Disassociate Address.
  3. In the Disassociate Address dialog box, review the details then click Yes, Disassociate.
  4. Select the same address, disassociated in the previous step, click the Actions dropdown button then select Associate Address.
  5. In the Associate Address dialog box, select the EC2 instance created at step no. 8 from Instance dropdown list then click Associate to attach the EIP.

10 Once you have verified that your new EC2 instance is working 100% within the new AZ, register the new instance to the load balancer, then remove the source backend instance from the ELB configuration by performing the following:

  1. In the left navigation panel, under LOAD BALANCING section, choose Load Balancers.
  2. Select the Amazon ELB that you want to reconfigure.
  3. Select Instances tab from the dashboard bottom panel and click Edit Instances.
  4. Inside Add and Remove Instances dialog box, check to add the new instance and uncheck to remove the old one (source instance) from the ELB configuration.
  5. Click Save to apply the new changes. Once all backend instances are successfully registered with the ELB, the instance count between the assigned AZs should be equal, e.g.
  6. instance count

11 Repeat steps no. 4 – 10 to reconfigure other Amazon ELBs available within the current region.

12 Change the AWS region from the navigation bar and repeat the audit process for other regions.

Using AWS CLI

01 Run enable-availability-zones-for-load-balancer command (OSX/Linux/UNIX) using the name of the ELB that you want to reconfigure (see Audit section part II to identify the right resource) to add a new Availability Zone (e.g. us-east-1c) to the selected load balancer configuration:

aws elb enable-availability-zones-for-load-balancer
	--region us-east-1
	--load-balancer-name MyCCProductionELB
	--availability-zones us-east-1c

02 The command output should list all Availability Zones assigned to the selected load balancer:

{
    "AvailabilityZones": [
        "us-east-1c",
        "us-east-1a",
        "us-east-1b"
    ]
}

03 Run create-image command (OSX/Linux/UNIX) to create an Amazon Machine Image (AMI) from the instance that you want to migrate (i.e. source instance). The image is required to re-create the instance within the newly added Availability Zone (us-east-1c), in the same AWS region. Include the –no-reboot command parameter to guarantee the file system integrity for the new AMI:

aws ec2 create-image
	--region us-east-1
	--instance-id i-0c5739f191e197977
	--name "US-EAST-1A EC2 Instance Image"
	--description "Instance AMI for AZ migration."
	--no-reboot

04 The command output should return the new AMI ID:

{
    "ImageId": "ami-cd071fc5"
}

05 Get the configuration details from the source backend instance, required for the next step. Run describe-instances command (OSX/Linux/UNIX) using the ID of the instance that you want to re-create (see Audit section part II to identify the right resource) to describe its configuration details:

aws ec2 describe-instances
	--region us-east-1
	--instance-ids i-0c5739f191e197977

06 The command output should return the source EC2 instance configuration metadata:

{
    "Reservations": [
        {
            "OwnerId": "123456789012",
            "ReservationId": "r-00b20341832e7c7fa",
            "Instances": [
                {
                    "Monitoring": {
                        "State": "disabled"
                    },
                    "EbsOptimized": false,
                    "LaunchTime": "2017-07-21T16:14:25.000Z",
                    "PublicIpAddress": "56.80.41.31",
                    "PrivateIpAddress": "172.31.7.105",
                    "ProductCodes": [],
                    "StateTransitionReason": "",
                    "InstanceId": "i-0c5739f191e197977",

                    ...

                    "EnaSupport": true,
                    "ImageId": "ami-0b33dc72",
                    "KeyName": "prod-web-key",
                    "Architecture": "x86_64",
                    "RootDeviceType": "ebs",
                    "RootDeviceName": "/dev/xvda",
                    "VirtualizationType": "hvm",
                    "AmiLaunchIndex": 0
                }
            ]
        }
    ]
}

07 Execute run-instances command (OSX/Linux/UNIX) using the configuration information returned at the previous step to launch an instance from the image created at step no. 3. The following command example re-creates an EC2 instance inside the us-east-1c Availability Zone (identified by the subnet ID subnet-73b8c319), within the US East region, using an AMI with the ID ami-cd071fc5:

aws ec2 run-instances
	--region us-east-1
	--image-id ami-cd071fc5
	--count 1
	--instance-type m3.medium
	--key-name prod-web-key
	--security-group-ids sg-df152af1
	--subnet-id subnet-73b8c319
	--no-ebs-optimized

08 The command output should return the new EC2 instance metadata:

{
    "Reservations": [
        {
            "OwnerId": "123456789012",
            "ReservationId": "r-0f823c80bcf85d5ca",
            "Instances": [
                {
                    "EbsOptimized": false,
                    "LaunchTime": "2017-09-11T19:56:20.000Z",
                    "PrivateIpAddress": "172.31.65.103",
                    "ProductCodes": [],
                    "StateTransitionReason": "",

                    ...

                    "ImageId": "ami-cd071fc5",
                    "KeyName": "prod-web-key",
                    "Architecture": "x86_64",
                    "RootDeviceType": "ebs",
                    "RootDeviceName": "/dev/xvda",
                    "VirtualizationType": "hvm",
                    "AmiLaunchIndex": 0
                }
            ]
        }
    ]
}

09 Now transfer the Elastic IP (EIP) from the source EC2 instance to the new instance in order to migrate the reference for the public IP (if any). To transfer the EIP, perform the following commands:

  1. Run disassociate-address command (OSX/Linux/UNIX) to detach the Elastic IP address from the source instance (the command does not produce an output):
    aws ec2 disassociate-address
    	--region us-east-1
    	--public-ip 56.80.41.31
    
  2. Run associate-address command (OSX/Linux/UNIX) to associate the EIP address detached at the previous step with the new EC2 instance, identified by the ID i-08e5da433bc07545d:
    aws ec2 associate-address
    	--instance-id i-08e5da433bc07545d
    	--allocation-id eipalloc-c83fe9e1
    

10 Once you have verified that your new EC2 instance is working 100% within the new AZ, register the new instance to the load balancer, then deregister the source backend instance from the ELB configuration by performing the following commands:

  1. First, run register-instances-with-load-balancer command (OSX/Linux/UNIX) to register the new backend instance to the selected AWS ELB (the command does not produce an output):
    aws elb register-instances-with-load-balancer
    	--region us-east-1
    	--load-balancer-name MyCCProductionELB
    	--instances i-08e5da433bc07545d
    
  2. Run deregister-instances-from-load-balancer command (OSX/Linux/UNIX) to deregister the necessary EC2 backend instance (source instance) from the selected Amazon ELB (the command does not return an output):
    aws elb deregister-instances-from-load-balancer
    	--region us-east-1
    	--load-balancer-name MyCCProductionELB
    	--instances i-0c5739f191e197977
    

11 Repeat steps no. 1 – 10 to reconfigure other Amazon ELBs available within the current region.

12 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 - 11 to perform the entire process for other regions.

References

Publication date 2017-13-09