Open menu

EC2 Instances with Scheduled Events

Cloud Conformity allows you to automate the auditing process of this resolution page. Register for a 14 day evaluation and check your compliance level for free!

Start a Free Trial Product features
Knowledge Base EC2 EC2 Instances with Scheduled Events
Last updated: 26 April 2018
Reliability

Risk level: High (not acceptable risk)

Determine if there are any EC2 instances scheduled for retirement and/or maintenance in your AWS account and take the necessary steps (reboot, restart or re-launch) to resolve them. The EC2 instances support multiple types of scheduled events such as Reboot (instance-reboot or system-reboot) - the instance or the underlying host machine is rebooted, Instance Stop (instance-stop) – the instance is stopped and started to migrate it to a new host machine, Instance Retirement (instance-retirement) - the instance is terminated and System Maintenance (system-maintenance) - the instance is taken offline for a short period of time, then rebooted.

This rule resolution is part of the Cloud Conformity Tool

Monitoring EC2 scheduled events within your AWS account will help you prevent unexpected downtime and data loss, improving the reliability and availability of your AWS EC2 fleet. Note: This guide assumes that your EC2 instances are associated with Elastic IPs. If your instances do not have Elastic IPs attached, you will have to update their public IP reference(s) in your application code or within the DNS zone file after you stop and start the necessary instances, as each restarted instance gets a new public IP address.

Audit

To determine if Connection Draining is enabled, perform the following:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to EC2 dashboard at https://console.aws.amazon.com/ec2/.

03 On the EC2 dashboard main page, verify the Scheduled Events section for any EC2 instances that have scheduled events assigned, available in the current AWS region. If the Scheduled Events current status is set to "No events":

If the Scheduled Events current status is set to No events

there are no EC2 instances scheduled for retirement/maintenance within the current region, otherwise, the dashboard will display the number of EC2 instances that have scheduled events assigned, e.g.

If the Scheduled Events status displays one or more instances

If the Scheduled Events status displays one or more instances, click on the status link to access the Events page and identify the type of the scheduled event for each EC2 instance, listed in the Event Type column.

04 Change the AWS region from the navigation bar and repeat the audit process for other regions.

Using AWS CLI

01 Run describe-instance-status command (OSX/Linux/UNIX) using predefined filters to list the IDs of all EC2 instances that have scheduled events assigned, available in the selected region: 

aws ec2 describe-instance-status
	--region us-east-1
	--filters"Name=event.code,Values=instance-reboot,system-reboot, system-maintenance,instance-retirement,instance-stop"
	--output table
	--query 'InstanceStatuses[*].InstanceId'

02 The command output should return a table with the requested information. If the DescribeInstanceStatus table returned is empty, there are no EC2 instances scheduled for retirement or maintenance within the selected region, otherwise, the table will display the ID(s) of the EC2 instance(s) that have scheduled events allocated (as shown in the output example below): 

-------------------------
|DescribeInstanceStatus |
+-----------------------+
|  i-06a56079863b82f8a  |
|  i-04c1691e53b1576b0  |
+-----------------------+

03 Run again describe-instance-status command (OSX/Linux/UNIX) using custom output filtering to expose the event type for each EC2 instance scheduled for retirement or maintenance returned at the previous step: 

aws ec2 describe-instance-status
	--region us-east-1
	--instance-id i-06a56079863b82f8a
	--query 'InstanceStatuses[*].Events.Code'

04 The command output should return the event type for the selected EC2 instance: 

[
    "instance-stop"
]

05 Repeat steps no. 1 – 4 to perform the audit process for other AWS regions.

Remediation / Resolution

To resolve EC2 instances scheduled for retirement/maintenance based on the event type (see Audit section to identify the event type(s) assigned to your instance(s)), perform the following:

Actions for instance-reboot event:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to EC2 dashboard at https://console.aws.amazon.com/ec2/.

03 In the left navigation panel, under INSTANCES section, choose Instances.

04 Select the running EC2 instance scheduled for instance-reboot (see the Audit section to identify the appropriate instance).

05 Click on the Actions dropdown button from the dashboard top menu, select Instance State and click Reboot. (!) IMPORTANT: Performing this step will result in a short downtime for the application(s) running on the selected instance.

06 In the Reboot Instances dialog box, review the instance identifier and click Yes, Reboot to confirm the action.

07 Repeat steps no. 4 – 6 to reboot any other scheduled instances available in the current region.

08 Change the AWS region from the navigation bar and repeat the process for other regions.

Using AWS CLI

01 Run describe-instances command (OSX/Linux/UNIX) using the instance ID as identifier to reboot the necessary EC2 instance (if the command succeeds, no output is returned): 

aws ec2 reboot-instances
	--region us-east-1
	--instance-ids i-06a56079863b82f8a

02 Repeat step no. 1 to reboot any other scheduled instances available in the selected region.

03 Change the AWS region to repeat the process for other regions.

Actions for system-reboot:

No action is required on your end. The system reboot is managed by AWS and occurs during its scheduled maintenance window. AWS will send you an email prior to the system-reboot event with all the necessary details, including the start and the end date of the event.

Actions for instance-stop and instance-retirement events:

Using AWS Console

01 Navigate to EC2 dashboard at https://console.aws.amazon.com/ec2/.

02 In the navigation panel, under INSTANCES section, choose IInstances.

03 Select the running EC2 instance scheduled for instance-stop or instance-retirement events.

04 Click on the Actions dropdown button from the dashboard top menu, select Instance State and click Stop. (!) IMPORTANT: This step will incur downtime for the application(s) running on the selected instance.

05 In the Stop Instances dialog box, review the instance details and click Yes, Stop to confirm the action. The instance status will change from running to stopping to stopped.

06 Click again on the Actions dropdown button, select Instance State and click Start to restart the instance.

07 Inside the Start Instances dialog box, review the details and click Yes, Start to confirm the action. The instance status will change from stopped to pending to running.

08 Repeat steps no. 4 – 8 to stop and start any other scheduled instances available in the current region.

09 Change the AWS region from the navigation bar and repeat the process for other regions.

Using AWS CLI

01 Run stop-instances command (OSX/Linux/UNIX) to stop the EC2 instance scheduled for instance-stop or instance-retirement (see Audit section to get the appropriate instance ID). (!) IMPORTANT: This step will incur downtime for the application(s) running on the selected instance. The following command example stops an EC2 instance with the ID i-06a56079863b82f8a within the US East region: 

aws ec2 stop-instances
	--region us-east-1
	--instance-ids i-06a56079863b82f8a

02 The command output should return the instance current state metadata (highlighted) after the request is performed: 

{
    "StoppingInstances": [
        {
            "InstanceId": "i-06a56079863b82f8a",
    		"CurrentState": {
                "Code": 64,
                "Name": "stopping"
            },
            "PreviousState": {
                "Code": 16,
                "Name": "running"
            }
        }
    ]
}

03 Run start-instances command (OSX/Linux/UNIX) using the stopped instance ID as identifier to restart the instance. The following command example restarts an EC2 instance with the ID i- 06a56079863b82f8a within the US East region: 

aws ec2 start-instances
	--region us-east-1
	--instance-ids i-06a56079863b82f8a

04 The command output should return the EC2 instance current state metadata (highlighted): 

{
    "StartingInstances": [
        {
            "InstanceId": "i-06a56079863b82f8a",
            "CurrentState": {
                "Code": 0,
                "Name": "pending"
            },
            "PreviousState": {
                "Code": 80,
                "Name": "stopped"
            }
        }
    ]
}

05 Repeat steps no. 1 – 4 to restart any other scheduled EC2 instances available in the selected region.

06 Change the AWS region to repeat the entire process for other regions.

Actions for system-maintenance event:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to EC2 dashboard at https://console.aws.amazon.com/ec2/.

03 In the navigation panel, under INSTANCES section, choose IInstances.

04 Select the running EC2 instance scheduled for system-maintenance event (see the Audit section to identify the instance marked for maintenance).

05 Click on the Actions dropdown button from the dashboard top menu, select Instance State and click Stop. (!) IMPORTANT: This step will incur downtime for the application(s) running on the selected instance.

06 In the Stop Instances dialog box, review the instance details and click Yes, Stop to confirm the action. The instance status will change from running to stopping to stopped.

07 Click again on the Actions dropdown button, select Instance State and click Start to restart the instance.

08 Inside the Start Instances dialog box, review the details and click Yes, Start to confirm the action. The instance status will change from stopped to pending to running.

09 Repeat steps no. 4 – 8 to restart any other scheduled instances available in the current region.

10 Change the AWS region from the navigation bar and repeat the process for other regions.

Using AWS CLI

01 Run stop-instances command (OSX/Linux/UNIX) to stop the EC2 instance scheduled for system-maintenance event. (!) IMPORTANT: This step will incur downtime for the application(s) running on the selected instance: 

aws ec2 stop-instances
	--region us-east-1
	--instance-ids i-04c1691e53b1576b0

02 The command output should return the instance current state metadata (highlighted) once the AWS API request is performed: 

{
    "StoppingInstances": [
        {
            "InstanceId": "i-04c1691e53b1576b0",
 		   	"CurrentState": {
                "Code": 64,
                "Name": "stopping"
            },
            "PreviousState": {
                "Code": 16,
                "Name": "running"
            }
        }
    ]
}

03 Run start-instances command (OSX/Linux/UNIX) using the stopped instance ID as identifier to restart the EC2 instance: 

aws ec2 start-instances
	--region us-east-1
	--instance-ids i-06a56079863b82f8a

04 The command output should return the EC2 instance current state metadata (highlighted): 

aws ec2 start-instances
	--region us-east-1
	--instance-ids i-04c1691e53b1576b0

05 The command output should return the EC2 instance current state metadata (highlighted): 

{
    "StartingInstances": [
        {
            "InstanceId": "i-04c1691e53b1576b0",
	        "CurrentState": {
                "Code": 0,
                "Name": "pending"
            },
            "PreviousState": {
                "Code": 80,
                "Name": "stopped"
            }
        }
    ]
}

06 Repeat steps no. 1 – 4 to restart any other scheduled EC2 instances available in the selected region.

07 Change the AWS region to repeat the entire process for other regions.

References

Publication date Jun 23, 2016

Related EC2 rules