Open menu

Enable S3 Encryption for AWS Glue

Cloud Conformity allows you to automate the auditing process of this resolution page. Register for a 14 day evaluation and check your compliance level for free!

Start a Free Trial Product features
Security

Risk level: Medium (should be achieved)

Ensure that encryption at rest is enabled within your Amazon Glue security configurations to meet regulatory requirements and prevent unauthorized users from getting access to the data written to Amazon S3. A security configuration is a set of security properties that can be used by AWS Glue to configure encryption for processes and resources associated with the security configuration such as jobs, crawlers and development endpoints.

Enable encryption when writing AWS Glue data to Amazon S3 in order to meet security and compliance requirements within your organization. With S3 encryption enabled, when you run crawlers, execute ETL jobs or start development endpoints, AWS Key Management Service (KMS) keys are used to encrypt your data at rest.

Audit

To determine if your Amazon Glue security configurations have S3 encryption mode enabled, perform the following actions:

Using AWS Console

01 Sign in to AWS Management Console.

02 Navigate to Glue service dashboard at https://console.aws.amazon.com/glue/.

03 In the navigation panel, under Security, choose Security configurations.

04 Select the security configuration that you want to examine, then click on the its name to access the resource details page.

05 On the security configuration details page, check the S3 encryption mode feature status. If the status is set to DISABLED, encryption at rest is not enabled when writing AWS Glue data to Amazon S3, therefore the selected security configuration is not compliant.

06 Repeat step no. 4 and 5 to verify other Amazon Glue security configurations available in the current region.

07 Change the AWS region from the console navigation bar and repeat the audit process for other regions.

Using AWS CLI

01 Run get-security-configurations command (OSX/Linux/UNIX) to list the names of all AWS Glue security configurations created within the selected AWS region:

aws glue get-security-configurations
	--region us-east-1
	--output table
	--query 'SecurityConfigurations[*].Name'

02 The command output should return a table with AWS Glue security configuration names:

-------------------------------
|  GetSecurityConfigurations  |
+-----------------------------+
|  cc-etl-project-sec-config  |
|  cc-glue-security-config    |
+-----------------------------+

03 Execute get-security-configuration command (OSX/Linux/UNIX) using the name of the Glue security configuration that you want to examine as identifier and custom query filters to determine the S3 encryption mode status:

aws glue get-security-configuration
	--name cc-etl-project-sec-config
	--region us-east-1
	--query 'SecurityConfiguration.EncryptionConfiguration.S3Encryption[*].S3EncryptionMode'

04 The command output should return the request information (i.e. S3 encryption mode status):

"DISABLED"

If get-security-configuration command output returns "DISABLED", as shown in the example above, encryption at rest is not enabled when writing Amazon Glue data to S3, therefore the selected AWS Glue security configuration is not compliant.

05 Repeat step no. 3 and 4 to check other Amazon Glue security configurations available in the selected region.

06 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 – 5 to perform the entire audit process for other regions.

Remediation / Resolution

To enable encryption when writing AWS Glue data to Amazon S3, you must to re-create the security configurations associated with your ETL jobs, crawlers and development endpoints, with the S3 encryption mode enabled. To create and configure a new Amazon Glue security configuration, perform the following:

Using AWS Console

01 Sign in to AWS Management Console.

02 Navigate to Glue service dashboard at https://console.aws.amazon.com/glue/.

03 In the left navigation panel, under Security, choose Security configurations.

04 Click Add security configuration to initiate the security configuration setup process.

05 On Add security configuration page, perform the following actions:

  1. Provide a unique name for your new configuration in the Security configuration name box.
  2. Select S3 encryption checkbox to enable at-rest encryption when writing data to Amazon S3, then choose the ARN of the AWS KMS key that you want to use for encryption, from AWS KMS key dropdown list.
  3. Click Finish to create your new AWS Glue security configuration.

06 Reconfigure any existing Amazon Glue ETL jobs, crawlers, and development endpoints to make use of the new security configuration created at the previous step.

07 Change the AWS region from the navigation bar and repeat the process for other regions.

Using AWS CLI

01 Define the required parameters for your new Amazon Glue security configuration and save the document to a JSON file named sec-config-s3-encrypted.json. Replace "KmsKeyArn" parameter value with the ARN of your own AWS KMS key:

{
  "S3Encryption": [
    {
      "S3EncryptionMode": "SSE-KMS",
      "KmsKeyArn": "arn:aws:kms:us-east-1:123456789012:key/aaaabbbb-aaaa-bbbb-cccc-aaaabbbbcccc"
    }
  ],
  "CloudWatchEncryption": {
    "CloudWatchEncryptionMode": "DISABLED"
  },
  "JobBookmarksEncryption": {
    "JobBookmarksEncryptionMode": "DISABLED"
  }
}

02 Run create-security-configuration command (OSX/Linux/UNIX) using the sec-config-s3-encrypted.json file defined at the previous step as command parameter to create a new Amazon Glue security configuration that has S3 encryption mode enabled:

aws glue create-security-configuration
	--region us-east-1
	--name cc-s3-encrypted-sec-config
	--encryption-configuration file://sec-config-s3-encrypted.json

03 The command output should return the command request metadata:

{
    "CreatedTimestamp": 1548347561.501,
    "Name": "cc-s3-encrypted-sec-config"
}

04 Reconfigure your existing Amazon Glue jobs, crawlers and development endpoints to make use of the new security configuration created at the previous steps.

05 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 – 4 to perform the remediation/resolution process for other regions.

References

Publication date Jan 17, 2019