Glue ETL Job CloudWatch Logs Encryption

Overview

This check verifies that your AWS Glue ETL jobs have CloudWatch Logs encryption enabled through a security configuration. When Glue jobs run, they write logs to CloudWatch. Without encryption, these logs are stored with AWS-managed encryption only, giving you less control over access to potentially sensitive data.

Risk

When Glue ETL job logs are not encrypted with a customer-managed KMS key:

• Sensitive data exposure: Logs can contain credentials, connection strings, database schema details, and PII from your data pipelines
• Lateral movement risk: Attackers with log storage access can harvest secrets to move deeper into your environment
• Reduced access control: You cannot define custom key policies to restrict who can decrypt your log data
• Compliance gaps: Many regulatory frameworks require customer-managed encryption for audit trails and sensitive data

Remediation Steps

Prerequisites

• AWS account access with permissions to manage Glue security configurations and KMS keys
• An existing KMS key, or permission to create one

Required IAM permissions

To create and manage Glue security configurations, you need:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "glue:CreateSecurityConfiguration",
        "glue:GetSecurityConfiguration",
        "glue:GetSecurityConfigurations",
        "glue:UpdateJob"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "kms:DescribeKey",
        "kms:CreateGrant",
        "kms:GenerateDataKey",
        "kms:Decrypt",
        "kms:Encrypt"
      ],
      "Resource": "arn:aws:kms:us-east-1:*:key/*"
    }
  ]
}

AWS Console Method

Step 1: Create or identify a KMS key

1. Open the AWS KMS Console at https://console.aws.amazon.com/kms
2. Ensure you are in the us-east-1 region (top-right corner)
3. Click Customer managed keys in the left sidebar
4. Either select an existing symmetric key, or click Create key:
   • Key type: Symmetric
   • Key usage: Encrypt and decrypt
   • Give it a name like glue-logs-encryption-key
5. Copy the Key ARN for later use

KMS key policy for Glue and CloudWatch Logs

Your KMS key policy must allow both Glue and CloudWatch Logs to use the key. Add these statements to your key policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Enable IAM User Permissions",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<YOUR_ACCOUNT_ID>:root"
      },
      "Action": "kms:*",
      "Resource": "*"
    },
    {
      "Sid": "Allow Glue Service",
      "Effect": "Allow",
      "Principal": {
        "Service": "glue.amazonaws.com"
      },
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:GenerateDataKey"
      ],
      "Resource": "*"
    },
    {
      "Sid": "Allow CloudWatch Logs",
      "Effect": "Allow",
      "Principal": {
        "Service": "logs.us-east-1.amazonaws.com"
      },
      "Action": [
        "kms:Encrypt*",
        "kms:Decrypt*",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:Describe*"
      ],
      "Resource": "*"
    }
  ]
}

Replace <YOUR_ACCOUNT_ID> with your 12-digit AWS account ID.

Step 2: Create a Glue security configuration

1. Open the AWS Glue Console at https://console.aws.amazon.com/glue
2. In the left sidebar, click Security configurations (under "Security")
3. Click Add security configuration
4. Enter a Name (e.g., glue-logs-encrypted)
5. In the Encryption settings section:
   • Check Enable CloudWatch logs encryption
   • For AWS KMS key, select your key from the dropdown or enter the Key ARN
6. Optionally enable S3 encryption and job bookmarks encryption for comprehensive protection
7. Click Create

Step 3: Attach the security configuration to your Glue job

1. In the Glue Console, click ETL jobs in the left sidebar
2. Select the job you want to update
3. Click Actions then Edit job
4. Scroll to Advanced properties and expand it
5. Under Security configuration, select the security configuration you created
6. Click Save

Note: Existing job runs are not affected. The new security configuration applies to future job runs.

AWS CLI (optional)

Create a security configuration with CloudWatch Logs encryption

aws glue create-security-configuration \
  --name glue-logs-encrypted \
  --encryption-configuration '{
    "CloudWatchEncryption": {
      "CloudWatchEncryptionMode": "SSE-KMS",
      "KmsKeyArn": "arn:aws:kms:us-east-1:<YOUR_ACCOUNT_ID>:key/<YOUR_KEY_ID>"
    },
    "S3Encryption": [
      {
        "S3EncryptionMode": "DISABLED"
      }
    ],
    "JobBookmarksEncryption": {
      "JobBookmarksEncryptionMode": "DISABLED"
    }
  }' \
  --region us-east-1

Replace:

• <YOUR_ACCOUNT_ID> with your 12-digit AWS account ID
• <YOUR_KEY_ID> with your KMS key ID

Update an existing job to use the security configuration

aws glue update-job \
  --job-name <YOUR_JOB_NAME> \
  --job-update '{
    "SecurityConfiguration": "glue-logs-encrypted"
  }' \
  --region us-east-1

View security configuration details

aws glue get-security-configuration \
  --name glue-logs-encrypted \
  --region us-east-1

CloudFormation (optional)

This template creates a KMS key, Glue security configuration, and a sample Glue job with encryption enabled.

AWSTemplateFormatVersion: '2010-09-09'
Description: Glue ETL job with CloudWatch Logs encryption enabled

Parameters:
  JobName:
    Type: String
    Description: Name of the Glue ETL job
    Default: my-encrypted-glue-job
  ScriptLocation:
    Type: String
    Description: S3 path to the Glue job script
    Default: s3://my-bucket/scripts/my-job.py
  GlueRoleArn:
    Type: String
    Description: ARN of the IAM role for the Glue job

Resources:
  GlueEncryptionKey:
    Type: AWS::KMS::Key
    Properties:
      Description: KMS key for Glue CloudWatch Logs encryption
      EnableKeyRotation: true
      KeyPolicy:
        Version: '2012-10-17'
        Statement:
          - Sid: Enable IAM User Permissions
            Effect: Allow
            Principal:
              AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
            Action: 'kms:*'
            Resource: '*'
          - Sid: Allow Glue Service
            Effect: Allow
            Principal:
              Service: glue.amazonaws.com
            Action:
              - kms:Encrypt
              - kms:Decrypt
              - kms:GenerateDataKey
            Resource: '*'
          - Sid: Allow CloudWatch Logs
            Effect: Allow
            Principal:
              Service: !Sub 'logs.${AWS::Region}.amazonaws.com'
            Action:
              - kms:Encrypt*
              - kms:Decrypt*
              - kms:ReEncrypt*
              - kms:GenerateDataKey*
              - kms:Describe*
            Resource: '*'

  GlueEncryptionKeyAlias:
    Type: AWS::KMS::Alias
    Properties:
      AliasName: alias/glue-logs-encryption-key
      TargetKeyId: !Ref GlueEncryptionKey

  GlueSecurityConfiguration:
    Type: AWS::Glue::SecurityConfiguration
    Properties:
      Name: glue-logs-encrypted
      EncryptionConfiguration:
        CloudWatchEncryption:
          CloudWatchEncryptionMode: SSE-KMS
          KmsKeyArn: !GetAtt GlueEncryptionKey.Arn
        S3Encryptions:
          - S3EncryptionMode: DISABLED
        JobBookmarksEncryption:
          JobBookmarksEncryptionMode: DISABLED

  GlueJob:
    Type: AWS::Glue::Job
    Properties:
      Name: !Ref JobName
      Role: !Ref GlueRoleArn
      Command:
        Name: glueetl
        ScriptLocation: !Ref ScriptLocation
        PythonVersion: '3'
      SecurityConfiguration: !Ref GlueSecurityConfiguration
      GlueVersion: '4.0'
      DefaultArguments:
        '--enable-continuous-cloudwatch-log': 'true'
        '--enable-metrics': 'true'

Outputs:
  SecurityConfigurationName:
    Description: Name of the Glue security configuration
    Value: !Ref GlueSecurityConfiguration
  KMSKeyArn:
    Description: ARN of the KMS key used for encryption
    Value: !GetAtt GlueEncryptionKey.Arn
  JobName:
    Description: Name of the Glue job
    Value: !Ref GlueJob

Deploy with:

aws cloudformation deploy \
  --template-file template.yaml \
  --stack-name glue-encrypted-job \
  --parameter-overrides \
    JobName=my-encrypted-job \
    ScriptLocation=s3://my-bucket/scripts/job.py \
    GlueRoleArn=arn:aws:iam::123456789012:role/GlueServiceRole \
  --region us-east-1

Terraform (optional)

variable "job_name" {
  description = "Name of the Glue ETL job"
  type        = string
  default     = "my-encrypted-glue-job"
}

variable "script_location" {
  description = "S3 path to the Glue job script"
  type        = string
}

variable "glue_role_arn" {
  description = "ARN of the IAM role for the Glue job"
  type        = string
}

data "aws_caller_identity" "current" {}
data "aws_region" "current" {}

resource "aws_kms_key" "glue_logs" {
  description             = "KMS key for Glue CloudWatch Logs encryption"
  deletion_window_in_days = 30
  enable_key_rotation     = true

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid       = "Enable IAM User Permissions"
        Effect    = "Allow"
        Principal = { AWS = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:root" }
        Action    = "kms:*"
        Resource  = "*"
      },
      {
        Sid       = "Allow Glue Service"
        Effect    = "Allow"
        Principal = { Service = "glue.amazonaws.com" }
        Action = [
          "kms:Encrypt",
          "kms:Decrypt",
          "kms:GenerateDataKey"
        ]
        Resource = "*"
      },
      {
        Sid       = "Allow CloudWatch Logs"
        Effect    = "Allow"
        Principal = { Service = "logs.${data.aws_region.current.name}.amazonaws.com" }
        Action = [
          "kms:Encrypt*",
          "kms:Decrypt*",
          "kms:ReEncrypt*",
          "kms:GenerateDataKey*",
          "kms:Describe*"
        ]
        Resource = "*"
      }
    ]
  })
}

resource "aws_kms_alias" "glue_logs" {
  name          = "alias/glue-logs-encryption-key"
  target_key_id = aws_kms_key.glue_logs.key_id
}

resource "aws_glue_security_configuration" "encrypted" {
  name = "glue-logs-encrypted"

  encryption_configuration {
    cloudwatch_encryption {
      cloudwatch_encryption_mode = "SSE-KMS"
      kms_key_arn                = aws_kms_key.glue_logs.arn
    }

    job_bookmarks_encryption {
      job_bookmarks_encryption_mode = "DISABLED"
    }

    s3_encryption {
      s3_encryption_mode = "DISABLED"
    }
  }
}

resource "aws_glue_job" "encrypted" {
  name                   = var.job_name
  role_arn               = var.glue_role_arn
  security_configuration = aws_glue_security_configuration.encrypted.name
  glue_version           = "4.0"

  command {
    name            = "glueetl"
    script_location = var.script_location
    python_version  = "3"
  }

  default_arguments = {
    "--enable-continuous-cloudwatch-log" = "true"
    "--enable-metrics"                   = "true"
  }
}

output "security_configuration_name" {
  description = "Name of the Glue security configuration"
  value       = aws_glue_security_configuration.encrypted.name
}

output "kms_key_arn" {
  description = "ARN of the KMS key used for encryption"
  value       = aws_kms_key.glue_logs.arn
}

output "job_name" {
  description = "Name of the Glue job"
  value       = aws_glue_job.encrypted.name
}

Deploy with:

terraform init
terraform plan -var="script_location=s3://my-bucket/scripts/job.py" -var="glue_role_arn=arn:aws:iam::123456789012:role/GlueServiceRole"
terraform apply

Verification

After making changes, verify encryption is enabled:

1. Open the AWS Glue Console at https://console.aws.amazon.com/glue
2. Click ETL jobs in the left sidebar
3. Select your job and view the Job details tab
4. Scroll to Advanced properties and confirm a Security configuration is attached
5. Click on the security configuration name to verify CloudWatch logs encryption shows "SSE-KMS" with your KMS key

Verify with AWS CLI

Check if a job has a security configuration

aws glue get-job \
  --job-name <YOUR_JOB_NAME> \
  --region us-east-1 \
  --query 'Job.{Name:Name,SecurityConfiguration:SecurityConfiguration}'

A compliant job will show the security configuration name:

{
  "Name": "my-glue-job",
  "SecurityConfiguration": "glue-logs-encrypted"
}

Verify the security configuration has CloudWatch encryption enabled

aws glue get-security-configuration \
  --name glue-logs-encrypted \
  --region us-east-1 \
  --query 'SecurityConfiguration.EncryptionConfiguration.CloudWatchEncryption'

Expected output:

{
  "CloudWatchEncryptionMode": "SSE-KMS",
  "KmsKeyArn": "arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012"
}

If CloudWatchEncryptionMode is DISABLED or the job has no security configuration, it is non-compliant.

Additional Resources

• AWS Glue Security Configurations
• AWS CLI Reference: create-security-configuration
• Terraform: aws_glue_security_configuration
• AWS KMS Developer Guide

Notes

• Existing job runs: The security configuration only applies to future job runs. Logs from previous runs remain with their original encryption settings.
• Symmetric keys only: AWS Glue supports only symmetric customer master keys (CMKs), not asymmetric keys.
• IAM role permissions: The Glue job's IAM role must have kms:GenerateDataKey, kms:Decrypt, and kms:Encrypt permissions on the KMS key.
• Key availability: If your KMS key is deleted or disabled, Glue jobs will fail to write logs. Ensure key policies allow necessary access.
• Comprehensive encryption: Consider also enabling S3 encryption and job bookmarks encryption in the security configuration for full protection.
• Cost considerations: Using customer-managed KMS keys incurs additional charges for key storage and API calls.