Skip to main content

Glue Development Endpoint CloudWatch Logs Encryption

Overview​

This check verifies that your AWS Glue development endpoints have CloudWatch Logs encryption enabled through an attached security configuration. When Glue jobs and development endpoints run, they can write logs to CloudWatch. Without encryption, these logs are stored in plain text, which may expose sensitive information.

Risk​

When Glue development endpoints do not have CloudWatch Logs encryption enabled:

  • Data exposure: Logs may contain credentials, connection strings, database queries, or data samples that could be visible to unauthorized users
  • Compliance violations: Many regulatory frameworks (HIPAA, PCI-DSS, SOC 2) require encryption of log data containing sensitive information
  • Lateral movement risk: Exposed credentials or connection details in logs could enable attackers to access additional systems
  • Audit trail gaps: Without KMS-backed encryption, you lose fine-grained access controls and auditability over who can read log data

Remediation Steps​

Prerequisites​

  • AWS account access with permissions to manage Glue security configurations and development endpoints
  • An existing KMS key, or permission to create one
Required IAM permissions

To create security configurations and development endpoints, you need:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "glue:CreateSecurityConfiguration",
        "glue:GetSecurityConfiguration",
        "glue:CreateDevEndpoint",
        "glue:GetDevEndpoint",
        "glue:DeleteDevEndpoint"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "kms:DescribeKey",
        "kms:CreateGrant"
      ],
      "Resource": "arn:aws:kms:us-east-1:*:key/*"
    }
  ]
}

AWS Console Method​

Step 1: Create a security configuration with CloudWatch Logs encryption​

  1. Open the AWS Glue Console at https://console.aws.amazon.com/glue
  2. Ensure you are in the us-east-1 region (top-right corner)
  3. In the left sidebar, click Security configurations (under "Security")
  4. Click Add security configuration
  5. Enter a Name (e.g., glue-cloudwatch-encrypted)
  6. In the Encryption configuration section:
    • Check CloudWatch logs
    • Select SSE-KMS for the encryption mode
    • Choose an existing KMS key from the dropdown, or enter a KMS key ARN
  7. Optionally enable S3 encryption and Job bookmarks encryption for additional security
  8. Click Create

Step 2: Create a new development endpoint with the security configuration​

Important: You cannot modify the security configuration of an existing development endpoint. You must create a new endpoint with the security configuration attached.

  1. In the Glue Console left sidebar, click Dev endpoints (under "ETL jobs")
  2. Click Add endpoint
  3. Enter an Endpoint name
  4. Select an IAM role with appropriate Glue permissions
  5. In the Security configuration section, select your newly created security configuration (e.g., glue-cloudwatch-encrypted)
  6. Configure other settings as needed (VPC, workers, etc.)
  7. Click Create endpoint
  8. Wait for the endpoint status to become READY (this may take several minutes)

Step 3: Migrate from existing unencrypted endpoints (if applicable)​

If you have existing development endpoints without encryption:

  1. Note the configuration of your existing endpoint (IAM role, VPC settings, libraries, etc.)
  2. Create a new endpoint with the same configuration plus the security configuration
  3. Update any notebooks or scripts to use the new endpoint
  4. Delete the old unencrypted endpoint
AWS CLI (optional)

Create a security configuration​

aws glue create-security-configuration \
  --name glue-cloudwatch-encrypted \
  --encryption-configuration '{
    "CloudWatchEncryption": {
      "CloudWatchEncryptionMode": "SSE-KMS",
      "KmsKeyArn": "arn:aws:kms:us-east-1:<YOUR_ACCOUNT_ID>:key/<YOUR_KEY_ID>"
    },
    "S3Encryption": [
      {
        "S3EncryptionMode": "SSE-KMS",
        "KmsKeyArn": "arn:aws:kms:us-east-1:<YOUR_ACCOUNT_ID>:key/<YOUR_KEY_ID>"
      }
    ],
    "JobBookmarksEncryption": {
      "JobBookmarksEncryptionMode": "CSE-KMS",
      "KmsKeyArn": "arn:aws:kms:us-east-1:<YOUR_ACCOUNT_ID>:key/<YOUR_KEY_ID>"
    }
  }' \
  --region us-east-1

Replace:

  • <YOUR_ACCOUNT_ID> with your 12-digit AWS account ID
  • <YOUR_KEY_ID> with your KMS key ID

Create a development endpoint with the security configuration​

aws glue create-dev-endpoint \
  --endpoint-name my-encrypted-endpoint \
  --role-arn arn:aws:iam::<YOUR_ACCOUNT_ID>:role/<YOUR_GLUE_ROLE> \
  --security-configuration glue-cloudwatch-encrypted \
  --worker-type G.1X \
  --number-of-workers 2 \
  --glue-version 3.0 \
  --region us-east-1

Replace:

  • <YOUR_ACCOUNT_ID> with your 12-digit AWS account ID
  • <YOUR_GLUE_ROLE> with your IAM role name for Glue

List existing development endpoints​

aws glue get-dev-endpoints \
  --region us-east-1 \
  --query 'DevEndpoints[*].{Name:EndpointName,SecurityConfig:SecurityConfiguration}'

Delete an unencrypted endpoint​

aws glue delete-dev-endpoint \
  --endpoint-name <ENDPOINT_NAME> \
  --region us-east-1
CloudFormation (optional)

This template creates a KMS key, a Glue security configuration with CloudWatch Logs encryption, and a development endpoint.

AWSTemplateFormatVersion: '2010-09-09'
Description: Glue Development Endpoint with CloudWatch Logs encryption

Parameters:
  EndpointName:
    Type: String
    Description: Name for the Glue development endpoint
    Default: my-encrypted-endpoint
  GlueRoleArn:
    Type: String
    Description: ARN of the IAM role for the Glue development endpoint

Resources:
  GlueKMSKey:
    Type: AWS::KMS::Key
    Properties:
      Description: KMS key for Glue encryption
      EnableKeyRotation: true
      KeyPolicy:
        Version: '2012-10-17'
        Statement:
          - Sid: Enable IAM User Permissions
            Effect: Allow
            Principal:
              AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
            Action: 'kms:*'
            Resource: '*'
          - Sid: Allow Glue Service
            Effect: Allow
            Principal:
              Service: glue.amazonaws.com
            Action:
              - kms:Encrypt*
              - kms:Decrypt*
              - kms:ReEncrypt*
              - kms:GenerateDataKey*
              - kms:Describe*
            Resource: '*'
          - Sid: Allow CloudWatch Logs
            Effect: Allow
            Principal:
              Service: !Sub 'logs.${AWS::Region}.amazonaws.com'
            Action:
              - kms:Encrypt*
              - kms:Decrypt*
              - kms:ReEncrypt*
              - kms:GenerateDataKey*
              - kms:Describe*
            Resource: '*'

  GlueKMSKeyAlias:
    Type: AWS::KMS::Alias
    Properties:
      AliasName: alias/glue-encryption-key
      TargetKeyId: !Ref GlueKMSKey

  GlueSecurityConfiguration:
    Type: AWS::Glue::SecurityConfiguration
    Properties:
      Name: glue-cloudwatch-encrypted
      EncryptionConfiguration:
        CloudWatchEncryption:
          CloudWatchEncryptionMode: SSE-KMS
          KmsKeyArn: !GetAtt GlueKMSKey.Arn
        S3Encryptions:
          - S3EncryptionMode: SSE-KMS
            KmsKeyArn: !GetAtt GlueKMSKey.Arn
        JobBookmarksEncryption:
          JobBookmarksEncryptionMode: CSE-KMS
          KmsKeyArn: !GetAtt GlueKMSKey.Arn

  GlueDevEndpoint:
    Type: AWS::Glue::DevEndpoint
    Properties:
      EndpointName: !Ref EndpointName
      RoleArn: !Ref GlueRoleArn
      SecurityConfiguration: !Ref GlueSecurityConfiguration
      GlueVersion: '3.0'
      WorkerType: G.1X
      NumberOfWorkers: 2

Outputs:
  SecurityConfigurationName:
    Description: Name of the Glue security configuration
    Value: !Ref GlueSecurityConfiguration
  DevEndpointName:
    Description: Name of the Glue development endpoint
    Value: !Ref GlueDevEndpoint
  KMSKeyArn:
    Description: ARN of the KMS key
    Value: !GetAtt GlueKMSKey.Arn

Deploy with:

aws cloudformation deploy \
  --template-file template.yaml \
  --stack-name glue-encrypted-endpoint \
  --parameter-overrides \
    EndpointName=my-encrypted-endpoint \
    GlueRoleArn=arn:aws:iam::123456789012:role/MyGlueRole \
  --region us-east-1
Terraform (optional)
variable "endpoint_name" {
  description = "Name for the Glue development endpoint"
  type        = string
  default     = "my-encrypted-endpoint"
}

variable "glue_role_arn" {
  description = "ARN of the IAM role for the Glue development endpoint"
  type        = string
}

data "aws_caller_identity" "current" {}
data "aws_region" "current" {}

resource "aws_kms_key" "glue" {
  description             = "KMS key for Glue encryption"
  deletion_window_in_days = 30
  enable_key_rotation     = true

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid       = "Enable IAM User Permissions"
        Effect    = "Allow"
        Principal = { AWS = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:root" }
        Action    = "kms:*"
        Resource  = "*"
      },
      {
        Sid       = "Allow Glue Service"
        Effect    = "Allow"
        Principal = { Service = "glue.amazonaws.com" }
        Action = [
          "kms:Encrypt*",
          "kms:Decrypt*",
          "kms:ReEncrypt*",
          "kms:GenerateDataKey*",
          "kms:Describe*"
        ]
        Resource = "*"
      },
      {
        Sid       = "Allow CloudWatch Logs"
        Effect    = "Allow"
        Principal = { Service = "logs.${data.aws_region.current.name}.amazonaws.com" }
        Action = [
          "kms:Encrypt*",
          "kms:Decrypt*",
          "kms:ReEncrypt*",
          "kms:GenerateDataKey*",
          "kms:Describe*"
        ]
        Resource = "*"
      }
    ]
  })
}

resource "aws_kms_alias" "glue" {
  name          = "alias/glue-encryption-key"
  target_key_id = aws_kms_key.glue.key_id
}

resource "aws_glue_security_configuration" "encrypted" {
  name = "glue-cloudwatch-encrypted"

  encryption_configuration {
    cloudwatch_encryption {
      cloudwatch_encryption_mode = "SSE-KMS"
      kms_key_arn                = aws_kms_key.glue.arn
    }

    s3_encryption {
      s3_encryption_mode = "SSE-KMS"
      kms_key_arn        = aws_kms_key.glue.arn
    }

    job_bookmarks_encryption {
      job_bookmarks_encryption_mode = "CSE-KMS"
      kms_key_arn                   = aws_kms_key.glue.arn
    }
  }
}

resource "aws_glue_dev_endpoint" "encrypted" {
  name                   = var.endpoint_name
  role_arn               = var.glue_role_arn
  security_configuration = aws_glue_security_configuration.encrypted.name
  glue_version           = "3.0"
  worker_type            = "G.1X"
  number_of_workers      = 2
}

output "security_configuration_name" {
  description = "Name of the Glue security configuration"
  value       = aws_glue_security_configuration.encrypted.name
}

output "dev_endpoint_name" {
  description = "Name of the Glue development endpoint"
  value       = aws_glue_dev_endpoint.encrypted.name
}

output "kms_key_arn" {
  description = "ARN of the KMS key"
  value       = aws_kms_key.glue.arn
}

Deploy with:

terraform init
terraform plan -var="glue_role_arn=arn:aws:iam::123456789012:role/MyGlueRole"
terraform apply -var="glue_role_arn=arn:aws:iam::123456789012:role/MyGlueRole"

Verification​

After creating your security configuration and development endpoint, verify that CloudWatch Logs encryption is enabled:

  1. Open the AWS Glue Console at https://console.aws.amazon.com/glue
  2. Click Security configurations in the left sidebar
  3. Click on your security configuration name
  4. Confirm that CloudWatch logs encryption shows SSE-KMS and displays your KMS key ARN
  5. Navigate to Dev endpoints and click on your endpoint name
  6. Verify that the Security configuration field shows your encrypted configuration name
Verify with AWS CLI

Check security configuration settings​

aws glue get-security-configuration \
  --name glue-cloudwatch-encrypted \
  --region us-east-1 \
  --query 'SecurityConfiguration.EncryptionConfiguration.CloudWatchEncryption'

Expected output:

{
  "CloudWatchEncryptionMode": "SSE-KMS",
  "KmsKeyArn": "arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012"
}

Check development endpoint security configuration​

aws glue get-dev-endpoint \
  --endpoint-name my-encrypted-endpoint \
  --region us-east-1 \
  --query 'DevEndpoint.{Name:EndpointName,SecurityConfig:SecurityConfiguration}'

Expected output:

{
  "Name": "my-encrypted-endpoint",
  "SecurityConfig": "glue-cloudwatch-encrypted"
}

If SecurityConfig is null or missing, the endpoint does not have encryption enabled.

Additional Resources​

Notes​

  • Cannot modify existing endpoints: Security configurations cannot be changed on existing development endpoints. You must create a new endpoint with the desired security configuration and migrate your workloads.
  • Development endpoints are deprecated: AWS recommends using AWS Glue interactive sessions instead of development endpoints for new projects. Interactive sessions offer similar functionality with better cost efficiency.
  • KMS key permissions: Ensure your KMS key policy allows the Glue service and CloudWatch Logs service to use the key for encryption operations.
  • Cost considerations: Using customer-managed KMS keys incurs additional charges for key storage and API calls.
  • Key rotation: Enable automatic key rotation on your KMS key for enhanced security.
  • Multiple encryption types: Consider enabling S3 encryption and Job bookmarks encryption in addition to CloudWatch Logs encryption for comprehensive data protection.