Ensure Kafka Cluster Encryption at Rest Uses Customer Managed Keys (CMK)

Overview

This check verifies that Amazon MSK (Managed Streaming for Apache Kafka) clusters use customer-managed KMS keys (CMK) for encrypting data at rest, rather than AWS-managed keys.

When you create an MSK cluster, AWS encrypts your data at rest by default. However, using a customer-managed key gives you greater control over the encryption process.

Risk

Using AWS-managed encryption keys (instead of customer-managed keys) limits your control over:

• Key rotation: You cannot control when or how often the key is rotated
• Access policies: You cannot define granular policies for who can use the key
• Audit visibility: You lose detailed CloudTrail logging of key usage
• Incident response: You cannot quickly disable a key to "crypto-shred" data during a security incident

For organizations with compliance requirements (C5, ISO27001, KISA-ISMS-P), customer-managed keys may be required.

Remediation Steps

Prerequisites

• AWS account access with permissions to manage MSK clusters and KMS keys
• An existing customer-managed KMS key, or permissions to create one

Important: MSK encryption settings cannot be changed after cluster creation. You must create a new cluster with the correct encryption settings and migrate your data.

AWS Console Method

Step 1: Create a Customer Managed Key (if needed)

1. Go to the AWS KMS Console: https://console.aws.amazon.com/kms
2. Make sure you are in the us-east-1 region (or your target region)
3. Click Create key
4. Choose Symmetric key type and Encrypt and decrypt usage
5. Click Next
6. Enter an alias (e.g., msk-encryption-key) and optional description
7. Click Next
8. Select key administrators who can manage the key
9. Click Next
10. Select users and roles that can use the key (include the role/user that will create MSK clusters)
11. Click Next and then Finish
12. Copy the key ARN for later use

Step 2: Create a New MSK Cluster with CMK Encryption

1. Go to the Amazon MSK Console: https://console.aws.amazon.com/msk
2. Click Create cluster
3. Choose Custom create for full configuration control
4. Enter a cluster name
5. Select your Apache Kafka version
6. Choose broker instance type and number of brokers
7. Configure networking (VPC, subnets, security groups)
8. In the Encryption section:
   • For Encryption at rest, select Use a customer-managed key
   • Choose your CMK from the dropdown (or paste the key ARN)
   • Do not select the alias/aws/kafka key (this is AWS-managed)
9. Configure other settings as needed
10. Click Create cluster

Step 3: Migrate from Existing Cluster (if applicable)

If you have an existing cluster using AWS-managed encryption:

1. Set up Apache Kafka MirrorMaker or use MSK Replicator to copy data
2. Update your applications to point to the new cluster
3. Verify data integrity
4. Delete the old cluster once migration is complete

AWS CLI (optional)

Create a KMS Key

# Create a customer-managed KMS key
aws kms create-key \
    --description "KMS key for MSK cluster encryption" \
    --region us-east-1

# Create an alias for easier reference
aws kms create-alias \
    --alias-name alias/msk-encryption-key \
    --target-key-id <key-id-from-previous-command> \
    --region us-east-1

Create MSK Cluster with CMK

# Create the MSK cluster with CMK encryption
aws kafka create-cluster \
    --cluster-name "my-secure-cluster" \
    --kafka-version "3.5.1" \
    --number-of-broker-nodes 3 \
    --broker-node-group-info '{
        "InstanceType": "kafka.m5.large",
        "ClientSubnets": [
            "subnet-xxxxxxxxx1",
            "subnet-xxxxxxxxx2",
            "subnet-xxxxxxxxx3"
        ],
        "SecurityGroups": ["sg-xxxxxxxxx"],
        "StorageInfo": {
            "EbsStorageInfo": {
                "VolumeSize": 100
            }
        }
    }' \
    --encryption-info '{
        "EncryptionAtRest": {
            "DataVolumeKMSKeyId": "arn:aws:kms:us-east-1:123456789012:key/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
        },
        "EncryptionInTransit": {
            "ClientBroker": "TLS",
            "InCluster": true
        }
    }' \
    --region us-east-1

Replace the placeholder values:

• subnet-xxxxxxxxx1, subnet-xxxxxxxxx2, subnet-xxxxxxxxx3: Your VPC subnet IDs
• sg-xxxxxxxxx: Your security group ID
• arn:aws:kms:...: Your customer-managed KMS key ARN

List Existing Clusters

# List all MSK clusters
aws kafka list-clusters --region us-east-1

Check Cluster Encryption Settings

# Get cluster details including encryption configuration
aws kafka describe-cluster \
    --cluster-arn "arn:aws:kafka:us-east-1:123456789012:cluster/my-cluster/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" \
    --query 'ClusterInfo.EncryptionInfo' \
    --region us-east-1

CloudFormation (optional)

AWSTemplateFormatVersion: '2010-09-09'
Description: MSK Cluster with Customer Managed Key Encryption

Parameters:
  ClusterName:
    Type: String
    Description: Name of the MSK cluster
  KMSKeyArn:
    Type: String
    Description: ARN of the Customer Managed KMS key
  KafkaVersion:
    Type: String
    Default: '3.5.1'
    Description: Apache Kafka version
  InstanceType:
    Type: String
    Default: kafka.m5.large
    Description: Instance type for broker nodes
  NumberOfBrokerNodes:
    Type: Number
    Default: 3
    Description: Number of broker nodes
  Subnet1:
    Type: AWS::EC2::Subnet::Id
    Description: First subnet for MSK brokers
  Subnet2:
    Type: AWS::EC2::Subnet::Id
    Description: Second subnet for MSK brokers
  Subnet3:
    Type: AWS::EC2::Subnet::Id
    Description: Third subnet for MSK brokers
  SecurityGroupId:
    Type: AWS::EC2::SecurityGroup::Id
    Description: Security group for MSK brokers

Resources:
  MSKCluster:
    Type: AWS::MSK::Cluster
    Properties:
      ClusterName: !Ref ClusterName
      KafkaVersion: !Ref KafkaVersion
      NumberOfBrokerNodes: !Ref NumberOfBrokerNodes
      BrokerNodeGroupInfo:
        InstanceType: !Ref InstanceType
        ClientSubnets:
          - !Ref Subnet1
          - !Ref Subnet2
          - !Ref Subnet3
        SecurityGroups:
          - !Ref SecurityGroupId
        StorageInfo:
          EBSStorageInfo:
            VolumeSize: 100
      EncryptionInfo:
        EncryptionAtRest:
          DataVolumeKMSKeyId: !Ref KMSKeyArn
        EncryptionInTransit:
          ClientBroker: TLS
          InCluster: true

Outputs:
  ClusterArn:
    Description: ARN of the MSK cluster
    Value: !Ref MSKCluster

Deploy the stack:

aws cloudformation create-stack \
    --stack-name msk-secure-cluster \
    --template-body file://msk-cluster.yaml \
    --parameters \
        ParameterKey=ClusterName,ParameterValue=my-secure-cluster \
        ParameterKey=KMSKeyArn,ParameterValue=arn:aws:kms:us-east-1:123456789012:key/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx \
        ParameterKey=Subnet1,ParameterValue=subnet-xxxxxxxxx1 \
        ParameterKey=Subnet2,ParameterValue=subnet-xxxxxxxxx2 \
        ParameterKey=Subnet3,ParameterValue=subnet-xxxxxxxxx3 \
        ParameterKey=SecurityGroupId,ParameterValue=sg-xxxxxxxxx \
    --region us-east-1

Terraform (optional)

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

variable "cluster_name" {
  description = "Name of the MSK cluster"
  type        = string
}

variable "kms_key_arn" {
  description = "ARN of the Customer Managed KMS key for encryption at rest"
  type        = string
}

variable "kafka_version" {
  description = "Apache Kafka version"
  type        = string
  default     = "3.5.1"
}

variable "instance_type" {
  description = "Instance type for broker nodes"
  type        = string
  default     = "kafka.m5.large"
}

variable "number_of_broker_nodes" {
  description = "Number of broker nodes (must be multiple of AZs)"
  type        = number
  default     = 3
}

variable "subnet_ids" {
  description = "List of subnet IDs for broker nodes"
  type        = list(string)
}

variable "security_group_ids" {
  description = "List of security group IDs for broker nodes"
  type        = list(string)
}

resource "aws_msk_cluster" "main" {
  cluster_name           = var.cluster_name
  kafka_version          = var.kafka_version
  number_of_broker_nodes = var.number_of_broker_nodes

  broker_node_group_info {
    instance_type   = var.instance_type
    client_subnets  = var.subnet_ids
    security_groups = var.security_group_ids

    storage_info {
      ebs_storage_info {
        volume_size = 100
      }
    }
  }

  encryption_info {
    encryption_at_rest_kms_key_arn = var.kms_key_arn

    encryption_in_transit {
      client_broker = "TLS"
      in_cluster    = true
    }
  }

  tags = {
    Environment = "production"
  }
}

output "cluster_arn" {
  description = "ARN of the MSK cluster"
  value       = aws_msk_cluster.main.arn
}

output "bootstrap_brokers_tls" {
  description = "TLS connection host:port pairs"
  value       = aws_msk_cluster.main.bootstrap_brokers_tls
}

Example terraform.tfvars:

cluster_name           = "my-secure-cluster"
kms_key_arn            = "arn:aws:kms:us-east-1:123456789012:key/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
subnet_ids             = ["subnet-xxxxxxxxx1", "subnet-xxxxxxxxx2", "subnet-xxxxxxxxx3"]
security_group_ids     = ["sg-xxxxxxxxx"]

Deploy:

terraform init
terraform plan
terraform apply

Verification

After creating your cluster, verify it uses a customer-managed key:

1. Go to the Amazon MSK Console
2. Click on your cluster name
3. In the Security section, check Encryption at rest
4. Confirm it shows your customer-managed key ARN (not alias/aws/kafka)

CLI verification

# Verify encryption configuration
aws kafka describe-cluster \
    --cluster-arn "arn:aws:kafka:us-east-1:123456789012:cluster/my-secure-cluster/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" \
    --query 'ClusterInfo.EncryptionInfo.EncryptionAtRest.DataVolumeKMSKeyId' \
    --output text \
    --region us-east-1

The output should show your customer-managed KMS key ARN, not the AWS-managed key.

Run Prowler to verify:

prowler aws --checks kafka_cluster_encryption_at_rest_uses_cmk

Additional Resources

• Amazon MSK encryption documentation
• AWS KMS key management
• MSK best practices
• Prowler MSK checks

Notes

• Encryption settings are immutable: You cannot change the encryption key after cluster creation. Plan accordingly.
• MSK Serverless: Serverless clusters automatically pass this check as they use AWS-managed encryption appropriately for the serverless model.
• Key policy requirements: Ensure your KMS key policy grants MSK the necessary permissions (kms:Encrypt, kms:Decrypt, kms:GenerateDataKey*, etc.)
• Cross-account keys: If using a KMS key from another account, additional key policy configuration is required.
• Key rotation: Enable automatic key rotation on your CMK for enhanced security (KMS handles this transparently).
• Cost consideration: Customer-managed KMS keys incur additional charges ($1/month per key plus API usage).