Skip to main content

Ensure Enhanced Monitoring is Enabled for MSK (Kafka) Brokers

Overview​

This check verifies that Amazon MSK (Managed Streaming for Apache Kafka) clusters have enhanced monitoring enabled beyond the default level. Enhanced monitoring provides deeper visibility into broker performance, replication health, and consumer lag.

Severity: Medium Service: Amazon MSK (Kafka)

Risk​

Without enhanced monitoring, you have limited visibility into your MSK cluster's health. This can lead to:

  • Delayed detection of performance issues like broker saturation or throttling
  • Missed warnings about under-replicated partitions that could cause data loss
  • Inability to identify consumer lag problems before they impact applications
  • Slower incident response due to lack of detailed metrics

Remediation Steps​

Prerequisites​

You need:

  • Access to the AWS Console with permissions to modify MSK clusters, OR
  • AWS CLI configured with appropriate IAM permissions (kafka:UpdateMonitoring, kafka:DescribeCluster)

AWS Console Method​

  1. Open the Amazon MSK console at https://console.aws.amazon.com/msk/
  2. Select Clusters from the left navigation
  3. Click on the cluster name you want to update
  4. Click the Properties tab
  5. In the Monitoring section, click Edit
  6. Under Enhanced monitoring, select one of:
    • Per broker - Metrics for each broker (recommended minimum)
    • Per topic per broker - Additional per-topic metrics
    • Per topic per partition - Most detailed metrics (higher CloudWatch costs)
  7. Click Save changes
  8. Wait for the cluster to finish updating (status returns to "Active")

Note: Serverless MSK clusters include enhanced monitoring by default. This remediation applies only to provisioned clusters.

AWS CLI

First, get your cluster ARN and current version:

# List clusters to find the ARN
aws kafka list-clusters --region us-east-1

# Get the current cluster version (required for updates)
aws kafka describe-cluster \
    --cluster-arn <your-cluster-arn> \
    --region us-east-1 \
    --query 'ClusterInfo.CurrentVersion' \
    --output text

Then enable enhanced monitoring:

aws kafka update-monitoring \
    --cluster-arn <your-cluster-arn> \
    --current-version <current-version> \
    --enhanced-monitoring PER_BROKER \
    --region us-east-1

Enhanced monitoring levels:

  • DEFAULT - Basic metrics only (does not pass this check)
  • PER_BROKER - Per-broker metrics (minimum recommended)
  • PER_TOPIC_PER_BROKER - Per-topic metrics for each broker
  • PER_TOPIC_PER_PARTITION - Most granular metrics
CloudFormation

Set the EnhancedMonitoring property to PER_BROKER or higher:

AWSTemplateFormatVersion: '2010-09-09'
Description: MSK Cluster with Enhanced Monitoring

Parameters:
  ClusterName:
    Type: String
    Description: Name of the MSK cluster
  SubnetIds:
    Type: List<AWS::EC2::Subnet::Id>
    Description: Subnet IDs for the MSK cluster (minimum 2)
  SecurityGroupId:
    Type: AWS::EC2::SecurityGroup::Id
    Description: Security group ID for the MSK cluster

Resources:
  MSKCluster:
    Type: AWS::MSK::Cluster
    Properties:
      ClusterName: !Ref ClusterName
      KafkaVersion: '3.5.1'
      NumberOfBrokerNodes: 3
      EnhancedMonitoring: PER_BROKER
      BrokerNodeGroupInfo:
        InstanceType: kafka.m5.large
        ClientSubnets: !Ref SubnetIds
        SecurityGroups:
          - !Ref SecurityGroupId
        StorageInfo:
          EBSStorageInfo:
            VolumeSize: 100

Outputs:
  ClusterArn:
    Description: ARN of the MSK cluster
    Value: !Ref MSKCluster

Valid values for EnhancedMonitoring:

  • DEFAULT
  • PER_BROKER
  • PER_TOPIC_PER_BROKER
  • PER_TOPIC_PER_PARTITION
Terraform

Set the enhanced_monitoring attribute to "PER_BROKER" or higher:

resource "aws_msk_cluster" "main" {
  cluster_name           = var.cluster_name
  kafka_version          = var.kafka_version
  number_of_broker_nodes = 3
  enhanced_monitoring    = "PER_BROKER"

  broker_node_group_info {
    instance_type   = "kafka.m5.large"
    client_subnets  = var.subnet_ids
    security_groups = var.security_group_ids

    storage_info {
      ebs_storage_info {
        volume_size = 100
      }
    }
  }
}

Valid values for enhanced_monitoring:

  • "DEFAULT"
  • "PER_BROKER"
  • "PER_TOPIC_PER_BROKER"
  • "PER_TOPIC_PER_PARTITION"

Verification​

After making changes, verify enhanced monitoring is enabled:

  1. In the AWS Console, navigate to your MSK cluster
  2. Check the Properties tab under Monitoring
  3. Confirm Enhanced monitoring shows PER_BROKER or higher
CLI Verification
aws kafka describe-cluster \
    --cluster-arn <your-cluster-arn> \
    --region us-east-1 \
    --query 'ClusterInfo.EnhancedMonitoring' \
    --output text

The output should be PER_BROKER, PER_TOPIC_PER_BROKER, or PER_TOPIC_PER_PARTITION.

Additional Resources​

Notes​

  • Cost consideration: Higher monitoring levels generate more CloudWatch metrics, which increases costs. PER_BROKER is usually sufficient for most use cases.
  • Update time: Changing the monitoring level triggers a rolling update that can take several minutes to complete.
  • Serverless clusters: MSK Serverless clusters include enhanced monitoring automatically and do not need this configuration.
  • Compliance frameworks: This check aligns with AWS Foundational Security Best Practices, KISA-ISMS-P, and NIS2 requirements.