OpenSearch CloudWatch Logging

Overview

This check verifies that your Amazon OpenSearch Service domains publish slow logs to CloudWatch Logs. Specifically, it looks for two log types:

• Search slow logs - Records queries that take longer than expected
• Index slow logs - Records indexing operations that take longer than expected

Enabling these logs gives you visibility into performance issues before they become outages.

Risk

Without slow logs enabled, you lose visibility into what is happening inside your OpenSearch domain:

• Performance problems go undetected - Slow queries and indexing can silently degrade user experience
• Capacity planning becomes guesswork - You cannot identify which operations need optimization
• Incident response is slower - When issues occur, you have no historical data to diagnose root causes
• Potential abuse is hidden - Malicious or inefficient queries may go unnoticed

Remediation Steps

Prerequisites

• Access to the AWS Console with permissions to modify OpenSearch domains
• The OpenSearch domain must be in "Active" status before making changes

Required IAM permissions

You need the following permissions:

• es:UpdateDomainConfig - To enable log publishing on the domain
• logs:CreateLogGroup - To create CloudWatch log groups
• logs:PutResourcePolicy - To allow OpenSearch to write to CloudWatch

AWS Console Method

1. Open the Amazon OpenSearch Service console at https://console.aws.amazon.com/aos/

2. Select your domain from the list

3. Click the Logs tab

4. Under Search slow logs:

   • Click Enable
   • Select an existing CloudWatch log group or click Create new log group
   • If creating new, use a name like /aws/opensearch/domains/<your-domain>/search-slow-logs
   • Click Save

5. Under Index slow logs:

   • Click Enable
   • Select an existing CloudWatch log group or click Create new log group
   • If creating new, use a name like /aws/opensearch/domains/<your-domain>/index-slow-logs
   • Click Save

6. Wait for the domain status to return to Active (this may take several minutes)

AWS CLI (optional)

First, create CloudWatch log groups for the logs:

# Create log groups
aws logs create-log-group \
  --log-group-name /aws/opensearch/domains/<your-domain>/search-slow-logs \
  --region us-east-1

aws logs create-log-group \
  --log-group-name /aws/opensearch/domains/<your-domain>/index-slow-logs \
  --region us-east-1

Next, create a resource policy to allow OpenSearch to write to these log groups:

aws logs put-resource-policy \
  --policy-name opensearch-logs-policy \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "Service": "es.amazonaws.com"
        },
        "Action": [
          "logs:PutLogEvents",
          "logs:CreateLogStream"
        ],
        "Resource": "arn:aws:logs:us-east-1:<your-account-id>:log-group:/aws/opensearch/domains/<your-domain>/*"
      }
    ]
  }' \
  --region us-east-1

Finally, enable log publishing on your domain:

aws opensearch update-domain-config \
  --domain-name <your-domain> \
  --log-publishing-options '{
    "SEARCH_SLOW_LOGS": {
      "CloudWatchLogsLogGroupArn": "arn:aws:logs:us-east-1:<your-account-id>:log-group:/aws/opensearch/domains/<your-domain>/search-slow-logs",
      "Enabled": true
    },
    "INDEX_SLOW_LOGS": {
      "CloudWatchLogsLogGroupArn": "arn:aws:logs:us-east-1:<your-account-id>:log-group:/aws/opensearch/domains/<your-domain>/index-slow-logs",
      "Enabled": true
    }
  }' \
  --region us-east-1

Replace:

• <your-domain> with your OpenSearch domain name
• <your-account-id> with your AWS account ID

CloudFormation (optional)

AWSTemplateFormatVersion: '2010-09-09'
Description: OpenSearch domain with CloudWatch logging enabled

Parameters:
  DomainName:
    Type: String
    Description: Name of the OpenSearch domain
    Default: my-opensearch-domain

Resources:
  # CloudWatch Log Group for Search Slow Logs
  SearchSlowLogsLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub /aws/opensearch/domains/${DomainName}/search-slow-logs
      RetentionInDays: 30

  # CloudWatch Log Group for Index Slow Logs
  IndexSlowLogsLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub /aws/opensearch/domains/${DomainName}/index-slow-logs
      RetentionInDays: 30

  # Resource Policy for CloudWatch Logs
  LogsResourcePolicy:
    Type: AWS::Logs::ResourcePolicy
    Properties:
      PolicyName: !Sub ${DomainName}-opensearch-logs-policy
      PolicyDocument: !Sub |
        {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Principal": {
                "Service": "es.amazonaws.com"
              },
              "Action": [
                "logs:PutLogEvents",
                "logs:CreateLogStream"
              ],
              "Resource": [
                "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/opensearch/domains/${DomainName}/*"
              ]
            }
          ]
        }

  # OpenSearch Domain with Logging Enabled
  OpenSearchDomain:
    Type: AWS::OpenSearchService::Domain
    DependsOn:
      - LogsResourcePolicy
    Properties:
      DomainName: !Ref DomainName
      EngineVersion: OpenSearch_2.11
      ClusterConfig:
        InstanceType: t3.small.search
        InstanceCount: 1
      EBSOptions:
        EBSEnabled: true
        VolumeType: gp3
        VolumeSize: 10
      LogPublishingOptions:
        SEARCH_SLOW_LOGS:
          CloudWatchLogsLogGroupArn: !GetAtt SearchSlowLogsLogGroup.Arn
          Enabled: true
        INDEX_SLOW_LOGS:
          CloudWatchLogsLogGroupArn: !GetAtt IndexSlowLogsLogGroup.Arn
          Enabled: true

Outputs:
  DomainEndpoint:
    Description: OpenSearch domain endpoint
    Value: !GetAtt OpenSearchDomain.DomainEndpoint
  SearchSlowLogsLogGroupArn:
    Description: ARN of the Search Slow Logs CloudWatch Log Group
    Value: !GetAtt SearchSlowLogsLogGroup.Arn
  IndexSlowLogsLogGroupArn:
    Description: ARN of the Index Slow Logs CloudWatch Log Group
    Value: !GetAtt IndexSlowLogsLogGroup.Arn

Deploy with:

aws cloudformation deploy \
  --template-file template.yaml \
  --stack-name opensearch-logging \
  --parameter-overrides DomainName=<your-domain> \
  --region us-east-1

Terraform (optional)

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

variable "domain_name" {
  description = "Name of the OpenSearch domain"
  type        = string
  default     = "my-opensearch-domain"
}

# CloudWatch Log Groups
resource "aws_cloudwatch_log_group" "search_slow_logs" {
  name              = "/aws/opensearch/domains/${var.domain_name}/search-slow-logs"
  retention_in_days = 30
}

resource "aws_cloudwatch_log_group" "index_slow_logs" {
  name              = "/aws/opensearch/domains/${var.domain_name}/index-slow-logs"
  retention_in_days = 30
}

# Resource Policy for OpenSearch to write to CloudWatch Logs
data "aws_caller_identity" "current" {}
data "aws_region" "current" {}

resource "aws_cloudwatch_log_resource_policy" "opensearch_logs" {
  policy_name = "${var.domain_name}-opensearch-logs-policy"

  policy_document = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "es.amazonaws.com"
        }
        Action = [
          "logs:PutLogEvents",
          "logs:CreateLogStream"
        ]
        Resource = "arn:aws:logs:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:log-group:/aws/opensearch/domains/${var.domain_name}/*"
      }
    ]
  })
}

# OpenSearch Domain with Logging Enabled
resource "aws_opensearch_domain" "main" {
  domain_name    = var.domain_name
  engine_version = "OpenSearch_2.11"

  cluster_config {
    instance_type  = "t3.small.search"
    instance_count = 1
  }

  ebs_options {
    ebs_enabled = true
    volume_type = "gp3"
    volume_size = 10
  }

  log_publishing_options {
    cloudwatch_log_group_arn = aws_cloudwatch_log_group.search_slow_logs.arn
    log_type                 = "SEARCH_SLOW_LOGS"
    enabled                  = true
  }

  log_publishing_options {
    cloudwatch_log_group_arn = aws_cloudwatch_log_group.index_slow_logs.arn
    log_type                 = "INDEX_SLOW_LOGS"
    enabled                  = true
  }

  depends_on = [aws_cloudwatch_log_resource_policy.opensearch_logs]
}

output "domain_endpoint" {
  description = "OpenSearch domain endpoint"
  value       = aws_opensearch_domain.main.endpoint
}

output "search_slow_logs_arn" {
  description = "ARN of the Search Slow Logs CloudWatch Log Group"
  value       = aws_cloudwatch_log_group.search_slow_logs.arn
}

output "index_slow_logs_arn" {
  description = "ARN of the Index Slow Logs CloudWatch Log Group"
  value       = aws_cloudwatch_log_group.index_slow_logs.arn
}

Apply with:

terraform init
terraform apply -var="domain_name=<your-domain>"

Verification

After enabling logging, verify the configuration:

1. Go to the Amazon OpenSearch Service console
2. Select your domain and click the Logs tab
3. Confirm both Search slow logs and Index slow logs show as Enabled
4. Check that each points to a valid CloudWatch log group

CLI verification

aws opensearch describe-domain \
  --domain-name <your-domain> \
  --query 'DomainStatus.LogPublishingOptions' \
  --region us-east-1

You should see output showing both SEARCH_SLOW_LOGS and INDEX_SLOW_LOGS with Enabled: true.

Additional Resources

• Monitoring OpenSearch logs with Amazon CloudWatch Logs
• Troubleshooting OpenSearch slow logs
• OpenSearch slow log configuration

Notes

• Logs require REST API configuration: After enabling log publishing, you must also configure slow log thresholds using the OpenSearch REST API. By default, thresholds are set very high, so no logs will appear until you adjust them.

• Log group permissions: The CloudWatch log group must have a resource policy allowing OpenSearch (es.amazonaws.com) to write to it. The console handles this automatically, but CLI/IaC approaches require explicit policy creation.

• Domain update time: Enabling logging triggers a domain update that can take several minutes. The domain remains available during this time.

• Cost considerations: CloudWatch Logs incur charges for ingestion and storage. Consider setting appropriate retention periods and log thresholds to manage costs.

• Additional log types: OpenSearch also supports ES_APPLICATION_LOGS (error logs) and AUDIT_LOGS (access audit). This check focuses on slow logs, but consider enabling application logs for troubleshooting.