OpenSearch Service Domain Should Have at Least Three Dedicated Master Nodes

Overview

This check verifies that your Amazon OpenSearch Service domains have dedicated master nodes enabled with a minimum count of three. Dedicated master nodes handle cluster management tasks like tracking nodes, managing indices, and coordinating shard allocation. Having at least three ensures the cluster can maintain a quorum (majority agreement) even if one master node fails.

Risk

Without at least three dedicated master nodes, your OpenSearch cluster is vulnerable to:

• Loss of quorum: If one of two master nodes fails, the cluster cannot elect a new leader
• Cluster unavailability: Read and write operations may fail during leader election problems
• Split-brain scenarios: The cluster could become inconsistent if nodes cannot agree on state
• Shard allocation failures: New data may not be properly distributed across nodes

Using an odd number (3 or 5) of master nodes is an industry best practice for distributed systems that rely on consensus algorithms.

Remediation Steps

Prerequisites

You need permission to modify OpenSearch domain configurations in your AWS account. Specifically, you need the es:UpdateDomainConfig permission.

Required IAM permissions

The following IAM policy grants the minimum permissions needed:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "es:DescribeDomain",
        "es:UpdateDomainConfig"
      ],
      "Resource": "arn:aws:es:us-east-1:<account-id>:domain/<domain-name>"
    }
  ]
}

AWS Console Method

1. Sign in to the AWS Management Console
2. Navigate to Amazon OpenSearch Service
3. Select the domain you want to update
4. Click Edit in the top right
5. Scroll down to the Cluster configuration section
6. Check the box for Enable dedicated master nodes
7. Set Number of dedicated master nodes to 3
8. Choose an appropriate Dedicated master node instance type (e.g., m5.large.search for most workloads)
9. Click Save changes

Note: The domain will enter a "Processing" state while the configuration is applied. This can take 15-30 minutes depending on your cluster size. The domain remains available during this time.

AWS CLI (optional)

Use the following command to enable three dedicated master nodes:

aws opensearch update-domain-config \
  --region us-east-1 \
  --domain-name <your-domain-name> \
  --cluster-config '{
    "DedicatedMasterEnabled": true,
    "DedicatedMasterCount": 3,
    "DedicatedMasterType": "m5.large.search"
  }'

Replace:

• <your-domain-name> with your OpenSearch domain name

To verify the current configuration before making changes:

aws opensearch describe-domain \
  --region us-east-1 \
  --domain-name <your-domain-name> \
  --query 'DomainStatus.ClusterConfig.{DedicatedMasterEnabled:DedicatedMasterEnabled,DedicatedMasterCount:DedicatedMasterCount,DedicatedMasterType:DedicatedMasterType}'

CloudFormation (optional)

Use this CloudFormation template to create or update an OpenSearch domain with fault-tolerant dedicated master nodes:

AWSTemplateFormatVersion: '2010-09-09'
Description: OpenSearch domain with fault-tolerant dedicated master nodes

Parameters:
  DomainName:
    Type: String
    Description: Name of the OpenSearch domain
    MinLength: 3
    MaxLength: 28
    AllowedPattern: '[a-z][a-z0-9\-]+'

  MasterNodeInstanceType:
    Type: String
    Description: Instance type for dedicated master nodes
    Default: m5.large.search
    AllowedValues:
      - m5.large.search
      - m5.xlarge.search
      - m5.2xlarge.search
      - r5.large.search
      - r5.xlarge.search

  DataNodeInstanceType:
    Type: String
    Description: Instance type for data nodes
    Default: r5.large.search

  DataNodeCount:
    Type: Number
    Description: Number of data nodes
    Default: 2
    MinValue: 1

Resources:
  OpenSearchDomain:
    Type: AWS::OpenSearchService::Domain
    Properties:
      DomainName: !Ref DomainName
      EngineVersion: OpenSearch_2.11
      ClusterConfig:
        DedicatedMasterEnabled: true
        DedicatedMasterCount: 3
        DedicatedMasterType: !Ref MasterNodeInstanceType
        InstanceCount: !Ref DataNodeCount
        InstanceType: !Ref DataNodeInstanceType
        ZoneAwarenessEnabled: true
        ZoneAwarenessConfig:
          AvailabilityZoneCount: 3
      EBSOptions:
        EBSEnabled: true
        VolumeType: gp3
        VolumeSize: 100
      NodeToNodeEncryptionOptions:
        Enabled: true
      EncryptionAtRestOptions:
        Enabled: true
      DomainEndpointOptions:
        EnforceHTTPS: true

Outputs:
  DomainArn:
    Description: ARN of the OpenSearch domain
    Value: !GetAtt OpenSearchDomain.Arn
  DomainEndpoint:
    Description: Endpoint of the OpenSearch domain
    Value: !GetAtt OpenSearchDomain.DomainEndpoint

Deploy the stack:

aws cloudformation deploy \
  --region us-east-1 \
  --template-file opensearch-domain.yaml \
  --stack-name opensearch-fault-tolerant \
  --parameter-overrides DomainName=my-domain

Terraform (optional)

Use this Terraform configuration to create or manage an OpenSearch domain with fault-tolerant dedicated master nodes:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

variable "domain_name" {
  description = "Name of the OpenSearch domain"
  type        = string
}

variable "master_node_instance_type" {
  description = "Instance type for dedicated master nodes"
  type        = string
  default     = "m5.large.search"
}

variable "data_node_instance_type" {
  description = "Instance type for data nodes"
  type        = string
  default     = "r5.large.search"
}

variable "data_node_count" {
  description = "Number of data nodes"
  type        = number
  default     = 2
}

resource "aws_opensearch_domain" "example" {
  domain_name    = var.domain_name
  engine_version = "OpenSearch_2.11"

  cluster_config {
    dedicated_master_enabled = true
    dedicated_master_count   = 3
    dedicated_master_type    = var.master_node_instance_type
    instance_count           = var.data_node_count
    instance_type            = var.data_node_instance_type
    zone_awareness_enabled   = true

    zone_awareness_config {
      availability_zone_count = 3
    }
  }

  ebs_options {
    ebs_enabled = true
    volume_type = "gp3"
    volume_size = 100
  }

  encrypt_at_rest {
    enabled = true
  }

  node_to_node_encryption {
    enabled = true
  }

  domain_endpoint_options {
    enforce_https = true
  }

  tags = {
    Environment = "production"
  }
}

output "domain_arn" {
  description = "ARN of the OpenSearch domain"
  value       = aws_opensearch_domain.example.arn
}

output "domain_endpoint" {
  description = "Endpoint of the OpenSearch domain"
  value       = aws_opensearch_domain.example.endpoint
}

Apply the configuration:

terraform init
terraform plan -var="domain_name=my-domain"
terraform apply -var="domain_name=my-domain"

Verification

After making changes, verify the configuration is correct:

1. In the AWS Console, go to Amazon OpenSearch Service
2. Select your domain
3. Under the Cluster configuration tab, confirm:
   • Dedicated master nodes shows Enabled
   • Number of dedicated master nodes shows 3

CLI verification commands

aws opensearch describe-domain \
  --region us-east-1 \
  --domain-name <your-domain-name> \
  --query 'DomainStatus.ClusterConfig.{DedicatedMasterEnabled:DedicatedMasterEnabled,DedicatedMasterCount:DedicatedMasterCount,DedicatedMasterType:DedicatedMasterType}'

Expected output:

{
    "DedicatedMasterEnabled": true,
    "DedicatedMasterCount": 3,
    "DedicatedMasterType": "m5.large.search"
}

Additional Resources

• Amazon OpenSearch Service Developer Guide - Dedicated Master Nodes
• Amazon OpenSearch Service Best Practices
• Sizing Amazon OpenSearch Service Domains
• Prowler Check Documentation

Notes

• Cost consideration: Dedicated master nodes add cost to your domain. Three m5.large.search instances will add approximately $0.50/hour to your bill. However, the stability benefits typically outweigh the cost for production workloads.
• Instance type selection: Choose a master node instance type based on your cluster size:
  • Small clusters (< 10 data nodes): m5.large.search or m6g.large.search
  • Medium clusters (10-30 data nodes): m5.xlarge.search or m6g.xlarge.search
  • Large clusters (30+ data nodes): m5.2xlarge.search or larger
• Zone awareness: For maximum fault tolerance, enable zone awareness alongside dedicated master nodes so masters are distributed across availability zones.
• DedicatedMasterCount limitation: AWS requires the count to be greater than 2 and not equal to 4. Valid values are 3 or 5 for most use cases.
• Update timing: Configuration changes are applied using a blue/green deployment. The domain remains available but performance may be slightly affected during the update.