EC2 Instance User Data Contains Secrets

Overview

This check inspects EC2 instance User Data for secret-like values including credentials, tokens, API keys, and passwords. User Data is a script or configuration that runs when an EC2 instance first starts. Unfortunately, it is a common (but insecure) practice to embed secrets directly in User Data for convenience.

The check analyzes both plain text and compressed content, looking for patterns that resemble sensitive material such as AWS access keys, database passwords, or API tokens.

Risk

Embedding secrets in User Data creates serious security vulnerabilities:

• Easy exposure: Anyone with EC2 read permissions can view User Data, including the embedded secrets
• Credential theft: Attackers who gain instance access can extract credentials and use them elsewhere
• Lateral movement: Stolen credentials often provide access to databases, APIs, or other AWS accounts
• Persistence in backups: Secrets in User Data persist across AMIs, snapshots, and backups, expanding exposure
• Audit trail gaps: Secrets in User Data bypass proper credential management and rotation workflows
• Compliance violations: Many security frameworks prohibit storing plaintext credentials in instance metadata

The severity is High because a single exposed secret can lead to widespread compromise of your environment.

Remediation Steps

Prerequisites

You need:

• AWS Console access with permissions to view and modify EC2 instances
• A plan for where to store the secrets securely (AWS Secrets Manager is recommended)

Required IAM permissions

Your IAM user or role needs these permissions:

For EC2 User Data remediation:

• ec2:DescribeInstances
• ec2:DescribeInstanceAttribute
• ec2:ModifyInstanceAttribute
• ec2:StopInstances (User Data can only be modified when instance is stopped)
• ec2:StartInstances

For setting up Secrets Manager (recommended alternative):

• secretsmanager:CreateSecret
• secretsmanager:PutSecretValue
• iam:CreateRole and iam:AttachRolePolicy (for instance role)

AWS Console Method

Step 1: Identify the Affected Instance

1. Go to the EC2 Console in us-east-1
2. Click Instances in the left sidebar
3. Find the flagged instance (note its Instance ID from the Prowler finding)
4. Select the instance by clicking its checkbox

Step 2: View Current User Data

1. With the instance selected, click Actions at the top
2. Go to Instance settings > Edit user data
3. Review the User Data content to identify the embedded secrets
4. Take note of what secrets exist and what they are used for

Step 3: Move Secrets to AWS Secrets Manager

Before removing secrets from User Data, store them securely:

1. Go to AWS Secrets Manager Console in us-east-1
2. Click Store a new secret
3. Select Other type of secret
4. Add your key-value pairs (e.g., DB_PASSWORD = your-password)
5. Click Next
6. Enter a secret name like myapp/database-credentials
7. Click Next through the rotation options (configure later if needed)
8. Click Store

Step 4: Update Instance Role for Secrets Manager Access

Your EC2 instance needs permission to retrieve secrets:

1. Go to IAM Console in us-east-1
2. Click Roles in the left sidebar
3. Find and click your EC2 instance role (or create a new one)
4. Click Add permissions > Attach policies
5. Search for and attach SecretsManagerReadWrite (or create a custom policy for least privilege)
6. Attach the role to your EC2 instance if not already attached:
   • Go back to EC2 Console > Instances
   • Select your instance > Actions > Security > Modify IAM role
   • Select the role and click Update IAM role

Step 5: Update Application Code

Modify your application to retrieve secrets at runtime instead of reading from environment variables or config files that were set by User Data.

Example approaches:

• Use the AWS SDK to call Secrets Manager when the application starts
• Use a wrapper script in User Data that retrieves secrets and sets environment variables (without storing the secrets in User Data itself)

Step 6: Remove Secrets from User Data

Important: The instance must be stopped to modify User Data.

1. Go to EC2 Console in us-east-1
2. Select the instance
3. Click Instance state > Stop instance
4. Wait for the instance to fully stop (State shows "Stopped")
5. Click Actions > Instance settings > Edit user data
6. Either:
   • Remove the secrets and keep the rest of the script
   • Clear User Data entirely if no longer needed
   • Replace with a script that retrieves secrets from Secrets Manager
7. Click Save
8. Click Instance state > Start instance

AWS CLI (optional)

View Current User Data

First, check the current User Data content (it will be base64-encoded):

aws ec2 describe-instance-attribute \
    --instance-id i-1234567890abcdef0 \
    --attribute userData \
    --region us-east-1 \
    --query 'UserData.Value' \
    --output text | base64 --decode

Replace i-1234567890abcdef0 with your actual instance ID.

Store Secrets in Secrets Manager

aws secretsmanager create-secret \
    --name myapp/database-credentials \
    --description "Database credentials for my application" \
    --secret-string '{"DB_USER":"admin","DB_PASSWORD":"your-secure-password"}' \
    --region us-east-1

Stop the Instance

User Data can only be modified when the instance is stopped:

aws ec2 stop-instances \
    --instance-ids i-1234567890abcdef0 \
    --region us-east-1

# Wait for instance to stop
aws ec2 wait instance-stopped \
    --instance-ids i-1234567890abcdef0 \
    --region us-east-1

Clear User Data

To completely clear User Data:

aws ec2 modify-instance-attribute \
    --instance-id i-1234567890abcdef0 \
    --user-data Value= \
    --region us-east-1

Replace User Data with Secure Script

If you need User Data for initialization but without embedded secrets, create a new script that retrieves secrets from Secrets Manager:

#!/bin/bash
# Retrieve database credentials from Secrets Manager
DB_CREDS=$(aws secretsmanager get-secret-value \
    --secret-id myapp/database-credentials \
    --region us-east-1 \
    --query SecretString \
    --output text)

# Export as environment variables
export DB_USER=$(echo $DB_CREDS | jq -r '.DB_USER')
export DB_PASSWORD=$(echo $DB_CREDS | jq -r '.DB_PASSWORD')

# Start your application
/opt/myapp/start.sh

Save this script, base64-encode it, and apply:

# Encode the new user data
base64 new-userdata.sh > new-userdata.b64

# Apply the new user data
aws ec2 modify-instance-attribute \
    --instance-id i-1234567890abcdef0 \
    --user-data file://new-userdata.b64 \
    --region us-east-1

Start the Instance

aws ec2 start-instances \
    --instance-ids i-1234567890abcdef0 \
    --region us-east-1

CloudFormation (optional)

When using CloudFormation, the best practice is to reference secrets from Secrets Manager directly rather than embedding them in UserData.

Insecure Pattern (DO NOT USE)

# BAD - Secrets embedded in UserData
Resources:
  MyInstance:
    Type: AWS::EC2::Instance
    Properties:
      UserData:
        Fn::Base64: |
          #!/bin/bash
          export DB_PASSWORD="hardcoded-secret"  # INSECURE!

Secure Pattern (RECOMMENDED)

AWSTemplateFormatVersion: '2010-09-09'
Description: EC2 instance that retrieves secrets from Secrets Manager at runtime

Parameters:
  InstanceType:
    Type: String
    Default: t3.micro

  AmiId:
    Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
    Default: /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2

  SecretArn:
    Type: String
    Description: ARN of the Secrets Manager secret containing application credentials

Resources:
  InstanceRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: ec2.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
      Policies:
        - PolicyName: SecretsManagerAccess
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - secretsmanager:GetSecretValue
                Resource: !Ref SecretArn

  InstanceProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
      Roles:
        - !Ref InstanceRole

  MyInstance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: !Ref InstanceType
      ImageId: !Ref AmiId
      IamInstanceProfile: !Ref InstanceProfile
      UserData:
        Fn::Base64: !Sub |
          #!/bin/bash
          # Retrieve secrets from Secrets Manager at runtime - no secrets in UserData!
          SECRET_JSON=$(aws secretsmanager get-secret-value \
              --secret-id ${SecretArn} \
              --region ${AWS::Region} \
              --query SecretString \
              --output text)

          # Parse and export as environment variables
          export DB_USER=$(echo $SECRET_JSON | jq -r '.DB_USER')
          export DB_PASSWORD=$(echo $SECRET_JSON | jq -r '.DB_PASSWORD')

          # Continue with application startup
          # /opt/myapp/start.sh

Outputs:
  InstanceId:
    Description: ID of the EC2 instance
    Value: !Ref MyInstance

Deploy with:

aws cloudformation deploy \
    --template-file ec2-with-secrets-manager.yaml \
    --stack-name my-secure-ec2 \
    --parameter-overrides \
        SecretArn=arn:aws:secretsmanager:us-east-1:123456789012:secret:myapp/database-credentials-AbCdEf \
    --capabilities CAPABILITY_IAM \
    --region us-east-1

Terraform (optional)

Insecure Pattern (DO NOT USE)

# BAD - Secrets embedded in user_data
resource "aws_instance" "bad_example" {
  ami           = data.aws_ami.amazon_linux.id
  instance_type = "t3.micro"

  user_data = <<-EOF
    #!/bin/bash
    export DB_PASSWORD="hardcoded-secret"  # INSECURE!
  EOF
}

Secure Pattern (RECOMMENDED)

# Variables
variable "secret_arn" {
  description = "ARN of the Secrets Manager secret"
  type        = string
}

# Data source for latest Amazon Linux 2 AMI
data "aws_ami" "amazon_linux" {
  most_recent = true
  owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["amzn2-ami-hvm-*-x86_64-gp2"]
  }
}

# IAM role for EC2 instance
resource "aws_iam_role" "instance_role" {
  name = "ec2-secrets-manager-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
      }
    ]
  })
}

# Policy to allow reading from Secrets Manager
resource "aws_iam_role_policy" "secrets_access" {
  name = "secrets-manager-access"
  role = aws_iam_role.instance_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "secretsmanager:GetSecretValue"
        ]
        Resource = var.secret_arn
      }
    ]
  })
}

# Instance profile
resource "aws_iam_instance_profile" "instance_profile" {
  name = "ec2-secrets-manager-profile"
  role = aws_iam_role.instance_role.name
}

# EC2 instance with secure user data
resource "aws_instance" "secure_example" {
  ami                  = data.aws_ami.amazon_linux.id
  instance_type        = "t3.micro"
  iam_instance_profile = aws_iam_instance_profile.instance_profile.name

  # User data retrieves secrets at runtime - no secrets embedded!
  user_data = <<-EOF
    #!/bin/bash
    # Retrieve secrets from Secrets Manager at runtime
    SECRET_JSON=$(aws secretsmanager get-secret-value \
        --secret-id ${var.secret_arn} \
        --region us-east-1 \
        --query SecretString \
        --output text)

    # Parse and export as environment variables
    export DB_USER=$(echo $SECRET_JSON | jq -r '.DB_USER')
    export DB_PASSWORD=$(echo $SECRET_JSON | jq -r '.DB_PASSWORD')

    # Continue with application startup
    # /opt/myapp/start.sh
  EOF

  tags = {
    Name = "secure-ec2-instance"
  }
}

output "instance_id" {
  description = "ID of the EC2 instance"
  value       = aws_instance.secure_example.id
}

Deploy with:

terraform init
terraform plan -var="secret_arn=arn:aws:secretsmanager:us-east-1:123456789012:secret:myapp/database-credentials-AbCdEf"
terraform apply -var="secret_arn=arn:aws:secretsmanager:us-east-1:123456789012:secret:myapp/database-credentials-AbCdEf"

Verification

After remediation, verify the secrets have been removed:

1. Check User Data in the Console:

   • Go to EC2 Console > Instances
   • Select the instance
   • Click Actions > Instance settings > Edit user data
   • Verify no secrets appear in the content

2. Re-run Prowler check:

   • Run the specific check again to confirm remediation:

   prowler aws --check ec2_instance_secrets_user_data

3. Test application functionality:

   • Verify your application still works correctly with secrets retrieved from Secrets Manager
   • Check application logs for any authentication errors

CLI verification commands

Check if User Data still contains secrets:

# Decode and view User Data
aws ec2 describe-instance-attribute \
    --instance-id i-1234567890abcdef0 \
    --attribute userData \
    --region us-east-1 \
    --query 'UserData.Value' \
    --output text | base64 --decode

The output should either be empty or contain a script that retrieves secrets from Secrets Manager rather than embedding them.

Verify the instance can access Secrets Manager:

# SSH into the instance and test secret retrieval
aws secretsmanager get-secret-value \
    --secret-id myapp/database-credentials \
    --region us-east-1

This command should succeed if the instance role is properly configured.

Additional Resources

• AWS Secrets Manager Documentation
• Tutorial: Rotating a Secret for an AWS Database
• EC2 User Data Best Practices
• IAM Roles for Amazon EC2
• Prowler AWS Security Tool

Notes

• Downtime required: Modifying User Data requires stopping the instance, which causes downtime. Plan accordingly and consider using Auto Scaling groups or load balancers to minimize impact.

• Existing secrets are compromised: If secrets were in User Data, assume they may have been exposed. Rotate all affected credentials after moving them to Secrets Manager.

• AMIs and snapshots: If you created AMIs or snapshots from an instance with secrets in User Data, those secrets are baked into those images. Consider deregistering affected AMIs and deleting snapshots after rotating credentials.

• Launch templates: If using Launch Templates, also update the User Data in your launch template to prevent new instances from being created with embedded secrets.

• Alternative secret stores: While this guide focuses on AWS Secrets Manager, you can also use:

  • AWS Systems Manager Parameter Store (SecureString parameters)
  • HashiCorp Vault
  • Environment variables injected by container orchestrators (ECS, EKS)

• Least privilege: When granting Secrets Manager access, use a custom IAM policy that only allows access to the specific secrets needed, rather than broad SecretsManagerReadWrite permissions.

• Secret rotation: AWS Secrets Manager supports automatic rotation for many secret types. Enable rotation to further reduce the risk of credential compromise.