Kinesis Streams Should Have an Adequate Data Retention Period

Overview

This check ensures that your Amazon Kinesis Data Streams have a retention period of at least 168 hours (7 days). The retention period determines how long data records remain accessible after being added to the stream.

By default, Kinesis streams retain data for only 24 hours. For production workloads, this is often too short to handle processing delays, consumer failures, or replay requirements.

Risk

If your retention period is too short:

Data loss during outages: If a consumer application goes down, records may expire before processing resumes.
No replay capability: You cannot reprocess historical data for debugging, analytics, or recovery.
Compliance gaps: Some regulations require data to be retained for specific periods.
Recovery difficulties: Troubleshooting issues becomes harder without access to recent data history.

Remediation Steps

Prerequisites

You need permission to modify Kinesis stream settings. This typically requires the kinesis:IncreaseStreamRetentionPeriod IAM permission.

AWS Console Method

Open the Amazon Kinesis console at https://console.aws.amazon.com/kinesis/
In the left navigation, select Data streams
Click on the stream name you want to modify
Choose the Configuration tab
In the Data retention period section, click Edit
Enter a value of 168 hours (or higher based on your requirements)
Click Save changes

The change takes effect immediately. Existing data that has not yet expired will remain available for the new retention period.

AWS CLI (optional)

Increase Retention Period

To increase the retention period to 168 hours (7 days):

aws kinesis increase-stream-retention-period \
    --stream-name <your-stream-name> \
    --retention-period-hours 168 \
    --region us-east-1

Replace <your-stream-name> with your actual stream name.

Check Current Retention Period

To view the current retention period for a stream:

aws kinesis describe-stream-summary \
    --stream-name <your-stream-name> \
    --region us-east-1 \
    --query 'StreamDescriptionSummary.RetentionPeriodHours'

Using Stream ARN Instead of Name

You can also specify the stream by ARN:

aws kinesis increase-stream-retention-period \
    --stream-arn arn:aws:kinesis:us-east-1:123456789012:stream/my-stream \
    --retention-period-hours 168 \
    --region us-east-1

Important Notes

You can only increase retention using increase-stream-retention-period. To decrease it, use decrease-stream-retention-period (but be aware this can cause data loss).
Maximum retention is 8760 hours (365 days).
Minimum retention is 24 hours.
Extended retention incurs additional costs. See Kinesis pricing for details.

CloudFormation (optional)

Use this template to create a new Kinesis stream with an adequate retention period, or update an existing stack.

AWSTemplateFormatVersion: '2010-09-09'
Description: Kinesis Data Stream with adequate retention period

Parameters:
  StreamName:
    Type: String
    Description: Name of the Kinesis stream
    Default: my-kinesis-stream
  RetentionPeriodHours:
    Type: Number
    Description: Data retention period in hours (24-8760)
    Default: 168
    MinValue: 24
    MaxValue: 8760
  ShardCount:
    Type: Number
    Description: Number of shards for the stream
    Default: 1
    MinValue: 1

Resources:
  KinesisStream:
    Type: AWS::Kinesis::Stream
    Properties:
      Name: !Ref StreamName
      RetentionPeriodHours: !Ref RetentionPeriodHours
      ShardCount: !Ref ShardCount
      Tags:
        - Key: Environment
          Value: production

Outputs:
  StreamArn:
    Description: ARN of the Kinesis stream
    Value: !GetAtt KinesisStream.Arn
  StreamName:
    Description: Name of the Kinesis stream
    Value: !Ref KinesisStream

Deploy the Stack

aws cloudformation deploy \
    --template-file kinesis-stream.yaml \
    --stack-name kinesis-retention-stack \
    --parameter-overrides \
        StreamName=my-stream \
        RetentionPeriodHours=168 \
    --region us-east-1

Terraform (optional)

Basic Configuration

resource "aws_kinesis_stream" "example" {
  name             = "my-kinesis-stream"
  shard_count      = 1
  retention_period = 168  # 7 days in hours

  tags = {
    Environment = "production"
  }
}

With Enhanced Fan-Out and Encryption

resource "aws_kinesis_stream" "secure_stream" {
  name             = "secure-kinesis-stream"
  shard_count      = 2
  retention_period = 168

  encryption_type = "KMS"
  kms_key_id      = aws_kms_key.kinesis.arn

  stream_mode_details {
    stream_mode = "PROVISIONED"
  }

  tags = {
    Environment = "production"
    Compliance  = "required"
  }
}

resource "aws_kms_key" "kinesis" {
  description             = "KMS key for Kinesis stream encryption"
  deletion_window_in_days = 7
}

Apply Changes

terraform plan
terraform apply

Updating Existing Streams

If you have an existing Terraform-managed stream, simply update the retention_period value and apply:

resource "aws_kinesis_stream" "existing" {
  name             = "existing-stream"
  shard_count      = 1
  retention_period = 168  # Changed from 24 to 168
}

Verification

After making changes, confirm the retention period was updated:

In the AWS Console, go to Kinesis > Data streams
Select your stream and check the Configuration tab
Verify the Data retention period shows your new value (e.g., 168 hours)

CLI Verification

aws kinesis describe-stream-summary \
    --stream-name <your-stream-name> \
    --region us-east-1 \
    --query 'StreamDescriptionSummary.{Name:StreamName,RetentionHours:RetentionPeriodHours}'

Expected output:

{
    "Name": "my-stream",
    "RetentionHours": 168
}

To verify all streams in your account:

aws kinesis list-streams --region us-east-1 --query 'StreamNames[]' --output text | \
while read stream; do
    echo "Stream: $stream"
    aws kinesis describe-stream-summary \
        --stream-name "$stream" \
        --region us-east-1 \
        --query 'StreamDescriptionSummary.RetentionPeriodHours' \
        --output text
done

Additional Resources

Notes

Cost implications: Extended retention (beyond 24 hours) incurs additional charges. Review pricing before setting very long retention periods.
Increasing vs. decreasing: You can increase retention at any time. Decreasing retention may cause immediate data loss for records older than the new period.
Historical data: Increasing retention does not recover data that has already expired. Only data added after the original retention period began will benefit.
Recommended minimum: 168 hours (7 days) provides a reasonable buffer for most production workloads, but adjust based on your specific recovery time objectives and compliance requirements.
Maximum retention: Kinesis supports up to 8760 hours (365 days) of retention.

Overview​

Risk​

Remediation Steps​

Prerequisites​

AWS Console Method​

Increase Retention Period​

Check Current Retention Period​

Using Stream ARN Instead of Name​

Important Notes​

Deploy the Stack​

Basic Configuration​

With Enhanced Fan-Out and Encryption​

Apply Changes​

Updating Existing Streams​

Verification​

Additional Resources​

Notes​