WAF++ WAF++
Back to WAF++ Homepage

Best Practice: Secrets Management

Context

Secrets – database passwords, API keys, TLS certificates, OAuth client secrets – are the most sensitive artifact in any cloud architecture. At the same time, secrets management is the most frequently misimplemented security concept.

Real consequences of poor secrets management:

  • GitHub secret leaks: Thousands of repositories contain active AWS access keys in the commit history – even after files are deleted via git log

  • CI/CD credential exposure: Build logs with echo $DATABASE_PASSWORD are stored in artifact storage and are visible to all CI users

  • Shared service credentials: A database password is distributed to 5 developers and 3 services – rotation breaks all of them simultaneously, so it is never done

  • Hardcoded in Docker images: ENV variables in Dockerfiles end up in the image layer and are visible in every copy of the image

Related Controls

Why Secrets Management Is Critical

A single exposed AWS access key with extensive permissions can lead within minutes to:

  • Complete data exfiltration (S3, RDS snapshots)

  • Crypto mining on all EC2 instances (cost potential: thousands of euros per day)

  • Creation of backdoor IAM users for persistent access

  • Deletion of critical resources (DynamoDB tables, S3 buckets)

AWS itself detects exposed keys in public GitHub repositories through the GitHub Secret Scanning integration – but the response often comes too late.

Tool Comparison: Secrets Manager, SSM Parameter Store, HashiCorp Vault

Criterion AWS Secrets Manager SSM Parameter Store HashiCorp Vault

Cost

~$0.40 per secret per month + API calls

Standard free; Advanced: ~$0.05 per parameter

Open source free; enterprise features paid

Automatic Rotation

Native for RDS, DocumentDB, Redshift; custom Lambda for all others

Not native; custom scripts required

Native for databases, cloud credentials, SSH

Dynamic Secrets

Not native

Not native

Native – key feature

Cross-account Access

Resource-based policy; IAM

IAM with account-specific ARNs

Token-based; cloud-agnostic

Audit Log

CloudTrail

CloudTrail

Vault audit log (more detailed)

KMS Integration

Native; CMK configurable per secret

Native for SecureString

Transit engine; own encryption

Multi-cloud

AWS only

AWS only

Cloud-agnostic

Recommendation

For AWS-native workloads; database passwords with auto-rotation

For configuration values and less sensitive secrets

For multi-cloud, dynamic secrets, complex hierarchies

Secret Rotation

Automatic Rotation with AWS Secrets Manager

resource "aws_secretsmanager_secret" "db_password" {
  name        = "production/payment-service/db-password"
  description = "PostgreSQL password for the payment service"

  # CMK encryption – no AWS-managed key
  kms_key_id = aws_kms_key.secrets_cmk.arn

  # Recovery window: at least 7 days (WAF-SEC-060)
  recovery_window_in_days = 30

  tags = {
    owner       = "platform-team"
    workload    = "payment-service"
    data-class  = "confidential"
    environment = "production"
  }
}

resource "aws_secretsmanager_secret_rotation" "db_password" {
  secret_id           = aws_secretsmanager_secret.db_password.id
  rotation_lambda_arn = aws_lambda_function.rds_rotation.arn

  rotation_rules {
    automatically_after_days = 30  # Rotation every 30 days
  }
}

AWS-managed Rotation for RDS

For RDS, Aurora, DocumentDB, ElastiCache, and Redshift, AWS provides ready-made rotation Lambda functions without custom code:

resource "aws_secretsmanager_secret" "rds_master" {
  name       = "production/payment-db/master-credentials"
  kms_key_id = aws_kms_key.secrets_cmk.arn

  recovery_window_in_days = 30

  tags = {
    owner      = "platform-team"
    data-class = "confidential"
  }
}

resource "aws_secretsmanager_secret_version" "rds_master_initial" {
  secret_id = aws_secretsmanager_secret.rds_master.id
  secret_string = jsonencode({
    username = "payment_svc"
    password = random_password.db_initial.result
    engine   = "postgres"
    host     = aws_db_instance.payment.address
    port     = 5432
    dbname   = "payment"
  })
}

# Rotation via AWS-managed Lambda – no custom Lambda required
resource "aws_secretsmanager_secret_rotation" "rds_master" {
  secret_id           = aws_secretsmanager_secret.rds_master.id
  rotation_lambda_arn = "arn:aws:lambda:eu-central-1:${data.aws_caller_identity.current.account_id}:function:SecretsManagerRDSPostgreSQLRotationSingleUser"

  rotation_rules {
    automatically_after_days = 30
  }
}

Dynamic Secrets Concept

Dynamic secrets are credentials that are generated on demand per session and automatically become invalid after a defined time. HashiCorp Vault implements this concept natively for databases:

Dynamic Secrets Flow (HashiCorp Vault Database Engine):

1. Payment service starts → authenticates with Vault via Kubernetes Auth
2. Vault generates a temporary database user:
   - Username: v-k8s-payment-abcd1234
   - Password: randomly generated
   - TTL: 1 hour (then automatically deleted)
3. Payment service connects to the database with these credentials
4. After 1 hour: Vault automatically deletes the database user
5. Payment service renews credentials 15 minutes before expiry

Advantages:

  • No permanent password – nothing to steal and abuse long-term

  • Full audit trail: every database access is attributable to a service token

  • Rotation is inherent – credentials are short-lived by design

Secrets in CI/CD: GitHub OIDC + AWS STS

Static AWS access keys in CI/CD are one of the greatest risks. The alternative: GitHub OIDC allows GitHub Actions to obtain temporary AWS credentials without static keys:

# Register OIDC provider for GitHub
resource "aws_iam_openid_connect_provider" "github" {
  url = "https://token.actions.githubusercontent.com"

  client_id_list = ["sts.amazonaws.com"]

  thumbprint_list = [
    "6938fd4d98bab03faadb97b34396831e3780aea1",
    "1c58a3a8518e8759bf075b76b750d4f2df264fcd"
  ]
}

# IAM role for the deployment workflow
resource "aws_iam_role" "github_deployment" {
  name = "github-actions-payment-service-deploy"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        Federated = aws_iam_openid_connect_provider.github.arn
      }
      Action = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "token.actions.githubusercontent.com:aud" = "sts.amazonaws.com"
        }
        StringLike = {
          # Only the main branch of the specific repository
          "token.actions.githubusercontent.com:sub" = "repo:acme-corp/payment-service:ref:refs/heads/main"
        }
      }
    }]
  })
}

GitHub Actions workflow configuration (no static key):

name: Deploy Payment Service

on:
  push:
    branches: [main]

permissions:
  id-token: write   # Request OIDC token
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS Credentials via OIDC
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-actions-payment-service-deploy
          aws-region: eu-central-1
          # No aws-access-key-id or aws-secret-access-key required

      - name: Deploy to ECS
        run: |
          aws ecs update-service \
            --cluster production \
            --service payment-service \
            --force-new-deployment

Anti-Patterns

Anti-Pattern Why Dangerous

Secrets in environment variables in Dockerfile

End up in image layers; visible in docker inspect; no rotation mechanism

Secrets in Terraform state

State file contains all values in plaintext; S3 backend needs CMK encryption

Secrets in Git (even after git rm)

Preserved in commit history; extractable at any time via git log -p

Never rotating passwords because rotation is work

Compromised credentials remain valid forever; GDPR Art. 32 and BSI C5 require periodic rotation

Sharing secrets between staging and production

Leak in staging compromises production; no audit separation possible

Monitoring and Drift Detection

  • wafpass check --pillar security --controls WAF-SEC-060 – checks Secrets Manager configuration

  • AWS Config Rule secretsmanager-rotation-enabled-check – checks rotation status

  • AWS Config Rule secretsmanager-using-cmk – checks CMK encryption

  • GuardDuty CredentialAccess:IAMUser/AnomalousBehavior – credential misuse

  • GitHub Advanced Security Secret Scanning – exposed keys in repositories

  • trufflehog / gitleaks in pre-commit hook – prevents commits with secrets