Service spotlight: wiring automatic rotation into Secrets Manager

Storing a database password in Secrets Manager instead of a .env file is the easy part. The part that actually earns the “we rotate credentials” line in a security questionnaire is automatic rotation — and that’s where most setups I’ve reviewed stop halfway, with a secret stored but never actually rotated because nobody wired up the rotation Lambda.

What it actually is

Secrets Manager stores secrets encrypted with KMS and, for the services that matter most, can rotate them on a schedule automatically via a rotation Lambda function that Secrets Manager invokes. For RDS, Aurora, DocumentDB, and Redshift, AWS provides pre-built rotation Lambda templates (deployed via a SAM app from the Secrets Manager console or CLI) that implement the standard four-step rotation process without you writing the logic yourself:

1. createSecret — generate a new password and stage it as AWSPENDING
2. setSecret — set that new password on the actual database
3. testSecret — verify the new credential actually authenticates
4. finishSecret — promote AWSPENDING to AWSCURRENT, completing the rotation

That staged, four-step model exists specifically so a failure partway through doesn’t lock you out: AWSCURRENT only moves to the new password after testSecret confirms it works, so a broken setSecret step leaves the database still accepting the old, still-AWSCURRENT password rather than stranding the app with a password nothing accepts.

Where it earns its keep

• No credential ever needs to be manually rotated by a human again once it’s wired up — the whole point of rotation is removing “someone remembers to change the password quarterly” from the list of things that depend on human follow-through, which is also the thing that reliably doesn’t happen.
• Application code stays credential-agnostic. Apps fetch the current secret value via GetSecretValue at connection time (ideally through the Secrets Manager RDS/JDBC connector libraries, which handle the AWSCURRENT/AWSPENDING transition transparently) instead of having a password baked into config, so a rotation doesn’t require an app deploy — as long as the app re-fetches rather than caching the credential for its entire process lifetime.
• Multi-user rotation strategy for zero-downtime cutover. For workloads that can’t tolerate any connection using a stale credential during rotation, Secrets Manager supports an alternating-user rotation strategy — two database users, rotation alternates which one is AWSCURRENT, so old connections using the previous user keep working until they naturally cycle rather than being cut off mid-rotation.

Where it goes wrong in practice

The most common failure isn’t rotation itself — it’s **rotation never running successfully because the Lambda can’t reach the database**. The rotation Lambda needs network access to the database (correct VPC subnets, security group rules allowing it in) and the Secrets Manager VPC endpoint if the Lambda runs without internet egress; get either wrong and rotation fails silently on schedule, over and over, until someone notices the secret’s LastRotatedDate hasn’t moved in months. Alarm on rotation failures explicitly (CloudWatch metric filter on the rotation Lambda’s error logs, or EventBridge on RotationFailed) rather than assuming “I set a rotation schedule” means it’s actually rotating.

The other gap: rotation changes the secret in Secrets Manager and on the database, but doesn’t retroactively fix every place a credential might be cached — a long-lived connection pool that doesn’t recycle connections, or a sidecar that read the secret once at container start and never again, keeps using the old credential until it happens to reconnect. Rotation strategy has to account for how long-lived your actual connections are, not just how often the schedule fires.

A practical tip

Set the rotation schedule’s testing in a non-production secret first, and deliberately break setSecret (point it at a nonexistent user, say) to confirm the four-step staging really does leave AWSCURRENT untouched on failure before you trust it against a production database credential — verifying the failure mode is safe is worth the ten minutes it takes.