5-Step Runbook: Containing a Leaked Terraform State File

A leaked Terraform state file is an active credential exposure, not a configuration mishap. The file is readable JSON that lists your resource IDs, network topology, and the secrets Terraform recorded when it created those resources. Your first moves are to rotate every credential found in the file, cut off write access to the state backend, and only then investigate how the leak happened. This runbook covers all three phases for teams running Terraform against AWS, and it closes with the pipeline changes that stop a repeat incident.

Terraform keeps sensitive values in state and plan files even when you mark them sensitive, so a leaked state file is a live credential disclosure, not just a map of your infrastructure. HashiCorp’s documentation is explicit on this point: state and plan files can contain sensitive values such as initial database passwords or API tokens, and anyone who can access those files can access your sensitive values. Redaction with the sensitive argument only affects CLI and UI output; the underlying values stay on disk, in full, ready for anyone who obtains the file.

What the state file exposes

Local runs write terraform.tfstate as a plaintext file in your working directory, and every apply refreshes it with current resource attributes. Remote backends move the file off developer laptops, but they do not change what the file contains: passwords set at creation time, generated API tokens, TLS private keys passed as arguments, and enough topology detail to aim the next attack. Before you rotate anything, build a written inventory of these categories so nothing gets missed.

The design fix is newer than most codebases. In current Terraform, ephemeral values and write-only arguments are the only mechanisms that keep secret material out of the state file entirely: an ephemeral block holds a value only for the duration of an operation, and a write-only argument such as password_wo reaches the provider without ever being persisted to state or plan files. Migrating credential creation to these constructs is the structural change that makes future leaks smaller.

Five-step containment runbook

1. Freeze the pipeline. Disable the CI job that runs apply, and revoke or temporarily constrain the credentials it uses to reach the state backend. The goal is to stop an attacker who holds the same access from poisoning state while you respond.
2. Inventory the file. Run terraform state pull and search the JSON for password, token, secret, key and private fields. Record each credential, the resource it belongs to, and its owner. This inventory becomes your rotation checklist and your incident record.
3. Rotate everything the file recorded. Rotate database passwords, API keys, tokens and keys even if you believe the backend encrypted them at rest; encryption protects you from storage theft, not from a copy that already left. Prioritize by blast radius: cloud provider credentials first, then data store credentials, then third-party tokens.
4. Diff state against reality. Compare the leaked file with the current remote state and its version history. Look for resources you do not recognize, edited security groups, or new IAM users, which would signal the file was already used operationally. Backend access logs, such as S3 data events in CloudTrail, tell you who read or wrote state and when.
5. Scrub the leak site. Remove the file from repository history, CI artifacts, logs and any bucket it landed in. Assume a public copy was indexed and cached; scrubbing reduces future exposure but does not replace rotation in step three.

Harden the remote backend

Once containment is done, re-check the backend that stores state. For the S3 backend, HashiCorp recommends enabling the encrypt option, and the baseline underneath it is stronger than many teams assume: every new object uploaded to an S3 bucket is encrypted at rest by default with SSE-S3, so the real work is access control rather than turning encryption on. Default encryption does nothing about who can read the object, so put the state bucket behind a customer-managed KMS key whose key policy allows Decrypt only to the single role your pipeline uses for apply.

Add versioning so poisoned or deleted state can be recovered, and block public access at the bucket and account level. The same outside-in review discipline you apply to public access settings on S3, Azure Blob and GCS applies to this bucket, and stored copies of your infrastructure deserve the same treatment as cloud snapshots before they leak: a scheduled review, not a one-time check.

Remove static pipeline keys

The leak path matters as much as the file. If your CI stored a long-lived AWS access key to run Terraform, that key is a second exposure with the same blast radius as the state itself. The modern replacement is federation: GitHub Actions workflows can authenticate to AWS through OpenID Connect without storing long-lived AWS credentials as repository secrets, exchanging a short-lived token issued per job run for temporary role credentials. Keep the trust policy tight: pin the sub claim to your exact repository and branch, or to a protected environment, so a pull request from an unrelated fork cannot assume the role.

Scope the federated role to what apply actually needs: read and write on the state bucket, the lock table if you use one, decrypt on the state KMS key, and the resource permissions your configuration manages. If an attacker compromises the pipeline again, a role that can read state but cannot exfiltrate data stores or pivot across accounts turns a repeat incident into a smaller one.

Verification checklist

• Every credential in the leaked file has a rotation timestamp recorded by its owner
• State bucket has versioning enabled, public access blocked, and a customer-managed key with a minimal decrypt policy
• CI authenticates through OIDC with the subject claim pinned to one repository and branch
• New credential creation uses ephemeral variables or write-only arguments instead of persisted values
• State version history reviewed for unauthorized applies, with findings written into the incident record
• This runbook stored where on-call can reach it, with contact owners for each affected provider

Sources

• HashiCorp — Manage sensitive data in your configuration
• AWS — Setting default server-side encryption behavior for Amazon S3 buckets
• GitHub Docs — Configuring OpenID Connect in Amazon Web Services