Kubernetes Secrets: etcd Encryption, RBAC and Vault

Kubernetes secrets are only as protected as the three layers underneath them: etcd storage, API permissions, and delivery to the pod. By default, the Kubernetes API server stores Secrets as plain text in etcd, with no at-rest encryption, so anyone who can read an etcd backup, a leaked snapshot, or a stolen node disk can read every credential in the cluster. The practical fix stacks three controls: encryption at rest with the KMS v2 provider, RBAC rules that stop pod-create rights from leaking entire namespaces, and an external vault for credentials that need rotation, short leases, and audit trails. The logic mirrors what session lifetime controls do against token theft: reduce the value of whatever an attacker manages to extract.

Why etcd encryption comes first

RBAC and network policy do nothing against an attacker who holds an etcd snapshot or physical access to a control-plane host. Encryption at rest is the only layer that protects Secret objects in that scenario, and it is not on by default. The kube-apiserver only encrypts API data when it is started with the --encryption-provider-config flag pointing at a valid EncryptionConfiguration file, and only when a provider other than identity is listed first. According to the upstream encryption task guide, the default identity provider provides no confidentiality protection at all.

One detail teams miss: enabling encryption does not retroactively protect existing objects. Secrets already written to etcd stay in their original encoding until they are written again, which is why the migration step matters as much as the configuration itself. After enabling a real provider, force a rewrite of every Secret with kubectl get secrets --all-namespaces -o json | kubectl replace -f - so old plain-text copies are replaced by encrypted ones.

Choose the encryption provider

Not every provider is worth running in production. The documentation is blunt about the trade-offs: AES-GCM keys must be rotated every 200,000 writes, which is why the aesgcm provider is only recommended when an automated rotation scheme exists. The aescbc provider is marked weak because CBC mode is vulnerable to padding oracle attacks. The strongest option, kms v2, uses envelope encryption with a data encryption key per API server, while the key encryption key lives in an external Key Management Service outside the cluster; kms v1 has been deprecated since Kubernetes v1.28.

Key rotation is a first-class operational task, not an afterthought. The documented zero-downtime sequence is: add the new key as the second entry, restart every kube-apiserver, make a secure backup, promote the new key to first position, restart again, re-encrypt all Secrets with the rewrite command, and only then remove the old key. If you lose every copy of a key, the only recourse for resources encrypted under it is deleting the entries directly from etcd, so back up the key material before each rotation.

RBAC limits secret exposure

Encryption at rest stops disk-level reads; it does not stop API-level reads. The Kubernetes Secrets documentation warns that anyone with API access can retrieve or modify a Secret. Worse, because of how the API works, anyone authorized to create a Pod in a namespace can read every Secret in that same namespace, including indirectly through a Deployment. Namespace-wide blast radius is the default, not the exception.

Three RBAC decisions close most of that gap. First, avoid granting list or watch on Secrets: those verbs let a subject read all Secret data in the namespace, not just the objects its pods reference — restrict bindings to get on named Secrets where possible. Second, reserve broad roles such as cluster-admin for genuine administrative use, and keep workloads in separate namespaces so one compromised app cannot enumerate its neighbors’ credentials. Third, remember what the node itself does: the kubelet stores mounted Secrets in tmpfs rather than durable storage and deletes its local copy when the consuming pod goes away — but any container running with privileged: true on a node can access all Secrets used on that node, which makes pod security context part of your secrets architecture.

When external vaults win

Native Secrets are static objects: they hold whatever you put in them until you change them. For database credentials that should expire, cloud keys that should be issued per workload, or certificates that need centralized policy, an external secrets store is the better primitive. Vault centralizes secret management, rotates old credentials, and generates credentials on demand, with an audit log of every client interaction — capabilities the HashiCorp Vault documentation positions as the core of the product. Cloud-native stores such as AWS Secrets Manager, GCP Secret Manager, and Azure Key Vault serve the same role inside their platforms.

The standard delivery mechanism is the Secrets Store CSI driver. A SecretProviderClass declares which external objects a pod needs, and on pod start the driver fetches the content over gRPC and mounts it as a tmpfs volume that is cleaned up when the pod is deleted. The blast-radius argument is real: instead of an ingress controller holding cluster-wide read access to every TLS Secret, it receives only the certificates it actually mounts. Know the cost: operationally, the Secrets Store CSI driver runs as a root, privileged DaemonSet on every node, because creating tmpfs mounts inside other pods’ filesystems requires the mount syscall and bidirectional mount propagation. Provider plugins also run as root, and on Windows hosts secret material can land on the node’s persistent filesystem — factor that into host hardening.

Rotation discipline transfers across domains: the same reasoning that pushes teams toward 47-day TLS certificate lifetimes pushes databases toward short-lived, vault-issued credentials instead of long-lived passwords in YAML.

A practical hardening checklist

1. Verify the current state: an API server running without --encryption-provider-config, or with identity listed first, has no confidentiality at the storage layer.
2. Configure EncryptionConfiguration with kms v2 first and identity last for the migration window, then restart API servers one at a time.
3. Re-encrypt existing Secrets with kubectl get secrets --all-namespaces -o json | kubectl replace -f - and confirm no plain-text copies remain.
4. Rotate keys on a schedule: add, restart, back up, promote, restart, rewrite, then remove the old key.
5. Audit RBAC for list/watch on Secrets and broad pod-create grants per namespace; narrow both to named resources.
6. Move high-value credentials to an external vault delivered through the CSI driver, with per-workload identity instead of shared tokens.
7. Treat privileged pods and Windows nodes as secrets-boundary exceptions and monitor them accordingly.

Sources

• Kubernetes – Encrypting Confidential Data at Rest
• Kubernetes – Secrets
• Secrets Store CSI Driver – Concepts
• HashiCorp – Vault Documentation