Skip to main content
6 min read·1,121 words

Provisioning Workload TLS Credentials with Pod Certificates

Workloads that speak TLS or mutual TLS need a private key and a certificate chain. The traditional pattern is to generate that key material out of band, store it in a Secret, and mount it into the pod — which leaves long-lived private keys sitting in etcd, copied across namespaces, and rotated by hand. The podCertificate projected volume source, together with the ClusterTrustBundle object, moves that responsibility into the kubelet: the node generates the key locally, requests a certificate from a named signer, writes the credential bundle into the pod, and refreshes it automatically before it expires.

Both the podCertificate and clusterTrustBundle projected volume sources are Stable since Kubernetes v1.37 and enabled by default. The podCertificate source was first available in the v1.34 release; the clusterTrustBundle source in the v1.29 release; the ClusterTrustBundle object was first available in v1.27.

1. Replace Long-Lived Secret-Mounted Keys with Kubelet-Issued Certificates​

Before: A private key and certificate are generated manually, committed to a Secret, and mounted into the pod. The key lives in etcd for its full validity period, is readable by anyone with get secrets on the namespace, and is rotated only when an operator remembers to.
After: The kubelet generates the private key on the node and never writes it to the API server. It creates a PodCertificateRequest addressed to a signer, and once the signer issues the certificate the kubelet writes the key and chain into the pod filesystem, refreshing them as expiration approaches.

Per the Kubernetes documentation, a podCertificate projected volume source "securely provisions a private key and X.509 certificate chain for pod to use as client or server credentials. Kubelet will then handle refreshing the private key and certificate chain when they get close to expiration."

Example Configuration​

apiVersion: v1
kind: Pod
metadata:
  name: tls-workload
spec:
  serviceAccountName: default
  containers:
    - name: app
      image: busybox:1.28
      command: ["sleep", "3600"]
      volumeMounts:
        - name: tls-creds
          mountPath: /var/run/my-x509-credentials
          readOnly: true
  volumes:
    - name: tls-creds
      projected:
        sources:
          - podCertificate:
              keyType: ED25519
              signerName: coolcert.example.com/foo
              credentialBundlePath: credentialbundle.pem

The kubelet writes /var/run/my-x509-credentials/credentialbundle.pem. The application must reload the file when it changes — for example with inotify or polling — because the kubelet rotates it in place.

2. Choose the Field Layout That Matches Your Consumer​

Issue: Splitting the private key and certificate chain into separate files can produce a mismatched pair during rotation.
Fix: Prefer credentialBundlePath — a single PEM file the kubelet writes atomically — unless your consumer strictly requires separate files.

Each podCertificate projection supports the following fields:

  • signerName — the signer you want to issue the certificate. Signers may have their own access requirements and may refuse to issue certificates to your pod.
  • keyType — the type of private key the kubelet generates. Valid values are ED25519, ECDSAP256, ECDSAP384, ECDSAP521, RSA3072, and RSA4096.
  • maxExpirationSeconds — the maximum certificate lifetime you will accept. If unset it defaults to 86400 (24 hours). It must be at least 3600 (1 hour) and at most 7862400 (91 days). Kubernetes built-in signers are restricted to a maximum lifetime of 86400 (1 day); a signer may issue a shorter lifetime than requested.
  • credentialBundlePath — a single PEM file where the first block is a PRIVATE KEY block containing a PKCS#8-serialized private key, followed by CERTIFICATE blocks that form the chain (leaf plus any intermediates).
  • keyPath and certificateChainPath — separate paths for just the private key or just the certificate chain.

The documentation is explicit about the trade-off: "Most applications should prefer using credentialBundlePath unless they need the key and certificates in separate files for compatibility reasons." The kubelet uses an atomic symlink-based write for the bundle, so a reader always sees either the old or the new content. When the key and chain are read from separate files, the kubelet may rotate between the two reads and hand the application a mismatched key and certificate.

3. Distribute Trust Anchors with ClusterTrustBundle​

Before: Each workload ships its own copy of the CA roots it must trust, baked into an image or a ConfigMap and updated independently.
After: A ClusterTrustBundle holds the trust anchors centrally, and the clusterTrustBundle projected volume source injects them as an automatically updating file.

A ClusterTrustBundle is a cluster-scoped object for distributing X.509 trust anchors (root certificates) to workloads. Its trustBundle field "must contain one or more X.509 certificates, DER-serialized, each wrapped in a PEM CERTIFICATE block," and each certificate must parse as a valid X.509 certificate.

A signer-linked bundle sets spec.signerName and takes a name derived from that signer:

apiVersion: certificates.k8s.io/v1
kind: ClusterTrustBundle
metadata:
  name: example.com:mysigner:foo
spec:
  signerName: example.com/mysigner
  trustBundle: "<... PEM data ...>"

To create or update a signer-linked ClusterTrustBundle, the caller must be permitted to attest on the signer (a custom authorization verb in the certificates.k8s.io API group). Signer-unlinked bundles leave spec.signerName empty, carry no attest requirement, and are intended for cluster-configuration use cases.

Consuming the Bundle in a Pod​

volumes:
  - name: tls
    projected:
      sources:
        - podCertificate:
            keyType: ED25519
            signerName: coolcert.example.com/foo
            credentialBundlePath: credentialbundle.pem
        - clusterTrustBundle:
            signerName: coolcert.example.com/foo
            path: signer-roots.pem

The clusterTrustBundle source selects bundles either by name (a single object) or by signerName with an optional labelSelector. The path field is required. By default the kubelet prevents the pod from starting if the named ClusterTrustBundle is not found, or if signerName/labelSelector match nothing; set optional: true to start with an empty file instead. The kubelet deduplicates and normalizes the certificates and keeps the file current as the selected bundles change.

4. Rely on Node Restriction to Scope Requests​

Issue: A pluggable certificate-issuance mechanism could let a compromised node request certificates for workloads it does not run, impersonating them.
Fix: Keep the NodeRestriction admission plugin enabled. It confines each node to certificate requests for its own pods.

Nodes automatically receive permission to create PodCertificateRequest objects and to read the ones related to them, as determined by the spec.nodeName field. On top of that, per the documentation, "the NodeRestriction admission plugin, if enabled, ensures that nodes can only create PodCertificateRequests that correspond to a real pod that is currently running on the node."

Two further properties bound the workflow:

  • After creation, the spec of a PodCertificateRequest is immutable.
  • Unlike CertificateSigningRequest objects, PodCertificateRequest objects have no approval phase. Once created, the signer's controller directly decides to issue, deny, or fail the request.

Because issuance is driven entirely by the signer's controller, this feature requires a signer for the signerName you reference — the certificate is only as trustworthy as that signer.

5. Audit Existing Usage and Verify Issuance​

Enumerate the trust bundles and outstanding certificate requests in the cluster:

kubectl get clustertrustbundles
kubectl get podcertificaterequests --all-namespaces

Find every pod that consumes a podCertificate projected volume:

kubectl get pods --all-namespaces -o json \
  | jq -r '.items[]
      | select(any(.spec.volumes[]?; .projected.sources[]?.podCertificate != null))
      | "\(.metadata.namespace)/\(.metadata.name)"'

After deploying the example, confirm the kubelet projected the bundle and that the first block is the private key:

kubectl exec tls-workload -- ls -l /var/run/my-x509-credentials/
kubectl exec tls-workload -- head -1 /var/run/my-x509-credentials/credentialbundle.pem

A correctly issued bundle shows credentialbundle.pem in the directory, and its first line is -----BEGIN PRIVATE KEY-----. Cross-check that the request was issued rather than left pending:

kubectl get podcertificaterequests --all-namespaces

Version Skew and Scope​

  • The podCertificate and clusterTrustBundle projected volume sources are GA and on by default only from Kubernetes v1.37. On clusters older than the release that introduced each stage — and on managed offerings, which frequently trail upstream by several minor versions — the sources may be absent or gated behind a feature gate and disabled. Confirm the feature state on your specific cluster version before relying on it.
  • This mechanism issues workload certificates for pods. It does not replace user client certificates (issued through the CertificateSigningRequest API) or the control-plane PKI.
  • Kubernetes built-in signers cap the certificate lifetime at one day. Longer lifetimes (up to 91 days) require a signer that permits them.
Recommended Reading
Learning Kubernetes Security
Learning Kubernetes Security
by Raul Lapaz
Its chapters on authentication and secrets management give the wider identity context for replacing Secret-mounted TLS keys with the kubelet-issued, auto-rotating pod certificates covered here.
View on Amazon
As an Amazon Associate, I earn from qualifying purchases.

References​

This article is based on information from the following official sources:

  1. Projected Volumes - Kubernetes Documentation
  2. Certificates and Certificate Signing Requests - Kubernetes Documentation