The cluster was private for the right reason, but that also meant a hosted CI runner had no route, no usable DNS answer, and no way to run kubectl. The job I needed to run was the one most likely to leak if handled carelessly: an inventory of every Kubernetes Secret. I built the access path into the private cluster, then built the report so secret values could not enter the output in the first place.
Ownership. Sole on this workstream: the CI access chain, drift-check tooling, leak guard, fixtures and both pipelines. This sits beside a larger infrastructure estate owned by other engineers; this write-up covers only the work I built.
A healthcare SaaS platform on an AWS account carrying SOC 2 Type 1 and HIPAA obligations · a private EKS control plane, unreachable from laptops · staging deploys through Flux GitOps: the app pipeline commits image-tag changes into the DevOps repo, and Flux reconciles the cluster to that Git state · production is a separate EC2 estate promoted through AWS CodePipeline, CodeDeploy and GitHub Releases · secrets could arrive through several mechanisms, with no confirmed live answer
The CI access path
The runner had to prove identity, join the private route, resolve the private endpoint, and only then ask Kubernetes for anything.
Per-run AWS identity, not a stored key
The runner mints a GitHub OIDC token and assumes an AWS role. No access key is stored in the repository or pipeline. That is the only acceptable starting point for a job that will query Secrets.
Join the overlay, then wait for the route
The runner installs NetBird, joins with a setup key, and polls the client status until the VPC route appears. That wait is the part I would defend in review. Without it, the failure shows up later as a kubectl timeout and looks like a cluster problem instead of an incomplete route.
Resolve the private endpoint deliberately
aws eks describe-cluster returns the endpoint hostname, but public DNS does not return the private address the runner needs. The job asks the VPC resolver directly, takes the private answer, pins it into /etc/hosts, and only then generates a kubeconfig.
The audit that must not leak
The obvious implementation is dangerous: fetch Secret objects, strip .data and .stringData, then publish the rest. That is a denylist. Denylists fail when the sensitive value appears somewhere you did not think to remove.
Kubernetes has a specific trap here. kubectl apply can store a complete copy of the object, including data, in the last-applied-configuration annotation. Strip the obvious fields and the report can still publish values from the annotation.
An allowlist projection that rebuilds a new object
Filtering sensitive fields out of the fetched object
Why: a filter is a denylist, and it only has to miss one field once — the last-applied-configuration annotation being the field almost everyone misses. The projection names the metadata it wants and constructs a fresh object from exactly those fields, so value data is not excluded, it is unreachable. The cost: adding a new metadata field to the report is an explicit code change.
Three rules make the projection hold end to end:
- Raw secret JSON never lands. The single query that touches Secrets pipes
kubectl ... -o jsonstraight into the projection. It is never redirected to a file and never echoed. - Read commands stay quiet. The scripts use
set -euo pipefailand deliberately do not useset -x. - Publication fails closed. Before anything uploads, the guard checks every output for value keys and the
last-applied-configurationannotation. On any hit, it shreds the output directory and exits non-zero.
A guard that fails closed and shreds its own output
A guard that flags the problem and continues
Why: a false positive costs a re-run and an investigation. A false negative costs secret values in a build artifact that has already been uploaded and summarized. The asymmetry is not close, so the guard is deliberately over-eager.
Fixtures that try to break it
The offline self-test uses fixtures that intentionally contain data, stringData, base64 payloads and a populated last-applied-configuration annotation.
The useful property is not that the self-test is green. It is that the fixture would publish secrets if the projection were wrong. That gives the test evidence value without pointing the code path at a live cluster.
What the report answers
The report answers two audit questions: what is the source of truth for secrets, and does the live cluster differ from the repo?
It classifies each candidate mechanism into three states:
| State | Meaning | |---|---| | active | Installed, and something is actually using it | | installed-but-unused | Controller or CRD present, zero custom resources | | absent | No controller, no CRD |
The candidate mechanisms are Flux with Kubernetes Secrets, git-crypt, SOPS, External Secrets Operator, CSI Secret Store and Sealed Secrets. The middle state is the one worth keeping: installed-but-unused is how teams end up saying they use a secrets system that nothing has touched for a year.
The repo-side expected model is git-crypt: secret material committed alongside charts is encrypted at rest by a filter on the secret file patterns, then delivered through Flux. The live-side check queries CRDs, controller Deployments, resource counts only when a CRD exists, Flux decryption settings, and every HelmRelease valuesFrom reference against collected name indexes. The output is compared with a hand-maintained expected model and includes a confidence score based on how many core queries succeeded.
Posture recorded, not asserted
Before the audit runs, the pipeline records what its own identity can do: kubectl auth can-i create secrets and kubectl auth can-i get secrets.
That changes the evidence. The report no longer just says the run was read-only; it records what the credential was permitted to do at the moment it ran.
Same access chain, different privilege
A second pipeline reuses the same private-cluster access path for a different job: forcing Flux to re-pull and re-run staging HelmRelease upgrades when a merged manifest change has not reached the pods and retries are exhausted.
This path is staging-only. Production does not run on this cluster and is not reconciled by Flux; it is a separate EC2 estate delivered through AWS CodePipeline and CodeDeploy, promoted by GitHub Releases. The reconcile job annotates custom resources, so it genuinely needs write access. The audit only reads. Both are manual-dispatch only, both use minimal GitHub token scopes, and the audit sits behind an environment approval gate.
What remains weak
- Both pipelines assume the same AWS role, and it has write access. The audit constrains itself by workflow behavior and leak guards, not by the credential. A separate read-only role is the first fix.
- The expected model is hand-maintained. It drifts when the GitOps secret flow changes and nobody updates it.
- The overlay is now on CI's critical path. A mesh outage is a pipeline outage.
- Point-in-time by design. Manual dispatch makes this evidence gathering, not monitoring. It says nothing between runs.
What changed
- CI can operate on a private cluster using a per-run federated identity instead of a stored credential.
- The question "what manages secrets here, and has it drifted?" has a repeatable answer.
- Six candidate mechanisms are classified as active, installed-but-unused or absent.
- Secret values are structurally unreachable in the output, and the offline fixtures would publish them if that stopped being true.
- The pipeline records what its own credential could do at run time, so the read-only posture is shown rather than merely asserted.
No before-and-after number is claimed. A drift count from one manual run is not a trend, and this is a control path rather than an optimization. Next changes, in order: a genuinely read-only role, then a schedule so drift can surface before someone remembers to ask.