The question was easy to ask: what actually manages secrets on this cluster, and does the live state match the repo? Answering it meant running kubectl from CI against an EKS control plane that was private for good reason, so a hosted runner had no route, no usable DNS answer and no way in. And the query at the heart of the job was a read of every Kubernetes Secret, the one object that must never end up in a log or a build artifact.
The situation
The platform is a healthcare SaaS product on an AWS account carrying SOC 2 Type 1 and HIPAA obligations. Its EKS control plane is private and unreachable from laptops. Staging deploys through Flux GitOps: the app pipeline commits image-tag changes into the DevOps repo, and Flux syncs the cluster to that Git state. Production is a separate EC2 estate, promoted through AWS CodePipeline, CodeDeploy and GitHub Releases.
Secrets could reach the cluster through several mechanisms, and nobody had confirmed which one was live. On an account with those obligations, that isn't a question to answer from memory. It needed a repeatable check that compared what the cluster was really doing with what the repo said it should be doing.
I did this workstream alone: the CI access chain, the drift-check tooling, the leak guard, the fixtures and both pipelines. The wider access and recovery work on the same estate has its own case study; this one covers only the audit and the path CI uses to reach the cluster.
Getting a hosted runner into a private cluster
The runner needed three things before it could ask Kubernetes for anything: an identity, a route and the right address. I built the chain in that order.
Identity first. The runner mints a GitHub OIDC token and uses it to assume an AWS role. OIDC lets GitHub vouch for that specific workflow run, so AWS hands back short-lived credentials and there's no access key in the repository or the pipeline. For a job that queries Secrets, I wouldn't accept any other starting point.
Then the route. The runner installs NetBird, joins the WireGuard overlay with a setup key, and polls the client status until the VPC route appears. That wait is the part I'd defend in review. A connected VPN client doesn't mean the route is ready, and without the wait the failure shows up later as a kubectl timeout, where it looks like a cluster problem when the real cause is a route that wasn't there yet.
Then DNS. aws eks describe-cluster returns the endpoint hostname, but public DNS doesn't give this runner the private address it needs. So the job asks the VPC resolver directly, pins that answer into /etc/hosts, and only then generates a kubeconfig.
The obvious report would have leaked
With access solved, the harder problem was the report itself. The obvious approach is to fetch the Secret objects, strip .data and .stringData, and publish the rest. That's a denylist, and denylists fail when the sensitive value turns up somewhere nobody thought to remove.
Kubernetes has a specific trap here. kubectl apply can store a full copy of the object, data included, in the last-applied-configuration annotation. Strip the obvious fields and the report can still publish values from that annotation. So I didn't try to make the filter more thorough. I stopped filtering.
Building the report from an allowlist instead
The projection names the metadata it wants and builds a fresh object from exactly those fields. It never edits the fetched Secret, so value data isn't removed from the output. It's never copied into it in the first place.
An allowlist projection that rebuilds a new object
Filtering sensitive fields out of the fetched object
Why: A filter is a denylist, and it only has to miss one field once. The last-applied-configuration annotation is the field almost everyone misses. The projection names the metadata it wants and builds a fresh object from exactly those fields. Value data isn't excluded. It's simply out of reach. The downside is that adding a metadata field to the report takes an explicit code change.
The projection only helps if the raw data never lands anywhere on the way to it, so the rest of the job is built around that. The one query that touches Secrets pipes kubectl ... -o json straight into the projection; it's never redirected to a file and never echoed. The scripts run with set -euo pipefail and deliberately leave out set -x, so commands aren't traced into the log. And before anything uploads, a guard checks every output for value keys and the last-applied-configuration annotation. On any hit, it shreds the output directory and exits non-zero, so there's nothing left to upload.
A guard that fails closed and shreds its own output
A guard that flags the problem and continues
Why: A false positive costs a re-run and an investigation. A false negative puts secret values in a build artifact that has already been uploaded and summarized. Those two costs aren't close, so I made the guard over-eager on purpose.
How I know it works
A green self-test on its own says little, so the offline self-test uses fixtures built to beat the projection. They deliberately contain data, stringData, base64 payloads and a filled-in last-applied-configuration annotation. If the projection ever regressed, these fixtures would publish secrets, so the test exercises the actual failure, and it never has to point at a live cluster to do it.
I also didn't want the report to simply claim that it ran read-only. Before the audit starts, the pipeline records what its own identity can do, with kubectl auth can-i create secrets and kubectl auth can-i get secrets. The report carries what the credential was allowed to do at the moment it ran. That record matters here, because the role this job assumes isn't a read-only one (more on that below).
The result
The question "what manages secrets here, and has it drifted?" now has a repeatable answer. The report puts each candidate mechanism into one of three states:
| State | Meaning | |---|---| | active | Installed, and something is actually using it | | installed-but-unused | Controller or CRD present, zero custom resources | | absent | No controller, no CRD |
The candidates are Flux with Kubernetes Secrets, git-crypt, SOPS, External Secrets Operator, CSI Secret Store and Sealed Secrets. The middle state is the valuable one: installed-but-unused is how teams end up saying they use a secrets system that nothing has touched for a year.
On the repo side, the expected model is git-crypt. Secret material committed alongside the charts is encrypted at rest by a filter on the secret file patterns, then delivered through Flux. On the live side, the check queries CRDs and controller Deployments, counts resources only when a CRD exists, reads the Flux decryption settings, and checks every HelmRelease valuesFrom reference against the name indexes it collected. It compares all of that with a hand-maintained expected model and attaches a confidence score based on how many core queries succeeded.
The same access chain also carries a second, separate job. When a merged manifest change hasn't reached the pods and retries are exhausted, a staging-only pipeline forces Flux to re-pull and re-run the HelmRelease upgrades. Production doesn't run on this cluster and isn't reconciled by Flux. That job annotates custom resources, so it really does need write access; the audit only reads. Both run only on manual dispatch with minimal GitHub token scopes, and the audit also sits behind an environment approval gate.
What's still open
- The audit's credential isn't read-only. Both pipelines assume the same AWS role, and it has write access. The audit limits itself through workflow behaviour and the leak guard, not through the credential. A separate read-only role is the first thing I'd change.
- It's point-in-time by design. Manual dispatch makes this evidence gathering, not monitoring, and it says nothing between runs. A schedule is the second change, so drift shows up before someone remembers to ask.
- The expected model is maintained by hand. It drifts when the GitOps secret flow changes and nobody updates it.
- The overlay is now on CI's critical path. A mesh outage is a pipeline outage.