On AWS, my client work is a healthcare estate someone else had built by hand, which I imported into Terraform and hardened in place. This one started from an empty AWS account, so I could design the whole path from nothing around three rules: no human runs kubectl apply, no AWS access key lands in a pod or a repo, and the whole environment can be destroyed and rebuilt from source.
The situation
This is a personal reference build, not client work. It has no users and no traffic, so nothing on this page is a production claim. I designed and wrote all of it, including the Terraform, the ArgoCD manifests and the External Secrets Operator wiring, and the code is open at talhaimtiaz09/eks-paved-road.
I built it because client work never lets me show the provisioning path from an empty account. On an inherited estate the cluster, the network and the access habits are already there, and the job is to change them without breaking anything. Here the question was different: if I got to decide everything up front, what would I refuse to allow in the first place?
The problem with the default path
The usual way to give a workload access to AWS is to hand it an access key. The key goes into a Kubernetes Secret or a CI variable, gets copied into each environment, and is rotated by hand when someone remembers. It leaks through logs and repos, and nothing about it expires on its own.
Moving secrets into AWS Secrets Manager doesn't remove that problem by itself. Something in the cluster still has to read from Secrets Manager, and the obvious way to let it do that is another AWS key, which is itself a secret that has to be stored, rotated and protected. The key that guards the secrets ends up being the weakest secret in the system.
The deploy path has the same shape. If people run kubectl apply by hand, the cluster's real state lives in whatever the last person ran, and a rebuild depends on someone remembering it.
What I built instead
Let the cluster vouch for the workload
I enabled the EKS OIDC provider, which lets AWS trust tokens that the cluster issues to its own service accounts. I bound External Secrets Operator's service account to a dedicated IAM role through IRSA (IAM Roles for Service Accounts). ESO assumes that role with a projected service-account token, a short-lived token Kubernetes mounts into the pod, reads the values from Secrets Manager, and writes them back as ordinary Kubernetes Secrets. There's no key anywhere in that chain to store or leak.
The part I care about most is where the boundary ended up. The application reads a plain Secret and knows nothing about AWS, so swapping the secret backend later is an operator change, not an application change.
IRSA + External Secrets Operator (keyless)
Injecting static AWS access keys into pods via Kubernetes Secrets
Why: A static key is a standing liability. It has to be stored somewhere, it spreads across environments, and it leaks through logs and repos. Rotating it is a manual event. IRSA replaces it with a short-lived token that AWS trusts through OIDC. The credential only lives for the length of a request and never sits in Git or the cluster. ESO then delivers the secret values as native K8s Secrets, so the app never has to know about AWS.
Give each layer one job
Around that identity path the platform has three layers. Terraform owns the cloud: the VPC, the EKS cluster and the IAM roles, including the OIDC provider and the role IRSA hands to workloads. ArgoCD owns the cluster's desired state. It watches Git and applies an app-of-apps hierarchy with Kustomize overlays per environment, so every deploy is a Git commit and nobody runs kubectl apply. ESO sits between the cluster and Secrets Manager.
The important boundary is between Terraform and ArgoCD. Terraform stops at the edge of the cluster and ArgoCD takes over inside it, and nothing manual bridges the two.
Keep the two control loops apart
I reached that boundary by watching the system run. Infrastructure state and application state drift independently, and a green ArgoCD sync doesn't mean the workload is healthy. Secret delivery also has to finish before a workload starts, not alongside it. Each layer has its own idea of "correct" and its own way of recovering, and forcing them through one pipeline would have made the setup fragile.
Keep Terraform and ArgoCD as two separate control loops
One combined provision-and-deploy pipeline
Why: Infrastructure and workloads really do run on different control loops. Terraform orders its work by resource dependencies. ArgoCD orders by sync waves and keeps reconciling against Git. Provisioning converges once. Workload state is reconciled forever and has to heal itself when it drifts. One pipeline would tie together two problems that fail differently and recover differently. It would add operational complexity and buy nothing.
How I know it works
There's no traffic to load-test, so the useful questions were whether the platform comes back the same way every time it's destroyed, and whether it heals when the live state drifts from Git. I tested three things an on-call engineer would eventually run into.
I tore the whole environment down and rebuilt it from Terraform plus Git, repeatedly, and checked that each rebuild landed in the same state instead of drifting from run to run. I rotated a value in AWS Secrets Manager and checked that ESO pushed the new value into the Kubernetes Secret, and that the app picked it up through the normal Secret, with no manual step. And I deleted live resources out from under ArgoCD to see whether reconciliation noticed, which it did, restoring them to the state declared in Git.
The result
None of these are performance numbers, and they aren't meant to be. They show the three rules holding and the control loops doing what the design says they should when something goes wrong.
What I'd harden before real traffic
A reference build skips the operational hardening that real traffic forces. If this were about to carry load, this is the order I'd work in:
- Tighter IRSA scoping. The ESO role is scoped enough to show the pattern. A real one needs least privilege per namespace and per secret, so a compromised operator can't read everything.
- Secret-rotation blast radius. Rotation propagates cleanly, but before trusting it under load I'd map exactly which workloads restart or reconnect when a secret rotates.
- ArgoCD drift alerting. Self-healing happens silently today. I'd want an alert whenever ArgoCD corrects drift, because a resource that keeps getting deleted is a signal worth seeing.
- Cluster autoscaling. There's none yet. With real workloads I'd add node and workload autoscaling before capacity becomes the incident.
- Backup and restore cadence. Reliable rebuilds cover most of disaster recovery, but I'd still formalize backups of Terraform state and any stateful data, with a real restore schedule, instead of relying on rebuild-from-source alone.