The estate had been built by hand before I joined, and it was already serving patients. Its Kubernetes API answered on the public internet, its worker nodes sat in public subnets, and two SSH bastions let in anyone coming from an allowlisted address. There was no rebuild on the table and no maintenance window, so whatever I changed had to land underneath the engineers and pipelines already using it.
The situation
The client runs a healthcare SaaS platform on an AWS account that carries SOC 2 Type 1 and HIPAA obligations. Production and pre-production ran on EC2 behind the two bastions, and staging ran on EKS with Flux. CI deployed with static IAM keys, recovery stopped at the region boundary, and an existing managed security tooling stack supplied part of the guardrail baseline. Health records can't be un-disclosed, so a bad access decision here can't be rolled back the way a dashboard outage can.
I did the infrastructure work alone: importing the estate into Terraform and hardening it there, the overlay routing, the identity migration, the access model, disaster recovery and the compliance documentation. The product team owned the application and the frontend.
What I saw
The obvious problem was that the admin paths were public. The bigger problem was that every control on them checked an address or a key, never a person.
An SSH allowlist trusts an IP range and says nothing about who is at the keyboard or whether they should still have access. The static CI keys, one of them cluster-admin, had the same weakness: they kept working until someone remembered to rotate or remove them. Cluster access lived in aws-auth, the ConfigMap EKS reads to map IAM identities to Kubernetes permissions, and because it was edited by hand, a typo could become a privilege grant.
So tighter IP rules weren't the goal. I wanted every path tied to an identity: people through SSO and overlay membership, CI through a short-lived workflow identity, and the cluster through grants that are logged and can be reviewed. The catch was that the estate was live, and the tidiest version of almost every control would have broken something already running.
What I did
Remove the public paths instead of narrowing them
The EKS cluster already existed, so I imported it into Terraform instead of rebuilding it. From there I made the API endpoint private, moved the nodes behind NAT, and emptied the ingress rules on both bastion security groups. Empty was deliberate: a narrow SSH rule still has to be audited and trusted, and its source range can go stale, while an empty list has nothing to go stale.
Engineers still needed a way in, so admin access moved onto a NetBird overlay built on WireGuard. The overlay reaches private addresses through routing peers, which are overlay members that advertise the VPC range so other peers can route through them. For staging a routing peer runs inside the cluster, and for production and pre-production the bastions that had just lost their public ingress became the routing peers.
NetBird
Tailscale
Why: Both solved the routing problem, and Tailscale was the better-known option. On this account's usage, though, the price difference was material. NetBird's free cloud tier covered what we needed. Price at this scale decided it, not features.
Take CI and people off standing credentials
The static keys could go because Flux already deployed the apps from the DevOps Git repo. The app pipeline built the image and committed the new tag to that repo, and Flux pulled the cluster back in line with Git. CI never needed a cluster credential at all, let alone a permanent cluster-admin one.
So I deactivated both static IAM keys outright instead of rotating them, and moved CI to GitHub OIDC: each workflow run gets a short-lived federated token that maps to an IAM role, and the role maps to a namespaced EKS access entry. For people, I set up IAM Identity Center with three permission sets and MFA on sign-in, replacing shared credentials, and moved cluster permissions from aws-auth to EKS access entries.
EKS access entries with namespaced RBAC
The hand-edited aws-auth ConfigMap
Why: The ConfigMap is one YAML file where a typo becomes a privilege grant. It keeps no audit trail of its own. On a cluster nobody has had time to think about, its default state is cluster-admin for everyone. Access entries move each grant into IAM, where it is logged and can be reviewed. The downside was two systems to understand during the migration.
Don't let the new control become the outage
VPN-only access was the right default, but it created a new dependency. Adding someone to the VPN needed a VPN administrator, so during an incident an engineer with valid AWS access could be locked out of the cluster just because the person who could enroll them wasn't available.
So I added an SSM bastion inside the private VPC, with no public IP and an empty ingress list. The instance dials out to AWS Systems Manager, and getting in is an IAM-authenticated API call, not a network connection. Its cluster role is read-only. The cost is real: it's a second path to audit, and it's convenient enough that it has to be kept from becoming the default.
Fit the remaining controls to the estate
The account-level hardening was where the textbook answer kept being the wrong one.
The AWS-managed EBS key
A customer-managed key
Why: A customer-managed key (CMK) needs explicit KMS grants for the EKS node role and the EC2 Auto Scaling service-linked role. Without them, new nodes fail to launch. That shows up later, during a scale-up or a node replacement, not during the change. The managed key looks weaker on a checklist, but it can't fail that way. The tradeoff is weaker key custody. I deferred the CMK migration until the prerequisite grants are in place.
Per-bucket public-access blocking on verified-private buckets
Account-level Block Public Access
Why: Account-level blocking overrides bucket policy, and one bucket is public by design. The one-setting option would have taken a working feature offline to satisfy a control. I first confirmed each bucket was not public, then blocked them one by one. No effective access changed. The tradeoff is seven resources to maintain instead of one, and account-level blocking is still deferred.
The rest followed the same rule. I turned on encryption by default only where it wouldn't change running volumes as a side effect, and handled existing unencrypted volumes by recycling node groups as a scheduled operation. I also turned on control-plane audit logging and extended threat detection to a region that had been left out. At close-out I kept what I built separate from what the managed security tooling already supplied, because verifying a control someone else built is a weaker claim than building it, and the honest one.
Add cross-region recovery, and publish the real number
The last gap was losing the region itself; point-in-time recovery only existed inside it. I designed hourly snapshots, copied cross-region into a vault encrypted with a separate customer-managed key, and set six production buckets to replicate cross-region, with a batch job to backfill their existing objects.
The backfill skipped one bucket. That bucket already replicated to another destination, so S3 had marked its objects COMPLETED, meaning complete somewhere, just not at the new destination. My job only picked up objects whose status was unset or failed, so it passed over all of them, and backfilling a new destination meant adding COMPLETED to the filter.
Then I measured the copy itself. There was a 40–50 minute lag between a snapshot completing and its copy landing in the second region, so the real recovery point is roughly 45 minutes to just under two hours, depending on where in the cycle the region fails.
Publishing a recovery point of up to ~1h50m
Publishing the ~1h target the design was scoped against
Why: The snapshots were hourly, the ticket said an hour, and every dashboard was green. I could have left the wider number out. A true seconds-level cross-region RPO needs a globally replicated database. That was scoped and costed, then deferred as a spending decision. An accepted posture with a real number behind it holds up. A target with an unmeasured number behind it is an audit finding waiting to happen.
How I know it works
I checked the access work by failing to connect, not by reading policies.
| Path | Off the overlay | On the overlay |
|---|---|---|
| Bastion SSH, production and pre-production | Times out | Connects |
| Kubernetes API | No reachable endpoint | kubectl works |
A screenshot of a security group only shows that a rule looked right when it was taken. A timeout from outside the overlay shows that the route doesn't exist.
The result
Every way into the admin surface now belongs to a person or a pipeline. Engineers come in through SSO and the overlay, CI through a token that lives for one workflow run, and cluster grants sit in IAM where they're logged. None of it needed a rebuild or a maintenance window.
The same idea closed one last grant. Developers had been given namespace access so they could port-forward to staging Postgres and dump realistic data to their laptops. I moved that job into a Lambda with its own role that writes the dump to S3, so developers now restore with a single read permission and kubectl is gone from the setup instructions. Move the capability, not the credential.
I don't have a cost figure or an incident-reduction figure. Price was part of why I chose the overlay, but I never produced a billing comparison, and the controls had no incident baseline to measure against.
What's still open
- Overlay activity logs sit apart. They live in the provider's dashboard and API, not in the same S3 and SIEM path as the rest of the compliance evidence.
- The quarterly access review is written down but hasn't run. Nobody knows how a process fails until someone runs it.
- Human identity is a stopgap. SSO users are created natively instead of federated from the company directory, so joiners, movers and leavers are still partly manual.
- The wider recovery point is accepted, not fixed. I measured the copy lag, but nothing monitors it.
- The customer-managed EBS key is deferred. I wrote down the prerequisite grants so nobody retries the migration blindly.
- The overlay is on the critical path for admin access. The SSM fallback lowers that risk but doesn't replace a break-glass procedure.