The estate did not need a cleaner checklist. It needed public administrative paths closed without breaking the people and pipelines already using them. The system carried SOC 2 Type 1 and HIPAA obligations, served patients, and had been built by hand: a public EKS API, public worker nodes, two SSH bastions, static CI keys, hand-edited cluster grants and no cross-region recovery path. I closed the paths in place, removed the standing credentials, and kept the write-up honest about the controls I deferred.
Ownership. Sole on the infrastructure work: Terraform import and hardening, overlay routing, identity migration, access model, disaster recovery and compliance evidence. The product team owned the application and frontend. This page describes infrastructure and access work only.
A healthcare SaaS platform on an AWS account carrying SOC 2 Type 1 and HIPAA obligations · production and pre-production on EC2 behind bastions · staging on EKS with Flux · hand-provisioned before the engagement · an existing managed security tooling stack already supplying part of the guardrail baseline
Health records cannot be un-disclosed. That is the operating constraint that matters here: a bad access decision is not a reversible dashboard outage.
From public administrative surface to identified access paths
What was actually risky
The visible problem was public reachability. The larger problem was that the controls described the wrong thing.
An SSH allowlist is a guess about trust expressed as an address range. It says nothing about the person at the keyboard, whether the address still belongs to them, or whether the grant should still exist. Static CI keys had the same failure shape: they worked until someone remembered to rotate or remove them. Cluster access lived in aws-auth, a single hand-edited ConfigMap where a typo is a privilege grant.
The target state was not "more restrictive IP rules." It was identified access: humans through SSO and overlay membership, CI through short-lived workflow identity, and the cluster through grants that are logged and reviewable.
The access rebuild
The access model
The work is easier to trust when the control plane, human access, CI access and fallback path are separated rather than described as one VPN migration.
Make public paths disappear
The EKS cluster already existed, so I imported it into Terraform instead of rebuilding it. From there the API endpoint went private, nodes moved behind NAT, and both bastion security groups had ingress emptied. A narrow SSH rule still has to be audited and trusted. An empty ingress list has no source range to stale out.
For staging, a routing peer runs inside the cluster and advertises the VPC range to the overlay. For production and pre-production, the bastions themselves became routing peers after their public ingress was removed.
Move grants out of static keys and hand-edited YAML
Flux already reconciled application deploys from the DevOps Git repo, so CI did not need standing cluster credentials. The application pipeline built the image, committed the new image tag into the DevOps repo, and Flux reconciled the cluster back to that Git state. CI needed to push an image and update Git; it did not need a permanent cluster-admin key. Both static IAM keys were deactivated rather than rotated, and CI moved to GitHub OIDC: a short-lived federated token per workflow run, mapped to a role, mapped to a namespaced access entry.
For humans, IAM Identity Center replaced shared credentials with three permission sets and MFA on sign-in. Cluster permissions moved from aws-auth to EKS access entries, where the grant is tied to IAM and leaves an audit trail.
Do not let the new control become the outage
VPN-only access was the right default, but it created a new bottleneck: onboarding someone to the VPN required a VPN administrator. During an incident, an engineer with valid AWS access could still be locked out of the cluster because the person who could enroll them was unavailable.
The fallback is an SSM bastion inside the private VPC. It has no public IP and an empty ingress list; the instance dials out to SSM, and access is an IAM-authenticated API call rather than a network connection. The cluster role on that path is read-only. The cost is real: it is a second path to audit, and it is convenient enough that it has to be kept from becoming the default.
NetBird
Tailscale
Why: both solved the routing problem and Tailscale was the better-known option, but on this account's usage the cost difference was material and NetBird's free cloud tier covered the requirement. Evaluated rather than assumed — the deciding factor was price at this scale, not features.
EKS access entries with namespaced RBAC
The hand-edited aws-auth ConfigMap
Why: the ConfigMap is a single YAML file in which a typo is a privilege grant, it has no audit trail of its own, and its default state on a cluster nobody has had time to think about is cluster-admin for everyone. Access entries move the grant into IAM where it is logged and reviewable. The cost: two systems to understand during the migration window.
Controls that had to fit the estate
Three account-level hardening items remained. The recurring pattern was that the tidiest version of the control would have broken something already serving traffic.
The AWS-managed EBS key
A customer-managed key
Why: a CMK requires explicit KMS grants for the EKS node role and the EC2 Auto Scaling service-linked role. Without them new nodes fail to launch, which surfaces during a scale-up or node replacement rather than during the change. The managed key is less impressive on a checklist and does not have that failure mode. The cost: weaker key custody, with a CMK migration deferred until the prerequisite grants are in place.
Per-bucket public-access blocking on verified-private buckets
Account-level Block Public Access
Why: account-level blocking overrides bucket policy, and one bucket is public by design. The one-setting option would have taken a working feature offline in the name of a control. Per-bucket blocking on buckets first verified non-public changed no effective access. The cost: seven resources to maintain instead of one, and account-level blocking still deferred.
Encryption-by-default was applied only where it would not mutate existing running volumes as a side effect. Existing unencrypted volumes were handled by recycling node groups as a scheduled operation. Threat detection was extended to a region that had been left out of coverage.
The close-out separated work I built from controls already supplied by the managed security tooling stack. That matters. "Verify and document" is weaker than "rebuild," but it is the defensible claim when a guardrail baseline already exists.
The recovery number I published
Cross-region disaster recovery was the last gap. Point-in-time recovery existed inside the region; nothing covered loss of the region itself.
The design used hourly snapshots copied cross-region into a vault encrypted with a separate customer-managed key. Six production buckets replicate cross-region, with existing objects backfilled by a batch job.
The measured gap between snapshot completion and copy landing was forty to fifty minutes. That makes the effective cross-region recovery point roughly forty-five minutes to just under two hours, depending on where in the cycle the region fails.
Publishing a recovery point of up to ~1h50m
Publishing the ~1h target the design was scoped against
Why: the snapshots were hourly, the ticket said an hour, and every dashboard was green — the wider number could have gone unwritten. A genuinely seconds-level cross-region RPO needs a globally replicated database, which was scoped, costed and deferred as a spending decision. An accepted posture with a real number behind it is defensible; a target with an unmeasured number behind it is a finding waiting for an auditor.
The backfill produced a separate finding. A bucket that already replicated to another destination was skipped by a batch job filtered on objects whose replication status was unset or failed. S3 marked those objects COMPLETED because they had completed replication somewhere, not because they had completed replication to the new destination. Backfilling a new destination required including COMPLETED in the filter.
The same move for developers
One more access grant existed because nobody had separated the capability from the credential. Developers needed realistic data on a laptop. The implementation had become "grant namespace access so they can port-forward to staging Postgres and dump data."
That capability now lives in a Lambda with its own role: open the port-forward, dump, upload to S3, and return the key. Developers restore from S3 with a single read permission, and kubectl left the setup instructions. There is also a path for someone with no AWS principal at all: a teammate hands over a file.
Same shape as the rest: move the capability, not the credential.
Verification
The access claim was verified by failing to connect, not by reading policies.
| Path | Off the overlay | On the overlay |
|---|---|---|
| Bastion SSH, production and pre-production | Times out | Connects |
| Kubernetes API | No reachable endpoint | kubectl works |
That distinction matters. A screenshot of a security group is evidence that a rule looked right when the screenshot was taken. A timeout from a valid path outside the overlay is evidence that the route does not exist.
What remains weak
- Overlay activity logs are not co-located. They live in the provider dashboard and API, not in the same S3 and SIEM path as the rest of the evidence.
- The quarterly access review is documented but has not run. A written process nobody has executed has an unknown failure rate.
- Human identity is interim. SSO users are native rather than federated from the company directory, so joiner-mover-leaver is still partly manual.
- The wider recovery point is accepted, not fixed. Backup copy lag was measured but is not monitored.
- The customer-managed EBS key is deferred. The prerequisite grants are written down so the migration is not retried blindly.
- The overlay is now critical path for administrative access. The SSM fallback reduces that risk; it does not remove the need for break-glass procedure.
What changed
- Four public administrative paths closed, verified by connection timeouts rather than policy review.
- Two static IAM keys deactivated, including one with cluster-admin, and replaced by per-run federated identity across seven repositories.
- Cluster permissions moved out of a hand-edited ConfigMap into IAM-logged access entries scoped per namespace.
- Encryption-by-default, public-access blocking and threat detection applied in forms that changed nothing already serving traffic.
- Production data went from no cross-region recovery path to an hourly one, published with the measured recovery point.
- Cluster access stopped being a prerequisite for local development.
No cost figure is claimed. No incident-reduction figure is claimed. The overlay was chosen partly on price, but no billing comparison was produced; the controls had no measured incident baseline. What is checkable is the state: empty ingress lists, deactivated keys, an endpoint that does not answer, and audit logs retained for review.