A regulated US lender was running an ACH-settling ledger on infrastructure where the Kubernetes API, all three databases, and every non-production app answered from the public internet. I rebuilt access as a zero-trust WireGuard overlay — and the hard part was never making things private, it was keeping them usable afterwards.
The platform moves investor money to small-business borrowers: about 22.8 million database queries and 3,600 reconciliation runs a month across six payment rails, with the highest-frequency settlement job firing every fifteen minutes. Capacity is governed by the ledger, not the web tier. The infrastructure under it was provisioned by hand in 2021 and served 292,709 requests in 30 days with 40 server errors — 0.014%. Performance was never the problem. The perimeter was.
One shared cluster with everything in the
defaultnamespace and deployments named-devserving the production domain. Three databases on public IPs. A Kubernetes API that answered from any network on the internet, with no allowlist configured at all. For a company touching ACH files and investor funds, each of those is an audit finding on its own.
From an estate on the public internet to one that answers only to identified peers
Making it private is easy — the bill comes afterwards
Flipping the control plane to a private endpoint and pulling the public IPs off the databases takes an afternoon. It also breaks engineers running kubectl, developers using SQL clients against dev and staging, CI deploying to the cluster, and the static-site build that fetches the API at build time from a runner whose address changes every run.
So the question was never how to close it, but what replaces the public internet as the thing those four groups arrive through. The three standard answers each fail differently. IP allowlists describe networks, not people: CI ranges rotate, home addresses are dynamic, and an over-broad entry fails silently forever. A bastion is a permanent public endpoint with key custody — the perimeter moves rather than shrinks. One flat VPN hands every peer a route to every environment, restoring the shared blast radius that splitting the estate was meant to remove.
The access model
Isolated networks, resource-level grants, tiers that fail safe, and one routing decision that kept non-production apps off the internet without breaking the build.
Three overlays that cannot see each other
Not one network with rules inside it — three separate networks, each served by its own router peer. There is no route from staging to production because no such route is ever created. Isolation is a property of the topology, not a rule someone can relax later.
Inside each environment I deliberately did not publish "the VPC" as one routable blob. Three ranges are published as separately grantable resources: cluster nodes, the private database range, and the small range the managed control plane sits on. Grant a subnet and you grant everything in it — that split is what makes the next step possible.
The routers themselves started as pods inside each cluster. That was tidy, needed no extra machine, and caused a node incident — but the incident mattered less than the topology it exposed: if the router lives in the cluster, the path you use to reach a broken cluster depends on that cluster being healthy. Each environment now runs its own small router instance outside the cluster it serves.
Two human tiers, one CI identity per environment
Most VPN setups have one tier: connected, or not. This one has two. Developers — where nearly everyone sits — reach databases and app domains in every environment, but no cluster nodes and no control-plane range. Not denied by RBAC: there is no route, so a perfectly valid kubeconfig in those hands is an inert file. Elevated peers get the full set, by a deliberate per-person act.
The default all-to-all policy is deleted, so every grant that exists is one somebody wrote down — and it comes out last, after the replacements are in place, or you lock the team out of the network they need to fix it. Any peer in no infrastructure group defaults to developer, which matters because the identity provider emits no groups claim: a peer can't infer its tier from its token. Someone reinstalls a laptop, re-enrols with a new cryptographic identity, and lands in the lower tier instead of silently keeping a stale grant.
CI is held to the same shape. The legacy pipeline used a static service-account key with no expiry, reaching whatever the project allowed; now each environment has its own CI identity scoped to its own overlay and, inside it, to only the cluster node range. Not the database, not the control plane. Runner peers are ephemeral and remove themselves when a job ends; router peers persist, because the thing granting access to everything else must survive a reboot without a human.
Keeping dev and staging off the internet without breaking the build
Dev and staging carry production-shaped data and get patched last, so they sit behind per-environment Cloud Armor policies that allow a very small set of source addresses and refuse everything else at the edge. That solves the human case. It breaks the build.
The static-site build fetches the API at build time from a hosted runner, at an address I can't predict. The standard workaround — allowlisting the CI provider's published ranges — means thousands of rotating addresses shared with every other customer of that provider, which would have made the policy decorative.
Instead I published the environment's own public ingress address as a resource inside the overlay. A runner that has joined the overlay sends its build-time request through that environment's router, so it leaves through that environment's Cloud NAT and arrives at the edge from the one stable address Cloud Armor already allows. Humans browsing dev or staging take the same path for free.
One allowlist entry per environment, permanently. It stopped describing networks that might contain someone I trust and started describing the egress identity of traffic the overlay already authenticated. Those NAT addresses are pinned for a second reason: the client's payment partners allowlist them too, so an egress IP here is a contract with a third party, not an implementation detail.
The latency decision nobody warns you about
The overlay's default is lazy connections: no tunnel until traffic needs one, then handshake on demand. At fleet scale that's obviously right. Here it produced a bug that looks nothing like a network setting — the engineers are in Asia, the estate is in us-central1, and that cold handshake costs 10 to 20 seconds. The day's first request to any internal resource stalls: a blank single-page app, a SQL client that appears to hang, and an entirely reasonable conclusion that the VPN is slow.
Every peer now holds a persistent tunnel. The cost is sustained hub capacity and it scales with fleet size — correct for a small team on a near-idle hub, wrong at several hundred peers. That's the condition to revisit it on, not a date.
An identity-scoped WireGuard overlay
IP allowlists on public control-plane and database endpoints
Why: an allowlist is a claim about networks, not about people — and an entry that is too broad, or outlived its owner, fails silently. The overlay makes access a property of an authenticated peer, so the endpoint can be genuinely private and a grant revoked in one place. The cost: the overlay becomes critical path for human access.
Hairpinning CI traffic out each environment's own NAT address
Allowlisting the CI provider's published runner ranges
Why: allowlisting thousands of rotating addresses shared with every other customer of that provider is a rule that exists and protects nothing. Routing through the environment's own egress means exactly one allowed source per environment, permanently, and what is trusted is an authenticated path rather than an address range. The cost: an overlay outage is now a build failure.
How I verified the perimeter
Confirmed by failing to connect. kubectl against the production API from outside the overlay — valid credentials, project-level IAM — times out. The credential is fine; the route does not exist.
Confirmed one environment at a time. Each cutover was verified end to end over the overlay — kubectl to the private control plane plus a live connection to the private database — before the next environment moved. Dev first, then staging and production.
True by construction, not by test. The developer tier has no route to a control plane and a CI identity reaches only its own node range, so both are bounded by topology rather than by policy correctness. A control that depends on a route not existing fails only when someone deliberately builds that route — far louder than someone editing a rule.
What it changed
Recovery went from "none" to a number. The ledger database ran an end-of-life engine with automated backups disabled, so a bad migration on the ledger that reconciles real money across six processors was unrecoverable. It now runs a supported engine with binary logging, deletion protection, and point-in-time recovery to any second in a 7-day window.
Growth stopped being a forced purchase. Measured under real traffic, not modelled: at effectively identical query volume — about 10.9 million a month each — the legacy database sits at 100% memory utilization and the new one at 16.1%. Legacy has zero headroom, so the next cohort of loans forces a paid upgrade; the new instance absorbs roughly six times the current ledger workload first.
Every credential has a written blast radius. The honest answer to "what could this key reach?" used to be "most of the project." Now it ends at one environment's app tier.
What I'd watch, and what is still open
- The overlay is critical path for human access. Routers are persistent, per-environment and outside the clusters they serve — but a break-glass path that doesn't depend on the overlay still needs writing down before it's needed.
- Everything is single-zone. All databases and all three clusters are zonal, as the legacy estate was. A zone failure is still a full outage — the one axis the rebuild didn't improve.
- A control regressed. The legacy cluster encrypted etcd with a customer-managed key; the new clusters don't yet. For a fintech estate that isn't a footnote.
- Collection is not alerting. Eleven metric streams are gathered instead of one, including the pod and job metrics whose absence once let a broken secrets operator halt payment reconciliation for seventeen days unnoticed.
- No cost figure is claimed. No billing export exists on either project and both estates ran in parallel during migration, so any spend comparison would be invented.
Key results
- Production's Kubernetes API went from answering the entire internet to answering one address, verified by failing to reach it from outside with valid credentials.
- All databases moved off public IPs — three of three exposed to zero of four — reachable only through an authenticated overlay path.
- Three isolated overlays replaced a shared perimeter, so cross-environment reachability doesn't exist rather than being denied.
- Dev and staging stayed off the public internet without breaking static-site builds, by turning a per-environment allowlist into a list of egress identities.