A registered funding portal — a US lending marketplace supervised by the securities regulator — was running its Kubernetes API, all three databases, and every non-production app on the public internet. I rebuilt access as three isolated WireGuard overlays and took every database off the public internet — all three of them, on an estate that has since grown to four, none of which is reachable from outside. The interesting part was not closing the estate; it was that closing it breaks four groups of people at once, and every standard fix for that reintroduces the original problem.
Ownership. Sole. I designed and built the access rebuild end to end: overlay topology, resource grants and access tiers, the CI identity model, the Cloud Armor and NAT routing, and the phased rollout across all three environments. No other engineer worked on it. The client team owned the application and its data layer. This write-up covers access and network security; other workstreams on the same engagement are out of scope here.
A registered funding portal: registered with the US securities regulator, a member of the industry SRO, operating under the retail crowdfunding exemption since the late 2010s · small businesses borrow from non-accredited retail investors · the platform underwrites the borrower, hosts the campaign, then services repayments to every individual noteholder for the life of the loan · a book of record subject to examination, not an application database · infrastructure hand-provisioned in 2021
What that means for an access model: this is a registered financial intermediary holding books and records, and the people whose money moves through it are retail investors who are, by the design of the exemption, not sophisticated. Non-production environments carrying that data are not a low-stakes tier.
Performance was never the problem. The estate was serving its load without complaint. The perimeter was the problem.
What was actually wrong
- One shared GKE cluster held dev, staging and production, all in the
defaultnamespace, with deployments suffixed-devserving the production domain. - The Kubernetes API answered from any network on the internet, with no allowlist configured at all.
- Three databases had public IPs.
- Dev and staging were loadable by anyone on the internet and carried production-shaped retail investor data — not synthetic fixtures. These are also the environments that get patched last.
- CI authenticated with a static service-account key with no expiry, scoped to whatever the project allowed.
No breach is claimed here, and none is known to me. What I can say is narrower and worse: nothing would have prevented one, and nothing would have shown it afterwards.
From an estate on the public internet to one that answers only to identified peers
Why this isn't a one-afternoon change
Flipping the control plane private and pulling public IPs off the databases genuinely does take an afternoon. It also breaks four groups the same day: engineers running kubectl, developers using SQL clients against dev and staging, CI deploying to the cluster, and a static-site build that fetches the API at build time from a runner whose address changes every run.
So the real question was what replaces the public internet for those four groups. The three obvious answers each fail differently:
- IP allowlists describe networks, not people. CI ranges rotate, home addresses are dynamic, and an entry that is too broad fails silently forever.
- A bastion is a permanent public endpoint with key custody attached. The perimeter moves rather than shrinks.
- One flat VPN gives every peer a route to every environment — which restores exactly the shared blast radius that splitting the estate was supposed to remove.
The decisions that mattered
The access model
Four decisions: how environments are separated, how humans are tiered, how CI gets in, and what the tunnel behaviour costs.
Three overlays instead of one network with rules in it
Each environment gets its own overlay and its own router peer. There is no route from staging to production because no such route is ever created — isolation is a property of the topology, not a rule someone can relax later.
Inside each environment I did not publish the VPC as one routable blob. Three ranges are published separately: cluster nodes, the private database range, and the small range the managed control plane sits on. Granting a subnet grants everything in it; splitting them is what makes tiered access possible at all.
One thing I got wrong first: the routers started as pods inside each cluster. Tidy, no extra machine, and it caused a node incident. The incident mattered less than what it exposed — if the router lives in the cluster, the path you use to reach a broken cluster depends on that cluster being healthy. Each environment now runs a small router instance outside the cluster it serves.
Two human tiers, separated by routing rather than RBAC
Most VPN setups have one tier: connected, or not. This has two. Developers, where nearly everyone sits, reach databases and app domains in every environment but no cluster nodes and no control-plane range. They aren't denied by RBAC — there is no route, so a valid kubeconfig in those hands is an inert file. Elevated peers get the full set, granted per person.
Two details make it fail safe. The identity provider emits no groups claim, so a peer can't infer its own tier from its token, and any peer in no infrastructure group defaults to developer. Someone reinstalling a laptop re-enrols with a new identity and lands in the lower tier instead of silently keeping a stale grant.
CI is held to the same shape: each environment has its own identity, scoped to its own overlay and, inside it, to the cluster node range only — not the database, not the control plane. Runner peers remove themselves when a job ends. Router peers persist, because the thing that grants access to everything else has to survive a reboot without a human.
The permissive default all-to-all policy came out last, after the replacements were verified. Delete it first and you lock the team out of the network they need to fix it.
Making the allowlist describe egress identity instead of networks
Dev and staging sit behind per-environment Cloud Armor policies that allow a small set of source addresses and refuse everything else at the edge. That covers humans. It breaks the build.
The static-site build fetches the API at build time from a hosted runner whose address I can't predict. The standard workaround is to allowlist the CI provider's published ranges — thousands of rotating addresses shared with every other customer of that provider. The rule would exist and protect nothing.
Instead I published the environment's own public ingress address as a resource inside the overlay. A runner that has joined the overlay sends its build-time request through that environment's router, leaves through that environment's Cloud NAT, and arrives at the edge from the one stable address Cloud Armor already allows. Humans browsing dev or staging take the same path for free. One allowlist entry per environment, permanently.
Then I found the constraint that changed the risk. Those NAT addresses were already load-bearing: the platform's payment partners allowlist them too. An egress IP on this estate is a term of a commercial agreement, not an implementation detail — which meant any change to egress had to preserve the exact addresses, and I had to check that before checking anything else. Re-provisioning a NAT address during a cutover would have looked like a purely internal change and produced a payment outage.
Eager tunnels, and what they cost
The overlay defaults to lazy connections: no tunnel until traffic needs one. At fleet scale that's right. Here it produced a bug that doesn't look like a network setting — engineers in Asia, estate in us-central1, and a cold handshake costing 10 to 20 seconds. The day's first request stalls: a blank single-page app, a SQL client that appears to hang, and a reasonable conclusion that the VPN is broken.
Every peer now holds a persistent tunnel. The cost is sustained hub capacity, which scales with fleet size — fine for a small team on a near-idle hub, wrong at several hundred peers. That's the revisit condition, not a date.
An identity-scoped WireGuard overlay
IP allowlists on public control-plane and database endpoints
Why: an allowlist is a claim about networks, not people — and an entry that is too broad, or outlived its owner, fails silently. The overlay makes access a property of an authenticated peer, so the endpoint can be genuinely private and a grant revoked in one place. The cost: the overlay becomes critical path for human access.
Hairpinning CI traffic out each environment's own NAT address
Allowlisting the CI provider's published runner ranges
Why: allowlisting thousands of rotating addresses shared with every other customer of that provider is a rule that exists and protects nothing. Routing through the environment's own egress means exactly one allowed source per environment, permanently. The cost: an overlay outage is now a build failure.
How I verified it
By failing to connect. kubectl against the production API from outside the overlay — valid credentials, project-level IAM — times out. The credential is fine; the route does not exist.
One environment at a time. Dev, then staging, then production. Each cutover was verified end to end over the overlay — kubectl to the private control plane plus a live connection to the private database — before the next environment moved, with both estates running in parallel throughout. There was no maintenance window the business would accept, so nothing could depend on one.
By construction, not by test. The developer tier has no route to a control plane and a CI identity reaches only its own node range. Both are bounded by topology rather than policy correctness, so they fail only when someone deliberately builds a route — which is far louder than someone editing a rule.
The estate in code
The legacy estate was provisioned by clicking, in 2021. Everything above is now defined in Pulumi, in TypeScript, with state on a GCS bucket rather than Pulumi Cloud, and a per-stack passphrase for secrets.
Four stacks, and the fourth is the interesting choice. dev, staging and prod are three stacks of one project. platform is a separate project with its own state namespace, holding the project-level singletons — fleet APIs, Connect Gateway IAM — that the three environment stacks would otherwise contend over. It can't be a fourth stack of the same project: the environment program hard-requires per-environment network and GKE config that a project-level stack has nothing to put in.
Nothing applies on merge. Apply is manual dispatch only, and it asks you to re-type the stack name first — a mis-click on prod shouldn't be one button from an apply.
That gate is built from what the platform actually enforces, not what the docs offer. On this org's GitHub plan a private repo gets no branch protection, no required status checks and no required reviewers. What is enforced — I probed it rather than assuming — is the environment deployment-branch policy: a push from a non-main branch at a prod environment was rejected in about a second, zero steps run. So the gate is four layers on top of that: manual dispatch, environment restricted to main, a provenance check refusing any commit that didn't arrive through a merged and approved PR, and GCP IAM scoped so a feature branch can't mutate anything even if it ran.
The honest limit: the provenance layer lives in the workflow file, so anyone with write access could edit it away. It stops accidents and process drift, not a determined admin — layers two and four are enforced outside the repo. The revisit condition is named rather than dated: the moment a third person has write access without already holding GCP owner.
Preview is read-only by construction. Every PR and every push to main previews all three environment stacks, and the only Pulumi verbs in that workflow are stack select and preview — no up, no destroy, no refresh, no state. A guard also fails the job if the preview proposes any create, delete or replace, so the pipeline that shows you a diff can't quietly become the one that applies it.
Drift needed its own job
A preview compares code to state, not to reality — and that gap is where the problems were. Four of them, each found only because somebody happened to look:
- a production node pool rebuilt out of band, under a new name
- a VPN authorized network added by hand to a development database
- a deleted VM
- all three clusters auto-upgrading a Kubernetes minor version
So drift detection is a separate read-only pipeline that runs on a daily schedule, ahead of the working day. It refreshes against live cloud before diffing, checks all three environment stacks, and raises an issue when it finds something — so the four events above surface on their own rather than waiting for somebody to happen to look.
It stays read-only in the same way the preview is: refresh reads live cloud, but the refreshed view is in-memory and never written back to state.
What changed
- Production's Kubernetes API went from answering the entire internet to answering one address.
- Every database in the estate is private-IP only, reachable only over an authenticated overlay path. Three were on public IPs when I arrived; there are four now, and none of them answers the internet.
- Three isolated overlays replaced a shared perimeter, so cross-environment reachability doesn't exist rather than being denied.
- The honest answer to "what could this CI key reach?" went from "most of the project" to one environment's app tier.
- An estate provisioned by clicking is now defined in Pulumi across four stacks, with a manual-only apply gate built from the controls the platform actually enforces.
- Dev and staging came off the public internet without breaking static-site builds.
What's still open
- The overlay is critical path for human access. Routers are persistent, per-environment and outside the clusters they serve, but a break-glass path that doesn't depend on the overlay still needs writing down before it's needed.
- Everything is single-zone. All databases and all three clusters are zonal, as the legacy estate was. A zone failure is still a full outage — the one axis the rebuild didn't improve.
- A control regressed. The legacy cluster encrypted etcd with a customer-managed key; the new clusters don't yet.
- Which parts predate me isn't written down. The estate was hand-provisioned in 2021 and is now in code, but this page doesn't separate what I brought under code from what was already there — and that's the more interesting half.
- No cost figure is claimed. No billing export existed on either project and both estates ran in parallel during migration, so any spend comparison would be invented.
- Scope. This covers access and network security. Database and observability workstreams on the same engagement are not described here.