Small businesses borrow from ordinary retail investors through this platform, and it keeps the books of record for every loan. When I started on its infrastructure, its Kubernetes API, its databases and every non-production app answered from the public internet. Making all of that private is an afternoon's work. It also locks out the engineers, the developers with SQL clients, the CI pipeline and a static-site build on the same day, and each of the standard ways to let them back in reopens the hole.
The situation
The client is a registered funding portal. It's registered with the US securities regulator, it's a member of the industry SRO, and it has operated under the retail crowdfunding exemption since the late 2010s. Small businesses borrow from non-accredited retail investors. The platform underwrites the borrower, hosts the campaign, and then services repayments to every individual noteholder for the life of the loan. Its database is a book of record subject to examination, not just an application database.
That shaped how I read everything else. The people whose money moves through this system are retail investors, and by the design of the exemption they aren't sophisticated ones, so an environment carrying their data can't be treated as low-stakes just because it's called dev. Performance was never the issue; the estate handled its load without complaint. The perimeter was. The infrastructure had been provisioned by hand in 2021, and any fix had to happen live, because the business wouldn't accept a maintenance window.
I did this alone. I designed and built the access rebuild end to end: the overlay topology, the resource grants and access tiers, the CI identity model, the Cloud Armor and NAT routing, and the phased rollout across all three environments. No other engineer worked on it. The client team owned the application and its data layer. This write-up covers access and network security only; other workstreams on the same engagement aren't described here.
What I found
One GKE cluster held dev, staging and production, all in the default namespace, and deployments suffixed -dev were serving the production domain. The Kubernetes API answered from any network on the internet, with no allowlist configured at all. Three databases had public IPs. Anyone on the internet could load dev and staging, and those environments carried production-shaped retail investor data rather than synthetic fixtures, in the places that usually get patched last. CI authenticated with a static service-account key that never expired, scoped to whatever the project allowed.
I don't know of any breach, and I'm not claiming one. What I can say is narrower and worse: nothing would have prevented one, and nothing would have shown it afterwards.
Why closing it wasn't the fix
Making the control plane private and pulling public IPs off the databases really does take an afternoon. It also breaks four groups on the same day. Engineers run kubectl. Developers use SQL clients against dev and staging. CI deploys to the cluster. And a static-site build fetches the API at build time, from a runner whose address changes on every run.
So the real question was what replaces the public internet for those four groups, and the three usual answers each fail in their own way. IP allowlists trust networks, not people: CI ranges rotate, home addresses change, and an entry that's too broad fails silently, forever. A bastion is a permanent public endpoint with key custody attached, so it moves the perimeter instead of shrinking it. And one flat VPN gives every peer a route to every environment, which brings back exactly the shared blast radius that separating the environments was meant to remove. What I wanted was access that belongs to an authenticated person or pipeline, scoped to what they actually need, in front of endpoints that are private rather than filtered.
An identity-scoped WireGuard overlay
IP allowlists on public control-plane and database endpoints
Why: An allowlist trusts networks, not people. An entry that's too broad, or that outlives its owner, fails silently. With the overlay, access belongs to an authenticated peer. The endpoint can be truly private, and a grant can be revoked in one place. The downside is that the overlay becomes the critical path for human access.
What I built instead
One overlay per environment, with the router in the wrong place at first
Each environment gets its own NetBird overlay, built on WireGuard, and its own router peer: the overlay member that publishes that environment's private ranges to other peers. There's no route from staging to production because I never create one. Isolation comes from how the network is built, not from a rule someone could relax later.
Inside each environment I didn't publish the VPC as one routable blob. I published three ranges separately: the cluster nodes, the private database range, and the small range the managed control plane sits on. Granting a subnet grants everything in it, so splitting the ranges is what makes tiered access possible at all. It's also why the Kubernetes API's allowlist can be that environment's router and nothing else.
I got one thing wrong at first. The routers started as pods inside each cluster. It was tidy and needed no extra machine, and it caused a node incident. The incident itself mattered less than what it exposed: with the router inside the cluster, the path I'd use to reach a broken cluster depended on that same cluster being healthy. So I moved each environment's router to a small instance outside the cluster it serves.
Two human tiers, separated by routing rather than RBAC
Most VPN setups have one tier: connected, or not. This one has two. Developers, where nearly everyone sits, reach databases and app domains in every environment, but not cluster nodes or the control-plane range. RBAC doesn't block them; there's simply no route, so a valid kubeconfig in their hands does nothing. Elevated peers get the full set, granted per person.
The tiers fail safe. The identity provider sends no groups claim, so a peer can't work out its own tier from its token, and any peer that isn't in an infrastructure group defaults to developer. Someone who reinstalls a laptop re-enrols with a new identity, so they land in the lower tier instead of quietly keeping an old grant.
CI follows the same pattern. Each environment has its own CI identity, scoped to its own overlay, and inside that overlay it reaches the cluster node range only, not the database and not the control plane. Runner peers remove themselves when a job ends. Router peers stay, because the thing that grants access to everything else has to survive a reboot without a human.
Making the allowlist describe egress instead of networks
Dev and staging sit behind per-environment Cloud Armor policies that allow a small set of source addresses and refuse everything else at the edge. That works for people, but it broke the build. The static-site build fetches the API from a hosted runner whose address I can't predict, and the usual workaround is to allowlist the CI provider's published ranges: thousands of rotating addresses shared with every other customer of that provider. The rule would exist and protect nothing.
So I turned the problem around. Instead of teaching the allowlist about the runner, I published the environment's own public ingress address as a resource inside the overlay. A runner joins the overlay and sends its build-time request through that environment's router. The request leaves through that environment's Cloud NAT and reaches the edge from the one stable address Cloud Armor already allows. People browsing dev or staging take the same path at no extra cost. That's one allowlist entry per environment, permanently.
Hairpinning CI traffic out each environment's own NAT address
Allowlisting the CI provider's published runner ranges
Why: Allowlisting thousands of rotating addresses shared with every other customer of that provider is a rule that exists and protects nothing. Routing through the environment's own egress means exactly one allowed source per environment, permanently. The downside is that an overlay outage now fails the build.
Then I found the constraint that changed the risk. Those NAT addresses were already allowlisted elsewhere, by the platform's payment partners. On this estate an egress IP is a term of a commercial agreement with a payment partner, not an implementation detail. So any change to egress had to keep the exact same addresses, and I had to confirm that before checking anything else. Re-provisioning a NAT address during a cutover would have looked like a purely internal change, and it would have caused a payment outage.
A default that was right for a fleet and wrong for this team
The overlay defaults to lazy connections, with no tunnel until traffic needs one. At fleet scale that's the right call. Here it produced a bug that didn't look like a network setting. The engineers were in Asia, the estate was in us-central1, and a cold handshake took 10 to 20 seconds, so the day's first request stalled. The single-page app showed a blank screen, the SQL client seemed to hang, and it was reasonable to conclude the VPN was broken.
I switched every peer to a persistent tunnel. The cost is sustained hub capacity, which grows with fleet size. That's fine for a small team on a near-idle hub and wrong at several hundred peers, and that fleet size, not a date, is when I'd revisit it.
How I know it works
The first check was a connection that failed. I ran kubectl against the production API from outside the overlay, with valid credentials and project-level IAM, and it timed out. The credential was fine; the route didn't exist.
The rollout went one environment at a time, dev first, then staging, then production, with both estates running in parallel throughout, since nothing could depend on a maintenance window. Before moving the next environment I verified each cutover end to end over the overlay: kubectl to the private control plane plus a live connection to the private database. I removed the permissive default all-to-all policy last, after the replacements were verified, because deleting it first would have locked the team out of the network they needed to fix it.
The tier and CI limits don't rely on getting a policy right. The developer tier has no route to a control plane, and a CI identity reaches only its own node range, because of how the topology is built. They only break if someone deliberately builds a route, and that's far more visible than someone editing a rule.
Keeping it closed
Closing an estate once doesn't keep it closed when it was provisioned by clicking. Four changes had been made out of band, each found only because somebody happened to look: a production node pool rebuilt under a new name, a VPN authorized network added by hand to a development database, a deleted VM, and all three clusters auto-upgrading a Kubernetes minor version.
So everything above is now defined in Pulumi, in TypeScript, with state in a GCS bucket rather than Pulumi Cloud and a per-stack passphrase for secrets. There are four stacks, and the fourth was the interesting choice. dev, staging and prod are three stacks of one project. platform is a separate project with its own state namespace, holding the project-level singletons, such as fleet APIs and Connect Gateway IAM, that the environment stacks would otherwise fight over. It can't be a fourth stack of the same project, because the environment program requires per-environment network and GKE config and a project-level stack has nothing to put there.
Nothing applies on merge. Apply runs only on manual dispatch, and it asks for the stack name to be re-typed first, so a mis-click on prod isn't one button away from an apply. I built that gate from what the platform actually enforces, not from what the docs offer. On this org's GitHub plan a private repo gets no branch protection, no required status checks and no required reviewers, so I tested what is enforced: the environment deployment-branch policy. A push from a non-main branch aimed at the prod environment was rejected in about a second, with zero steps run. The gate adds four layers on top of that: apply by manual dispatch only, the environment restricted to main, a provenance check that refuses any commit that didn't arrive through a merged and approved PR, and GCP IAM scoped so a feature branch couldn't change anything even if it ran.
That gate has a real limit. The provenance layer lives in the workflow file, so anyone with write access could edit it out; it stops accidents and process drift, not a determined admin. The second and fourth layers are enforced outside the repo, and I'll revisit the rest the moment a third person has write access without already holding GCP owner.
The pipelines that look at the estate can't change it. Every PR and every push to main previews all three environment stacks, the only Pulumi verbs in that workflow are stack select and preview, and a guard fails the job if the preview proposes any create, delete or replace. But a preview compares code to state, not to what's actually running, and the four changes above lived in exactly that gap. So drift detection is a separate read-only pipeline. It runs daily, before the working day starts, refreshes against live cloud before diffing across all three environment stacks, and raises an issue when something has moved. The refreshed view stays in memory and is never written back to state.
The result
Every database in the estate is private-IP only. Three were on public IPs when I arrived; there are four now, and none of them answers the internet. Three isolated overlays replaced a shared perimeter, so routes between environments don't exist and nothing has to deny them. Dev and staging came off the public internet without breaking the static-site build, and an estate provisioned by clicking is now in code, behind an apply gate built from controls the platform actually enforces.
What's still open
- The overlay is the critical path for human access. Routers are persistent, per-environment and outside the clusters they serve, but a break-glass path that doesn't depend on the overlay still needs writing down before it's needed.
- Everything is single-zone. All databases and all three clusters are zonal, as they were in the legacy estate. A zone failure is still a full outage. It's the one area the rebuild didn't improve.
- A control regressed. The legacy cluster encrypted etcd with a customer-managed key. The new clusters don't yet.
- It isn't written down which parts predate me. The estate was hand-provisioned in 2021 and is now in code, but this page doesn't separate what I brought under code from what was already there, and that's the more interesting half.
- I have no cost figure. Neither project had a billing export, and both estates ran in parallel during the migration, so any spend comparison would be made up.