A live host-led music-bingo platform crashed once a session passed 70–80 players. I came in as the DevOps engineer to make it hold real audiences. The obvious fix — more collaboration-server pods — made it measurably worse, because each pod broadcasts to every other pod and that overhead grows n×(n−1). The working shape was the opposite: fewer, larger pods sized from load tests, the API split into its own pod, and KEDA scaling the frontend on waiting-room player count instead of CPU. Load-verified to 400+ players in one session.
Ownership. Sole. The infrastructure work was all mine: the load testing, the pod topology, the KEDA trigger and its custom metric API, the observability stack, and the CI build changes. The client's product team owned the game itself and the Hocuspocus integration.
A host plays a set of songs; players hold a bingo card on their own phone and mark squares as they recognise tracks · players join by scanning a QR code — no app, no account — because zero friction on the player side is the product's selling point · so an entire room enters the lobby within seconds and transitions into gameplay together · a live show has a paying host and a full room, and there is no way to redeploy and try again
That last point is what makes this different from a traffic spike. The load is a step function by design — frictionless joining is the selling point, so a whole room can arrive in the time it takes a host to say "scan the code." Better provisioning doesn't smooth it out, because the product would be worse if it did.
From crash-at-80 to 400+ stable
Diagnosis: the spike had a shape, and it wasn't CPU
The first useful step was making the failure visible. I put kube-prometheus-stack in front of all three services and used Headlamp for runtime inspection, then watched real sessions fail.
Two things came out of that.
The frontend peaked at one specific moment — when a room moved from the waiting room into gameplay, and every client loaded at once. Not a climb; a step. HPA on CPU or memory reacts after the resource curve moves, which is structurally too late for that. By the time CPU rises, the players are already in.
Hocuspocus and the API were sharing a pod. When Hocuspocus fell over under socket load, it took the API with it. That's why the failure looked like a total session collapse rather than degraded collaboration.
The decisions
1. Split the backend before anything else
One container per pod, API and Hocuspocus separated. This is routine, but it was the right first move because it changed what the failures looked like — after the split, a Hocuspocus problem stayed a Hocuspocus problem, and stability improved immediately. Cheap, and it made the next round of testing interpretable.
2. Scaling Hocuspocus out — tested, then rejected
My first instinct was to scale Hocuspocus horizontally with a Redis extension for shared state. It's the standard answer and I built it.
Load testing said no. Because every pod broadcasts document updates to every other pod, n pods produce roughly n×(n−1) inter-pod interactions. The overhead grew faster than the capacity being added, so past a small number of replicas, adding a pod reduced the number of players the tier could hold.
So this tier scales vertically: fewer, larger pods, sized to where the tests kept pointing — 100–150 players each.
Fewer, larger Hocuspocus pods (100–150 players each)
Many small pods scaled horizontally with a Redis extension
Why: each pod broadcasts updates to every other pod — n×(n−1) inter-pod overhead that made scaling out actively worse. The trade-off is real: room size is now bounded by per-pod memory, which is a hard ceiling rather than an elastic one.
The trade-off is worth naming. Vertical scaling means the ceiling is per-pod memory, and it doesn't stretch. That is a worse property than horizontal scaling would have had — it just happens to be true, and pretending otherwise would have cost capacity rather than bought it.
3. Scale the frontend on a product signal, not a resource metric
CPU is a lagging indicator here. The waiting-room player count is a leading one, and the game already maintains it — the number exists seconds before the transition it predicts.
So a small metric API polls waiting-room occupancy every few seconds and exposes it to KEDA, which scales the Next.js pods the moment the threshold is crossed. Pods start coming up while players are still in the lobby, not after they've all loaded the game.
KEDA with a custom player-threshold metric
Standard HPA on CPU/memory
Why: the spike is tied to a specific gameplay event (waiting room → gameplay) rather than being proportional to CPU. The product's own state already contains the leading signal; the resource metrics only contain the lagging one.
Verification
The 400+ figure is load-verified, and worth being precise about: it's what the rebuilt topology held under tests that reproduced the exact flow that broke production — players filling the waiting room, transitioning into gameplay together, then generating socket-heavy Hocuspocus traffic. Not a synthetic request rate against an endpoint.
Three signals through each run: frontend behaviour across the waiting-room-to-gameplay transition, Hocuspocus CPU and memory as socket activity climbed, and whether the API stayed stable when collaboration traffic got noisy.
What I'm deliberately not claiming is a production peak of 400. I didn't record one. The claim is that the tested flow — the one that used to fail at 70–80 — holds at 400+.
Faster CI: 6 min → 2 min
Separate work on the same project. The Next.js build ran over 6 minutes, which throttled iteration. Four changes removed avoidable work: Turbo prune so each build pulled only the dependencies it needed, Next.js standalone output in a multi-stage Docker build, GitHub Actions layer caching, and Turborepo remote caching across branches. Measured on the same pipeline before and after.
The largest single win was the Docker context — it had been copying the entire monorepo into every image.
Known limitations
- Reconnect storms are an operating condition, not a hypothetical. The platform runs in venues on shared wifi and virtually over video bridges, so mass reconnects happen. To the infrastructure a reconnect storm looks almost identical to a join burst — except a join burst is preceded by the waiting-room signal that triggers scale-up, and a reconnect storm isn't. It arrives with no warning from the product's state. This is the thing I'd instrument first.
- KEDA scale-up latency versus join speed. If a room fills faster than pods come up, the transition can still outrun the metric. I don't have the measured gap between threshold crossing and pod-ready against the observed join window; that's the number I'd want next.
- Per-pod memory is a hard ceiling. 100–150 is the tested band. A longer or heavier session could push a pod past it, and the tier doesn't scale out to absorb that.
- Cold-start timing. A cold Next.js pod arriving mid-transition is the worst case to design against.
What changed
- Load-verified capacity from ~80 to 400+ concurrent players in one session on GKE.
- API and Hocuspocus isolated, ending the shared failure domain.
- Frontend autoscaling driven by a product signal that leads the load rather than a resource metric that trails it.
- Pod sizing chosen from measurement, against a scale-out approach that testing showed was actively harmful.
- Next.js CI build time from 6 min to 2 min.