A host tells the room to scan the code, and within seconds everyone in it is in the lobby of a live music-bingo game on their own phone. Then the host starts the game and the whole room moves into play at once. When I came in as the DevOps engineer, that second moment was the one the platform couldn't survive once a session passed 70–80 players.
The situation
The product is a platform for live, host-led music bingo. A host plays a set of songs, and players mark squares on a bingo card on their own phone as they recognise the tracks. Players join by scanning a QR code, with no app to install and no account to create, because zero friction on the player side is what the product sells. So an entire room enters the lobby within seconds and then moves into gameplay together. The load isn't a traffic curve that happens to spike; it's a step, and the product creates it on purpose. Better provisioning can't smooth that out, and the product would be worse if it did.
That is also what made the failures expensive. A live show has a paying host and a full room, and there is no redeploying and trying again halfway through. My job was to make the platform hold real audiences.
The infrastructure work was mine, and I did it alone: the observability stack, the load testing, the pod topology, the KEDA trigger and its custom metric API, and the CI build changes. The client's product team owned the game itself and its integration with Hocuspocus, the real-time collaboration server that carries the game's socket traffic.
What I saw
First I needed to see the failure. I put kube-prometheus-stack in front of all three services (the Next.js frontend, the API and Hocuspocus), used Headlamp to inspect them at runtime, and then watched real sessions fail. Two things stood out.
The first was the shape of the frontend load. It didn't build up as players arrived. It peaked at one specific moment, when the room moved from the waiting room into gameplay and every client loaded the game at once. That mattered because the frontend was scaled by HPA on CPU and memory, and CPU is a lagging proxy for this kind of load. HPA only reacts after the resource curve moves, so by the time CPU rises the players are already in. Against a step, a CPU-based autoscaler is always late.
The second was that Hocuspocus and the API were running in the same pod. When Hocuspocus fell over under socket load, it took the API down with it. That's why every failure looked like the whole session collapsing, not just collaboration getting slow.
The obvious fix made it worse
Hocuspocus was the part falling over, so the next question was how to give it more capacity. The standard answer is to scale it horizontally, running more replicas with the Redis extension so they share state. That was my first instinct too, and I built it.
Load testing ruled it out. For players on different pods to stay in the same game, every pod broadcasts its document updates to every other pod. With n pods that comes to roughly n×(n−1) inter-pod interactions, so each new replica added one pod's worth of capacity but also a new set of broadcast paths to and from every pod already running. The overhead grew faster than the capacity I was adding. Past a small number of replicas, each new pod lowered the number of players the tier could hold, which is the opposite of what scaling out is for.
Shipping it would have cost capacity instead of adding it.
What I did instead
Split the backend first
The first change came before any of the scaling tests: one container per pod, with the API and Hocuspocus separated. It's a routine change, but it was the right first move because it changed what failures looked like. After the split, a Hocuspocus problem stayed a Hocuspocus problem, and stability improved right away. It was also cheap, and it made the test results that followed readable, because a failure under load now pointed at one service instead of taking both down together.
Fewer, larger Hocuspocus pods
With scale-out ruled out, the collaboration tier scales vertically. It runs fewer, larger pods, sized where the load tests kept pointing: 100–150 players each.
Fewer, larger Hocuspocus pods (100–150 players each)
Many small pods scaled horizontally with a Redis extension
Why: Each pod broadcasts updates to every other pod. That n×(n−1) inter-pod overhead made scaling out actively worse. The trade-off is real. Room size is now limited by per-pod memory, and that limit is fixed, not elastic.
That fixed ceiling is the price of this design. Horizontal scaling would have been better in that one respect, if it had worked.
Scale the frontend on the signal that comes first
For the frontend, the question was what moves before the load does. CPU moves after it. Waiting-room player count moves before it, the game already tracks that number, and it exists seconds before the transition it predicts. So I scaled on the product's own state instead of a resource metric.
A small metric API polls waiting-room occupancy every few seconds and exposes it to KEDA, a Kubernetes autoscaler that can scale on external signals like this one. When the count crosses the threshold, KEDA scales the Next.js pods, so they start coming up while players are still in the lobby rather than after they've all loaded the game.
KEDA with a custom player-threshold metric
Standard HPA on CPU/memory
Why: The spike comes from one gameplay event, the move from waiting room to gameplay. It doesn't track CPU. The product's own state already holds the early signal. The resource metrics only show the load after it lands.
How I know it works
I want to be precise about the 400+ figure. It's load-verified: it's what the rebuilt topology held under tests that reproduced the exact flow that broke production. Players filled the waiting room, moved into gameplay together, and then generated socket-heavy Hocuspocus traffic. That is a different thing from a synthetic request rate against an endpoint.
Through each run I watched three signals: how the frontend behaved across the waiting-room-to-gameplay transition, how Hocuspocus CPU and memory grew as socket activity climbed, and whether the API stayed stable when collaboration traffic got noisy.
I'm not claiming a production peak of 400. I never recorded one. What I'm claiming is that the tested flow, the one that used to fail at 70–80, holds at 400+.
The result
The API and Hocuspocus no longer fail together, the frontend scales on a signal that comes before the load instead of one that trails it, and the collaboration tier is sized from measurement rather than from the assumption that more replicas always means more room.
The build-time figure is separate work on the same project: Turbo prune, Next.js standalone output in a multi-stage Docker build, GitHub Actions layer caching and Turborepo remote caching, measured on the same pipeline before and after. The biggest single win was that Docker builds stopped copying the entire monorepo into every image.
What's still open
- Reconnect storms. They're a routine operating condition, not an edge case, because the platform runs in venues on shared wifi and virtually over video bridges. To the infrastructure a reconnect storm looks almost the same as a join burst, except that it doesn't follow a waiting-room signal, so nothing triggers scale-up ahead of it. This is the first thing I'd instrument.
- KEDA scale-up latency versus join speed. If a room fills faster than pods come up, the transition can still outrun the metric. I haven't measured the gap between the threshold being crossed and pods being ready against the observed join window, and it's the number I'd want next.
- The per-pod memory ceiling. 100–150 is the tested band. A longer or heavier session could push a pod past it, and the tier doesn't scale out to absorb that.
- Cold starts. A cold Next.js pod arriving mid-transition is the worst case to design for.