React Native Release Pipeline: Diff-Routed OTA vs Native Builds with Real-Device QA Gates

This React Native app ships through two channels. Native binaries go through app-store review. JavaScript-only changes ship over the air, reaching installed devices in minutes with no external reviewer and no staged rollout — and once shipped, you can't recall them from a phone. That makes the CI pipeline the only review the OTA path gets. I built the routing that decides which channel a change takes, based on the changed file set rather than a person's judgement, plus a real-device Appium gate on BrowserStack that fits inside a one-session plan.

Ownership. Sole. I built both pipelines end to end: the release routing, the parent-child pipeline generation, the device-farm evaluation and the QA pipeline around it, the build-reuse check, the session serialisation, and the reporting. The client's product and QA teams owned the app itself and the content of the test suites.

Two of those facts change how much the OTA channel matters. There is a wallet with real balances in the app, so a billing bug is a financial incident rather than a UX one. And there is personal data about people who never signed up for anything. Both are reachable by a release no reviewer outside the team ever looks at.

Before

Expand

After

Expand

From a judgement call to a routing rule

§Routing the release from the diff

A React Native release isn't "build the latest commit and push it." Some changes need a full native binary; JavaScript-only changes can ship over the air. Get it wrong in one direction and you burn CI time on a native build nobody needed. Get it wrong in the other and you ship a JS bundle that depends on native code the installed binary doesn't have.

The useful realisation is that this isn't a judgement call at all. The changed file set doesn't predict the release type — it determines it. Changes touching package.json, eas.json, app.json, ios/ or android/ are native-sensitive. Everything else is JavaScript.

So the routing is deliberately blunt:

if native_sensitive_files_changed; then
  trigger_native_build
else
  trigger_ota_update
fi

That's the whole rule, and being boring is the point — it decides the same way every time, which a person under release pressure does not.

The pipeline is split parent-child: the parent runs validation and the routing decision, then generates the right child pipeline for the release type. Preview builds, production builds, OTA updates, store submission and QA triggers all want different jobs, and cramming them into one .gitlab-ci.yml with rules everywhere is how mobile CI becomes unreadable.

A small Cloud Run service bridges build-lifecycle events to the automation downstream, which is what lets a finished EAS build trigger the QA pipeline without someone watching a queue. I'm describing its role, not its internals — how much logic lives inside it isn't something this write-up documents.

§Choosing the device farm before building against it

Real-device E2E needed a cloud device farm, and I compared four before writing pipeline code: BrowserStack, Sauce Labs, AWS Device Farm and Firebase Test Lab.

That order was deliberate. A device farm isn't a library you swap later — it sits in the release path, carries a contract and a plan tier, and the test framework binds to it. Choosing after the pipeline exists means either rewriting the pipeline or living with whatever seemed fine first.

The deciding factor was operability rather than a feature comparison. A farm that wins on capabilities but is annoying to run day to day produces a gate people route around — which is worse than no gate, because it looks like coverage.

§Working inside a one-session plan

Android and iOS run as separate matrix jobs, which keeps each platform's context clean. The plan allowed one concurrent BrowserStack session, so both starting at once meant one failing on the session limit.

resource_group: "browserstack_global_limit"

There were two ways to handle that. Design for the plan — one hard-coded serial path, rebuilt when the budget changes. Or design for the shape of the problem, keeping the matrix jobs independent and letting a shared resource group be the only thing enforcing the limit.

One related optimisation: BrowserStack needs the .apk or .ipa uploaded before Appium can run, and re-running tests against a build you already tested shouldn't re-upload it. The pipeline checks whether the build already exists by signature, reuses the app reference if so, and uploads only when it doesn't.

Expand

§What the runs leave behind

When a suite finishes, the pipeline generates an HTML report for review and a PDF for the record, emails them, and uploads both as GitLab artifacts. Every run is attached to the pipeline that produced it: which build, which platform, which suite, what failed.

That's a deliberate choice about where QA results live. A report in someone's local folder or a Slack thread isn't a record — you can't go back six weeks and answer "was this tested before it shipped?"

§What the suites actually assert — an honest gap

This write-up describes the delivery system in detail and the test content not at all, and that asymmetry is real.

What's documented: the suites are Appium, they run smoke or regression selections against a real build on real Android and iOS devices, they run inside the release lifecycle, and every run files a report naming the build, platform, suite and failure.

What isn't: which user flows they cover. For a product whose critical paths are authentication, property search, a wallet-debiting lookup and an export, "we run Appium on real devices" describes the machinery, not the coverage. A pipeline that runs a thin suite very reliably is a gate in name. Naming even four flows would turn this from process into coverage, and I'd rather say that than imply otherwise.

§What I'd watch

• Flake rate on real devices. Real hardware surfaces timing and environment flakiness that simulators hide. A rising flake rate is what quietly decides whether anyone still trusts the gate a year in.
• Queue depth. With sessions serialised through one resource group, growing test volume shows up as wait time before it shows up as anything else.
• OTA rollout health. The path with no external reviewer deserves its own signal, and it doesn't have one here.
• Native build failure rate. EAS failures block the native path entirely.
• Bot mitigation. The platform runs a commercial anti-scraping service, and automated sessions driving real devices can resemble the traffic it exists to block. Whether that ever interfered here isn't something I'm going to claim in either direction.

§Measurement

Nothing on this pipeline was instrumented, so there are no before-and-after numbers here — no release frequency, escaped-defect rate, or merge-to-TestFlight time.

What the system does is verifiable from the configuration: the release path is decided by the diff, every build runs on real hardware before it ships, and every run leaves a durable record. Measurement is where I'd start if I picked this up again — merge-to-TestFlight time and real-device flake rate first.