Level Up Your Reliability: Lessons from the T-Mobile Outage for Building Multi-Region Test Infrastructure

On July 27, 2026, T-Mobile customers across the United States reported phones dropping into SOS mode. Tom's Guide's live report recorded a peak of roughly 64,000 user reports on Downdetector before the count declined. That figure is a report count, not a verified customer-impact total. T-Mobile acknowledged technical challenges affecting some customers, but, as of this article's publication, it had not published a root-cause analysis.

That uncertainty is itself useful. A professional reliability practice does not wait for someone else's postmortem before checking its own assumptions. A dashboard that says “two regions healthy” may hide one shared identity provider, one global database, one DNS control plane, or one retry storm capable of taking both regions down together.

You do not need a telecom-sized platform to learn this lesson. If your app started in Replit, Lovable, or a single cloud project, the next step is not buying an enterprise resilience diagram. It is converting “region B exists” into an executable claim. This guide builds that claim with Playwright, Node.js, guarded failure injection, and evidence your CI system can keep.

What did the T-Mobile outage teach software teams?

Redundancy becomes resilience only when tests prove traffic, state, dependencies, and recovery paths still work after one region disappears.

For the causal lesson, the best public evidence is the FCC's investigation of T-Mobile's June 15, 2020 outage. The outage lasted more than twelve hours. The FCC estimated that at least 41% of calls attempting to use T-Mobile's network failed, including 23,621 calls to 911. It also estimated that more than 250 million calls from other providers to T-Mobile subscribers failed.

The incident did not have one magical “root cause.” A fiber link failed. Incorrect OSPF weights sent signaling traffic to a router that could not pass it. A latent software defect repeatedly directed devices toward an unavailable registration node. Those retries produced a registration storm, and congestion spread beyond the original failure domain. Engineers then acted on an incorrect diagnosis and recreated the initial conditions while losing useful access to the affected router.

The FCC specifically recommended validating new procedures, commands, and devices in a lab that simulates the target network and its load. That maps directly to your test environment: reproduce topology, meaningful traffic, failure, and recovery—not just application code.

How should you test a multi-region application?

Probe every region directly and through global routing, then rehearse failover with bounded timeouts, isolated state, and auditable evidence.

Split the problem into three layers. First, prove each regional data plane can serve reads and writes without calling a global deployment API. Second, prove the global entry point routes only to capable regions. Third, prove state converges within your declared recovery point objective and service returns within your recovery time objective. RPO describes acceptable data loss; RTO describes acceptable recovery time. Neither is a slogan until a drill measures it.

AWS calls a related property static stability: a working data plane should continue without requiring a successful control-plane action during the incident. That matters because auto-scaling, deploying, changing DNS, and minting new secrets may all depend on impaired control planes.

Upgrade the test model, not just the deployment count

If your existing suite still assumes one URL, first follow our Playwright browser-context testing guide to separate environment configuration from test behavior. Then use the following migration in small, reviewable steps.

Step 1: turn regions into first-class Playwright projects

Do not copy and paste one config block per region. Parse a deployment manifest, reject unsafe input early, and let Playwright produce separate reports for each regional project. This complete configuration expectsREGIONS_JSON such as [{"name":"eu-west","baseURL":"https://eu.example.test"},{"name":"eu-central","baseURL":"https://de.example.test"}].

// playwright.config.ts
import { defineConfig, devices } from "@playwright/test";

type Region = { name: string; baseURL: string };

function loadRegions(): Region[] {
  try {
    const raw = process.env.REGIONS_JSON;
    if (!raw) throw new Error("REGIONS_JSON is required");

    const value: unknown = JSON.parse(raw);
    if (!Array.isArray(value) || value.length < 2) {
      throw new Error("Configure at least two regions");
    }

    const seenNames = new Set<string>();
    const seenUrls = new Set<string>();
    return value.map((item, index) => {
      if (!item || typeof item !== "object") {
        throw new Error(`Region ${index} must be an object`);
      }

      const { name, baseURL } = item as Partial<Region>;
      if (!name || !baseURL) {
        throw new Error(`Region ${index} needs name and baseURL`);
      }
      if (seenNames.has(name)) throw new Error(`Duplicate region: ${name}`);

      const url = new URL(baseURL);
      const isLocal = ["localhost", "127.0.0.1"].includes(url.hostname);
      if (url.protocol !== "https:" && !isLocal) {
        throw new Error(`${name} must use HTTPS outside local development`);
      }

      const normalized = url.origin;
      if (seenUrls.has(normalized)) {
        throw new Error(`Two regions resolve to the same origin: ${normalized}`);
      }
      seenNames.add(name);
      seenUrls.add(normalized);
      return { name, baseURL: normalized };
    });
  } catch (error) {
    const message = error instanceof Error ? error.message : String(error);
    throw new Error(`Invalid regional test configuration: ${message}`);
  }
}

const regions = loadRegions();

export default defineConfig({
  testDir: "./tests",
  fullyParallel: true,
  forbidOnly: Boolean(process.env.CI),
  retries: process.env.CI ? 1 : 0,
  reporter: [["html", { open: "never" }], ["json", { outputFile: "test-results/results.json" }]],
  use: {
    ...devices["Desktop Chrome"],
    trace: "retain-on-failure",
    screenshot: "only-on-failure",
  },
  projects: regions.map((region) => ({
    name: region.name,
    use: {
      baseURL: region.baseURL,
      extraHTTPHeaders: { "x-test-source": "multi-region-ci" },
    },
  })),
});

The validation is part of the reliability feature. A misspelled secondary URL must fail before the suite creates a comforting green report from the primary twice. The code also handles two common edge cases: local HTTP remains usable, while duplicate names or origins fail closed. One CI retry is diagnostic protection against transport noise; it is not permission to hide a deterministic regional failure.

Step 2: test state across the regional boundary

A /health endpoint often proves only that one process can return 200. The next test creates a uniquely keyed order in the current Playwright project, reads it through a peer region, and always attempts cleanup. The fixture API is intentionally restricted to test environments.

// tests/cross-region-state.spec.ts
import { expect, test } from "@playwright/test";

type Order = { id: string; testKey: string; status: "created" };
type RegionMap = Record<string, string>;

function loadPeers(): RegionMap {
  try {
    const peers = JSON.parse(process.env.REGION_URLS ?? "{}") as RegionMap;
    if (Object.keys(peers).length < 2) throw new Error("Need two REGION_URLS entries");
    for (const [name, rawUrl] of Object.entries(peers)) {
      if (!name || new URL(rawUrl).protocol !== "https:") {
        throw new Error(`Invalid peer region: ${name || "<empty>"}`);
      }
    }
    return peers;
  } catch (error) {
    throw new Error(
      `Cannot load peer regions: ${error instanceof Error ? error.message : String(error)}`,
    );
  }
}

test("a committed write becomes readable from a peer region", async (
  { request },
  testInfo,
) => {
  const peers = loadPeers();
  const source = testInfo.project.name;
  const peer = Object.entries(peers).find(([name]) => name !== source);
  if (!peer) throw new Error(`No peer configured for project ${source}`);

  const testKey = `pw-${source}-${crypto.randomUUID()}`;
  let order: Order | undefined;

  try {
    const create = await request.post("/__test/orders", {
      headers: { "idempotency-key": testKey },
      data: { testKey, amountCents: 1 },
      failOnStatusCode: false,
      timeout: 8_000,
    });

    if (create.status() === 409) {
      throw new Error(`Idempotency collision for unique key ${testKey}`);
    }
    if (!create.ok()) {
      throw new Error(`Create failed in ${source}: HTTP ${create.status()} ${await create.text()}`);
    }
    order = (await create.json()) as Order;
    expect(order.testKey).toBe(testKey);

    const [peerName, peerUrl] = peer;
    await expect
      .poll(
        async () => {
          const response = await fetch(
            `${peerUrl}/__test/orders/${encodeURIComponent(order!.id)}`,
            { headers: { "x-test-key": testKey }, signal: AbortSignal.timeout(3_000) },
          );
          if (response.status === 404) return "replicating";
          if (response.status === 429) return "backpressure";
          if (!response.ok) throw new Error(`${peerName} returned HTTP ${response.status}`);
          const body = (await response.json()) as Order;
          return body.testKey === testKey ? body.status : "wrong-record";
        },
        { message: `Order never converged from ${source} to ${peerName}`, timeout: 30_000 },
      )
      .toBe("created");
  } finally {
    if (order) {
      const cleanup = await request.delete(`/__test/orders/${encodeURIComponent(order.id)}`, {
        failOnStatusCode: false,
      });
      if (![204, 404].includes(cleanup.status())) {
        throw new Error(`Cleanup failed: HTTP ${cleanup.status()}`);
      }
    }
  }
});

Why poll instead of sleep? Asynchronous replication does not promise one exact completion instant. Polling expresses the real contract: convergence before the RPO-derived deadline. A 404 is temporarily acceptable, 429 means the peer is applying backpressure, and any other server error fails immediately. The unique idempotency key prevents retries from creating duplicate business state, while the finally block keeps repeated runs isolated.

Gotcha: replication success is not proof that writes are safe during a partition. If both regions accept the same logical operation, test your conflict rule—single writer, compare-and-swap version, globally unique idempotency key, or deterministic merge. “Last write wins” can silently lose a payment or account change when clocks skew.

Step 3: run a guarded failover drill

Now test the path your architecture diagram promises. This Node.js 20 script disables one staging region through a protected chaos endpoint, watches the global data plane move to the secondary, and restores the primary even when an assertion throws. It refuses to run without an explicit environment gate and token.

// scripts/failover-drill.ts — run with: npx tsx scripts/failover-drill.ts
type Status = { ok: boolean; servedBy: string };

const globalUrl = mustUrl("GLOBAL_URL");
const controlUrl = mustUrl("CHAOS_CONTROL_URL");
const primary = process.env.PRIMARY_REGION ?? "";
const token = process.env.CHAOS_TOKEN ?? "";

function mustUrl(name: string): URL {
  const raw = process.env[name];
  if (!raw) throw new Error(`${name} is required`);
  const url = new URL(raw);
  if (url.protocol !== "https:") throw new Error(`${name} must use HTTPS`);
  return url;
}

async function control(action: "disable" | "restore"): Promise<void> {
  const response = await fetch(new URL(`/regions/${encodeURIComponent(primary)}/${action}`, controlUrl), {
    method: "POST",
    headers: { authorization: `Bearer ${token}`, "content-type": "application/json" },
    body: JSON.stringify({ reason: "automated-failover-drill" }),
    signal: AbortSignal.timeout(5_000),
  });
  if (!response.ok) {
    throw new Error(`Control action ${action} failed: HTTP ${response.status} ${await response.text()}`);
  }
}

async function sample(): Promise<Status> {
  const response = await fetch(new URL("/readyz", globalUrl), {
    cache: "no-store",
    headers: { "x-drill-id": crypto.randomUUID() },
    signal: AbortSignal.timeout(3_000),
  });
  if (!response.ok) throw new Error(`Global endpoint returned HTTP ${response.status}`);
  const body = (await response.json()) as Partial<Status>;
  if (typeof body.ok !== "boolean" || !body.servedBy) {
    throw new Error("Malformed readiness payload; refusing to infer failover");
  }
  return body as Status;
}

async function waitForSecondary(deadlineMs: number): Promise<Status> {
  let last: Status | undefined;
  while (Date.now() < deadlineMs) {
    try {
      last = await sample();
      if (last.ok && last.servedBy !== primary) return last;
    } catch (error) {
      console.warn("Probe failed during convergence:", error);
    }
    await new Promise((resolve) => setTimeout(resolve, 1_000));
  }
  throw new Error(`Failover deadline exceeded; last region was ${last?.servedBy ?? "unreachable"}`);
}

async function main(): Promise<void> {
  if (process.env.ALLOW_FAILOVER_DRILL !== "true") {
    throw new Error("Set ALLOW_FAILOVER_DRILL=true after change approval");
  }
  if (!primary || !token) throw new Error("PRIMARY_REGION and CHAOS_TOKEN are required");
  if (process.env.DRILL_ENV !== "staging") {
    throw new Error("This script is staging-only; production needs a separate game-day approval path");
  }

  const before = await sample();
  if (!before.ok || before.servedBy !== primary) {
    throw new Error(`Unsafe starting state: expected healthy primary ${primary}, got ${before.servedBy}`);
  }

  let disabled = false;
  try {
    await control("disable");
    disabled = true;
    const after = await waitForSecondary(Date.now() + 60_000);
    console.log(JSON.stringify({ outcome: "passed", before, after }));
  } finally {
    if (disabled) await control("restore");
  }
}

main().catch((error) => {
  console.error("FAILOVER_DRILL_FAILED", error);
  process.exitCode = 1;
});

This is failure injection, not chaos cosplay. The start-state check prevents a drill from worsening an existing incident. The restoration runs from finally. Malformed health responses fail closed. Probe errors are tolerated only inside the bounded convergence window. A production version should also emit an audit event, page an owner if restoration fails, and use an out-of-band control path.

You can extend the same runner to inject dependency failures: deny the primary database route, expire a regional secret, throttle the queue, or return 503 from one identity edge. Keep one failure variable per drill until you can explain the result. For a broader test design, use our end-to-end testing workflow to connect these infrastructure checks to real user journeys.

Design the test infrastructure around failure domains

Running the same tests twice from one CI machine leaves a blind spot. An application may be reachable from GitHub-hosted runners in one country while private DNS, a peering route, or a certificate chain is broken for users elsewhere. Run probes both against every region and from independent locations.

• Give each runner its own network path and credentials. Shared NAT, VPN, or secret distribution can become a correlated failure.
• Keep the assertion bundle identical. Region-specific exceptions should be explicit data, reviewed like code, and temporary.
• Upload traces and JSON results to storage outside the region under test. Evidence that disappears with the failure cannot help diagnosis.
• Probe the global hostname and direct regional hostnames. One tests routing; the others preserve visibility when routing is wrong.
• Separate synthetic test tenants and cleanup policies by region. Data-residency rules may forbid copying even fake-but-derived customer fixtures across borders.

Start with active-passive if you are an indie developer. It reduces write-conflict complexity while still giving you a recovery target. Active-active earns its keep when latency or availability requirements justify conflict resolution, replicated queues, and a much larger test matrix. Professional does not mean maximum complexity; it means your chosen complexity is explicit and verified.

Troubleshooting multi-region tests without guessing

A failed drill is useful only if you can distinguish application, routing, state, and test-harness failures. Preserve the first error, timestamps, resolved IPs, response headers, served-region identity, and replication version. Then diagnose from the outside inward.

Your practical level-up plan

Resist the urge to jump directly from one deployment to global active-active. In week one, inventory shared dependencies and write a one-page RTO/RPO decision. In week two, deploy a warm secondary and run the same smoke contract directly against both regions. In week three, add cross-region state verification. In week four, rehearse a guarded staging failover and turn every surprise into a regression test.

Measure the things the user experiences: successful transactions, bounded recovery, preserved committed data, and clear communication. Do not grade yourself on the number of cloud logos in the diagram. The 2020 T-Mobile report shows how ordinary faults become extraordinary when routing, software, retries, and recovery tooling interact. Your advantage is that you can encode those interactions before the next outage chooses the test schedule for you.