Level Up Your Distributed Systems: Architecture Patterns from PlayStation Plus for Global Test Infrastructure

Your first end-to-end suite probably lived on one laptop or one CI job. That was the right architecture: one process, one browser, one obvious log. Then customers arrived in Europe and North America, authentication gained regional policies, and a checkout that passed from Madrid failed from Virginia. Adding ten identical CI jobs made the suite faster, but it did not make it trustworthy. A network partition could still strand the run, retries could charge a test account twice, and the final report could turn missing evidence into a green check.

A service such as PlayStation Plus gives us a useful public design prompt. Sony says PlayStation Network served more than 124 million monthly active users as of March 31, 2025. During the 2022 PlayStation Plus rollout, the company described a phased launch across Asia, Japan, the Americas, Europe, Australia, and New Zealand, plus cloud streaming in30 markets. Those are facts about scale and rollout, not a disclosure of Sony's private topology. The patterns below are our engineering inference from the public constraints: regional variation, huge concurrency, staged releases, and a product that must degrade cleanly.

What can PlayStation Plus teach us about global test infrastructure?

Separate global decisions from regional execution, contain failures in cells, and make every result durable, attributable, and safe to replay.

A global subscription product cannot assume that every capability exists in every market. Catalogs, streaming availability, prices, regulations, and rollout dates vary. Tests should model the same truth. “Run everything everywhere” is expensive and often incorrect; “run only in the cheapest region” misses the behavior users actually see. Express requirements such as locale, data residency, feature flags, payment sandbox, and browser engine as scheduling constraints.

This architecture has two layers. A small global control plane accepts a run, freezes a manifest, assigns shards, and reconciles completion. Regional data-plane cells execute browsers, cache dependencies, upload traces, and can be drained independently. A cell may be a managed CI runner pool today and a Kubernetes cluster later. The contract matters more than the scheduler.

From a single CI job to failure-contained regional cells

Keep each cell deliberately boring: a queue, workers, a nearby artifact bucket, and health signals. Do not let workers in Europe depend synchronously on a coordinator in the United States for every test step. The dispatcher assigns work; after that, a worker needs only the immutable manifest, secrets scoped to its region, and the application under test. This limits the blast radius and reduces cross-region chatter.

If your suite is still coupled to a single provider, first move credentials and runtime assumptions behind a narrow worker contract. Our Playwright sharding guide covers the test-side boundaries; this article focuses on the distributed control path.

Pattern 1: route shards deterministically, then fail over deliberately

The dispatcher should first remove ineligible cells, then choose among healthy ones. Rendezvous hashing is ideal here: compute a score for every eligible cell from the run key and cell name, then pick the highest. Adding or removing a cell moves only part of the keyspace, while retrying the same shard normally selects the same environment. That stability is valuable when a failure depends on a local cache, feature rollout, or regional backend.

This runnable Node 20+ dispatcher accepts a shard request, applies residency and browser constraints, probes health with a deadline, and emits a deterministic assignment. Its important edge case is zero healthy eligible cells: it fails closed instead of silently routing regulated traffic somewhere forbidden.

// scripts/dispatch-shard.ts — run with: npx tsx scripts/dispatch-shard.ts
import { createHash } from 'node:crypto';

type Cell = {
  name: string;
  healthUrl: string;
  residencies: string[];
  browsers: string[];
};

const cells: Cell[] = [
  { name: 'eu-west', healthUrl: 'https://eu-runner.example.com/health', residencies: ['eu'], browsers: ['chromium', 'firefox'] },
  { name: 'us-east', healthUrl: 'https://us-runner.example.com/health', residencies: ['us'], browsers: ['chromium', 'webkit'] },
];

async function isHealthy(cell: Cell): Promise<boolean> {
  try {
    const response = await fetch(cell.healthUrl, {
      signal: AbortSignal.timeout(2_000),
      headers: { accept: 'application/json' },
    });
    if (!response.ok) return false;
    const body = (await response.json()) as { acceptingWork?: boolean };
    return body.acceptingWork === true;
  } catch (error) {
    console.warn('Health probe failed', { cell: cell.name, error: String(error) });
    return false; // Timeout, invalid JSON, and DNS failure all remove the cell.
  }
}

function score(key: string, cell: string): bigint {
  const hex = createHash('sha256').update(key + ':' + cell).digest('hex').slice(0, 16);
  return BigInt('0x' + hex);
}

async function dispatch(input: { runId: string; shard: number; residency: string; browser: string }) {
  if (!input.runId || !Number.isInteger(input.shard) || input.shard < 1) {
    throw new Error('runId is required and shard must be a positive integer');
  }
  const eligible = cells.filter(
    (cell) => cell.residencies.includes(input.residency) && cell.browsers.includes(input.browser),
  );
  const health = await Promise.all(eligible.map(async (cell) => ({ cell, ok: await isHealthy(cell) })));
  const healthy = health.filter((item) => item.ok).map((item) => item.cell);
  if (healthy.length === 0) {
    throw new Error('No healthy eligible cell; refusing unsafe cross-residency fallback');
  }
  const key = input.runId + '/shard-' + input.shard;
  return healthy.sort((a, b) => (score(key, a.name) > score(key, b.name) ? -1 : 1))[0];
}

const input = JSON.parse(process.argv[2] ?? '{}') as Parameters<typeof dispatch>[0];
dispatch(input)
  .then((cell) => console.log(JSON.stringify({ assignedCell: cell.name })))
  .catch((error) => {
    console.error('Dispatch rejected:', error instanceof Error ? error.message : error);
    process.exitCode = 1;
  });

Do not make automatic failover unconditional. A WebKit shard cannot move to a cell that only has Chromium; an EU-only test identity cannot move to a US cell; a latency test becomes meaningless if it changes origin. Record both the requested constraints and the selected cell in the run manifest so the report explains what actually happened.

Pattern 2: make execution region-aware without cloning the suite

Regional projects should share test logic and vary only configuration. Playwright projects provide an explicit environment matrix, while tags travel into blob reports. Avoid branching on machine hostnames: it creates invisible behavior and makes local reproduction painful. Pass a small, validated environment contract instead.

The next two files are a complete example. The configuration refuses unknown regions or missing base URLs at startup. The test treats absent routing headers as a diagnosable edge case, attaches response evidence on failure, and rethrows the original error so CI cannot convert a capture problem into a pass.

// playwright.config.ts
import { defineConfig, devices } from '@playwright/test';

const regions = ['eu-west', 'us-east'] as const;
type Region = (typeof regions)[number];
const requested = (process.env.TEST_REGIONS ?? 'eu-west').split(',').filter(Boolean);

function baseURL(region: Region): string {
  const key = 'BASE_URL_' + region.replace('-', '_').toUpperCase();
  const value = process.env[key];
  if (!value) throw new Error('Missing required environment variable ' + key);
  try { return new URL(value).toString(); }
  catch (error) { throw new Error(key + ' is not a valid URL', { cause: error }); }
}

for (const region of requested) {
  if (!regions.includes(region as Region)) {
    throw new Error('Unsupported region "' + region + '"; allowed: ' + regions.join(', '));
  }
}

export default defineConfig({
  testDir: './e2e',
  timeout: 45_000,
  expect: { timeout: 8_000 },
  retries: process.env.CI ? 1 : 0,
  reporter: process.env.CI ? [['blob'], ['line']] : [['html', { open: 'never' }]],
  projects: requested.map((value) => {
    const region = value as Region;
    return {
      name: region + '-chromium',
      metadata: { region },
      use: {
        ...devices['Desktop Chrome'],
        baseURL: baseURL(region),
        extraHTTPHeaders: { 'x-test-region': region },
        trace: 'retain-on-failure' as const,
      },
    };
  }),
});

// e2e/regional-catalog.spec.ts
import { expect, test } from '@playwright/test';

test('catalog is served by the requested region', async ({ request }, testInfo) => {
  const region = String(testInfo.project.metadata.region ?? '');
  if (!region) throw new Error('Project metadata.region is required');

  let status: number | undefined;
  let headers: Record<string, string> = {};
  try {
    const response = await request.get('/api/catalog', { timeout: 10_000 });
    status = response.status();
    headers = response.headers();
    const body = await response.text();
    await testInfo.attach('catalog-response', { body, contentType: 'application/json' });
    expect(response.ok(), 'catalog request failed with ' + status).toBeTruthy();
    expect(headers['x-served-region'], 'routing evidence header is missing').toBe(region);
  } catch (error) {
    await testInfo.attach('routing-diagnostics', {
      body: JSON.stringify({ region, status, headers }, null, 2),
      contentType: 'application/json',
    });
    throw error; // Preserve failure semantics after collecting evidence.
  }
});

Playwright's official sharding documentation recommends blob reports for merging shards; blob files include test results and attachments, and shard numbers prevent name clashes. Pin the same Playwright version and container digest in every cell. Different versions or test roots can make otherwise valid blobs incompatible or ambiguous during merge.

There is another gotcha: sharding and environment matrices are not the same operation. A shard divides one logical suite; a project intentionally repeats behavior in a different environment. Include region and browser in the logical shard ID, or two cells can upload different evidence under the same key. See how to run Playwright in CI before adding the multi-region layer.

How do leases prevent duplicate distributed test execution?

A lease grants temporary ownership; an attempt ID makes writes idempotent, so expired workers cannot overwrite evidence from the accepted retry.

Queues usually promise at-least-once delivery, not exactly-once execution. A worker can finish a test, upload its trace, and crash before acknowledging the message. The queue redelivers it. Exactly-once side effects across browsers, APIs, queues, and storage would require a transaction boundary they do not share. Design for replay instead.

Give each assignment an attempt ID and a short lease. Workers heartbeat while active. The reconciler may issue a new attempt after expiry, but completion is accepted only if its attempt still owns the lease. Artifact uploads use keys such asruns/run-42/eu-west/3/attempt-a/blob.zip, never a shared mutable filename. If a stale worker returns late, retain its evidence for debugging but do not let it change the canonical verdict.

Pattern 3: reconcile evidence before producing one global verdict

Report merging is a consistency problem. The control plane knows the expected shard set; object storage knows which artifacts arrived. A trustworthy merger compares the two. It must never infer “no failures” from “no report.” Missing evidence is an infrastructure failure with a distinct exit code, not a passing test run.

This Node 20+ script validates a manifest, rejects duplicate logical shards, checks ZIP signatures to catch truncated downloads, and only then invokes Playwright. The empty manifest edge case is rejected because merging nothing can otherwise create a convincing but meaningless artifact.

// scripts/merge-global-reports.ts
// Run: npx tsx scripts/merge-global-reports.ts artifacts/manifest.json artifacts/blobs
import { execFile } from 'node:child_process';
import { readFile, readdir } from 'node:fs/promises';
import { promisify } from 'node:util';
import path from 'node:path';

const execFileAsync = promisify(execFile);
type Entry = { shardId: string; file: string; sha256?: string };

async function main(manifestPath: string, blobDir: string) {
  if (!manifestPath || !blobDir) throw new Error('Usage: merge-global-reports <manifest> <blob-dir>');
  let expected: Entry[];
  try {
    expected = JSON.parse(await readFile(manifestPath, 'utf8')) as Entry[];
  } catch (error) {
    throw new Error('Manifest is unreadable or invalid JSON', { cause: error });
  }
  if (!Array.isArray(expected) || expected.length === 0) throw new Error('Manifest has no expected shards');

  const ids = new Set<string>();
  for (const item of expected) {
    if (!item.shardId || !item.file) throw new Error('Every manifest entry needs shardId and file');
    if (ids.has(item.shardId)) throw new Error('Duplicate logical shard: ' + item.shardId);
    ids.add(item.shardId);
  }

  const actual = new Set(await readdir(blobDir));
  const missing = expected.filter((item) => !actual.has(item.file));
  if (missing.length) {
    throw new Error('Missing shard evidence: ' + missing.map((item) => item.shardId).join(', '));
  }

  for (const item of expected) {
    const bytes = await readFile(path.join(blobDir, item.file));
    const isZip = bytes.length >= 4 && bytes[0] === 0x50 && bytes[1] === 0x4b;
    if (!isZip) throw new Error('Corrupt or truncated blob for shard ' + item.shardId);
  }

  try {
    const { stdout, stderr } = await execFileAsync(
      process.platform === 'win32' ? 'npx.cmd' : 'npx',
      ['playwright', 'merge-reports', '--reporter=html', blobDir],
      { timeout: 120_000, maxBuffer: 10 * 1024 * 1024 },
    );
    if (stdout) console.log(stdout);
    if (stderr) console.error(stderr);
  } catch (error) {
    throw new Error('Playwright could not merge validated blob reports', { cause: error });
  }
}

main(process.argv[2], process.argv[3]).catch((error) => {
  console.error(error instanceof Error ? error.stack : error);
  process.exitCode = 2; // Distinguish infrastructure failure from test failure.
});

Production code should also verify the optional SHA-256 in the manifest after download; the example checks the file signature to stay focused. Store manifests and artifacts with retention rules, encryption, and least-privilege credentials. Browser traces can contain tokens, user data, request bodies, and screenshots. A global test system is also a global data-handling system.

Build the migration in four safe stages

First, freeze the run. Generate an immutable manifest containing commit SHA, test list, Playwright version, container digest, projects, shard count, and configuration hash. A retry reads that manifest; it does not rediscover tests from a branch that may have moved.

Second, add one remote cell while keeping your current CI job as the control. Run a small, side-effect-safe slice in shadow mode and compare evidence manually. Do not gate releases yet. You are validating identity, networking, time zones, DNS, certificates, artifact upload, and secret scope—not just test assertions.

Third, introduce leases and reconciliation before adding more regions. Scale amplifies ambiguous ownership. Simulate a worker killed after upload but before acknowledgment, a delayed completion event, a duplicate queue delivery, an expired signed URL, and one missing blob. The expected outcome should be deterministic for every case.

Fourth, make the global verdict release-relevant. Define required cells for each change. A localization-only patch may not require the payment sandbox; an authentication change probably requires every residency boundary. Policy belongs in versioned code, and the report should state which rule selected each environment.

Troubleshooting distributed Playwright failures

Debug from durable facts, not from the last console line. Start with run ID, manifest hash, logical shard ID, attempt ID, cell, worker, application release, and artifact key. Carry those fields through queue headers, structured logs, traces, and reports. Timestamps alone are not correlation IDs, especially when clocks drift.

• Shard stays running after the worker vanished: inspect lease expiry and heartbeat age. If expiry keeps extending, fence heartbeats by attempt ID so a stale process cannot renew a replacement's lease.
• Retry passes in another region: compare assignment constraints and manifest hashes. Pin retries to the original cell for product diagnosis; fail over only after classifying an infrastructure outage.
• Merged report omits tests: compare expected logical shard IDs with object keys before calling merge-reports. Check that every cell uses the same Playwright version, testDir, tag format, and container digest.
• Tests fail only around midnight: log application time zone, worker time zone, locale, and a server-provided clock. Freeze time where possible and avoid deriving regional expectations from the runner clock.
• Healthy cell rejects every job: separate liveness from readiness. A process can answer HTTP while its browser image, secret mount, queue permission, or artifact bucket is unavailable.
• Artifacts upload but cannot merge: verify content length and SHA-256, not merely object existence. Multipart or interrupted uploads can leave a key that is not a valid blob ZIP.

Edge cases that deserve an architecture test

Drain a cell while it owns leases. Rotate a secret halfway through a run. Remove a test after the manifest is frozen. Let the control plane become temporarily unavailable while workers continue. Deliver completion twice and out of order. Make one region return a localized consent screen. Exhaust artifact storage. Give two reports the same filename. These are not exotic; they are the ordinary seams exposed by distribution.

Also decide what “degraded” means. If an optional browser cell is down, you may publish a qualified result. If a required residency check is absent, the correct verdict is blocked, not failed and certainly not passed. Use separate states—passed, product-failed, infrastructure-failed, blocked, and cancelled—so dashboards and release automation do not collapse unlike situations into red or green.

The professional habit: preserve truth under partial failure

Global infrastructure is not defined by the number of clouds on a diagram. It is defined by what happens when only half the diagram works. A control plane that freezes intent, regional cells that contain failure, deterministic routing, leased ownership, idempotent artifacts, and manifest-based reconciliation give you a system whose verdict still means something on a bad day.

Start with two cells and one critical journey. Kill a worker at the worst possible moment. Prove that the retry is safe and that missing evidence cannot become green. That exercise will teach you more about professional distributed systems than adding fifty parallel jobs to a pipeline—and it gives every future test a foundation you can trust.