Level Up Your Load Testing: Building PS6-Scale Distributed Infrastructure with Kubernetes and Playwright

Your first load test probably looked sensible: open a page, loop a request, watch the terminal, then celebrate when the server stayed up. That is useful exploration. It is not yet launch infrastructure. A console-scale release—our deliberately hypothetical “PS6 launch” scenario—mixes sign-in, queues, regional CDNs, inventory, checkout, WebSockets, third-party identity, and impatient humans. The hard question is no longer “can I generate traffic?” It is “can I generate controlled, representative demand without the generator becoming the bottleneck or corrupting the result?”

This guide is the bridge from a local script to a professional test plane. Playwright supplies real browser behavior; Kubernetes supplies scheduling, isolation, retries, and finite work; your telemetry supplies the verdict. If you are still choosing which journeys deserve browser coverage, start with our Playwright end-to-end guide, then return here to distribute them.

What does PS6-scale load testing actually require?

It requires representative journeys, bounded concurrency, deterministic shards, clear aborts, and generators proven faster than the target.

“PS6-scale” is a design prompt, not a traffic forecast or a claim about Sony. Before touching YAML, write a workload contract: which journeys run, their relative weights, regions, ramp shape, test duration, success criteria, and abort conditions. Separate arrival rate—new journeys started per second—from concurrency—journeys currently in flight. If a checkout takes longer under stress, closed-loop workers complete fewer iterations and quietly reduce offered load. An open-loop controller preserves arrivals, but must shed or queue work when the generator saturates. Choose consciously.

Kubernetes documents design limits of 5,000 nodes, 110 pods per node, 150,000 total pods, and 300,000 total containers for current large-cluster configurations. Those are platform ceilings, not a sizing recommendation. They also expose a useful truth: control-plane objects, IP addresses, log streams, and image pulls are load too. Meanwhile, Playwright isolates tests with fast, incognito-like BrowserContexts, each with separate cookies and storage. Contexts reduce cross-user contamination; they do not make Chromium free.

Move from a script to a layered test plane

Keep four layers separate. The scenario layer owns user intent and assertions. The worker layer owns browsers, concurrency, deadlines, and graceful shutdown. The scheduler layer assigns stable shards and resources. The evidence layer stores metrics, traces, logs, and a manifest describing exactly what ran. That separation lets you replace one tool without rewriting the experiment.

Example 1: make one browser journey honest

Start with a scenario that distinguishes product failure from infrastructure failure. This runnable TypeScript file uses Playwright directly, validates configuration, bounds every navigation, treats expected queue responses explicitly, and always closes its context. It also handles the edge case where a sold-out result is valid business behavior rather than a failed test.

// scenarios/reserve-console.ts
import { chromium, type Browser } from 'playwright';

const baseURL = process.env.BASE_URL;
if (!baseURL || !/^https:///.test(baseURL)) {
  throw new Error('BASE_URL must be an https URL');
}

async function reserveConsole(browser: Browser, userId: string) {
  const context = await browser.newContext({
    extraHTTPHeaders: { 'x-load-test': 'ps6-rehearsal' },
  });
  const page = await context.newPage();
  page.setDefaultTimeout(10_000);

  try {
    const response = await page.goto(baseURL, {
      waitUntil: 'domcontentloaded', timeout: 20_000,
    });
    if (!response) throw new Error('Navigation produced no HTTP response');
    if (![200, 202].includes(response.status())) {
      throw new Error(`Unexpected landing status: ${response.status()}`);
    }

    await page.getByTestId('email').fill(`load+${userId}@example.test`);
    await page.getByTestId('join-queue').click();

    const outcome = page.getByTestId('reservation-confirmed')
      .or(page.getByTestId('sold-out'));
    await outcome.waitFor({ state: 'visible', timeout: 15_000 });

    return await page.getByTestId('sold-out').isVisible()
      ? { outcome: 'sold-out' as const }
      : { outcome: 'reserved' as const };
  } catch (error) {
    await page.screenshot({ path: `artifacts/failure-${userId}.png`, fullPage: true })
      .catch(() => undefined); // Never mask the original failure.
    throw new Error(`Journey failed for ${userId}`, { cause: error });
  } finally {
    await context.close().catch(() => undefined);
  }
}

const browser = await chromium.launch({ headless: true });
try {
  const userId = process.env.USER_ID ?? crypto.randomUUID();
  console.log(JSON.stringify(await reserveConsole(browser, userId)));
} finally {
  await browser.close();
}

Do not retry the whole journey merely because the assertion failed; that changes the measured workload and can turn one reservation into two. Retry only operations you know are idempotent, and give each write a unique idempotency key understood by the application. Test accounts must be synthetic, rate-limit rules must be agreed, and production runs require a kill switch owned by an identifiable operator.

Example 2: build a bounded worker, not a browser bomb

A pod should translate one stable shard index into bounded local work. This worker rejects malformed environment variables, stops accepting journeys on SIGTERM, drains in-flight promises, and returns a non-zero exit when any journey fails. The partial final batch is an important edge case: a loop that always launches the configured concurrency overshoots its assigned iterations.

// workers/run-shard.ts
import { chromium } from 'playwright';

function positiveInt(name: string, fallback?: number): number {
  const raw = process.env[name] ?? (fallback === undefined ? '' : String(fallback));
  const value = Number(raw);
  if (!Number.isSafeInteger(value) || value < 1) {
    throw new Error(`${name} must be a positive integer; received "${raw}"`);
  }
  return value;
}

const shard = Number(process.env.JOB_COMPLETION_INDEX);
const totalShards = positiveInt('TOTAL_SHARDS');
const iterations = positiveInt('ITERATIONS_PER_SHARD');
const concurrency = positiveInt('CONCURRENCY', 2);
if (!Number.isInteger(shard) || shard < 0 || shard >= totalShards) {
  throw new Error(`Invalid shard ${shard}; expected 0..${totalShards - 1}`);
}

let stopping = false;
process.once('SIGTERM', () => { stopping = true; });
const browser = await chromium.launch({ headless: true });
let failures = 0;

async function runUser(sequence: number) {
  const context = await browser.newContext();
  const page = await context.newPage();
  try {
    const id = `s${shard}-i${sequence}`;
    const response = await page.goto(`${process.env.BASE_URL}/queue?testUser=${id}`, {
      waitUntil: 'domcontentloaded', timeout: 20_000,
    });
    if (!response || response.status() >= 500) {
      throw new Error(`Bad response: ${response?.status() ?? 'none'}`);
    }
    await page.getByTestId('queue-status').waitFor({ timeout: 10_000 });
  } catch (error) {
    failures += 1;
    console.error(JSON.stringify({ shard, sequence, error: String(error) }));
  } finally {
    await context.close().catch(() => undefined);
  }
}

try {
  for (let offset = 0; offset < iterations && !stopping; offset += concurrency) {
    const batchSize = Math.min(concurrency, iterations - offset);
    await Promise.all(Array.from({ length: batchSize }, (_, i) => runUser(offset + i)));
  }
} finally {
  await browser.close().catch(() => undefined);
}

console.log(JSON.stringify({ shard, iterations, failures, stopped: stopping }));
if (stopping || failures > 0) process.exitCode = 1;

This is closed-loop by design: each slot starts its next journey only after the previous one ends. For arrival-rate testing, place a token scheduler in front and record dropped starts when it cannot keep pace. Never “catch up” by launching an unbounded burst; that measures your backlog implementation, not the intended ramp.

How should Kubernetes distribute Playwright load?

Use an Indexed Job for finite shards, explicit resources and deadlines, immutable images, topology spreading, and per-index artifacts.

Kubernetes Indexed Jobs assign each completion a stable index through the JOB_COMPLETION_INDEX environment variable. Official Job documentation notes that the controller may occasionally start the same program twice, even with one completion and one parallel pod. Therefore IDs, artifact paths, and application writes must be idempotent. The manifest below also avoids the default retry surprise: Kubernetes documents a default Job backoffLimit of 6 and exponential delays capped at six minutes. Explicit policy makes the experiment reproducible.

# k8s/playwright-load-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: ps6-rehearsal
  labels: { app: playwright-load, run-id: rehearsal-20260818 }
spec:
  completions: 24
  parallelism: 8
  completionMode: Indexed
  backoffLimitPerIndex: 1
  maxFailedIndexes: 2
  activeDeadlineSeconds: 1800
  ttlSecondsAfterFinished: 3600
  podFailurePolicy:
    rules:
      - action: FailJob
        onExitCodes: { containerName: runner, operator: In, values: [78] }
      - action: Ignore
        onPodConditions: [{ type: DisruptionTarget }]
  template:
    metadata:
      labels: { app: playwright-load }
    spec:
      restartPolicy: Never
      terminationGracePeriodSeconds: 45
      serviceAccountName: load-runner
      containers:
        - name: runner
          image: ghcr.io/example/playwright-load@sha256:REPLACE_WITH_DIGEST
          command: ['node', 'dist/workers/run-shard.js']
          env:
            - { name: BASE_URL, value: 'https://staging.example.com' }
            - { name: TOTAL_SHARDS, value: '24' }
            - { name: ITERATIONS_PER_SHARD, value: '250' }
            - { name: CONCURRENCY, value: '2' }
          resources:
            requests: { cpu: '2', memory: 3Gi, ephemeral-storage: 2Gi }
            limits: { cpu: '2', memory: 3Gi, ephemeral-storage: 4Gi }
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            capabilities: { drop: ['ALL'] }
          volumeMounts:
            - { name: tmp, mountPath: /tmp }
            - { name: artifacts, mountPath: /app/artifacts }
      volumes:
        - name: tmp
          emptyDir: { sizeLimit: 1Gi }
        - name: artifacts
          emptyDir: { sizeLimit: 2Gi }
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector: { matchLabels: { app: playwright-load } }

Replace the image placeholder with a real immutable digest; failing to do so should be a deployment-time validation error. The emptyDir artifact volume disappears with the pod, so a production image needs a sidecar or shutdown hook that uploads results to object storage using a shard-and-run key. Upload to a temporary key, verify checksum, then publish the final key. If a pod is duplicated, conditional writes prevent the late copy from overwriting good evidence.

Avoid mounting cloud credentials as static environment variables. Prefer workload identity with write access limited to one run prefix. Apply a dedicated namespace, ResourceQuota, NetworkPolicy, Pod Security controls, and a PriorityClass lower than the application under test. A load generator must never evict the service it is meant to measure. For a practical preflight checklist, see how to run Playwright safely on Kubernetes.

Example 3: gate the run before creating expensive pods

A tiny preflight script prevents the most embarrassing failures: testing localhost from inside the cluster, hitting production by accident, using an unresolved DNS name, or accepting a health endpoint that returns a login page. It includes timeouts, redirect control, content validation, and DNS error reporting.

// scripts/preflight.ts (Node 20+)
import { lookup } from 'node:dns/promises';

const raw = process.env.BASE_URL;
if (!raw) throw new Error('BASE_URL is required');
const target = new URL(raw);
if (target.protocol !== 'https:') throw new Error('Refusing a non-HTTPS target');
if (['localhost', '127.0.0.1', '::1'].includes(target.hostname)) {
  throw new Error('localhost inside a pod is the pod, not your application');
}
if (target.hostname === 'www.example.com' && process.env.ALLOW_PRODUCTION !== 'yes') {
  throw new Error('Production target requires ALLOW_PRODUCTION=yes');
}

try {
  const addresses = await lookup(target.hostname, { all: true });
  if (addresses.length === 0) throw new Error('DNS returned no addresses');

  const response = await fetch(new URL('/health/ready', target), {
    redirect: 'manual',
    signal: AbortSignal.timeout(5_000),
    headers: { 'x-load-test-preflight': 'true' },
  });
  if (response.status !== 200) {
    throw new Error(`Readiness returned ${response.status}`);
  }
  const body = await response.json() as { ready?: unknown };
  if (body.ready !== true) throw new Error('Readiness body did not contain ready=true');
  console.log(JSON.stringify({ ok: true, hostname: target.hostname, addresses }));
} catch (error) {
  console.error(JSON.stringify({ ok: false, target: target.origin, error: String(error) }));
  process.exitCode = 78; // Manifest treats configuration failures as FailJob.
}

Run this once from the same namespace and network policy as the workers, then create the Job only if it succeeds. Edge cases matter: split-horizon DNS may resolve differently outside the cluster; a service mesh may require sidecar readiness; a 302 redirect may hide missing authentication; and an IPv6 address does not prove the pod has an IPv6 route.

Observe both sides of the experiment

Application dashboards alone cannot tell you whether Playwright stalled. Record generator CPU throttling, working-set memory, OOM kills, event-loop lag, DNS time, connect time, TLS time, browser navigation timing, active contexts, scheduled versus started journeys, and artifact upload failures. On the system under test, correlate request rate, queue depth, dependency latency, saturation, and business outcomes using a unique run ID—not user email or other personal data.

Treat coordinated omission as a first-class risk. A blocked closed-loop user stops generating new work exactly when the service slows, making the worst interval look quieter. Preserve scheduled timestamps in open-loop runs and measure lateness. For browser tests, use Playwright traces selectively: traces and video consume CPU, disk, and upload bandwidth. A representative sample plus traces-on-failure is usually more honest than tracing every journey.

Playwright’s official sharding guide recommends the blob reporter because shard reports retain test results and attachments and can be merged with npx playwright merge-reports. Use a unique run ID and shard index in every blob filename. Refuse to merge if expected shards are missing; a beautiful partial report is not a successful load test.

Troubleshooting: when the graph lies or the pods stall

• Pods stay Pending. Run kubectl describe pod and inspect Events. Look for insufficient CPU or memory, untolerated taints, topology constraints, quota, and unavailable node IPs. Do not lower requests until measurement proves they are inflated.
• Chromium exits with OOMKilled or signal 9. Compare container working set with the memory limit and inspect lastState.terminated. Reduce contexts per worker, disable video, bound traces, or raise memory. Retrying unchanged only repeats the generator failure.
• Results improve as load rises. Check scheduled versus started journeys. Your closed-loop generator may be backing off unintentionally, or generator CPU throttling may suppress arrivals. Add event-loop lag and start-lateness histograms.
• Every shard tests the same users. Confirm JOB_COMPLETION_INDEX is present and in range, and include it in deterministic IDs. Never default a missing index to zero; fail fast.
• Only some shards have reports. Inspect Job failedIndexes, pod termination reasons, artifact sidecar logs, and object-store conditional-write errors. Merge only after the expected index set is complete.
• DNS or TLS spikes dominate. Compare pod-local DNS metrics, CoreDNS saturation, connection reuse, certificate chains, and service-mesh telemetry. Browser DNS caches can make later iterations differ from cold users.
• 429 responses appear. Decide before the run whether rate limiting is the subject, a safety rail, or noise. Record Retry-After; do not automatically retry writes, and do not bypass protection without explicit authorization.
• Shutdown loses evidence. Ensure the worker handles SIGTERM, stops admission, drains within terminationGracePeriodSeconds, and uploads atomically. Test node drain before the real rehearsal.

Gotchas that only appear at scale

NAT port exhaustion, cloud API quotas, pod IP scarcity, registry throttling, CoreDNS saturation, and log ingestion limits can all cap the generator before the application. Browser clock skew can corrupt cross-pod timing. Shared accounts serialize through locks. Third-party CAPTCHA and fraud systems may block synthetic users. CDN caches can make a single-region generator unrealistically warm. A test that crosses regions also measures the internet; label results by source region.

Warm the infrastructure, not the application result you want to measure. Pre-pull the immutable browser image and provision nodes before the ramp, but decide whether DNS, TLS sessions, CDN objects, and application caches should be cold or warm based on the launch hypothesis. Keep a control shard at low load: if it degrades alongside high-load shards, you have evidence of a shared bottleneck rather than a scenario bug.

Your level-up runbook

• Model a few revenue- or trust-critical journeys; send bulk background traffic with a protocol-level tool.
• Validate targets, authorization, synthetic data, safety limits, abort ownership, and the cleanup plan.
• Benchmark one pod until you find its safe concurrency below CPU, memory, disk, and event-loop saturation.
• Choose shard count from measured pod capacity, then preflight cluster quotas, IPs, nodes, registry, DNS, and storage.
• Run a small canary, verify both generator and application telemetry, then ramp in observable stages.
• Stop automatically on agreed safety conditions; preserve immutable configuration and complete shard evidence.
• Write the conclusion in terms of the hypothesis. “We launched 24 pods” is activity; “the queue objective held while generators remained healthy” is evidence.

You do not need hyperscale on day one. You need a system whose next step is obvious: one honest journey, one bounded worker, one deterministic shard, then a fleet. That is the real level up—from generating noise to running an experiment your team can trust.