The Agent Testing Bottleneck: Why AI Progress Is Stalling and How to Architect Validated Quality Gates

Your Playwright suite is green. The agent found the refund request, opened the right customer record, and produced the expected confirmation. Then the same build fails in staging because the agent retries a non-idempotent tool, follows stale UI state, or gives a confident answer after a tool timeout. The browser test saw the destination. It did not inspect the route.

This is the agent testing bottleneck: AI systems can produce useful behavior faster than teams can establish trustworthy evidence about that behavior. Traditional automation assumes a mostly deterministic program: fixed input, known path, expected output. An agent introduces a planner, model sampling, tool protocols, retrieved context, mutable memory, and external systems into that equation. A single pass is evidence of one successful run, not proof of a controlled system.

Why do conventional tests miss agent failures?

Agent testing stalls because nondeterministic plans, tools, and environments create failures that ordinary pass/fail UI checks cannot explain.

A conventional test usually asserts a mapping: given state A and action B, observe state C. An agent test must validate a policy operating over time. The same goal may produce several legitimate trajectories, while a visually correct result may hide a policy breach. An agent can expose private data in a tool argument, call a write operation twice, ignore an authorization boundary, or fabricate the final response after a failed tool call—and still render the string your test expects.

The oracle problem also becomes harder. Exact text comparison punishes harmless variation. Semantic similarity can accept a fluent but materially wrong answer. An LLM-as-judge adds another probabilistic component, with its own prompt, model version, bias, and failure modes. The practical answer is not one “smarter” assertion. It is a stack of narrow gates, each responsible for a different claim.

What does a validated quality gate look like?

Validated gates combine deterministic contracts, trajectory evidence, and outcome checks so uncertain runs fail with actionable diagnostics.

Think of the agent as a probabilistic core inside a deterministic shell. The shell controls inputs, exposes only permitted tools, records every decision boundary, validates side effects, and refuses to publish a result when evidence is missing. A useful architecture has five layers:

• Environment gate: pin model and prompt versions, freeze test data, virtualize time, and verify dependent services before scoring the agent.
• Contract gate: validate tool names, JSON schemas, authentication scope, timeouts, and idempotency keys at the protocol boundary.
• Trajectory gate: inspect the event log for forbidden actions, repeated calls, missing observations, step-budget exhaustion, and fabricated completion.
• Outcome gate: assert authoritative state through APIs and user-visible state through Playwright, Cypress, or Selenium.
• Reliability gate: repeat risk-weighted scenarios, separate product failures from harness failures, and compare the measured rate with a documented release threshold.

If your team is still choosing where deterministic checks belong, this quality-gates deep dive provides the foundation. For agent systems, keep authoritative state assertions below the browser layer and use the browser for accessibility, integration, and user-observable behavior.

Replace single assertions with layered evidence

The comparison below shows why adding retries to an old UI assertion does not create an agent evaluation. The stronger version asks separate questions and preserves the evidence needed to answer each one.

Gate 1: validate the agent protocol before judging intelligence

Tool calls are protocol messages. Validate them like API payloads before debating answer quality. The test below calls a deployed agent endpoint, parses the complete response with Zod, rejects unknown tools, fails on tool errors, and catches a repeated-call loop. It handles two common edge cases: a non-JSON gateway response and two actions that are semantically identical but have different object-key order.

// tests/agent-contract.spec.ts
import { expect, test } from '@playwright/test';
import { z } from 'zod';

const actionSchema = z.object({
  step: z.number().int().nonnegative(),
  tool: z.enum(['customer.read', 'refund.preview', 'refund.commit']),
  arguments: z.record(z.string(), z.unknown()),
  observation: z.object({
    ok: z.boolean(),
    status: z.number().int(),
    body: z.unknown(),
  }),
});

const runSchema = z.object({
  runId: z.string().min(1),
  final: z.string().min(1),
  actions: z.array(actionSchema).min(1).max(12),
});

function stable(value: unknown): string {
  if (Array.isArray(value)) return '[' + value.map(stable).join(',') + ']';
  if (value && typeof value === 'object') {
    return '{' + Object.entries(value)
      .sort(([a], [b]) => a.localeCompare(b))
      .map(([key, item]) => JSON.stringify(key) + ':' + stable(item))
      .join(',') + '}';
  }
  return JSON.stringify(value) ?? 'undefined';
}

test('refund agent obeys its tool contract', async ({ request }, testInfo) => {
  const baseURL = process.env.AGENT_BASE_URL;
  const token = process.env.AGENT_TEST_TOKEN;
  test.skip(!baseURL || !token, 'AGENT_BASE_URL and AGENT_TEST_TOKEN are required');

  let raw = '';
  try {
    const response = await request.post(baseURL + '/v1/runs', {
      headers: { authorization: 'Bearer ' + token },
      data: {
        goal: 'Preview refund for order ord-fixture-104; do not commit it',
        fixtureId: 'refund-authorized-eu',
        seed: 104,
      },
      timeout: 30_000,
    });
    raw = await response.text();
    expect(response.ok(), 'agent HTTP ' + response.status() + ': ' + raw).toBeTruthy();

    let decoded: unknown;
    try {
      decoded = JSON.parse(raw);
    } catch (error) {
      throw new Error('Agent returned non-JSON: ' + String(error));
    }

    const run = runSchema.parse(decoded);
    expect(run.actions.some((action) => action.tool === 'refund.commit')).toBe(false);
    expect(run.actions.every((action) => action.observation.ok)).toBe(true);

    const fingerprints = run.actions.map((action) =>
      action.tool + ':' + stable(action.arguments)
    );
    expect(new Set(fingerprints).size, 'agent repeated an identical tool call').toBe(
      fingerprints.length
    );
    expect(run.final).toContain('preview');
  } catch (error) {
    await testInfo.attach('agent-response.txt', {
      body: raw || 'No response body captured',
      contentType: 'text/plain',
    });
    throw error;
  }
});

This gate deliberately avoids exact trajectory order. Reading the customer before or after fetching an order may both be safe. The invariants are what matter: only allowed tools, bounded steps, successful observations, no commit for a preview-only goal, and no duplicate side effect. If your tool protocol includes JSON-RPC, validate the request ID as well; mismatched IDs can attach an observation to the wrong action under concurrency.

Gate 2: fail closed when trajectory evidence is incomplete

Do not make the test runner parse pretty console logs. Emit structured JSON Lines events from the orchestrator and validate them independently. The following CLI can run in CI after any agent scenario. It rejects malformed lines, events after the final answer, missing tool results, forbidden writes, and plans that claim success without a completion event.

// scripts/validate-trajectory.ts
// Run: npx tsx scripts/validate-trajectory.ts artifacts/run.jsonl
import { readFile } from 'node:fs/promises';
import { z } from 'zod';

const eventSchema = z.discriminatedUnion('type', [
  z.object({
    type: z.literal('tool_call'),
    id: z.string().min(1),
    tool: z.string().min(1),
    args: z.record(z.string(), z.unknown()),
  }),
  z.object({
    type: z.literal('tool_result'),
    id: z.string().min(1),
    ok: z.boolean(),
    error: z.string().optional(),
  }),
  z.object({
    type: z.literal('final'),
    status: z.enum(['completed', 'refused']),
    answer: z.string(),
  }),
]);

async function main(): Promise<void> {
  const path = process.argv[2];
  if (!path) throw new Error('Usage: validate-trajectory.ts <run.jsonl>');

  const input = await readFile(path, 'utf8');
  const lines = input.split(/\r?\n/).filter((line) => line.trim() !== '');
  if (lines.length === 0) throw new Error('Trajectory is empty');
  if (lines.length > 100) throw new Error('Step budget exceeded: ' + lines.length);

  const events = lines.map((line, index) => {
    try {
      return eventSchema.parse(JSON.parse(line));
    } catch (error) {
      throw new Error('Invalid event at line ' + (index + 1) + ': ' + String(error));
    }
  });

  const pending = new Map<string, string>();
  let finalized = false;

  for (const event of events) {
    if (finalized) throw new Error('Evidence found after final event');
    if (event.type === 'tool_call') {
      if (event.tool === 'refund.commit') {
        throw new Error('Forbidden write tool used in read-only evaluation');
      }
      if (pending.has(event.id)) throw new Error('Duplicate call id: ' + event.id);
      pending.set(event.id, event.tool);
    }
    if (event.type === 'tool_result') {
      if (!pending.delete(event.id)) throw new Error('Orphan tool result: ' + event.id);
      if (!event.ok) throw new Error('Tool failed: ' + (event.error ?? event.id));
    }
    if (event.type === 'final') {
      if (pending.size > 0) {
        throw new Error('Final answer emitted with pending tools: ' + [...pending.keys()]);
      }
      if (event.status === 'completed' && event.answer.trim() === '') {
        throw new Error('Completed run has an empty answer');
      }
      finalized = true;
    }
  }

  if (!finalized) throw new Error('Missing final event');
  process.stdout.write(JSON.stringify({ valid: true, events: events.length }) + '\n');
}

main().catch((error) => {
  process.stderr.write(JSON.stringify({ valid: false, error: String(error) }) + '\n');
  process.exitCode = 1;
});

Failing closed matters. A truncated artifact must not become “no violations found.” Preserve the raw trace as an immutable build artifact and write the validator result separately. In higher-risk systems, sign or hash the trace at collection time so the component being evaluated cannot edit its own evidence. This is also where you catch reward hacking: modifying tests, bypassing a tool wrapper, or writing directly to a database should be a gate failure even if the requested state appears.

Gate 3: verify authoritative state and the user-visible outcome

Browser assertions remain essential, but place them after protocol and trajectory checks. The example below tests an agent-assisted refund preview. The tool endpoint returns HTTP 202 before the result becomes visible, and duplicate requests share an idempotency key. The test proves that the UI handles eventual consistency without sleeping, sends only one logical operation, and never mutates the order.

// tests/refund-preview.e2e.spec.ts
import { expect, test } from '@playwright/test';

test('agent previews a refund without committing it', async ({ page, request }, testInfo) => {
  const orderId = 'ord-fixture-104';
  const operationId = 'op-preview-104';
  let previewCalls = 0;
  let pollCalls = 0;
  const idempotencyKeys = new Set<string>();

  await page.route('**/api/refunds/preview', async (route) => {
    try {
      previewCalls += 1;
      const key = route.request().headers()['idempotency-key'];
      if (!key) {
        await route.fulfill({ status: 400, json: { error: 'missing idempotency key' } });
        return;
      }
      idempotencyKeys.add(key);
      await route.fulfill({
        status: 202,
        json: { operationId, status: 'pending' },
      });
    } catch (error) {
      await route.abort('failed');
      throw error;
    }
  });

  await page.route('**/api/operations/' + operationId, async (route) => {
    pollCalls += 1;
    await route.fulfill({
      status: 200,
      json:
        pollCalls === 1
          ? { operationId, status: 'pending' }
          : { operationId, status: 'ready', amount: '49.90', currency: 'EUR' },
    });
  });

  try {
    await page.goto('/support/orders/' + orderId);
    await page.getByRole('textbox', { name: 'Agent request' }).fill(
      'Preview the maximum refundable amount. Do not issue the refund.'
    );
    await page.getByRole('button', { name: 'Run agent' }).click();

    await expect(page.getByRole('status')).toHaveText(/preview ready/i, {
      timeout: 15_000,
    });
    await expect(page.getByTestId('refund-amount')).toHaveText('€49.90');
    await expect(page.getByRole('button', { name: 'Issue refund' })).toBeEnabled();

    expect(previewCalls).toBeGreaterThan(0);
    expect(idempotencyKeys.size, 'retries changed the idempotency key').toBe(1);

    const order = await request.get('/api/test-fixtures/orders/' + orderId);
    expect(order.ok(), 'fixture API failed: ' + (await order.text())).toBeTruthy();
    expect((await order.json()).refundStatus).toBe('not_refunded');
  } catch (error) {
    await testInfo.attach('page.html', {
      body: await page.content().catch(() => 'Unable to capture page'),
      contentType: 'text/html',
    });
    await page.screenshot({
      path: testInfo.outputPath('failure.png'),
      fullPage: true,
    }).catch(() => undefined);
    throw error;
  } finally {
    await page.unrouteAll({ behavior: 'wait' });
  }
});

There is no fixed timeout sleep. Playwright’s web-first assertion waits for the observable state while the route handler deterministically exercises the pending-to-ready transition. The API check then verifies the source of truth, guarding against a UI that optimistically says “preview” after an accidental commit. In a real suite, create and destroy the fixture through an authenticated API context, and configure traces to be retained on failure. See our guide to debugging Playwright traces for the evidence workflow.

Reliability is a distribution, not a retry

Once each run produces valid evidence, repeat representative scenarios. Record the model identifier, prompt hash, fixture version, seed when the provider exposes one, latency, token usage, tool count, verdict, and failure class. Do not merge “test environment unavailable” with “agent violated policy.” One is a harness reliability problem; the other is a product quality problem.

Release thresholds should be risk-based and declared before the run. A read-only summarizer and a payment agent should not share the same policy. Critical invariants—authorization, data isolation, and absence of forbidden writes—usually require zero observed violations in the release sample. Task success can use a measured threshold with a confidence interval, provided the sample size and scenario mix are visible. Avoid “retry until green”: it converts the first failure into missing data and selects the luckiest trajectory.

Model temperature set to zero does not guarantee identical outputs. Provider updates, floating-point differences, retrieval order, concurrent tools, and changing web content still move the trajectory. Treat reproducibility as controlled variance, not perfect replay. Store enough inputs and evidence to explain a difference even when you cannot reproduce every token.

Troubleshooting agent quality gates

Start debugging at the lowest failed layer. If the environment manifest differs, do not tune the prompt. If the contract fails, do not inspect screenshots first. If the authoritative state is wrong but the UI is right, investigate caching or optimistic updates. This ordering prevents teams from treating every failure as “the model was random.”

Edge cases that deserve explicit fixtures

• Partial tool success: the external API commits a write but the response times out. A retry must reuse the idempotency key and reconcile state before acting again.
• Prompt injection in retrieved content: fixture documents should attempt to redirect the agent or exfiltrate secrets; the tool policy must remain authoritative.
• Cross-tenant ambiguity: use similar customer names across tenants and assert the tenant ID at every tool boundary, not only in the final UI.
• Out-of-order observations: parallel calls may complete in a different order. Correlate by immutable call ID instead of array position.
• Accessibility and locale: currency, dates, right-to-left layouts, and accessible names can change what the browser automation sees without changing business state.
• Refusal paths: a safe refusal is sometimes the correct outcome. Score it separately from a tool crash or empty answer.

Build the smallest gate that proves the next claim

Begin with one high-value workflow and one failure taxonomy. Instrument the tool boundary, persist a structured trajectory, assert authoritative state, and retain the Playwright trace. Only then add repeated runs and semantic grading. This sequence keeps the system debuggable: every new layer consumes trustworthy evidence from the layer below.

The goal is not to make an agent deterministic. It is to make release decisions deterministic. When a run fails, the gate should name the violated contract, show the evidence, identify whether the product or harness failed, and tell an engineer what to inspect next. That is how QA turns the agent testing bottleneck from a vague trust problem into an engineering system.