The Anatomy of an AI Agent: Architecting Reliable Quality Gates for Non-Deterministic Workflows

A conventional Playwright test is reassuringly binary: click a button, observe a state, pass or fail. An AI agent breaks that mental model. The same instruction can produce different plans, tool calls, wording, latency, and token usage on consecutive runs. A run may even reach the correct screen through an unsafe sequence. If your only assertion is “the record exists,” an agent can pass after editing the wrong customer, leaking data into a prompt, or repeating an irreversible action.

The practical problem is not making generation deterministic. It is making acceptance deterministic. Treat the agent as an untrusted proposal engine and put a typed, observable, fail-closed quality boundary between its reasoning loop and the systems it can change. This article develops that boundary for engineers who already know selectors, fixtures, API mocks, and CI, but now need to test decisions as well as code.

What is an AI agent, architecturally?

An AI agent observes state, chooses a tool action, records evidence, and stops only when a verifier accepts the resulting outcome.

An agent is not merely an LLM with function calling. It is a feedback system with five separable parts: an objective, an observation builder, a policy that proposes the next action, tools that change or inspect the environment, and a termination policy. Memory and retrieval enrich the observation, while guards constrain what tools may do. Reliability problems appear at the joins: stale observations, malformed tool arguments, partial writes, duplicated retries, and a model that declares victory without evidence.

This distinction matters because benchmark results measure end-to-end completion, not whether a model sounded confident. The original WebArena paper reported 14.41% end-to-end success for its best GPT-4-based agent versus 78.24% for humans. The original SWE-bench paper evaluated 2,294 issues from 12 Python repositories; its best reported baseline resolved 1.96%. These are historical results on specific scaffolds, not claims about today’s best models. They demonstrate a durable testing lesson: fluent output is not functional correctness.

Why do AI agent quality gates need multiple layers?

A reliable gate turns agent evidence into a deterministic pass, retry, or stop decision before any irreversible side effect can run.

No single assertion covers agent correctness. JSON Schema can prove shape, not truth. An LLM judge can assess meaning, but it is itself non-deterministic. A browser assertion can confirm visible state while missing an unauthorized API call. Build a conjunction of independent layers, ordered from cheapest and most objective to most contextual.

• Contract gate: parse typed output; reject missing, extra, or invalid fields.
• Policy gate: enforce tool allowlists, tenant scope, monetary limits, and human-approval requirements.
• Evidence gate: require fresh receipts from the system of record, not the agent’s narration.
• Semantic gate: score qualities such as relevance or completeness with a rubric and explicit uncertainty handling.
• Side-effect gate: require idempotency, authorization, and a current lease immediately before committing.

QA teams can extend patterns they already use. A tool schema resembles an API contract. A recorded trajectory resembles a Playwright trace. A model rubric resembles visual regression: valuable but tolerant, so it cannot replace exact invariants. For more on separating checks by risk, see the test automation pyramid beyond coverage percentages.

Code comparison: from hopeful assertions to a quality boundary

Example 1: make evidence a typed contract

The agent should never return a bare “done.” Require claims tied to machine-checkable evidence. This runnable TypeScript example validates unknown model output, rejects duplicate evidence IDs, treats stale receipts as a retryable failure, and fails closed on unexpected status values. Install Zod with npm i zod, then run it with npx tsx gate-contract.ts.

import { z } from 'zod';

const Evidence = z.object({
  id: z.string().min(1),
  kind: z.enum(['api-receipt', 'dom-assertion', 'policy-check']),
  observedAt: z.string().datetime(),
  subjectId: z.string().min(1),
  passed: z.boolean(),
}).strict();

const Candidate = z.object({
  runId: z.string().uuid(),
  status: z.enum(['proposed', 'completed']),
  targetTenantId: z.string().min(1),
  evidence: z.array(Evidence).min(1),
}).strict();

type Decision =
  | { kind: 'pass'; runId: string }
  | { kind: 'retry'; reason: string }
  | { kind: 'stop'; reason: string };

export function validateCandidate(
  raw: unknown,
  expectedTenant: string,
  now = Date.now(),
): Decision {
  const parsed = Candidate.safeParse(raw);
  if (!parsed.success) {
    return { kind: 'stop', reason: parsed.error.issues.map(i => i.message).join('; ') };
  }

  const value = parsed.data;
  if (value.targetTenantId !== expectedTenant) {
    return { kind: 'stop', reason: 'tenant boundary violation' };
  }

  const ids = value.evidence.map(item => item.id);
  if (new Set(ids).size !== ids.length) {
    return { kind: 'stop', reason: 'duplicate evidence cannot increase confidence' };
  }

  if (value.evidence.some(item => now - Date.parse(item.observedAt) > 60_000)) {
    return { kind: 'retry', reason: 'evidence is older than 60 seconds' };
  }

  if (value.status !== 'completed' || value.evidence.some(item => !item.passed)) {
    return { kind: 'stop', reason: 'completion claim is not supported' };
  }
  return { kind: 'pass', runId: value.runId };
}

try {
  const raw = JSON.parse(process.argv[2] ?? '{}') as unknown;
  console.log(validateCandidate(raw, process.env.TENANT_ID ?? 'tenant-demo'));
} catch (error) {
  console.error('Input was not valid JSON:', error instanceof Error ? error.message : error);
  process.exitCode = 1;
}

The key design choice is returning a discriminated decision rather than a boolean. “Retry” means the objective may still be valid but the observation is transient. “Stop” means repeating the same plan would be unsafe or pointless. That difference prevents the common failure mode where a generic catch block retries policy violations until a rate limit or budget finally stops it.

Example 2: verify a browser-agent trajectory with Playwright

Final-state assertions cannot reveal every unsafe intermediate action. Intercept requests, reject unapproved hosts and methods, assert selector uniqueness, and verify the server receipt. This Playwright test handles absent or ambiguous elements, network failures, and a success banner that appears without a matching backend record.

import { test, expect, type Page } from '@playwright/test';

type Step = { action: 'click' | 'fill'; testId: string; value?: string };

async function executePlan(page: Page, steps: Step[]): Promise<void> {
  for (const [index, step] of steps.entries()) {
    const target = page.getByTestId(step.testId);
    const count = await target.count();
    if (count !== 1) throw new Error('Step ' + index + ': expected one ' + step.testId + ', found ' + count);

    try {
      if (step.action === 'fill') {
        if (step.value === undefined) throw new Error('fill step has no value');
        await target.fill(step.value, { timeout: 5_000 });
      } else {
        await target.click({ timeout: 5_000 });
      }
    } catch (error) {
      throw new Error('Step ' + index + ' failed', { cause: error });
    }
  }
}

test('agent creates an invoice only inside the authorized tenant', async ({ page, request }) => {
  const violations: string[] = [];
  page.on('request', req => {
    const url = new URL(req.url());
    const mutates = !['GET', 'HEAD', 'OPTIONS'].includes(req.method());
    if (url.origin !== 'https://billing.test' || (mutates && !url.pathname.startsWith('/api/invoices'))) {
      violations.push(req.method() + ' ' + url.href);
    }
  });

  await page.goto('https://billing.test/tenants/acme/invoices');
  const plan: Step[] = [
    { action: 'click', testId: 'new-invoice' },
    { action: 'fill', testId: 'customer-id', value: 'customer-42' },
    { action: 'fill', testId: 'amount', value: '125.00' },
    { action: 'click', testId: 'save-invoice' },
  ];

  await executePlan(page, plan);
  await expect(page.getByRole('status')).toContainText('Invoice created');

  const response = await request.get('/api/invoices?tenant=acme&customer=customer-42');
  if (!response.ok()) throw new Error('Receipt lookup failed with ' + response.status());
  const body = await response.json() as { items?: Array<{ amount: number }> };
  expect(body.items).toHaveLength(1); // also catches accidental duplicate writes
  expect(body.items?.[0]?.amount).toBe(125);
  expect(violations, 'agent attempted an unauthorized request').toEqual([]);
});

In CI, retain the Playwright trace, screenshots, console messages, and intercepted request summary under the same runId used by the agent. A trace without prompt and model versions is incomplete; a prompt log without the external state observed at the time is equally weak. The debugging unit is the whole trajectory.

Example 3: use semantic judges without trusting them blindly

Some requirements cannot be reduced to exact assertions: “the support reply addresses the customer’s concern” is semantic. Use a judge only after deterministic checks, request structured output, run more than one independent judgment when the risk warrants it, and route disagreement to review. This Node 20 example includes a timeout, malformed-response handling, an explicit abstain state, and disagreement handling.

type Verdict = { decision: 'pass' | 'fail' | 'abstain'; reasons: string[] };

async function judge(candidate: string, seed: number): Promise<Verdict> {
  const endpoint = process.env.JUDGE_URL;
  const token = process.env.JUDGE_TOKEN;
  if (!endpoint || !token) throw new Error('JUDGE_URL and JUDGE_TOKEN are required');

  const response = await fetch(endpoint, {
    method: 'POST',
    headers: { authorization: 'Bearer ' + token, 'content-type': 'application/json' },
    body: JSON.stringify({
      seed,
      temperature: 0,
      rubric: 'Pass only if the reply answers the issue and invents no policy.',
      candidate,
      outputSchema: {
        type: 'object', required: ['decision', 'reasons'], additionalProperties: false,
        properties: {
          decision: { enum: ['pass', 'fail', 'abstain'] },
          reasons: { type: 'array', items: { type: 'string' }, minItems: 1 },
        },
      },
    }),
    signal: AbortSignal.timeout(10_000),
  });

  if (!response.ok()) throw new Error('Judge returned HTTP ' + response.status());
  const value = await response.json() as Partial<Verdict>;
  if (!['pass', 'fail', 'abstain'].includes(value.decision ?? '') || !value.reasons?.length) {
    throw new Error('Judge response violated its contract');
  }
  return value as Verdict;
}

export async function semanticGate(candidate: string): Promise<Verdict> {
  if (candidate.trim().length < 40) return { decision: 'fail', reasons: ['reply is empty or trivial'] };

  try {
    const verdicts = await Promise.all([judge(candidate, 17), judge(candidate, 29)]);
    if (verdicts.some(v => v.decision === 'abstain')) {
      return { decision: 'abstain', reasons: verdicts.flatMap(v => v.reasons) };
    }
    if (verdicts[0].decision !== verdicts[1].decision) {
      return { decision: 'abstain', reasons: ['judges disagreed; human review required'] };
    }
    return verdicts[0];
  } catch (error) {
    return {
      decision: 'abstain',
      reasons: [error instanceof Error ? error.message : 'unknown judge failure'],
    };
  }
}

semanticGate(process.argv.slice(2).join(' '))
  .then(result => { console.log(JSON.stringify(result)); process.exitCode = result.decision === 'pass' ? 0 : 2; })
  .catch(error => { console.error(error); process.exitCode = 1; });

Agreement does not prove truth; correlated judges can share the same blind spot. Calibrate the rubric against a versioned human-labeled set, track false accepts and false rejects separately, and test the gate against known-good production examples before making it blocking. Over-broad regexes and vague rubrics often reject excellent work while accepting polished nonsense.

Example 4: guard the irreversible side effect

Validation can become stale between checking and writing. Place authorization at the commit boundary and make retries idempotent. The endpoint below expects the service to enforce a unique idempotency key. It handles expired approvals, duplicate requests, timeouts, and a backend that acknowledges without returning a receipt.

type Approval = {
  runId: string;
  tenantId: string;
  decision: 'pass';
  expiresAt: string;
  evidenceHash: string;
};

export async function publishInvoice(approval: Approval, payload: unknown) {
  if (Date.parse(approval.expiresAt) <= Date.now()) throw new Error('approval expired');
  if (!/^[a-f0-9]{64}$/.test(approval.evidenceHash)) throw new Error('invalid evidence hash');

  const response = await fetch('https://billing.test/api/invoices', {
    method: 'POST',
    headers: {
      'content-type': 'application/json',
      'x-tenant-id': approval.tenantId,
      'idempotency-key': approval.runId + ':' + approval.evidenceHash,
    },
    body: JSON.stringify({ payload, approval }),
    signal: AbortSignal.timeout(8_000),
  });

  if (response.status === 409) {
    const existing = await response.json() as { receiptId?: string };
    if (!existing.receiptId) throw new Error('duplicate response omitted its receipt');
    return { duplicate: true, receiptId: existing.receiptId };
  }
  if (!response.ok()) throw new Error('publish failed with HTTP ' + response.status());

  const receipt = await response.json() as { receiptId?: string; tenantId?: string };
  if (!receipt.receiptId || receipt.tenantId !== approval.tenantId) {
    throw new Error('publish receipt failed tenant or identity verification');
  }
  return { duplicate: false, receiptId: receipt.receiptId };
}

async function main() {
  const approval = JSON.parse(process.env.APPROVAL_JSON ?? '{}') as Approval;
  console.log(await publishInvoice(approval, { customerId: 'customer-42', amount: 125 }));
}

main().catch(error => {
  console.error('Invoice was not safely published:', error instanceof Error ? error.message : error);
  process.exitCode = 1;
});

A client-provided idempotency key helps only if the server stores it atomically with the write and returns the original receipt on replay. A check-then-insert implementation still races. For payments, deployment, deletion, access grants, and customer messages, require a database uniqueness constraint or transactional outbox on the server side.

Design the retry policy as a state machine

“Retry three times” is not a policy; it is a loop. Classify failures before retrying. A transport timeout can reuse the same idempotency key. Invalid structured output can receive one repair prompt containing schema errors. Missing evidence should trigger a fresh observation, not a replayed mutation. Policy violations and tenant mismatches should stop immediately and alert.

• Transient: rate limit, timeout, or temporary dependency failure; back off with jitter and preserve operation identity.
• Repairable: schema-invalid proposal or missing citation; return precise validation feedback within a bounded attempt budget.
• Stale: changed DOM, expired approval, or old receipt; re-observe before planning again.
• Terminal: forbidden tool, cross-tenant target, budget breach, or repeated no-progress trajectory; stop and escalate.

Persist state transitions such as PROPOSED → VALIDATING → APPROVED → COMMITTING → COMMITTED. Reject illegal transitions. This makes crash recovery testable: a worker restarting in COMMITTING looks up the idempotency receipt instead of blindly publishing again. See how to test AI agents with Playwright for fixture and trace patterns.

Troubleshooting: diagnose the gate, not just the model

When an agent workflow flakes, first identify whether generation, observation, evaluation, orchestration, or the side-effect boundary failed. Re-running without a hypothesis destroys evidence and can repeat damage.

• The gate passes locally but fails in CI: compare timezone, locale, model and prompt versions, seeded test data, feature flags, and clock skew. Freeze external dependencies or record their versioned responses.
• The agent says done but the receipt is absent: treat narration as untrusted. Query the system of record with the run’s tenant and operation ID; inspect whether the tool timed out after committing.
• A judge flips between pass and fail: log raw verdicts, run the calibration corpus, tighten rubric examples, and introduce abstention. Do not hide instability by averaging arbitrary numeric scores.
• Retries create duplicates: verify the same idempotency key crosses every layer and that the server enforces uniqueness atomically. A new key per retry defeats the mechanism.
• The gate rejects known-good work: test rules against a gold corpus and inspect each predicate. Regex-based style rules often confuse valid inventories, URLs, or domain terms with violations.
• The trajectory never terminates: detect repeated tool-call signatures, no-change observations, token or cost ceilings, and wall-clock deadlines. Return a typed no_progress reason.

Edge cases that escape happy-path testing

Test ambiguous DOM matches, empty retrieval results, malformed Unicode, oversized tool output, prompt injection inside retrieved documents, daylight-saving boundaries, approval expiry during a slow call, and partial success across multiple tools. Also test cancellation: aborting the orchestrator does not guarantee the remote tool stopped.

Multi-agent workflows add more traps. Two workers can approve different snapshots; a late result can overwrite a newer one; shared memory can leak one tenant’s evidence into another; and a reviewer model may inherit the generator’s assumptions. Bind every artifact to runId, tenant, objective hash, evidence hash, and version. Use optimistic concurrency or a lease for state updates, and reject late writes from superseded attempts.

A release checklist for reliable agent gates

• The model proposes; deterministic code authorizes.
• Every success claim points to fresh, independently retrieved evidence.
• Pass, retry, abstain, and stop are distinct machine-readable outcomes.
• Semantic judges are calibrated, versioned, and unable to bypass hard policy.
• Retries preserve identity and vary by failure class.
• Irreversible tools require idempotency plus server-side atomic enforcement.
• Traces connect prompt, trajectory, gate verdict, and receipt under one run ID.
• Known-good, known-bad, adversarial, and stale-state cases run in CI.

Reliable agents do not come from eliminating variability. They come from containing it. Let the model search a wide solution space, but narrow the path to production through typed evidence, independent verification, bounded recovery, and an atomic commit boundary. That is the bridge from an impressive demo to a system a QA team can defend.