Agentic Coding vs. Traditional SDLC: A Senior QA Architect’s Guide to Reliable AI Systems

A conventional application waits for an input, executes code selected by developers, and returns an output. An agentic coding system interprets a goal, chooses tools, reads mutable context, edits files, runs commands, observes results, and decides whether to continue. The final diff may look ordinary; the path that produced it is not. That distinction changes the QA problem from “does this function return the expected value?” to “did a probabilistic controller reach an acceptable state through an authorized, bounded, recoverable path?”

The pressure is real. GitHub’s controlled study of 95 professional developers reported that the Copilot group completed a bounded JavaScript HTTP-server task 55% faster. That result does not prove that autonomous changes are reliable, but it explains why delivery teams adopt them. Meanwhile, the Stack Overflow 2024 Developer Survey found that 45% of professional developers rated AI tools bad or very bad at complex tasks. Faster generation and uncertain correctness can coexist.

Why does agentic coding break the traditional test model?

Agentic coding adds nondeterministic decisions, mutable memory, tool side effects, and multi-step loops, so QA must verify trajectory and outcome.

Traditional SDLC quality gates assume that the implementation is the main source of change. Requirements become code; code is checked with unit, integration, and end-to-end tests; a known artifact is promoted. An agent adds a runtime policy layer. The same prompt can take different routes because retrieved context changed, a tool returned a different ordering, a model version moved, or the agent summarized earlier observations differently.

This does not invalidate your Playwright, Cypress, or Selenium investment. It means those suites cover only the outermost observable layer. A green checkout test can coexist with an agent that read a forbidden file before producing the right UI. A correct patch can hide a duplicated deployment call. A polished explanation can be unsupported by the tool trace. Reliable testing therefore needs multiple oracles: protocol validity, policy compliance, state transitions, side-effect correctness, and user-visible outcome.

What should a production agentic test architecture verify?

Use layered contracts: deterministic components at the base, controlled trajectories in the middle, and outcome evaluations plus telemetry at the top.

Start with the cheapest oracle that can detect a defect. Validate JSON Schemas, tool names, arguments, authorization, timeouts, and idempotency before asking another model to judge semantics. Then replay representative trajectories. Finally, run a small live-model suite for risks that fixtures cannot expose. This ordering keeps merge checks reproducible while still detecting provider drift.

• Component contracts: parsers, prompt builders, retrieval filters, policy engines, and tool adapters fail closed on malformed input.
• Trajectory contracts: every step has a correlation ID; tool calls are allowed, bounded, and paired with results; irreversible actions require explicit authority.
• Outcome invariants: generated code compiles, tests pass, required files change, protected files do not, and claims point to trace evidence.
• Adversarial scenarios: indirect prompt injection, poisoned repository instructions, oversized context, stale memory, permission denial, rate limits, and partial tool failure.
• Operational signals: termination reason, tool error class, retry count, token and wall-clock budget, policy denials, and human overrides are queryable by release.

This structure also clarifies ownership. Developers own deterministic contracts near the code. QA owns risk coverage, scenario design, and release evidence. Security owns tool permissions and data boundaries. Product owners decide which outcomes are acceptable. For a practical companion on translating risks into browser coverage, see our AI test automation deep dive.

Example 1: test the user journey and the hidden trace with Playwright

A browser test should not stop at “Patch ready.” Intercept the agent boundary, return a known trajectory, and assert that the UI exposes terminal failure safely. The example below is a complete Playwright test file. It handles malformed request JSON and includes the edge case where a tool result is missing.

// tests/agent-run.spec.ts — run: npx playwright test
import { test, expect, type Route } from '@playwright/test';

type Step = { id: string; kind: 'tool_call' | 'tool_result'; tool: string };

async function mockRun(route: Route, steps: Step[]) {
  try {
    const body = route.request().postDataJSON() as {
      goal?: string; idempotencyKey?: string;
    };
    if (!body.goal?.trim() || !body.idempotencyKey?.trim()) {
      return await route.fulfill({
        status: 400,
        contentType: 'application/json',
        body: JSON.stringify({ error: 'goal and idempotencyKey are required' }),
      });
    }
    await route.fulfill({
      status: 200,
      contentType: 'application/json',
      body: JSON.stringify({ runId: 'run-fixture-1', status: 'completed', steps }),
    });
  } catch (error) {
    await route.fulfill({
      status: 422,
      contentType: 'application/json',
      body: JSON.stringify({
        error: error instanceof Error ? error.message : 'invalid request',
      }),
    });
  }
}

test('shows evidence for a completed, paired tool call', async ({ page }) => {
  await page.route('**/api/agent/runs', route => mockRun(route, [
    { id: '1', kind: 'tool_call', tool: 'run_tests' },
    { id: '1', kind: 'tool_result', tool: 'run_tests' },
  ]));
  await page.goto('/agent');
  await page.getByLabel('Goal').fill('Fix the failing checkout test');
  await page.getByRole('button', { name: 'Run agent' }).click();

  await expect(page.getByRole('status')).toHaveText('Patch ready');
  await expect(page.getByTestId('trace')).toContainText('run_tests');
  await expect(page.getByTestId('run-id')).toHaveText('run-fixture-1');
});

test('fails closed when a tool result is missing', async ({ page }) => {
  await page.route('**/api/agent/runs', route => mockRun(route, [
    { id: 'orphan', kind: 'tool_call', tool: 'write_file' },
  ]));
  await page.goto('/agent');
  await page.getByLabel('Goal').fill('Update checkout validation');
  await page.getByRole('button', { name: 'Run agent' }).click();

  await expect(page.getByRole('alert')).toContainText('Incomplete tool trace');
  await expect(page.getByRole('button', { name: 'Apply patch' })).toBeDisabled();
});

The important seam is the HTTP boundary, not the model temperature. A temperature of zero does not guarantee identical outputs across infrastructure, model revisions, or parallel decoding. Route interception makes this merge-blocking test deterministic. Keep a separate scheduled suite that calls the real service; do not weaken this test with retries until a random answer happens to pass.

Example 2: enforce a trace protocol before evaluating prose

Treat the agent loop as a protocol. Calls require unique IDs, allowed tool names, matching results, and a maximum step count. The validator below is runnable with Node’s test runner and TypeScript stripping. Its explicit edge cases are duplicate side effects, unknown tools, orphan results, and runaway loops.

// agent-trace.test.ts — run: node --test --experimental-strip-types agent-trace.test.ts
import test from 'node:test';
import assert from 'node:assert/strict';

type Step =
  | { type: 'call'; id: string; tool: string; args: unknown }
  | { type: 'result'; id: string; ok: boolean; output?: unknown; error?: string };

const ALLOWED_TOOLS = new Set(['read_file', 'write_file', 'run_tests']);

export function validateTrace(steps: Step[], maxSteps = 12): void {
  if (!Array.isArray(steps) || steps.length === 0) throw new Error('trace is empty');
  if (steps.length > maxSteps) throw new Error('step budget exceeded');

  const pending = new Map<string, string>();
  for (const step of steps) {
    if (!step.id?.trim()) throw new Error('step id is required');
    if (step.type === 'call') {
      if (!ALLOWED_TOOLS.has(step.tool)) throw new Error('tool not allowed: ' + step.tool);
      if (pending.has(step.id)) throw new Error('duplicate call id: ' + step.id);
      pending.set(step.id, step.tool);
      continue;
    }

    if (!pending.has(step.id)) throw new Error('orphan result: ' + step.id);
    if (!step.ok && !step.error?.trim()) throw new Error('failed result lacks error');
    pending.delete(step.id);
  }

  if (pending.size > 0) {
    throw new Error('missing results for: ' + [...pending.keys()].join(','));
  }
}

test('accepts a bounded, paired trace', () => {
  assert.doesNotThrow(() => validateTrace([
    { type: 'call', id: 'read-1', tool: 'read_file', args: { path: 'src/cart.ts' } },
    { type: 'result', id: 'read-1', ok: true, output: 'export const cart = {}' },
    { type: 'call', id: 'test-1', tool: 'run_tests', args: { suite: 'cart' } },
    { type: 'result', id: 'test-1', ok: false, error: '1 assertion failed' },
  ]));
});

test('rejects a duplicated call before a side effect can repeat', () => {
  assert.throws(() => validateTrace([
    { type: 'call', id: 'write-1', tool: 'write_file', args: { path: 'src/cart.ts' } },
    { type: 'call', id: 'write-1', tool: 'write_file', args: { path: 'src/cart.ts' } },
  ]), /duplicate call id/);
});

test('rejects unknown tools and runaway traces', () => {
  assert.throws(() => validateTrace([
    { type: 'call', id: 'x', tool: 'shell_as_root', args: {} },
  ]), /tool not allowed/);
  assert.throws(() => validateTrace(new Array(13).fill({
    type: 'result', id: 'x', ok: true,
  }) as Step[]), /step budget exceeded/);
});

This validator intentionally says nothing about whether the patch is good. It answers a prior question: is the execution structurally admissible? Separating protocol validation from semantic evaluation produces better defect localization. If both live in one “agent score,” a policy violation can disappear inside a high average.

Example 3: build a fail-closed CI evaluation gate

Semantic scenarios still need a release decision. Store per-scenario results as evidence, then apply non-compensating gates: zero policy violations, zero unsupported high-risk actions, and a minimum outcome pass rate. This complete Node script handles missing or corrupt reports and rejects the edge case of an empty suite instead of dividing by zero and passing accidentally.

// scripts/agent-quality-gate.mjs — run: node scripts/agent-quality-gate.mjs reports/agent-eval.json
import { readFile } from 'node:fs/promises';

function assertReport(value) {
  if (!value || !Array.isArray(value.scenarios)) throw new Error('scenarios array missing');
  if (value.scenarios.length === 0) throw new Error('evaluation suite is empty');
  for (const [index, row] of value.scenarios.entries()) {
    if (typeof row.passed !== 'boolean') throw new Error('scenario ' + index + ' lacks passed');
    for (const key of ['policyViolations', 'unsupportedActions']) {
      if (!Number.isInteger(row[key]) || row[key] < 0) {
        throw new Error('scenario ' + index + ' has invalid ' + key);
      }
    }
  }
  return value;
}

async function main() {
  const path = process.argv[2];
  if (!path) throw new Error('usage: node agent-quality-gate.mjs REPORT.json');

  let report;
  try {
    report = assertReport(JSON.parse(await readFile(path, 'utf8')));
  } catch (error) {
    throw new Error('cannot load evaluation report: ' +
      (error instanceof Error ? error.message : String(error)));
  }

  const passed = report.scenarios.filter(row => row.passed).length;
  const passRate = passed / report.scenarios.length;
  const policyViolations = report.scenarios.reduce((n, row) => n + row.policyViolations, 0);
  const unsupportedActions = report.scenarios.reduce((n, row) => n + row.unsupportedActions, 0);

  const failures = [];
  if (passRate < 0.90) failures.push('outcome pass rate below 90%');
  if (policyViolations > 0) failures.push('policy violations must equal zero');
  if (unsupportedActions > 0) failures.push('unsupported actions must equal zero');

  if (failures.length) throw new Error(failures.join('; '));
  console.log(JSON.stringify({ passRate, policyViolations, unsupportedActions }));
}

main().catch(error => {
  console.error('[agent-quality-gate]', error instanceof Error ? error.message : error);
  process.exitCode = 1;
});

The 90% threshold is illustrative, not an industry benchmark. Choose it from your risk appetite and scenario severity. Averages must never compensate for a catastrophic class: ten easy successes cannot cancel one unauthorized production write. Version the scenario set, model identifier, prompt, tools, and evaluator rubric with every report so a score is reproducible evidence rather than an orphan number.

Design deterministic tests without pretending the model is deterministic

Reproducibility comes from controlled seams. Record model responses at the adapter boundary, stub tools below their authorization layer, freeze clocks, seed generated IDs, and create a fresh workspace per scenario. Replay should preserve the exact request, response, tool schema version, and correlation IDs. Redact secrets before fixtures enter source control.

Do not replay everything. A mocked policy engine cannot prove the real policy. A stubbed filesystem cannot expose path normalization bugs. Maintain a contract pyramid: many offline tests, fewer sandbox integrations using real tool adapters, and a small number of live-model evaluations. Run irreversible tools only against disposable infrastructure. Use idempotency keys and verify that retries return the prior result rather than repeating the action.

For semantic variability, prefer invariants and metamorphic relations. If a repository is renamed, the plan should remain equivalent. If irrelevant documentation is added, protected-file behavior must not change. If permission is removed, the agent must stop or request approval—not invent a workaround. Exact prose snapshots are useful only for protocol fixtures; they are a poor oracle for meaning.

Troubleshooting: diagnose agent failures by layer

Debug the earliest violated contract, not the final symptom. A weak patch may originate in retrieval, a tool timeout, context compaction, or an evaluator that rewarded verbosity. Preserve a structured event stream with run ID, parent step, model and prompt versions, tool arguments after safe redaction, duration, result class, and termination reason.

• The same scenario passes locally and fails in CI: compare model, prompt, tool schema, locale, timezone, filesystem ordering, and fixture hashes. Freeze all six before adding retries.
• The agent loops on one tool: inspect whether failures are machine-readable, confirm the observation reaches the next model turn, and enforce per-tool plus total step budgets. Terminate with a typed reason.
• A plausible answer has no support: require every factual claim to reference a tool-result ID. Reject references to missing, failed, or superseded observations.
• A write happens twice after a timeout: the client could not distinguish “failed” from “completed but response lost.” Add an idempotency key and query operation status before retrying.
• Replay passes but live evaluation regresses: diff model version, system prompt, retrieval corpus, tool descriptions, and safety policy independently. Promote only one changed variable at a time.
• Judge scores drift: rerun a frozen calibration set with human-agreed labels. If disagreement grows, block the judge upgrade rather than adjusting thresholds to hide it.

Keep the raw trace as a restricted artifact, not an unrestricted log stream. Tool output may contain source code, customer data, or credentials. Test redaction itself with canary secrets and nested structures; simple key-name filters miss tokens embedded in error messages. Our guide to testing AI agents safely extends this threat model into sandbox and access-control design.

Release strategy: preserve SDLC controls while changing their evidence

Agentic coding does not replace change management. It makes small batches, independent review, and automated evidence more important. The DORA Impact of Generative AI in Software Development report found that a 25% increase in AI adoption was associated with a 1.5% decrease in delivery throughput and a 7.2% decrease in delivery stability. DORA describes associations, not proof of causation, but the finding is a useful warning against equating faster code generation with better delivery.

Require the agent to submit the same reviewable unit as a human: scoped diff, rationale, tests, risk notes, and provenance. Add machine evidence: trace validator result, protected-path check, scenario report, dependency scan, and termination reason. A human reviewer should be able to reject the change without understanding hidden chain-of-thought. Expose concise decisions and tool evidence, not private reasoning tokens.

• Block merges on deterministic contracts, policy violations, compilation, and regression tests.
• Run live-model scenarios in a sandbox and quarantine provider outages from product defects.
• Canary model, prompt, retrieval, or tool-schema changes separately; never upgrade them as one opaque bundle.
• Monitor failure classes and human overrides after release, then turn new incidents into replayable regression scenarios.

The QA architect’s operating model

Begin with a risk register, not a leaderboard. Identify assets the agent can read, actions it can take, irreversible boundaries, and the evidence needed to prove safe completion. Map each risk to the lowest deterministic oracle available. Use model-based evaluation only where semantic judgment is genuinely required.

Next, define an agent run as a versioned test artifact: input goal, context manifest, model and prompt versions, tool schemas, event trace, output diff, test results, evaluator decision, and budget consumption. This makes failures reproducible across QA, engineering, security, and audit. It also prevents a common gotcha: comparing scores from different scenario sets as if they measured the same system.

Finally, optimize only after the safety envelope is visible. Faster token generation is irrelevant if the agent spends its savings in retries, review rework, or unstable deployments. The durable advantage comes from shortening feedback loops while strengthening evidence. Traditional SDLC supplies the discipline; agent-aware automation expands what that discipline can observe.