Level Up Your Agent Harness: Parallel CI Orchestration for LLM Agent Loops

The first time an AI coding agent fixes a test for you, it feels like cheating. The tenth time, you notice the pattern: it is fast when the problem is local, slow when the repository is noisy, and expensive when CI gives it one giant, ambiguous failure log. That is the moment to level up from agent-as-chat-window to agent-as-test-infrastructure.

This guide is for vibe coders and indie developers who already use AI tools, but now need professional reliability. We will build a Node.js agent harness that splits work across CI, gives every loop a contract, fails clearly, retries only the right failures, and merges results without letting parallel agents overwrite each other.

What Problem Does an Agent Harness Actually Solve?

An agent harness turns LLM output into a bounded engineering loop: gather context, propose a change, run checks, inspect failures, revise, and return a structured verdict. The harness is not the model. It is the machinery around the model that decides what files it sees, what commands it can run, how long it may run, and what counts as success.

Without a harness, an agent sees a giant prompt and a terminal. With a harness, it receives a contract: you own shard 2, these files changed, these checks failed, write a patch or explain why not. That contract is what makes parallelization possible. One agent can investigate browser tests, another can inspect type errors, and a third can validate docs or schema output.

This is where beginners often get stuck. They parallelize too early by launching three agents with the same prompt. That creates duplicated work, conflicting edits, and vague summaries. Professional orchestration parallelizes the problem space, not the vibes. For more on hardening test loops before adding agents, see our deep dive on test infrastructure flakiness.

The Mental Model: From Prompt Loop to CI Control Plane

A single local agent loop is usually sequential: prompt, edit, run tests, prompt again. CI orchestration changes the shape of the problem. CI is event-driven, isolated, time-boxed, and artifact-oriented. Each job starts from a clean checkout. Each process has a fresh environment. Logs are streamed, truncated, and billed. Files do not persist between jobs unless you upload artifacts or write to a shared store.

That means the harness must treat every agent run like a distributed system component. Inputs must be explicit. Outputs must be machine-readable. Timeouts must be enforced outside the model. Retries must distinguish model mistakes from infrastructure failures. A flaky npm registry request can be retried. A failing assertion caused by a bad patch should not be retried blindly.

Step 1: Build a CI Matrix from the Files That Actually Changed

The first upgrade is deterministic sharding. Instead of asking an agent to inspect the whole repo, generate a matrix from changed files. The edge case is important: on the first run of a branch, shallow clones and missing merge bases often make git diff origin/main... fail. Production code needs a fallback and a clear error when it cannot determine scope.

#!/usr/bin/env node
// ci/build-agent-matrix.mjs
import { execFileSync } from 'node:child_process';
import { writeFileSync } from 'node:fs';

const DEFAULT_BASE = process.env.CI_BASE_REF || 'origin/main';
const MAX_FILES_PER_SHARD = Number(process.env.MAX_FILES_PER_SHARD || 30);

function git(args) {
  try {
    return execFileSync('git', args, { encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'] }).trim();
  } catch (error) {
    const stderr = error.stderr?.toString() || error.message;
    throw new Error('git ' + args.join(' ') + ' failed: ' + stderr);
  }
}

function changedFiles() {
  const strategies = [
    ['diff', '--name-only', DEFAULT_BASE + '...HEAD'],
    ['diff', '--name-only', 'HEAD~1', 'HEAD'],
  ];

  for (const args of strategies) {
    try {
      const out = git(args);
      if (out) return out.split('\n').filter(Boolean);
    } catch {
      // Shallow clones may not have origin/main or HEAD~1. Try the next strategy.
    }
  }

  throw new Error('Could not determine changed files. Fetch the base branch or pass CI_BASE_REF.');
}

function classify(file) {
  if (/\.(spec|test)\.(ts|tsx|js|jsx)$/.test(file)) return 'test';
  if (/^(src|app|packages)\//.test(file)) return 'implementation';
  if (/^(docs|content|blog)\//.test(file) || /\.mdx?$/.test(file)) return 'docs';
  if (/^(package.json|pnpm-lock.yaml|bun.lockb|yarn.lock)$/.test(file)) return 'dependency';
  return 'misc';
}

function chunk(items, size) {
  const chunks = [];
  for (let i = 0; i < items.length; i += size) chunks.push(items.slice(i, i + size));
  return chunks;
}

try {
  const files = changedFiles().filter((file) => !file.startsWith('vendor/'));
  if (files.length === 0) {
    console.log(JSON.stringify({ include: [{ shard: 'noop', kind: 'noop', files: [] }] }));
    process.exit(0);
  }

  const groups = new Map();
  for (const file of files) {
    const kind = classify(file);
    if (!groups.has(kind)) groups.set(kind, []);
    groups.get(kind).push(file);
  }

  const include = [];
  for (const [kind, groupedFiles] of groups) {
    chunk(groupedFiles, MAX_FILES_PER_SHARD).forEach((shardFiles, index) => {
      include.push({ shard: kind + '-' + (index + 1), kind, files: shardFiles });
    });
  }

  writeFileSync('agent-matrix.debug.json', JSON.stringify({ generatedAt: new Date().toISOString(), include }, null, 2));
  console.log(JSON.stringify({ include }));
} catch (error) {
  console.error(JSON.stringify({ level: 'error', message: error.message }));
  process.exit(1);
}

This script is intentionally conservative. It ignores vendored files, emits a no-op shard when there is nothing to do, and writes a debug artifact. The debug file matters because the fastest way to diagnose bad agent behavior is often to inspect the input contract it received. If the matrix is wrong, the model is not the root cause.

Step 2: Run Each Agent Loop with Timeouts, Typed Output, and Retry Classification

Now wrap the agent invocation. The exact model provider does not matter for the harness design. The important part is that every shard gets the same contract shape and must return JSON. Replace AGENT_COMMAND with your tool, but keep the timeout, artifact, and classification behavior.

#!/usr/bin/env node
// ci/run-agent-shard.mjs
import { spawn } from 'node:child_process';
import { mkdirSync, writeFileSync } from 'node:fs';

const TIMEOUT_MS = Number(process.env.AGENT_TIMEOUT_MS || 12 * 60 * 1000);
const MAX_ATTEMPTS = Number(process.env.AGENT_MAX_ATTEMPTS || 2);
const AGENT_COMMAND = process.env.AGENT_COMMAND || 'node';
const AGENT_ARGS = process.env.AGENT_ARGS ? JSON.parse(process.env.AGENT_ARGS) : ['ci/mock-agent.js'];

function parseShard() {
  if (!process.env.SHARD) throw new Error('Missing SHARD JSON environment variable');
  const shard = JSON.parse(process.env.SHARD);
  if (!shard.shard || !Array.isArray(shard.files)) throw new Error('Invalid shard contract');
  if (shard.files.some((file) => file.includes('..'))) throw new Error('Refusing unsafe path traversal in shard files');
  return shard;
}

function classifyFailure(output, code) {
  if (/ECONNRESET|ETIMEDOUT|rate limit|temporarily unavailable/i.test(output)) return 'infra';
  if (code === 124 || /timeout/i.test(output)) return 'timeout';
  if (/SyntaxError|TypeError|AssertionError|expect\(/i.test(output)) return 'code';
  return 'unknown';
}

function runOnce(shard, attempt) {
  return new Promise((resolve) => {
    const child = spawn(AGENT_COMMAND, AGENT_ARGS, {
      env: { ...process.env, AGENT_SHARD: JSON.stringify(shard), AGENT_ATTEMPT: String(attempt) },
      stdio: ['ignore', 'pipe', 'pipe'],
    });

    let output = '';
    const timer = setTimeout(() => {
      output += '\n[harness] timeout after ' + TIMEOUT_MS + 'ms';
      child.kill('SIGTERM');
      setTimeout(() => child.kill('SIGKILL'), 5000).unref();
    }, TIMEOUT_MS);

    child.stdout.on('data', (chunk) => { output += chunk.toString(); });
    child.stderr.on('data', (chunk) => { output += chunk.toString(); });
    child.on('error', (error) => { output += '\n[harness] spawn error: ' + error.message; });
    child.on('close', (code) => {
      clearTimeout(timer);
      const failureClass = code === 0 ? null : classifyFailure(output, code);
      resolve({ code, output, failureClass });
    });
  });
}

function extractJson(output) {
  const match = output.match(/AGENT_RESULT_JSON:(\{.*\})/s);
  if (!match) throw new Error('Agent did not emit AGENT_RESULT_JSON marker');
  const result = JSON.parse(match[1]);
  if (!['pass', 'patch', 'needs-human', 'fail'].includes(result.verdict)) {
    throw new Error('Invalid agent verdict: ' + result.verdict);
  }
  return result;
}

try {
  mkdirSync('agent-results', { recursive: true });
  const shard = parseShard();
  let last;

  for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {
    last = await runOnce(shard, attempt);
    writeFileSync('agent-results/' + shard.shard + '.attempt-' + attempt + '.log', last.output);

    if (last.code === 0) {
      const result = extractJson(last.output);
      writeFileSync('agent-results/' + shard.shard + '.json', JSON.stringify({ shard, result }, null, 2));
      process.exit(0);
    }

    if (!['infra', 'timeout'].includes(last.failureClass) || attempt === MAX_ATTEMPTS) break;
  }

  writeFileSync('agent-results/' + shard.shard + '.json', JSON.stringify({
    shard,
    result: { verdict: 'fail', reason: 'Agent failed with ' + last.failureClass, retryable: false },
  }, null, 2));
  console.error(last.output);
  process.exit(1);
} catch (error) {
  console.error(JSON.stringify({ level: 'error', message: error.message }));
  process.exit(1);
}

Notice the marker protocol: AGENT_RESULT_JSON:. CI logs contain warnings, package manager noise, and tool banners. A marker gives the harness a stable extraction point. If the marker is missing, that is a harness failure, not a passing result with a messy summary.

Can Parallel Agent Loops Be Trusted in CI?

Trust comes from constraints, not personality. A parallel loop is trustworthy when every shard has a small scope, a time budget, and a schema. It becomes risky when any shard can silently edit shared files, skip tests, or summarize a failed command as success. This is why the harness should fail closed. If the model forgets the output marker, the job fails. If two shards patch the same file, the merge step escalates.

Step 3: Merge Results Without Letting Agents Fight

The merge step is where many DIY systems break. Parallel agents can produce individually reasonable patches that conflict when combined. They can also generate incompatible recommendations: one says update a test snapshot, another says the snapshot reveals a product bug. The aggregator should be a deterministic gate first, and only then a human-readable summary.

#!/usr/bin/env node
// ci/merge-agent-results.mjs
import { readFileSync, writeFileSync } from 'node:fs';

const files = process.argv.slice(2);
if (files.length === 0) {
  console.error('No result files provided');
  process.exit(1);
}

function load(file) {
  try {
    const parsed = JSON.parse(readFileSync(file, 'utf8'));
    if (!parsed.shard?.shard || !parsed.result?.verdict) throw new Error('missing shard or verdict');
    return parsed;
  } catch (error) {
    throw new Error(file + ': invalid result JSON: ' + error.message);
  }
}

try {
  const results = files.map(load);
  const touched = new Map();
  const failures = [];
  const needsHuman = [];

  for (const item of results) {
    const verdict = item.result.verdict;
    if (verdict === 'fail') failures.push(item);
    if (verdict === 'needs-human') needsHuman.push(item);

    for (const file of item.result.touchedFiles || []) {
      const owners = touched.get(file) || [];
      owners.push(item.shard.shard);
      touched.set(file, owners);
    }
  }

  const conflicts = [...touched.entries()]
    .filter(([, owners]) => new Set(owners).size > 1)
    .map(([file, owners]) => ({ file, owners: [...new Set(owners)] }));

  const finalVerdict = failures.length > 0 || conflicts.length > 0
    ? 'fail'
    : needsHuman.length > 0
      ? 'needs-human'
      : 'pass';

  const summary = {
    verdict: finalVerdict,
    resultCount: results.length,
    failures: failures.map((item) => ({ shard: item.shard.shard, reason: item.result.reason })),
    conflicts,
    needsHuman: needsHuman.map((item) => ({ shard: item.shard.shard, reason: item.result.reason })),
    generatedAt: new Date().toISOString(),
  };

  writeFileSync('agent-results/summary.json', JSON.stringify(summary, null, 2));
  writeFileSync('agent-results/summary.md', [
    '# Agent CI Summary',
    '',
    'Verdict: **' + summary.verdict + '**',
    'Results: ' + summary.resultCount,
    'Conflicts: ' + summary.conflicts.length,
    'Failures: ' + summary.failures.length,
  ].join('\n'));

  if (summary.verdict === 'fail') {
    console.error(JSON.stringify(summary, null, 2));
    process.exit(1);
  }

  console.log(JSON.stringify(summary));
} catch (error) {
  console.error(error.message);
  process.exit(1);
}

The key gotcha is shared ownership. Package manifests, generated clients, snapshots, and lockfiles are conflict magnets. Either route them to a dedicated dependency shard or mark them as protected so the harness requires human review. If you are building this into a larger developer workflow, our guide to choosing AI testing tools covers where orchestration fits among recorders, runners, and review assistants.

A GitHub Actions Workflow That Wires the Pieces Together

The scripts above are useful locally, but the payoff comes from running them as a CI matrix. This workflow builds the matrix, runs one harness per shard, uploads artifacts even on failure, and merges results in a final job. It is intentionally plain GitHub Actions YAML because plain orchestration is easier to debug than clever wrappers.

name: agent-harness-ci

on:
  pull_request:
    types: [opened, synchronize, reopened]

jobs:
  matrix:
    runs-on: ubuntu-latest
    outputs:
      matrix: \${{ steps.matrix.outputs.matrix }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 2
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - id: matrix
        run: |
          node ci/build-agent-matrix.mjs > matrix.json
          echo "matrix=$(cat matrix.json)" >> "$GITHUB_OUTPUT"
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: agent-matrix
          path: agent-matrix.debug.json

  agent:
    needs: matrix
    if: \${{ fromJson(needs.matrix.outputs.matrix).include[0].kind != 'noop' }}
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix: \${{ fromJson(needs.matrix.outputs.matrix) }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 2
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - name: Run agent shard
        env:
          SHARD: \${{ toJson(matrix) }}
          AGENT_COMMAND: node
          AGENT_ARGS: '["ci/mock-agent.js"]'
        run: node ci/run-agent-shard.mjs
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: agent-result-\${{ matrix.shard }}
          path: agent-results/*

  merge:
    needs: [matrix, agent]
    if: always()
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/download-artifact@v4
        with:
          pattern: agent-result-*
          path: agent-results
          merge-multiple: true
      - run: node ci/merge-agent-results.mjs agent-results/*.json
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: agent-summary
          path: agent-results/summary.*

There are two edge cases to call out. First, fail-fast: false is deliberate. If one shard fails, you still want the other shards to produce diagnostics. Second, artifacts are uploaded with if: always(). A failed agent job without artifacts is just an expensive mystery.

Troubleshooting: When the Harness Fails Before the Code Does

Debugging agent CI is different from debugging ordinary tests because there are three failure planes: the system under test, the harness, and the model/provider. Start by identifying which plane failed. Do not ask the model to explain a missing artifact until you know whether the job uploaded one.

• Matrix is empty on a real PR: fetch depth is too shallow or the base ref is missing. Inspectagent-matrix.debug.json and fetch the base branch before rerunning.
• Agent says pass but merge fails: the agent emitted an invalid schema or omitted touched files. Treat this as a harness contract failure, not a flaky test.
• Every shard times out: check dependency install time, provider latency, and whether the prompt includes full logs. Shard smaller before raising timeouts.
• Two shards edit the same lockfile: route dependency files to one owner. Parallel lockfile edits are almost never worth automatic merging.
• Retries make failures worse: your classifier is too broad. Retry network and provider failures; do not retry assertion failures or type errors without new information.

In our experience, the most useful debugging artifact is not the final patch. It is the exact input contract for each shard plus the raw command log. Those two files tell you whether the agent received the wrong task, the right task with too much noise, or the right task with an insufficient permission boundary.

Gotchas That Separate Demos from Production

Agent orchestration demos often skip the unglamorous cases. Production systems cannot. Generated files need ownership. Secrets must never be printed into prompts. Pull requests from forks may not have access to provider credentials. CI logs may truncate long outputs. Hosted runners can evict caches. Package managers can fail before the model starts.

• Use read-only mode for forked PRs and produce recommendations instead of patches.
• Cap prompt size before calling the model; summarize logs with file paths, failing tests, and stack roots.
• Store provider request IDs when available so rate-limit and outage tickets are diagnosable.
• Never let a model decide whether required checks should be skipped. Checks are policy, not suggestions.
• Prefer additive artifacts over direct pushes until your merge rules have survived real failure modes.

The Migration Path: From Solo Vibes to Team-Ready Infrastructure

You do not need to build a platform on day one. Start by wrapping your current AI tool with a contract and a timeout. Then add matrix sharding for changed files. Then upload artifacts. Then add a deterministic merge gate. Only after those pieces work should you add more agents, queues, dashboards, or auto-commit behavior.

The professional move is not replacing intuition with bureaucracy. It is turning the good parts of your intuition into repeatable guardrails. You still decide what matters. The harness makes sure every agent loop gets the same facts, follows the same limits, and leaves behind evidence.

That is how you level up: not by trusting AI more, but by trusting your delivery system because it can absorb imperfect AI output and still produce a clear engineering result.