Deterministic AI agent tests with AIMock — mocking LLMs, vector DBs, MCP, and A2A on one port
AIMock mocks the whole AI stack on one port (LLM, vector DB, MCP, A2A), so an agent test suite runs in under three seconds with identical output every time.

On this page
Introduction
Testing an application built on an LLM has an awkward problem: send the same prompt twice and you get two slightly different responses. A test that passed on Monday fails on Tuesday with the code unchanged, because the model is non-deterministic.
A 2026 agent application makes this worse. A single request touches six or seven services: not just the LLM but a vector database for retrieval-augmented generation (RAG), Model Context Protocol (MCP) tools, and inter-agent traffic over the Agent2Agent (A2A) protocol. How to mock each of them becomes the real question.
AIMock, the open-source mock server CopilotKit released in April 2026, answers all of them on one port. The test suite this article builds runs 18 cases in 2.35 seconds against mocked LLM, vector DB, and A2A services, with byte-identical output every run.
This article builds that test suite, then records real fixtures with the llmock CLI and injects failures with LLMock’s chaos API.
What AIMock is
AIMock is an npm package that mocks every service an AI application talks to.
- 11+ LLM providers (OpenAI, Claude, Gemini, Bedrock, Azure, Vertex AI, Ollama, Cohere, and others)
- Full MCP JSON-RPC 2.0 support
- A2A agent card discovery and server-sent events (SSE) streaming
- Vector database mocks (Pinecone, Qdrant, ChromaDB)
- Chaos testing: injecting 500s, malformed JSON, and mid-stream disconnects,
both from a hand-rolled server and through the
setChaos()API - Zero dependencies; Node.js built-ins only
The concept is one config file and one port for the entire AI stack.
Where it sits among similar tools
I went looking for competitors and found that this niche has almost no direct ones.
| Tool | Focus | Overlap with AIMock |
|---|---|---|
| Keploy | Records real traffic and generates tests. General-purpose APIs | No support for AI-specific protocols (MCP, A2A, streaming SSE) |
| VCR.py / Polly.js | Classic HTTP record and replay | Usable for LLM mocking, but no AI-specific features such as tool calls |
| Portkey / Helicone | LLM gateways with caching and replay | Production-oriented. No fault injection, no MCP mocking |
If you want to test an agent application including MCP and A2A, AIMock is currently the only option. If all you need is a mock of the OpenAI API, a VCR-style HTTP replay is enough.
Recording and replaying fixtures, and why it matters
Record and replay is one of AIMock’s core features: capture a real API response, then serve it back verbatim during tests.
What record and replay solves:
# Monday: test passesassert response.includes("SQL injection risk") ✅
# Tuesday: same code, same prompt, OpenAI rephrasedassert response.includes("SQL injection risk") ❌# it returned "potential SQL vulnerability"With fixtures, an LLM call behaves like a pure function. Same input, same output, every time.
| Aspect | Without fixtures | With fixtures |
|---|---|---|
| Cost | A real API call per test run | Zero |
| Speed | 2–30s per LLM call | Under 1ms |
| Determinism | Different output every time | Byte-identical |
| CI stability | Flaky on network, rate limits, API changes | Stable |
| Edge cases | Not reproducible | Captured permanently |
Is mocking in CI a cop-out?
A mocked CI tests code logic. Prompt quality still needs the real API:
| What you are testing | Needs the real API? |
|---|---|
| Does the JSON parser handle this response shape? | No, a fixture is enough |
| Does the retry logic work on a 500? | No, chaos injection is better |
| Does the prompt produce good output? | Yes, impossible against a mock |
| Does the UI render the response correctly? | No, a fixture is enough |
In practice the best pattern is a two-tier CI pipeline:
- Every PR: fast mocked tests that verify code logic
- Nightly or weekly: integration tests against the real API, for drift detection and prompt validation
AIMock’s drift detection complements this. Run daily in CI, it compares the SDK type definitions, the real API response, and AIMock’s own output three ways, catching a provider’s API change before your users do.
Building the demo: a smart purchasing concierge
To exercise the main features I wrote tests for a purchasing agent.
Architecture
One request leaves the agent four times. Every call lands on a mock running on localhost.
Every call in the figure lands on one of these mocks:
- LLMock: mocks the OpenAI API (the
LLMockclass from@copilotkit/aimock) - VectorMock: a Pinecone-compatible vector DB mock (the
VectorMockclass) - An A2A mock: a seller agent I wrote by hand with
http.createServer
Project layout
aimock-a2a-demo/├── src/│ ├── buyer-agent/│ │ ├── index.ts # the main BuyerAgent orchestrator│ │ ├── a2a-client.ts # A2A protocol client│ │ ├── knowledge-base.ts # vector DB / RAG layer│ │ └── reasoner.ts # LLM reasoning│ └── types/│ ├── a2a.ts # A2A protocol types│ └── product.ts # product types├── tests/│ ├── setup.ts # AIMock setup│ ├── buyer-agent.test.ts # main test suite│ └── chaos.test.ts # chaos tests├── fixtures/llm/ # recorded LLM responses├── aimock.json # AIMock config└── package.jsonStep 1: setting up LLMock
LLMock registers fixtures by pattern-matching on systemMessage or
userMessage.
import { LLMock, VectorMock } from "@copilotkit/aimock";
const llmock = new LLMock({ port: 0 }); // random port
// route fixtures by the content of the system promptllmock.on({ systemMessage: "evaluating product offers" }, { content: JSON.stringify({ topPickIndex: 0, reasoning: "The MacBook Air M4 offers the best combination...", confidence: 0.92, }),});
llmock.on({ systemMessage: "purchasing concierge" }, { content: JSON.stringify({ refinedQuery: "ultrabook laptop under 1.3kg...", priorities: ["weight under 1.3kg", "battery life 15+ hours"], priceRange: { min: 900, max: 1300 }, }),});
await llmock.start();process.env.OPENAI_BASE_URL = `${llmock.url}/v1`;process.env.OPENAI_API_KEY = "mock-key";onMessage() matches only the text of the user message. When an application calls the LLM more than once (twice here, for planning the
search strategy and for evaluating offers), similar user messages can match an
unintended fixture. Both user messages carry the same shopping request, but each
call has its own system prompt, so routing on on({ systemMessage: "..." }) by
the content of the system prompt is reliable.
Both calls carry the same user message, so only the system prompt tells them apart.
LLMock returns the first fixture that matches, so register the more specific patterns first and the catch-all
(onMessage(/.*/)) last.
Step 2: setting up VectorMock
VectorMock exposes a Pinecone-compatible API.
const vectorMock = new VectorMock({ port: 0 });
vectorMock.addCollection("product-knowledge", { dimension: 1536 });
// register static results — the same results come back every timevectorMock.onQuery("product-knowledge", [ { id: "review-macbook-air", score: 0.94, metadata: { brand: "Apple", model: "MacBook Air M4", text: "MacBook Air M4: 1.24kg, 18hr battery life...", }, }, // ... more entries]);
await vectorMock.start();The QueryResult type has no text field. Text data
goes in metadata.text, and the application reads it back as m.metadata?.text.
VectorMock can in theory be mounted on the same port as LLMock via LLMock’s
mount(). But mount() does not strip the path prefix before passing a request
on, so starting VectorMock on its own port is what actually works.
Mounted, the /vector prefix reaches VectorMock intact. On its own port, the query path matches.
Step 3: mocking the A2A protocol
A2A is the open inter-agent protocol from Google and the Linux Foundation (v1.2, 150+ participating organizations).
A2A is built for cross-organization agent communication: your agent talking to another company’s agent, say a payment processor or a logistics agent. Using A2A between sub-agents inside one application is overkill; a function call is enough.
In this demo A2A sits between the buyer agent and the seller agent. Those are meant to be agents from different organizations, so it is a legitimate use.
The buyer agent and the seller agent belong to different companies, and each exchange between them is an A2A request and response over HTTP.
I wrote the A2A mock by hand with http.createServer, conforming to A2A v1.0:
const server = createServer((req, res) => { // Agent card discovery: GET /.well-known/agent-card.json (v1.0) if (req.url === "/.well-known/agent-card.json" && req.method === "GET") { res.writeHead(200, { "Content-Type": "application/json" }); res.end(JSON.stringify({ name: "Mock Electronics Seller", skills: [{ id: "product-search", name: "Product Search", ... }], ... })); return; }
// JSON-RPC: POST / — SendMessage (v1.0) if (req.method === "POST") { const rpc = JSON.parse(body); if (rpc.method === "SendMessage") { res.end(JSON.stringify({ jsonrpc: "2.0", id: rpc.id, result: { message: { role: "ROLE_AGENT", parts: [{ data: { offers: mockOffers } }], messageId: `reply-${rpc.id}`, contextId: rpc.params?.message?.contextId, }, }, })); } }});The buyer agent’s A2A client talks to this mock exactly as it would to a real A2A v1.0 agent, so pointing it at a real agent changes only the URL. I did that with Google’s codelab agent on Cloud Run, which is a separate article.
Step 4: running the tests
Eighteen test cases, all finishing inside three seconds. They fall into four groups:
| Test group | Cases | What it proves |
|---|---|---|
| A2A: Seller Agent Discovery | 3 | The buyer agent reads the seller’s agent card and gets offers over SendMessage |
| Vector DB: Knowledge Base | 3 | The same query returns the same results, and an unreachable server returns an empty result |
| Integration: Full Purchasing Flow | 2 | The whole query → RAG → LLM → A2A → LLM path runs, with and without RAG |
| Chaos and Resilience | 10 | The vector DB and the LLM fail on purpose, and the agent handles each failure as the test expects |
$ pnpm test
Test Files 2 passed (2) Tests 18 passed (18) Duration 2.35sDeterminism:
The test queries the knowledge base twice with the same input and asserts that the two results are identical. Against VectorMock that holds on every run; real Pinecone gives no such guarantee.
test("same query twice returns identical results (determinism proof)", async () => { const results1 = await kb.query("laptop"); const results2 = await kb.query("laptop");
// real Pinecone gives no such guarantee expect(results1).toEqual(results2);});Graceful degradation:
The test points the knowledge base at a port nothing listens on, so every RAG call fails, and asserts that the agent still returns a recommendation using the LLM and A2A alone. Taking RAG down on demand like this is something a hosted Pinecone index cannot do.
test("works without RAG knowledge (graceful degradation)", async () => { const agent = new BuyerAgent({ sellerAgentUrl: sellerUrl, knowledgeBase: { apiKey: "mock-key", baseUrl: "http://localhost:1", // unreachable → falls back indexName: "product-knowledge", }, ragTimeoutMs: 500, });
const result = await agent.findProduct({ query: "I need a lightweight laptop for travel", });
// even with RAG down, the agent still works via LLM + A2A expect(result.topPick).toBeDefined(); expect(result.knowledgeUsed).toHaveLength(0);});Recording fixtures with the llmock CLI
Having covered record and replay in the abstract, I ran it.
Starting llmock
The @copilotkit/aimock package ships a CLI binary called llmock. The
--record flag starts it as a recording proxy.
# start llmock in record mode, listening on port 4010pnpm mock:record# → llmock --record --provider-openai https://api.openai.com -f ./fixtures/llmRunning the app through llmock
Point OPENAI_BASE_URL at llmock and every OpenAI SDK call from the application
goes to llmock. The application still reads the API key from .env and attaches
it to each request; llmock forwards the request, key included, to OpenAI
unchanged and records the response as a fixture.
OPENAI_BASE_URL=http://localhost:4010/v1 \SELLER_URL=https://burger-agent-XXXXXXXXXX.us-central1.run.app \pnpm devThe recorded fixture
JSON like this lands in fixtures/llm/recorded/:
{ "fixtures": [{ "match": { "userMessage": "User request: \"I need a lightweight laptop...\"", "model": "gpt-4o-mini", "turnIndex": 0, "hasToolResult": false }, "response": { "content": "{\"refinedQuery\": \"lightweight travel laptop under $1200\", ...}" }, "metadata": { "systemHash": "a11524e7" } }]}The fields under match (userMessage, model, turnIndex) are the matching
conditions. From then on, a request meeting the same conditions gets this
response without OpenAI being called.
What it feels like in practice
pnpm mock:recordstarts the proxyOPENAI_BASE_URL=http://localhost:4010/v1 pnpm devruns the app normally- Fixtures save themselves to
fixtures/llm/recorded/ - From then on,
pnpm mock:startreplays them
No code changes at all. Only where OPENAI_BASE_URL points.
LLMock’s chaos injection
The Chaos and Resilience group in Step 4 includes tests built on setChaos() and
nextRequestError(). They use LLMock’s programmatic chaos injection API, which
reproduces LLM failure scenarios without the hand-rolled HTTP server that the
Chaos: Vector DB failures and Chaos: LLM failures tests stand up to fake a
500 or a timeout. The vector DB tests have no alternative, since VectorMock has
no chaos API; the LLM tests could have used setChaos() from the start.
setChaos(): probabilistic fault injection
setChaos() sets a failure probability applied to every request.
test("setChaos({ malformedRate: 1 }) — every response is garbled", async () => { const mock = new LLMock({ port: 0 });
// register a valid fixture (/.*/ matches every message) mock.onMessage(/.*/, { content: '{"refinedQuery": "test", "priorities": [], "priceRange": {"min": 0, "max": 100}}', });
// chaos: corrupt the response 100% of the time mock.setChaos({ malformedRate: 1.0 });
await mock.start(); process.env.OPENAI_BASE_URL = `http://localhost:${mock.port}/v1`;
const reasoner = new Reasoner("gpt-4o-mini");
// the fixture matches, but chaos breaks the response await expect( reasoner.buildSearchStrategy("laptop", []) ).rejects.toThrow();
await mock.stop();});ChaosConfig takes a probability per failure mode:
| Parameter | Effect | Works without a fixture |
|---|---|---|
dropRate | Returns a 500 | Yes |
malformedRate | Returns {malformed json: <<<chaos>>>} | No, a match is required |
disconnectRate | Cuts the TCP connection mid-stream | No, a match is required |
One gotcha: onMessage("*") matches the literal string *, not a wildcard.
To match every message, use the regular expression /.*/.
clearChaos(): turning it off
clearChaos() removes the chaos configuration and normal fixture responses
resume. Useful for testing recovery from a failure.
test("clearChaos() restores normal operation", async () => { const mock = new LLMock({ port: 0 }); mock.onMessage(/.*/, { content: validJson }); mock.setChaos({ malformedRate: 1.0 });
await mock.start(); // ... env setup ...
const reasoner = new Reasoner("gpt-4o-mini");
// chaos on — it fails await expect( reasoner.buildSearchStrategy("laptop", []) ).rejects.toThrow();
// chaos off — it works mock.clearChaos();
const strategy = await reasoner.buildSearchStrategy("laptop", []); expect(strategy.refinedQuery).toBe("laptop"); // ✓ valid JSON});nextRequestError(): one-shot error injection
nextRequestError() returns an error for the next request only, then
consumes itself. Ideal for a “fails once, recovers on retry” scenario.
test("nextRequestError(400) — single request fails, next succeeds", async () => { const mock = new LLMock({ port: 0 }); mock.onMessage(/.*/, { content: validJson });
// one-shot: return 400 for the next request only mock.nextRequestError(400, { message: "Bad Request" });
await mock.start(); // ... env setup ...
const reasoner = new Reasoner("gpt-4o-mini");
// first call: 400 await expect( reasoner.buildSearchStrategy("laptop", []) ).rejects.toThrow();
// second call: the one-shot is spent → normal response const strategy = await reasoner.buildSearchStrategy("laptop", []); expect(strategy.refinedQuery).toBe("laptop"); // ✓});The OpenAI SDK retries 5xx errors (500, 503, and so on) automatically, so a one-shot error test uses a 4xx, which is not retried.
Test run
$ pnpm test
Test Files 2 passed (2) Tests 18 passed (18) Duration 2.2sBoth approaches work. The Chaos: Vector DB failures and Chaos: LLM failures
tests use a hand-rolled HTTP server, the setChaos() and nextRequestError()
tests use LLMock’s built-in chaos API, and agent degrades gracefully belongs to
neither. The API version is less code and states its intent more clearly.
Summary
I built a test suite for an AI agent with AIMock, mocking the LLM, the vector DB, and the A2A seller agent. It runs in under three seconds with identical output every time.
Where AIMock earns its place:
- Testing an agent application that combines an LLM, a vector DB, and several other services
- Fast, cheap, stable test runs in CI/CD
- Verifying fault tolerance through fault injection
Where it does not fit:
- Mocking only the OpenAI API, where a VCR-style HTTP replay is enough
- Evaluating prompt quality, which is impossible against a mock
Implementation notes for AIMock:
onMessage()matches only the user message.on({ systemMessage })lets you route on the system promptonMessage("*")matches the literal*, not a wildcard. UseonMessage(/.*/)to match everything- Fixture registration order matters: the first match wins
QueryResultin VectorMock has notextfield. Put it inmetadata- Start VectorMock on its own port, separate from LLMock
malformedRateanddisconnectRateinsetChaos()need a fixture match. OnlydropRateworks without one- The OpenAI SDK auto-retries 5xx. Use a 4xx when testing errors with
nextRequestError()


