Interrogating Agents, Multi-Agent LLM Evaluation Harness

Problem

An LLM judge scoring a treatment against a control in separate calls carries its own calibration drift into the comparison, so the measured difference is partly the judge and partly the thing being measured.

Approach

Score both legs in a single batched, blinded call returning structured JSON. Because the judge sees both in the same context with the same calibration, its bias appears on both sides and cancels in the delta. Underneath sits a two-channel RAG pipeline — ChromaDB, MiniLM embeddings, dialogue-phase metadata filters — over 18 technique cards.

What it measured

4 matched control/treatment legs per topic, scored in 1 batched blinded-judge call, across 18 technique cards.

Result

Reproducibility owned for a 4-person team: the stack runs offline on Ollama with Llama 3.1 8B at zero marginal API cost, sets up with one command, and is verified by a GitHub Actions CI matrix across 3 operating systems and 2 Python versions. Moving off paid APIs mattered more than it sounds — it meant trial count stopped being decided by budget. The study was also scoped around a hard constraint: testing the hypothesis on real users was unsafe, so it runs against a simulated proxy over non-partisan policy topics with no personal data.

On the engineer resume

• Built a two-channel RAG harness (ChromaDB, MiniLM embeddings, dialogue-phase filters) over 18 technique cards, scoring matched control/treatment legs in 1 batched blinded-judge call returning structured JSON, cancelling judge calibration bias.
• Owned reproducibility for a 4-person team: offline Ollama (Llama 3.1 8B) inference at zero API cost, one-command setup, and a GitHub Actions CI matrix across 3 OSes and 2 Python versions.