One session.
Every model.

Act one — the uncomfortable first result

On a set of well-known defects (SQL injection, XSS, plaintext passwords, TOCTOU…), two different-vendor frontier models — OpenAI's codex and xAI's grok — caught the same 10 of 11 defects and missed the identical one.

Different vendors, perfectly correlated errors. On familiar bug classes, cross-vendor diversity was theater — exactly as ensemble theory predicts when errors aren't decorrelated.

That's not a failure of the mission — it sharpens it. xllm's answer is to spend diversity where decorrelation is measured, not assumed: the panel ledger tracks pairwise agreement, and tiebreakers go to the model with the lowest measured agreement — never to a prettier pedigree. An instrument that can prove its own product wrong is the honest difference from "claims benefits without evidence."

Act two — on hard problems, the dividend is real

On a harder set (6 tasks, 21 subtle defects — boundary windows, retry jitter, LRU eviction, timezone edges) a five-model calibration showed the errors decorrelating: mean pairwise agreement fell from 1.0 to 0.746. A blind panel of claude / grok / gemma4 then recovered a defect the best single model missed.

claude and grok each missed two defects — but different ones, and the panel bought one back. Diversity pays exactly when errors decorrelate, and error correlation is a function of problem difficulty.

One defect remained a shared blind spot — both strong models missed it, so no union can recover it. That is the case for escalating to a fourth, measured-decorrelated model: the strong pair agrees at 0.905, while the weaker, different model decorrelates most (0.667 against each).

Act three — when one model dominates, the dividend is a routing question

A rerun of the full hard set (codex vs grok, zero provider errors) confirmed the decorrelation — pairwise error correlation 0.762, one shared blind spot — but this time one panelist dominated:

The union added nothing over the dominant single — the dividend isn't a property of "diversity" in the abstract. It depends on who your best single is. Which is a routing question.

So that is what happens now: those detection cells become trait profiles (Wilson 95% lower bounds, sample sizes attached), and the very next judgment-role pick consumed them — pick verify moved to grok citing LCB 0.7733 vs 0.5004 over 21 shared opportunities. What the benchmark measured is what the router does.

Act four — a fourth, decorrelated model recovers the shared blind spot

Act two predicted it: when both strong models share a blind spot, escalate to a fourth, measured-decorrelated model. So we did — anchoring on grok and adding three cloud models. The LRU off-by-one that both codex and grok missed on the rerun is now caught by gemma4 and nemotron-3-super (which scored 3/3 on that task).

Escalation works — but diversity is not a universal solvent. One defect (a once-emitter double-fire) is now missed by every model measured across every run: a deep blind spot no panel here can buy.

Act five — debate, measured for the first time: no quality dividend here

Debate and council shipped on design + live e2e only — the quality claim ("plausible-but-wrong claims die, correct ones survive") was never measured. Now it is: the bench grades the actual adversarial protocol, mapping each surviving claim to the seeded defects. A grounded claim maps to a real defect; a surplus one does not.

On a strong, aligned pair (codex vs grok), debate killed only 6 of 48 claims — and split them 3 grounded / 3 surplus. SURVIVED did not track truth. This is the debate analogue of act one: the dividend is conditional, not free.

The deliverable is not a positive result — it is that the instrument now exists to detect when deliberation helps. Every result also tags its measurement surface (cli-agentic vs http-completion), so a strong CLI score is never silently mistaken for raw model quality.

Act six — the cleanest dividend yet: a balanced, same-surface panel

Every panel so far carried a near-ceiling model whose high best-single ate the dividend. Ensemble theory says the dividend grows when no member dominates — so we tested it with three comparable, different-lab, all-http-completion models (no surface confound), run three times to separate signal from run-to-run variance.

Dividend +2 on every run (σ=0) — double the +1 of any dominated panel. The model ranks rotate run-to-run and individual scores swing ±3, yet the ensemble dividend does not move: the panel is more stable than any of its members.

Mean pairwise agreement 0.735 (the hard-set decorrelation regime, now with no surface confound), and zero permanent blind spots across the three runs. Three cheap, free, local models reached a per-run union of 19–20/21 — on par with a single frontier model. The mission hypothesis, cleanly confirmed: diversity pays when errors decorrelate and no single member dominates — measured, not assumed.

Act seven — lineage does not predict decorrelation

The founding question, tested head-on: does shared lineage predict correlated errors? We panelled three sizes of one family (nemotron ultra / super / nano — same lab, NVIDIA) against a different-lab model (gpt-oss:120b), 3×. Dividend rose to a mean of +2.33 (a run hit +3, the largest yet) as agreement fell to 0.691.

The same-lab pairs span the whole range and overlap the cross-lab ones entirely. The most decorrelated pair in the panel is same-lab (two nemotrons, 0.603); a cross-lab pair is among the most correlated. Shared lineage carries no reliable signal.

This is the vindication of a core design choice: the panel tiebreaker picks by lowest measured agreement, never by vendor pedigree. Routing diversity by lineage would have discarded the panel's most decorrelated pair and kept a more correlated one. The benchmark retires cross-vendor — the original v0.1 heuristic — in favour of cross-decorrelation, measured per pair.

Act eight — the maximally cross-vendor panel had the lowest dividend

The opposite extreme: three frontier vendor CLIs, one per lab, all pinned to the lowest effort — claude:sonnet (Anthropic) · codex:gpt-5.6-luna (OpenAI) · grok-composer-2.5-fast (xAI). Maximally cross-vendor, same surface, hard set, 3×.

Mean pairwise agreement 0.841 — frontier models agree, even throttled to low effort across three different labs. Two near-ceiling members (20, 19) leave no room to recover. The most cross-vendor panel produced the least diversity dividend.

The capstone: the most correlated pair in the whole benchmark is claude ↔ grok (0.920, different vendors); the most decorrelated is super ↔ nano (0.603, same lab). Pedigree diversity is here anti-correlated with decorrelation. The dividend lives in the mid-tier decorrelated regime — and only measurement tells you which panel you hold.

Act nine — three light models on one GPU: the largest dividend yet

Every dividend so far came from cloud models, and the SOTA-heavy panels recovered the least. So we went the other way: three genuinely light models whose weights fit on one consumer GPU — ornith (5.6GB) · gemma4 (9.6GB) · qwen3-coder (18GB) — all raw HTTP completion, hard set, run 3×. A full run takes ~10 minutes and costs nothing.

Mean dividend +3.33 — the largest measured in the benchmark's history — at mean pairwise agreement 0.609, the lowest ever. Across the three runs the union covered all 21 seeded defects with zero permanent blind spots; the 5.6GB model caught the double-fire bug that every frontier model had missed.

The decorrelation–dividend curve is now four points and strictly monotonic: agreement 0.841 → dividend +0.33 (frontier trio) · 0.735 → +2.0 · 0.691 → +2.33 · 0.609 → +3.33 (this panel). The weaker and more different the members, the more the ensemble recovers — three free local models reached a union within one defect of a single frontier model. What diversity cannot buy at the frontier, it pays handsomely at the light-local end — and the panel measures which regime you are in.

Act ten — council, measured: aggressive kills, zero discrimination

The last unmeasured deliberation mode. Council (blind panel → adversarial debate over the panel's claims) ran on the same light trio over every hard task, and each surviving claim was graded against the seeded defects — does hostile refutation keep the true claims and kill the rest?

Light models refuted aggressively — half of all claims died — but killed blind: real-defect claims and unmapped claims died at exactly the same rate. Capable models (frontier and mid-tier alike) barely kill; light models kill indiscriminately. No measured regime shows the deliberation story tracking truth on this instrument.

And deliberation costs recall: council's surviving claims covered only 8 of 21 seeded defects, versus 19 per run for the same trio as a blind union. The routing lesson is now measured, not aesthetic: for light local models, spend your calls on the blind panel (union, dividend +3.33) — the adversarial phase subtracts detections without adding measured precision. An honest instrument reports its negatives; this one just did.

Act eleven — high effort buys diversity, not skill

Does the balanced panel's dividend grow at higher effort? Unanswerable until v0.29 taught the ollama transport to forward @effort as the API's think parameter. Then the act-six trio ran again at @high, three times.

Thinking did not lift the ceiling — the union stayed flat and best-single actually fell. What high effort bought was decorrelation: members diverged further, headroom widened, and the union-over-best gap grew from +2 to +3. More effort made each model noisier and the ensemble better.

With five panels measured, the agreement→dividend relationship stays strongly negative but gains an honest wrinkle: this panel (0.704 → +3.0) out-dividends a more decorrelated one (0.691 → +2.33) because its best single left more room to fill. Dividend = decorrelation × headroom — both drivers, measured: 0.841→+0.33 · 0.735→+2.0 · 0.704→+3.0 · 0.691→+2.33 · 0.609→+3.33.