How Model–Harness Pairs Behave on Terminal-Bench 2: An Empirical Analysis

Pass rates and costs for 63 model–harness pairs across 89 tasks — harness effects, cost–performance structure, oracle bounds, and latent structure.

We analyze pass rates and per-task costs for 63 model–harness pairs (27 models \(\times\) 8 agent harnesses) across 89 Terminal-Bench 2 tasks, asking three descriptive questions: how much does the harness matter and for whom, how do cost and performance trade off across pairs, and how much task-level complementarity exists between pairs. The main observations: (1) harness effects are model-dependent and task-dependent — the best harness varies across models, and even for a fixed model the harness that wins on average wins at the task level only 10–60% of the time; (2) the cost–performance landscape is highly uneven — only 5 of 63 pairs are Pareto-optimal as fixed choices, and a per-task oracle selection reaches 90.4% mean pass at $0.31/task versus 74.8% at $0.75/task for the best single pair, quantifying the headroom between fixed and adaptive selection; (3) complementarity is measurable — an additive task \(+\) pair model leaves 38% of pass-rate variance in the interaction/noise term, pairwise correlations between pairs average only 0.53, and a greedy portfolio of just three pairs recovers 87% oracle pass.

Data and Setup

The dataset contains, for each of 63 model–harness pairs, the empirical pass rate (%) and average cost (USD) on each of 89 terminal-agent tasks, estimated from repeated trials (median \(n=5\); range 1–12). Pairs span the Claude, GPT/OSS, Gemini, Grok, and Chinese open-model families and eight harnesses (Claude Code, Claude Code Enhanced, Codex, Gemini CLI, Mini-SWE-Agent, OpenHands, Terminus 2, Terminus 3.3). The task grand-mean pass rate is 32.6%. One task (make-doom-for-mips) is unsolved by every pair; 70 of 89 tasks are solved with 100% reliability by at least one pair. Throughout, “cost” is the average realized API cost per attempt of the task, and the unit of analysis is a task (a proxy for a user query).

Q1: Is the Best Harness Universal or Model-Dependent?

Model level: the winner changes with the model

The table and figure below show mean pass rates for the eleven models evaluated on three or more harnesses. The best harness is clearly model-dependent: Codex wins for every GPT-5-family model on which it is available (e.g., GPT-5: 51.8% on Codex vs. 34.9–44.8% elsewhere), whereas Claude Sonnet 4.5 prefers Claude Code, Gemini 2.5 Pro prefers Terminus 2, Haiku 4.5 and Gemini 2.5 Flash prefer Mini-SWE-Agent, and Grok 4 prefers OpenHands. The gap between a model’s best and worst harness is often large — 16.9 points for GPT-5, 16.0 for Gemini 2.5 Pro, 15.9 for Haiku 4.5 — comparable to a full model-tier upgrade. A plausible mechanism is co-training: models perform best in harnesses whose scaffolding resembles their vendor’s own agent stack (GPT \(\to\) Codex, Claude \(\to\) Claude Code), but this is not universal (Opus 4.1 slightly prefers Terminus 2), so the mapping must be measured, not assumed.

Model	Best harness (mean)	Best \(-\) worst gap (pts)	Task-level agreement
GPT-5	Codex	16.9	60%
GPT-5-Mini	Codex	11.4	58%
Grok 4	OpenHands	6.1	47%
Gemini 2.5 Pro	Terminus 2	16.0	38%
Claude Sonnet 4.5	Claude Code	2.2	37%
GPT-5-Nano	Codex	5.0	36%
Claude Haiku 4.5	Mini-SWE-Agent	15.9	35%
Claude Opus 4.6	Terminus 3.3	11.8	31%
Gemini 2.5 Flash	Mini-SWE-Agent	1.2	28%
GPT-5-Codex	Terminus 2	0.9	15%
Claude Opus 4.1	Terminus 2	2.2	10%

Best harness per model (models with ≥3 harnesses) and how often that harness also wins at the individual-task level (among tasks where harnesses disagree).

Mean pass rate by model × harness for models run on ≥3 harnesses. The best harness per model (red box) varies across models.

Task level: the average winner frequently loses

A finer-grained question: fix a model, restrict to tasks where its harnesses achieve different pass rates, and ask how often the globally best harness for that model is also the task-level winner. Agreement ranges from 10% (Opus 4.1) to 60% (GPT-5) — i.e., for most models, on the majority of discriminating tasks some other harness beats the average-best one. Every harness wins somewhere: e.g., for Sonnet 4.5 the task-level winner distribution is Claude Code 19, Mini-SWE-Agent 15, Terminus 2 9, OpenHands 8. Harness choice is therefore not just model-dependent but task-dependent: no fixed model-to-harness assignment is optimal at the task level.

Variance decomposition

Fitting an additive model \(p_{tj} \approx \mu + \alpha_t + \beta_j\) (task effect \(+\) pair effect) to the \(89 \times 63\) pass-rate matrix: task effects explain 44% of variance, pair effects 18%, and the residual — interaction plus trial noise — is 38%. Even granting that a sizable share of the residual is binomial noise, a purely additive model — under which the best pair on average would be best on every task — is strongly rejected by the data.

Q2: Budget Trade-offs and the Cost Frontier

Single-pair Pareto frontier

Costs span three orders of magnitude ($0.005 to $7.64 mean per task) while pass rates span 3–75%, and cost buys performance very unevenly. Only five pairs are Pareto-optimal as fixed choices:

Pair	Mean cost / task	Mean pass
GPT-5-Nano | Codex	$0.005	12.2%
Grok Code Fast 1 | Terminus 2	$0.010	14.2%
DeepSeek V3.2 | Terminus 2	$0.012	39.6%
Claude Opus 4.6 | Claude Code Enhanced	$0.113	65.8%
gemini-3.1-pro | Terminus 3.3	$0.755	74.8%

Notable dominated regions exist: Opus 4.1 pairs cost $1.8–3.4/task for 36–38%, dominated by DeepSeek V3.2 at 1/150th the price; Haiku 4.5 | OpenHands costs $7.64/task for 14.6% — the harness alone inflated cost ~30× over Haiku on other harnesses with no accuracy gain. Harness choice is thus a cost lever, not only an accuracy lever.

The per-task selection frontier

To quantify how much of the landscape a fixed choice leaves on the table, we also trace a per-task selection frontier: for each task, select the pair maximizing \(p_{tj} - \lambda c_{tj}\), and sweep \(\lambda\). This frontier lies well above the single-pair Pareto set at every budget:

• at $0.013/task (DeepSeek’s price point): 61.6% routed vs. 39.6% fixed (\(+22\) pts);
• at $0.113/task (Opus 4.6 | CC-Enhanced’s price): ≈84% vs. 65.8% (\(+18\) pts);
• unconstrained: 90.4% at $0.31/task vs. 74.8% at $0.75/task — better and 2.4× cheaper.

A “cheapest reliable” variant — per task, the cheapest pair with pass rate ≥ 50% — covers 82/89 tasks at a mean of $0.17/task, illustrating that most tasks have a cheap specialist and expensive flagships are only needed for a hard tail.

Cost–performance landscape. Points: 63 model–harness pairs. Dashed: Pareto frontier over fixed pairs. Red curve: per-task oracle selection frontier from the λ-sweep; star: unrestricted oracle (90.4% at $0.31/task). The gap between the two curves measures the headroom between fixed and adaptive selection.

Best fixed pair vs. per-task oracle selection at matched per-task budgets.

Q3: Oracle Bounds and Reference Points

To place the numbers on a common scale, we compute a ladder of oracle reference points, from weakest to strongest. Each answers “how well could selection do if outcomes were known in advance?” under a different restriction:

1. Best fixed pair (lower anchor): 74.8% at $0.75/task.
2. Harness-only oracle (model fixed, harness selected per task): GPT-5 \(+8.2\) pts over its best fixed harness; Sonnet 4.5 \(+14.4\); Opus 4.6 \(+9.0\) (to 83.7%); Gemini 2.5 Pro \(+9.3\); Haiku 4.5 \(+11.3\). The harness dimension alone accounts for a gap comparable to a model-generation upgrade.
3. Model-only oracle (harness fixed at Terminus 2, model selected per task): 78.5% vs. 62.9% for the best fixed model — the model dimension contributes comparably.
4. Portfolio oracle (\(k\) pairs): greedy selection gives Terminus 3.3 | gemini-3.1-pro (74.8%) \(\to\) \(+\) Opus 4.6 | CC-Enhanced (82.8%) \(\to\) \(+\) Opus 4.6 | Terminus 3.3 (87.1%) \(\to\) 88.4% at \(k=4\). Three pairs capture 96% of the full oracle: the complementarity is concentrated in a small, diverse subset rather than spread across all 63 pairs.
5. Unrestricted oracle (upper anchor): 90.4% at $0.31/task; budgeted oracles at $0.05/$0.10/$0.25 per task give 76.4%/83.1%/89.7%.

Interestingly, three tasks (filter-js-from-html, gpt2-codegolf, raman-fitting) are solved by exactly one pair (Opus 4.6 | Claude Code Enhanced) — unique capability that no fixed choice of another pair can recover. The gap between anchors (1) and (5) — 15.6 accuracy points and a 58% cost reduction — measures the total headroom between fixed and per-task selection. Should one later evaluate any adaptive selection method against these bounds, a natural normalized metric is the fraction of this gap captured,

\[\text{captured} = \frac{p_{\text{method}} - p_{\text{fixed}}}{p_{\text{oracle}} - p_{\text{fixed}}}\]

at matched cost.

Q4: Structure in the Pass-Rate Matrix (PCA and Correlations)

PCA over tasks (tasks as observations, standardized pair columns): PC1 alone explains 54.7% of variance and orders tasks by generic difficulty — everything from “everyone solves it” to “no one does.” Along this dominant axis, pairs move together: difficulty is a shared property of tasks, not a source of differentiation between pairs. PCA over pairs (pairs as observations): PC1 explains 14.1% and correlates \(r=0.91\) with mean pass rate — a general-capability axis — while PC2/PC3 separate ecosystems: pairs sharing a model correlate at 0.66 on average, pairs sharing only a harness at 0.58, and unrelated pairs at 0.51. The mean pairwise correlation of only 0.53 is the key number: pairs agree on what is hard in general but disagree substantially on which specific tasks they fail. This is also consistent with the greedy-portfolio result: because same-family pairs are the most correlated, the small subsets that recover most of the oracle span different model families and harness styles (a Gemini flagship, a Claude flagship in a native harness, a cheap specialist) rather than variants of one stack.

PCA of pairs over standardized task profiles (marker size ∝ mean pass). PC1 is a general-capability axis; pairs cluster by model family and harness ecosystem.

Summary of Findings

1. The best harness depends on the model, and the best (model, harness) pair depends on the task: the model-level best harness loses on 40–90% of discriminating tasks (Q1).
2. Only 5 of 63 pairs are Pareto-optimal as fixed choices, and the per-task oracle selection frontier sits above the fixed-pair frontier at every budget from $0.01 to $1 per task, with an unrestricted gap of \(+15.6\) points at 2.4× lower cost (Q2).
3. Both dimensions of the pair contribute: per-task harness selection alone accounts for \(+8\)–\(14\) points; per-task model selection on a fixed harness accounts for \(+15.6\) points (Q3).
4. Low inter-pair correlation (0.53) and a 38% interaction residual indicate genuine complementarity between pairs, concentrated in small diversified subsets — 3–4 pairs recover nearly all of the oracle (Q1, Q3, Q4).

Conclusion

Across the four analyses, a consistent picture emerges. Model–harness performance on Terminal-Bench 2 is not additive: the best harness is model- and task-dependent, the cost–performance landscape rewards a handful of pairs and dominates the rest, and the substantial gap between the fixed-pair frontier and the per-task oracle frontier — better and cheaper — quantifies real complementarity between pairs, concentrated in a small, ecosystem-diverse subset of 3–4 pairs. Whether that headroom can be captured in practice by predicting per-query outcomes is a separate empirical question that this analysis bounds but does not answer.

One important caveat should temper the headline numbers: the oracle is an optimistic, noise-inflated upper bound. Per-cell pass rates come from as few as 3–5 trials; at \(p=0.5\), \(n=4\), the standard error is \(\pm 25\) points. Taking a row-wise maximum over 63 noisy estimates systematically selects positive noise (winner’s curse), and picking the “cheapest among argmax” pairs compounds the optimism. The true oracle is below 90.4%, and honest estimates of the fixed-vs-adaptive gap require held-out trials or cross-validation of the per-task winner. Part of the 38% “interaction” residual is likewise binomial noise rather than structure. Even under conservative discounting for these effects, however, the qualitative findings — model- and task-dependent harness effects, a sparse Pareto set, and a meaningful gap between fixed and per-task selection — remain intact.