Reproducing LLM-as-a-Verifier with open-source models on Terminal-Bench 2.0

We set out to verify the core claim of the LLM-as-a-Verifier paper by Kwok et al. [1]: reading the probability distribution over score tokens, instead of a single score token, makes an LLM a far more reliable selector.

Three results stand out:

• Zero ties. The continuous verifier never tied. The discrete judge tied on 14% of comparisons.
• 8× fewer calls. Two verifier passes matched the accuracy a judge needed sixteen passes to reach.
• Model-dependent hyperparameters. Score granularity, which drove the paper’s gains, did nothing for our verifier model, GLM-5.2.

In this post, I explain what trajectory-level benchmarking measures, which verifier settings to tune first, and how to benchmark your own coding agent usage. The paper also covers progress tracking and reinforcement-learning rewards; this post covers verification only. However, we see progress tracking as an important capability for runtime guardrails.

Your model isn’t the bottleneck

Maybe you already write evals for your coding agent. These check outcomes: did the patch apply, and did the tests pass? They cannot tell you which of five attempts to ship, or whether the agent’s process—its tool calls, error recovery, or verification habits—was any good. So instead of choosing a model consciously, you default to the biggest model you are allowed to use.

That default wastes capability you already have. The paper’s authors pooled candidate trajectories from across the Terminal-Bench 2.0 leaderboard and found that an oracle selector—a hypothetical, all-perfect agent that always picks the best attempt—solves 98.9% of the benchmark. In other words, a solution to nearly every task exists in the pool of generated trajectories.

The bottleneck is the selector: something that can tell a correct trajectory from a plausible-looking failure. Once you can measure selection quality, you do not have to guess which model to use. You can start experimenting.

Judge vs. verifier: one token vs. the distribution

A standard LLM judge reads a trajectory and outputs one score token on a fixed scale, say 1 to 20. You take that token—the argmax of the model’s distribution—as the verdict. Trajectories that differ in important ways collapse onto the same integer, so comparisons end in ties and best-of-N selection stalls.

As the paper’s authors point out, Gemini 2.5 Flash reliably identified failure modes but expressed its conclusions in graded, hedged language such as “slightly cleaner” or “marginally more direct.” An LLM-as-a-Judge using a 1-to-5 scale collapses these nuanced assessments by forcing the model to map them to fixed integers.

LLM-as-a-Verifier reads the top logprobs—the log probabilities for the next possible token—and uses them to calculate a continuous score. These logprobs show how likely the model considers each token to be the next one. The continuous score preserves the model’s uncertainty and depends on three verifier hyperparameters:

• Granularity (G): the size of the score scale, from token A to token T on a 20-point scale.
• Repetition (K): the number of evaluations to run and average.
• Criteria (C): the decomposition of the main verification goal—correctness—into sub-criteria. These criteria vary across benchmarks.

Only certain inference providers expose top logprobs for certain open-source models. We used Fireworks AI and Together AI in our work.

The setup: Harbor with sandboxes and 100 trajectory pairs

We ran Terminal-Bench 2.0 through Harbor [2], the Terminal-Bench team’s framework for evaluating agents in containerized sandboxes. Harbor executes each task in isolation and runs the benchmark’s hidden checks, so every trajectory carries a ground-truth pass/fail label. From these runs, we built 100 pairs: for each task, one passing trajectory and one failing trajectory.

The metric is pairwise accuracy: does the scorer rate the passing trajectory above the failing one? A tie counts against the scorer because you paid for an evaluation and learned nothing.

The verifier is GLM-5.2, Z.ai’s MIT-licensed open-weights flagship, with thinking effort set to maximum. We served it through Fireworks AI and Together AI because both return top logprobs. The judge baseline uses the same model and prompt on the same 1-to-20 scale; the only difference is argmax versus expectation.

Ties drop to zero

The following comparison shows the head-to-head results at G=20:

At repetition K=1, the judge ended in a tie 14% of the time. Averaging across multiple repetitions gradually erodes the number of ties: 6% at K=4 and 2% at K=16. The important result is that the verifier never tied at any budget.

Two passes beat sixteen

The verifier scores 64% at K=1, jumps to 74% at K=2, and then stays flat through K=16. Further repetitions do not provide any measurable benefit to the score. The judge climbs from 65% to 71% to 73% and needs all sixteen passes to approach what the verifier reaches with two.

The paper reports a single-pass verifier matching the judge; we did not see that match in our work. A single evaluation carries the run-to-run variance of one long thinking trace. Future work on the verifier context and criteria definition is needed to improve the zero-shot accuracy of the verifier setup.

Granularity did nothing—and that is by design

Along the x-axis, from G=1—which uses the model’s most likely score—to G=20—which uses all scores over the full 20-point distribution—accuracy moved from 73% to 74% and then remained flat, all within the confidence band.

The paper found the opposite with Gemini 2.5 Flash as the verifier, where granularity alone lifted accuracy from 73.1% to 77.5%. Gemini 2.5 Flash and subsequent Gemini models removed logprobs from their output, so the paper cannot be reproduced in its original form with those models.

We hypothesize that this shows model-dependent behavior in pairwise accuracy with respect to the granularity parameter G. Higher granularity has no cost component, but it limits the inference providers and models eligible for verification. Most models and inference providers cap the number of top logprobs at G=5. We plan to investigate scoring granularity further.

Three narrow questions beat one broad question

At repetition K=16, each criterion alone lands between 62% and 70%: specification adherence at 70%, output correctness at 68%, and error-freeness at 62%.

The ensemble of all three reaches 74%, above the best single criterion. This mechanism closely resembles the traditional machine-learning concept of boosting: combining a set of less accurate “weak learners” to create a highly accurate “strong learner.”

For example, in a Terminal-Bench task called gcode-to-text, the agent had to inspect a 3D-printer file and identify hidden text. One trajectory produced the incorrect answer flag{gcode3_iz_chALLenGing}, while another produced the correct text, flag{gc0d3_iz_ch4LLenGiNg}.

After 16 repeated evaluations, specification adherence strongly favored the correct trajectory, with a score of 18.38 against 12.79 for the incorrect one. Output correctness scored the correct trajectory at 15.20 and the incorrect one at 7.56. The error-free criterion, however, made a mistake and narrowly preferred the incorrect trajectory, 19.50 versus 19.44. This may be because the incorrect trajectory still appeared to be a clean attempt to solve the task from a technical point of view.

What to do with this

Here is how to start verifying your own agent:

1. Log full trajectories, not just outcomes. The verifier scores the process, and outcome evals do not capture the process.
2. Sample several candidates on hard tasks. The pass@k uplift is substantial, but only if you have a selector that captures it.
3. Use three narrow criteria and repeated evaluations. Score with continuous top logprobs, average across K evaluations, and combine narrow criteria to improve final verification accuracy.

Benchmark your own agent

We measured everything above on Terminal-Bench tasks. Your agent runs on your tasks, in your harness, with your failure modes. Verifier behavior does not transfer automatically between stacks, but the trajectory-verification approach does generalize.

The raw material for creating your own benchmark already exists: every production run is a trajectory.

References

1. Kwok et al., “LLM-as-a-Verifier,” Stanford University, University of California, Berkeley, and NVIDIA. Read the paper on AlphaXiv.
2. Harbor, a framework for evaluating agents in containerized environments. Visit the Harbor Framework website.