Process rewards without labels · our notebook

Following the DPO-PRM hybrid thread, on a task we actually care about.

Process reward models work. Everyone agrees since the math-reasoning results. The catch has always been the same: step-level labels are brutally expensive, and nobody wants to pay annotators to grade 40-step agent traces. The DPO-PRM hybrid thread asks the obvious question — can you get step-level credit assignment out of nothing but outcome signal? Our notebook says: on our task, mostly yes.

Setup

Task: our internal code-review agent — read a diff, trace the call sites, flag real issues, draft the review. Outcome signal is nearly free: did the flagged issue survive human review, did the patch land. We had ~2k agent traces sitting in logs with exactly that signal attached, and zero step labels.

~2k traces, outcome-labeled only
  → pair traces sharing a prefix, diverging outcome
  → divergence step inherits the credit        # the trick
  → train PRM on inferred step preferences
  → re-rank / early-stop the agent at inference

The trick is embarrassingly simple: traces that share a common prefix and then diverge in outcome are treated as a preference pair at the divergence step. Aggregate enough pairs and the steps sort themselves — steps that appear disproportionately in failing branches accumulate negative credit without any human ever grading them. It's noisy per-pair and surprisingly clean in aggregate.

Numbers

With the trained PRM re-ranking candidate steps and early-stopping doomed branches: 71% → 84% pass rate on our held-out review set. For scale: our previous best intervention (prompt surgery, two weeks of it) was worth four points. This was thirteen, from data we already had.

Honest caveats, because that's the house style: one task, one domain, and the prefix-pairing trick wants dense trace coverage — thin logs make weak pairs. The PRM also inherits every bias of the outcome signal; ours over-rewards caution, because human reviewers do. And no, it's not in production yet — the eval suite that would let us trust it there is the current work. But we're close.

Writeup and eval set attached for clients; if you have a pile of outcome-labeled traces and a suspicion they're worth something — they probably are. That pile is the cheapest post-training data you own.