Judging the judges: what 895 adversarial turns taught us about guardrail models

Why we ran this

Orithos Guard sits inline on agent tool calls and conversation turns. That means it needs a judge that answers in milliseconds, costs little enough to run on every turn, and returns a calibrated confidence we can route on.

Two candidates went to the bench:

• Jev (TypeSafe AI) — a "System One" decision model: non-generative, answers typed questions (probability, one-of-N choice, scored rubric) with calibrated confidence, in 70–500 ms.
• Laya — an open-source decision head (ModernBERT-large backbone, 421M params, Apache 2.0): same answer shape, self-hostable.

Every model card in this category carries the same line: evaluate on your own distribution. So we did. One replay harness, three suites:

Zero transport or parse errors across all runs — 1,848 calls scored cleanly. This is a capability comparison, not an integration artefact.

The numbers

Read the table slowly — the ranking isn't the story. The failure modes are.

Lesson 1: How a judge fails matters more than how often it agrees

Laya's second endpoint agreed on 61.2% of turns — superficially "three-quarters of the way to the leader." Then you look at the failure direction:

• It missed 85% of labeled breaches (recall 0.15) — including a tool-misuse case where its own two answer heads contradicted each other.
• It leaked 68 breaches into the auto-skip band (3.6× more than the alternative).
• It blocked or escalated 32 of 34 benign tool calls — read-only lookups like listing scans. That's the failure direction that breaks real operations.

The first endpoint failed the other way: systematic over-flagging (V-FULL on 51% of turns; flat confidence — no separation between right and wrong answers).

For a security gate, one failure direction is disqualifying; the other is merely annoying. Decide which one you can tolerate before you benchmark, or the numbers will decide for you.

Lesson 2: The serving endpoint is part of the model

Both Laya deployments reported the same model name: laya-rl-agent. On identical states, 71.8% of verdicts flipped between them — and the failure mode inverted (over-flagging → under-detection).

Same name, different behaviour, no version metadata to tell them apart. Treat every deployment as a versioned artefact — pin the serving build, or your benchmark measured a URL, not a model. It's also why we don't quote endpoint 1's numbers as "Laya's behaviour" — they're one deployment's behaviour, and the owner's suspicion about that deployment was consistent with what we measured.

Lesson 3: Route around the weak class — don't pretend it away

Even the winner has a weak spot: jailbreak turns. Jev agreed with our judge on 40% of them — a weakness documented in its own materials, and confirmed on our data.

So jailbreak verdicts never gate anything. They're logged, advisory, and routed to the full judge. The architecture absorbs the known weakness instead of asserting it away. A single judge with no routing logic is a single point of failure wearing a confidence score.

Lesson 4: Tuning until a model wins is not a benchmark

We didn't prompt-tune the question set per model. The questions were authored for one contract and accepted natively by both — zero parse errors, zero retries. If we'd tuned each model until it scored best, we'd have measured our tuning, not the models.

What we decided

• Jev stays as the judge for Guard's inline layer: 81.9% agreement, 0.76 breach recall (vs 0.15 on the stronger alternative endpoint), 100% detection on the crafted adversarial corpus, and zero false positives across 90 benign tool calls (34 crafted-benign + 56 organic-shaped).
• Laya is parked — with one live option: Apache 2.0 and self-hostable means it stays a plausible fine-tuning target on our labeled data (895 shadow turns + guardrail sets + the tool-call corpus). That's a build decision, not a drop-in swap.
• Fairness notes, stated plainly: both Laya endpoints were community-style deployments (quick tunnels, no stable version metadata) — a self-hosted build may behave differently, and we'd retest before quoting any of this as the model's ceiling. 39 turns were unrecoverable. Our questions were authored for a different contract; no per-model tuning. And the same rule applies to us: evaluate on your own distribution — these numbers are ours.

The three questions we'd ask any judge vendor now

1. What failure direction can your product tolerate — missed breaches or blocked operations?
2. What catches the judge when it's wrong — which weak classes route to what?
3. Can you pin the exact model build we tested — or is "the model" a moving endpoint?

If a vendor answers "our accuracy is X%" and nothing else, you've learned the least interesting number.

Postscript: we ran that retest

The fairness note above promised we'd retest on stable serving before quoting these numbers as any model's ceiling. We did — same checkpoint family, healthy hardware, same inputs, same questions, same harness:

• The stronger endpoint's profile reproduced cell for cell. Agreement, breach precision and recall, skip-band leaks, adversarial detection, tool-call behaviour — identical, to the decision.
• The incumbent held too. Our production judge re-ran the same day: 81.2–81.9% agreement (vs 81.9% above), 100% detection on the crafted corpus, zero blocks across the benign tool-call set.
• One line stays open: the over-flagging build's family (endpoint 1) hasn't been re-measured on healthy hardware.

So this is no longer "a benchmark of two URLs." It's reproducible — same states in, same decisions out, which is the exact property we were shopping for. Numbers you can't reproduce aren't results; they're anecdotes.

Methodology and raw results: this evaluation replayed production states through an identical harness for every candidate — same questions, same parser, same labels. The full numbers live in our reports; the harness re-runs as the labeled set grows.