The A/B we couldn't win: our decision model picks attack techniques now — here's what the data actually said

What we built (and why the split)

Red-team engines have one knee-jerk design: let the big generative model decide everything — what to try, how to write it, when to stop. The problem with a generative decider is the same problem we keep writing about: it's steerable by the material it reads, its choices live in prose nobody audits, and it costs frontier money on every turn of every scan.

So we split decide from write:

• The decision model chooses. "Which of eight named techniques does the conversation state call for?" — a typed question with a bounded answer set, answered in milliseconds, logged with a digest of the exact state it saw.
• The LLM writes. Given the chosen technique, the generative model writes the actual attack message. It doesn't pick the strategy anymore.
• Every choice is enumerable. Each turn carries the chosen technique in the run's record — who picked what, from what state, is a query, not an archaeology project.

This went live this week. In the first full production-shaped run, the model made 114 selection decisions across a scan — and all eight named techniques got used in proportion to the conversation states (context anchoring and direct elicitation leading; meta-layer roleplay reserved for late-game turns). The writer followed the mandate each time. Mechanically, the split works.

The experiment that could have proven it

Mechanically working isn't the interesting question. The interesting question: does the decision model pick better techniques than the LLM did?

We ran matched scans against a live staging endpoint — same agent, same seven attack-suite groups, same conversation budgets:

The LLM arm swung from 3 breaches to zero between identical runs. The decision-model arm was steadier — one leak almost every time — but "steadier" is not "better," and at n≈40 per arm, nothing here statistically separates the two choosers. The strongest honest statement is the boring one: on this target, at this sample, both approaches find roughly the same things, sometimes.

We could have buried this run and waited for one where our arm won. Publishing it is the point — a red-team engine that only shows you a favorable quarter is a demo, not a tool.

Why we shipped it anyway

Because the case for the split was never "the decision model out-thinks the frontier model at strategy." It's four properties the LLM chooser can't offer:

1. Decisions you can enumerate. Typed choices + state digests. You can walk every strategic decision a scan made and why. "The model decided in prose" is not a record you can query.
2. Determinism where it counts. Same state, same typed question — near-same answer. Strategy stops being a dice roll across reruns (which, as the variance above shows, even a good chooser badly needs).
3. Cost and latency at the decision point. Milliseconds and rounding errors per decision instead of a frontier call — which is what lets the loop run per turn, not per epoch.
4. Not steerable in the same way. A non-generative model doesn't take instructions from the conversation it's reading. That property has a fresh, important asterisk — see below.

The asterisk we're adding to our own story

Three days ago a paper landed — "Decision Hijacking: Prompt Injection Attacks on Jev's Typed Probabilistic Decisions" (Wu & Lim) — testing exactly the class of model we use as the chooser and judge: non-generative, schema-typed outputs. Their findings, on 510 reconstructed attack cases: malicious content shifts action probabilities but rarely flips the decision itself. Under adaptive, feedback-driven attacks, success on fresh calls rose to 3.5%.

That is a good result for the architecture and a bad result for anyone (including us, previously) who casually says "it can't be talked to." Schema-typed decisions change the prompt-injection problem; they don't eliminate it. Rare is not zero.

So the intake pipeline got its own defense this week: every piece of untrusted content bound for a judge now passes a screening call first — "is this text trying to steer its own evaluation?" — and flagged content rides along with a caution note outside the untrusted text, in the trusted region of the prompt. In the live run: 30 of 30 judge-bound turns screened; benign traffic scored 0.04–0.26; a deliberately steering test string got flagged at 0.93. The screen never decides anything — it can't gate, can't block. It just makes sure the judge knows when it's being pitched.

The honest summary

• The chooser works, is enumerable from the record, and is live.
• The claim "our chooser finds more than an LLM chooser" is not supported by our data. Two arms, four runs, overlapping distributions. Revisit with more runs and harder targets.
• "Schema-typed decisions are beyond prompt injection" is not true — a fresh paper puts adaptive success at ~3.5%, and we've screened the intake accordingly.
• The value we're actually selling here is the boring architectural stuff: enumerable, deterministic, cheap, and harder to steer — harder, not immune.

If your red-team engine's strategy is "trust the big model to decide everything," you can't inspect it, can't reproduce it, and can't reason about its failure modes. That's the case. Not a leaderboard.

Methodology: matched adaptive scans against the same staging target (same suites, same turn budgets), run-to-run comparisons disclosed. The selection mechanism, decision logs, and the contested-turn adjudication live in our engineering reports; the intake-screen numbers come from the live staging run.