Put a Cheap Judgment in Front of Your Expensive One: Cutting LLM Latency and Cost with a Typed Gate
Three frontier models deliberate for fifteen seconds. A 300 ms typed judgment in front of them refuses junk and skips a wait 82% of runs never needed.
Council asks three frontier models the same question, in parallel, then synthesises one verdict out of their answers. A run takes fifteen to twenty-five seconds and costs four paid API calls. That is the product, and the slowness is the point — you are watching a panel deliberate.
The problem with a pipeline shaped like that is it commits before anything has looked at the input. Every question gets the full fifteen seconds, including the ones that did not deserve four calls, and including the ones that were not questions at all.
Yesterday I put a 300 ms judgment in front of it. Not a fourth model in the panel — a gate: one call that answers two yes/no questions about the raw input and hands code two probabilities. This is the pattern, the numbers behind each decision, and the two places I tried it and backed out.
The two wastes at the top of every run
The research planner. Some questions need current facts — prices, weather, schedules, anything after a model’s knowledge cutoff. A small Gemini call decides whether to look something up and writes the search queries if so. It runs concurrently with the quota check, and the expert fan-out cannot start until both settle: median 1,087 ms, max 2,532 ms, on a 4 s deadline.
Only 18% of production runs end up researched — 24 of 135 over thirty days. So 82% of runs pay a second of latency to be told “no”.
Non-questions. Nothing checked that the input was a question. “Привет”,
“тут ?”, “Ej mam pytanie” and “عطني معلومات اول مره اسمعها” each ran three
experts plus synthesis, took fifteen to twenty-five seconds, cost four calls,
and were persisted as public /s/[id] verdict pages — in the sitemap, where
Google could index a council’s careful deliberation on the word “hello”.
Both are the same bug. The pipeline had no cheap opinion about its own input.
The pattern: 300 ms of judgment, then 15 seconds of reasoning
What goes in front cannot be another chat model. A small LLM returns text, which means you parse it, which means you own the malformed-output problem in the one place that has to be reliable — and you pay a second of latency to save a second of latency.
What works is a model whose output is already typed. I used TypeSafe’s Jev, a System One model: it reads natural language and returns calibrated probabilities over answers you define, with no text and no reasoning trace. One POST, five questions in the body, and the shape of the reply is a contract rather than a hope.
The whole call is the question text, today’s date, and the asker’s approximate city:
const result = await askSystemOne(
{ question, today, approximate_user_location },
{ answerable, needs_live_data, stakes, taste, pick },
{ signal: AbortSignal.timeout(1_500), label: "preflight" }
);Measured over 46 questions: median 283 ms, max 933 ms. It runs alongside the quota check, which the run was waiting on anyway, so on the happy path it costs nothing at all.
Then code — not a model — decides:
answerablebelow 0.2 refuses the run before a single expert call.needs_live_databelow 0.5 stops waiting for the planner and proceeds with no research.
Pick the primitive, not the prompt
The part I got wrong on the first attempt was reaching for a prompt when I should have been choosing a return type. System One gives you three, and the choice is about what the answer means:
| Primitive | Returns | Use it for |
|---|---|---|
| Noul | P(yes), no separate confidence | Whether a condition holds |
| Score | Distribution over ordered levels | Degree along a described dimension |
| Choice | One option, plus a distribution over all of them | One of a defined set |
Both gate questions are Nouls, because both are genuinely yes/no. A Noul at 0.5 does not mean “medium” — it means yes and no are equally likely, which for a gate is exactly the reading you want.
What separated a real question from noise was not the instruction. It was the criteria, with examples:
answerable: {
type: "noul",
instructions:
"Is `question` an actual question or decision a panel of advisors could " +
"give a useful verdict on, rather than a greeting, a test message, a " +
"fragment with no ask, or noise?",
criteria: {
true: {
what: "Contains a real ask, even if short or badly phrased",
examples: ["Тонзилгон применение", "Buy or rent in Madrid?"],
},
false: {
what: "Greeting, 'are you there', 'I have a question' with no question, gibberish",
examples: ["Привет", "hello?", "I have a question"],
},
},
}Those examples are load-bearing. “Тонзилгон применение” is two Russian words,
no verb, no question mark, and a real ask — it scores 0.91. “Ej mam
pytanie” is a fluent Polish sentence that asks nothing — 0.05. Without
the examples the model reads punctuation and fluency; with them it reads
whether there is an ask. That is one edit to criteria, not prompt
engineering.
Thresholds are measured, not guessed
0.2 and 0.5 look arbitrary because, written down, they are. What makes
them defensible is that both sit in a measured gap.
| Gate | Threshold | Junk / negatives | Real / positives |
|---|---|---|---|
answerable |
0.2 | 0.01 – 0.16 | ≥ 0.63, most ≥ 0.9 |
needs_live_data |
0.5 | ≤ 0.70 | ≥ 0.92 |
The planner fixture is thirty questions, half of them research-positive. Every positive scored 0.92 or above and every negative 0.70 or below, so 0.5 leaves 0.2 of margin on each side — I can move the threshold a long way before either class crosses it.
The threshold is not the deliverable, though. The bench script is. It runs both gates over the fixtures and fails the build on recall below 1.0, precision below 0.9, any junk item let through, any terse real question refused, or a median over 600 ms. Current run: precision 0.94, recall 1.00, median 296 ms.
That gate is what makes the number a product decision rather than a superstition. A threshold with no bench behind it is a guess you will be afraid to touch in three months.
Keep the policy in code, where you can change it for free
The temptation with a calibrated model is to ask it for the conclusion. Don’t — ask for the judgment and keep the conclusion in your own code, because that is the half you will want to change.
Council also asks Jev how reversible a decision is, as a Score over three levels: easily reversible, costly to reverse, one-way door. The obvious read is the expected level the API returns. It is also wrong. A question split 50/50 between “easily reversible” and “one-way door” has an expectation of 1.0, which rounds to “costly to reverse” — a level the model considered the least likely of the three.
So the policy is one line of code: take the highest-probability bucket, and on an exact tie keep the lower level.
// argmax over the distribution, not the rounded expectation
for (const l of [0, 1, 2] as const) {
const p = stakes.probabilities[String(l)];
if (p !== undefined && p > best) { best = p; level = l; }
}The same goes for display thresholds. Whether a question reads as “mostly
personal taste” or “mostly facts” is TASTE_MIN = 0.6 and FACTS_MAX = 0.4
in a constants block. Moving either is a deploy, not a re-run: the evidence
and the question’s meaning have not changed, so there is nothing to re-infer.
Raw judgments stay reusable; policy stays explicit and cheap.
Fail open, or don’t ship it
A gate in front of a working pipeline is a new way for that pipeline to break.
The rule I would not bend: every failure path returns null, and null
means the route behaves exactly as it did before the gate existed.
No key, 1.5 s timeout, HTTP error after one retry, a body that isn’t JSON — all of them log a warning and yield no verdict, and no verdict means no refusal and no planner skip. The feature can be switched off by deleting an environment variable.
Two details earned their place:
One retry, then stop. 408, 429, 5xx and TypeSafe’s 529 retry once after 300 ms. Not twice — the caller’s deadline is 1.5 s, so a second retry could never land inside it. The retry policy is a function of the deadline, not of what feels robust.
A malformed answer is dropped, not thrown. One request carries five questions, and the two gate questions ride in the same body as three user-facing ones. If the response is not an object, or has no answers at all, that throws. But a single answer that fails validation is dropped with one warning and the rest are handed to their callers. Otherwise one bad read on a cosmetic feature would cost the run its “is this even a question” gate — the blast radius of a body I do not control would be the whole pipeline.
And log the decision. Every run past the quota check records both probabilities, what the route did with them, and — when the planner was awaited — Gemini’s own verdict on the same question. Jev-versus-Gemini agreement is now a query rather than a belief. I have shipped a routing rule on a number I had never checked before, and I would rather not do it twice.
Once every input goes through the gate, more questions are almost free
The unexpected dividend: the expensive part of this pattern is the round trip, not the question count. Once every submitted question is already going to Jev, additional judgments over the same state cost almost nothing, and they run in parallel — they cannot see each other’s answers, so there is no ordering to pay for.
Council’s gate went from two questions to five: the two gates, plus how reversible the decision is, plus whether it turns on taste or on facts, plus — when the question names alternatives — which one Jev would pick. Median went from 283 ms to 313 ms, p90 710 ms, max 851 ms.
Those three extra reads are a user-facing feature. An “Instinct” card appears above the expert seats about half a second after submit, with the stakes and taste pills and a bar per option, while the panel is still thinking — a snap judgment beside a deliberation, and later a line saying whether the council agreed with it. The latency budget for it was zero, because the request was already in flight for the gate.
That is the part worth stealing even if you never build a gate: if you are paying for a round trip to a judgment model, ask everything you will want to know about that state while you are there.
The two places it didn’t work
Missing-context detection. The idea was to notice when a question omits the one fact that decides it and ask the user before running the panel. Jev’s answers came back at 0.2–0.6 confidence — a distribution too flat to threshold. That is a signal to redesign the question and build a labelled set, not to ship it at a lower bar. Cut.
Replacing the planner outright. A judgment model returns judgments, not
search queries, and the research step needs queries. So a “yes” on
needs_live_data still waits for Gemini exactly as before. The gate only
removed the wait on the 82% of runs that were always going to get a “no” —
which was the whole win, and is a smaller claim than “we replaced a model”.
One thing to be straight about: the gate call sends the question text, the date and the approximate city. It never sends user ids, anonymous ids, session ids or emails. But there is no per-request zero-retention flag on that API today, the way there is on the search call this pipeline makes elsewhere, so those inputs sit under the provider’s standard retention. That belongs in your privacy copy before the feature belongs in production.
What transfers
If you run an expensive AI pipeline — several models, a long chain, anything where one input costs real money and real seconds:
- Find what the pipeline commits to before it knows anything. In Council it was a planner wait and an unvalidated input. Both were measurable in production data I already had.
- Gate it with typed output, not text. The thing in front has to be more reliable than what it protects.
- Choose the primitive by what the answer means. Condition, degree, or one-of-a-set.
- Put the examples in the criteria. That is where the accuracy came from.
- Measure the gap, then set the threshold inside it — and keep the bench.
- Fail open on every path, and log every decision.
The honest caveat: these are bench numbers and thirty days of production history behind the problem, but the gate itself has been live for a day. I know it refuses junk and skips the planner. Whether the 300 ms judgment agrees with three frontier models often enough to widen its role is a query I cannot run yet, and I have written down what it would take to answer it before I do anything else with it.
Council is live at trycouncil.app, on the App Store and on Google Play. If you have an AI pipeline that pays full price for inputs that don’t deserve it, and you want a second pair of eyes on where a cheap gate would go, get in touch.