Where a $0.04 model beats a frontier model: how Tulsk uses Jev

The decision nobody budgets for

An agent run is mostly small decisions. Did this run actually finish? Does this message want a task created? Which of these eighty tools is this turn even about? Is this task high-stakes?

Today most products answer those questions one of two ways. Either they call the same reasoning model that writes the code — seconds of latency and real money for a yes/no — or they use a keyword list, which is free and wrong in both directions, with no signal that it was wrong.

Tulsk had both. Our output-quality check was a seven-phrase blocklist. Our Slack "does this want a task" detector was a regex. Our "should we load this integration for this turn" gate was a keyword match with a documented production miss. And the quality stack that could have done better — contracts, evaluator, self-verify — was gated off in production, because running a reasoning model on every run to grade the run cost more than the run.

TypeSafe's Jev is built for exactly this gap. It is a System One model: you hand it a state and a set of typed questions, and it returns answers with calibrated probabilities. It never writes prose. Three primitives — Noul (probability a condition holds), Choice (one of a defined set, with a distribution), Score (a position on ordered levels) — cover most of the decisions above. It costs $0.042 per million input tokens, nothing for output, and answers in a few hundred milliseconds.

The interesting question was never "is Jev good". It was "where, in a real product, does it beat what is already there — and where would it just be a cheaper way to be wrong?"

Rule one: no judgment ships without a comparison

Before writing any integration we read through the codebase for decision points: places where code branches on something a model or a heuristic decided. We found about two hundred. Thirty-one were plausible fits for a typed judgment. We shipped six.

The rule we set for the platform's own decisions is simple and we have not broken it:

That rule is why this post can quote numbers instead of adjectives.

What we built

Judgments you define, as tools your agents can call

The first thing we shipped is not a platform decision at all. It is a product feature: a workspace judgment is a saved set of typed questions — say, "is this support message a bug report?" (Noul), "which team owns it?" (Choice), "how frustrated is the customer?" (Score) — that you configure once in a dialog and that becomes a tool.

In Workspace Chat it shows up as judgment_<slug>; the assistant calls it with whatever state is in play and gets probabilities back, which it can act on or show you. For an agent, a judgment is granted by slug the same way an integration is, so an agent only reaches the judgments its owner allowed. There is one seam in the codebase that talks to TypeSafe, and both surfaces go through it: one request shape, typed answers, retries, and a cost event per call.

Why this first? Because the article that made Jev popular lists support routing, refund triage, lead qualification and sales policy as its business use cases — and those are your rules, not ours. We are not going to ship a built-in "is this a refund request" classifier tuned on nobody's data. We ship the primitive and the wiring, and your workspace owns the questions.

The platform's own decisions, in shadow

Six places in Tulsk now run a Jev judgment on the platform's behalf. One is in advise mode; five are shadow-only.

Site	What it asks	Mode
Run completion (when an agent run settles)	Did the agent complete the task? If not, why — missing access, unclear task, external failure, gave up?	Advise: below 30% completion, one system comment on the task asks a person to look before relying on the result
Run failure type	What kind of failure is this error message?	Shadow, beside the substring classifier
Chat integration gate	Does this turn need each connected integration?	Shadow, beside the keyword match
Slack create intent	Does this message want a task created, and at what priority?	Shadow, beside the regex
Task criticality	How high-stakes is this task for an autonomous agent?	Shadow
Chat tool-group gate	Which families of tools does this turn need?	Shadow, with the real outcome written back

Each call costs the platform roughly 1,700 to 3,000 microcents — a few thousandths of a cent — and takes 100 to 400 ms warm. The closest reasoning call we compared it against was about eighty times more expensive and returned free text that code then had to parse.

The run-completion judgment is the one we let speak, because the thing it replaces was the worst offender: a run that said "I could not find any P0 defects" used to be flagged as a failed run by the blocklist and re-queued at the workspace's expense. Now the question is asked properly, and the answer is a probability with a reason attached.

Rule two: measure before you believe the use-case list

Here is the part I actually want to write about.

The Jev use-case lists circulating right now are good. They are also lists, and a list cannot tell you which item matters in your system. So before adding anything beyond the six above, we took the four candidates that sounded most promising for Tulsk — rerank workspace memory before injecting it, add an observability judgment over agent traces, make agent permissions dynamic per task, route platform-utility calls to cheaper models — and measured each against production data. Three of them lost.

Reranking memory lost because memory is small. The context we inject from workspace memory is a few thousand tokens at most, out of a median model call of about 57,000 input tokens. A reranker would sit serially in front of prompt assembly, adding latency to trim the smallest part of the prompt.

Trace observability lost because our failures were not mysterious. When we looked at why runs had failed, the causes were a misconfigured model id, a step limit, and a wall-clock limit — all deterministic, all already classified. There was nothing semantic left for a judgment to find.

Dynamic permissions lost on evidence, not on merit: the approvals path simply did not have the volume to evaluate a gate against. A judgment you cannot measure is a judgment you cannot promote.

Model routing lost because the calls it would route were already on the cheapest model and were a rounding error of spend.

And then the measurement pointed somewhere none of the lists had.

The one that pays: which tools does this turn need?

Workspace Chat is our general harness. Nearly every feature is chat-controllable, which means the assistant has 78 registered tools — projects and tasks, agents and runs, schedules, memory, budgets, risks, stakeholders, change requests, retros, web search, billing, and more.

Every one of those tools is described to the model, with its input schema, on every step of every turn. We measured the registry: 69,622 characters of description plus schema, roughly 17,400 tokens — before the conversation, before the workspace context, before anything the user typed.

Then we looked at what turns actually do. The most-used tool by far is the sandbox shell. After it comes a long tail: task context, task updates, web search, task search, task creation — about 25 distinct tools in a month. The median turn calls no tool at all. The 90th-percentile turn takes four tool steps, and every one of those steps re-sends all 17,400 tokens.

That is 15 to 30 percent of a median call, spent describing tools the turn was never going to use.

So we asked Jev one Noul per tool family — ten families, "does this turn need tools from this group?" — over the last user message and the two before it. The state is about 685 tokens. That is one call of about 2,900 microcents per turn.

What it saves, per step, for 15,000 dropped definition tokens at the prices in our live catalogue: 53,000 microcents on our default model, 1.1 to 1.4 million on the mid-tier models people chose last month, 7.5 million on Claude Opus. The gate pays for itself 18 times over on the first step of the cheapest model, and 400 to 2,500 times over on the others.

We ran it retrospectively on 130 real turns from our own development workspace. At a 0.2 threshold it kept every tool family the reply actually used, and would have dropped 88 percent of the definition tokens. Median latency 313 ms; it overlaps the tool setup that already happens before the first token, so it costs the user nothing to wait for.

Two more things the measurement forced on the design:

Families, not tools. "Is this turn about the agents?" is answerable from a message. "Will it need get_run_result or get_agent_runs?" usually is not. The model still picks the tool; the gate only decides which families are worth describing. Ten questions is ten chances to miss, not seventy-eight.

A tool cannot ship ungrouped. The group table is code, and a test walks the real registry and fails when a tool is missing from it or listed twice. It failed on its first run — two tools added the day before had no group. That is the kind of failure you want.

What we would tell you if you are adding Jev to your own system

Start from the decision, not the model. Find the branch in your code where a heuristic or a reasoning call decides something small, and ask what it costs to be wrong there today.

Then measure before you believe a use-case list, including this one. Three of the four "obvious" fits for Tulsk were not fits for Tulsk. The one that was did not appear on any list, because it came out of a token histogram, not a brainstorm.

Ship every platform judgment in shadow with the real outcome written back beside it. Calibrated probabilities are only useful if you check them against what happened; a threshold you assumed is a keyword list with extra steps.

And keep the model's job small. Jev cannot write code, and it cannot invent a fourth option when you gave it three. That is the point. It answers the question you asked, with a probability you can act on, for a price that lets you ask on every turn.

If you want to see the judgment tool in your own workspace, it is in Workspace Chat on every plan — including the free one. Pricing is here.

Try Tulsk free→