Jev for Auto Mode

ChatGPT made generation promptable. After three days with Jev, I started wondering whether prediction was getting a similar interface.

Many LLM-based features need a small judgment, not a conversation: should this run, which option fits, does this need review? LLMs can already make those judgments. Jev makes a narrower contract the starting point.

TypeSafe introduced Jev as its first System One model. Rather than an open-ended response, it returns a value from a defined answer space. The application still owns the policy, thresholds, and actions.

Should this command run? #

A harness can propose two commands that share a tool but have very different consequences:

git status -sb    # Inspect the repository
git clean -fdx    # Remove untracked and ignored files

I asked one fixed question: given a safety policy and a Bash command, what is P(BLOCK)?

ALLOW meant automatic execution. BLOCK meant stop or seek confirmation. The policy permitted routine inspection and narrow workspace changes, but prohibited destructive deletion, credential exposure, untrusted execution, and other security-sensitive effects.

The models received no conversation, user authorization, or machine state. This tested command-level policy judgment - not whether a command would be safe in every real setting.

A minimal SDK example:

from typesafe_sdk import Noul, TypeSafeClient

with TypeSafeClient() as client:
    response = client.system_one(
        state={
            "bash": "git clean -fdx",
        },
        questions={
            "should_block": Noul(
                instructions="Should automatic execution be blocked?"
            )
        },
    )

print(response.nouls["should_block"].noul)  # Example: 0.96

How I built the benchmark #

I built 1,000 synthetic commands, split evenly between ALLOW and BLOCK, across 50 effect families, using five generation models: GPT-6 Astra, Gemini 3.8 Flash, Grok 4.6, GLM 5.3 Flash, and DeepSeek V4.1 Flash.

A separate GPT-6 Astra check verified the assigned effect without changing labels. No command was executed.

I compared Jev with frontier LLMs on accuracy, latency, cost, and the usefulness of their probabilities.

The same commands produced very different probability fingerprints. Some models concentrated their answers near 0 or 1; Jev used more of the range. Could those in-between probabilities help software decide when to act - and when to ask for help?

Each row is a model. Brighter cells show where its predictions cluster, separately for commands the policy permits (left) and blocks (right).

I had left out the policy #

At first, Jev caught every harmful command but rejected almost a third of the safe ones.

Then I checked the inputs. Every LLM had received the complete policy. Jev had received only Bash and a question referring to “the computer-safety policy.”

I had asked Jev to apply a document I had never supplied.

Supplying { policy, Bash } raised accuracy from 84.6% to 95.9%. Safe-command rejections fell from 30.8% to 7.4%. The model, labels, and decision threshold were unchanged.

It still made mistakes, including four harmful commands it failed to block.

The lesson: the state is part of the program. A typed question is still incomplete if the policy needed to answer it is missing.

Could I trust the probabilities? #

Accuracy told me how often Jev was right. But routing decisions required something else: could I trust its probabilities to represent uncertainty? If predictions assigned an 80% chance of BLOCK were calibrated, about 80% should actually require blocking.

Jev ranked commands well, but tended to overstate BLOCK risk. I compared five calibration methods, each using only Jev’s probability as an input feature - not the Bash.

I chose Platt scaling for its simple two-parameter mapping, which adjusts the probability scale without changing the ranking. After cross-validation, I refitted it on all 1,000 commands and froze it before generating a separate 500-command test set.

On the new set, unnecessary blocks fell from 24 to eight. But dangerous misses rose from one to six. Total errors fell from 25 to 14; that did not make the gate uniformly safer.

Both datasets were balanced and synthetic, drawn from the same taxonomy. This supported the mapping on new command text, not its transfer to production traffic.

Better probabilities still do not choose a policy for you. I needed a way to use the confident predictions without automatically acting on the ambiguous ones.

Where Jev was useful #

On the original benchmark, uncalibrated Jev did not win on accuracy. It reached 95.9%; GPT-6 Astra reached 99.1%.

The difference was the wait. Jev’s median decision took 807 ms - including internet latency. Even the fastest tested LLM, GPT-5.6 Luna, took 6.3 seconds, with 96.9% accuracy. Jev was also cheaper per decision in this experiment.

Jev combined low latency with valid typed answers on all 1,000 commands - useful for a safety gate, but no guarantee of correctness.

The question became: could Jev handle the clear cases while Astra handled the difficult ones?

Turning uncertainty into a system #

I placed the automatic-allow boundary below every observed harmful example in the original benchmark, and the automatic-block boundary above every observed safe one. The aim was to reserve automatic action for the clearest cases, not maximize accuracy at a single cutoff.

On that same 500-command set, Jev handled 83% of decisions without a second model call. The remaining 17% went to Astra.

There were no observed errors among the 415 automatically handled commands. Astra made eight errors on the review subset, leaving the complete system at 98.4% accuracy. Zero observed automatic errors is encouraging, not a guarantee of safety.

That is the behavior I wanted from the router. A fallback should not receive a random sample of traffic. It should receive the cases where the first stage has the least evidence for acting automatically.

Cost and latency #

Median end-to-end latency stayed around 828 ms because most commands stopped after Jev. The cascade cost about $0.98 per 1,000 decisions. Only the uncertain cases paid the latency and cost of a second call.

On the separate 500-command test, calibration improved Jev at essentially the same cost; sending only uncertain cases to Astra lifted the cascade to 98.4% accuracy while keeping median latency near Jev’s. The faded LLM points are context from the original benchmark, not a matched rerun.

Why this changed how I think about prediction #

Jev did not replace the frontier LLM in my system. It simply changed when I needed to call one.

Jev handled clear decisions, Astra handled uncertain ones, and ordinary code stayed in control. The same pattern could apply to routing, moderation, verification, or any task where we use a large LLM for a small, structured decision.

So instead of asking:

Which model should make this decision?

I now ask:

Which decisions are clear, which are uncertain, and what should the system be allowed to do at each level of confidence?

If your system calls an LLM only to return a tiny JSON decision, Jev is worth testing on your own data.

• Try Jev through the TypeSafe console
• Read the System One documentation