Wiring JEV Into Your System: Harness, Cost and Known Limits

The sentence repeated most often about JEV is this: "without a system around it, it is useless." That is not an exaggeration, it is the definition. JEV does not give you an action; it gives you a cloud of probabilities. You write the machine that turns that cloud into a decision and that decision into an action.

In the harness article I described the model as a function and the tool as the program wrapped around it. JEV is the purest form of that split: the model really is just a function, and everything else is yours.

This article is about that everything else.

The five jobs of the harness

In a JEV integration, your code owns the following:

1. Preparing the state

You decide what goes to the model, and this single step drives the quality of the whole integration. The documentation is explicit about a limit here: as the state grows with content unrelated to the decision, accuracy falls. Unrelated detail acts as a distractor.

So "send everything and let the model filter" does not work. Select the relevant fields in code, name them, and refer to those names in the question.

2. Designing the questions

Put every question that shares a state into one call. Questions are evaluated in parallel, so an extra question costs almost nothing in latency. That makes asking the "might need it" questions rational; your code decides later which answers it uses.

3. Using confidence as a gate

The answer tells you what; confidence tells you whether to act. Your code should implement a three-way split: act, ask for confirmation, hand over to a human.

answer = response.answers["request_type"]

if answer.confidence < 0.5:
    escalate_to_human(ticket)
elif answer.confidence < 0.9 and is_risky(answer.choice):
    ask_user_to_confirm(answer.choice)
else:
    apply(answer.choice)

Set those thresholds from your own data, not from intuition — and do not carry them from one question to another.

4. Turning the decision into an action

JEV says "use this tool"; you call it. It says "this ticket is urgent"; you put it in the queue. This layer looks boring but it is where the system's real behaviour lives: retries, rollbacks, logging, audit trail.

5. Measuring

Skip this and the rest is meaningless. Hand-label at least 50–100 real examples, compare against JEV's answers, and break accuracy down by confidence band. The typical finding: accuracy is very high above 0.9 confidence and close to random below 0.5. That table, not your intuition, sets your threshold.

Four architecture patterns

TypeSafe's documentation names the patterns; all of them hold up in practice.

Speculative fan-out. In a single call, ask the "might need it" questions alongside the ones you certainly need. Whichever way the user's flow branches, the answer is in hand and no second round trip is needed.

Confidence-gated routing. Treat confidence as a second axis. The same answer is processed automatically at high confidence and sent to a human at low confidence.

Composite scoring. Do not squeeze a complex judgement into one question; break it into atomic scores and do the weighting in code. You can then validate each component separately and change the weights without retraining anything.

Intent routing. Classify the incoming request and send each to the best handler: deterministic code, a specialist LLM, or a person. In the documentation's example an order-status query goes straight to the database without ever touching an LLM, a product question goes to a model loaded with product context, and a complaint goes to a model or a human depending on its complexity score.

Combine the four and the architecture that falls out is this: expensive intelligence runs only when it is genuinely needed, and with the right context. Everything else is handled by a cheap decision layer and plain code.

Known limits

TypeSafe publishes the known weak spots of jev-1.13 on a dedicated page. That transparency makes your job easier, because every item below will bite you in production if you are not aware of it.

Literal reading. The model answers the question you wrote, not the one you meant. Scoping words, negations, and implied conditions are taken at face value. State conditions explicitly, spell out boundary cases, split ambiguous questions and combine them in code.

Counting and maths. It does not count reliably — characters in a word, occurrences of a term, items in a long list. Do counting in code with a regex or a parser; if needed, ask one question per item and sum the results.

Numeric representations. Hex values, RGB triples, and binary-encoded data are weak areas. Convert them into meaningful buckets in code and ask about those.

Score interpolation. Score levels are weakly calibrated numerically. Do not try to interpolate between two levels; check whether a threshold was crossed.

Dates and time. It reads dates as text, not as ordered quantities. Which date comes first, how far apart they are, whether one falls in a window — all unreliable. Extract the date parts with Choice and compare in code.

Indirection. Double negatives, a property of a property, or anything needing several hops of reasoning costs accuracy. Write directly.

Large state with irrelevant detail. Mentioned above, worth repeating: unrelated content lowers accuracy.

Adversarial content. Text written to steer the model — an injected instruction, a deliberately misleading framing, or text arguing for its own classification — can move the answer. Write explicit criteria and test this scenario specifically before going live.

Contradictory instructions and criteria. If the two ask for different things, the model gets confused. Treat criteria as an extension of the instruction.

Structural invariants. Do not assume identities like "P(yes) = 1 − P(no)" hold across separate questions. Word every question for its direct meaning and do not transfer thresholds between primitives.

Generation. Not trained for it. Forcing it by chaining choices is possible but works poorly and is very slow.

Look at that list and a pattern emerges: everywhere JEV is weak is exactly where code is strong. Counting, comparing, arithmetic, sorting — that is deterministic code's job anyway. Leave the model only the part that requires genuine judgement.

Working out the cost

Pricing fits in two sentences: input is $0.042 per million tokens, output is free. Context is 64k tokens total per request, with 32k for the state plus the longest question. Rate limits are given as 250,000 tokens per second and 1,200 requests per minute, with a note that they are adjusted dynamically under load.

The maths:

daily cost ≈ (daily requests × average input tokens) / 1,000,000 × 0.042

The key point: every additional question about the same state costs only its own tokens. Because you are not resending the state text, batching 13 questions into one call came out roughly 12x cheaper and 10x faster than separate calls in a published cookbook.

Practical conclusion: optimise questions per call, not calls per day. Most teams do the opposite.

A shipping order

The sequence I would recommend:

1. Pick one decision. Start with the highest-volume, lowest-risk one. A moderation label is a good start; a payment approval is not.
2. Run it in shadow mode. Record JEV's decision without acting on it. Put it side by side with your current system.
3. Set the threshold from your data. Build an accuracy table broken down by confidence band.
4. Open it gradually. Automate the high-confidence cases first and leave the rest to humans. Raise the automation rate over time.
5. Keep records. Store the state, the questions, the answers, and the confidence for every decision. When the model version changes (jev-latest is an alias, not a fixed version) that log is the only way to see what moved.

The fifth point matters more than it sounds. Aliases are convenient, but pinning a version — jev-1.13.0 — is more predictable in production. Test a new version against your own data before moving.

Letting your coding agent do it

You do not have to write this integration by hand. TypeSafe publishes an official agent skill:

claude plugin marketplace add typesafe-ai/skills
claude plugin install typesafe@typesafe-ai

For other agent environments, npx skills add typesafe-ai/skills --skill typesafe-ai does the same. The skill teaches the agent the question types, the architecture patterns, and the evaluation practices, so you can ask it to "find the places where fragile parsing code could be replaced by intelligent judgement."

For how skills work and what to check before installing one, see the agent skill article.

Closing thought

The question to ask about JEV is not "how smart is it?" but "should my code make this decision, or should a model?" Three tests settle it: does the answer come from a closed set, can a rule be written, and what does a wrong decision cost?

If a rule can be written, write code. If the answer is open-ended, use an LLM. The wide space in between — decisions where no rule can be written but the answer is short — was until now handled either crudely with keywords or wastefully with an expensive model. That gap is exactly what JEV fills.

The rest of the series: what JEV is, Choice, Score and Noul, 10 use cases.

Limits and pricing come from TypeSafe's jev-1.13 documentation and can change between versions.

Frequently Asked Questions

Can I use JEV on its own?

No. JEV produces a decision, not an answer: an option, a score, or a probability. The flow that acts on it — preparing the state, branching on confidence, performing the action, and logging the result — is your responsibility. Without a system around it you are left holding numbers.

What are JEV's known weaknesses?

It does not count, does not compare dates, does not do arithmetic, is weak on numeric representations such as hex and RGB, does not interpolate between Score levels, loses accuracy on double negatives and indirect questions, is affected by large irrelevant state, and can be moved by adversarially written text. All of these appear on the vendor's own published list.

What confidence threshold should I use?

There is no fixed answer; derive it from your data. As a starting point the documentation suggests routing below 0.5 to a human, acting automatically between 0.5 and 0.9 for low-stakes actions and with confirmation for high-stakes ones, and confirming even above 0.9 for irreversible operations. Do not carry a threshold from one question to another.

How is JEV's cost calculated?

Input is $0.042 per million tokens and output is free, so daily cost is roughly "(requests × average input tokens) / 1,000,000 × 0.042". Because the state is not resent for each additional question about it, extra questions add very little.

Should I pin the model version?

For production, yes. jev-latest is an alias and may point at a different version over time, which means behaviour can change. Pinning a specific version such as jev-1.13.0 and moving only after a new one passes your own accuracy test is more predictable.

Where should I start with a JEV integration?

Pick your highest-volume, lowest-risk decision, run it in shadow mode first (record the decision without acting on it), build an accuracy-versus-confidence table from your own data, then automate only the high-confidence cases. Raise the automation rate as the measurements justify it.