JEV Question Types: Using Choice, Score and Noul Properly

In the article on what JEV is I mentioned the three question types in passing. In practice this is exactly where quality is decided: the value of the answer you get depends on the shape of the question you asked. Picking the wrong type, or writing sloppy criteria, is the single most common reason people end up blaming the model.

Here I go through all three with working code and a rule for choosing between them. At the end I get to where the real win is: asking dozens of questions in one call.

Three types, three answer shapes

Type	Question	Fields returned
Choice	"Which of these?"	choice, probabilities, confidence
Score	"Which level?"	score, legend, probabilities, confidence
Noul	"Is this true?"	noul (probability between 0 and 1)

All three can be mixed in a single request, and they all evaluate the same state.

Choice: a closed set with no ordering

Use Choice when the answer is one of a known set with no order between the options: routing a ticket to a department, classifying a document type, detecting a programming language.

from typesafe_sdk import Choice, TypeSafeClient

client = TypeSafeClient()

response = client.system_one(
    state={"email": "Hello, please find September's invoice attached."},
    questions={
        "email_type": Choice(
            instructions="Which category does `email` belong to?",
            criteria={
                "invoice": "A payment document or invoice is being sent.",
                "sponsorship": "A partnership or advertising offer.",
                "support": "A problem with a product or service is reported.",
                "spam": "Unsolicited bulk mail.",
            },
        ),
    },
)

answer = response.answers["email_type"]
print(answer.choice)         # "invoice"
print(answer.probabilities)  # {"invoice": 0.94, "spam": 0.03, ...}
print(answer.confidence)     # 0.91

A single Choice question accepts up to 255 options. That sounds excessive until you look at TypeSafe's own cookbooks: picking one capability out of a 182-item catalogue, or placing a document into one of 75 industry groups. 255 is a design decision, not decoration.

Two practical rules:

• Write descriptions, not just labels. Instead of "invoice", write "invoice": "A payment document or invoice is being sent." The model decides from the definition, not the label.
• Close the set. Leave an "other" or "unclear" option for inputs that fit nothing. Without it, the model is forced into the nearest wrong box.

Score: a scale you can describe

Use Score when the answer falls on a spectrum and you can put each point of that spectrum into words: bug severity, customer frustration, candidate experience.

from typesafe_sdk import Score

questions = {
    "urgency": Score(
        instructions="How urgent is this support request?",
        criteria=[
            "Informational, can wait.",
            "The user is annoyed but not blocked.",
            "Work has stopped, same-day action required.",
            "Data loss or a security risk is involved.",
        ],
    ),
}

Score needs at least two levels and the API accepts up to ten. Adding levels does not buy accuracy; levels that cannot be told apart just blur the distribution. Three or four well-described levels beat eight vague ones.

One important limit: the documentation states that Score levels are weakly calibrated numerically. Do not try to reconstruct an intermediate value like "somewhere around 2.4". Use Score to test a threshold ("is it above 2?"), not to measure a magnitude.

Noul: when the probability itself is the signal

Use Noul for a clean yes/no question where the probability itself is what you need: does this message contain personal data, is the customer asking for a refund, does this sentence make a firm factual claim without a source.

from typesafe_sdk import Noul

questions = {
    "personal_data": Noul(
        instructions="Does `message` contain a phone number, address, or ID number?",
    ),
}
# response.answers["personal_data"].noul -> 0.02

The most common mistake here is reading 0.5 as "medium". It does not mean medium, it means undecided. It is not "the candidate has a medium skill level"; it is "yes and no are equally likely". If you want a middle level, that is a Score question, not a Noul.

Rules for writing instructions and criteria

All three types take the same two inputs: instructions (the question you are asking) and criteria (options, levels, or a yes/no clarification).

JEV answers the question you wrote, not the one you meant. Scoping words, negations, and implied conditions are read at face value. So:

• Refer to state fields by name, like `ticket.messages[0].text`. Do not say "in the text" — say which field.
• Avoid double negatives and indirection. "Isn't this content not inappropriate?" loses accuracy. The docs call this indirection and note that every extra hop of reasoning costs you.
• Never let instructions and criteria disagree. If they ask for different things the model gets confused. Write criteria as a continuation of the instruction.
• Split ambiguous questions. "Is this request both urgent and privileged?" is two questions. Ask both and combine them in code.
• Clean up the state. Content unrelated to the decision lowers accuracy. Select the relevant fields in code rather than dumping everything and hoping the model filters.

Applying those five rules moves the needle more than switching models does.

Probability and confidence are not the same thing

Choice and Score answers return two different numbers, and they are widely confused.

Probabilities are the likelihood assigned to each individual option. Confidence is a single number summarising the shape of that distribution: high when probability concentrates on one option, low when it spreads out.

Consider a three-option question returning [0.90 / 0.06 / 0.04] versus [0.40 / 0.33 / 0.27]. Both have the same winner, but in the second the model is close to flipping a coin. Confidence compresses that difference into one number; for three options it is approximately (3 × largest probability − 1) / 2. Full concentration gives 1.0, a uniform spread gives 0.0.

The thresholds the documentation suggests:

• Below 0.5: the model is genuinely unsure. Do not guess — route to a human or stop the flow.
• 0.5 to 0.9: act directly for low-stakes actions (showing information, tagging). Ask for confirmation on high-stakes ones.
• Above 0.9: you can proceed even on irreversible operations, but still confirm.

The value here is that you get two axes. The answer tells you what; confidence tells you whether to act. A classic classifier has no second axis: you either invent a threshold or treat every prediction the same.

A warning: do not carry thresholds between types. A 0.7 that works for a Noul does not mean the same thing as 0.7 confidence on a Choice. Calibrate each question on your own data.

The real win: many questions, one call

Now the important part. JEV evaluates every question in a request in parallel. Adding a question barely changes the response time and adds only the cost of that question's tokens — which is tiny.

response = client.system_one(
    state=state,
    questions={
        "refund_requested": Noul(instructions="Is the customer requesting a refund?"),
        "request_type": Choice(instructions="What is the main request?", criteria={...}),
        "frustration": Score(instructions="How frustrated is the customer?", criteria=[...]),
        "personal_data": Noul(instructions="Does the message contain personal data?"),
        "needs_approval": Noul(instructions="Does this require manager approval?"),
    },
)

In a published TypeSafe cookbook, batching a 13-question briefing into one call comes out roughly 12x cheaper and 10x faster than 13 separate calls, with no change in the answers.

This inverts the usual habit. Normally we think "don't ask unless you need it"; here the opposite is rational: ask the questions you only might need. The docs call the pattern speculative fan-out. Whichever way the user's flow branches, the answer is already in hand and no second round trip is needed.

The same logic unlocks checks you skipped because they were too expensive: a personal-data check on every incoming message, a relevance and prompt-injection check on every retrieved passage, a policy check on every response. Expensive one at a time, nearly free as an extra question in a call you were already making.

If you cannot decide which type to use

A simple decision flow:

1. Does the answer come from a closed set with no ordering? → Choice
2. Does it fall on a spectrum whose points you can describe? → Score
3. Is it a clean yes/no where the probability itself is useful? → Noul
4. None of the above? It is probably a generation task, and not JEV's job.

The fourth case comes up more often than you would think. "Summarise this email" is not a JEV question. "Which department should this email go to" absolutely is.

To see the patterns applied to real scenarios, read JEV use cases; for what to know before shipping, read harness, cost and limits.

Technical values come from TypeSafe's official documentation and can change between versions.

Frequently Asked Questions

What is the difference between Choice and Score in JEV?

Choice picks one option from a closed set with no ordering (department, document type, language). Score places the input on a scale of ordered levels (urgency, frustration, severity). If your answers have a "less than / more than" relationship use Score; otherwise use Choice.

What does 0.5 mean in a Noul answer?

It means undecided, not "medium". Yes and no are equally likely. If you want to measure an intermediate level you need a Score question instead of a Noul.

How many options can a Choice question have?

The API accepts up to 255. In practice accuracy improves noticeably when each option carries a short description instead of being a bare label, and it is good practice to include an "unclear" option for inputs that fit nothing.

What is the difference between confidence and probability?

Probability is the likelihood assigned to each individual option; confidence is a single number describing how concentrated that distribution is. You can see the same winning option with very different confidence values, and confidence is what you use to decide whether to act on the answer.

How many questions should I put in one call?

All of the ones that share the same state. Questions are evaluated in parallel, so an extra question barely affects latency and adds only its own tokens to the bill. In a published example, batching 13 questions into one call was roughly 12x cheaper and 10x faster than asking them separately.

Should I write my questions in English?

The documentation notes English is the primary training language. Writing instructions and criteria in English while leaving the evaluated text in its original language is a reasonable starting point, but compare both setups on 50–100 examples from your own data before committing.