Understanding Confidence and Calibration

Confidence is not an extra opinion

The most common misreading of Jev's API is treating confidence as a second output: the answer, and then the model's separate self-assessment of that answer. It is not. confidence is a statistic computed from probabilities, which you already have in the same response.

From TypeSafe's confidence docs: "confidence is a statistic computed from the probability distribution the answer already gives you."

This matters because self-reported confidence from a text-generating model is notoriously unreliable. Ask an LLM to rate its certainty and you get a number generated by the same process that generated the answer, with all the same failure modes. Jev's confidence is not generated at all. It is arithmetic over the distribution.

Which means it carries no information the distribution did not already have. It is a convenience: a single scalar to threshold on so you do not have to compute a spread statistic yourself. When the default does not suit your use case, you have the raw probabilities and can compute your own. TypeSafe is explicit that you are not locked into their definition.

The shape of the distribution is the signal

Every Choice and Score answer includes probabilities, a map summing to 1. All the certainty information lives in the shape of that map.

Probability concentrated on one outcome is a confident answer. Probability spread evenly is an uncertain one. That is the entire intuition, and confidence just collapses it to a number in [0, 1].

A clean case:

{
  "department": {
    "type": "choice",
    "choice": "returns",
    "confidence": 1.0,
    "probabilities": { "shipping": 0.0, "returns": 1.0, "billing": 0.0 }
  }
}

And a genuinely ambiguous one, from a ticket that mentions a late delivery, a wrong size, and a double charge:

{
  "department": {
    "type": "choice",
    "choice": "returns",
    "confidence": 0.42,
    "probabilities": { "shipping": 0.04, "billing": 0.35, "returns": 0.61 }
  },
  "requested_resolution": {
    "type": "choice",
    "choice": "refund",
    "confidence": 0.2,
    "probabilities": { "replacement": 0.34, "refund": 0.4, "information": 0.02, "exchange": 0.24 }
  }
}

Read the second one carefully. choice is refund, which a naive integration would act on. But refund holds only 0.40, with replacement at 0.34 and exchange at 0.24 right behind it, and confidence is 0.20. The customer genuinely never said what they wanted. The top-line choice field is nearly meaningless here, and the distribution is telling you so clearly. Acting on .choice without checking .confidence throws away the most useful part of the response.

The formula, with a caveat

TypeSafe's docs include an interactive demo, and that demo discloses its own arithmetic. For a Choice with n options, where peak is the largest probability:

confidence = (n × peak − 1) / (n − 1),  clamped to [0, 1]

The docs word it as approximating confidence for three options: (3 × largest probability − 1) / 2.

Label this carefully: it is a demo approximation, not a documented specification of the production statistic. The docs describe it as what "this demo uses" to approximate confidence. TypeSafe's actual computation is not published, and they explicitly reserve the right to have you compute your own instead.

That said, the formula is instructive about the intended semantics. Sanity-check it on three options: all probability on one option gives (3 − 1) / 2 = 1.0; a perfectly even split at 0.333 gives 0.0; a clear winner at 0.90 gives (2.7 − 1) / 2 = 0.85.

So it is a rescaling of the peak probability, mapping "uniform" to 0 and "certain" to 1, normalized for how many options there are. Note the consequence: 0.50 on two options is maximally uncertain, but 0.50 on ten options is a strong signal, and the normalization handles that. Note too that it only reads the peak. Two distributions with the same maximum get the same confidence even if the runner-up differs wildly. If the gap between first and second place matters to your routing, compute a margin from probabilities yourself.

Why Noul has no confidence field

This surprises people, and the reason is clean.

A Noul's distribution has exactly two outcomes, yes and no. A two-outcome distribution is fully described by one number: if yes is 0.93, no is necessarily 0.07. There is nothing left for a separate statistic to summarize.

So the noul value is the answer and the certainty at once. Near 1 is a strong yes, near 0 a strong no, near 0.5 is the model saying it splits evenly.

Real recorded jev-1.13.0 values for "Is the customer asking for a human agent?":

"I need this sorted today" is urgent but never asks for a person: 0.26. "Are you a bot?" gestures at wanting a human without asking: 0.40. Those middle values are not defects. They are the model correctly reporting that the proposition is genuinely borderline, and they are exactly the cases your code should route to a person.

Which gives Noul a three-way threshold rather than a two-way one:

YES = 0.8
NO = 0.2
 
wants_human = answers["is_human_escalation"].noul
 
if NO < wants_human < YES:
    # The model isn't sure either way. Let a person decide.
    send_to_review(message)
elif wants_human > YES:
    route_to_agent(message)
else:
    route_to_bot(message)

One caution that belongs here. A Noul value is not a scale of the thing you asked about; it is the probability the proposition is true. Asking "Is the candidate strong in Python?" and reading 0.5 as "medium skill" is wrong. TypeSafe's docs show a candidate with two years of daily Python use scoring 0.81 on that Noul, against 2.05 on a four-level Score. If you want degree, use a Score with levels you wrote, so the model is judging descriptions you chose rather than a boundary you left implicit.

Score confidence and the expectation trap

Score confidence works the same way as Choice confidence, over levels rather than options. But Score has an extra hazard, because score is an expectation.

{
  "bug_severity": {
    "type": "score",
    "score": 1.43,
    "confidence": 0.35,
    "legend": {
      "0": "Cosmetic; no impact to functionality",
      "1": "Broken or degraded feature, but workaround exists",
      "2": "Blocking issue; no workaround exists"
    },
    "probabilities": { "0": 0.0, "1": 0.57, "2": 0.43 }
  }
}

score is the probability-weighted mean: 0 × 0.0 + 1 × 0.57 + 2 × 0.43 = 1.43. Confidence is 0.35 because the model is genuinely torn between "workaround exists" and "no workaround", which matches the input: Chrome works, but some customers only use Safari.

The trap is that different distributions produce the same score. A score of 1.0 can mean all the probability sat on level 1, or that it split evenly between levels 0 and 2. The first is a confident middle reading. The second is a confident bimodal reading that happens to average to the middle, and it means something completely different. Confidence distinguishes them, which is why you should never act on score alone.

And from TypeSafe's jaggedness notes: do not use a Score expectation to reconstruct an exact numeric magnitude by interpolating between levels. Thresholding on the expectation is supported. Treating 1.43 as "43% of the way to blocking" is not.

Low confidence on a Score usually means one of three things: the levels overlap for this state, the question is measuring more than one dimension, or the state does not contain enough to place it. The second is the most common and the most fixable. A level described as "punctual and smart and experienced" is three questions wearing a trenchcoat, and an input that is high on one and low on another cannot be placed anywhere. Split it.

Thresholds scale with consequences

There is no universally correct confidence threshold. There is a correct threshold per action, determined by what happens if that action fires on a wrong answer.

The docs' own worked example, from a voice banking interface, makes the structure obvious:

action = response.answers["intent"]
 
# Below 0.6 confidence on any action, route to a human
if action.confidence < 0.6:
    route_to_support_agent(account_id)
 
elif action.choice == "check_balance":
    # Low stakes. 0.6 confidence is sufficient.
    show_balance(account_id)
 
elif action.choice == "approve_transfer":
    if action.confidence > 0.85:
        # High stakes, but high confidence. Safe to act automatically.
        approve_transfer(account_id)
    else:
        # High stakes, moderate confidence. Verify intent first.
        ask_user_to_confirm(
            "Just to confirm: you would like to approve this transfer, is that correct?"
        )
 
else:
    route_to_support_agent(account_id)

Three things to take from this shape:

There is a global floor. Below 0.6 nothing automated happens regardless of what was chosen. This catches the case where the model is telling you it does not have a read.

Above the floor, thresholds differ per action. check_balance acts at 0.6. approve_transfer needs 0.85. Same model, same call, same response. The difference lives entirely in your code, because the risk lives entirely in the consequence.

The middle band has a third behavior. approve_transfer between 0.6 and 0.85 does not act and does not escalate to support. It asks the user. Binary act-or-escalate throws away the most useful thing about a medium-confidence answer, which is that it is probably right and cheap to confirm.

Note also that the confidence docs use a 0.5 floor with a >0.9 bar for the transfer, while the routing pattern page uses 0.6 and >0.85 for the same scenario. That inconsistency between two TypeSafe pages is itself the lesson: these are illustrative starting points, not derived constants. TypeSafe says so directly, advising you to start conservative, test on your own data, and adjust.

Does calibration actually hold?

Thresholding is only sound if confidence tracks correctness. That is a testable claim, and someone outside TypeSafe tested it.

Emil Lindfors (lindfors.no) evaluated Jev on 24 Norwegian policy documents and reported that calibration held: at confidence of 0.9 and above, answers were correct about 94% of the time.

Two reasons this is a stronger result than it looks. First, it is independent, so it is not TypeSafe's benchmark on TypeSafe's chosen tasks. Second, Norwegian is not Jev's primary training language, and TypeSafe explicitly warns that non-English accuracy is lower and that you should pay close attention to confidence when routing non-English workloads. Calibration surviving a domain where raw accuracy is expected to be weaker is exactly the property you want, because it means confidence degrades honestly instead of staying falsely high.

Accuracy tells you how often the model is right. Calibration tells you whether you can know when it is likely wrong. The second is what lets you build a system with a predictable error budget.

Read the number precisely, though. 94% correct at 0.9+ confidence means roughly 1 in 17 high-confidence answers was still wrong. If your action at that threshold is irreversible, a 6% error rate may be far too high, and the fix is a higher bar plus a confirmation step, not a belief that 0.95 means certain.

This is also the practical answer to the "it can't hallucinate" debate. Jev can be confidently wrong, as its own founder conceded. What calibration buys you is that confidently wrong is rare and measurably rare, on your own data, with a number you can tune against.

How to actually set your thresholds

Do not copy the numbers from the docs. Derive yours:

1. Label a few hundred real examples from your own distribution. Not synthetic, not cherry-picked.
2. Run them and bucket by confidence. 0.9+, 0.8-0.9, 0.7-0.8, and so on.
3. Compute accuracy per bucket. This is your calibration curve, and it is specific to your task.
4. Price your errors. What does a false positive cost? A false negative? They are rarely symmetric.
5. Set the threshold where expected cost crosses the cost of escalation. Escalation is not free either, and sending everything to a human defeats the purpose.
6. Pin the model version. Aliases move. jev-latest resolves to jev-1.13.0 today, and a future release can shift the distributions your thresholds were tuned against. If you have tuned carefully, send the versioned ID and upgrade deliberately.

That last point is easy to skip and expensive to skip. Your thresholds are calibrated against a specific set of weights. TypeSafe says as much: pin the version and move on your own schedule.