probabilities, a distribution over your options, and confidence, the margin between the two best options.
Probabilities
probabilities maps each of your option ids to a probability:
- The keys are exactly the option ids you sent, in request order. Krun never renames, drops or adds options.
- The values form a distribution over the options you provided, and add up to about 1. Rounding can make the sum differ slightly.
- They are relative to your option set. Adding, removing or rewording options changes every probability.
probabilitiesis always present, also when the model abstains.
Confidence
confidence is the difference between the highest and the second-highest probability:
confidence ranges from 0 to 1:
- Close to 1: one option clearly wins.
- Close to 0: the two best options are nearly tied.
Why a margin
Krun uses the margin because, in Krun One’s evaluation, it separated answerable requests from requests where no option applies as well as the alternatives tested, with the lowest error rate on accepted answers. The abstention threshold is applied to this value: whenconfidence is below the threshold, the model abstains.
A margin also catches cases a single probability misses. With three options at 0.48, 0.47 and 0.05, the top probability looks moderate, but the margin of 0.01 shows that the model can’t tell the first two apart.
Using the signals
- Branch on
choiceandabstainfirst. They already apply the model’s threshold. - If you need a stricter policy, for example before an irreversible action, add your own threshold on
confidence, tuned on your traffic. - Log
confidenceand send feedback so you can see how it relates to errors in your data.