Skip to main content
Krun One v0 is a fully fine-tuned version of Laya, an open decision model. The table compares the two under the same evaluation protocol. Laya was evaluated as released, without additional training.

​
Results

Higher is better, except ECE where lower is better.

​
What each metric measures

Share of messages where the chosen option is the correct intent. Data: KrunBench, built from Banking77, HWU64 and MASSIVE (English, European Portuguese and Spanish). The final test row covers Banking77 and MASSIVE. Each decision offers a subset of candidate intents, including semantically close distractors.
Share of requests where the chosen tool is the one that should be called. Validation and held-out sets are built from xLAM and ToolACE function-calling data. The held-out set includes tools that never appeared in training. BFCL (Berkeley Function Calling Leaderboard, KrunBench test subset) was used for evaluation only: Krun One was never trained or tuned on it.
How well confidence separates requests that have a correct option from requests where none of the options applies (out-of-distribution). 0.5 is chance, 1.0 is perfect separation. This is what makes abstention useful.
Average gap between predicted probability and observed accuracy. Low ECE means that, on this data, answers given with probability 0.9 are correct about 90% of the time.
Share of decisions where the model picks the same option after the options are shuffled. A model that depends on option order gives different answers to the same question.

​
Methodology

  • Same protocol for both models: identical decisions, options, input serialization and metrics.
  • Splits: validation numbers are measured on a held-back half of the validation data. Temperatures and abstention thresholds were fitted on the other half. The KrunBench test split, BFCL and the held-out tool set (“final” rows) were evaluated once, after all modeling choices were frozen.
  • Abstention score: the top-1 minus top-2 probability margin (the confidence field), with thresholds chosen for 98% selective accuracy on validation data.
  • Laya configuration: Laya was evaluated with its default input budgets and with the budgets it ships with. For the tool rows, the Laya column shows its better configuration. OOD AUROC and ECE use its default configuration.
  • Single training run: one seed per configuration, so there is no variance estimate. The differences in accuracy are large, but small differences such as ECE should not be over-read.

​
Read these numbers carefully

These results are measured on public benchmarks built from English, European Portuguese and Spanish intent data and public function-calling data. They show that Krun One v0 improves substantially on its base model under this protocol. They do not show that Krun One is better than every model or approach, or that it will reach the same accuracy on your data. Two results matter for production:
  • On BFCL live irrelevance (real requests where no tool applies), OOD AUROC is 0.804, and about 39% of these requests are accepted at the current tool threshold. This is why tool abstention is advisory.
  • Calibration is measured on these benchmarks. On data that looks very different, probabilities can be less reliable.
Measure on a sample of your own traffic before relying on a specific accuracy or threshold, and send feedback so errors can be tracked. See Limitations for the full list.