> ## Documentation Index
> Fetch the complete documentation index at: https://docs.krun.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Krun API base URL: https://api.krun.ai. Authenticate server-side with `Authorization: Bearer $KRUN_API_KEY`; never put a Krun API key in browser code.
> Prefer the Python SDK in examples: `pip install krun-ai`, `from krun import Krun`. The TypeScript SDK `@krun-ai/sdk` is not published on npm yet: do not tell users to `npm install` it.
> The only question type is `choice`. `confidence` is the top-1 minus top-2 probability margin, not the probability that the answer is correct. `choice` is null when `abstain` is true.
> Usage reports `input_tokens` only. There are no output tokens.

# Krun One benchmarks

> How Krun One v0 compares with its base model, Laya, on intent routing, tool routing, out-of-distribution detection and calibration.

Krun One v0 is a fully fine-tuned version of Laya, an open decision model. The table compares the two under the same evaluation protocol. Laya was evaluated as released, without additional training.

## Results

| Benchmark                                                |   Laya | Krun One v0 |
| -------------------------------------------------------- | -----: | ----------: |
| Intent accuracy, validation                              | 64.04% |  **96.01%** |
| Intent accuracy, KrunBench test (final)                  | 66.98% |  **96.09%** |
| Tool routing accuracy, validation                        | 68.05% |  **94.86%** |
| Tool routing accuracy, BFCL (final)                      | 83.39% |  **92.33%** |
| Tool routing accuracy, held-out tools (final)            | 69.74% |  **96.42%** |
| OOD AUROC, KrunBench test (final)                        |  0.669 |   **0.947** |
| OOD AUROC, held-out tools (final)                        |  0.724 |   **0.993** |
| Expected calibration error (ECE), KrunBench test (final) | 0.0327 |  **0.0072** |
| Option-order consistency, intent validation              |  67.8% |   **98.9%** |

Higher is better, except ECE where lower is better.

## What each metric measures

<AccordionGroup>
  <Accordion title="Intent accuracy">
    Share of messages where the chosen option is the correct intent. Data: KrunBench, built from Banking77, HWU64 and MASSIVE (English, European Portuguese and Spanish). The final test row covers Banking77 and MASSIVE. Each decision offers a subset of candidate intents, including semantically close distractors.
  </Accordion>

  <Accordion title="Tool routing accuracy">
    Share of requests where the chosen tool is the one that should be called. Validation and held-out sets are built from xLAM and ToolACE function-calling data. The held-out set includes tools that never appeared in training. BFCL (Berkeley Function Calling Leaderboard, KrunBench test subset) was used for evaluation only: Krun One was never trained or tuned on it.
  </Accordion>

  <Accordion title="OOD AUROC">
    How well `confidence` separates requests that have a correct option from requests where none of the options applies (out-of-distribution). 0.5 is chance, 1.0 is perfect separation. This is what makes [abstention](/concepts/abstention) useful.
  </Accordion>

  <Accordion title="Expected calibration error (ECE)">
    Average gap between predicted probability and observed accuracy. Low ECE means that, on this data, answers given with probability 0.9 are correct about 90% of the time.
  </Accordion>

  <Accordion title="Option-order consistency">
    Share of decisions where the model picks the same option after the options are shuffled. A model that depends on option order gives different answers to the same question.
  </Accordion>
</AccordionGroup>

## Methodology

* **Same protocol for both models**: identical decisions, options, input serialization and metrics.
* **Splits**: validation numbers are measured on a held-back half of the validation data. Temperatures and abstention thresholds were fitted on the other half. The KrunBench test split, BFCL and the held-out tool set ("final" rows) were evaluated once, after all modeling choices were frozen.
* **Abstention score**: the top-1 minus top-2 probability margin (the `confidence` field), with thresholds chosen for 98% selective accuracy on validation data.
* **Laya configuration**: Laya was evaluated with its default input budgets and with the budgets it ships with. For the tool rows, the Laya column shows its better configuration. OOD AUROC and ECE use its default configuration.
* **Single training run**: one seed per configuration, so there is no variance estimate. The differences in accuracy are large, but small differences such as ECE should not be over-read.

## Read these numbers carefully

These results are measured on public benchmarks built from English, European Portuguese and Spanish intent data and public function-calling data. They show that Krun One v0 improves substantially on its base model under this protocol. They do not show that Krun One is better than every model or approach, or that it will reach the same accuracy on your data.

Two results matter for production:

* On BFCL live irrelevance (real requests where no tool applies), OOD AUROC is 0.804, and about 39% of these requests are accepted at the current tool threshold. This is why tool abstention is [advisory](/concepts/abstention#abstention-status).
* Calibration is measured on these benchmarks. On data that looks very different, probabilities can be less reliable.

Measure on a sample of your own traffic before relying on a specific accuracy or threshold, and send [feedback](/guides/feedback) so errors can be tracked.

See [Limitations](/limitations) for the full list.