Timeouts and cold starts
Krun One runs on serverless GPUs that scale to zero when there is no traffic. After a period without requests, the next request starts a model worker first. This cold start, while the model loads, can take up to about 2 minutes. While workers are warm, requests are much faster. Krun does not promise a fixed end-to-end latency. Plan for both cases:- Set client timeouts of at least 180 seconds. The API waits up to 150 seconds for the model backend, including a cold start, and needs a few seconds more to respond. The Python SDK defaults to 180 seconds from
krun-ai0.3.0 (pip install -U krun-ai), and so does@krun-ai/sdk0.3.0; with an earlier SDK version, set the timeout to 180 seconds yourself. - Don’t block a user-facing request on a cold start if you can avoid it. For interactive flows, consider a fallback path, such as a default route, when a decision takes too long for your UX.
- Retry
504 UPSTREAM_TIMEOUT. It can happen when a cold start takes longer than the API’s deadline. Retrying usually reaches the worker that has started in the meantime.
Retries
Retry only errors that are transient:
Use exponential backoff with jitter, and honor the
Retry-After header when it is present. Keep the number of retries small: the API already retries its model backend internally.
The Python SDK does this for you. decide() and models() retry once by default after connection errors and 502, 503 or 504, waiting for Retry-After (up to 10 seconds) or 0.5 s, 1 s, 2 s and so on. Set max_retries to change it. The SDK never retries its own timeout, and never retries feedback().
A retried decision can at worst be counted twice for usage. It has no other side effects.
Protect your API key
- Call Krun only from your server. Never put a Krun API key in browser, mobile or desktop client code.
- Load the key from an environment variable or a secret manager, as
KRUN_API_KEY. Don’t commit it. - Use separate keys per service or environment, so you can revoke one without affecting the others.
- If a key leaks, ask Krun to revoke it and issue a new one.
Privacy
The API does not store or log the content of your requests: the context, option ids, descriptions, instructions, criteria, levels and probability distributions are not persisted. For each request, Krun records usage metadata such as the request id, model, status, input tokens, the number of questions and options, and latency. For each answer it records the question id, the question type and the model’s numeric outputs (thechoice confidence margin and whether it abstained, the noul value, the score level), which is what lets Krun measure quality without keeping your text.
Feedback stores what you send in expected, expected_decision and metadata. Don’t put personal data in them.
Request content is retained only if an owner of your organization turns on data controls. It is off by default.
For multimodal inputs, media files, OCR text and transcripts are never logged or
returned, and uploaded assets expire after 24 hours by default and can be deleted earlier.
Log request ids
Log theX-Request-ID (result.request_id in Python) next to each decision in your system. You need it to send feedback, and it lets Krun find the request if you contact support. Error responses include it too, in error.request_id.
Monitor decisions
- Track the abstention rate per question. A rising rate can mean your traffic changed or an option is missing.
- Log
confidenceand send feedback, so you can check accuracy on your own data. - Watch for
429responses and theX-RateLimit-Remainingheader. See Rate limits and quotas.