Skip to main content

Timeouts and cold starts

Krun One runs on serverless GPUs that scale to zero when there is no traffic. After a period without requests, the next request starts a model worker first. This cold start, while the model loads, can take up to about 2 minutes. While workers are warm, requests are much faster. Krun does not promise a fixed end-to-end latency. Plan for both cases:
  • Set client timeouts of at least 180 seconds. The API waits up to 150 seconds for the model backend, including a cold start, and needs a few seconds more to respond. The Python SDK defaults to 180 seconds from krun-ai 0.3.0 (pip install -U krun-ai), and so does @krun-ai/sdk 0.3.0; with an earlier SDK version, set the timeout to 180 seconds yourself.
  • Don’t block a user-facing request on a cold start if you can avoid it. For interactive flows, consider a fallback path, such as a default route, when a decision takes too long for your UX.
  • Retry 504 UPSTREAM_TIMEOUT. It can happen when a cold start takes longer than the API’s deadline. Retrying usually reaches the worker that has started in the meantime.

Retries

Retry only errors that are transient: Use exponential backoff with jitter, and honor the Retry-After header when it is present. Keep the number of retries small: the API already retries its model backend internally. The Python SDK does this for you. decide() and models() retry once by default after connection errors and 502, 503 or 504, waiting for Retry-After (up to 10 seconds) or 0.5 s, 1 s, 2 s and so on. Set max_retries to change it. The SDK never retries its own timeout, and never retries feedback(). A retried decision can at worst be counted twice for usage. It has no other side effects.

Protect your API key

  • Call Krun only from your server. Never put a Krun API key in browser, mobile or desktop client code.
  • Load the key from an environment variable or a secret manager, as KRUN_API_KEY. Don’t commit it.
  • Use separate keys per service or environment, so you can revoke one without affecting the others.
  • If a key leaks, ask Krun to revoke it and issue a new one.
See Authentication.

Privacy

The API does not store or log the content of your requests: the context, option ids, descriptions, instructions, criteria, levels and probability distributions are not persisted. For each request, Krun records usage metadata such as the request id, model, status, input tokens, the number of questions and options, and latency. For each answer it records the question id, the question type and the model’s numeric outputs (the choice confidence margin and whether it abstained, the noul value, the score level), which is what lets Krun measure quality without keeping your text. Feedback stores what you send in expected, expected_decision and metadata. Don’t put personal data in them. Request content is retained only if an owner of your organization turns on data controls. It is off by default. For multimodal inputs, media files, OCR text and transcripts are never logged or returned, and uploaded assets expire after 24 hours by default and can be deleted earlier.

Log request ids

Log the X-Request-ID (result.request_id in Python) next to each decision in your system. You need it to send feedback, and it lets Krun find the request if you contact support. Error responses include it too, in error.request_id.

Monitor decisions

  • Track the abstention rate per question. A rising rate can mean your traffic changed or an option is missing.
  • Log confidence and send feedback, so you can check accuracy on your own data.
  • Watch for 429 responses and the X-RateLimit-Remaining header. See Rate limits and quotas.

Keep the option set stable

Probabilities are relative to the options you send. Changing option ids, descriptions or the number of options changes the answers. Version your option sets and compare accuracy before and after a change.