Skip to main content
Request-rate and in-flight concurrency limits apply globally, per IP, per organization, and per credential. Read the headers rather than assuming numbers.

​
Headers

The bucket described is whichever scope has the fewest requests left. The headers cover request rate only, not in-flight concurrency, plan usage, prepaid balance, or upstream limits.

​
A 429

An in-flight concurrency rejection is also 429 rate_limited, with no fixed window: retry after one of your own requests completes.

​
Quota

insufficient_quota also returns 429 but is a funding decision: prepaid balance, plan allowance, or the Free daily allowance is exhausted. Retrying does not clear it.

​
Notes

  • Limits are checked before a stream commits, so an over-limit streaming request returns a normal JSON error.
  • If the limiter backend is unavailable, inference and MCP routes fail closed with 503 unavailable.
  • Honor Retry-After; otherwise back off exponentially with jitter. See Backoff. Bound your own concurrency. Never retry insufficient_quota in a loop.

​
Inspect current capacity

GET /api/v1/rate-limits returns snapshots for the authenticated caller without consuming capacity in the groups being inspected. Use an organization API key or an OAuth bearer with cli:inference. No admin-key permission is required. Anonymous callers and MCP tokens are not supported.
Do not send query parameters, including model or scope selectors. The gateway derives scopes from your credential and connection. An optional X-Org-ID must match your credential’s organization. Illustrative response; the numbers are not fixed plan limits:
limit, remaining, and reset_at are null when no applicable request bucket exists, including when enforcement is disabled. Zero remaining on a finite bucket means exhausted. If enforcement is disabled, enabled is false; rows allow requests and have zero retry_after. The generation row excludes model-, stream-, and command-specific policies. Snapshots also exclude concurrency, MCP limits, other HTTP groups, billing/spend caps, and Free-plan quotas. They neither reserve capacity nor guarantee admission; concurrent requests can change capacity immediately, and the buckets are not read atomically.

​
Polling and failures

Polling uses a separate management request quota. Response X-RateLimit-* headers describe that polling request, not the inspected groups; the JSON body describes the inspected capacity. Ordinary enforcement headers report capacity after the request, whereas snapshot remaining is before the next request.
  • An exhausted inspected group returns HTTP 200 with allowed: false and a retry_after value.
  • Exhausting the polling quota returns HTTP 429; honor Retry-After when supplied.
  • Limiter backend failure returns 503, not unlimited capacity or a partial snapshot.
  • Unsupported query parameters return 400; authentication/organization failures use 401 or 403.
Responses use Cache-Control: no-store. See the endpoint schema.