Skip to main content

​
Stream

stream: true returns tokens as they are produced. Total generation time is unchanged; time to first visible token drops, which is what people feel.
Available at every level; at Encrypted the deltas are ciphertext.

​
Cache Repeated Prefixes

A long, stable prefix costs processing time on every call. Mark it with cache_control and a supporting provider serves it from cache. Stable content first, the question last. Confirm hits in the cache accounting in usage. See Prompt Caching.

​
Right-Size the Model

A smaller model that is good enough responds faster. Compact models for classification, extraction, and short replies; heavyweight models for hard reasoning. Select from GET /v1/models. See Choosing a Model.

​
Stay Under the Body Caps

Chat request bodies: 64 MiB. Other inference routes: 4 MiB. Inline attachments: 1 MiB decoded. Oversized requests are rejected before inference. Trim history you do not need and supply large documents by URL.

​
The Privacy Trade-Off

Confidential variants run only where the model has an attested build, which can mean higher latency or lower availability. Reserve stronger levels for work that needs them.

​
Checklist

  • Streaming on for anything a person watches.
  • Caching on for repeated prefixes.
  • Smallest adequate model.
  • Bodies under the caps; large documents by URL.
  • Stronger privacy levels where they earn their keep.

How the API Works

The request lifecycle.

Streaming

Token by token.

Prompt Caching

Cache markers.

Choosing a Model

Task to model.

Privacy Levels

The four levels.

Cost Control

Spend.