API Reference

llmsoup exposes an OpenAI-compatible HTTP API. Any application or SDK that works with the OpenAI Chat Completions API can point at llmsoup with zero code changes — just swap the base URL.

Authentication

All endpoints except /metrics require a Bearer token in the Authorization header.

Authorization: Bearer <your-token>

Tokens are configured in the llmsoup config file via auth.tokens (inline list) or auth.tokens_file (external file). When authentication is enabled and the token is missing or invalid, llmsoup returns:

HTTP/1.1 401 Unauthorized
Content-Type: application/json

{
  "error": {
    "message": "Invalid or missing authentication token",
    "type": "invalid_request_error",
    "code": "invalid_api_key"
  }
}

If authentication is disabled in the config (auth.enabled: false), all requests are accepted without a token.

---

Endpoints

Method	Path	Auth Required	Description
POST	/v1/chat/completions	Yes	Chat completion (non-streaming and streaming)
GET	/v1/usage	Yes	Calling user’s own usage snapshot (JSON)
GET	/metrics	No	Prometheus metrics

---

POST /v1/chat/completions

Request

Send a JSON body matching the OpenAI Chat Completions format.

Request fields

Field	Type	Required	Description
model	string	Yes	Model name. llmsoup routes to the best backend model based on your routing rules; you can also target a specific configured model by name.
messages	array	Yes	Conversation messages. See Message object.
temperature	number	No	Sampling temperature (0–2). Passed through to the routed model.
top_p	number	No	Nucleus sampling parameter. Passed through to the routed model.
max_tokens	integer	No	Maximum tokens to generate (deprecated — use max_completion_tokens).
max_completion_tokens	integer	No	Maximum completion tokens. Preferred over max_tokens.
stream	boolean	No	Set true for Server-Sent Events streaming. Default: false.

llmsoup tolerates additional fields (e.g., tools, tool_choice, response_format, logprobs) and passes them through to the upstream model unchanged. This means tool/function calling, structured output, and other OpenAI features work as long as the routed model supports them.

Message object

Field	Type	Required	Description
role	string	Yes	One of system, user, assistant, tool, or developer.
content	string or array	No	Text string, or an array of content parts for multimodal input. Omit for tool-call assistant messages.
tool_calls	array	No	Tool calls made by the assistant (assistant messages only).
tool_call_id	string	No	ID of the tool call being responded to (tool messages only).

Content parts (when content is an array):

[
  { "type": "text", "text": "Describe this image" },
  { "type": "image_url", "image_url": { "url": "https://example.com/photo.png" } }
]

Multimodal content parts (images, audio) are passed through to the upstream model. Support depends on the routed model’s capabilities.

Tool call object

Field	Type	Description
id	string	Unique identifier for the tool call.
type	string	Always "function".
function.name	string	Name of the function to call.
function.arguments	string	JSON-encoded arguments.

Example request

curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer your-token" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Explain quicksort in two sentences."}
    ],
    "temperature": 0.7,
    "max_completion_tokens": 256
  }'

Quota enforcement: When per-user quotas are enabled, a request from a user who is over quota is rejected with HTTP 429 and an OpenAI-compatible error of type insufficient_quota before any model is called. The quota check reads only in-memory usage state, so it adds no upstream latency. See Quotas for configuration.

---

Non-streaming response

When stream is false (default), llmsoup returns a single JSON object.

Response fields

Field	Type	Description
id	string	Unique request identifier (format: chatcmpl-{hex}-{hex}).
object	string	Always "chat.completion".
created	integer	Unix timestamp (seconds) when the completion was created.
model	string	The model that actually served the request.
choices	array	Array of completion choices (typically one). See Choice object.
usage	object	Token usage statistics. See Usage object.
service_tier	string	Service tier (optional, omitted when not applicable).

Choice object

Field	Type	Description
index	integer	Zero-based index of this choice.
message	object	The assistant’s response message.
message.role	string	Always "assistant".
message.content	string or null	Text content. null when tool calls are made.
message.tool_calls	array or null	Tool calls, if any.
finish_reason	string	Why generation stopped: "stop", "length", "content_filter", "tool_calls", or "function_call".
logprobs	object or absent	Log probability information. Currently omitted from responses (not present in JSON). Reserved for future use.

Usage object

Field	Type	Description
prompt_tokens	integer	Number of tokens in the prompt.
completion_tokens	integer	Number of tokens in the completion.
total_tokens	integer	Sum of prompt and completion tokens.

Example response

{
  "id": "chatcmpl-67a3f2c6-1a",
  "object": "chat.completion",
  "created": 1738867398,
  "model": "gpt-4o-mini",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Quicksort is a divide-and-conquer algorithm that selects a pivot element and partitions the array into elements less than and greater than the pivot. It then recursively sorts each partition, achieving O(n log n) average-case performance."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 28,
    "completion_tokens": 42,
    "total_tokens": 70
  }
}

---

Streaming response (SSE)

When stream is true, llmsoup returns a stream of Server-Sent Events. Each event is a data: line containing a JSON chunk, terminated by a data: [DONE] sentinel.

Passthrough streaming: By default, llmsoup requests SSE from the upstream provider and forwards each chunk to the client as it arrives. Delta payloads are byte-identical to what the upstream sends, and time-to-first-token is the real upstream TTFT — not the full generation time. Set defaults.stream_passthrough: false to restore the legacy buffered behavior (full completion fetched, then replayed as SSE events).

Passthrough applies when the routed decision uses the default or fallback strategy and has no response-body plugins. Decisions using the parallel strategy, or configured with semantic-cache, hallucination, or router_replay plugins, automatically degrade to the buffered path so plugin and strategy guarantees are preserved. Providers that ignore stream: true (returning a buffered JSON completion) are detected and served via the buffered path; the detection is remembered for 10 minutes, then re-probed. Non-streaming requests are completely unchanged.

Streamed requests bypass the model response cache entirely — no cache read, no cache write.

Chunk fields

Field	Type	Description
id	string	Same request identifier as non-streaming.
object	string	Always "chat.completion.chunk".
created	integer	Unix timestamp (seconds).
model	string	Model that served the request.
choices	array	Array with one chunk choice. Empty on the final usage chunk.
usage	object	Token usage. Present only on the final usage chunk, which is delivered only when the request set stream_options: {"include_usage": true}. See Usage chunk.

Chunk choice fields

Field	Type	Description
index	integer	Choice index (always 0).
delta	object	Incremental content update.
delta.role	string	"assistant" — present in the first chunk only.
delta.content	string	Content fragment.
delta.tool_calls	array	Tool call deltas, if any.
finish_reason	string or null	null until the final chunk, then "stop", "length", etc.

Usage chunk

llmsoup always injects stream_options: {"include_usage": true} into the upstream request (retrying once without it if the provider rejects it) so it can account cost, usage, and quotas. Providers that support it send a final chunk with an empty choices array and a usage object.

Matching OpenAI semantics exactly, that usage chunk is forwarded to the client only when the request itself set stream_options: {"include_usage": true}. Otherwise llmsoup consumes it internally for accounting and strips it from the stream, so clients that never asked for a usage chunk never see one.

If the usage chunk never arrives — the provider does not support it, or the client disconnects mid-stream — usage is estimated from the forwarded content (~4 characters per token) and the llmsoup_usage_estimated_total metric is incremented.

SSE wire format

data: {"id":"chatcmpl-67a3f2c6-1a","object":"chat.completion.chunk","created":1738867398,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"role":"assistant","content":"Quick"},"finish_reason":null}]}

data: {"id":"chatcmpl-67a3f2c6-1a","object":"chat.completion.chunk","created":1738867398,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"content":"sort is"},"finish_reason":null}]}

data: {"id":"chatcmpl-67a3f2c6-1a","object":"chat.completion.chunk","created":1738867398,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-67a3f2c6-1a","object":"chat.completion.chunk","created":1738867398,"model":"gpt-4o-mini","choices":[],"usage":{"prompt_tokens":28,"completion_tokens":42,"total_tokens":70}}

data: [DONE]

Each data: line is followed by two newlines (\n\n). The [DONE] sentinel signals the end of the stream. The usage chunk shown above appears only when the request opted into stream_options.include_usage.

Fallback and mid-stream errors

If a model ref fails before the first chunk is forwarded (connection error, non-2xx response, or first-chunk timeout), llmsoup tries the next ref in the fallback chain — the client sees one uninterrupted stream with no visible retry.

Once the first chunk has been forwarded, no model switch happens. An upstream failure mid-stream produces an OpenAI-format error event followed by data: [DONE], then the stream closes:

data: {"error":{"message":"Model call failed: connection reset","type":"internal_error","code":"internal_error"}}

data: [DONE]

Streaming response headers

Streaming responses include Cache-Control: no-cache and X-Accel-Buffering: no so intermediary proxies do not buffer the stream.

Because headers are sent before generation finishes, cost headers on streamed responses carry routing-time values only: x-llmsoup-cost covers routing cost (classifier calls etc.), x-llmsoup-tokens-prompt is the routing-time estimate, and x-llmsoup-tokens-completion is 0. Final completion usage arrives in the usage chunk. See Cost headers.

Streaming curl example

curl -N -X POST http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer your-token" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": true
  }'

Streaming Python example

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="your-token",
)

stream = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
print()

---

Error responses

All errors follow the OpenAI error response format:

{
  "error": {
    "message": "Human-readable error description",
    "type": "error_type",
    "code": "error_code"
  }
}

Error types

HTTP Status	type	code	When
400	invalid_request_error	null	Malformed JSON, missing required fields
400	invalid_request_error	context_length_exceeded	Request terminally exceeds the model’s context length (non-retryable; returned for both buffered and streaming requests, before the first chunk)
401	invalid_request_error	invalid_api_key	Missing or invalid Bearer token
429	insufficient_quota	insufficient_quota	Caller is over their per-user usage quota (rejected before any model call)
500	internal_error	internal_error	Routing failure, model call failure, serialization error

Common error scenarios

Scenario	Status	Message
Invalid JSON body	400	Invalid JSON: {details}
Missing model field	400	Missing required field: model
Missing messages field	400	Missing required field: messages
Request exceeds context window	400	Request of ~{tokens} tokens exceeds model '{model}' effective context window of {n} tokens ({declared} declared minus headroom)
User over quota	429	You have exceeded your usage quota for the current period.
No routing decision available	500	No routing decision available
Upstream model call failed	500	Model call failed: {error}
Response serialization error	500	Failed to serialize response

A context-window 400 is a last resort: llmsoup first prefers decision refs whose context window fits the request, truncates per defaults.context_overflow, and retries an upstream context-length 400 once after forced truncation. When none of these recover the request — or the strategy is stop_at_limit — llmsoup returns HTTP 400 with "type": "invalid_request_error" and "code": "context_length_exceeded", for both buffered and streaming (pre-first-chunk) requests. The error is non-retryable: the prompt must shrink or route to a larger-window model. See Context overflow strategies.

---

Response headers

Cost headers

When defaults.include_cost_headers is true (the default), llmsoup adds cost and routing information to every response:

Header	Type	Description
x-llmsoup-cost	string	Total request cost (model + routing) formatted to 6 decimal places (e.g., "0.001250"). Currency is set in model pricing config.
x-llmsoup-model	string	Name of the model that served the request (e.g., "gpt-4o-mini").
x-llmsoup-tokens-prompt	string	Prompt token count (e.g., "150").
x-llmsoup-tokens-completion	string	Completion token count (e.g., "42").

Cost headers are included on both non-streaming and streaming responses, but their meaning differs. On non-streaming responses they reflect the full request (model + routing cost, final token counts). On streaming responses, headers are sent before generation finishes, so x-llmsoup-cost carries the routing-time cost only (classifier calls etc.), x-llmsoup-tokens-prompt is the routing-time estimate, and x-llmsoup-tokens-completion is 0 — final completion usage is delivered in the stream’s usage chunk.

Example response headers

HTTP/1.1 200 OK
content-type: application/json
x-llmsoup-cost: 0.000125
x-llmsoup-model: gpt-4o-mini
x-llmsoup-tokens-prompt: 28
x-llmsoup-tokens-completion: 42

Plugin headers

Plugins can inject additional response headers after the standard cost headers. The built-in security plugins add the following headers when triggered:

Header	Type	Description
x-llmsoup-jailbreak-blocked	string	"true" when the jailbreak detection plugin blocks a request.
x-llmsoup-jailbreak-confidence	string	Confidence score (e.g., "0.95") for the jailbreak detection.
x-llmsoup-pii-blocked	string	"true" when the PII detection plugin blocks a request.
x-llmsoup-pii-types	string	Comma-separated PII types detected (e.g., "email,phone").

See the Plugins Reference for full plugin documentation and configuration.

---

GET /v1/usage

Returns the calling user’s own usage as a JSON snapshot. It is mounted behind the same identity-bearing auth layer as /v1/chat/completions — it requires a valid Bearer token and auth.tokens_file (which maps each token to a user_id) — and is never exposed on the unauthenticated /metrics route. A caller only ever sees the usage attributed to their own token; there is no way to read another user’s data.

An unauthenticated or invalid request is rejected with 401 in the standard OpenAI error format (type: invalid_request_error, code: invalid_api_key), identical to the Authentication section above.

Response body

The body is an envelope with two keys: usage (the per-user metrics snapshot for the requested window) and, when the caller has a quota configured, quota (their current-period limit / used / remaining).

usage — fields with no per-user source (such as active_connections) are always 0.

Field	Type	Description
total_cost	number	Total cost across the window.
total_savings	number	Total estimated savings versus the baseline model.
total_requests	number	Number of requests.
active_connections	number	Always 0 for per-user snapshots (no per-user source).
errors	number	Total error count.
model_tokens	object	Map of model name → { input, output } token counts.
model_costs	object	Map of model name → cost.
model_latencies	object	Map of model name → average latency in seconds.
model_selections	object	Model-selection algorithm counts (not populated in per-user snapshots).
triggered_routes	object	Map of matched routing rule name → count.
errors_by_type	object	Map of error type → count.

quota — present only if the caller has a quota configured; omitted entirely otherwise. Always reflects the current quota period, independent of the window parameter. Each dimension reports limit / used / remaining, where limit and remaining are null for an unlimited dimension. remaining is limit − used, clamped at 0.

Field	Type	Description
period	string	The accounting period the quota resets over (daily, weekly, monthly, or total).
requests	object	{ limit, used, remaining } request counts for the current period.
cost	object	{ limit, used, remaining } in dollars for the current period.
tokens	object	{ limit, used, remaining } token counts (prompt + completion) for the current period.

Example response

{
  "usage": {
    "total_cost": 1.234,
    "total_savings": 0.456,
    "total_requests": 128,
    "active_connections": 0,
    "errors": 2,
    "model_tokens": { "gpt-4o-mini": { "input": 40100, "output": 15980 } },
    "model_costs": { "gpt-4o-mini": 1.234 },
    "model_latencies": { "gpt-4o-mini": 0.82 },
    "model_selections": {},
    "triggered_routes": { "code_review": 40 },
    "errors_by_type": { "model_error": 2 }
  },
  "quota": {
    "period": "monthly",
    "requests": { "limit": 10000, "used": 128, "remaining": 9872 },
    "cost":     { "limit": null,  "used": 1.234, "remaining": null },
    "tokens":   { "limit": null,  "used": 56080, "remaining": null }
  }
}

Query parameters

Use window to select a preset range, or from/to for an explicit date range.

Parameter	Value	Description
window	lifetime	All-time totals with full per-model detail (default).
window	period	Current quota period (aggregate totals).
window	7d	Last 7 days.
window	30d	Last 30 days.
window	mtd	Month-to-date.
from / to	YYYY-MM-DD	Explicit inclusive date range.

lifetime and period are served from the live in-memory accumulator. The period window reports only the current quota period’s consumption and reads zero once a period boundary has rolled over, consistent with the quota block. Bounded ranges (7d, 30d, mtd, and explicit from/to) are summed from the daily rollups and carry headline totals only — per-model, latency, and route detail is available for lifetime. Bounded ranges require persistence to be enabled.

A requested start date earlier than the retained rollups is clamped to the earliest retained day; the end date is passed through as requested. The resolved window is reported in response headers:

Header	Description
x-llmsoup-usage-window	Resolved window kind: lifetime, period, or range.
x-llmsoup-usage-from	Effective start date YYYY-MM-DD, clamped to the earliest retained day. Present for bounded ranges only.
x-llmsoup-usage-to	End date YYYY-MM-DD as requested (not clamped). Present for bounded ranges only.

Example

curl http://localhost:8080/v1/usage \
  -H "Authorization: Bearer your-token"

Last 30 days:

curl "http://localhost:8080/v1/usage?window=30d" \
  -H "Authorization: Bearer your-token"

Explicit date range:

curl "http://localhost:8080/v1/usage?from=2026-01-01&to=2026-01-31" \
  -H "Authorization: Bearer your-token"

See Self-service usage for the companion llmsoup usage terminal dashboard, and Quotas for per-user limits.

---

GET /metrics

Returns Prometheus-formatted metrics. This endpoint does not require authentication.

Content-Type: text/plain; version=0.0.4

Example

curl http://localhost:8080/metrics

# HELP llmsoup_requests_total Total number of requests
# TYPE llmsoup_requests_total counter
llmsoup_requests_total{method="POST",endpoint="/v1/chat/completions",status="200"} 1523

# HELP llmsoup_active_connections Current number of active connections
# TYPE llmsoup_active_connections gauge
llmsoup_active_connections 3

# HELP llmsoup_model_request_duration_seconds Model request duration
# TYPE llmsoup_model_request_duration_seconds histogram
llmsoup_model_request_duration_seconds_bucket{model="gpt-4o-mini",le="0.5"} 1200
llmsoup_model_request_duration_seconds_bucket{model="gpt-4o-mini",le="1.0"} 1490
llmsoup_model_request_duration_seconds_bucket{model="gpt-4o-mini",le="+Inf"} 1523

# HELP llmsoup_tokens_total Total tokens processed
# TYPE llmsoup_tokens_total counter
llmsoup_tokens_total{type="prompt"} 45200
llmsoup_tokens_total{type="completion"} 12800

All metric names use the llmsoup_ prefix with snake_case naming and unit suffixes (_total, _seconds, _bytes). A full metrics reference is available in the Metrics Reference documentation page.

---

curl examples

Basic completion

curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer your-token" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [
      {"role": "user", "content": "What is the capital of France?"}
    ]
  }'

Streaming completion

curl -N -X POST http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer your-token" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [
      {"role": "user", "content": "Write a haiku about coding."}
    ],
    "stream": true
  }'

With tool calling

curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer your-token" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [
      {"role": "user", "content": "What is the weather in Paris?"}
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_weather",
          "description": "Get current weather for a city",
          "parameters": {
            "type": "object",
            "properties": {
              "city": {"type": "string", "description": "City name"}
            },
            "required": ["city"]
          }
        }
      }
    ]
  }'

Using the OpenAI Python SDK

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="your-token",
)

response = client.chat.completions.create(
    model="auto",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain recursion simply."},
    ],
    temperature=0.5,
    max_completion_tokens=200,
)

print(response.choices[0].message.content)