Advanced Configuration

This page covers advanced configuration topics that go beyond the basics in the Configuration Reference. Each section dives deeper into tuning, behavior details, and practical examples.

Multi-endpoint failover

A model can have multiple endpoints for redundancy. When more than one endpoint is configured, llmsoup selects the first healthy endpoint in declaration order.

Ordered failover

Endpoints are tried in the order they are declared. The first one that passes health checks serves the request; the rest are standby. There is no traffic splitting.

models:
  - name: gpt-5-mini
    provider: openai
    access_key:
      env: OPENAI_API_KEY
    endpoints:
      - url: https://primary.openai.example.com/v1/chat/completions
        timeout_ms: 5000
        description: "Primary region (us-east)"
      - url: https://secondary.openai.example.com/v1/chat/completions
        timeout_ms: 8000
        description: "Secondary region (eu-west)"

In this example every request goes to the primary endpoint while it is healthy, and to the secondary only when the primary is not.

Endpoint fields

Field	Type	Default	Description
`url`	string	(required)	Full URL to the chat completions endpoint.
`weight`	integer (>0)	1	Accepted and validated, but ignored by routing — reserved for planned weighted distribution.
`timeout_ms`	integer	global default	Per-endpoint timeout override in milliseconds.
`description`	string	—	Human-readable label (appears in logs and metrics).

Upstream provider routing

endpoints control which URL llmsoup calls. On an aggregator, provider_routing controls which upstream provider the aggregator then hands the request to — a second, independent hop that llmsoup would otherwise have no say over.

This matters more than it sounds. A single OpenRouter model id routinely fans out to 15–35 upstream endpoints whose input prices differ by an order of magnitude, some of which do not price prompt caching at all, some of which do not support tool calling, and some of which serve a heavily quantised build. Left unstated, which one you get is the aggregator’s choice on every request.

Pinning a first-party endpoint

The cheapest endpoint for a model is often the model vendor’s own, with resellers charging a multiple. Pin it, and disallow fallbacks if you would rather fail than silently pay the reseller rate:

models:
  - name: deepseek/deepseek-v4-pro-0813
    provider: openrouter
    provider_routing:
      order: [DeepSeek]
      allow_fallbacks: false

Leave allow_fallbacks unset (or true) to keep the pin as a preference — the aggregator tries DeepSeek first and moves on if it is unavailable. That is usually the right trade for production: a price preference should not become an availability risk.

Requiring capabilities

On a tool-calling workload, an endpoint that does not support tools is a live failure waiting for a failover. require_parameters drops any endpoint that cannot serve the parameters actually present on the request:

    provider_routing:
      require_parameters: true

This is request-sensitive, not static: the same model is eligible for a plain completion and ineligible for a tool-calling one.

Excluding quantisations

Cheap endpoints are frequently cheap because they serve a quantised build. Benchmarks are published at reference precision, so a model that benchmarks well may not be the model you are served:

    provider_routing:
      quantizations: [bf16, fp8]

Protecting the cache

On a prompt-heavy workload the cached-input price is the bill, and an endpoint that prices no cache read charges full rate on every request. If a model’s cache hit rate is mysteriously zero, the usual cause is the aggregator routing to endpoints that do not cache. Restrict to the ones that do:

    provider_routing:
      only: [DeepSeek, GMICloud, Parasail]

only and ignore are mutually exclusive — pick the shorter list.

Scope and validation

The block applies to OpenRouter-backed models only. Validation rejects it on a model whose provider is anything else, or unset, rather than accepting and ignoring it. Unset fields are omitted from the outbound request, and a block that sets no field at all is a validation error rather than a silent no-op — it would read like a pin while sending no preference. Validation also rejects only and ignore together, a sort outside price / throughput / latency, and an empty list or blank entry in order, only, ignore or quantizations.

Two further checks exist because both failure modes would otherwise look like a working pin:

Unknown keys are rejected, even alongside valid ones. A misspelled alow_fallbacks next to a real order fails at load, naming the key and the valid set, rather than being dropped and leaving a block that reads as pinned and only half is. This strictness applies to provider_routing only — it is not a schema-wide guarantee. The error names the config file but not a line and column, since the check runs after the document is parsed.
max_price is shape-checked. It must be a non-empty mapping of prompt and/or completion to a finite non-negative number, e.g. max_price: {prompt: 0.7, completion: 2.0}. Without the check a scalar would validate cleanly and then be rejected by the aggregator on every single call.

Entries in the list fields are trimmed as they are read, so a padded name binds upstream rather than matching nothing; an entry that is only whitespace is still rejected as blank.

The preference applies to every call llmsoup makes for that model — streaming and non-streaming, routed completions and the preference classifier’s own requests.

Upstream selection is server-controlled: any provider object a client puts in the request body is stripped before the outbound call, on every model, pinned or not. Otherwise a caller could steer an unpinned model to a pricier upstream while still being billed at the static pricing: rate that per-user cost quotas are computed from.

Verifying a pin

A pin is a preference, not a guarantee, and allow_fallbacks decides how hard it binds. llmsoup_model_provider_total reports which upstream actually served each call, labelled by model and provider, on both streamed and buffered calls:

sum by (provider) (rate(llmsoup_model_provider_total{model="deepseek/deepseek-v4-pro-0813"}[1h]))

Traffic still landing on a provider you excluded means the pin is not doing what the config says. A series that stays empty means the upstream reported no provider — the counter records nothing rather than inventing an unknown bucket.

An empty series has one other cause worth ruling out first: llmsoup only scans for a serving provider when the model’s configured provider: is an aggregator (openrouter) or is left unset. A model that declares provider: openai while actually pointing at OpenRouter is never scanned, so it reports nothing however the pin behaves — and it would have failed validation for declaring provider_routing in the first place.

When to use multiple endpoints

Geographic redundancy — Declare the nearest region first, fail over to others.
Provider diversity — Fail over between OpenAI and Azure OpenAI for the same model.
Rate limit management — Fail over across multiple API keys or accounts.

Reasoning configuration

Reasoning configuration controls how llmsoup tells upstream models to use chain-of-thought reasoning. This is configured per-model with reasoning_family and overridden per-rule with model_refs.

reasoning_family

The reasoning_family field on a model definition tells llmsoup how to pass reasoning parameters to that model’s API. It accepts two forms:

String shorthand — for providers that use a standard parameter name:

models:
  - name: gpt-5.2
    reasoning_family: reasoning_effort

This tells llmsoup the model supports a reasoning_effort parameter directly in the API request.

Object form — for providers with custom parameter names:

models:
  - name: deepseek-r1
    reasoning_family:
      type: chat_template_kwargs
      parameter: thinking_mode

This tells llmsoup to pass reasoning controls via the thinking_mode parameter in chat_template_kwargs.

Model reference overrides

Within a routing rule, model_refs can override reasoning behavior per-model:

rules:
  - name: deep-analysis
    priority: 100
    conditions:
      - signal: keyword.complex_query
    action:
      strategy: default
      primary_model: gpt-5.2
      model_refs:
        - model: gpt-5.2
          use_reasoning: true
          reasoning_effort: high
        - model: gpt-5-mini
          use_reasoning: false

Field	Type	Description
`model`	string	Model name (must reference a configured model).
`use_reasoning`	boolean	Enable or disable reasoning for this rule.
`reasoning_effort`	string	Effort level, validated against the model’s `supported_efforts` when declared, or against `low`, `medium`, `high` when it is not.

When use_reasoning is true, llmsoup mutates the outbound request to include the reasoning parameter appropriate for the model’s reasoning_family. When false, reasoning parameters are stripped even if the model supports them.

Pinning effort on a mandatory-reasoning model

Some models cannot be told not to reason, only to reason less. On those, reasoning_effort is the only lever on generation cost — so a pin that silently fails to apply is expensive. A pin needs three coordinated pieces, and any one missing is a no-op that upstream reports as success.

1. Declare the model’s ladder. Providers disagree on what the rungs are. Declaring them turns a wrong effort into a load-time error instead of a silent fallback to the provider’s default, which on a mandatory-reasoning model is typically the most expensive rung.

models:
  - name: z-ai/glm-5.2
    reasoning_family: reasoning_effort
    supported_efforts: [xhigh, high]

With that ladder, reasoning_effort: low fails llmsoup validate with a file:line:column error naming the value, the model, and the accepted efforts.

2. Pin the effort — with use_reasoning. The effort alone does nothing: omitting use_reasoning leaves the outbound body untouched.

rules:
  - name: deep-analysis
    priority: 100
    conditions:
      - signal: keyword.complex_query
    action:
      strategy: default
      primary_model: z-ai/glm-5.2
      model_refs:
        - model: z-ai/glm-5.2
          use_reasoning: true      # without this the effort below is inert
          reasoning_effort: high

A second gap worth knowing: defaults.default_model and defaults.default_fallback_models carry no model_refs, so no reasoning mutation is reachable on a default decision at all. An effort pinned in a rule does not apply to requests that fall through to the default model.

3. Confirm the pin applied. Configuration is not evidence — read the effort that actually reached the wire from llmsoup_delivered_reasoning_effort_total:

sum by (effort) (rate(llmsoup_delivered_reasoning_effort_total{model="z-ai/glm-5.2"}[1h]))

An effort="high" series means the pin is on the wire. An effort="none" series means the body carried no effort at all — one of the three pieces is missing, or the request came through a default decision. The configured-effort metric, llmsoup_reasoning_effort_usage_total, cannot tell these apart: it restates the config.

Pair it with llmsoup_reasoning_tokens_total to see what the reasoning actually cost:

sum by (model) (rate(llmsoup_reasoning_tokens_total[1h]))

Generation bounds

Nothing about an upstream response is self-limiting. A provider that returns HTTP 200 and then emits one byte every 89 seconds holds a request open indefinitely; one that decides to think for 400,000 tokens is billed in full. Two defaults settings bound each.

Total duration

defaults.max_generation_duration_ms is a wall-clock bound on receiving a whole upstream response. It applies to the streamed and the buffered path alike, including the preference classifier’s calls.

defaults:
  max_generation_duration_ms: 600000   # 0 disables

It is not the same as an endpoint’s timeout_ms, and does not replace it:

Bound	Question it answers	Restarts?
`endpoints[].timeout_ms`	”Has the provider stalled?” — time to first chunk, and each gap between chunks.	Yes, on every chunk.
`defaults.max_generation_duration_ms`	”Has the provider had long enough?” — the lifetime of the call.	No. Fixed at request dispatch.

Because the idle bound restarts on every chunk, a stream producing output more often than that bound never trips it, however long it runs. The duration bound is what ends it.

The clock starts at request dispatch, not at the first byte — a deadline starting at first byte would let a provider stall for the full idle bound, emit one byte, and then collect the whole duration budget on top. Time spent queued upstream therefore counts against the budget: from llmsoup’s side a queued request and a slow one are the same exposure.

When the bound trips:

Before any output — the client gets the standard OpenAI error envelope naming the bound.
Mid-stream — the SSE stream is terminated cleanly (the client still sees data: [DONE]) and usage is finalised from the content already forwarded, counted in llmsoup_usage_estimated_total.

A tripped bound is terminal for that attempt. Whether the request then survives is governed by the existing fallback chain, unchanged. Terminating the connection does not stop a provider that has already generated — the bound limits llmsoup’s exposure and the client’s wait, not the upstream meter.

Ship it generous and read the trip rate before tightening it.

Output ceiling

defaults.max_output_tokens injects an output ceiling into a request that carries neither max_completion_tokens nor max_tokens, under whichever key prefer_max_completion_tokens selects.

defaults:
  prefer_max_completion_tokens: true
  max_output_tokens: 8192             # omit or 0 to disable injection

A client-supplied budget is never raised and never lowered. A caller asking for 32K keeps 32K; a caller asking for 512 keeps 512. The ceiling bounds only the unspecified case — which is what agent loops send, and where the unbounded tail comes from. Silently truncating an explicit instruction would be an OpenAI-compatibility break: a caller that asked for 32K tokens and received 8K has no way to tell that apart from the model stopping early. Operators who want a hard clamp have provider_routing.max_price and per-user quotas.

The injected value counts toward the context-window fit check like a client-supplied one, so enabling injection can move requests onto a larger-window ref that previously fit against the internal default. Size the ceiling from the observed completion-token distribution rather than a guess, and watch the reroute rate on first deploy.

Plugins overview

Plugins add pre-processing and post-processing to routing rules. They are configured in the plugins array on each rule and execute in order.

rules:
  - name: secure-route
    priority: 100
    conditions:
      - signal: keyword.sensitive
    action:
      strategy: default
      primary_model: gpt-5.2
    plugins:
      - type: jailbreak
        configuration:
          enabled: true
          threshold: 0.8
      - type: system_prompt
        configuration:
          system_prompt: "Answer carefully and factually."

Every plugin has a type field and a configuration block. The available plugin types are:

Plugin type	Purpose
`system_prompt`	Inject or replace system prompts
`semantic-cache`	Cache responses by semantic similarity
`jailbreak`	Detect prompt injection attempts
`pii`	Detect personally identifiable information
`header_mutation`	Modify HTTP headers on requests/responses
`hallucination`	Flag potential hallucinations in responses
`router_replay`	Record routing decisions for debugging

For complete per-plugin configuration details, field descriptions, and examples, see the Plugins Reference.

Cache tuning

llmsoup uses three independent caching layers, each with different eviction strategies and tuning knobs.

Model response cache (TTL)

Caches full model responses keyed by request content. When a cache hit occurs, the response is returned immediately without calling the upstream model.

defaults:
  model_cache_ttl_seconds: 3600
  model_cache_max_capacity: 1000

Setting	Default	Description
`model_cache_ttl_seconds`	3600	Time-to-live in seconds. Entries expire after this duration regardless of access.
`model_cache_max_capacity`	1000	Maximum number of cached model response entries. Oldest entries are evicted when capacity is reached (LRU).

Eviction behavior: Entries are evicted after the TTL expires or when the cache reaches model_cache_max_capacity entries (whichever comes first). The capacity limit uses LRU eviction — the least recently used entry is removed to make room for new ones. This prevents unbounded memory growth under heavy traffic.

When to tune:

Lower TTL (e.g., 60–300) for rapidly changing data or when freshness matters.
Higher TTL (e.g., 7200+) for stable queries where the same prompt always has the same answer.
Set to 0 to effectively disable model response caching.

Embedding cache (LRU)

Caches computed embedding vectors to avoid re-running BERT inference for repeated text. This primarily benefits the embedding signal evaluator.

defaults:
  embedding_cache_capacity: 1000

Setting	Default	Description
`embedding_cache_capacity`	1000	Maximum number of cached embedding entries. Oldest entries are evicted when capacity is reached.

Eviction behavior: Least Recently Used (LRU). When the cache is full, the entry that hasn’t been accessed for the longest time is evicted to make room.

When to tune:

Increase if your workload has many unique prompts and you see high embedding computation times in metrics.
Decrease if memory is constrained — each embedding entry holds a vector of floats.

TPOT cache (EMA)

Tracks per-model Time Per Output Token (TPOT) using Exponential Moving Average smoothing. This is used internally by the latency signal evaluator — it is not directly configurable via YAML.

How it works: After each model response, llmsoup computes the actual TPOT and updates the smoothed average:

smoothed_tpot = alpha × new_tpot + (1 - alpha) × previous_smoothed_tpot

The default EMA alpha is 0.3, which means:

30% weight to the most recent observation
70% weight to historical average

This smoothing prevents a single slow response from drastically changing the model’s latency estimate. The latency signal evaluator compares the smoothed TPOT against the max_tpot threshold configured on each latency signal.

When multiple embedding signals reference the same model (e.g., sentence-transformers/all-MiniLM-L6-v2), llmsoup automatically shares a single loaded model instance across all signals. This is transparent — no configuration is needed.

Impact: Each BERT embedding model uses ~130MB of RAM. Without sharing, six signals using the same model would consume ~780MB. With sharing, they use ~130MB total.

How it works: During startup, llmsoup groups embedding signals by their resolved model path. Signals that share a path receive a shared handle to a single model instance. Each signal still maintains its own reference text, candidates, and cache — only the underlying BERT model weights are shared.

Memory tuning

llmsoup loads its BERT models lazily on the first request, so idle memory stays low. Under load, resident memory is driven by two things: the resident model weights and the per-request activation/scratch buffers of concurrent forward passes. The knobs below keep both bounded so the process stays within the memory budget (idle ≤ 500 MB, loaded ≤ 1 GB).

Bounded inference concurrency

Every embedding and classifier forward pass runs on a blocking thread. Without a limit, N concurrent requests run N concurrent forwards, and peak memory scales with request concurrency. defaults.max_inference_concurrency caps how many forward passes execute at once, so peak memory scales with a small fixed limit instead.

defaults:
  max_inference_concurrency: 4

Default: min(number_of_cores, 4), floored at 2. On Linux the core count is cgroup-aware, so a container with a CPU quota gets a sensible value automatically.
Override at runtime: set the LLMSOUP_MAX_INFERENCE_CONCURRENCY environment variable (used when the config value is omitted). An explicit config value always wins.
Validation: the value must be greater than 0 and no larger than 1024; out-of-range values are rejected at startup with a file:line:column error (the ceiling prevents an oversized value from aborting the process when the semaphore is built).
Backpressure, not bypass: requests wait for a permit before running a forward pass — the cap is never silently exceeded under load. Because bounding peak memory is the goal, the queue applies as hard backpressure rather than flooding the blocking pool, so a burst trades latency (not memory) for the guarantee.
Backward compatibility: setting the value at or above your maximum expected in-flight requests restores the prior effectively-unbounded behavior.

Because routing inference is CPU-bound and runs batch-1 with short prompts, concurrency beyond a few gives essentially no throughput gain but grows memory linearly — so a small limit is the right default. The only realistic cost of a low limit is worst-case tail latency under a hard burst (roughly ceil(burst / limit) × per-forward-ms), not steady-state throughput.

Inference thread cap (environment)

LLMSOUP_INFERENCE_THREADS caps the number of intra-op (gemm/rayon) threads used per forward pass. It is applied at startup before the first matmul by seeding RAYON_NUM_THREADS (an explicitly set RAYON_NUM_THREADS takes precedence). Lowering it reduces duplicated gemm scratch on many-core hosts without changing routing decisions.

LLMSOUP_INFERENCE_THREADS=2 llmsoup serve

glibc arena capping (environment)

On the glibc/Linux target, MALLOC_ARENA_MAX=2 limits the number of per-thread malloc arenas, which curbs arena retention from freed inference scratch. The production container image sets this by default. It has no effect on musl/Alpine, which is why the shipped image is glibc-based (Debian slim).

Tuned global allocator (build option)

llmsoup can be built with a tuned global allocator via the jemalloc cargo feature:

cargo build --release --features jemalloc

This installs jemalloc as the global allocator on the Linux target (with background_thread:true,dirty_decay_ms:1000 to purge dirty pages promptly) and leaves macOS on the system allocator. On the Linux/glibc target this is the single most effective memory lever: in the container benchmark at 50-concurrency it cut loaded RSS from ~2.8 GB (glibc, MALLOC_ARENA_MAX=2) to ~1.2 GB, because glibc retains freed inference scratch in its arenas while jemalloc returns dirty pages to the OS. When jemalloc is active it supersedes MALLOC_ARENA_MAX.

Because the saving is decisive on the production target, the shipped Docker image is built with jemalloc by default, and the same feature is applied to the downloadable Linux release binaries (*-unknown-linux-gnu). Plain cargo build (e.g. local dev on macOS) stays on the system allocator; rebuild the image without it via --build-arg CARGO_FEATURES="" if you need the glibc allocator.

Docker deployment

The shipped Dockerfile is a multi-stage, glibc (Debian slim) build that carries only the release binary, so memory behavior can be measured on and deployed to the production target rather than only macOS. It applies the memory-tuned default MALLOC_ARENA_MAX=2 and leaves inference concurrency at the cgroup-aware auto-default (so a small CPU quota yields a small limit automatically — set LLMSOUP_MAX_INFERENCE_CONCURRENCY to override), runs as a non-root user, exposes port 8080, and loads its configuration from the path in LLMSOUP_CONFIG (mount your own; no secrets are baked into the image).

# Build (context is the repo root; the crate lives in llmsoup/)
docker build -t llmsoup .

# Run with a mounted config and a persistent HuggingFace model cache
docker run --rm -p 8080:8080 \
  -v "$PWD/config.yaml:/etc/llmsoup/config.yaml:ro" \
  -v llmsoup-models:/models \
  llmsoup

The same image doubles as the reproducible memory-measurement harness — run llmsoup benchmark --use-models inside the container to record idle and loaded RSS on the Linux/glibc target. (macOS numbers come from the Metal backend and are indicative only.)

Secret resolution

llmsoup resolves secrets (API keys, authentication tokens) from external sources at startup. Secrets are never stored in the configuration file itself. The resolution methods are tried in the order they appear in the configuration.

Environment variable

The simplest and most common method. Reads the secret from an environment variable.

access_key:
  env: OPENAI_API_KEY

Behavior: Looks up the environment variable at config load time. Fails validation if the variable is not set or is empty.

Error: Secret resolution failed: environment variable 'OPENAI_API_KEY' not set

File

Reads the secret from a file path. Ideal for Docker secrets and Kubernetes secret volumes.

access_key:
  file: /run/secrets/openai_api_key

Behavior: Reads the entire file content and trims leading/trailing whitespace. Fails if the file does not exist or is not readable.

Error: Secret resolution failed: cannot read file '/run/secrets/openai_api_key'

Common patterns:

# Docker secret
access_key:
  file: /run/secrets/api_key

# Kubernetes secret volume
access_key:
  file: /etc/llmsoup/secrets/api-key

Command

Executes a shell command and captures its stdout as the secret value. Useful for integration with secret managers like AWS Secrets Manager or 1Password CLI.

access_key:
  command: "aws secretsmanager get-secret-value --secret-id openai-key --query SecretString --output text"

Behavior: Runs the command via the system shell, captures stdout, and trims whitespace. The command must exit with code 0.

Command-based secret resolution is disabled by default for security. You must explicitly enable it by setting the environment variable:

export LLMSOUP_ALLOW_COMMAND_SECRETS=1

Without this variable, any command secret reference will fail validation with an error explaining the security gate.

Error (when disabled): Secret resolution failed: command-based secrets are disabled. Set LLMSOUP_ALLOW_COMMAND_SECRETS=1 to enable.

Error (when command fails): Secret resolution failed: command exited with non-zero status

Vault (planned)

access_key:
  vault: "secret/data/openai/api-key"

Error: Secret resolution failed: vault secret resolution is not yet implemented

Resolution priority

When multiple methods are specified in the same secret reference, they are resolved in order: env → file → vault → command. The first successful resolution wins.

# Tries env first, falls back to file
access_key:
  env: OPENAI_API_KEY
  file: /run/secrets/openai_api_key

Context overflow strategies

When a request exceeds the model’s effective context window (see Context headroom), llmsoup applies a context overflow strategy to fit the conversation within the limit. The strategy is set globally in defaults.

defaults:
  context_overflow: truncate_middle
  context_headroom_ratio: 0.1

Context headroom

llmsoup estimates request tokens with a ~4 characters/token heuristic, which under-counts code-dense content. To compensate, a fraction of each model’s declared context_window is reserved as headroom: the effective window used for all fit checks and truncation budgets is context_window * (1 - context_headroom_ratio).

The fit check covers the whole request: messages, tools, and the output budget (max_completion_tokens if set, else max_tokens, else 4096).

context_headroom_ratio defaults to 0.1 and must be between 0.0 and 0.5 — out-of-range values fail config validation at load time with a file:line:column error. Set it to 0.0 to disable the headroom and check against the raw window.

truncate_middle (default)

Keeps the leading system and developer messages (the “protected prefix”) and the most recent messages. Drops messages from the middle of the conversation.

Before (6 messages, context exceeded):

[system]   You are a helpful assistant.
[user]     What is Rust?                    ← DROPPED
[assistant] Rust is a systems language...   ← DROPPED
[user]     How about Go?                    ← DROPPED
[assistant] Go is a compiled language...    ← kept
[user]     Compare their async models.      ← kept

After:

[system]   You are a helpful assistant.
[assistant] Go is a compiled language...
[user]     Compare their async models.

Best for: General-purpose use. Preserves the system/developer instructions and the most recent turns, which are usually the most relevant.

rolling_window

Drops the oldest non-system messages first, keeping the most recent conversation.

Before (6 messages, context exceeded):

[system]   You are a helpful assistant.
[user]     What is Rust?                    ← DROPPED
[assistant] Rust is a systems language...   ← DROPPED
[user]     How about Go?                    ← DROPPED
[assistant] Go is a compiled language...    ← kept
[user]     Compare their async models.      ← kept

After:

[system]   You are a helpful assistant.
[assistant] Go is a compiled language...
[user]     Compare their async models.

Best for: Long-running conversations where only the recent context matters. Similar to a sliding window over the chat history.

stop_at_limit

Returns an error immediately without sending the request. No messages are dropped.

Behavior: Returns HTTP 400 in OpenAI format. The message cites the effective window — the declared window minus headroom — so clients that resize their prompt use a limit that is actually enforced:

{
  "error": {
    "message": "Request of ~130000 tokens exceeds model 'gpt-4o' effective context window of 115200 tokens (128000 declared minus headroom). Reduce the message length or use a model with a larger context window.",
    "type": "invalid_request_error",
    "code": "context_length_exceeded"
  }
}

Best for: Applications that need to know when context is exceeded so they can handle it themselves (e.g., summarize the conversation before retrying).

Fit-aware model selection

When a matched decision uses a confidence or ratings algorithm to select among multiple model refs, llmsoup estimates the request’s tokens once (messages + tools + output budget) and skips refs whose effective window cannot hold the estimate — provided at least one ref on the decision fits. Oversized prompts route to a larger-window ref instead of losing messages to truncation.

If no ref fits, the algorithm’s normal ranking stands and the configured overflow strategy truncates the request as a last resort. Skipped refs are logged with the model name, estimated tokens, and effective window. Models without metadata.context_window are assumed to fit. See Algorithms.

The decision’s fallback chain is ordered the same way: refs that fit the request are listed before refs that don’t, so a transient failure falls back to another fitting model instead of one that would truncate.

Truncate-and-retry on upstream errors

Token estimates are heuristic, so an oversized request can still reach a model and be rejected upstream. When an upstream call returns a 400 whose body matches a context-length signature — a message containing “maximum context length” (case-insensitive) or error code context_length_exceeded — llmsoup truncates the request per the configured overflow strategy and retries the same model exactly once.

The retry is skipped — the error surfaces immediately — when a second attempt cannot succeed:

Truncation cannot actually shrink the request, e.g. content the token estimator cannot see, such as image parts.
The output budget (max_completion_tokens/max_tokens) alone exceeds the effective window — dropping messages cannot fix that.
Dropping messages would leave only system/developer messages; llmsoup never silently answers a gutted conversation.
The first attempt already consumed half or more of the per-ref time budget (defaults.request_timeout_ms, default 30000 ms), so a retry would be killed by the executor timeout.

If the retry also fails with a context-length error, execution advances to the decision’s remaining refs (fallback). Other 4xx errors remain terminal for that ref and are not retried. This applies to both buffered and streaming (SSE) upstream calls. Every retry decision is counted in llmsoup_context_length_retries_total.

Advanced model metadata

Model metadata fields influence routing decisions beyond simple name matching.

models:
  - name: gpt-5.2
    metadata:
      context_window: 128000
      parameter_count: 520000
      latency_seconds: 1.2

Field	Type	Used by
`context_window`	integer	Context overflow strategies, fit-aware model selection, prompt validation.
`parameter_count`	integer	Reserved. Intended for the confidence algorithm’s planned `escalation_order: size`, which is not implemented — see below.
`latency_seconds`	float	Initial latency estimates before real TPOT data is collected.

How parameter_count affects routing

What the confidence algorithm actually does: model_refs are evaluated in declaration order. If the confidence score meets the threshold, the first eligible ref is used; otherwise the last ref in the list is selected. Order your refs so that the first entry is the intended primary and the last is the escalation (or relief-valve) target:

action:
  algorithm:
    type: confidence
    confidence:
      threshold: 0.8
  model_refs:
    - model: gpt-5-mini      # intended primary — used when confidence >= 0.8
    - model: gpt-5.2         # escalation target — used when confidence < 0.8

Every ref starts from the same signal-derived confidence score. With cost-aware routing off — defaults.cost_aware_routing unset or false — that score is also what each ref is compared against, so either the first ref clears the threshold or none does, and an intermediate ref in a list of three or more is unreachable by the threshold and acts only as a failover target.

With cost_aware_routing: true the refs no longer share one score. Each is charged its own pricing-derived penalty — the average of its prompt and completion rates per 1M tokens, divided by $100 and scaled by the effective cost_quality_tradeoff, which is 0.3 when cost-aware routing is on and no value is set — and the penalty is subtracted from the shared score before the comparison. An expensive ref can therefore fall below the threshold while a cheaper ref further down the list clears it and is selected, which makes intermediate refs reachable and a three-tier ladder behave as a ladder. A ref whose model has no pricing takes no penalty.

Algorithm error handling

Both the confidence and ratings algorithms support an on_error field that controls behavior when a model call fails during algorithm evaluation.

algorithm:
  type: confidence
  confidence:
    threshold: 0.8
    on_error: skip

Value	Default	Behavior
`skip`	Yes	Skip the failed model and try the next one in the list.
`fail`	—	Return an error immediately without trying other models.

When on_error is skip (the default) and a model returns an error or times out, the algorithm moves to the next model in the evaluation order instead of failing the entire request. When set to fail, the first model error aborts the algorithm and returns the error to the caller.

Confidence method options

The confidence_method field controls how the confidence algorithm calculates confidence from model responses:

Method	Description
`margin`	Uses the difference between the top token probability and the second-highest. Larger margins indicate higher confidence.
`avg_logprob`	Uses the average log probability across all output tokens. Higher average means more confident.
`hybrid`	Combines both methods using configurable weights.

algorithm:
  type: confidence
  confidence:
    threshold: 0.75
    confidence_method: hybrid
    hybrid_weights:
      logprob_weight: 0.6
      margin_weight: 0.4

Cost-quality tradeoff in algorithms

Both algorithm types support a per-rule cost_quality_tradeoff that overrides the global defaults.cost_quality_tradeoff:

rules:
  - name: budget-route
    action:
      algorithm:
        type: confidence
        confidence:
          threshold: 0.7
          cost_quality_tradeoff: 0.8  # strongly prefer cheaper models
  - name: quality-route
    action:
      algorithm:
        type: ratings
        ratings:
          policy: highest
          cost_quality_tradeoff: 0.1  # strongly prefer quality

The value ranges from 0.0 (pure quality) to 1.0 (pure cost savings). This requires cost_aware_routing: true in defaults and pricing configured on models.