Configuration Reference

llmsoup uses a single YAML configuration file to define models, signals, routing rules, plugins, and authentication. This reference covers every configuration option available.

Overview

llmsoup loads configuration from a YAML file at startup. The file location is resolved in this order:

1. --config CLI flag (e.g., llmsoup serve --config my-config.yaml)
2. LLMSOUP_CONFIG environment variable
3. config.yaml in the current directory (default)

Generate a starter config with all options commented:

llmsoup prepare

Note

Configuration is loaded once at startup. Changes require a server restart — there is no hot reload.

Top-level structure

A config file has the following top-level keys:

version: v0.1            # Optional version identifier
defaults:                 # Global routing defaults
models:                   # Model definitions (min 2)
signals:                  # Signal evaluators
rules:                    # Routing rules (priority-ordered)
classifier:               # Domain classifier model (optional)
auth:                     # Authentication (optional)

The version field is an optional string carried forward from the config file — it has no effect on behavior but can be used to track config revisions.

Semantic-router compatibility

llmsoup is backward-compatible with the semantic-router configuration format. The parser auto-normalizes legacy field names:

Legacy field	Normalized to
providers.default_model	defaults.default_model
providers.models	models
decisions	rules
modelRefs	action.model_refs + action.primary_model / fallback_models
Condition type field	{type}.{name} signal reference format

Both formats are accepted. This reference documents the normalized format; the Getting Started guide uses the legacy decisions / modelRefs format for compatibility.

Defaults

The defaults block sets global behavior for the routing engine.

defaults:
  default_model: gpt-5-mini
  preference_model: gpt-5-mini
  request_timeout_ms: 60000
  model_cache_ttl_seconds: 300
  embedding_cache_capacity: 1000
  prefer_max_completion_tokens: false
  preference_confidence: false
  preference_structured_outputs: false
  preference_classification_budget_ms: 20000
  cost_aware_routing: false
  cost_quality_tradeoff: 0.3
  include_cost_headers: true
  stream_passthrough: true
  context_overflow: truncate_middle
  context_headroom_ratio: 0.1
  default_fallback_models: []
  cost_baseline_model: gpt-5.2
  model_cache_max_capacity: 1000
  semantic_cache_max_entries: 2000
  max_inference_concurrency: 4
  max_generation_duration_ms: 600000
  max_output_tokens: 0

Field	Type	Default	Description
default_model	string	—	Model used when no routing rule matches. Must reference a configured model.
preference_model	string	—	Model used for preference signal classification.
request_timeout_ms	integer	—	Global request timeout in milliseconds.
model_cache_ttl_seconds	integer	—	TTL for model response cache entries.
embedding_cache_capacity	integer	—	Maximum entries in the embedding LRU cache.
prefer_max_completion_tokens	boolean	—	Send max_completion_tokens instead of max_tokens to upstream models.
preference_confidence	boolean	false	Ask the preference classifier for its top-2 routes with scores and use the score margin (best − second) as the preference signal confidence. When off, the classifier reports a constant confidence of 1.0, so confidence algorithms always select the first model ref. In either mode an answer the router had to rewrite to other — off-list or matching several routes — reports 0.0 instead. See Signals & Routing — confidence.
preference_structured_outputs	boolean	false	Constrain the preference classifier’s response with JSON-schema structured outputs (response_format), locking the route (and, in margin mode, second_route) to the configured route names. On models that support structured outputs this eliminates malformed-JSON parse failures and off-list route hallucinations. Models or providers that reject response_format transparently fall back to prompt-only parsing, and a transient empty choices response is retried once before the signal degrades. See Signals & Routing — preference.
preference_classification_budget_ms	integer	derived	Wall-clock budget, in milliseconds, covering one whole preference classification — the first attempt plus every degradation retry. Must be between 100 and 300000; an out-of-range value fails validation at load time. When omitted the budget is derived as twice the classifier model’s longest endpoint timeout_ms, so an existing config is unchanged. Lowering an endpoint timeout_ms is not an equivalent bound: it also shortens each individual classifier call. See Signals & Routing — preference.
cost_aware_routing	boolean	false	Enable cost-aware routing algorithm.
cost_quality_tradeoff	float	0.3	Balance between quality (0.0) and cost savings (1.0).
include_cost_headers	boolean	true	Include cost data in response headers (X-LLMSoup-Cost-*).
stream_passthrough	boolean	true	Forward SSE chunks from the upstream provider to the client as they arrive for stream: true requests. Applies to default/fallback strategy decisions without response-body plugins; parallel decisions and those with semantic-cache, hallucination, or router_replay plugins always use the buffered path. Set false to buffer the full completion and replay it as SSE events.
context_overflow	string	truncate_middle	Strategy when prompt exceeds context window: rolling_window, truncate_middle, or stop_at_limit.
context_headroom_ratio	float	0.1	Fraction of each model’s context_window reserved as headroom for token-estimation error. Fit checks and truncation budgets use context_window * (1 - context_headroom_ratio). Must be between 0.0 and 0.5 — out-of-range values fail validation at load time. Set 0.0 to disable. See Context overflow strategies.
default_fallback_models	array	[]	Fallback model chain when the default model fails.
cost_baseline_model	string	—	Model used as baseline for cost savings calculation. Uses the most expensive model if omitted.
model_cache_max_capacity	integer	1000	Maximum entries in the model response cache. Oldest entries are evicted when capacity is reached.
semantic_cache_max_entries	integer	2000	Maximum entries in the semantic cache store. Oldest entries are evicted when capacity is reached.
max_generation_duration_ms	integer	600000	Wall-clock bound, in milliseconds, on receiving a whole upstream response — streamed and buffered alike, including preference-classifier calls. Independent of each endpoint’s timeout_ms, which bounds only the gap between chunks: a provider that emits a byte more often than that gap streams indefinitely without this bound. Measured from request dispatch, so time spent queued upstream counts. Set 0 to disable. A negative value fails validation at load time. See Generation bounds.
max_output_tokens	integer	—	Output ceiling injected into an outbound request that carries neither max_completion_tokens nor max_tokens, under the key prefer_max_completion_tokens selects. A client-supplied budget is forwarded unchanged — never raised and never lowered — so this bounds only the unspecified case, which is what agent loops send. The injected value also counts toward the context-window fit check. Omit or set 0 to disable injection. A negative value fails validation at load time. See Generation bounds.
max_inference_concurrency	integer	min(cores, 4), floored at 2	Maximum model forward passes (embedding + classifier inference) allowed to run concurrently. Bounds peak resident memory under load. Must be greater than 0. Overridable with the LLMSOUP_MAX_INFERENCE_CONCURRENCY environment variable. See Memory tuning.

To restore the legacy buffered streaming behavior (full completion fetched, then replayed as SSE events):

defaults:
  stream_passthrough: false

Models

The models array defines all available LLM endpoints. A minimum of 2 models is required.

models:
  - name: gpt-5-mini
    provider: openai
    access_key:
      env: OPENAI_API_KEY
    endpoints:
      - url: https://api.openai.com/v1/chat/completions
        weight: 1
    metadata:
      context_window: 8000
      parameter_count: 120000
      latency_seconds: 0.5
    pricing:
      prompt_per_1m: 0.25
      completion_per_1m: 2.00
      currency: USD

  - name: gpt-5.2
    provider: openai
    access_key:
      env: OPENAI_API_KEY
    endpoints:
      - url: https://api.openai.com/v1/chat/completions
    metadata:
      context_window: 128000
      parameter_count: 520000
      latency_seconds: 1.2
    pricing:
      prompt_per_1m: 1.75
      completion_per_1m: 14.00
    reasoning_family: reasoning_effort

Model fields

Field	Required	Type	Description
name	yes	string	Unique model identifier used in routing rules.
provider	no	string	Provider name (e.g., openai, anthropic).
access_key	no	string or object	API key — see Secret Resolution.
endpoints	yes	array	At least one endpoint.
metadata	no	object	Model metadata for routing decisions.
pricing	no	object	Token pricing for cost tracking.
reasoning_family	no	string or object	Reasoning control configuration.
supported_efforts	no	array	Reasoning efforts this model actually serves — see Supported efforts.
provider_routing	no	object	Upstream provider preference. OpenRouter-backed models only — see Provider routing.

Provider routing

On an aggregator such as OpenRouter a single model id fans out to many upstream endpoints at different prices, quantisations and capabilities. Without a stated preference the aggregator chooses, which leaves the effective price, the cache behaviour and even tool support undetermined. provider_routing states that preference; the fields map 1:1 onto OpenRouter’s provider request object.

models:
  - name: deepseek/deepseek-v4-pro-0813
    provider: openrouter
    provider_routing:
      order: [DeepSeek, GMICloud]
      allow_fallbacks: false
      require_parameters: true

Field	Type	Description
order	array of strings	Ordered provider preference, most preferred first.
only	array of strings	Allow list — route only to these providers. Mutually exclusive with ignore.
ignore	array of strings	Deny list — never route to these providers. Mutually exclusive with only.
allow_fallbacks	boolean	Whether the aggregator may route outside order/only.
require_parameters	boolean	Drop endpoints that do not support the request’s parameters.
quantizations	array of strings	Permitted quantisation levels (e.g. fp8, bf16).
sort	string	Endpoint sort preference: price, throughput or latency.
max_price	object	Per-token price ceiling: a mapping of price dimension to a non-negative number. The accepted dimensions are prompt and completion, e.g. {prompt: 0.7, completion: 2.0}. Validated for shape, then passed through verbatim.

Every field is optional and unset fields are omitted from the outbound request, so the upstream sees only what you configured. A block that sets no field at all is rejected at load time rather than accepted as a no-op, since it would read like a pin while sending no preference.

Unknown keys are rejected. A provider_routing block containing a key that is not in the table above fails to load, naming the offending key and listing the valid ones — including when it sits alongside perfectly good keys. The point of the block is that a preference which never reaches the aggregator must not read like a working pin, and a misspelled key next to a valid one is exactly that. This strictness is specific to provider_routing; the rest of the schema does not currently reject unrecognized keys, so do not read it as a config-wide guarantee.

Validation also rejects:

• only and ignore set together — use one or the other.
• a sort value other than price, throughput or latency.
• an empty order, only, ignore or quantizations list, or a blank entry in one — omit the field instead of declaring it empty.
• a max_price that is not a mapping, is an empty mapping, names a dimension other than prompt or completion, or gives a dimension anything but a finite non-negative number.

Entries in order, only, ignore and quantizations are trimmed as they are read, so a padded name (order: [" DeepSeek"] — invisible in review) still binds upstream instead of quietly matching nothing. An entry that is only whitespace trims to empty and is rejected as a blank entry.

OpenRouter-backed models only

provider_routing is only meaningful for a backend that accepts a provider object on the request. Validation rejects the block on a model whose provider is anything other than openrouter, including a model with no provider set — a silently ignored block would look like a working pin while traffic kept landing wherever the upstream chose.

The preference applies to every call llmsoup makes for that model, streaming and non-streaming, including the preference classifier’s own requests. Which upstream actually served a call is reported by llmsoup_model_provider_total.

Upstream selection is server-controlled

Any provider object a client puts in the request body is stripped before the outbound call, on every model, whether or not it declares provider_routing. Otherwise a caller could steer an unpinned model to a pricier upstream while still being billed at the static pricing: rate that per-user cost quotas are computed from.

Endpoints

Each endpoint requires a url. Optional fields control load balancing and timeouts.

Field	Required	Type	Description
url	yes	string	Full URL to the chat completions endpoint.
weight	no	integer (>0)	Accepted and validated, but ignored by routing — endpoints are tried in declaration order. Reserved for planned weighted distribution.
timeout_ms	no	integer	Per-endpoint timeout override.
description	no	string	Human-readable description.

Metadata

Field	Type	Description
context_window	integer (>0)	Maximum token context window.
parameter_count	integer (>0)	Model parameter count (used for size-based escalation).
latency_seconds	float (>=0)	Expected response latency.

Pricing

Field	Type	Description
prompt_per_1m	float	Cost per 1M prompt tokens.
completion_per_1m	float	Cost per 1M completion tokens.
cached_prompt_per_1m	float	Cost per 1M cached prompt tokens.
cached_completion_per_1m	float	Cost per 1M cached completion tokens.
currency	string	Currency code (default: USD).

Reasoning family

Controls how reasoning parameters are passed to upstream models.

# String shorthand
reasoning_family: reasoning_effort

# Object form
reasoning_family:
  type: chat_template_kwargs
  parameter: thinking_mode

Supported efforts

supported_efforts declares the reasoning efforts a model actually serves. Any reasoning_effort on a modelRef targeting that model is validated against the list at config-load time, and a value outside it fails to load with a file:line:column error naming the value, the model, and the accepted efforts.

models:
  - name: z-ai/glm-5.2
    reasoning_family: reasoning_effort
    supported_efforts: [xhigh, high]

This matters because an unsupported effort is not an upstream error: the provider silently falls back to its own default, which on a model that cannot be told to stop reasoning is typically the most expensive rung. Without the declared ladder there is nothing to catch the mistake before it bills.

The check covers the effort you don’t write too. A ref that sets use_reasoning: true without a reasoning_effort still sends one — the default medium — so against the ladder above it fails to load rather than shipping an effort the model does not serve. Declaring a ladder that contains medium, or declaring none at all, leaves that case accepted exactly as before.

Providers disagree on the ladder — max, xhigh and none are all real values on live models. Declare what the model documents, not a house standard.

Behaviour	Detail
Omitted	Falls back to low, medium, high — the set validated before this field existed, so existing configs load unchanged.
Declared	Only the listed values are accepted for that model. supported_efforts: [max, high, low] rejects medium.
Empty list	Fails validation. An empty list would reject every effort while reading as a permissive declaration.
Blank entry	Fails validation — an entry that can never match an effort.
Padded entry	Trimmed on ingest, so [" high"] behaves as [high] rather than rejecting the effort it declares.

The list is declared rather than fetched: llmsoup validate is the pre-deploy gate and runs in CI, so it must stay offline and reproducible. The cost is that a declared ladder can go stale relative to the provider — the same maintenance property pricing and metadata.context_window already have. A stale ladder fails at load time with a named error, not at runtime.

Signals

Signals evaluate incoming requests and produce scores used by routing rules. Configure them under the signals block, grouped by type.

Keyword

Match requests based on keyword presence in the prompt.

signals:
  keyword:
    - name: code_keywords
      operator: OR
      keywords: ["code", "function", "implement", "debug"]
      case_sensitive: false

Field	Required	Default	Description
name	yes	—	Unique signal identifier.
keywords	yes	—	Array of trigger phrases (minimum 1).
operator	no	OR	Match logic: AND (all must match), OR (any), NOR (none).
case_sensitive	no	false	Enable case-sensitive matching.

Embedding

Semantic similarity matching using BERT embeddings.

signals:
  embedding:
    - name: quick_answer
      model: sentence-transformers/all-MiniLM-L6-v2
      threshold: 0.90
      candidates:
        - "quick question"
        - "simple answer"
        - "brief response"
      aggregation_method: max

The shipped default configuration uses sentence-transformers/all-MiniLM-L6-v2: it is ~40 MB lighter and roughly 2× faster than all-MiniLM-L12-v2, with a negligible routing-accuracy difference on cosine-threshold matching. Operators who prefer the larger model can pin all-MiniLM-L12-v2 explicitly.

Field	Required	Default	Description
name	yes	—	Unique signal identifier.
model	yes	—	Embedding model ID (e.g., sentence-transformers/all-MiniLM-L6-v2).
threshold	yes	—	Similarity threshold (0.0–1.0).
reference_text	no	—	Single reference text for comparison.
candidates	no	—	Array of candidate texts for comparison.
aggregation_method	no	—	How to combine candidate scores: max, avg, or any. Requires candidates.

Tip

Embedding models are downloaded from Hugging Face on first use. Set HUGGINGFACE_HUB_TOKEN for access. See the Getting Started guide for setup.

Domain

Classify requests into subject domains using MMLU categories or embedding-based matching.

signals:
  domain:
    - name: math
      mmlu_categories: [math]
      threshold: 0.6

    - name: programming
      examples:
        - "Write a Python function"
        - "Debug this JavaScript code"
      description: "Programming and software development"
      threshold: 0.7

Field	Required	Default	Description
name	yes	—	Unique signal identifier.
examples	no	—	Example phrases for embedding-based matching.
description	no	—	Human-readable domain description.
mmlu_categories	no	—	MMLU category aliases (e.g., math, physics, computer_science).
threshold	no	—	Confidence threshold (0.0–1.0).

Note

At least one of examples, description, or mmlu_categories is required.

Language

Detect the language of incoming requests.

signals:
  language:
    - name: english
      language: en

    - name: chinese
      language: zh

Field	Required	Description
name	yes	Unique signal identifier.
language	yes	ISO 639-1 code (en, es, fr, de, zh, ja, ru, etc.).

Latency

Route based on time-per-output-token (TPOT) requirements.

signals:
  latency:
    - name: low_latency
      max_tpot: 0.050

    - name: medium_latency
      max_tpot: 0.150

Field	Required	Description
name	yes	Unique signal identifier.
max_tpot	yes	Maximum time per output token in seconds (must be >0).
description	no	Human-readable description.

Fact check

Flag requests that may require factual verification.

signals:
  fact_check:
    - name: needs_fact_check
      description: "Requests requiring factual accuracy"

Field	Required	Description
name	yes	Unique signal identifier.
description	no	Human-readable description.

User feedback

Classify user feedback signals for adaptive routing.

signals:
  user_feedback:
    - name: clarification_needed
      description: "User needs clarification"

    - name: satisfied
      description: "User is satisfied with the response"

Field	Required	Description
name	yes	Unique signal identifier.
description	no	Human-readable description.

Preference

Route based on user preference classification (uses an external LLM for classification).

signals:
  preference:
    - name: code_generation
      description: "User wants code generation"

    - name: bug_fixing
      description: "User wants help fixing bugs"

Field	Required	Description
name	yes	Unique signal identifier.
description	no	Human-readable description.

Note

Preference signals use the model configured in defaults.preference_model for classification.

Classifier

The optional classifier block configures the MMLU-based domain classification model used by domain signals.

classifier:
  category_model:
    model_id: "models/mom-domain-classifier"
    threshold: 0.6
    category_mapping_path: "models/mom-domain-classifier/category_mapping.json"

Field	Default	Description
category_model.model_id	—	HuggingFace repo ID or local path to the classification model.
category_model.threshold	0.6	Confidence threshold (0.0–1.0) for classification.
category_model.category_mapping_path	—	Path to the category mapping JSON file.

Tip

If classifier is omitted, llmsoup uses built-in defaults for domain classification. Only configure this if you need a custom model or category mapping.

Routing Rules

Routing rules (also called “decisions” in semantic-router format) evaluate signal results and route requests to models. Rules are evaluated in priority order (highest priority wins).

rules:
  - name: code-routing
    priority: 100
    operator: AND
    conditions:
      - signal: keyword.code_keywords
    action:
      strategy: default
      primary_model: gpt-5.2
      fallback_models: [gpt-5-mini]

Rule fields

Field	Required	Default	Description
name	yes	—	Unique rule identifier.
priority	yes	—	Integer >0. Higher priority rules are evaluated first.
conditions	yes	—	Array of conditions (minimum 1).
operator	no	AND	How conditions combine: AND (all must match), OR (any), NOR (none).
action	yes	—	What happens when the rule matches.
plugins	no	—	Plugins to apply for this rule (see Plugins).

Conditions

Each condition references a signal and optionally applies a comparison.

conditions:
  - signal: keyword.code_keywords
  - signal: domain.math
    operator: greater-than
    value: 0.8
  - signal: language.english
    negate: true

Field	Required	Default	Description
signal	yes	—	Signal reference in {type}.{name} format (e.g., keyword.code_keywords).
operator	no	equals	Comparison: equals, contains, greater-than, less-than, in.
value	no	—	Comparison value. Numeric for greater-than/less-than, array for in.
negate	no	false	Invert the condition result.

Action

The action block defines the execution strategy when a rule matches.

action:
  strategy: fallback
  primary_model: gpt-5.2
  fallback_models: [gpt-5-mini]

Field	Required	Description
strategy	yes	Execution strategy: default, parallel, or fallback.
primary_model	yes	Model to use (must reference a configured model).
fallback_models	no	Ordered fallback chain if primary fails.
model_refs	no	Model references with reasoning control overrides.
algorithm	no	Selection algorithm (confidence or ratings).

Strategies:

• default — Send to the primary model. On failure, try fallback models in order.
• parallel — Send to multiple models simultaneously. Return the first successful response.
• fallback — Try primary model first, then each fallback in order until one succeeds.

Algorithms

Algorithms control how models are selected within a rule.

Confidence algorithm — Escalate to larger models when confidence is low:

algorithm:
  type: confidence
  confidence:
    threshold: 0.75
    confidence_method: margin
    on_error: skip

Field	Required	Default	Description
threshold	yes	—	Confidence threshold (0.0–1.0). At or above it the first ref is used; below it the last ref is selected.
confidence_method	no	—	Method: margin, avg_logprob, or hybrid. When unset, the signal score is used directly.
escalation_order	no	—	Reserved for future use — parsed and validated but ignored by routing. Planned modes: size, cost, automix.
on_error	no	—	Error handling: skip the model on failure. Only gates logprob-extraction failure, so it is unreachable when confidence_method is unset.

Ratings algorithm — Select among multiple models by policy:

algorithm:
  type: ratings
  ratings:
    policy: highest
    on_error: skip

Field	Required	Default	Description
policy	no	—	Selection policy: highest, lowest, or prefer_first.
on_error	no	—	Error handling: skip the model on failure.

Plugins

Plugins add pre-processing and post-processing capabilities to routing rules. Configure them in the plugins array on each rule. Each plugin requires a type field and a configuration block with plugin-specific options.

rules:
  - name: secure-routing
    priority: 100
    conditions:
      - signal: keyword.code_keywords
    action:
      strategy: default
      primary_model: gpt-5.2
    plugins:
      - type: jailbreak
        configuration:
          enabled: true
          threshold: 0.8
          action: block
      - type: pii
        configuration:
          enabled: true
          threshold: 0.7
      - type: system_prompt
        configuration:
          system_prompt: "You are a helpful coding assistant."

Built-in plugins

system_prompt

Inject a system prompt into the request.

- type: system_prompt
  configuration:
    system_prompt: "You are a helpful assistant specialized in mathematics."
    mode: replace

Field	Default	Description
system_prompt	—	System prompt text to inject.
mode	replace	How to apply: replace (overwrite existing), prepend, or append.

semantic-cache

Cache responses based on semantic similarity of prompts.

- type: semantic-cache
  configuration:
    enabled: true
    similarity_threshold: 0.92
    ttl_seconds: 3600

Field	Default	Description
enabled	true	Enable or disable the plugin.
similarity_threshold	—	Minimum similarity (0.0–1.0) to return a cached response.
ttl_seconds	—	Cache entry time-to-live.

Note

The maximum number of entries in the semantic cache store is controlled globally via defaults.semantic_cache_max_entries, not per-plugin.

jailbreak

Detect and handle prompt injection attempts.

- type: jailbreak
  configuration:
    enabled: true
    threshold: 0.8
    action: block

Field	Default	Description
enabled	true	Enable or disable the plugin.
threshold	—	Detection threshold (0.0–1.0).
action	—	Response action when detected.

pii

Detect personally identifiable information in prompts.

- type: pii
  configuration:
    enabled: true
    threshold: 0.7
    pii_types_allowed: []

Field	Default	Description
enabled	true	Enable or disable the plugin.
threshold	—	Detection threshold (0.0–1.0).
pii_types_allowed	[]	Array of PII types to allow through (empty = block all).

header_mutation

Modify HTTP headers on proxied requests and responses using a mutations array.

- type: header_mutation
  configuration:
    enabled: true
    mutations:
      - header: X-Routed-By
        operation: set
        value: llmsoup
        phase: response
      - header: X-Request-Source
        operation: add
        value: llmsoup-proxy
        phase: request

Field	Description
enabled	Enable or disable the plugin.
mutations	Array of mutation rules (see below).

Each mutation entry:

Field	Description
header	Header name.
operation	set (overwrite), add (append), or remove.
value	Header value (not required for remove).
phase	When to apply: request or response.

hallucination

Detect potential hallucinations in model responses.

- type: hallucination
  configuration:
    enabled: true
    threshold: 0.7
    action: header
    heuristic_sensitivity: 0.5

Field	Default	Description
enabled	true	Enable or disable the plugin.
threshold	—	Detection threshold (0.0–1.0).
action	—	Response action: header, body, block, or log.
heuristic_sensitivity	0.5	Heuristic sensitivity (0.0–1.0). Higher values are more sensitive.

router_replay

Capture routing decisions for debugging and analysis.

- type: router_replay
  configuration:
    enabled: true
    max_records: 200
    capture_request_body: false
    max_body_bytes: 4096

Field	Default	Description
enabled	true	Enable or disable the plugin.
max_records	200	Maximum routing records stored in memory.
capture_request_body	false	Whether to capture request payloads.
max_body_bytes	4096	Maximum bytes per captured body (truncated if exceeded).

Authentication

llmsoup supports token-based authentication on all endpoints except /metrics.

auth:
  enabled: true
  tokens:
    - env: MY_API_TOKEN
  tokens_file: /etc/llmsoup/tokens.yaml

Field	Type	Default	Description
enabled	boolean	true	Enable authentication (defaults to true when auth section exists).
tokens	array	—	Inline token definitions using secret references.
tokens_file	string	—	Path to an external tokens file (preferred over inline).

Tokens file format

tokens:
  - id: alice
    description: "Alice - Data team"
    secret:
      env: ALICE_TOKEN

  - id: bob
    description: "Bob - Engineering"
    secret:
      file: /run/secrets/bob_token

  - id: ci-pipeline
    description: "CI/CD service account"
    secret:
      env: CI_API_TOKEN

Tip

Use a separate tokens file for easier rotation. Token IDs appear in logs and metrics for auditing.

Caution

Never store actual token values in configuration files. Always use secret references (environment variables, files, or vault).

Persistence

The optional persistence block enables durable per-user usage tracking. When enabled, per-user usage aggregates are written to an embedded SQLite store by a single background writer and restored on startup. Persistence is disabled by default.

persistence:
  enabled: true
  path: usage.db
  flush_interval_seconds: 10
  quota_period: monthly
  daily_retention_days: 90
  anonymous_quota: exempt

Field	Type	Default	Description
enabled	boolean	false	Enable durable usage persistence.
path	string	usage.db	Path to the embedded SQLite store file. A single process owns this file.
flush_interval_seconds	integer (>0)	10	Interval in seconds between background flushes of buffered usage to the store.
quota_period	string	monthly	Accounting period applied to per-user quotas: daily, weekly, monthly, or total. Boundaries are evaluated in UTC, and total never resets.
daily_retention_days	integer	90	Days of daily rollup rows to retain before pruning.
anonymous_quota	string	exempt	How the reserved anonymous bucket is treated under quotas: exempt (never rejected) or reject (rejected while quotas are enforced).

Note

Meaningful per-user tracking (and per-user quotas) requires auth.tokens_file so that requests carry a token identity. Without identified tokens, all usage collapses into a single reserved anonymous bucket.

Per-user quotas

Per-user quotas are defined on individual token entries in the tokens file via a quota block. Any dimension left unset is unlimited; each dimension that is set must be positive.

tokens:
  - id: alice
    description: "Alice - Data team"
    secret:
      env: ALICE_TOKEN
    quota:
      max_cost: 50.0       # dollars per quota_period
      max_tokens: 1000000  # prompt + completion tokens per period
      max_requests: 5000   # requests per period

Field	Type	Description
max_cost	float	Maximum cost per period, in dollars.
max_tokens	integer	Maximum tokens (prompt + completion) per period.
max_requests	integer	Maximum requests per period.

Caution

Unlimited is expressed by omitting a dimension, not by setting it to 0. A max_requests, max_tokens, or max_cost of 0 (or any non-positive max_cost) is rejected at load time — e.g. quota.max_requests must be greater than 0 (omit the field for unlimited).

Caution

Invalid persistence config is rejected at load time with a file:line:column error. This includes an empty path, a non-writable parent directory, or a flush_interval_seconds of 0.

See Persisted usage and Quotas for the full treatment.

Secret Resolution

All access_key and token secret fields support multiple resolution methods.

Environment variable

access_key:
  env: OPENAI_API_KEY

Reads the value from the specified environment variable.

File

access_key:
  file: /run/secrets/api_key

Reads the value from a file path. Useful with Docker/Kubernetes secrets.

Command

access_key:
  command: "cat /etc/secret/key"

Caution

Command-based secret resolution is disabled by default. Set LLMSOUP_ALLOW_COMMAND_SECRETS=1 to enable it.

Vault

access_key:
  vault: "secret/data/api_key"

Note

Vault integration is planned for a future release. Using this method currently returns an error.

Environment Variables

Variable	Purpose	Default
LLMSOUP_CONFIG	Config file path	config.yaml
LLMSOUP_HOST	Server bind address	127.0.0.1
LLMSOUP_PORT	Server port	8080
LLMSOUP_LOG	Log level filter	info
LLMSOUP_ALLOW_COMMAND_SECRETS	Enable command-based secret resolution	unset
LLMSOUP_PREPARE_OUTPUT	Override output path for llmsoup prepare	config.yaml
LLMSOUP_ONNX_MODELS_DIR	Override directory for ML model storage	~/.llmsoup/models
LLMSOUP_SKIP_MODEL_DOWNLOAD	Skip downloading embedding/domain models (for CI/testing)	unset

Complete Example

A full configuration with two models, multiple signal types, routing rules, authentication, and plugins:

version: v0.1

defaults:
  default_model: gpt-5-mini
  preference_model: gpt-5-mini
  request_timeout_ms: 60000
  model_cache_ttl_seconds: 300
  embedding_cache_capacity: 1000
  cost_aware_routing: true
  cost_quality_tradeoff: 0.3
  include_cost_headers: true
  context_overflow: truncate_middle
  default_fallback_models: [gpt-5-mini]
  cost_baseline_model: gpt-5.2

models:
  - name: gpt-5-mini
    provider: openai
    access_key:
      env: OPENAI_API_KEY
    endpoints:
      - url: https://api.openai.com/v1/chat/completions
    metadata:
      context_window: 8000
      parameter_count: 120000
      latency_seconds: 0.5
    pricing:
      prompt_per_1m: 0.25
      completion_per_1m: 2.00

  - name: gpt-5.2
    provider: openai
    access_key:
      env: OPENAI_API_KEY
    endpoints:
      - url: https://api.openai.com/v1/chat/completions
    metadata:
      context_window: 128000
      parameter_count: 520000
      latency_seconds: 1.2
    pricing:
      prompt_per_1m: 1.75
      completion_per_1m: 14.00
    reasoning_family: reasoning_effort

signals:
  keyword:
    - name: code_keywords
      operator: OR
      keywords: ["code", "function", "implement", "debug", "program"]

    - name: math_keywords
      operator: OR
      keywords: ["calculate", "equation", "formula", "solve"]

  embedding:
    - name: quick_answer
      model: sentence-transformers/all-MiniLM-L12-v2
      threshold: 0.90
      candidates:
        - "quick question"
        - "simple answer"
      aggregation_method: max

  domain:
    - name: math
      mmlu_categories: [math]
      threshold: 0.6

    - name: programming
      examples:
        - "Write a function"
        - "Debug this code"
      threshold: 0.7

  language:
    - name: english
      language: en

    - name: chinese
      language: zh

  latency:
    - name: low_latency
      max_tpot: 0.050

  fact_check:
    - name: needs_fact_check
      description: "Requests requiring factual accuracy"

  preference:
    - name: code_generation
      description: "User wants code generation"

rules:
  - name: code-routing
    priority: 100
    operator: AND
    conditions:
      - signal: keyword.code_keywords
    action:
      strategy: fallback
      primary_model: gpt-5.2
      fallback_models: [gpt-5-mini]
    plugins:
      - type: system_prompt
        configuration:
          system_prompt: "You are a senior software engineer."
      - type: jailbreak
        configuration:
          enabled: true
          threshold: 0.8
          action: block

  - name: math-routing
    priority: 90
    operator: OR
    conditions:
      - signal: keyword.math_keywords
      - signal: domain.math
        operator: greater-than
        value: 0.6
    action:
      strategy: default
      primary_model: gpt-5.2
    plugins:
      - type: system_prompt
        configuration:
          system_prompt: "Show your work step by step."

  - name: quick-answers
    priority: 80
    conditions:
      - signal: embedding.quick_answer
    action:
      strategy: default
      primary_model: gpt-5-mini

  - name: cost-efficient
    priority: 40
    conditions:
      - signal: domain.programming
    action:
      strategy: default
      primary_model: gpt-5-mini
      fallback_models: [gpt-5.2]
      algorithm:
        type: confidence
        confidence:
          threshold: 0.75
          confidence_method: margin
          on_error: skip

auth:
  enabled: true
  tokens_file: /etc/llmsoup/tokens.yaml