Signals & Routing

llmsoup evaluates configurable signals on every incoming request, matches routing rules based on signal results, selects the best model using algorithms, and executes the request through a chosen strategy. This page explains the entire pipeline in depth.

For YAML configuration syntax and field-level details, see the Configuration Reference.

Request flow

Every chat completion request follows this pipeline:

HTTP request
    │
    ▼
Auth middleware ── rejects if token invalid
    │
    ▼
Signal evaluation ── all signals run concurrently
    │
    ▼
Rule matching ── rules evaluated in priority order (highest first)
    │
    ▼
Algorithm (optional) ── confidence or ratings model selection
    │
    ▼
Strategy execution ── default, fallback, or parallel
    │
    ▼
Model HTTP call ── forwarded as OpenAI-compatible request
    │
    ▼
Plugin execution ── pre/post-routing plugins, if configured (see Plugins Reference)
    │
    ▼
Response ── returned in OpenAI format

1. Signal evaluation — All configured signals evaluate concurrently against the request payload. Each signal produces a score (0.0–1.0), a triggered boolean, and metadata. Signal failures are isolated — routing continues without the failed signal (see Graceful degradation).
2. Rule matching — The decision engine evaluates rules in descending priority order. Conditions within a rule use AND, OR, or NOR logic with five condition operators. The first matching rule wins.
3. Algorithm selection — If the matched rule specifies an algorithm (confidence or ratings), it selects the optimal model from the rule’s model_refs list.
4. Strategy execution — The selected model(s) are called according to the execution strategy: single call (default), sequential retry (fallback), or concurrent race (parallel).
5. Response — The model response is returned in OpenAI-compatible format.

---

Signal types

Signals are the building blocks of routing decisions. Each signal evaluates one aspect of the incoming request and produces a result that conditions can match against.

All signals share these result properties:

Property	Type	Description
score	0.0–1.0	Confidence or similarity score
triggered	boolean	Whether the signal matched
metadata	key-value pairs	Signal-specific details for debugging

keyword

Performs substring matching against the request payload.

Field	Type	Default	Description
name	string	required	Signal identifier
keywords	list	required	Phrases to match against
case_sensitive	boolean	false	Enable case-sensitive matching
operator	string	OR	Match logic: AND, OR, or NOR

Operators:

• OR — triggers if any keyword is found in the payload
• AND — triggers if all keywords are found in the payload
• NOR — triggers if no keywords are found (inverse of OR)

Score: 1.0 when triggered, 0.0 otherwise.

signals:
  keyword:
    - name: urgent
      keywords: [urgent, critical, emergency]
      case_sensitive: false
      operator: OR
    - name: coding
      keywords: [code, program, function, debug]

embedding

Computes semantic similarity between the request payload and reference text or candidate phrases using BERT embeddings (via Candle).

Field	Type	Default	Description
name	string	required	Signal identifier
model	string	required	HuggingFace model ID or local path
threshold	float	required	Similarity threshold (0.0–1.0)
context_window	integer	512	Maximum token length
reference_text	string	—	Single reference phrase
candidates	list	—	Multiple candidate phrases
aggregation_method	string	max	How candidate scores combine: max, avg, or any

How it works:

1. The payload and reference/candidates are embedded using the configured model.
2. Cosine similarity is computed between the payload and each reference.
3. The raw cosine similarity (range -1.0 to 1.0) is normalized to 0.0–1.0 using (cosine + 1.0) / 2.0.
4. The normalized score is compared against the threshold.

Aggregation methods (when using candidates):

Method	Behavior
max	Highest similarity score across all candidates (default)
avg	Average similarity score across all candidates
any	Triggers if any single candidate exceeds the threshold

The key difference: max and avg produce a single aggregate score compared to the threshold, while any checks each candidate individually — the signal triggers as soon as one candidate passes.

Model loading: Models are loaded lazily on first use. Supported sources:

• HuggingFace repository ID (downloaded automatically, e.g., sentence-transformers/all-MiniLM-L12-v2)
• Local file path (must contain config.json, tokenizer.json, and weight files)

Caching: Embeddings are cached per-request (shared across evaluators) and per-evaluator (LRU). Batch precomputation runs before individual evaluators for efficiency.

signals:
  embedding:
    - name: semantic-match
      model: sentence-transformers/all-MiniLM-L12-v2
      threshold: 0.8
      candidates:
        - technical documentation
        - API reference
        - code examples
      aggregation_method: max

domain

Classifies the request into subject-matter categories using a BERT-based classifier trained on MMLU categories.

Field	Type	Default	Description
name	string	required	Domain name to match
examples	list	—	Example phrases for the domain
description	string	—	Human-readable domain description
mmlu_categories	list	—	MMLU category aliases to match
threshold	float	see below	Minimum confidence for triggering

Threshold resolution order:

1. Signal-level threshold field
2. Global classifier.category_model.threshold
3. Default: 0.5

How it works:

1. The classifier predicts an MMLU category for the request payload.
2. The predicted category is compared against the signal’s name and mmlu_categories aliases.
3. If the category matches and confidence meets the threshold, the signal triggers.

All domain signals share a single classifier instance for memory efficiency.

classifier:
  category_model:
    model_id: LLM-Semantic-Router/lora_intent_classifier_bert-base-uncased_model
    category_mapping_path: tests/data/domain/category_mapping.json
    threshold: 0.5

signals:
  domain:
    - name: math
      mmlu_categories: [math, abstract_algebra, college_mathematics]
      threshold: 0.7
    - name: science
      examples: [physics, chemistry, biology]

language

Detects the natural language of the request payload using the whatlang library (no ML model required).

Field	Type	Default	Description
name	string	required	Signal identifier
language	string	required	Expected ISO 639-1 code (case-insensitive)

How it works:

1. The payload text is analyzed by whatlang for language detection.
2. The detected ISO 639-3 code is mapped to ISO 639-1 (e.g., eng → en).
3. The signal triggers when the detected language matches the configured language.

Score: whatlang confidence score (0.0–1.0). Returns 0.0 if detection fails (e.g., very short or ambiguous text).

Supported languages (ISO 639-1 codes):

Code	Language	Code	Language	Code	Language
af	Afrikaans	hi	Hindi	pa	Punjabi
ak	Akan	hr	Croatian	pl	Polish
am	Amharic	hu	Hungarian	pt	Portuguese
ar	Arabic	hy	Armenian	ro	Romanian
az	Azerbaijani	id	Indonesian	ru	Russian
be	Belarusian	it	Italian	si	Sinhala
bg	Bulgarian	ja	Japanese	sk	Slovak
bn	Bengali	jv	Javanese	sl	Slovenian
ca	Catalan	ka	Georgian	sn	Shona
cs	Czech	km	Khmer	sr	Serbian
cy	Welsh	kn	Kannada	sv	Swedish
da	Danish	ko	Korean	ta	Tamil
de	German	la	Latin	te	Telugu
el	Greek	lt	Lithuanian	th	Thai
en	English	lv	Latvian	tk	Turkmen
eo	Esperanto	mk	Macedonian	tl	Tagalog
es	Spanish	ml	Malayalam	tr	Turkish
et	Estonian	mr	Marathi	uk	Ukrainian
fa	Persian	my	Myanmar	ur	Urdu
fi	Finnish	nb	Norwegian Bokmal	uz	Uzbek
fr	French	ne	Nepali	vi	Vietnamese
gu	Gujarati	nl	Dutch	yi	Yiddish
he	Hebrew	or	Odia	zh	Chinese
				zu	Zulu

signals:
  language:
    - name: english
      language: en
    - name: french
      language: fr

latency

Evaluates model performance based on TPOT (Time Per Output Token) metrics collected from previous requests. TPOT values are smoothed using exponential moving average (EMA).

Field	Type	Default	Description
name	string	required	Signal identifier
max_tpot	float	required	TPOT threshold in seconds

How it works:

1. The evaluator reads TPOT metrics from the shared cache (populated by model responses).
2. If the routing context includes candidate_models metadata, only those models are checked. Otherwise, all models with TPOT data are considered.
3. The best (lowest) TPOT value among candidates is compared against the threshold.
4. Triggers when the best TPOT is at or below max_tpot.

Score: Confidence is calculated as 1.0 - (best_tpot / max_tpot), clamped to 0.0–1.0. Higher scores indicate faster models relative to the threshold.

TPOT calculation: total_latency_seconds / completion_token_count per response, smoothed with EMA across requests.

Note: TPOT data is only available after at least one response from a model. On first request (cold start), no TPOT data exists and the signal returns score 0.0 without triggering.

signals:
  latency:
    - name: fast_response
      max_tpot: 0.05
    - name: moderate_latency
      max_tpot: 0.15

preference

Routes requests based on LLM-classified user preferences. An external LLM analyzes the conversation to determine which preference route best matches.

Field	Type	Default	Description
name	string	required	Preference label (route name)
description	string	—	Description of this preference style

How it works:

1. All preference signals are collected and sent to an external LLM classifier along with the conversation context. The prompt lists the allowed route names and instructs the model to copy one verbatim.
2. The LLM determines which preference label (signal name) best matches the request.
3. The signal whose name matches the classifier’s output triggers.
4. Classification results are cached per-request — multiple preference signals share one LLM call.

What the classifier actually sees: not the whole request. System and developer prompts are excluded, and the context is anchored on the most recent user turn — in an agent loop the intent does not change between tool iterations, so the last human instruction is the stable signal. The anchor is served first from a 4000-character budget; the remaining slots (6 turns in total) go to the assistant turns that follow the anchor, then to the turns preceding it, nearest first, so an ordinary follow-up like “now make it async” keeps its subject. Selected turns are rendered chronologically as role: text lines. A turn too large for what is left of the budget keeps its head and its tail, joined by a [...] marker — a pasted payload usually carries the instruction on its last line, so truncating from the front would drop the very thing being classified.

This anchoring matters for agent traffic. Assistant turns that only call tools carry content: null, and tool results are neither user nor assistant messages, so both are skipped. Without an anchor, a long agent session fills the window with narration, the original instruction scrolls out, and the classifier sees only text like “now let me check the tests” — which classifies as a catch-all route with high confidence, i.e. confidently wrong rather than ambiguous. If no user turn carries text at all — there is no user message, or every one of them is image- or audio-only — no anchored context is produced and the classifier falls back to the plain request payload instead — every system and developer prompt verbatim, the input the anchored context exists to keep out, and with no cap of any kind, so such a request can carry far more than the 4000 characters.

How the budget is shared. The anchor draws on it first and uncapped, since it is the intent being classified. Every other selected turn is then capped at an even share of what remains — the remainder divided by the turns still to place — so a single long neighbour cannot swallow the window. A turn shorter than its share hands the surplus to the next one, and the last turn placed may use everything left, so no budget is stranded. Turns are dropped once the budget runs out. The practical effect for agent traffic: a question followed by a multi-thousand-character assistant answer and a short “now make it async” no longer produces a prompt that is ~99% assistant code and ~0.6% user intent — the oversized answer is elided to roughly its fair share and the surrounding user turns survive.

Tolerant route resolution: small classifier models occasionally return a route that is not in the configured list — most often by echoing a content word from the prompt (e.g. motion) instead of a real route name. Rather than fail the whole preference signal set, the router resolves such answers tolerantly, in this order:

1. Exact — the answer is a configured route name, or the reserved other.
2. Normalized exact — case- and separator-insensitive (Code-Review resolves to code_review).
3. Containment, either direction, and only for normalized names of at least four characters, so a short echo cannot latch onto a longer route. Containment collects every matching route: exactly one candidate resolves to it, two or more are ambiguous.
4. Fallback — nothing matched.

Both ambiguous and unmatched answers degrade to the reserved other route rather than being guessed. review sits inside both code_review and complex_review; picking one would be a coin flip.

What you observe for each outcome:

Outcome	Route taken	Reported confidence	Log line
Exact	the named route	as normal — the margin, or the constant 1.0	—
Normalized or single-candidate containment	the matched route	as normal — the margin, or the constant 1.0	WARN naming the requested and resolved route
Ambiguous (two or more candidates)	other	0.0	WARN naming the requested route and every candidate
Unmatched	other	0.0	WARN naming the unrecognized route

The 0.0 matters: a rewritten answer reports no confidence at all, in margin mode and in the default mode where the classifier otherwise reports a constant 1.0. The classifier’s own score described a route the router did not take, so carrying it over would report a coin flip — or a route that exists nowhere — as a certainty. A model that deliberately answers other is unaffected and keeps its real score.

What that 0.0 costs you. It does not send the request to default_model. The other signal still triggers — its name matches the rewritten label — and a condition written without a value defaults to comparing true against triggered, so the catch-all rule matches on the trigger alone; default_model is reached only when no rule matches. What the zero changes is which ref of that rule runs. With a non-zero confidence threshold no ref can clear it, so the algorithm takes its escalation branch and selects the last ref in model_refs — which is where escalation targets belong, and so is normally the most expensive model on the route. Every ambiguous or unrecognized answer is therefore billed at the catch-all’s escalation price rather than its primary price, and an intermediate ref is never the one that runs. Order the catch-all’s refs with that in mind, or set its threshold to 0.0, which makes a rewritten answer commit to the first ref instead.

These WARN lines are the only evidence of a rewrite — there is no dedicated metric for it. other is always a valid classifier answer even if you do not configure an other signal (it simply triggers no rule unless an other route exists). If off-list fallbacks are frequent, enrich your route descriptions or use a stronger classifier model. Ambiguity in particular is a naming problem, not a model problem: two route names that contain a common token (code_review and complex_review) collide on every echo of it, and only non-colliding names fix that.

Structured outputs (recommended for weak classifiers): set defaults.preference_structured_outputs: true to send the classifier request with a JSON-schema response_format whose route (and, in margin mode, second_route) is enum-locked to your configured route names. On models that support structured outputs this makes two common failure modes structurally impossible: malformed JSON (Failed to parse JSON: trailing characters, extra prose, markdown fences) and off-list route echoes — the model can only return one of your routes. Providers or models that reject response_format fall back to prompt-only parsing automatically, so it is safe to leave on — the retry fires only when the upstream error text actually names the parameter as unsupported, not merely because a 400 body echoes the classifier’s own request back, so an unrelated failure surfaces on the first attempt instead of being retried blind; and a transient empty choices response (a provider-side hiccup, logged as No choices in LLM response) is retried once before the signal degrades to default_model routing. Use it together with tolerant route resolution above as defense in depth.

Important: Preference evaluation makes an external LLM call during routing, which consumes tokens and adds latency. This cost is tracked separately from the main model call.

Classification deadline: one classification — the first attempt plus any of the retries above — is bounded end to end by a single wall-clock budget. A single first attempt is unaffected; the deadline exists so a flaky classifier cannot chain several full-timeout calls and stall routing for minutes before the routed model is contacted. On expiry the signal fails with a timeout and graceful degradation drops it, so the request routes on the remaining signals or on default_model.

Set the budget explicitly with defaults.preference_classification_budget_ms, an integer number of milliseconds between 100 and 300000; a value outside that range fails validation at load time with the offending file, line and column. When the key is omitted the budget is derived as twice the classifier model’s longest configured endpoint timeout_ms (which is defaults.request_timeout_ms unless an endpoint overrides it), so an existing config behaves exactly as before.

Do not reach for a lower endpoint timeout_ms to bound classification: that is not an equivalent knob. It shortens every individual call to the classifier model as well, so it trades a bounded worst case for first attempts that time out on a merely slow provider. preference_classification_budget_ms caps the aggregate without touching the per-call timeout.

An expired budget is visible in metrics as llmsoup_signal_errors_total{signal_type="preference", error_type="timeout"} — a bucket distinct from the inference one that upstream and parse failures land in, so it can be alerted on directly.

Signal score: by default the classifier is only asked for a route name and the signal score is a constant 1.0 — confidence algorithms then always select the first model ref. Set defaults.preference_confidence: true to ask the classifier for its top-2 routes with scores; the signal score becomes the margin (best − second, clamped to 0.0–1.0), so ambiguous classifications score low and confidence thresholds become meaningful. Margins are a separation measure, not a probability — clear winners typically land around 0.5–0.8, coin flips near 0.0. In either mode, an answer the router had to rewrite to other — ambiguous or unrecognized, per the resolution table above — scores 0.0 rather than the constant 1.0 or the classifier’s own margin.

Beware margin quantisation when picking a threshold. Classifiers report scores to one decimal place, so the margins you actually observe are quantised — 0.30, 0.50, 0.80, 0.90, with nothing in between. A threshold placed inside the common band, such as 0.35, therefore turns the gap between two adjacent levels into the decision: the same kind of prompt escalates at 0.30 and does not at 0.50, which reads as random routing. Place the threshold between quantisation levels but outside the band where real classifications cluster — 0.25 (escalate only when the classifier is genuinely torn) or 0.45 (escalate on anything short of a clear winner). Calibrate from the llmsoup_decision_confidence metric.

signals:
  preference:
    - name: concise
      description: User prefers short, direct answers
    - name: detailed
      description: User prefers thorough, comprehensive answers
    - name: creative
      description: User wants creative or exploratory responses

Reading the confusion matrix

In margin mode the classifier names its second choice as well as its first, and both survive as labels on llmsoup_classification_total: category is the route taken, runner_up the route that came closest. The runner-up is resolved through the same tolerant path as the selected route, so separator drift normalises onto the configured name rather than unbounding the label set. Two label values are reserved: other means the model named the catch-all, and unresolved means its second route matched nothing configured — they are kept apart so a hallucinated slot is never read as a real second choice. Without a runner-up the label is empty, which Prometheus treats as absent.

That empty value covers more than you might expect, because this counter is shared: preference signals in the default single-route mode leave it empty, and so do the domain and fact_check classifiers, which increment the same counter and never report a runner-up. The counter carries no label naming the classifier, so runner_up!="" is what isolates margin-mode preference traffic — every query below depends on it. This also matters for reading the catch-all: domain has its own literal other category, so category="other" on its own mixes two unrelated things, while category="other", runner_up!="" is unambiguously the preference catch-all.

That filter is not free, and the queries below are shaped by what it drops. Three populations fall outside it, all of them margin-mode: an answer where the model named no second route, an answer the router had to rewrite to other (its runner-up is dropped along with its score, for the same reason — the model’s ranking described routes the router did not take), and an answer whose second route was the selected route. The first is the important one: those are the classifications where the classifier was most decisive. So a category total read under this filter counts contested classifications only, and systematically understates how often a route wins outright. Read the matrix for the shape of contention between routes, not for volume. For how often a route was actually selected, drop the filter — but the counter is still shared and still per-evaluator, so the honest form names your routes explicitly and keeps the divisor below:

ceil(
  sum by (category) (increase(llmsoup_classification_total{category=~"route-a|route-b|other"}[7d]))
    / scalar(count(count by (signal_name) (increase(llmsoup_signal_extraction_total{signal_type="preference"}[7d]) > 0)))
)

Substitute your own signals.preference[].name values for the matcher. Even then it cannot separate a preference other from the domain classifier’s own other category — there is no label naming the classifier, and runner_up!="" is not available here because dropping it is the point.

The pair is a confusion matrix over your route taxonomy:

ceil(
  sum by (category, runner_up) (increase(llmsoup_classification_total{runner_up!=""}[7d]))
    / scalar(count(count by (signal_name) (increase(llmsoup_signal_extraction_total{signal_type="preference"}[7d]) > 0)))
)

That divisor is not decoration. llmsoup_classification_total is incremented once per signal evaluation, not once per classification — every preference evaluator increments it for the one classification they all share — so raw cells are inflated by the number of configured preference routes. Counting the distinct signal_names that saw traffic on llmsoup_signal_extraction_total recovers the factor. Two details matter for getting a readable answer out of it:

• Keep the divisor on the same range as the numerator. An instant count(count by (signal_name) (llmsoup_signal_extraction_total{...})) looks equivalent and is not: after five minutes without traffic — a quiet weekend, a restarted pod, a user_id selection with nothing recent — it returns an empty vector, scalar() of which is NaN, and every cell blanks even though the numerator has a week of data.
• Round up, not to nearest. increase() extrapolates over the range, so a cell whose true value is 1 can compute to 0.4; round() would then display 0 for a pair that genuinely occurred, which is the one reading the matrix must never get wrong. ceil() can overstate a cell by one and never zeroes a real pair, so 0 keeps its meaning: that pair did not occur in this range.

Read it column-wise to answer the question a top-1 counter cannot: is a route that never fires losing narrowly, or is it never in contention at all?

• (a) Losing narrowly — a high runner_up total against a low category total. The classifier recognises the intent and keeps ranking the route second. That is a threshold or wording problem: the two routes’ descriptions overlap, or the margin threshold is set where the pair’s margins cluster.
• (b) Never in contention — absent from both columns. The classifier is not seeing the intent at all, which is a description problem (or the route does not describe real traffic).
• (c) The catch-all row — a large category="other" row tells you which real route each catch-all classification was closest to, which is the evidence for merging, rewording, or splitting a route. Keep the runner_up!="" matcher on it, or the domain classifier’s own other category is counted alongside. This row holds only answers where the model chose other; ones the router rewrote to it are excluded by design, and the WARN naming the requested route is the evidence for those.

Pair it with the margin distribution to size the gap. Note what this second query can and cannot tell you: llmsoup_classification_confidence carries category but not runner_up, so a margin belongs to the selected route across all of its runner-ups — it cannot be attributed to one specific pair.

histogram_quantile(0.5, sum by (le, category) (rate(llmsoup_classification_confidence_bucket{classification_method="preference"}[7d])))

This histogram has no label for the mode either, and single-route mode records a constant 1.0 into it. A flat p50 = p90 = 1.0 therefore means the deployment is not in margin mode — not that every route separates perfectly.

A series pinned to the bottom bucket is the mirror artifact, and it has two causes. category="other" sits there because answers rewritten to the catch-all record 0.0 to mark themselves unscored. Any other route can sit there too, for an unrelated reason: a classification whose own score was fine but whose second score came back unusable ("NaN", "high", a boolean) also records 0.0, because a margin cannot be computed from it. That route still appears normally in the confusion matrix, so a bottom-bucket route that looks healthy in the matrix is a malformed-output problem with the classifier model, not a coin flip between two routes.

Read the two together: the matrix names the pair, the histogram sizes the gap. A route that both wins by margins clustered near 0.0 and has one dominant runner-up in its matrix row is a naming collision between those two routes. A route winning at wide margins is separating cleanly, whatever sits second.

One case where absence is not evidence. If two of your route names share a word — code_review and complex_review, say — a model that answers with the bare token review names neither unambiguously, so it resolves to unresolved rather than to a route. Those near-misses drain out of both routes’ runner_up columns, which can make a genuinely contested pair look like case (b) above. Structured outputs (defaults.preference_structured_outputs) close this off by constraining the model to the exact route names, so the path only opens when a provider rejects response_format — check llmsoup_classification_degraded_mode before reading a shared-word route as never in contention, and treat a large unresolved column as a signal to rename the overlapping pair.

The bundled Grafana dashboard (monitoring/grafana/dashboards/llmsoup.json) ships these in its Classification row — the matrix itself, a per-route selected-versus-runner-up table, a breakdown of what other was closest to, and the margin distribution — so none of these queries has to be typed by hand. Two more panels beside them cover the classifier’s own health: whether it is currently suppressing a provider capability it had rejected, and the rate of those degradation events.

One limit to keep in mind: both queries can only report routes the classifier has actually emitted. A route that is never selected and never a runner-up produces no series at all, so it is absent from the matrix rather than showing as a zero row — which is why case (b) has to be read against your configured route list, not off the matrix alone.

fact_check (stub)

Stub implementation

The fact_check signal uses a local binary classifier that is not yet fully integrated. The classifier model may not be available in all environments. This signal is planned for full implementation in a future release.

Classifies whether content requires fact-checking using a local binary classifier.

Field	Type	Default	Description
name	string	required	Must be needs_fact_check or no_fact_check_needed
description	string	—	Optional description

Recognized signal names:

• needs_fact_check — triggers when the classifier predicts FACT_CHECK_NEEDED
• no_fact_check_needed — triggers when the classifier predicts NO_FACT_CHECK_NEEDED
• Any other name will never trigger

signals:
  fact_check:
    - name: needs_fact_check
      description: Content requiring verification
    - name: no_fact_check_needed
      description: Creative content not needing verification

user_feedback (stub)

Stub implementation

The user_feedback signal uses a local classifier that is not yet fully integrated. The classifier model may not be available in all environments. This signal is planned for full implementation in a future release.

Categorizes user follow-up messages using a local classifier.

Field	Type	Default	Description
name	string	required	One of the recognized feedback labels
description	string	—	Optional description

Recognized signal names:

• satisfied — user is satisfied with the response
• need_clarification — user needs more detail or explanation
• want_different — user wants a different type of response
• wrong_answer — user indicates the response was incorrect

The evaluator uses follow_up_message context (if available) rather than the original payload.

signals:
  user_feedback:
    - name: satisfied
    - name: need_clarification
    - name: want_different
    - name: wrong_answer

---

Condition operators

Routing rules use conditions to check signal results. Each condition specifies a signal name, an operator, and a value to compare against.

equals

Compares signal results for exact match. This is the default operator when none is specified.

• Boolean comparison: Matches triggered state — true means the signal fired, false means it did not.
• Numeric comparison: Matches the signal score with floating-point tolerance (±0.0001).
• String comparison: Matches signal metadata values (e.g., detected language code).

conditions:
  - signal: keyword.urgent
    operator: equals
    value: true         # keyword signal triggered

  - signal: language.english
    operator: equals
    value: true         # detected language is English

  - signal: embedding.semantic
    operator: equals
    value: 0.95         # exact score match (with tolerance)

contains

String substring matching on signal metadata.

conditions:
  - signal: keyword.tech
    operator: contains
    value: "router"     # "router" appears in matched keywords

  - signal: domain.science
    operator: contains
    value: "physics"    # "physics" is a substring of the predicted category

greater-than

Numeric comparison — signal score exceeds the value. For latency signals, compares against the best TPOT value.

conditions:
  - signal: embedding.relevance
    operator: greater-than
    value: 0.85         # similarity score > 0.85

  - signal: latency.speed
    operator: greater-than
    value: 0.1          # best TPOT > 0.1 seconds (slow models)

less-than

Numeric comparison — signal score is below the value. For latency signals, compares against the best TPOT value.

conditions:
  - signal: embedding.relevance
    operator: less-than
    value: 0.3          # low similarity

  - signal: latency.speed
    operator: less-than
    value: 0.05         # best TPOT < 50ms (fast models)

in

Checks whether the signal score or metadata value is contained in a list.

conditions:
  - signal: domain.category
    operator: in
    value: [math, physics, chemistry]  # predicted domain in set

  - signal: language.detected
    operator: in
    value: [en, fr, de]   # detected language in set

Rule logic operators

Conditions within a rule are combined using the rule’s operator field:

Operator	Behavior
AND (default)	All conditions must match
OR	At least one condition must match
NOR	No conditions must match

routing:
  rules:
    - name: complex_query
      priority: 100
      operator: AND          # both conditions required
      conditions:
        - signal: keyword.technical
          operator: equals
          value: true
        - signal: embedding.complexity
          operator: greater-than
          value: 0.8
      action:
        primary_model: gpt-4

---

Algorithms

When a routing rule includes an algorithm configuration, the decision engine uses it to select the optimal model from the rule’s model_refs list. Two algorithms are available: confidence and ratings.

Both algorithms are fit-aware: refs whose effective context window cannot hold the request’s estimated tokens (messages + tools + output budget) are skipped, as long as at least one ref on the decision fits. If no ref fits, the ranking described below stands and the configured overflow strategy truncates the request. The decision’s fallback chain lists fitting refs before non-fitting ones, so a transient failure falls back to another model that fits rather than one that would truncate. Skipped refs are logged with the model name, estimated tokens, and effective window; models without metadata.context_window always qualify. See Context overflow strategies.

confidence

Selects the first model that meets a confidence threshold, with escalation to the last model if none qualify.

Field	Type	Default	Description
threshold	float	required	Minimum confidence score (0.0–1.0)
confidence_method	string	null	Scoring method: null, avg_logprob, margin, or hybrid
hybrid_weights	object	—	Weights for hybrid method
escalation_order	string	—	Reserved for future use (planned modes: size, cost, automix)
cost_quality_tradeoff	float	—	Per-algorithm override (0.0 = quality, 1.0 = cost)
on_error	string	skip	Error handling: skip or fail

Confidence methods:

Method	How score is computed
null (default)	Maximum signal score from the matched rule’s conditions
avg_logprob	Normalized average log-probability from model response tokens
margin	Normalized average margin between top-2 token probabilities
hybrid	Weighted combination of logprob and margin scores

Logprob normalization: Average log-probability is mapped from [-3.0, 0.0] to [0.0, 1.0]:

normalized = (avg_logprob + 3.0) / 3.0   (clamped to 0.0–1.0)

Margin normalization: Average margin between top-1 and top-2 token log-probabilities:

normalized = 1.0 - exp(-avg_margin / 3.0)   (clamped to 0.0–1.0)

When fewer than 2 top logprobs are available for a token, a default margin of 2.0 is used.

Hybrid method: Combines logprob and margin scores:

score = logprob_weight × normalized_logprob + margin_weight × normalized_margin

hybrid_weights:
  logprob_weight: 0.6
  margin_weight: 0.4

Selection behavior:

1. Each model in model_refs receives a confidence score.
2. If cost-aware routing is enabled, a cost penalty is subtracted from each score.
3. The first model meeting the threshold is selected.
4. If no model meets the threshold, the last model in model_refs is selected (escalation).

Decision-time scoring

Routing decisions are made before the model responds, so response logprobs are not available at decision time and the avg_logprob, margin, and hybrid methods fall back to signal-based scoring (the matched rule’s maximum signal score). For preference-routed rules that score is the classifier confidence, which is a constant 1.0 unless defaults.preference_confidence: true is set — see preference. Without it, thresholds never trigger escalation and the first model ref is always selected.

Error handling (on_error):

• skip (default) — if the confidence method (e.g., avg_logprob) is unavailable, fall back to signal-based scoring
• fail — return an error immediately

algorithm:
  type: confidence
  confidence:
    threshold: 0.7
    confidence_method: margin
    cost_quality_tradeoff: 0.5
    on_error: skip
model_refs:
  - gpt-3.5-turbo
  - gpt-4
  - gpt-4-turbo

In this example, if no model reaches 0.7 confidence, gpt-4-turbo (last) is selected as escalation.

ratings

Computes a score per model using a configurable policy and selects the highest-rated model.

Field	Type	Default	Description
policy	string	highest	Scoring policy
cost_quality_tradeoff	float	—	Per-algorithm override (0.0 = quality, 1.0 = cost)
on_error	string	skip	Error handling: skip or fail

Scoring policies:

Policy	Behavior
highest (default)	Select the model with the highest signal score
lowest	Select the model with the lowest signal score (inverted: 1.0 - score)
prefer_first	Tie-break favoring earlier models in model_refs order
prefer_last	Tie-break favoring later models in model_refs order

prefer_first and prefer_last add a tiny position-based bonus (0.0001 per position) to break ties deterministically.

Selection behavior:

1. Each model receives a score from the policy.
2. If cost-aware routing is enabled, a cost penalty is subtracted.
3. The model with the highest final score is selected.
4. Ties are broken by model_refs order.

algorithm:
  type: ratings
  ratings:
    policy: highest
    cost_quality_tradeoff: 0.3
model_refs:
  - claude-3-haiku
  - gpt-3.5-turbo
  - gpt-4

Cost-aware routing

Both algorithms support cost-aware model selection. When enabled, model pricing is factored into the selection score.

Tradeoff resolution order:

1. Per-algorithm cost_quality_tradeoff (overrides everything)
2. Global defaults.cost_quality_tradeoff
3. Default: 0.3 (when cost-aware routing is enabled but no tradeoff specified)

Tradeoff values:

• 0.0 — pure quality (no cost penalty)
• 0.3 — default balance (slight cost preference)
• 0.5 — equal weight to quality and cost
• 1.0 — pure cost minimization

Cost penalty formula:

cost_penalty = normalized_cost × tradeoff
adjusted_score = max(confidence_score - cost_penalty, 0.0)

Cost normalization: avg(prompt_per_1m, completion_per_1m) / 100.0, clamped to 0.0–1.0.

defaults:
  cost_aware_routing: true
  cost_quality_tradeoff: 0.3

routing:
  rules:
    - name: budget_query
      action:
        algorithm:
          type: confidence
          confidence:
            threshold: 0.6
            cost_quality_tradeoff: 0.8  # override: prefer cheap models

---

Execution strategies

After model selection, the execution strategy determines how the request is sent to model endpoint(s).

default

Routes to a single model (the primary model). This is the simplest strategy.

Behavior:

1. Send request to the primary model.
2. Return response or timeout error.

action:
  strategy: default
  primary_model: gpt-4

fallback

Tries the primary model first, then falls back to alternative models sequentially on failure.

Behavior:

1. Send request to the primary model.
2. If primary fails (error or timeout), try the first fallback model.
3. Continue through fallback models in order until one succeeds.
4. If all models fail, return a combined error.

The response indicates whether a fallback was used via metadata.

action:
  strategy: fallback
  primary_model: gpt-4-turbo
  fallback_models:
    - gpt-4
    - gpt-3.5-turbo

parallel

Sends the request to the primary model and all fallbacks concurrently, returning the first successful response with deterministic priority.

Behavior:

1. Launch concurrent requests to all models (primary + fallbacks).
2. As responses arrive, select the winner using priority order (primary first).
3. A later model only wins if all earlier models have failed.
4. Abort remaining requests once a winner is determined.

Determinism: The primary model always wins over fallbacks if both succeed, regardless of which responds first. This prevents non-deterministic routing behavior.

action:
  strategy: parallel
  primary_model: gpt-4-turbo
  fallback_models:
    - gpt-4
    - claude-3-opus

---

Graceful degradation

llmsoup is designed to continue routing even when individual components fail. Non-critical operations — metrics recording, cache reads/writes, and tracing — are wrapped in error isolation boundaries.

Signal evaluation failures:

• If a signal evaluator fails (model loading error, timeout, invalid context), the signal is excluded from the evaluation results.
• Routing continues with the remaining successful signals.
• The failure is logged as a warning but does not produce a hard error.

Non-critical operation failures:

• Metrics recording — if Prometheus metrics fail to record, the request proceeds normally.
• Cache operations — if embedding or response cache fails, the request falls through to a direct model call (cache miss behavior).
• Tracing — if distributed tracing fails, request processing is unaffected.

What does cause hard errors:

• Authentication failure (invalid or missing token)
• No model available to handle the request (no matching rule and no default model)
• All models in a strategy execution fail (all fallbacks exhausted)

This design ensures that observability and optimization features never block request processing. A signal failure simply means routing has less information — it does not mean the request fails.