Metrics Reference

llmsoup exposes 42 Prometheus metrics covering requests, routing, signals, caching, plugins, costs, models, and reasoning. All metrics are available at the /metrics endpoint in Prometheus text exposition format.

Endpoint behavior

Property	Value
URL	GET /metrics
Authentication	Not required
Content-Type	text/plain; version=0.0.4
Format	Prometheus text exposition format

curl http://localhost:8080/metrics

For Prometheus scrape configuration and Docker Compose monitoring setup, see the Deployment Guide.

---

Build information

llmsoup_build_info

Type: Gauge — Value: always 1

Identifies the running binary. The value carries no information; the version does, in the label. This is the Prometheus convention for build metadata.

Label	Description
version	Crate version of the running binary

llmsoup_build_info{version="0.1.4"} 1

The version is read from the binary itself, not from configuration, so it stays truthful when a deployment fails to roll the image forward — a case that is otherwise invisible, since llmsoup exposes no version endpoint and /metrics is typically restricted to the internal network.

Useful queries:

# Which version is running, per instance
llmsoup_build_info

# Annotate any series with the running version
llmsoup_requests_total * on(instance) group_left(version) llmsoup_build_info

# More than one version live at once — a rollout in flight, or a stuck one
count(count by (version) (llmsoup_build_info)) > 1

# An instance whose version changed in the last hour
count by (instance) (count_over_time(llmsoup_build_info[1h])) > 1

Both rollout queries count series, not values. changes(llmsoup_build_info[1h]) is always 0: the value of each series is the literal 1 written once at startup, and a new version starts a new series under a new version label rather than moving the old one. Counting distinct version labels is what actually observes a rollout.

---

Naming conventions

All llmsoup metrics follow Prometheus best practices:

• Prefix: Every metric starts with llmsoup_
• Case: snake_case for metric names and label names
• Unit suffixes:
  • _total — counters (monotonically increasing)
  • _seconds — duration histograms
  • _dollars — cost values (project-specific, not a Prometheus standard)
  • _bytes — sizes (reserved, not currently used)
• Label conventions:
  • user_id — authenticated user or "anonymous"
  • model — model name from routing configuration
  • decision_name — routing rule/decision name
  • status — operation outcome (success, error, hit, miss)

---

Request metrics

Core HTTP request tracking.

llmsoup_requests_total

Property	Value
Type	Counter
Labels	method, status_code, endpoint, user_id
Description	Total HTTP requests received

Labels:

• method — HTTP method (GET, POST)
• status_code — HTTP status code (200, 401, 500)
• endpoint — Request path (/v1/chat/completions, /metrics)
• user_id — Authenticated user ID or "anonymous"

llmsoup_active_connections

Property	Value
Type	Gauge
Labels	None
Description	Current active HTTP connections

llmsoup_errors_total

Property	Value
Type	Counter
Labels	error_type, user_id
Description	Total errors by type

Labels:

• error_type — Error category (routing_error, model_error, config_error)
• user_id — Authenticated user ID or "anonymous"

llmsoup_quota_exceeded_total

Property	Value
Type	Counter
Labels	user_id
Description	Requests rejected for exceeding a per-user quota

Increments each time a request is rejected with an HTTP 429 insufficient_quota error because the authenticated user exceeded their configured per-user quota.

Labels:

• user_id — Authenticated user ID or "anonymous"

Example PromQL — Request rate by endpoint:

rate(llmsoup_requests_total{endpoint="/v1/chat/completions"}[5m])

Example PromQL — Error rate percentage:

sum(rate(llmsoup_errors_total[5m])) / sum(rate(llmsoup_requests_total[5m])) * 100

---

Routing metrics

Routing decision evaluation, rule matching, and model selection.

llmsoup_routing_decisions_total

Property	Value
Type	Counter
Labels	outcome, decision_name, user_id
Description	Routing decision outcomes

Labels:

• outcome — Result of routing (success, failed, fallback, default)
• decision_name — Name of the matched routing rule
• user_id — Authenticated user ID or "anonymous"

llmsoup_routing_duration_seconds

Property	Value
Type	Histogram
Labels	decision_name, user_id
Description	Time spent making routing decisions
Buckets	0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0 (Prometheus defaults)

llmsoup_decision_evaluation_total

Property	Value
Type	Counter
Labels	user_id
Description	Total routing rule evaluations

llmsoup_decision_match_total

Property	Value
Type	Counter
Labels	decision_name, user_id
Description	Total routing rule matches by decision name

llmsoup_decision_confidence

Property	Value
Type	Histogram
Labels	decision_name, user_id
Description	Algorithm confidence/score for routing decisions
Buckets	0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 0.99

llmsoup_model_selection_total

Property	Value
Type	Counter
Labels	algorithm_type, model, decision_name, user_id
Description	Total algorithm-based model selections

Labels:

• algorithm_type — Algorithm used ("confidence" or "ratings")
• model — Selected model name
• decision_name — Name of the matched routing rule
• user_id — Authenticated user ID or "anonymous"

llmsoup_model_routing_modifications_total

Property	Value
Type	Counter
Labels	source_model, target_model, user_id
Description	Total fallback/parallel re-routing events

Labels:

• source_model — Original primary model before re-routing
• target_model — Actual model that responded after re-routing

Example PromQL — Routing decisions per second by outcome:

sum by (outcome) (rate(llmsoup_routing_decisions_total[5m]))

Example PromQL — Average routing latency:

rate(llmsoup_routing_duration_seconds_sum[5m]) / rate(llmsoup_routing_duration_seconds_count[5m])

---

Signal metrics

Per-signal evaluation tracking across all signal types: keyword, embedding, domain, language, latency, fact_check, user_feedback, preference.

llmsoup_signal_extraction_total

Property	Value
Type	Counter
Labels	signal_type, signal_name, user_id
Description	Signal extraction attempts

Labels:

• signal_type — Signal evaluator type (keyword, embedding, domain, language, latency, fact_check, user_feedback, preference)
• signal_name — Specific signal name from evaluator configuration

llmsoup_signal_match_total

Property	Value
Type	Counter
Labels	signal_type, signal_name, user_id
Description	Successful signal matches (triggered=true)

llmsoup_signal_extraction_duration_seconds

Property	Value
Type	Histogram
Labels	signal_type, user_id
Description	Per-signal evaluation latency
Buckets	0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0, 5.0

llmsoup_signal_errors_total

Property	Value
Type	Counter
Labels	signal_type, error_type, user_id
Description	Signal evaluation failures

Labels:

• signal_type — Signal family that failed (keyword, embedding, domain, language, latency, preference, fact_check, user_feedback)
• error_type — Failure kind: timeout, inference, invalid_context, duplicate_signal, or task_panic

A preference classification that exhausts its budget is reported as error_type="timeout", distinct from the inference bucket that upstream and parse failures land in, so it can be alerted on directly.

Read it as a count of evaluators, not of classifications. All preference signals share one classifier call, but each signal has its own evaluator and each reports the shared failure independently, so a single expired budget increments this counter once per configured preference signal — twelve times on a config with twelve preference signals. Divide by that number to get classifications:

sum(rate(llmsoup_signal_errors_total{signal_type="preference", error_type="timeout"}[5m]))
  /
count(count by (signal_name) (llmsoup_signal_extraction_total{signal_type="preference"}))

llmsoup_signal_errors_total carries no signal_name label, so all of those increments collapse onto one series and cannot be separated after the fact. llmsoup_signal_extraction_total does carry signal_name and is incremented once per evaluator on every request, so counting its distinct signal_names recovers the divisor. Compare the undivided rate against request rate and the classifier-timeout rate reads high by exactly that factor.

Sustained increments there mean the classifier is not answering inside defaults.preference_classification_budget_ms — raise the budget or move to a faster classifier model.

llmsoup_classification_confidence

Property	Value
Type	Histogram
Labels	category, classification_method, user_id
Description	Classification confidence scores
Buckets	0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 0.99

Labels:

• category — Predicted classification category (e.g., "math", "computer_science")
• classification_method — Classifier type ("domain", "fact_check", "preference")

llmsoup_classification_total

Property	Value
Type	Counter
Labels	category, runner_up, user_id
Description	Classification outcomes across all classifier-based evaluators

Tracks domain, fact_check, and preference classification events. Incremented once per signal evaluation.

Labels:

• category — the selected classification category
• runner_up — the classifier’s second choice, when it reports one. Only preference signals with defaults.preference_confidence enabled do; the domain and fact_check classifiers share this counter and always leave it empty, which Prometheus treats as an absent label. Since no label names the classifier, runner_up!="" is what isolates margin-mode preference traffic. Two values are reserved: other (the model named the catch-all) and unresolved (its second route matched nothing configured). Paired with category it makes the counter a confusion matrix — read that section before trusting the totals, because the runner_up!="" filter excludes decisive classifications.
• user_id — authenticated user ID, or "anonymous"

Cardinality

category and runner_up are each bounded by your configured route set, but they multiply, and user_id is unbounded in a multi-tenant deployment: worst case is roughly routes × (routes + 1) × distinct_users series. Six preference routes and 200 authenticated users is ~11k series on this counter alone. runner_up is only populated in margin mode, so the extra factor is 1 with preference_confidence off. Drop user_id in a scrape relabel rule if the product is too large for your Prometheus.

Example PromQL — Signal evaluation rate by type:

sum by (signal_type) (rate(llmsoup_signal_extraction_total[5m]))

Example PromQL — Signal match ratio (how often signals trigger):

sum by (signal_type) (rate(llmsoup_signal_match_total[5m]))
  / sum by (signal_type) (rate(llmsoup_signal_extraction_total[5m]))

---

llmsoup_classification_degradations_total

Property	Value
Type	Counter
Labels	kind
Description	Fallbacks applied by the preference classifier to obtain an answer

Counted as each fallback is applied rather than on the success path, so an attempt sequence that ultimately fails or exhausts its classification budget still leaves the evidence of what it tried.

Labels:

• kind — response_format_dropped (the provider rejected response_format, so structured outputs were dropped), temperature_dropped (the model only accepts the default temperature), or empty_choices_retried (a provider returned an empty choices array and the call was retried once)

A confirmed response_format or temperature rejection is remembered on the classifier and re-probed every 256 classifications, so against a provider that genuinely rejects one this counter is a slow heartbeat — roughly one increment per 256 classifications, indefinitely — not a rate proportional to traffic. At low volume that can look like silence for hours. Either way the counter answers when the rejection happened, never whether the capability is off now: use the gauge below for that.

empty_choices_retried is not memoized and is a genuine per-request rate.

Example PromQL — Degradation rate by kind:

sum by (kind) (rate(llmsoup_classification_degradations_total[5m]))

llmsoup_classification_degraded_mode

Property	Value
Type	Gauge
Labels	kind
Description	1 while the preference classifier suppresses a rejected request capability

Answers the state question the counter cannot: is structured output off right now? 1 means the capability is currently suppressed, 0 that a re-probe found it accepted. Only response_format_dropped and temperature_dropped appear — empty_choices_retried is a per-request retry, not a mode.

A sustained 1 on response_format_dropped means the classifier is running without the JSON-schema enum lock, so off-list route hallucinations are possible again; cross-check against the preference classifier returned an off-list route WARN.

Example PromQL — Capabilities currently suppressed:

max by (kind) (llmsoup_classification_degraded_mode) == 1

---

Cache metrics

Observability for three cache backends tracked by Prometheus metrics:

• model_response — TTL-based cache for model responses
• embedding — Capacity-based LRU cache for embedding vectors
• semantic — Semantic cache plugin cache

Note: llmsoup also maintains an internal TPOT (Time Per Output Token) EMA cache for latency-aware routing. TPOT values are tracked via llmsoup_model_tpot_seconds in the Model metrics section rather than through cache-level metrics.

llmsoup_cache_operations_total

Property	Value
Type	Counter
Labels	backend, operation, status
Description	Cache operations by backend, operation, and status

Labels:

• backend — Cache backend (model_response, embedding, semantic)
• operation — Operation type (get, put, evict)
• status — Operation outcome (hit, miss, error)

llmsoup_cache_operation_duration_seconds

Property	Value
Type	Histogram
Labels	backend, operation
Description	Cache operation latency
Buckets	0.00001, 0.00005, 0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5

Buckets are microsecond-to-millisecond focused for in-memory caches.

llmsoup_cache_entries

Property	Value
Type	Gauge
Labels	backend
Description	Current cache entry count by backend

llmsoup_cache_plugin_hits_total

Property	Value
Type	Counter
Labels	decision_name
Description	Semantic cache plugin hits by decision name

llmsoup_cache_plugin_misses_total

Property	Value
Type	Counter
Labels	decision_name
Description	Semantic cache plugin misses by decision name

Example PromQL — Cache hit rate by backend:

sum by (backend) (rate(llmsoup_cache_operations_total{status="hit"}[5m]))
  / sum by (backend) (rate(llmsoup_cache_operations_total{operation="get"}[5m]))

Example PromQL — Semantic cache plugin effectiveness:

sum(rate(llmsoup_cache_plugin_hits_total[5m]))
  / (sum(rate(llmsoup_cache_plugin_hits_total[5m])) + sum(rate(llmsoup_cache_plugin_misses_total[5m])))

---

Plugin metrics

Execution tracking for all plugin types: semantic-cache, jailbreak, pii, system_prompt, header_mutation, hallucination, router_replay.

llmsoup_plugin_execution_total

Property	Value
Type	Counter
Labels	plugin_type, decision_name, status, user_id
Description	Plugin executions by type, decision, status, and user

Labels:

• plugin_type — Plugin type (semantic-cache, jailbreak, pii, system_prompt, header_mutation, hallucination, router_replay)
• decision_name — Name of the matched routing rule
• status — Execution status (success, error)

llmsoup_plugin_execution_duration_seconds

Property	Value
Type	Histogram
Labels	plugin_type, user_id
Description	Plugin execution latency
Buckets	0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0, 5.0

Expect wide variance: regex-only plugins run in ~10-100us, embedding-based in ~10-500ms, and LLM-calling plugins in ~100ms-5s.

llmsoup_plugin_errors_total

Property	Value
Type	Counter
Labels	plugin_type, error_reason, user_id
Description	Plugin execution failures by type and error reason

Labels:

• error_reason — Error category (execution_failed, configuration_error, timeout, internal_error)

llmsoup_pii_violations_total

Property	Value
Type	Counter
Labels	model, pii_type, user_id
Description	PII violations detected by model, type, and user

Labels:

• pii_type — PII type detected (EMAIL_ADDRESS, PHONE_NUMBER, etc.)

Example PromQL — Plugin error rate by type:

sum by (plugin_type) (rate(llmsoup_plugin_errors_total[5m]))
  / sum by (plugin_type) (rate(llmsoup_plugin_execution_total[5m]))

Example PromQL — PII violations over time:

sum by (pii_type) (rate(llmsoup_pii_violations_total[5m]))

---

Cost metrics

Per-request cost tracking and cost-aware routing savings.

llmsoup_request_cost_dollars_total

Property	Value
Type	Counter
Labels	model, user_id, currency
Description	Cumulative request cost in dollars by model

llmsoup_request_cost_dollars

Property	Value
Type	Histogram
Labels	model, user_id
Description	Per-request cost distribution in dollars
Buckets	0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0

llmsoup_routing_cost_dollars_total

Property	Value
Type	Counter
Labels	user_id, currency
Description	Cumulative cost of external routing calls (e.g., preference signal LLM calls)

llmsoup_cost_savings_dollars_total

Property	Value
Type	Counter
Labels	decision_name, user_id, currency
Description	Estimated cost savings from cost-aware routing

llmsoup_tokens_total

Property	Value
Type	Counter
Labels	model, user_id, token_type
Description	Total tokens processed by model and type

Labels:

• token_type — One of "prompt", "completion", "cached_prompt", "cached_completion"

llmsoup_tokens_per_request

Property	Value
Type	Histogram
Labels	model, user_id, token_type
Description	Token count distribution per request
Buckets	10, 50, 100, 500, 1000, 2000, 5000, 10000, 50000, 100000

Example PromQL — Hourly cost by model:

sum by (model) (increase(llmsoup_request_cost_dollars_total[1h]))

Example PromQL — Total cost savings:

sum(increase(llmsoup_cost_savings_dollars_total[24h]))

---

Model metrics

Per-model latency, throughput, and error tracking.

llmsoup_model_request_duration_seconds

Property	Value
Type	Histogram
Labels	model, user_id
Description	End-to-end model call latency in seconds
Buckets	0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0 (Prometheus defaults)

llmsoup_model_tpot_seconds

Property	Value
Type	Histogram
Labels	model
Description	Time Per Output Token (TPOT) in seconds
Buckets	0.001, 0.005, 0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1.0

TPOT measures how long each output token takes to generate. llmsoup uses Exponential Moving Average (EMA) smoothing for TPOT values in its internal cache, but the raw per-request TPOT is recorded in this histogram. Lower TPOT indicates faster model generation speed.

For streamed (passthrough) calls, TPOT is computed from real inter-chunk timing: (last chunk time − first chunk time) / completion tokens. For buffered calls it remains total request duration / completion tokens.

llmsoup_model_ttft_seconds

Property	Value
Type	Histogram
Labels	model
Description	Time to first streamed token in seconds
Buckets	0.05, 0.1, 0.25, 0.5, 1.0, 2.0, 3.0, 5.0, 8.0, 13.0, 21.0, 34.0

Time from the start of the serving model ref’s own upstream call to its first content chunk. Time spent on earlier failed fallback refs is excluded, so the histogram stays attributable to the model it is labeled with. Recorded for streamed (passthrough) calls only — buffered and non-streaming requests do not observe this histogram. Buckets span fast streaming starts (50 ms) to slow reasoning warmups (34 s).

llmsoup_model_request_errors_total

Property	Value
Type	Counter
Labels	model, reason, user_id
Description	Model request errors by model and reason

Labels:

• reason — Error reason code (timeout, http_error, parse_error, connection_error)

llmsoup_context_length_retries_total

Property	Value
Type	Counter
Labels	model, outcome
Description	Truncate-and-retry attempts after upstream context-length rejections

Incremented by the context-aware model caller each time an upstream call is rejected for exceeding the model’s context length. Frequent increments mean the token estimator is under-counting for your traffic — this is the signal for calibrating defaults.context_headroom_ratio.

Labels:

• outcome — Retry result: recovered (retry succeeded), failed (retry rejected again), skipped (retry not attempted — no meaningful truncation possible or time budget exhausted)

llmsoup_usage_estimated_total

Property	Value
Type	Counter
Labels	model, reason
Description	Streamed requests whose token usage was estimated instead of provider-reported

llmsoup asks upstream providers for a final usage chunk (stream_options.include_usage). When that chunk never arrives — the provider does not support it, or the client disconnects mid-stream — usage is estimated from the forwarded content (~4 characters per token) and this counter increments.

Labels:

• reason — Why usage was estimated (e.g., missing_usage, malformed_usage)

Example PromQL — P95 time-to-first-token by model:

histogram_quantile(0.95, sum by (model, le) (rate(llmsoup_model_ttft_seconds_bucket[5m])))

Example PromQL — P99 model latency:

histogram_quantile(0.99, sum by (model, le) (rate(llmsoup_model_request_duration_seconds_bucket[5m])))

Example PromQL — Model error rate:

sum by (model) (rate(llmsoup_model_request_errors_total[5m]))
  / sum by (model) (rate(llmsoup_model_request_duration_seconds_count[5m]))

llmsoup_model_provider_total

Property	Value
Type	Counter
Labels	model, provider
Description	Upstream provider that served each model call

On an aggregator a single model id fans out to many upstream providers at different prices, quantisations and capabilities. This counter reports which one actually served each call, so a provider_routing pin can be verified against traffic instead of assumed.

Labels:

• provider — Provider name as reported by the upstream response (e.g. DeepSeek, Google AI Studio)

Recorded on both buffered and streamed (passthrough) calls. Streamed calls increment once per call, from the first chunk that names a provider — aggregators repeat the field on every chunk, so counting each one would measure generation length rather than provider dispersion. Recording happens in the HTTP model caller, so only calls that actually reached an upstream count: a cache hit records nothing, and on a fallback chain each ref that served records under its own model label.

A response that reports no provider records nothing rather than filling an unknown bucket that would dilute the distribution. Backends that are not aggregators have no such field, so this counter stays empty for them — and llmsoup skips the body scan entirely for them, keyed off the model’s configured provider:. Only openrouter counts as an aggregator; a model left without a provider: is treated as unknown and still scanned. The practical consequence is that an OpenRouter endpoint mislabelled as, say, provider: openai reports no serving provider at all, so an empty counter on a model you believe is aggregator-backed is worth checking against that field.

The provider label is upstream-controlled and every distinct value is a new time series, so it is bounded before it is recorded: values are trimmed, capped at 48 bytes, and dropped entirely if they contain anything outside ASCII alphanumerics, -, _, . and space. A dropped value is logged at debug level, not bucketed — a counter that stays empty while traffic flows points at the upstream, not at routing.

Example PromQL — Provider share for one model:

sum by (provider) (rate(llmsoup_model_provider_total{model="deepseek/deepseek-v4-pro-0813"}[1h]))

---

Reasoning metrics

Reasoning control decisions and effort distribution for models that support reasoning parameters, plus what actually reached the upstream.

llmsoup_reasoning_decisions_total

Property	Value
Type	Counter
Labels	category, model, enabled, effort, user_id
Description	Reasoning decisions by category, model, enabled state, effort level, and user

Labels:

• category — Matched rule name or "default"
• model — Routed model name
• enabled — Whether reasoning is enabled ("true", "false")
• effort — Reasoning effort level ("low", "medium", "high", "default")

llmsoup_reasoning_effort_usage_total

Property	Value
Type	Counter
Labels	family, effort, user_id
Description	Reasoning effort distribution by family and effort level

Labels:

• family — Reasoning family type ("reasoning_effort", "chat_template_kwargs")
• effort — Reasoning effort level ("low", "medium", "high", "default")

Example PromQL — Reasoning usage by effort level:

sum by (effort) (rate(llmsoup_reasoning_decisions_total[5m]))

Example PromQL — Reasoning family distribution:

sum by (family, effort) (rate(llmsoup_reasoning_effort_usage_total[5m]))

llmsoup_delivered_reasoning_effort_total

Property	Value
Type	Counter
Labels	model, effort
Description	Reasoning control present on the outbound request body, by model

The two counters above record what the configuration says. This one records what actually reached the wire, read from the serialized outbound request body.

The distinction is load-bearing. A reasoning pin needs three coordinated pieces — reasoning_family on the model, plus use_reasoning and reasoning_effort on every ref — and any one missing leaves the body untouched. Upstream, an absent or unsupported effort is not an error: the provider falls back to its own default, which on a mandatory-reasoning model is typically the most expensive rung. Recording the configured value would restate the config and detect none of this.

Labels:

• model — Routed model name
• effort — The value on the body. "none" when the key is absent, which is what an incomplete pin looks like. A chat_template_kwargs family carries a toggle rather than an effort, so it reports "enabled" / "disabled".

Recorded once per routed request, from the body as serialized before dispatch, and only for models that declare a reasoning_family — a model with none has no effort to deliver and emits no series at all rather than an "unknown" bucket. A fallback or parallel strategy sends that same body to further refs without emitting additional series, so the counter tracks requests rather than upstream calls and is labelled with the routed model, not whichever ref ended up serving.

The value can originate in a client request body — reasoning_effort survives the request passthrough and reaches the outbound body verbatim when no reasoning mutation replaces it — so it is bounded twice before being recorded. Its shape must be at most 24 bytes of ASCII alphanumerics, - and _; and its value must be one llmsoup’s own configuration knows: any effort a model declares in supported_efforts, any effort a rule pins, the low / medium / high fallback set, and the intrinsic none / enabled / disabled labels. Everything else folds into "other", so the series count is bounded by your config rather than by what callers send.

Note that defaults.default_model and defaults.default_fallback_models carry no model refs, so no reasoning mutation is reachable on a default decision. Requests that fall through to the default model record effort="none" even when a rule pins an effort for the same model.

This is a dedicated metric rather than a label on the cost or latency series: those back shipped Grafana dashboards and already carry user_id, which a new dimension would multiply.

Example PromQL — Delivered vs configured effort for one model:

sum by (effort) (rate(llmsoup_delivered_reasoning_effort_total{model="z-ai/glm-5.2"}[1h]))

A persistent effort="none" series on a model you believe is pinned means the pin is inert. See Pinning effort on a mandatory-reasoning model.

llmsoup_reasoning_tokens_total

Property	Value
Type	Counter
Labels	model
Description	Reasoning tokens reported by upstream responses, by model

Makes “billed for reasoning while configured not to reason” visible, which no configuration-side metric can show.

Read from usage.completion_tokens_details.reasoning_tokens, falling back to a flat usage.reasoning_tokens for providers that report it there. Recorded on both the buffered and the streamed path; on a stream, once per call from the last chunk carrying a usage object, so a provider that reports cumulative usage on every chunk is counted at its final value rather than its first, and a provider that merely repeats the same usage is not multiplied by it.

Unlike llmsoup_delivered_reasoning_effort_total above, the model label here is the model that served the request, not the routed one — on a fallback the two differ, and these tokens were produced by whichever ref actually answered. Take care joining the two metrics by (model) on traffic that falls back.

A provider that never reports reasoning tokens records nothing — not a zero, which would read as “reasoned for free”. Consistency across providers is therefore worth confirming on your own traffic before alerting on this series.

Example PromQL — Reasoning tokens per model:

sum by (model) (rate(llmsoup_reasoning_tokens_total[1h]))

Example PromQL — Share of completion tokens spent reasoning:

sum by (model) (rate(llmsoup_reasoning_tokens_total[1h]))
  / sum by (model) (rate(llmsoup_tokens_total{token_type="completion"}[1h]))

---

Complete metric index

Metric	Type	Category
llmsoup_build_info	Gauge	Build
llmsoup_requests_total	Counter	Request
llmsoup_active_connections	Gauge	Request
llmsoup_errors_total	Counter	Request
llmsoup_quota_exceeded_total	Counter	Request
llmsoup_routing_decisions_total	Counter	Routing
llmsoup_routing_duration_seconds	Histogram	Routing
llmsoup_decision_evaluation_total	Counter	Routing
llmsoup_decision_match_total	Counter	Routing
llmsoup_decision_confidence	Histogram	Routing
llmsoup_model_selection_total	Counter	Routing
llmsoup_model_routing_modifications_total	Counter	Routing
llmsoup_signal_extraction_total	Counter	Signal
llmsoup_signal_match_total	Counter	Signal
llmsoup_signal_extraction_duration_seconds	Histogram	Signal
llmsoup_signal_errors_total	Counter	Signal
llmsoup_classification_confidence	Histogram	Signal
llmsoup_classification_total	Counter	Signal
llmsoup_classification_degradations_total	Counter	Signal
llmsoup_classification_degraded_mode	Gauge	Signal
llmsoup_cache_operations_total	Counter	Cache
llmsoup_cache_operation_duration_seconds	Histogram	Cache
llmsoup_cache_entries	Gauge	Cache
llmsoup_cache_plugin_hits_total	Counter	Cache
llmsoup_cache_plugin_misses_total	Counter	Cache
llmsoup_plugin_execution_total	Counter	Plugin
llmsoup_plugin_execution_duration_seconds	Histogram	Plugin
llmsoup_plugin_errors_total	Counter	Plugin
llmsoup_pii_violations_total	Counter	Plugin
llmsoup_request_cost_dollars_total	Counter	Cost
llmsoup_request_cost_dollars	Histogram	Cost
llmsoup_routing_cost_dollars_total	Counter	Cost
llmsoup_cost_savings_dollars_total	Counter	Cost
llmsoup_tokens_total	Counter	Cost
llmsoup_tokens_per_request	Histogram	Cost
llmsoup_model_request_duration_seconds	Histogram	Model
llmsoup_model_tpot_seconds	Histogram	Model
llmsoup_model_ttft_seconds	Histogram	Model
llmsoup_model_request_errors_total	Counter	Model
llmsoup_context_length_retries_total	Counter	Model
llmsoup_usage_estimated_total	Counter	Model
llmsoup_model_provider_total	Counter	Model
llmsoup_reasoning_decisions_total	Counter	Reasoning
llmsoup_reasoning_effort_usage_total	Counter	Reasoning
llmsoup_delivered_reasoning_effort_total	Counter	Reasoning
llmsoup_reasoning_tokens_total	Counter	Reasoning