Persisted usage

By default every usage and cost number llmsoup tracks lives only in an in-memory Prometheus registry, so it resets to zero on every restart. Enabling persistence keeps durable, per-user aggregates in an embedded SQLite store so your stats survive restarts and can back per-user reporting and quotas.

Enabling persistence

Add a persistence section to your config. It is disabled by default:

persistence:
  enabled: true
  # Store file path (created on first run). One process owns this file.
  path: "usage.db"
  # Seconds between background flushes to disk (must be > 0).
  flush_interval_seconds: 10
  # Quota accounting period: daily | weekly | monthly | total.
  quota_period: monthly
  # Days of daily rollup rows to retain before pruning.
  daily_retention_days: 90

Invalid values (an empty path, a non-writable parent directory, or a flush_interval_seconds of 0) are rejected at startup with a file:line:column error.

Per-user tracking requires auth.tokens_file

Recorded usage is attributed to the user_id that a Bearer token maps to via auth.tokens_file. Requests that carry no token identity — including every request on the deprecated inline-tokens auth path — are recorded under a single reserved anonymous bucket. Meaningful per-user tracking therefore requires auth.tokens_file.

What persists vs. what stays volatile

Persisted, per user, and restored on startup:

• Lifetime totals: requests, errors, cost, and savings.
• Per-model breakdown: input/output tokens, cached tokens, cost, and request latency (kept as a sum and a sample count so the average is reproducible).
• Per-route match counts and per-error-type counts.
• Current quota-period consumption and the period boundary.
• Per-UTC-day rollups powering date-range reporting.

Stays volatile (never persisted):

• Response, embedding, and semantic caches (persisting them would risk serving stale answers).
• The model health registry (it self-heals on restart).
• Prometheus latency/confidence distributions — the store keeps aggregates and learned state, not a full time-series database. Keep scraping /metrics with Prometheus if you need histograms over time.

Restart behavior

On startup, before the network listener binds, llmsoup opens and migrates the store, seeds the in-memory accumulators from the persisted totals, and seeds the global Prometheus counters from those same totals. As a result the --stats view and /metrics reflect history immediately rather than catching a transient zero.

Money is stored as integer micro-USD to avoid floating-point drift once values feed billing and quotas.

Latency-average asymmetry

The global model-latency average shown in --stats still resets on restart, because the underlying Prometheus histogram has no sum/count setter to reseed. The per-user latency in the self-service view is reproduced from the persisted sum and count and is correct across restarts.

Clearing usage: serve --reset

llmsoup serve --reset clears all persisted per-user usage in a single transaction before restore, then starts with counters at zero and logs a warning recording the reset. It is non-interactive (no confirmation prompt) and safe to run in automation. Without --reset, persisted usage is always preserved across restarts.

llmsoup serve --reset

Durability and scaling caveats

• Flush interval: usage is flushed to disk every flush_interval_seconds and once more on graceful shutdown. A crash between flushes can lose up to one interval of usage. Lower the interval for tighter durability. The store keeps aggregates and quota state, not a per-request ledger — for penny-exact billing reconciliation, derive it from your upstream providers’ invoices.
• Single writer, single process: one process owns the store file (WAL journal, a single background writer). Running multiple instances against the same file is not supported; a horizontally scaled deployment needs a shared external store.
• Footprint: the embedded store adds only low-single-digit megabytes (a ~2 MB page cache plus the vendored SQLite code), well within the idle RSS budget. Accumulator memory scales with your finite token-file user set (roughly ~1.5 MB for ~1,000 users × ~20 models), dwarfed by the ML models.