Methodology

Overview

InferenceBench measures how the same model behaves on different inference infrastructure. The unit of observation is the execution path — Model × Provider × Region × Deployment configuration — because that is what an engineering team actually ships against. Provider rankings are an explicit aggregation over those paths, never the raw unit.

A benchmark is only as credible as its measurement process. This page specifies what is measured, how it is executed, how results are normalized and aggregated, what is excluded, and where the limits are.

Measurement model

Every number on InferenceBench belongs to exactly one of three classes. The class is part of the metric's definition and is shown in the metrics registry below.

Observed

Measured directly by InferenceBench runners against live provider endpoints: TTFT, p95 TTFT, throughput, reliability, tool-calling success, structured-output validity.

Declared

Taken from official provider documentation at observation time: input/output prices, regions, data residency, context length, supported models, hosting type. Declared data never overrides an observation.

Derived

Computed by InferenceBench from observed and declared inputs: the overall score, E2E latency, percentiles, ranks, 30-day changes, provider aggregation.

Entity model

Provider

The commercial platform operating inference endpoints (e.g. Fireworks, Nebius). Provider rows in the ranking are aggregates, one per provider.

Model

The logical model (e.g. DeepSeek-V3), independent of where it runs.

Model × Provider

The basic unit of observation: a specific model served by a specific provider through a specific endpoint and region, under a stated deployment configuration. Internally: ExecutionPath.

ModelProviderCombination = {
  modelId, providerId,
  regionId?, endpointId?, deploymentConfig?
}

Benchmark environment

Benchmarks run as raw HTTPS requests from dedicated runner locations — no provider SDKs, so client-library overhead cannot differ between providers. The first runner operates from Falkenstein (DE); additional client regions are planned and every observation records its client region.

Workloads

Workloads are versioned entities, not ad-hoc prompt sets. Each defines its prompt corpus size, input-length range, target output length, sampling parameters and the capabilities it exercises. Scores always name the workload profile they were computed under.

Request parameters are normalized per provider through recorded, versioned quirks, applied identically to every run of that provider: providers that reject an explicit temperature parameter (currently OpenAI and Anthropic) run with the parameter omitted at their documented default, and providers that reject response_format run structured-output cases prompt-only. A quirk is never inferred silently — each one is recorded the day it is observed.

Corpus sizes are deliberately small and versioned. Note the interaction with the sample gates: frontier-lab paths run correctness weekly, and one weekly sweep (7 tool cases, 6 structured) sits below the 12-result rate gate — their tool-calling and structured columns publish only after a second sweep lands in the window. Statistical weight comes from repetition — every case runs on every sweep. At the published cadence a hosted-model path accumulates ~217 eligible results per 7-day window (crossing the high-confidence threshold below); frontier-lab paths run correctness suites weekly for cost reasons and accumulate ~139, which is medium confidence and is labelled as such. Suites are hash-locked: changing any case without a version bump makes the loader refuse to run. Corpus growth is a funded-capacity decision and arrives as new suite versions, never as silent edits. Responses are never deliberately cut: every stream runs to the model's natural completion, and the per-case token ceiling exists only to bound runaway generation — it sits far above any well-formed interactive answer, reasoning burn included, with the 60 s per-operation timeout and the 120 s wall-clock bound as the outer bounds. This regime dates from 2026-08-19: earlier suite versions imposed output budgets sized for terse open-weight models, which turned reasoning-class models' internal deliberation into truncations (v0.2 of General generation raised those budgets the same day before budgets were dropped entirely). Reasoning burn is never scored as a reliability failure: a case whose entire budget is consumed by reasoning fails only that case's own capability metric, and it surfaces where a caller actually feels it — in measured TTFT and in token cost. Runs recorded under earlier versions keep citing them.

Model selection is operator-curated and never automatic, under three published rules: (1) first-party labs we measure are represented by their current flagship API models — a successor release enters, its predecessor stays measured while providers still serve it; (2) an open-weight model enters when at least two covered providers serve it, because same-model cross-provider comparison is the point of this observatory; (3) every credentialed provider keeps at least two measured models where its catalogue allows. Freshness is enforced by machine, decided by human: catalogues are re-discovered daily, new model ids surface as events the day they appear, and a daily coverage report cross-checks every tracked model against every credentialed provider's newest catalogue — naming the exact missing ids and flagging providers below the two-model minimum. Admission of a new model spends budget daily, so it stays an explicit operator decision; the mirror guarantees the decision is prompted the same day, never missed.

TTFT

TTFT measures when the model demonstrably starts responding: the first streamed delta of any channel, reasoning included — which keeps reasoning and non-reasoning models comparable on the same scale. The wait until the first user-visible output (time-to-visible) is captured separately on every run; for non-reasoning models the two are usually identical (a leading whitespace delta can separate them by one frame), for reasoning models the difference IS the thinking phase, and it is never silently mixed into either number. TTFT includes DNS/TLS (amortized by connection reuse), provider queueing, prompt processing and the first delta's generation; it excludes benchmark-client processing. The table shows p50 as the primary metric; p95 is reported alongside because tail latency determines how a provider feels under production load. Percentiles are computed over the rolling 7-day sampling window per path, never pooled across models. All latency figures are measured from the canonical European vantage (Falkenstein, DE): for providers serving from outside Europe, TTFT therefore includes genuine transatlantic network distance. This is by design, stated rather than hidden — the benchmark answers “what does this path feel like from Europe”, which is the question its audience is asking. Additional vantage regions are on the roadmap; when they ship, every figure will carry its vantage explicitly.

End-to-end latency

End-to-end latency is only comparable when output length is controlled, so InferenceBench normalizes it to a 400-output-token response using each path's observed sustained throughput. It is a derived metric and is labelled as such.

Throughput

Measured over the full delta span — TTFT is excluded, so the metric captures generation speed rather than queueing. The numerator is the provider-reported completion token count (reasoning tokens included) and the denominator spans every streamed delta, reasoning included: the same population on both sides. A response without reported usage contributes no throughput sample — SSE chunk counts undercount batched tokens and are never used as a token count — and responses shorter than 20 output tokens are excluded as degenerate speed readings. Spans under 250 ms and computed speeds above 5,000 tok/s are burst-delivery measurement artifacts, discarded at capture and retroactively excluded from aggregates. Unit: tokens per second.

Reliability

Hitting the per-case token ceiling (finish_reason=length) is a generation outcome, not a serving defect: the response was delivered, so it never lowers reliability — it fails the case, which for the tool-calling and structured suites shows in their published rates; the health suites record the failure but have no published pass rate yet. Excluded throttles stay visible: dossiers surface each provider's throttle count so the signal is published without becoming a score. Every exclusion persists a human-readable reason alongside the result, and when classification rules change, stored verdicts are re-classified under the new rules with the correction itself recorded. A run aborted by a runner or scheduler error is never written at all. Retries are disabled in the runner, so reliability reflects first-attempt behaviour.

Structured output

Structured output is not reduced to 'the JSON parses'. Each run is validated in three mechanical stages; a run only passes if every applicable stage passes.

JSON validity

The output parses as JSON.

Schema compliance

The parsed object validates against the requested JSON schema.

Constraint compliance

Stated constraints hold (enums, ranges, required fields, formats).

Tool calling

Tool-calling runs present a tool inventory and a task. A run passes only when all of the following hold:

• the correct tool is selected (including a correct no-tool decision when no tool applies);
• the argument object validates against the tool's schema;
• all required arguments are present;
• no hallucinated arguments are added;

Pricing

Input and output prices are tracked separately, per million tokens, from official provider documentation for the public serverless tier, and recorded in the currency the provider declares. For comparison — tables, rankings, the price dimension of scores and price/performance charts — non-USD prices are converted to USD at the ECB daily euro reference rate; the rate is fetched daily, stored dated with its source, and travels with each converted figure so the conversion is citable, never silent. Dossiers keep the declared figure alongside. A currency without a stored official rate is not converted and stays out of cross-currency comparisons. The score's cost dimension is the effective cost of the benchmarked exchange — the workload's published input size plus the same normalized 400-token response the E2E metric uses, at both declared prices — and it enters a score only when both prices are recorded:

Input sizes per workload: General 400 tokens, Tool calling 600, Structured output 400, Realtime 200. Once measured per-path token mixes are materialized, the cost dimension will move to them as a versioned methodology change — that is what will make reasoning burn visible in cost.

Pricing is enforced as an invariant, not an aspiration: the scheduler refuses to plan new measurements for any path without a current declared price, a pricing gap surfaces loudly on every operations cycle until the catalogue or an operator resolves it, and operator-recorded prices are flagged for re-verification after 30 days.

Dedicated, batch and cached-input pricing are recorded when published but do not enter the public serverless ranking.

Scoring

The overall score is a weighted sum of market-normalized metric values under the active workload profile, shaded by measurement confidence. The weights below are read from the exact configuration the engine executes — the General profile is the homepage default; each workload has its own published profile.

Normalization runs against a fixed reference population — every measured path in the index — using the market's winsorized band: each dimension's range is the reference population's p5–p95, so a single outlier path clamps to the floor or ceiling instead of defining the whole scale. Published minimum spans per dimension (reliability 5 pp, tool-calling 5 pp, structured output 10 pp, latency 1,000 ms of end-to-end time, throughput 50 tok/s, effective workload cost $0.0002 per benchmarked exchange) keep a saturated market from amplifying sampling noise; the span anchors at the good extreme and extends only the weak side. Within the band, values map linearly onto 55–100 — the weak edge reads 'weak', not 'zero'. Only measured values shape a range; an unmeasured field never injects a fixture number into the reference. Weights renormalize over the dimensions actually measured for a path. An optimization tilt shifts 12 points of weight toward the user's selected priority; confidence applies the factors in the Sampling section.

Provider aggregation

A provider aggregate that averages path scores inherits the provider's catalogue: a lab serving only its own expensive reasoning model would get that model's properties billed as provider quality. The provider ranking's Overall is therefore Serving Quality — the provider's effect measured within the same model, which is the only place a provider effect is identifiable. These are the rules the engine runs:

• Serving Quality: for every model measured on two or more providers, each serving dimension (reliability, latency as the completed exchange, throughput, tool calling, structured output) normalizes inside that model's own cross-provider band — same 55-floor grammar and published minimum spans as path scores. Price never enters (see Value below); the composite quality dimension never enters. A provider's Overall is the mean of its within-model effects across shared models × confidence factor × freshness factor.
• Sole servers: a provider whose models nobody else serves (currently the frontier labs) has no counterfactual — its serving effect is unidentifiable. The row is flagged non-comparable and is never ranked on this axis; its paths remain fully measured, priced and comparable in Model × Provider mode.
• Coverage: displayed on the row (models benchmarked vs the index's best coverage) and multiplies nothing. Cherry-picking resistance comes from the within-model design itself: every shared model the provider serves enters its mean.
• Value: Serving Quality per effective dollar of the use-case's benchmarked exchange (workload input tokens + the normalized 400-token response, both prices USD-converted at the ECB reference rate). Shown as a percentile across comparable providers; exact prices live in Model × Provider mode.
• quality metrics (reliability, tools, structured): mean across the provider's included paths; TTFT, P95, E2E, prices: medians; throughput: mean;
• freshness factor = 1.0 (≤30 min), 0.995 (≤60 min), 0.99 (older); confidence factor = 1.0 / 0.988 / 0.968 (high / medium / low).

Sampling & confidence

Every path carries a run count and a confidence level over the rolling 7-day window. Confidence shades the score multiplicatively and gates winner eligibility.

Confidence levels map to eligible results in the rolling 7-day window: high at 200 or more, medium at 50 or more, low below 50 — computed per path from that path's own window evidence, never inherited from the provider's other models. Run counts include only canonical, finished runs. Below minimum window samples a statistic is not published at all: p50 needs 10 samples, p95 needs 30, p99 needs 100, rates need a 12-result denominator (two full sweeps of the smallest suite — the excluded failure mode is single-sweep publication), throughput needs 10 — the surface renders '—' instead of a number whose variance would swamp its meaning. Measured paths report real counts; in zero-fixture production an unmeasured pair is excluded entirely, while development fixtures stay labelled DEV DATA.

Outlier handling

Bad results are never silently deleted. Provider-attributable failures stay in reliability. Latency samples are winsorized at the 99th percentile within each path's window before p50/p95 computation — extreme values are capped, not removed. The stored p99 deliberately reads the uncapped tail (the p99 of a p99-winsorized list is the cap itself). Every exclusion under the client-fault rules of the Reliability section is logged with a reason.

Geography

'Europe' is not one property. InferenceBench tracks six distinct geographic dimensions per path and never collapses them:

Provider HQ

Where the operating company is headquartered.

Ownership region

Region of the controlling entity — an EU endpoint of a US-owned provider is not EU-owned.

Endpoint region

Where the inference request is processed.

Benchmark client region

Where the measurement originated (currently Falkenstein, DE).

Data residency

Where the provider guarantees request data remains.

Legal jurisdiction

The law governing the service contract.

Freshness & scheduling

Scheduling is tiered and published: health suites run daily on every measured path; correctness suites run daily on hosted-model paths and weekly on frontier-lab paths (a cost decision, stated as such); heavy suites run weekly. The stalest due (path, suite) pairs are enqueued first on every operations cycle, and detected provider changes (price edits, new endpoints, new model ids) surface as events the same day.

Two hard guards bound every cycle: a daily spend ceiling checked before any billable call, and the never-unpriced invariant — a path without a current declared price is not scheduled at all.

Every surface shows freshness: global ('updated 4m ago'), provider ('last benchmark 8m ago') and path level (run count and last-run age in the score breakdown).

Fairness

The following principles are structural, not aspirational — the pipeline has no code path that lets any of them be violated:

• Providers cannot pay for ranking position.
• Sponsorship does not change scores.
• All providers are measured with the same methodology, workloads and runner locations.
• Provider-supplied benchmark numbers never replace InferenceBench observations.
• Corrections are versioned and published in the changelog.

Limitations

Honest limits of the current design — none of these are hidden from readers of the numbers:

• Internet variability: measurements traverse the public internet; regional network effects are partially controlled by fixed runner locations.
• Hidden routing: some providers route internally across clusters or vendors; InferenceBench observes the endpoint, not the internals.
• Hardware opacity: provider hardware is not always disclosed and is never guessed.
• Silent model updates and aliasing: providers may update weights behind a stable model name; re-benchmarks catch drift with a delay.
• Quantization differences: declared serving precision is published as a variant chip on the provider's rows; unless declared, quantization is unknown and is not inferred.
• Load and batching: provider-side batching and load vary; sampling windows smooth but cannot eliminate this.
• Cold starts: the current design measures warm behaviour after one warm-up request; cold-start latency is not yet a published metric.
• Workload representation: four workload profiles cannot represent every production pattern.
• Coverage: the measured corpus is deliberately dev-scale and grows as an explicit budget decision; a pair not yet measured is excluded from rankings rather than estimated.
• History: the 30-day score-change column stays empty until a path accumulates 30 days of measured history — it is never filled from fixtures in production.
• Transport: benchmark traffic runs over HTTP/1.1; enabling HTTP/2 would change measured latencies and will only ever arrive as a versioned methodology change.

Reproducibility

The methodology is specified so that a third party could reimplement it: raw HTTPS, streaming timestamps, the formulas above, published workload parameters and the scoring configuration. A public runner CLI is planned but not yet available — no claim of current public tooling is made.

# planned public runner (not yet available)
inferencebench run --model deepseek-v3 --provider nebius --suite general

Versioning

Every observation and score records its methodology version. Historical rankings remain reconstructable under the methodology that produced them; a methodology change never silently rewrites the past. Research articles pin both a methodology version and a dataset snapshot.

Commercial independence

InferenceBench is funded through commercial partnerships that do not influence benchmark results. Partners support the engineering, infrastructure and research behind the benchmark; they cannot purchase ranking position, influence methodology, change scores, suppress results or control editorial conclusions.

• Scores, rankings, winners and badges are computed by the published pipeline — there is no commercial input anywhere in it.
• Partner identity lives on separate commercial surfaces (the partner rail, /partners); it never appears inside benchmark tables, winners, the ticker or Compare.
• Research supported by a partner is disclosed on the article, and the partner has no control over methodology, provider selection, results or conclusions.
• Conflicts are handled by disclosure and versioned corrections, never by silent edits.

InferenceBench is operated by SnowStorm Solutions S.L. and may be funded through commercial partnerships and related services.

Metrics registry

One definition per metric. The benchmark table tooltips and detail pages read from this registry — no surface redefines a metric locally.

Score weights

Read directly from the scoring configuration the engine executes. The General profile is the homepage default; every workload page shows its own profile.

The deterministic pipeline that consumes these weights — market-reference normalization, optimization tilt and confidence shading — is specified in Scoring and Provider aggregation.

Changelog