Methodology

Overview

InferenceBench measures how the same model behaves on different inference infrastructure. The unit of observation is the execution path — Model × Provider × Region × Deployment configuration — because that is what an engineering team actually ships against. Provider rankings are an explicit aggregation over those paths, never the raw unit.

A benchmark is only as credible as its measurement process. This page specifies what is measured, how it is executed, how results are normalized and aggregated, what is excluded, and where the limits are.

Measurement model

Every number on InferenceBench belongs to exactly one of three classes. The class is part of the metric's definition and is shown in the metrics registry below.

Observed

Measured directly by InferenceBench runners against live provider endpoints: TTFT, p95 TTFT, throughput, reliability, tool-calling success, structured-output validity.

Declared

Taken from official provider documentation at observation time: input/output prices, regions, data residency, context length, supported models, hosting type. Declared data never overrides an observation.

Derived

Computed by InferenceBench from observed and declared inputs: the overall score, E2E latency, percentiles, ranks, 30-day changes, provider aggregation.

Entity model

Provider

The commercial platform operating inference endpoints (e.g. Fireworks, Nebius). Provider rows in the ranking are aggregates, one per provider.

Model

The logical model (e.g. DeepSeek-V3), independent of where it runs.

Model × Provider

The basic unit of observation: a specific model served by a specific provider through a specific endpoint and region, under a stated deployment configuration. Internally: ExecutionPath.

ModelProviderCombination = {
  modelId, providerId,
  regionId?, endpointId?, deploymentConfig?
}

Benchmark environment

Benchmarks run as raw HTTPS requests from dedicated runner locations — no provider SDKs, so client-library overhead cannot differ between providers. The first runner operates from Falkenstein (DE); additional client regions are planned and every observation records its client region.

Workloads

Workloads are versioned entities, not ad-hoc prompt sets. Each defines its prompt corpus size, input-length range, target output length, sampling parameters and the capabilities it exercises. Scores always name the workload profile they were computed under.

Corpus sizes are deliberately small and versioned. Statistical weight comes from repetition — every case runs on every sweep, so a path accumulates roughly 215 samples per 7-day window at daily cadence, which crosses the high-confidence threshold below. Suites are hash-locked: changing any case without a version bump makes the loader refuse to run. Corpus growth is a funded-capacity decision and arrives as new suite versions, never as silent edits.

TTFT

TTFT includes DNS/TLS (amortized by connection reuse), provider queueing, prompt processing and the first token's generation. It excludes benchmark-client processing. The table shows p50 as the primary metric; p95 is reported alongside because tail latency determines how a provider feels under production load. Percentiles are computed over the rolling sampling window per path, never pooled across models.

End-to-end latency

End-to-end latency is only comparable when output length is controlled, so InferenceBench normalizes it to a 400-output-token response using each path's observed sustained throughput. It is a derived metric and is labelled as such.

Throughput

Measured between the first and last output token — TTFT is explicitly excluded, so the metric captures generation speed rather than queueing. Unit: tokens per second.

Reliability

Retries are disabled in the runner, so reliability reflects first-attempt behaviour — what an application without aggressive retry logic would experience.

Structured output

Structured output is not reduced to 'the JSON parses'. Each run is validated in four stages; a run only passes if every applicable stage passes.

JSON validity

The output parses as JSON.

Schema compliance

The parsed object validates against the requested JSON schema.

Constraint compliance

Stated constraints hold (enums, ranges, required fields, formats).

Semantic correctness

Where the workload defines ground truth, values are checked against it.

Tool calling

Tool-calling runs present a tool inventory and a task. A run passes only when all of the following hold:

• the correct tool is selected (including a correct no-tool decision when no tool applies);
• the argument object validates against the tool's schema;
• all required arguments are present;
• no hallucinated arguments are added;
• multi-tool tasks execute the expected call sequence.

Pricing

Input and output prices are tracked separately, in USD per million tokens, from official provider documentation for the public serverless tier. Cost efficiency inside the score uses the effective cost of the workload actually benchmarked:

Dedicated, batch and cached-input pricing are recorded when published but do not enter the public serverless ranking.

Scoring

The overall score is a weighted sum of normalized metric values under the active workload profile, adjusted by deployment fit and measurement confidence. The weights below are read from the exact configuration the engine executes — the General profile is the homepage default; each workload has its own published profile.

Normalization is min–max within the filtered candidate set with a floor of 55, so the weakest candidate in scope reads 'weak', not 'zero', and the score stays contextual to the query. An optimization tilt shifts 12 points of weight toward the user's selected priority. Deployment fit applies a 1.5% penalty when execution leaves the user's region; confidence applies the factors in the Sampling section.

Provider aggregation

A provider ranking must not reward cherry-picking (three excellent models) or punish coverage (seventy-five mixed models). Provider rows aggregate the provider's included paths as follows — these constants are the ones the engine runs:

• quality metrics (reliability, tools, structured): mean across the provider's included paths — path scores already carry per-provider confidence;
• TTFT, P95, E2E, prices: medians, so a single expensive or slow model cannot dominate;
• provider score = mean path score × coverage factor × freshness factor;
• coverage factor = 0.92 + 0.08 × (models benchmarked / best coverage in scope);
• freshness factor = 1.0 (≤30 min), 0.995 (≤60 min), 0.99 (older).

Sampling & confidence

Every path carries a run count and a confidence level over the rolling window. Confidence shades the score multiplicatively and gates winner eligibility.

Confidence levels map to eligible results in the rolling window: high at 200 or more, medium at 50 or more, low below 50. Run counts are shown in score breakdowns and tooltips. Measured paths report real counts; fixture rows remain labelled DEV DATA until measured.

Outlier handling

Bad results are never silently deleted. Provider-attributable failures stay in reliability. Latency samples are winsorized at the 99th percentile within each path's window before percentile computation — extreme values are capped, not removed — and every exclusion under the client-fault rules of the Reliability section is logged with a reason.

Geography

'Europe' is not one property. InferenceBench tracks six distinct geographic dimensions per path and never collapses them:

Provider HQ

Where the operating company is headquartered.

Ownership region

Region of the controlling entity — an EU endpoint of a US-owned provider is not EU-owned.

Endpoint region

Where the inference request is processed.

Benchmark client region

Where the measurement originated (currently Falkenstein, DE).

Data residency

Where the provider guarantees request data remains.

Legal jurisdiction

The law governing the service contract.

Freshness & scheduling

Benchmarks are scheduled adaptively rather than on a flat cycle. Priority rises with observation age, metric volatility, path popularity, confidence deficit and detected provider changes (price edits, new endpoints, version bumps). Exact scheduler parameters are withheld to reduce gaming; the principles are public.

Every surface shows freshness: global ('Updated 12m ago'), provider ('last benchmark 8m ago') and path level ('126 runs · last run 4m ago'). Data ages through fresh → aging → stale states.

Fairness

The following principles are structural, not aspirational — the pipeline has no code path that lets any of them be violated:

• Providers cannot pay for ranking position.
• Sponsorship does not change scores.
• All providers are measured with the same methodology, workloads and runner locations.
• Provider-supplied benchmark numbers never replace InferenceBench observations.
• Corrections are versioned and published in the changelog.

Limitations

Honest limits of the current design — none of these are hidden from readers of the numbers:

• Internet variability: measurements traverse the public internet; regional network effects are partially controlled by fixed runner locations.
• Hidden routing: some providers route internally across clusters or vendors; InferenceBench observes the endpoint, not the internals.
• Hardware opacity: provider hardware is not always disclosed and is never guessed.
• Silent model updates and aliasing: providers may update weights behind a stable model name; re-benchmarks catch drift with a delay.
• Quantization differences: unless declared, quantization is unknown and is not inferred.
• Load and batching: provider-side batching and load vary; sampling windows smooth but cannot eliminate this.
• Cold starts: the current design measures warm behaviour after one warm-up request; cold-start latency is not yet a published metric.
• Workload representation: four workload profiles cannot represent every production pattern.
• Development fixtures: until the measurement engine ships, all published figures are deterministic fixtures labelled DEV DATA.

Reproducibility

The methodology is specified so that a third party could reimplement it: raw HTTPS, streaming timestamps, the formulas above, published workload parameters and the scoring configuration. A public runner CLI is planned but not yet available — no claim of current public tooling is made.

# planned public runner (not yet available)
inferencebench run --model deepseek-v3 --provider nebius --suite general

Versioning

Every observation and score records its methodology version. Historical rankings remain reconstructable under the methodology that produced them; a methodology change never silently rewrites the past. Research articles pin both a methodology version and a dataset snapshot.

Commercial independence

InferenceBench is funded through commercial partnerships that do not influence benchmark results. Partners support the engineering, infrastructure and research behind the benchmark; they cannot purchase ranking position, influence methodology, change scores, suppress results or control editorial conclusions.

• Scores, rankings, winners and badges are computed by the published pipeline — there is no commercial input anywhere in it.
• Partner identity lives on separate commercial surfaces (the partner rail, /partners); it never appears inside benchmark tables, winners, the ticker or Compare.
• Research supported by a partner is disclosed on the article, and the partner has no control over methodology, provider selection, results or conclusions.
• Conflicts are handled by disclosure and versioned corrections, never by silent edits.

InferenceBench is operated by SnowStorm Solutions S.L. and may be funded through commercial partnerships and related services.

Metrics registry

One definition per metric. The benchmark table tooltips and detail pages read from this registry — no surface redefines a metric locally.

Score weights

Read directly from the scoring configuration the engine executes. The General profile is the homepage default; every workload page shows its own profile.

The deterministic pipeline that consumes these weights — normalization floor, optimization tilt, deployment fit and confidence shading — is specified in Scoring and Provider aggregation.

Changelog