Skip to content

Metrics

GuideLLM provides a comprehensive set of metrics to evaluate and optimize the performance of large language model (LLM) deployments. These metrics are designed to help users understand the behavior of their models under various conditions, identify bottlenecks, and make informed decisions about scaling and resource allocation. Below, we outline the key metrics measured by GuideLLM, their definitions, use cases, and how they can be interpreted.

Request Status Metrics

Successful, Incomplete, and Error Requests

  • Successful Requests: The number of requests that were completed successfully without any errors.
  • Incomplete Requests: The number of requests that were started but not completed, often due to timeouts or interruptions.
  • Error Requests: The number of requests that failed due to errors, such as invalid inputs or server issues.

These metrics provide a breakdown of the overall request statuses, helping users identify the reliability and stability of their LLM deployment.

Requests Made

  • Definition: The total number of requests made during a benchmark run, broken down by status (successful, incomplete, error).
  • Use Case: Helps gauge the workload handled by the system and identify the proportion of requests that were successful versus those that failed or were incomplete.

Token Metrics

Prompt Tokens and Counts

  • Definition: The number of tokens in the input prompts sent to the LLM.
  • Use Case: Useful for understanding the complexity of the input data and its impact on model performance.

Output Tokens and Counts

  • Definition: The number of tokens generated by the LLM in response to the input prompts.
  • Use Case: Helps evaluate the model's output length and its correlation with latency and resource usage.

Performance Metrics

Request Rate (Requests Per Second)

  • Definition: The number of requests processed per second.
  • Use Case: Indicates the throughput of the system and its ability to handle concurrent workloads.

Request Concurrency

  • Definition: The number of requests being processed simultaneously.
  • Use Case: Helps evaluate the system's capacity to handle parallel workloads.

SLO Attainment

  • Definition: The fraction of requests meeting every configured latency objective. Reported only when objectives are set through --metrics. Errored requests count as non-conforming. Requests cancelled at the run's duration limit, and requests whose objectives cannot be evaluated (such as time to first token on a non-streaming backend), are excluded from both the numerator and the denominator.
  • Use Case: States directly whether a deployment meets a target such as "TTFT under 200ms for 99% of requests". Because it is a ratio rather than a rate, it does not change with the length of the measurement window.

Request Goodput

  • Definition: The number of objective-conforming requests completed per second. Requests that complete but breach an objective count toward request rate and not toward goodput, so goodput is always at or below the request rate.
  • Use Case: Separates useful capacity from raw capacity. Past a server's saturation point the request rate can stay flat while goodput falls, because requests still complete but no longer complete quickly enough to be useful.

Output Tokens Per Second

  • Definition: The average number of output tokens generated per second as a throughput metric across all requests.
  • Use Case: Provides insights into the server's performance and efficiency in generating output tokens.

Total Tokens Per Second

  • Definition: The combined rate of prompt and output tokens processed per second as a throughput metric across all requests.
  • Use Case: Provides insights into the server's overall performance and efficiency in processing both prompt and output tokens.

Request Latency

  • Definition: The time taken to process a single request, from start to finish.
  • Use Case: A critical metric for evaluating the responsiveness of the system.

Generation Delay

  • Definition: generation_delay is the time between the scheduler beginning to produce the next conversation from requests_generator and creating that conversation's request infos. It is recorded once per conversation on scheduler state, not on individual requests.
  • Use Case: Shows how long the load generator spends materializing and enqueueing each conversation (dataset iteration plus coordinator prep). A delay near zero means request production is keeping up; a large delay means the dataset or enqueue path is the bottleneck, independent of backend latency and dispatch scheduling.

Dispatch Delay

  • Definition: request_dispatch_delay is request_start - targeted_start: the time between when a request was scheduled to arrive and when it was actually sent.
  • Use Case: Reveals whether the benchmark itself kept up with the configured arrival schedule. A request cannot be dispatched while the concurrency limit is saturated, and that wait is not part of Request Latency. A delay near zero means the reported latencies reflect the full picture; a large delay means the load generator fell behind and the server saw a lower arrival rate than was requested. Dispatch Delay does not say why the request was late. Turn Predecessor Delay and Turn Scheduling Delay split that wait.

Turn Predecessor Delay

  • Definition: turn_predecessor_delay is the part of Dispatch Delay spent waiting because a prior turn was still running after this request's targeted start. It is zero when the request has no predecessor, or the predecessor finished before the target. When both happen, the wait until the predecessor finishes is Turn Predecessor Delay and any further wait is Turn Scheduling Delay. The two sum to Dispatch Delay.
  • Use Case: Shows that the previous turn, not the load generator, kept this request from starting on time.

Turn Scheduling Delay

  • Definition: turn_scheduling_delay is Dispatch Delay minus Turn Predecessor Delay. It is the wait after the request was both due and no longer blocked by a predecessor, including time to deserialize the trace, time spent queued, and worker turnaround. Think time (requeue_delay) that runs past the target is included here. Trace replay does not set think time.
  • Use Case: Can indicate when GuideLLM could not keep up or the machine was undersized for the benchmark.

Scheduled Latency

  • Definition: request_scheduled_latency is request_end - targeted_start: request latency measured from the scheduled arrival time rather than from dispatch. When all three timestamps are present it equals Dispatch Delay plus Request Latency.
  • Use Case: Describes what a client holding to the configured arrival schedule would have experienced, including time spent waiting to be dispatched. When the benchmark keeps up this matches Request Latency; when it falls behind, the gap between the two is latency that Request Latency alone does not show.

Dispatch Delay, Turn Predecessor Delay, Turn Scheduling Delay, and Scheduled Latency only apply to some of the scheduling strategies. See Applicability of Dispatch Delay and Scheduled Latency below. In relative trace replay, targeted_start moves with the prior turn, so these delays are measured against that shifted target rather than the original trace clock.

Time to First Token (TTFT)

  • Definition: The time taken to generate the first token of the output.
  • Use Case: Indicates the initial response time of the model, which is crucial for user-facing applications.

Inter-Token Latency (ITL)

  • Definition: The average time between generating consecutive tokens in the output, excluding the first token.
  • Use Case: Helps assess the smoothness and speed of token generation.

Time Per Output Token

  • Definition: The average time taken to generate each output token, including the first token.
  • Use Case: Provides a detailed view of the model's token generation efficiency.

Time To Last Round Trip

  • Definition: For the realtime WebSocket backend (openai_websocket), the time from the last sent packet to the last received token.
  • Use Case: Measures tail latency of a streaming exchange (how long the final output lags the final input).

Average Round-Trip Time (Avg RTT)

  • Definition: For the WebSocket backend, the mean of received-token timestamps minus the mean of sent-packet timestamps.
  • Use Case: Estimates the average send-to-receive lag across a request. It is approximate, since it assumes sent packets and received tokens line up evenly in time.

Measurement Window, Warmup, and Cooldown

Benchmark profiles can configure warmup and cooldown periods that bracket the active measurement window. Requests sent during warmup still run to completion, but reported metrics are scoped to the interval between measure_start (warmup ends) and measure_end (cooldown begins).

Not every metric applies that window the same way:

  • Request-level metrics: request totals, request latency, concurrency, and per-request token counts, etc. will include any request whose lifetime overlaps the measurement window, even when the request started during warmup or finished during cooldown.
  • Event-level metrics: TTFT, time to first output token (TTFOT), inter-token latency (ITL), and per-token throughput rates will include only events whose timestamps fall inside the window. A request can therefore appear in request totals while some or all of its token events are excluded.

For token latencies specifically, GuideLLM filters on when the event occurs rather than on the whole request span:

  • TTFT is included when the interval from request start to first token partially overlaps the measurement window. A request that starts during warmup but streams its first token after warmup ends contributes a TTFT sample; a request whose first token arrives before measure_start does not, even if the request completes during the active phase.
  • TTFOT uses the first content token timestamp (first_output_token_iteration), which can differ from TTFT when reasoning or tool tokens precede visible output.
  • ITL is included when the span from first token through request completion partially overlaps the window.

The purpose of warmup is meant to let the system reach steady state before measurement and event-level filtering keeps pre-warmup token timing from skewing latency and throughput figures while still counting long-running requests that cross the boundary.

Statistical Summaries

GuideLLM provides detailed statistical summaries for each of the above metrics using the StatusDistributionSummary and DistributionSummary models. These summaries include the following statistics:

Summary Statistics

  • Mean: The average value of the metric.
  • Median: The middle value of the metric when sorted.
  • Mode: The most frequently occurring value of the metric.
  • Variance: The measure of how much the values of the metric vary.
  • Standard Deviation (Std Dev): The square root of the variance, indicating the spread of the values.
  • Min: The minimum value of the metric.
  • Max: The maximum value of the metric.
  • Count: The total number of data points for the metric.
  • Total Sum: The sum of all values for the metric.

Percentiles

GuideLLM calculates a comprehensive set of percentiles for each metric, including:

  • 0.1th Percentile (p001): The value below which 0.1% of the data falls.
  • 1st Percentile (p01): The value below which 1% of the data falls.
  • 5th Percentile (p05): The value below which 5% of the data falls.
  • 10th Percentile (p10): The value below which 10% of the data falls.
  • 25th Percentile (p25): The value below which 25% of the data falls.
  • 75th Percentile (p75): The value below which 75% of the data falls.
  • 90th Percentile (p90): The value below which 90% of the data falls.
  • 95th Percentile (p95): The value below which 95% of the data falls.
  • 99th Percentile (p99): The value below which 99% of the data falls.
  • 99.9th Percentile (p999): The value below which 99.9% of the data falls.

Confidence Intervals

Each summary statistic above is an estimate made from a finite number of requests, and GuideLLM reports how precisely each one was measured alongside the value itself.

  • mean_ci: A two-sided confidence interval for the mean.
  • percentile_cis: A confidence interval for each reported percentile, or null where the run is too short to place one.

Set the level with --metrics kind=generative,confidence=0.95. It defaults to 0.95, and null reports the metrics without intervals. The level is recorded once per benchmark as config.confidence.

The console shows the mean and the half-width of its interval together, for example 80.5 ± 3.9, and marks a percentile the sample cannot bound with *. The CSV appends a Mean CI and a Percentile CIs column for each metric that carries intervals, the latter keyed by percentile, plus the confidence level. Those columns go at the end of each row so that existing column positions are unchanged.

Which metrics report intervals

Intervals are reported only for metrics recorded exactly once per request, where each value is one observation: request latency, TTFT, time to first output token, the round-trip metrics, the token counts, streaming iteration count, dispatch delay, and scheduled latency.

Two groups are reported without intervals:

  • Inter-token latency and time per output token. Their reported mean is weighted by output tokens, so it is a ratio of total generation time to total tokens rather than a mean over interchangeable observations. Note that their count is the summed token exposure, not the number of requests.
  • Rate distributions, such as requests per second and output tokens per second. These are derived from event timings rather than per-request observations.

Why a percentile interval can be absent

The reported percentiles are order statistics: p99 is the value at rank ceil(0.99n) in the sorted sample. A two-sided interval needs observations on both sides of that rank, which requires at least ceil(log(alpha / 2) / log(q)) requests. At the default 0.95 confidence that is 72 requests for p95, 368 for p99 and 3688 for p999.

Below those sizes the interval is reported as null. This is worth noticing: with fewer than 100 successful requests, the reported p99 is the single slowest request in the run, and with fewer than 1000 the reported p999 is as well.

What the interval does and does not say

The interval describes how precisely this run measured its own conditions, treating the requests within it as independent observations. It does not describe how far the value would move if the same benchmark were run again. Those two differ when successive requests share a condition that varies between runs, such as the server's load during generation, so a run-to-run comparison needs repeated runs rather than a single run's interval.

Use Cases for Statistical Summaries

  • Mean and Median: Provide a central tendency of the metric values.
  • Variance and Std Dev: Indicate the variability and consistency of the metric.
  • Min and Max: Highlight the range of the metric values.
  • Percentiles: Offer a detailed view of the distribution, helping identify outliers and performance at different levels of service.

By combining these metrics and statistical summaries, GuideLLM enables users to gain a deep understanding of their LLM deployments, optimize performance, and ensure scalability and cost-effectiveness.

Applicability of Dispatch Delay and Scheduled Latency

Dispatch Delay, Turn Predecessor Delay, Turn Scheduling Delay, and Scheduled Latency are derived from targeted_start, so they only describe arrival-schedule delay for strategies that define an arrival schedule:

  • constant and poisson derive each target from the configured rate.
  • trace derives each target from the replayed dataset timestamps. Trace requests that carry no relative timestamp fall back to the benchmark start time and take on the throughput caveat below.

The synchronous, concurrent, and throughput strategies set an ASAP-style target instead, and each is unusable for a different reason:

  • synchronous and concurrent target the previous request's completion, apart from the requests staggered across a configured rampup. The target is therefore derived from the system's own responses, so a delay measured against it is circular: it describes harness turnaround rather than lag against an arrival schedule.
  • throughput targets the benchmark start time for every request, so the value grows with elapsed run time and is not a delay at all.

For those three strategies these metrics are reported as null rather than as zero, and their columns do not appear in the CSV. Reporting zero would read as "no delay measured", which is a stronger and more misleading claim than "not applicable". This matters for the default sweep profile, which runs synchronous and throughput alongside its rate-based strategies.

For multi-turn conversation datasets, a turn's target is fixed when a scheduler slot opens rather than when the preceding turn completes, so configured think time between turns is counted as Dispatch Delay and as Turn Scheduling Delay.

Where the metrics do apply they are recorded in the serialized report and the CSV output for offline analysis. They are omitted from the final console latency table in all cases, since that table is shared across every profile in a run.