Skip to content

Metrics

GuideLLM provides a comprehensive set of metrics to evaluate and optimize the performance of large language model (LLM) deployments. These metrics are designed to help users understand the behavior of their models under various conditions, identify bottlenecks, and make informed decisions about scaling and resource allocation. Below, we outline the key metrics measured by GuideLLM, their definitions, use cases, and how they can be interpreted.

Request Status Metrics

Successful, Incomplete, and Error Requests

  • Successful Requests: The number of requests that were completed successfully without any errors.
  • Incomplete Requests: The number of requests that were started but not completed, often due to timeouts or interruptions.
  • Error Requests: The number of requests that failed due to errors, such as invalid inputs or server issues.

These metrics provide a breakdown of the overall request statuses, helping users identify the reliability and stability of their LLM deployment.

Requests Made

  • Definition: The total number of requests made during a benchmark run, broken down by status (successful, incomplete, error).
  • Use Case: Helps gauge the workload handled by the system and identify the proportion of requests that were successful versus those that failed or were incomplete.

Token Metrics

Prompt Tokens and Counts

  • Definition: The number of tokens in the input prompts sent to the LLM.
  • Use Case: Useful for understanding the complexity of the input data and its impact on model performance.

Output Tokens and Counts

  • Definition: The number of tokens generated by the LLM in response to the input prompts.
  • Use Case: Helps evaluate the model's output length and its correlation with latency and resource usage.

Performance Metrics

Request Rate (Requests Per Second)

  • Definition: The number of requests processed per second.
  • Use Case: Indicates the throughput of the system and its ability to handle concurrent workloads.

Request Concurrency

  • Definition: The number of requests being processed simultaneously.
  • Use Case: Helps evaluate the system's capacity to handle parallel workloads.

SLO Attainment

  • Definition: The fraction of requests meeting every configured latency objective. Reported only when objectives are set through --metrics. Errored requests count as non-conforming. Requests cancelled at the run's duration limit, and requests whose objectives cannot be evaluated (such as time to first token on a non-streaming backend), are excluded from both the numerator and the denominator.
  • Use Case: States directly whether a deployment meets a target such as "TTFT under 200ms for 99% of requests". Because it is a ratio rather than a rate, it does not change with the length of the measurement window.

Request Goodput

  • Definition: The number of objective-conforming requests completed per second. Requests that complete but breach an objective count toward request rate and not toward goodput, so goodput is always at or below the request rate.
  • Use Case: Separates useful capacity from raw capacity. Past a server's saturation point the request rate can stay flat while goodput falls, because requests still complete but no longer complete quickly enough to be useful.

Output Tokens Per Second

  • Definition: The average number of output tokens generated per second as a throughput metric across all requests.
  • Use Case: Provides insights into the server's performance and efficiency in generating output tokens.

Total Tokens Per Second

  • Definition: The combined rate of prompt and output tokens processed per second as a throughput metric across all requests.
  • Use Case: Provides insights into the server's overall performance and efficiency in processing both prompt and output tokens.

Request Latency

  • Definition: The time taken to process a single request, from start to finish.
  • Use Case: A critical metric for evaluating the responsiveness of the system.

Dispatch Delay

  • Definition: request_dispatch_delay is request_start - targeted_start: the time between when a request was scheduled to arrive and when it was actually sent.
  • Use Case: Reveals whether the benchmark itself kept up with the configured arrival schedule. A request cannot be dispatched while the concurrency limit is saturated, and that wait is not part of Request Latency. A delay near zero means the reported latencies reflect the full picture; a large delay means the load generator fell behind and the server saw a lower arrival rate than was requested.

Scheduled Latency

  • Definition: request_scheduled_latency is request_end - targeted_start: request latency measured from the scheduled arrival time rather than from dispatch. When all three timestamps are present it equals Dispatch Delay plus Request Latency.
  • Use Case: Describes what a client holding to the configured arrival schedule would have experienced, including time spent waiting to be dispatched. When the benchmark keeps up this matches Request Latency; when it falls behind, the gap between the two is latency that Request Latency alone does not show.

Dispatch Delay and Scheduled Latency only apply to some of the scheduling strategies. See Applicability of Dispatch Delay and Scheduled Latency below.

Time to First Token (TTFT)

  • Definition: The time taken to generate the first token of the output.
  • Use Case: Indicates the initial response time of the model, which is crucial for user-facing applications.

Inter-Token Latency (ITL)

  • Definition: The average time between generating consecutive tokens in the output, excluding the first token.
  • Use Case: Helps assess the smoothness and speed of token generation.

Time Per Output Token

  • Definition: The average time taken to generate each output token, including the first token.
  • Use Case: Provides a detailed view of the model's token generation efficiency.

Time To Last Round Trip

  • Definition: For the realtime WebSocket backend (openai_websocket), the time from the last sent packet to the last received token.
  • Use Case: Measures tail latency of a streaming exchange (how long the final output lags the final input).

Average Round-Trip Time (Avg RTT)

  • Definition: For the WebSocket backend, the mean of received-token timestamps minus the mean of sent-packet timestamps.
  • Use Case: Estimates the average send-to-receive lag across a request. It is approximate, since it assumes sent packets and received tokens line up evenly in time.

Measurement Window, Warmup, and Cooldown

Benchmark profiles can configure warmup and cooldown periods that bracket the active measurement window. Requests sent during warmup still run to completion, but reported metrics are scoped to the interval between measure_start (warmup ends) and measure_end (cooldown begins).

Not every metric applies that window the same way:

  • Request-level metrics: request totals, request latency, concurrency, and per-request token counts, etc. will include any request whose lifetime overlaps the measurement window, even when the request started during warmup or finished during cooldown.
  • Event-level metrics: TTFT, time to first output token (TTFOT), inter-token latency (ITL), and per-token throughput rates will include only events whose timestamps fall inside the window. A request can therefore appear in request totals while some or all of its token events are excluded.

For token latencies specifically, GuideLLM filters on when the event occurs rather than on the whole request span:

  • TTFT is included when the interval from request start to first token partially overlaps the measurement window. A request that starts during warmup but streams its first token after warmup ends contributes a TTFT sample; a request whose first token arrives before measure_start does not, even if the request completes during the active phase.
  • TTFOT uses the first content token timestamp (first_output_token_iteration), which can differ from TTFT when reasoning or tool tokens precede visible output.
  • ITL is included when the span from first token through request completion partially overlaps the window.

The purpose of warmup is meant to let the system reach steady state before measurement and event-level filtering keeps pre-warmup token timing from skewing latency and throughput figures while still counting long-running requests that cross the boundary.

Statistical Summaries

GuideLLM provides detailed statistical summaries for each of the above metrics using the StatusDistributionSummary and DistributionSummary models. These summaries include the following statistics:

Summary Statistics

  • Mean: The average value of the metric.
  • Median: The middle value of the metric when sorted.
  • Mode: The most frequently occurring value of the metric.
  • Variance: The measure of how much the values of the metric vary.
  • Standard Deviation (Std Dev): The square root of the variance, indicating the spread of the values.
  • Min: The minimum value of the metric.
  • Max: The maximum value of the metric.
  • Count: The total number of data points for the metric.
  • Total Sum: The sum of all values for the metric.

Percentiles

GuideLLM calculates a comprehensive set of percentiles for each metric, including:

  • 0.1th Percentile (p001): The value below which 0.1% of the data falls.
  • 1st Percentile (p01): The value below which 1% of the data falls.
  • 5th Percentile (p05): The value below which 5% of the data falls.
  • 10th Percentile (p10): The value below which 10% of the data falls.
  • 25th Percentile (p25): The value below which 25% of the data falls.
  • 75th Percentile (p75): The value below which 75% of the data falls.
  • 90th Percentile (p90): The value below which 90% of the data falls.
  • 95th Percentile (p95): The value below which 95% of the data falls.
  • 99th Percentile (p99): The value below which 99% of the data falls.
  • 99.9th Percentile (p999): The value below which 99.9% of the data falls.

Use Cases for Statistical Summaries

  • Mean and Median: Provide a central tendency of the metric values.
  • Variance and Std Dev: Indicate the variability and consistency of the metric.
  • Min and Max: Highlight the range of the metric values.
  • Percentiles: Offer a detailed view of the distribution, helping identify outliers and performance at different levels of service.

By combining these metrics and statistical summaries, GuideLLM enables users to gain a deep understanding of their LLM deployments, optimize performance, and ensure scalability and cost-effectiveness.

Applicability of Dispatch Delay and Scheduled Latency

Both metrics are derived from targeted_start, so they only describe arrival-schedule delay for strategies that define an arrival schedule:

  • constant and poisson derive each target from the configured rate.
  • trace derives each target from the replayed dataset timestamps. Trace requests that carry no relative timestamp fall back to the benchmark start time and take on the throughput caveat below.

The synchronous, concurrent, and throughput strategies set an ASAP-style target instead, and each is unusable for a different reason:

  • synchronous and concurrent target the previous request's completion, apart from the requests staggered across a configured rampup. The target is therefore derived from the system's own responses, so a delay measured against it is circular: it describes harness turnaround rather than lag against an arrival schedule.
  • throughput targets the benchmark start time for every request, so the value grows with elapsed run time and is not a delay at all.

For those three strategies both metrics are reported as null rather than as zero, and their columns do not appear in the CSV. Reporting zero would read as "no delay measured", which is a stronger and more misleading claim than "not applicable". This matters for the default sweep profile, which runs synchronous and throughput alongside its rate-based strategies.

For multi-turn conversation datasets, a turn's target is fixed when a scheduler slot opens rather than when the preceding turn completes, so configured think time between turns is counted as Dispatch Delay.

Where the metrics do apply they are recorded in the serialized report and the CSV output for offline analysis. They are omitted from the final console latency table in all cases, since that table is shared across every profile in a run.