Skip to content

Analyze Results

After running a benchmark, GuideLLM provides comprehensive results that help you understand your LLM deployment's performance. This guide explains how to interpret both console output and file-based results.

Understanding Console Output

Upon benchmark completion, GuideLLM automatically displays results in the console, divided into three main sections:

1. Benchmarks Metadata

This section provides a high-level summary of the benchmark run, including:

  • Server configuration: Target URL, model name, and backend details
  • Data configuration: Data source, token counts, and dataset properties
  • Profile arguments: Rate type, maximum duration, request limits, etc.
  • Extras: Any additional metadata provided via the --output-extras argument

Example:

Benchmarks Metadata
------------------
Args:        {"backend_type": "openai", "target": "http://localhost:8000", "model": "Meta-Llama-3.1-8B-Instruct-quantized", ...}
Worker:      {"type_": "generative", "backend_type": "openai", "backend_args": {"timeout": 120.0, ...}, ...}
Request Loader: {"type_": "generative", "data_args": {"prompt_tokens": 256, "output_tokens": 128, ...}, ...}
Extras:      {}

2. Benchmarks Info

This section summarizes the key information about each benchmark run, presented as a table with columns:

  • Type: The benchmark type (e.g., synchronous, constant, poisson, etc.)
  • Start/End Time: When the benchmark started and ended
  • Duration: Total duration of the benchmark in seconds
  • Requests: Count of successful, incomplete, and errored requests
  • Token Stats: Average token counts and totals for prompts and outputs

This section helps you understand what was executed and provides a quick overview of the results.

3. Benchmarks Stats

This is the most critical section for performance analysis. It displays detailed statistics for each metric:

  • Throughput Metrics:

  • Requests per second (RPS)

  • Request concurrency
  • Output tokens per second

  • Total tokens per second

  • Latency Metrics:

  • Request latency (mean, median, p99)

  • Time to first token (TTFT) (mean, median, p99)
  • Inter-token latency (ITL) (mean, median, p99)
  • Time per output token (mean, median, p99)

The p99 (99th percentile) values are particularly important for SLO analysis, as they represent the worst-case performance for 99% of requests.

Analyzing Saved Results

For deeper analysis, GuideLLM saves detailed results to these files by default:

  • benchmarks.json: Complete benchmark data in JSON format
  • benchmarks.csv: Summary of key metrics in CSV format

The files are written to the directory configured by GUIDELLM__DEFAULT_RESULTS_DIR, or to the current directory when the variable is not set. See configuring file outputs to choose filenames and destinations.

File Formats

See supported file formats for the available reports and output configuration for examples of selecting them. Explicit --output options replace the default JSON and CSV selection. Console display controls are described in console output.

Programmatic Analysis

For custom analysis, you can reload the results into Python:

from guidellm.benchmark import GenerativeBenchmarksReport

# Load results from file
report = GenerativeBenchmarksReport.load_file("benchmarks.json")

# Access individual benchmarks
for benchmark in report.benchmarks:
    # Print basic info
    print(f"Benchmark: {benchmark.id_}")
    print(f"Type: {benchmark.type_}")

    # Access metrics
    print(f"Avg RPS: {benchmark.metrics.requests_per_second.successful.mean}")
    print(f"p99 latency: {benchmark.metrics.request_latency.successful.percentiles.p99}")
    print(f"TTFT (p99): {benchmark.metrics.time_to_first_token_ms.successful.percentiles.p99}")

Key Performance Indicators

When analyzing your results, focus on these key indicators:

1. Throughput and Capacity

  • Maximum RPS: What's the highest request rate your server can handle?
  • Concurrency: How many concurrent requests can your server process?
  • Token Throughput: How many tokens per second can your server generate?

2. Latency and Responsiveness

  • Time to First Token (TTFT): How quickly does the model start generating output?
  • Inter-Token Latency (ITL): How smoothly does the model generate subsequent tokens?
  • Total Request Latency: How long do complete requests take end-to-end?

3. Reliability and Error Rates

  • Success Rate: What percentage of requests completes successfully?
  • Error Distribution: What types of errors occur and at what rates?

Additional Analysis Techniques

Comparing Different Models or Hardware

Run benchmarks with different models or hardware configurations, then compare:

guidellm run \
  --backend kind=openai_http,target=http://server1:8000 \
  --data kind=synthetic_text,prompt_tokens=256,output_tokens=128 \
  --output kind=json,path=model1/benchmarks.json

guidellm run \
  --backend kind=openai_http,target=http://server2:8000 \
  --data kind=synthetic_text,prompt_tokens=256,output_tokens=128 \
  --output kind=json,path=model2/benchmarks.json

Cost Optimization

Calculate cost-effectiveness by analyzing:

  • Tokens per second per dollar of hardware cost
  • Maximum throughput for different hardware configurations
  • Optimal batch size vs. latency tradeoffs

Determining Scaling Requirements

Use your benchmark results to plan:

  • How many servers you need to handle your expected load
  • When to automatically scale up or down based on demand
  • What hardware provides the best price/performance for your workload