Architecture
GuideLLM evaluates and optimizes large language model (LLM) deployments by simulating real-world inference workloads. Its components are decoupled so that data sources, backends, scheduling strategies, and reporting can be changed independently.
Component Flow
flowchart LR
A["Configuration<br/>CLI, scenario file, or env vars"] --> B["Data pipeline<br/>conversation graphs"]
A --> P["Profile<br/>generate strategies"]
P --> G["Benchmarker<br/>run strategies and accumulate results"]
B --> C["Scheduler<br/>execute one strategy and enforce constraints"]
G --> C
C --> D["Worker processes"]
D --> E["Backend"]
G --> F["Collectors"]
E --> C
C --> G
F --> G
G --> H["GenerativeBenchmarksReport"]
H --> I["Outputs<br/>console, JSON/YAML, CSV, HTML, plot"] Core Components
1. Configuration
Benchmarks are configured through the CLI (guidellm run), one or more scenario files (--config), or GUIDELLM__SPEC__... environment variables. All three forms populate the same BenchmarkScenario schema, whose spec field is a BenchmarkArgs object containing the backend, profile, constraints, data pipeline, tokenizer, metrics, collectors, and outputs.
- Responsibilities:
- Select registered implementations with the
kind=prefix (for example,--profile kind=sweep) - Merge scenario files and command-line overrides in order
- Validate the complete configuration before any requests are sent
2. Data Pipeline
The data pipeline lives in guidellm/data and converts dataset definitions into a stream of GenerativeConversationGraph objects containing GenerationRequest nodes. Single-turn data also produces conversation graphs, with one request node each.
- Deserializers turn a source into a Hugging Face dataset. Supported kinds include
synthetic_text,synthetic_image,synthetic_video,huggingface/hf, file kinds such asjson_fileandparquet_file, in-memory kinds, and trace formats such astrace_syntheticandotel. - Preprocessors map dataset columns (
generative_column_mapper,pooling_column_mapper), encode media (encode_media), transpose turn columns (turn_pivot), and extract tool-call data from messages (tool_calling_message_extractor). - The
generativefinalizer builds request nodes and their conversation graph, including expansion of client-managed tool-call turns.conversation_graphis an internal helper module, not a selectable finalizer kind. - Data loaders (
pytorch) stream rows and applysampleslimits. - Tokenizers (
huggingface_auto) compute token counts, defaulting to the model reported by the backend.
3. Scheduler
Profiles generate scheduling strategies, and the Benchmarker iterates over them. The Scheduler in guidellm/scheduler executes one strategy at a time and enforces constraints while requests run.
- Responsibilities:
- Run the strategies generated by the configured profile (
synchronous,throughput,concurrent,constant/poisson,sweep,knee,goodput, orreplay) - Apply constraints such as
max_duration,max_requests,min_requests,max_errors,max_error_rate,max_global_error_rate, andover_saturation - Track warmup and cooldown windows that are excluded from the measured results
- Dispatch requests to worker processes and enforce turn dependencies for multiturn conversations
4. Worker Processes
Worker processes send requests concurrently and return responses plus timing information to the scheduler. They are started with spawn or forkserver multiprocessing; fork is not supported. Worker pool size and messaging behavior are controlled by GUIDELLM__... settings such as GUIDELLM__MAX_WORKER_PROCESSES.
5. Backend
The Backend is an abstract interface for interacting with inference servers and local inference engines. GuideLLM ships with OpenAI-compatible HTTP (openai_http) and WebSocket (openai_websocket) backends, an in-process vLLM Python backend (vllm_python_async, also accepted as vllm_python), and a vLLM Python batch backend (vllm_python_batch).
- Responsibilities:
- Validate backend readiness and resolve the model when it is not configured explicitly
- Format and send requests
- Parse streaming and non-streaming responses, including token usage and tool calls
6. Collectors
Collectors in guidellm/collectors gather data from outside GuideLLM while benchmarks run. The built-in prometheus collector scrapes a metrics endpoint in a separate process and attaches its summary to each benchmark result.
7. Benchmarker and Results
The Benchmarker in guidellm/benchmark runs a profile's strategies and aggregates each one with GenerativeBenchmarkAccumulator into a GenerativeBenchmark. Each benchmark contains SchedulerMetrics and GenerativeMetrics, including status breakdowns, percentile distributions, confidence intervals, SLO attainment, warnings, and collector results.
GenerativeBenchmarksReport collects the benchmarks, the merged configuration, and any profile conclusions (for example, knee or goodput analysis), and writes the requested outputs from guidellm/benchmark/outputs: console, CSV, HTML, plot, and JSON/YAML.
8. Progress and Warnings
guidellm/benchmark/progress.py reports setup and benchmark progress to the console and logs. After each benchmark, warning rules from guidellm/schemas/benchmark/warnings.py are evaluated against the compiled metrics and recorded in the report; warnings never stop the run.
Package Layout
| Package | Responsibility |
|---|---|
guidellm.cli | Click-based commands for run, preprocess, export, and more |
guidellm.schemas | Pydantic argument, request, response, metric, and report schemas |
guidellm.data | Deserializers, preprocessors, finalizers, loaders, and tokenizers |
guidellm.backends | Backend implementations and request handlers |
guidellm.scheduler | Strategies, constraints, worker processes, and DAG execution |
guidellm.collectors | Collectors that gather external data during a benchmark |
guidellm.benchmark | Benchmark orchestration, metrics, progress, warnings, and outputs |