# agentic-api Roadmap `agentic-api`, also referred to as vLLM Agentic API, is the stateful agentic API layer for [vLLM core](https://github.com/vllm-project/vllm). It is optimized for vLLM core and implemented in Rust. Its job is to own stateful agentic APIs, including Responses and Messages, and to orchestrate server-side tool execution for tool calls generated by vLLM core. This roadmap uses: - **vLLM core** for [`vllm-project/vllm`](https://github.com/vllm-project/vllm), the inference engine and model-serving project. - **vLLM Agentic API** for this repository, `vllm-project/agentic-api`. vLLM Agentic API should let vLLM core focus on inference while this repository handles API state, continuation, tool-call execution, and the higher-level orchestration needed by stateful agentic applications. ## Project Goals ### 1. Responses API Hydration The first project goal is reliable state hydration for the Responses API. This means supporting `previous_response_id` so clients can continue from a stored response without replaying the full conversation history. The server must persist the model-visible items needed for continuation, rehydrate them in the right order, append new input, and send the resulting request to vLLM core. Core work: - Store Responses API state needed for continuation. - Rehydrate prior input, assistant output, tool calls, and tool outputs. - Preserve OpenAI-compatible request and response shapes. - Support streaming and non-streaming Responses API flows. - Keep storage semantics explicit and testable. ### 2. Codex Support Through Responses The second project goal is to support Codex through the Responses API. Codex depends on stateful continuation, tool-call preservation, and a broad set of agentic tool and response shapes. `agentic-api` should accept Codex-compatible Responses traffic, preserve the fields Codex needs, and execute or return tool calls according to the ownership model for each tool. Core work: - Preserve Codex-used tool declarations and response item shapes. - Support client-owned tools, gateway-owned tools, and provider-owned tools. - Implement the tool calls and API behavior needed for practical Codex sessions. ### 3. Server-Side Tool Execution The project should execute server-side tool calls generated by vLLM core. When vLLM core produces a tool call that is owned by the gateway, vLLM Agentic API should dispatch it, capture the result, append the tool output, and continue the agentic loop. The tool system should make ownership explicit so each tool call has a clear execution path. Tool ownership means: - **Gateway-owned tools** are executed by vLLM Agentic API. - **Client-owned tools** are preserved and returned to the client for execution. - **Provider-owned tools** are passed through to vLLM core or another upstream provider surface; vLLM Agentic API does not execute them. Ownership should be resolved from the request, the configured tool registry, and the tool type. vLLM Agentic API should only execute tools that resolve to a configured gateway-owned handler. Unknown, unsupported, or ambiguous tool shapes are preserved and returned or passed through; they are never executed by default. Requests may opt into parallel tool calling for gateway-owned built-in tools: a single turn can invoke the same built-in tool more than once (for example, two web searches). Agentic API forwards that model-generation preference upstream, then executes emitted gateway calls through a bounded, configurable window. Calls to different tool names can overlap; calls to the same name overlap only when that handler declares it safe. Results retain model call order and are all appended before continuing the agentic loop ([#181](https://github.com/vllm-project/agentic-api/issues/181)). Initial and expected tool areas include: - File search. - Web search. - Web fetch. - Computer use. - Shell tools. - MCP-backed tools and resources. - Function tools owned by the gateway. ### 4. Messages API The third major API goal is Messages. Messages should build on the same persistence, rehydration, and execution foundations as Responses, while exposing the API surface expected by agentic clients that organize state around message objects. Messages should adapt into shared execution primitives for items, tool calls, and tool outputs instead of forking a Responses-specific execution loop. Core work: - Define the Messages API state model. - Map Messages state into the shared execution and storage primitives. - Preserve compatibility with Responses where the APIs overlap. - Add tests that cover continuation and tool-call behavior through Messages. ### 5. Broader Stateful Agentic APIs After Responses, Codex support, and Messages, the project should add the remaining stateful APIs needed by core agentic applications. This should be driven by concrete client needs and vLLM core integration points, not by adding broad abstractions ahead of use. Each new API should reuse the common storage, continuation, and tool-execution foundations wherever possible. ## Near-Term Focus The near-term focus is to make the Rust implementation dependable and easy to extend. - Keep orchestration logic reusable across standalone and gateway deployments. - Keep HTTP serving, gateway integration, and tool execution thin around the shared stateful agentic API implementation. - Keep the implementation Rust-first and optimized around vLLM core's inference surface. - Treat model aliasing and routing as deployment concerns that may live at the standalone server or gateway edge; they are not part of tool execution semantics. - Expand compatibility tests for Responses, Codex-shaped traffic, tool calls, streaming, and continuation. - Document behavior as it stabilizes, especially where compatibility or tool ownership rules are subtle. ## Responses API Compatibility and Interactive Execution The [GPT-6 Astra API guide](https://developers.openai.com/api/docs/guides/latest-model#gpt-6-astra-update-api-and-model-parameters) identifies newer Responses capabilities needed by agentic clients. Track these as API and orchestration work while keeping model inference, prompt rendering, and KV-cache execution in the upstream. Accepting a field does not establish support: each deployment must preserve, deliberately transform, or explicitly reject it across HTTP, WebSocket, storage, and continuation paths. ### First: Request and Continuation Compatibility - **Prompt caching and service tiers:** preserve typed cache controls, cache keys, supported breakpoints, and processing-tier metadata; validate unsupported model/backend combinations without imposing one provider's restrictions on every vLLM model ([#330](https://github.com/vllm-project/agentic-api/issues/330)). - **Cache-preserving reasoning updates:** support `configuration_update` items, their ordered history and effective effort, and explicit compatibility rules for compaction, truncation, and multi-agent mode ([#331](https://github.com/vllm-project/agentic-api/issues/331)). - **Provider-aware reasoning replay:** retain safe plaintext replay for vLLM while allowing compatible upstreams to consume their own opaque reasoning state. Cross-provider decryption is outside scope ([#335](https://github.com/vllm-project/agentic-api/issues/335)). These issues extend the capability contract in [#314](https://github.com/vllm-project/agentic-api/issues/314). Document support per execution path and prioritize preventing silent loss of request semantics. Provider-specific replay is conditional on a supported deployment need. ### Next: Interactive Execution - **Async function and custom tools:** preserve async declarations and calls, retain pending call identity across responses, and allow independent model work before the application returns a tool call output. Define bounded lifecycle rules for any gateway-managed pending work ([#332](https://github.com/vllm-project/agentic-api/issues/332)). - **Mid-turn steering:** accept WebSocket `response.steer`, queue user updates, emit steering lifecycle events, and continue from completed work with correct handling of pending tool outputs, approvals, and disconnects ([#333](https://github.com/vllm-project/agentic-api/issues/333)). Reuse the shared typed ingestion, orchestration, and ordered delivery boundaries tracked in [#241](https://github.com/vllm-project/agentic-api/issues/241) and [#244](https://github.com/vllm-project/agentic-api/issues/244). Parallel tool calls, independent WebSocket lanes, async calls, and steering have distinct semantics. ### Broader Tool and Multi-Agent Capabilities - **Programmatic Tool Calling:** define execution ownership and upstream requirements, then implement typed programmatic calls, caller policies, and continuation for supported profiles. Gateway execution requires an isolated, resource-bounded runtime ([#334](https://github.com/vllm-project/agentic-api/issues/334)). - **Multi-agent orchestration:** implement isolated agent contexts and shared scheduling through the existing tracker ([#298](https://github.com/vllm-project/agentic-api/issues/298)), including HTTP execution ([#299](https://github.com/vllm-project/agentic-api/issues/299)) and WebSocket injection ([#300](https://github.com/vllm-project/agentic-api/issues/300)). - **Computer use:** add typed computer calls, ordered actions, screenshots, and explicit execution-location handling through the existing feature issue ([#171](https://github.com/vllm-project/agentic-api/issues/171)). Use reference recordings and matching gateway scenarios to qualify each capability. Reuse the cassette recorder workflow; document intentional differences and unsupported upstream combinations rather than implying full OpenAI conformance from request-schema coverage alone. ## Enterprise Readiness [Enterprise Readiness](https://github.com/vllm-project/agentic-api/issues/316) groups the enhancements needed to operate Responses and Messages as a secure, predictable service with explicit capabilities and repeatable release checks. - **Authenticated state ownership:** scope persisted responses, conversations, items, and continuation to the authenticated user and tenant ([#107](https://github.com/vllm-project/agentic-api/issues/107)). - **Predictable Messages execution:** validate streaming lifecycles and bound retained content, pending tool calls, and delivery resources ([#313](https://github.com/vllm-project/agentic-api/issues/313)). - **Request and conversation compatibility:** preserve supported fields and transport metadata across execution paths, with explicit handling of unsupported capabilities ([#314](https://github.com/vllm-project/agentic-api/issues/314)); complete standard conversation references, creation payloads, and resource operations ([#155](https://github.com/vllm-project/agentic-api/issues/155)). - **MCP tool approvals:** support approval policies and resumable, authorized execution of selected Model Context Protocol tools ([#145](https://github.com/vllm-project/agentic-api/issues/145)). - **Complete usage accounting:** report inference usage across all Messages rounds, including supported cache counters, without double-counting ([#315](https://github.com/vllm-project/agentic-api/issues/315)). - **Release qualification:** verify authentication, persistence, tools, continuation, streaming, WebSockets, and failure behavior through the actual service in standalone and reverse-proxy deployments ([#110](https://github.com/vllm-project/agentic-api/issues/110)), building on the engine-conformance program ([#211](https://github.com/vllm-project/agentic-api/issues/211)). Prioritize ownership, Messages reliability, and request fidelity; then complete conversation, approval, and metering workflows. Develop qualification tests throughout the work. Each supported deployment profile must document its capabilities and meet the linked acceptance criteria before release. ## Longer-Term Direction Longer-term work should improve production readiness and performance without moving inference responsibilities out of vLLM core. - Production storage backends, retention policy, and compaction. - Observability for request lifecycle, tool execution, and continuation. - Integration with reverse proxies, ingress, and other deployment surfaces. - Cached prefix continuation and other latency optimizations where vLLM core owns rendering, tokenization, and KV-cache execution. - Coordination with vLLM core and llm-d where token identity, prefix routing, and renderer boundaries matter. ## Non-Goals For Now These are intentionally out of scope until the core APIs and execution model are stable. - Reimplementing tokenization, chat templates, or model-specific rendering in this repository. - Building a broad plugin platform before the built-in execution model is settled. - Adding stateful agentic APIs without a concrete client or vLLM core integration need. - Moving vLLM core inference responsibilities into vLLM Agentic API.