Proxy Endpoints¶

DeltaLLM's proxy API is OpenAI-compatible. In most clients, you only change the base_url and API key.

Quick Start¶

Use the same auth header for every proxy endpoint: Authorization: Bearer YOUR_API_KEY

Check which models are available:

curl http://localhost:8000/v1/models \
  -H "Authorization: Bearer YOUR_API_KEY"

Send a chat request:

curl http://localhost:8000/v1/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [
      {"role": "user", "content": "Hello from DeltaLLM"}
    ]
  }'

Endpoint Map¶

Chat Completions accepts max_tokens or max_completion_tokens. Supply one. When a caller has an output_tpm_limit, the gateway checks recorded output before the request and counts actual output after each provider attempt. Admitted calls can exceed the remaining quota; subsequent calls are blocked until the fixed UTC minute resets. Output TPM does not require or change generation caps. See Output TPM limits and headers.

Endpoint	Purpose
`POST /v1/chat/completions`	Chat completions, including streaming
`POST /v1/messages`	Anthropic Messages API-compatible subset, see Anthropic Messages
`POST /v1/completions`	Legacy prompt-style completions
`POST /v1/responses`	Responses API compatible subset
`POST /v1/embeddings`	Text embeddings
`POST /v1/images/generations`	Image generation
`POST /v1/audio/speech`	Text-to-speech
`POST /v1/audio/transcriptions`	Speech-to-text
`POST /v1/rerank`	Reranking
`GET /v1/models`	Available public model names
`POST /v1/files`	Upload batch input files
`GET /v1/files/{file_id}`	Inspect batch files
`GET /v1/files/{file_id}/content`	Download batch file content
`POST /v1/batches`	Create embedding or non-streaming chat completion batches
`GET /v1/batches`	List batches
`GET /v1/batches/{batch_id}`	Inspect one batch
`POST /v1/batches/{batch_id}/cancel`	Cancel a batch

Text Endpoints¶

Chat Completions¶

POST /v1/chat/completions

This is the main endpoint most applications should start with.

Chat requests also support DeltaLLM-managed MCP tools through tools: [{ "type": "mcp", ... }] on non-streaming requests. See MCP Gateway & Tooling.

OpenAI-compatible client-managed tool histories may omit assistant content, or set it to null, when that assistant message contains non-empty tool_calls. Follow the assistant turn with the matching tool result:

{
  "model": "gpt-4o-mini",
  "messages": [
    {"role": "user", "content": "Search the documentation."},
    {
      "role": "assistant",
      "tool_calls": [
        {
          "id": "call_1",
          "type": "function",
          "function": {"name": "search", "arguments": "{\"query\":\"DeltaLLM\"}"}
        }
      ]
    },
    {"role": "tool", "tool_call_id": "call_1", "content": "Search result"}
  ]
}

System, user, and tool messages still require non-null content. Use POST /v1/chat/completions, not the legacy POST /v1/completions, for tool-calling requests.

Provider error bodies have size limits. DeltaLLM never returns them verbatim. The gateway does not decompress encoded error bodies. It classifies these errors with trusted status and Retry-After metadata. Bounded identity bodies can also supply provider-specific classifications. Before response commit, DeltaLLM uses the sanitized classification for retry and failover.

After a streaming response sends content, DeltaLLM does not retry it. A provider failure closes the OpenAI-compatible stream without a [DONE] marker. Clients must treat a missing terminal marker as an incomplete response.

Streaming accounting¶

For successful text streams, DeltaLLM awaits the configured answer-accounting writer and required audit before forwarding [DONE] (or the translated message_stop event). Clients may close at that marker without draining the HTTP connection to EOF. Content tokens are still delivered immediately. This ordering applies to Chat, Completions, Responses, and the Messages-compatible wrapper.

Model-router selectors require general_settings.spend_ingestion_mode: outbox. In this mode, durable accounting must accept the answer charge before the terminal marker. The stream ends without a successful terminal marker if this acceptance or required audit fails. This also occurs if finalization exceeds the request deadline. The provider can still charge for tokens already generated. Durable acceptance does not mean that the background worker completed each reporting or ledger update.

The default legacy spend mode is also awaited, but its existing writer logs and swallows database-write failures and does not atomically update all ledgers. Consequently, [DONE] or message_stop in legacy mode is not proof that the charge was persisted. Awaiting this writer fixes close-at-terminal cancellation, not its historical durability limitations. Use outbox mode when durable charge acceptance is required.

Completions (Legacy)¶

POST /v1/completions

Use this only if you still have prompt-based clients. DeltaLLM translates prompt into a chat-style user message internally.

Unsupported request fields currently return 400:

echo
best_of > 1
logprobs
suffix

Responses¶

POST /v1/responses

DeltaLLM supports a compatible subset of the Responses API and translates these requests into chat completions internally.

curl http://localhost:8000/v1/responses \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "input": "Write a one-line summary of DeltaLLM.",
    "stream": false
  }'

Responses requests also support DeltaLLM-managed MCP tools on non-streaming requests. See MCP Gateway & Tooling.

Embeddings¶

POST /v1/embeddings

curl http://localhost:8000/v1/embeddings \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "text-embedding-3-small",
    "input": "The quick brown fox"
  }'

Multimodal and Specialized Endpoints¶

Image Generation¶

POST /v1/images/generations

curl http://localhost:8000/v1/images/generations \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "dall-e-3",
    "prompt": "A sunset over mountains",
    "size": "1024x1024"
  }'

Audio Speech¶

POST /v1/audio/speech

This endpoint returns audio bytes, not JSON.

Some providers stream WAV output with placeholder sizes in the file header (for example Groq Orpheus). When response_format is wav, DeltaLLM rewrites those placeholders with the real sizes so strict players accept the file.

Public speech requests remain OpenAI-compatible even when the selected deployment is ElevenLabs. For ElevenLabs, DeltaLLM maps input to native text, resolves the ElevenLabs voice ID from request voice or deployment model_info.default_params.voice_id, and calls POST /v1/text-to-speech/{voice_id} upstream with xi-api-key.

curl http://localhost:8000/v1/audio/speech \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "tts-1",
    "input": "Hello world",
    "voice": "alloy",
    "response_format": "mp3"
  }' \
  --output speech.mp3

Example using an ElevenLabs-backed public model:

curl http://localhost:8000/v1/audio/speech \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "elevenlabs-tts",
    "input": "Hello from ElevenLabs through DeltaLLM",
    "voice": "21m00Tcm4TlvDq8ikWAM",
    "response_format": "mp3"
  }' \
  --output elevenlabs-speech.mp3

Audio Transcription¶

POST /v1/audio/transcriptions

This endpoint accepts multipart form data.

Public transcription requests remain OpenAI-compatible when the selected deployment is ElevenLabs. DeltaLLM sends uploaded audio to the native POST /v1/speech-to-text endpoint. It maps public language to language_code and maps temperature. It converts the provider response to the requested public response format.

curl http://localhost:8000/v1/audio/transcriptions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -F "file=@sample.wav" \
  -F "model=whisper-large" \
  -F "response_format=json"

Example using an ElevenLabs-backed public model:

curl http://localhost:8000/v1/audio/transcriptions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -F "file=@sample.wav" \
  -F "model=elevenlabs-stt" \
  -F "language=en" \
  -F "response_format=verbose_json"

Rerank¶

POST /v1/rerank

curl http://localhost:8000/v1/rerank \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "rerank-english-v2.0",
    "query": "What is machine learning?",
    "documents": [
      "Machine learning is a subset of AI.",
      "The weather is sunny today.",
      "Deep learning uses neural networks."
    ],
    "top_n": 2
  }'

Batch Endpoints¶

Files¶

POST /v1/files
GET /v1/files/{file_id}
GET /v1/files/{file_id}/content

Use files as the input and output artifacts for batch jobs.

curl http://localhost:8000/v1/files \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -F "purpose=batch" \
  -F "file=@input.jsonl"

Batches¶

POST /v1/batches
GET /v1/batches
GET /v1/batches/{batch_id}
POST /v1/batches/{batch_id}/cancel

Create a batch:

curl http://localhost:8000/v1/batches \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "input_file_id": "file_123",
    "endpoint": "/v1/embeddings",
    "completion_window": "24h"
  }'

DeltaLLM supports the OpenAI-compatible 24h batch completion window. Missing or null completion_window values default to 24h; unsupported values return HTTP 400.

Inspect a batch:

curl http://localhost:8000/v1/batches/batch_123 \
  -H "Authorization: Bearer YOUR_API_KEY"

Current behavior:

batch endpoints are available only when general_settings.embeddings_batch_enabled: true
supported endpoints are "/v1/embeddings" and "/v1/chat/completions"
public batch responses include OpenAI-compatible completion_window, errors, expires_at, failed_at, and expired_at fields
chat batch requests must be non-streaming; stream: true is rejected
MCP tools are not supported in chat batch requests yet

Model Discovery¶

List Models¶

GET /v1/models

DeltaLLM returns the public model names that clients can request. This list is built from the current runtime model registry.

{
  "object": "list",
  "data": [
    {
      "id": "gpt-4o-mini",
      "object": "model",
      "owned_by": "deltallm"
    }
  ]
}