Design an LLM API Service (ChatGPT-scale)

#Problem Statement

Design a scalable API service for serving large language model inference. Users send prompts and receive streamed completions. The system must handle high request volume, long-running inference, multi-tenant isolation, and token-based billing.

📖

Real-world: This is the most current system-design problem, and its constraints are unlike the classics. The bottleneck isn't a database — it's scarce, expensive GPUs: inference is slow and memory-bound, so the whole design orbits a request queue feeding a GPU worker pool, with tricks like continuous/in-flight batching (packing multiple users' tokens through the GPU together) to maximize utilization. Responses stream token-by-token over SSE because a full completion takes seconds, and users tolerate "typing" but not a long blank wait. ChatGPT's launch in late 2022 famously hit 1M users in 5 days and repeatedly showed "at capacity" — a vivid lesson in GPU scarcity as the scaling limit. For a security audience the rich angles are right here: multi-tenant isolation (one customer's prompt/data must never leak into another's context), prompt-injection and abuse filtering on inputs, token-based billing integrity (and the "denial-of-wallet" abuse where an attacker runs up your inference bill), and rate limiting as a cost-control and safety control — see ../ai-security/owasp-llm-2025.md.

#Requirements

#Functional

• Accept prompt + parameters → return completion (full or streamed)
• Maintain conversation context (multi-turn chat)
• Token counting for billing
• Model selection (different model sizes/capabilities)
• Streaming output (token by token as generated)

#Non-Functional

• 10M requests/day → ~115 requests/s average; 1000 req/s peak
• Time to first token (TTFT): < 500ms
• Single inference: 10-60 seconds for long outputs
• GPU utilization: maximize (GPUs are expensive)
• Multi-tenant: one user's slow request cannot starve others
• Token limits enforced per API key

#Core Design

#Why This Is Hard

LLM inference is:

Standard request/response architecture doesn't scale for this.

#Request Queuing with Priority Lanes

Client → API Gateway → Token Counter + Auth → Request Queue
                                                    ↓
                                          Inference Scheduler
                                                    ↓
                                        GPU Worker Pool
                                          (batch requests)

Priority lanes

• Real-time API (low latency, interactive): premium lane, preemptible
• Batch API (high throughput, non-interactive): best-effort lane, up to 50% cheaper

#Continuous Batching

Naïve: one GPU processes one request at a time → 1/60th of GPU time utilized.

Continuous batching (key LLM optimization): multiple requests share the same GPU forward pass. As one request finishes, a new one joins the batch without waiting for all current requests to complete.

Time →
Req A: ████████ (done)
Req B:   ████████████████ (long)
Req C:         ████ (done)
Req D:             ████████ (started when A finished)

GPU is never idle waiting for one long request. Throughput scales with batch size.

#Key-Value (KV) Cache

During inference, transformer attention layers compute key-value tensors that can be reused across tokens in the same sequence. Caching these (KV cache) dramatically speeds up autoregressive generation.

Prefix cachingif two requests share the same system prompt, the KV cache for the prefix is computed once and reused. Important for:

• System prompt reuse (same system prompt across all requests in a session)
• RAG document context (same document used in many queries)

#Streaming via SSE

HTTP Server-Sent Events (SSE) is the standard for streaming tokens:

HTTP/1.1 200 OK
Content-Type: text/event-stream

data: {"token": "The"}
data: {"token": " quick"}
data: {"token": " brown"}
data: [DONE]

Client reads the stream token by token, rendering as it arrives.

#Architecture

Client → API Gateway (auth, rate limit, token count)
              ↓
        Request Router (model selection, priority)
              ↓
        Queue (Redis Streams / SQS per model tier)
              ↓
        Inference Cluster (Kubernetes + GPU nodes)
              |
     ┌────────────────────────┐
     │ Inference Server        │ ← vLLM / TensorRT-LLM
     │ (continuous batching)   │
     │ KV cache manager        │
     └────────────────────────┘
              ↓
        Response Streamer → SSE → Client
              ↓
        Usage DB (token counts, latency, model used)

#Context Window Management

Each conversation has a history that grows with each turn. LLMs have a context window limit (e.g., 128k tokens). Strategy:

• Keep last N turns
• Summarize old context when approaching limit
• Always include system prompt + last M turns

Conversation history stored in Redis (TTL = session length); if user returns after TTL, start fresh.

#Model Serving Infrastructure

#Key Design Decisions

#Security Considerations

#Interview Tips