How vLLM Processes a Prompt: My First Deep Dive into Inference

Today I started digging into vLLM and, more importantly, what actually happens inside an inference engine when we send a prompt to an LLM.

I had used LLM APIs before, but I never really thought about what happens between:

"Hello, how are you?"

and

"I'm doing great..."

So I decided to trace the journey.

1. Everything starts with tokenization

The first step is simple:

Prompt → Tokenizer → Tokens

The text we provide isn't directly processed by the transformer. The tokenizer converts the prompt into token IDs, which are then used by the model.

For example:

"Hello, how are you?"  →  [15496, 11, 703, 389, 345, 30]

The exact tokens depend on the tokenizer/model. These tokens become the input to the inference engine.

2. The scheduler decides what gets processed

This is where things start getting interesting. The requests don't simply enter the GPU and get processed one by one. The inference engine maintains requests in different states, such as:

• Waiting
• Running
• Finished

The scheduler decides which requests should be processed in each iteration. One important distinction is between:

• Prefill requests — processing the prompt for the first time.
• Decode requests — generating subsequent tokens.

Decode work matters because already-running sequences need to keep generating tokens, while new prompts are still waiting to be processed. This is one of the reasons scheduling becomes such an important part of inference performance.

3. What actually happens inside the Transformer?

Now the interesting part. The tokens enter the Transformer, which consists of multiple layers. Inside the attention mechanism, we derive Query (Q), Key (K), and Value (V) from the hidden states.

Attention is then calculated using the relationship between Q and K:

Attention(Q, K, V) = softmax(QKᵀ / √d) V

The resulting representation is passed through the rest of the layer, including the feed-forward network. After passing through all the layers, the model produces logits representing the probability distribution for the next token. The highest-probability token isn't necessarily always selected — sampling strategies can also be applied.

4. Prefill: processing the prompt

This initial processing of the entire prompt is called prefill. Suppose the user sends:

"Explain how neural networks work"

The model processes all of those input tokens through the Transformer. While doing this, it produces the Key and Value states needed for attention, and stores them in the KV cache.

Why? Because when we generate the next token, we don't want to recompute the entire prompt from scratch.

5. Decode: generating the next token

Now comes the decode phase. Suppose the model generates "Neural". That token is appended to the sequence, and the model predicts the next one.

Prompt → Prefill → Token 1 → Decode → Token 2 → Decode → Token 3 → ...

The key idea is that the previously computed K/V states are reused from the KV cache. Only the newly generated token needs to go through the incremental decode computation. This is a huge part of making autoregressive generation practical.

6. So where does the KV cache live?

This is where vLLM gets really interesting. The KV cache consumes a significant amount of GPU memory. Instead of treating it as one giant contiguous chunk, vLLM uses a paged/block-based approach: the cache is divided into fixed-size blocks.

Block 0 → tokens 0–15
Block 1 → tokens 16–31
Block 2 → tokens 32–47

Each block stores Key and Value tensors for the relevant transformer layers. So if a sequence occupies multiple blocks, those blocks don't need to be physically contiguous in GPU memory. That's one of the important ideas behind PagedAttention.

7. How much GPU memory does a KV block consume?

Now we get into the mathematics. A simplified way to think about the KV-cache memory for a block is:

2 × num_layers × num_KV_heads × head_dim × block_size × bytes_per_element

Why the 2? Because we store both K (Key) and V (Value).

For example, with layers = 32, KV heads = 32, head dimension = 128, block size = 16 tokens, and FP16:

KV memory per block = 2 × 32 × 32 × 128 × 16 × 2 bytes

Understanding this makes something very clear: the KV cache can consume a lot of GPU memory, and inference engines need to manage that memory extremely carefully.

8. The block table

If blocks aren't necessarily contiguous, the system needs to know which logical tokens are stored in which physical GPU blocks.

Logical tokens:  0 1 2 ... 15 | 16 17 ... 31
                     ↓              ↓
                  Block 7       Block 12

The engine maintains metadata that maps the logical sequence to the physical KV-cache blocks. This lets vLLM manage GPU memory dynamically — when a block becomes free, it can be reused by another request.

9. Continuous batching

Now imagine multiple users:

Request 1 → 8 tokens
Request 2 → 5 tokens
Request 3 → 7 tokens
Request 4 → 20 tokens
Request 5 → 10 tokens

The engine doesn't wait for one request to completely finish before processing another. Instead, the scheduler dynamically combines work from different requests — this is continuous batching. At every iteration it decides which sequences participate and how much work fits within the available token budget. This is much more efficient than static batching when requests arrive and finish at different times.

10. Token budget

The scheduler also respects a limit on how many tokens it can process in an iteration. Think of it as:

Token budget = 2048

If several requests are waiting, it considers how many tokens each needs and figures out what fits into the current iteration:

Request 1 → 5 tokens
Request 2 → 4 tokens
Request 3 → 6 tokens
Request 4 → 5 tokens
Total     → 20 tokens

If the budget allows, they're processed together. This is where scheduling, batching, and GPU memory management start interacting.

11. The decode loop

After prefill, requests enter the running state and the decode loop begins:

Scheduler
   ↓
Select sequences
   ↓
Transformer layers
   ↓
Predict next token
   ↓
Update KV cache
   ↓
Append new token
   ↓
Scheduler again  →  repeat

This continues until a stopping condition is reached — an EOS token, a maximum generation length, a stop sequence, or another request-specific condition — and then the request is finished.

12. Why vLLM is interesting

Before learning this, I mostly thought of inference as Prompt → Model → Answer. Now I'm starting to see that inference is actually a systems problem. Several things happen at once:

Scheduler
   ↓
Continuous batching
   ↓
Transformer
   ↓
KV cache (block-based memory)
   ↓
Next token

And this is what I'm beginning to understand about inference engineering: it's not just about running a model. It's about figuring out how to use limited GPU memory and compute as efficiently as possible while serving many requests.

Still early in the journey, and I'm definitely going to get things wrong along the way. But that's the point of learning in public.

Today it was vLLM and understanding what happens between a prompt and the next token. Tomorrow, we go one layer deeper.

Thanks for reading. More to come.