Why move past Ollama at all

Ollama's value is "running in five minutes": one ollama run command pulls the model, quantizes it, starts a server, and gives you an OpenAI-compatible endpoint. But its default scheduling is built for a single user with low concurrency. Once you start running batch inference, serving an internal team gateway, or handling multiple agent sessions at once, you hit two hard walls:

  • Throughput plateaus: requests queue up badly, and GPU utilization sits below 30% for long stretches.
  • Poor VRAM utilization: the KV cache is pre-allocated at maximum length, which wastes a lot in long-context scenarios.

vLLM targets exactly these two problems. It manages the KV cache with PagedAttention (allocating VRAM in blocks, cutting fragmentation dramatically) and pairs that with continuous batching. On the same GPU you'll typically see several times the throughput. The trade-off is heavier configuration and more sensitivity to VRAM.

The route below assumes you have a machine with an NVIDIA GPU (8GB to 24GB consumer cards or datacenter cards all work) and that you've already used Ollama.

Step 1: Pick the model and quantization tier for your VRAM

Subtract first; don't grab the biggest model out of the gate. Rough guidance (measure on your own hardware):

| VRAM | Model size | Quantization | Typical use |

| --- | --- | --- | --- |

| 8GB | 7B-8B | Q4 / AWQ 4bit | Single-session chat, code completion |

| 12-16GB | 8B-14B | Q4 / FP8 | Small-team internal assistant |

| 24GB | 14B-32B | Q4 / AWQ | Batch data processing |

| Multi-GPU 48GB+ | 32B-70B | FP8 / AWQ | High-concurrency gateway |

vLLM is friendlier to unquantized precision, but if VRAM is tight, AWQ and GPTQ are mature options. FP8 needs newer cards (Hopper and later).

The migration trap: Ollama uses GGUF, which vLLM does not natively support. You need to find the safetensors version of the same model on Hugging Face, or an AWQ/GPTQ quantized build. This is where most people get stuck, so confirm matching weights exist before you start.

Step 2: Install vLLM and start an OpenAI-compatible server

The official Docker image saves you from CUDA version alignment headaches:

```bash

docker run --runtime nvidia --gpus all \

-v ~/.cache/huggingface:/root/.cache/huggingface \

-p 8000:8000 \

--ipc=host \

vllm/vllm-openai:latest \

--model Qwen/Qwen2.5-7B-Instruct-AWQ \

--quantization awq \

--max-model-len 8192 \

--gpu-memory-utilization 0.90

```

What the key flags mean:

  • --max-model-len: directly determines KV cache size. Too large and you OOM; too small and long documents get truncated. Start at 8192.
  • --gpu-memory-utilization: the fraction of VRAM vLLM may occupy, default 0.9. If other processes share the GPU, drop it to 0.7-0.8.
  • --tensor-parallel-size: set to your GPU count for multi-GPU (e.g. 2 for two cards), keep at 1 for single GPU.

Bare-metal install via pip install vllm works too, but CUDA, PyTorch, and driver versions must all line up. Docker saves a lot of debugging.

Step 3: Point your existing code at vLLM

This is the easiest part: vLLM exposes an OpenAI-compatible API, so code that pointed at Ollama mostly just needs a new base_url and model name.

```python

from openai import OpenAI

client = OpenAI(

base_url="http://localhost:8000/v1",

api_key="EMPTY" # vLLM does not validate by default; placeholder is fine

)

resp = client.chat.completions.create(

model="Qwen/Qwen2.5-7B-Instruct-AWQ",

messages=[{"role": "user", "content": "Explain PagedAttention in one sentence"}],

temperature=0.7,

)

print(resp.choices[0].message.content)

```

If you were using Ollama's /api/chat endpoint, switch to the OpenAI-style call above. If you were already using the OpenAI SDK pointed at Ollama, it's nearly a no-op.

Add an environment variable (e.g. LLM_BASE_URL) so switching between local dev and production needs no code changes.

Step 4: Verify throughput actually improved

Don't guess. Benchmark with vLLM's built-in script:

```bash

vllm bench serve \

--backend openai-chat \

--base-url http://localhost:8000 \

--model Qwen/Qwen2.5-7B-Instruct-AWQ \

--num-prompts 200 \

--request-rate 8

```

Watch two numbers: output token throughput (tokens per second) and TTFT (time to first token). Compared with Ollama under the same conditions, you'll usually see a multiple-fold gap, especially as concurrency rises.

If throughput didn't improve, first check whether --max-model-len is so large that the KV cache is thrashing, or whether your request rate simply isn't hitting the bottleneck.

Step 5: When not to use vLLM

vLLM isn't universal. These cases favor Ollama / llama.cpp instead:

  • CPU-only or Apple Silicon: vLLM's CPU inference support is limited; llama.cpp (Ollama's underlying engine) is more mature on that hardware.
  • Very small VRAM (< 8GB) with single-session chat: vLLM's scheduling overhead isn't worth it; Ollama is lighter.
  • Frequent switching between many models: Ollama's model management is nicer; vLLM requires restarting the server per model swap.
  • Running GGUF quantizations: vLLM doesn't read GGUF; stay with the llama.cpp family.

A middle path: use Ollama for local dev, vLLM for benchmarking and production. Both speak OpenAI-compatible APIs, so switching is cheap.

Caveats

  1. VRAM is a hard constraint. Budget model weights + KV cache + activations. On OOM, lower --max-model-len first, then --gpu-memory-utilization, then consider a smaller quantization.
  2. Read the license. Open weights don't mean unrestricted commercial use. Llama, Qwen, and Mistral each have their own terms; confirm compliance before internal deployment.
  3. Don't drop Docker's --ipc=host. vLLM's multi-process shared memory needs it, or you may hit insufficient shared memory errors.
  4. Don't expose the server publicly. vLLM does no auth by default; api_key is decorative. Put a gateway in front for auth and rate limiting.
  5. Versions move fast. vLLM updates frequently and flag names occasionally change. Check the docs for your version rather than copying old tutorials.
  6. Trust the model source. When pulling weights from Hugging Face, check the repo owner and prefer official organization releases.

When this fits

  • A team needs an internal inference gateway with no API bill.
  • Batch processing large volumes of text (classification, extraction, summarization) where throughput matters more than latency.
  • Data-sensitive scenarios that cannot leave the intranet.
  • Agent / RAG systems that need high-concurrency calls to a local model.

If your need is "ask a few questions occasionally," Ollama is enough; don't bother with vLLM. The real signal to migrate is when you start worrying about queueing and idle GPUs.

Cost accounting

"Free" here means zero API fees, not zero cost. The real costs are:

  • Hardware depreciation (your own GPU or a rented cloud GPU instance).
  • Electricity (consumer cards typically draw 200-400W under load).
  • Your time.

For light use, a pay-as-you-go cloud GPU may beat buying a card; for heavy long-term use, owned hardware amortizes better. Run your own numbers rather than getting carried away by the word "free."