Why Start at the Client Integration Layer

The most overlooked step in local deployment isn't the model — it's how the client points at it. Most SDKs (the official OpenAI library, LangChain, LlamaIndex, IDE plugins) accept only one base_url. As long as both backends expose the same API shape, a single config change lets you move between a lightweight Ollama instance and a high-concurrency vLLM instance without touching business code.

Two practical payoffs:

  1. Switchable capacity: route to vLLM during daytime shared use, back to Ollama for solo debugging at night — one supply, allocated on demand.
  2. Rollback on failure: if the vLLM instance OOMs or restarts, flip one environment variable and you're back on Ollama instead of losing the whole line.

Steps

Step 1: Expose Ollama as an OpenAI-compatible endpoint

Ollama ships a /v1 compatibility layer on port 11434 by default. Verify after starting:

```bash

ollama serve

curl http://localhost:11434/v1/models

```

On the client side:

```python

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

```

Any non-empty api_key works; Ollama doesn't validate it. Note: the compatibility layer covers common fields for chat/completions and embeddings, but tools, strict JSON Schema for response_format, and some logprobs behavior vary by model. Get the main path working, then verify field by field.

Step 2: Pull a model under the same name as the vLLM side

The key trick is exposing the same model name from both backends, so switching requires zero client changes. Alias it on the Ollama side with a Modelfile:

```bash

ollama pull qwen2.5:7b-instruct

cat > Modelfile <<'EOF'

FROM qwen2.5:7b-instruct

PARAMETER num_ctx 8192

EOF

ollama create team-default -f Modelfile

```

From then on, always send model="team-default".

Step 3: Decide whether vLLM is needed, based on VRAM budget

vLLM's value is continuous batching and tensor parallelism (TP). If one GPU fits, skip TP — TP adds inter-GPU communication overhead, and 2-way TP typically delivers noticeably less than a linear extrapolation of single-GPU throughput. Rules of thumb:

  • Weights + KV cache fit on one GPU, concurrency < roughly 8: Ollama is enough.
  • Need 10+ concurrent requests, or long-context batching: move to single-GPU vLLM.
  • Weights don't fit on one GPU (e.g. 32B+ in BF16): only then enable TP=2.

Launch example (2-way TP):

```bash

vllm serve Qwen/Qwen2.5-14B-Instruct \

--tensor-parallel-size 2 \

--served-model-name team-default \

--max-model-len 8192 \

--gpu-memory-utilization 0.90 \

--port 8000

```

--served-model-name team-default is what makes the client switch transparent.

Step 4: One-switch via an environment variable

Don't hardcode addresses in two places; route through one variable:

```bash

export LLM_BASE_URL=http://localhost:11434/v1 # change to :8000/v1 for vLLM

```

Add a minimal health check to decide whether to switch:

```bash

curl -fsS "${LLM_BASE_URL}/models" >/dev/null && echo UP || echo DOWN

```

Drop it into your launch script or IDE pre-launch task so the client never idles against a dead backend.

Step 5: Map quantization tiers to VRAM

On the same GPU, quantization directly determines the largest model you can run:

  • Q4_K_M (GGUF, Ollama): about 1/4 the VRAM of BF16; quality loss acceptable for most chat tasks.
  • AWQ / GPTQ 4-bit (vLLM): also about 1/4, but requires a vLLM build supporting that quantization backend.
  • FP8 (vLLM, newer GPUs): about 1/2, quality close to BF16.

Practical advice: get the workload running on Ollama + Q4 first, confirm the model choice, then decide whether to move to vLLM + AWQ. Doing it in the reverse order means repeated quantization-compatibility pain.

Step 6: Track "capacity" as observable metrics

Local tokens aren't claimed — they're produced. Record at least:

  • Output tokens per second (tokens/s)
  • Average time to first token (TTFT)
  • Peak concurrency and queue time

vLLM exposes /metrics (Prometheus format); Ollama can be measured via --verbose or a thin proxy. With these three numbers you'll know when to switch from Ollama to vLLM instead of guessing.

Caveats

  1. Compatibility isn't equivalence: Ollama's /v1 and vLLM's OpenAI-compatible API differ on tool calling, structured output, and streaming chunk boundaries. Run a regression set of identical requests before switching backends.
  2. TP isn't free: tensor parallelism wants fast inter-GPU links (NVLink best, PCIe works but with efficiency loss). Two mismatched GPUs often fail to launch or perform terribly — use identical cards.
  3. Don't max out --gpu-memory-utilization: 0.90 is a common safe value; 0.95+ tends to OOM on long contexts.
  4. Name consistency is discipline: maintain served-model-name and the Ollama create alias as a pair; rename one without the other and switching breaks.
  5. Licensing and compliance: running open weights locally still requires following model licenses (Qwen, Llama families each have terms). Check before commercial use.
  6. This isn't free tokens: you save API fees but pay electricity, hardware depreciation, and ops time. Small single-machine self-use usually pays off; at scale, do the full math.

When This Fits

  • Solo developers or small teams with 1–2 consumer/workstation GPUs who want everyday inference cost near zero.
  • Offline or intranet environments where external APIs aren't allowed.
  • Workloads with clear peaks and troughs (shared by day, solo by night) that suit dual-backend switching.
  • Model-selection comparisons where one client codebase must switch models and quantization tiers quickly.

Not a fit: production services needing high availability (SLA), environments without local GPUs, or cases with strict consistency requirements on tool calling and structured output — hosted APIs remain the lower-effort choice there.