Concurrency Is Your Quota: Turning Ollama Queues + vLLM Continuous Batching into a Metered Local Token Supply in 2026

Why start from concurrency, not the model

The most common local-deployment mistake is swapping to a bigger model or a newer quantization while never touching concurrency. In practice, single-box token output ≈ effective concurrency × per-request decode speed × time. Once the model is fixed, almost every remaining lever is concurrency and batching.

The 2026 reality: Ollama fits single-user or small-team interaction, vLLM fits multi-request throughput. They are not either/or — they are a front-end/back-end split.

Steps

Step 1: Use Ollama as the rate-limited front door

After installing Ollama, control concurrency and queuing via environment variables:

OLLAMA_NUM_PARALLEL: how many requests a single model handles at once. Set to 1 for serial requests when VRAM is tight; 2–4 for smoother interaction at higher VRAM cost.
OLLAMA_MAX_QUEUE: max queue length; requests beyond it are rejected, preventing unbounded pile-ups that destroy latency.
OLLAMA_MAX_LOADED_MODELS: how many distinct models stay resident. Model switching causes load jitter; keep only 1–2 hot models.

Start with ollama serve, then call /api/generate or the OpenAI-compatible /v1/chat/completions.
Keep only interactive work (chat, completion, coding assistant) on the front door; push long batch jobs to the back end.

Step 2: Use vLLM as the throughput engine

Launch as an OpenAI-compatible server, e.g. vllm serve <model> --max-model-len <len> --gpu-memory-utilization 0.85 --max-num-seqs <N>.
--max-num-seqs sets how many sequences are batched concurrently — the core throughput knob. --gpu-memory-utilization sets the VRAM share reserved for KV cache and therefore caps feasible concurrency.
Continuous batching (vLLM's default) lets new requests slot into decode gaps instead of waiting for a whole batch to finish.
Quantized weights (AWQ / GPTQ / FP8 depending on hardware) cut VRAM and let you raise --max-num-seqs, but low-bit quantization degrades long-context quality.

Step 3: Benchmark your "metered capacity"

Run a fixed prompt length + fixed output length benchmark and record: output tokens/sec, time to first token (TTFT), P95 latency.
Raise concurrency step by step (1→2→4→8…) and plot both a concurrency–throughput and a concurrency–latency curve.
Find the knee where throughput flattens and latency starts climbing steeply; set production concurrency slightly below it. That knee value is your local supply line's "quota".
Write benchmark results into config or docs and re-run whenever the model or hardware changes.

Step 4: Route and monitor

Put a lightweight gateway (LiteLLM Proxy or a small FastAPI service) in front to split by task type: interactive to Ollama, batch to vLLM.
Log input/output tokens per request and aggregate daily to get a "local tokens produced" metric.
Alert on queue length and VRAM usage to avoid OOM or request avalanches.

Caveats

VRAM is a hard wall: raising concurrency linearly raises KV cache usage; exceeding the --gpu-memory-utilization cap causes OOM or preemption.
Quantization is not free: 4-bit buys concurrency but clearly hurts long-context, math and coding quality; use it per task tier.
Don't let Ollama and vLLM fight over one GPU: sharing a card causes VRAM contention; split cards or time slots.
Local tokens are "free" only within limits: you save API fees but pay electricity, hardware depreciation and ops time. It pays off for small-scale, high-frequency workloads.
Compliance and data: local deployment keeps data in-house, but serving externally still requires obeying model licenses (custom Llama licenses, Apache 2.0 commercial terms).

When it fits

Internal coding assistants, document summarization, batch translation — high-frequency, low-value tasks.
Privacy-sensitive workloads that cannot call external APIs.
Offline or poor-network environments needing continuous token output.
As a fallback when external API quota runs out or rate-limits: auto-switch to local.

When it doesn't

Tasks needing frontier capability (complex reasoning, multimodality).
One-off, low-frequency, latency-insensitive long jobs — buying API access is simpler.
No GPU, or only small-VRAM consumer cards, yet high concurrency is required.

FAQ

Is a higher OLLAMA_NUM_PARALLEL always better?

No. It directly raises VRAM and KV cache pressure; exceeding the cap causes OOM or queuing. Start at 1, raise gradually toward the benchmark knee, and set OLLAMA_MAX_QUEUE to prevent pile-ups.

How do --max-num-seqs and --gpu-memory-utilization work together in vLLM?

The former sets batch concurrency; the latter sets the VRAM share available for KV cache. Higher share supports higher concurrency. In practice set gpu-memory-utilization to 0.8–0.9, then raise max-num-seqs until the throughput knee or OOM boundary.

Is local deployment really cheaper than buying API access?

It depends on scale and whether hardware already exists. With existing GPUs and high-frequency, short-output tasks, marginal cost approaches electricity and is usually cheaper; if you must buy hardware or tasks are infrequent, APIs usually win.

How far should I quantize?

Tier by task: 4-bit is fine for chat and summarization; use 8-bit or FP8/FP16 for code, math and long context. Validate concurrency gains from quantization with quality regression tests, not just throughput numbers.