Why Local Deployment Is the Real Zero Marginal Cost

Any cloud free tier has a ceiling: quota runs out, rate limits cap concurrency, terms change and it's gone. Local deployment has a completely different cost structure — one-time hardware spend, then marginal cost per million tokens approaches the electricity bill. In 2026 this path is mature enough: Ollama handles usability, vLLM handles throughput, and both expose OpenAI-compatible endpoints, so application code barely changes.

Key insight: local token "quota" isn't claimed, it's computed from VRAM and quantization tier. Get that math right and you know exactly how much capacity you have.

Step 1: Budget VRAM First, Then Pick a Model

Rough formula (inference, not training):

```

weight VRAM ≈ params × bytes per param

FP16 ≈ 2 bytes/param

INT8 ≈ 1 byte/param

INT4 ≈ 0.5 bytes/param

```

Add KV cache and framework overhead — in practice reserve 1.2–1.5× the weights. Examples:

  • 7B INT4 ≈ 3.5 GB weights → fits a 12 GB card with context headroom
  • 14B INT4 ≈ 7 GB → comfortable on 16 GB
  • 32B INT4 ≈ 16 GB → needs 24 GB or dual cards
  • 70B INT4 ≈ 35 GB → multi-GPU or more aggressive quantization

Measure VRAM, then pick model size, then pick quantization tier. Reverse the order and you'll OOM repeatedly.

Step 2: Ollama for Daily Use, vLLM for Throughput

Don't mix their roles:

Ollama (daily, low concurrency, hot switching)

```bash

ollama pull qwen2.5:14b-instruct-q4_K_M

ollama run qwen2.5:14b-instruct-q4_K_M

# OpenAI-compatible endpoint at http://localhost:11434/v1

```

Good for: solo coding, single-user Q&A, frequent model swaps. Model management is pleasant, and Modelfile lets you pin system prompts and parameters.

vLLM (high concurrency, batching, saturating VRAM)

```bash

pip install vllm

vllm serve Qwen/Qwen2.5-14B-Instruct-AWQ \

--quantization awq \

--max-model-len 8192 \

--gpu-memory-utilization 0.90 \

--port 8000

```

--gpu-memory-utilization 0.90 is the key flag: it lets vLLM devote 90% of VRAM to KV cache, supporting higher concurrency. PagedAttention cuts fragmentation and continuous batching lets many requests share one forward pass — that's how local throughput approaches small cloud models.

Step 3: Turn Concurrency Ceiling into Measurable Capacity

Concurrency isn't a guess. The relationship:

```

max concurrency ≈ (available VRAM - weight VRAM) / per-request KV cache

per-request KV cache ≈ 2 × layers × hidden dim × seq len × bytes

```

In practice: start --max-num-seqs at 8, load test, raise gradually, watch VRAM and P99 latency. If you exceed VRAM, lower --max-model-len — cheaper than cutting concurrency.

Capacity math: a 24 GB card running 14B INT4 typically outputs ~40–80 tok/s depending on card and batch size. At 8 hours full load and 60 tok/s average, that's roughly 1.7M output tokens per day. That's your "daily quota" — no platform approval needed.

Step 4: Wire It as an OpenAI-Compatible Endpoint

Both frameworks expose /v1/chat/completions, so apps only change base_url and api_key:

```python

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

```

Recommended: a thin gateway with vLLM as primary and Ollama as fallback, switching on timeout or overload. Even a single machine then feels like uninterrupted supply.

Caveats

  • Quantization has a cost: INT4 shows noticeable accuracy loss on long context, math, and code. Switch to FP16/INT8 for eval work, or use a larger model at a smaller quant tier.
  • Longer context eats KV cache: setting --max-model-len too high causes concurrency to collapse. Set it to your real task length, don't max it out by default.
  • Consumer cards lack NVLink: tensor parallelism over PCIe has limited returns; dual cards are better for "one model each" than "one giant model".
  • Power and cooling are real costs: factor in electricity and fan noise; cap power for long runs.
  • Not everything belongs locally: huge models need multi-GPU even quantized, where pay-per-token cloud may be cheaper.
  • Compliance and data: the advantage is data staying on-prem, but model licenses (Apache 2.0, custom commercial terms) still need per-model review.

When to Use This

  • Solo developers: daily coding help and document processing, fully offline
  • Small teams on-prem: keep sensitive data local, serve multiple users via vLLM
  • App prototypes: validate the pipeline locally before deciding on cloud
  • Batch offline jobs: cleaning, classification, summarization overnight on the GPU

Conclusion

Local deployment isn't a "cheaper substitute" — it's a controllable, measurable token supply line immune to terms changes. Budget VRAM first, use Ollama for daily use and vLLM for throughput, then expose both as an OpenAI-compatible endpoint. Your "free quota" becomes stable daily output.