Why Local Deployment Is About *Getting* Tokens, Not Saving Them

Official free tiers reset, require cards, and vary by region. Once open weights land on your own disk, inference is no longer bounded by any platform quota — you pay a one-time hardware cost plus electricity, and marginal cost approaches zero. In 2026 the bar is low enough that a single 16GB-VRAM consumer GPU can reliably serve 7B–14B quantized models for high-frequency, low-difficulty work: summarization, classification, extraction, code completion, batch translation.

The key mindset: don't use a local model to replace a flagship model — use it as your traffic floor. Keep 80% of simple requests local and call cloud APIs only when real reasoning is needed. Your cloud quota suddenly goes a long way.

Steps

Step 1: Measure hardware first, then pick a model

Size the model to VRAM. Rules of thumb for INT4/GGUF quantization (2026 mainstream):

  • 8GB VRAM: 3B–4B models (small Qwen variants, Llama 3.2 3B, Gemma 3 4B class), context capped under 8K
  • 12–16GB VRAM: 7B–9B models, 16K–32K context feasible
  • 24GB VRAM (3090/4090 class): 14B at INT4 or 32B at low-bit quantization — good for a shared team endpoint
  • Multi-GPU or 48GB+: 32B–70B; switch to vLLM with tensor parallelism here

Run nvidia-smi first, then choose quantization. Better to drop a model tier than to max out VRAM and thrash — throughput falls by an order of magnitude.

Step 2: Ollama as a single-box always-on service

Ollama suits personal machines and small teams. One command after install:

```bash

ollama pull qwen3:8b # or your chosen quantized variant

OLLAMA_HOST=0.0.0.0:11434 OLLAMA_NUM_PARALLEL=4 OLLAMA_MAX_LOADED_MODELS=1 ollama serve

```

Key points:

  • OLLAMA_NUM_PARALLEL sets concurrency slots; 2–4 per GPU is sensible. Too high and you queue on VRAM
  • OLLAMA_MAX_LOADED_MODELS=1 prevents multiple models resident and blowing up VRAM
  • It ships an OpenAI-compatible endpoint at http://localhost:11434/v1 — just point your SDK's base_url there, no code changes
  • Keep model files on a large SSD and set the path via OLLAMA_MODELS

Step 3: vLLM as a high-throughput shared endpoint

When several people share the endpoint or you need batch inference, Ollama's scheduler becomes the bottleneck. vLLM's PagedAttention and continuous batching multiply throughput:

```bash

vllm serve Qwen/Qwen3-14B-Instruct \

--quantization awq \

--max-model-len 16384 \

--gpu-memory-utilization 0.90 \

--max-num-seqs 64 \

--api-key local-only-key \

--port 8000

```

Key points:

  • Start --gpu-memory-utilization at 0.85 and leave headroom for KV cache growth
  • --max-num-seqs is your concurrency ceiling — derive it from VRAM and context length, don't copy defaults
  • For multiple GPUs add --tensor-parallel-size N where N is the GPU count
  • Always set --api-key, even on an internal network, so a stray script can't hammer it

Step 4: Add a gateway for routing and fallback

Put LiteLLM (or a thin proxy you write) in front of the local endpoint to do three things:

  1. Routing: simple tasks go local, hard tasks go to the cloud
  2. Rate limiting: even local endpoints need throttling so one script can't saturate the GPU
  3. Fallback: if the local service dies, traffic falls back to a cloud API transparently

In LiteLLM, define a local and a cloud deployment and chain them with fallbacks.

Step 5: The economics of quantization and context

  • Prefer INT4 (AWQ / GPTQ / GGUF Q4); quality loss is usually acceptable for summarization and classification
  • Context length directly drives KV cache footprint — if 8K works, don't open 32K
  • Enable prefix caching (vLLM's automatic prefix caching) for fixed-system-prompt workloads

Caveats

  • Encode quality boundaries in code: small local models are clearly weaker at long-chain reasoning and multi-step tool calls. Set thresholds in the routing layer so users don't hit the cliff
  • Don't expect consumer GPUs to handle long context: longer context degrades both VRAM headroom and latency, and real usability drops fast
  • Thermals and power are real bottlenecks: 24/7 inference is demanding; consider lowering the power limit for long-running setups
  • Compliance and data: local deployment keeps data inside your network, but model licenses (Apache 2.0, Llama community license, etc.) each have commercial terms — review them before shipping
  • Don't invent throughput numbers: actual tokens/s depends on model, quantization, batch size, and GPU. Benchmark yourself before setting an SLA

Where It Fits

  • Batch offline jobs: document summarization, corpus cleaning, label extraction, multilingual translation
  • Privacy-sensitive work: contracts, medical records, internal code that cannot leave the network
  • High-frequency, low-value requests: support intent classification, log triage, form validation
  • Shared team base: one stable OpenAI-compatible endpoint for the whole group, reserving cloud quota for hard tasks

Requirements and Limits

You need at least one GPU with 8GB+ VRAM (or an Apple Silicon unified-memory machine), tolerance for a one-tier quality drop, and basic Linux and container ops skills. If you have no hardware at all, local deployment isn't your best move — prioritize hosted inference free tiers instead.