Why self-hosting is the most durable form of "free tokens"

Platform credits expire, get rate-limited, and change rules. Local inference has a different cost structure: it replaces per-token billing with one-time hardware plus electricity. As long as the machine runs, marginal token cost is near zero. In 2026, open-weight models in the 7B–32B range (Llama, Qwen, Mistral, Gemma, DeepSeek distills, and others) are good enough for summarization, rewriting, classification, extraction, code completion, and RAG Q&A — the bulk of everyday work.

The real question isn't whether it runs, but splitting light interactive use from high-throughput batch work into two tiers: Ollama for daily chat (simple, fast startup, easy model management) and vLLM for batch jobs (continuous batching, PagedAttention, several times the throughput). Both expose OpenAI-compatible endpoints, so existing clients need almost no changes.

Hardware thresholds and quantization choices

Work backwards from VRAM (rules of thumb; actual limits depend on context length):

| VRAM | Realistic models | Suggested quantization |

|---|---|---|

| 8 GB | 7B–8B, short context | Q4_K_M |

| 12–16 GB | 7B–14B, medium context | Q4_K_M / Q5_K_M |

| 24 GB | 14B comfortable, 32B marginal | Q4_K_M |

| 48 GB+ | 32B comfortable, larger context | Q5_K_M / Q6_K |

CPU-only works but a 7B Q4 model typically produces only a few tokens per second — fine for offline batch or low-frequency calls. Apple Silicon unified memory is a cost-effective path, and Ollama's Metal backend is mature.

Selection principle: pick "good enough", not "strongest". Use 7B–8B for classification/extraction, 14B–32B for writing and code, and prioritize context window over parameter count for long-document summarization.

Steps

Step 1: Build the daily inference tier with Ollama

  1. Install the build for your platform from ollama.com and confirm ollama --version works.
  2. Pull models based on your VRAM, e.g. ollama pull qwen3:8b, ollama pull llama3.3:latest, ollama pull mistral-small (check the official model library for current tags).
  3. Verify the local endpoint: it listens on http://localhost:11434 by default, with an OpenAI-compatible endpoint at http://localhost:11434/v1; any non-empty string works as the API key.
  4. Point any OpenAI SDK at that base_url and run one chat completion to confirm.
  5. To share on a LAN, set OLLAMA_HOST=0.0.0.0:11434 and restart. This exposes the service — always pair it with a firewall or an authenticating reverse proxy.

Step 2: Build the batch tier with vLLM

  1. Install vLLM on a machine with an NVIDIA GPU (an isolated Python environment or the official container image is recommended).
  2. Start the OpenAI-compatible server, for example:

```

python -m vllm.entrypoints.openai.api_server \

--model Qwen/Qwen3-14B-Instruct \

--served-model-name qwen3-14b \

--max-model-len 16384 \

--gpu-memory-utilization 0.90 \

--port 8000

```

  1. Hit http://localhost:8000/v1 and verify with the OpenAI SDK.
  2. If VRAM is tight, lower --max-model-len or --gpu-memory-utilization; insufficient KV cache reservation is the most common startup failure.
  3. Submit batch jobs in larger batches — continuous batching in vLLM raises throughput substantially, while single interactive requests are easier on Ollama.

Step 3: Add a simple router

Configure two providers in your client or a local gateway (e.g. a LiteLLM proxy):

  • Interactive, low-latency, single requests → Ollama
  • Offline batch, long documents, high throughput → vLLM

Route by task type rather than model name so each tier plays to its strengths and a small machine isn't crushed by batch jobs.

Step 4: Make "free" measurable

Local isn't costless — the cost is front-loaded. Track tokens per task, latency, and GPU occupancy. Use that data to decide whether a task belongs locally or should go to a cloud endpoint with free quota.

Caveats

  • Quality doesn't fully match closed frontier models. 7B–32B open models still lag on complex reasoning and long tool-call chains; keep human review or a cloud fallback for critical work.
  • Context length is a hard constraint. Large contexts on quantized models consume significant VRAM; estimate KV cache before long-document jobs.
  • Read the license. Most open weights allow commercial use, but some carry terms (MAU thresholds, attribution). Check the model card before deploying.
  • Electricity and thermals are real costs. A desktop at full load draws noticeable power; laptops throttle on long inference runs.
  • Never expose it to the public internet. Ollama and vLLM have no strong auth by default; open ports are a common breach vector.
  • Versions move fast. Model tags and vLLM flags change between releases; defer to current official docs.

Where it fits

  • High-frequency, low-value text processing: classification, tagging, cleaning, field extraction
  • Internal RAG Q&A where data must stay on-prem
  • Code completion and a local IDE assistant
  • Offline batch translation, summarization, rewriting
  • A fallback and traffic-splitter for cloud free tiers, so work continues when quota runs out

Where it doesn't fit

  • Complex tasks needing top-tier reasoning
  • Heavy multimodal work (runnable locally, but weigh quality against cost)
  • Temporary environments without stable power and cooling

Bottom line

Self-hosting isn't about grabbing credits — it turns tokens from a consumable into owned capacity. Ollama for daily use, vLLM for throughput, and a task-type router between them gives you a stable supply line in 2026 that no platform's rule change can take away.