Why Local Deployment Is the Only Truly Free Option

Every cloud free tier shares three limits: credits expire, rate limits apply, and terms change. Local deployment works on a different logic — you pay a one-time hardware cost plus electricity and get unmetered, unthrottled, offline inference in return.

The trade-off is equally clear: model capability is capped by your VRAM, and you own the ops burden. So this is not a replacement for frontier cloud models — it is a cost-to-zero strategy for high-frequency, low-sensitivity, batchable workloads.

Option 1: Ollama — Lowest-Barrier Local Inference

Ollama is currently the easiest local model runtime to get started with. It supports macOS / Linux / Windows, handles model pulls and quantized variants, and exposes an OpenAI-compatible endpoint.

Steps

  1. Install: download the installer for your platform from ollama.com; on Linux the official one-line script works fine.
  2. Pull a model: ollama pull qwen3:8b, ollama pull llama3.1:8b, ollama pull deepseek-r1:7b. The tag after the colon indicates size and quantization; the default is usually 4-bit.
  3. Run it: ollama run qwen3:8b for interactive use, or ollama serve to start the service on 127.0.0.1:11434.
  4. Wire it into your tools: Ollama exposes an OpenAI-compatible endpoint at http://localhost:11434/v1. Put any non-empty string as the API key. Most clients that let you set a custom base_url (IDE plugins, chat frontends, SDKs) connect directly.
  5. Keep it running: use a systemd unit on Linux, or launchd / the menu-bar startup toggle on macOS.

Hardware thresholds (rules of thumb, not hard limits)

  • 7B–8B at 4-bit: roughly 5–6 GB VRAM, or a 16 GB unified-memory Apple Silicon machine.
  • 14B at 4-bit: roughly 9–10 GB VRAM.
  • 32B at 4-bit: roughly 20 GB VRAM; a 24 GB consumer card is the usual entry point.
  • CPU-only works, but a 7B model typically produces only a few tokens per second — offline batch jobs only.

Limitations

  • Longer context means more VRAM; leave headroom for long-document tasks.
  • Weak concurrency; Ollama suits single-user, single-session use.
  • Model updates are manual (pull), with no auto-upgrade.

Option 2: vLLM — High-Throughput Batch Inference

If you need to process thousands of records at once (classification, extraction, rewriting, labeling), Ollama's throughput becomes the bottleneck. vLLM uses PagedAttention and continuous batching to raise throughput several-fold, sometimes more than ten-fold, on the same GPU.

Steps

  1. Environment: Linux + NVIDIA GPU recommended; the official Docker image is the least painful path. pip install vllm also works but is CUDA-version sensitive.
  2. Start the server:

```bash

vllm serve Qwen/Qwen3-8B \

--served-model-name qwen3-8b \

--max-model-len 8192 \

--gpu-memory-utilization 0.90 \

--port 8000

```

It exposes an OpenAI-compatible /v1/chat/completions endpoint.

  1. Tune it: --max-model-len caps context and directly drives VRAM use; --gpu-memory-utilization controls KV cache reservation; raise --max-num-seqs when throughput matters most.
  2. Batch calls: point the official openai Python SDK at your local address and fire requests concurrently with asyncio to saturate the GPU.
  3. Quantize if needed: AWQ / GPTQ weights, or FP8 on Hopper and newer architectures.

Limitations

  • Installation is sensitive to CUDA, driver, and PyTorch versions; mismatches are the most common source of errors.
  • First load takes tens of seconds to minutes — not suited to frequent restarts.
  • Multi-GPU requires correct tensor-parallel configuration, otherwise you hit OOM.

How to Choose

| Scenario | Recommendation |

| --- | --- |

| Personal Q&A, writing assistance | Ollama |

| Local IDE code completion | Ollama |

| Batch processing tens of thousands of records | vLLM |

| Shared inference service for multiple users | vLLM |

| Mac only, no discrete GPU | Ollama |

Cautions

  1. Never expose the service to the public internet. Neither Ollama nor vLLM authenticates by default; mapping the port is equivalent to giving away your GPU. Use an SSH tunnel or a reverse proxy with auth for remote access.
  2. Quantization has a cost. 4-bit losses are acceptable on most tasks but degrade noticeably on math reasoning and long-chain logic. Use 8-bit or full precision for sensitive work.
  3. Read the license. Open weights do not automatically mean free commercial use; some models ship custom terms. Verify before shipping.
  4. Electricity is a cost too. A 300 W card at full load for a day is roughly 7 kWh. Factor it in for sustained heavy use.
  5. Model choice beats parameter count. Same-size models differ widely in practice. Run a small benchmark on your own task rather than trusting leaderboards alone.

When This Fits

  • Compliance scenarios where data cannot leave the intranet.
  • Development phases with heavy prompt iteration and huge call volume.
  • Air-gapped environments, edge devices, offline batch jobs.
  • As a fallback for cloud APIs: when the cloud throttles or raises prices, the local service takes over immediately.

Bottom Line

If you have a usable GPU or a large-memory Mac, local deployment is the only approach that drives token cost down to "electricity only." Start with Ollama, scale with vLLM — both speak the OpenAI-compatible API, so switching costs are near zero.