Why move inference on-prem

Platform credits expire, throttle, and vary by region. Open-weight models (Llama family, Qwen family, Mistral family, DeepSeek family, etc.) can be downloaded freely, so the only cost left is electricity and hardware depreciation. For individual developers and small teams, this turns token supply from "claim per use" into "produce on demand."

Steps

1. Check your hardware threshold first

  • 7B–9B at 4-bit quantization: runs on about 8GB VRAM; a 16GB-RAM CPU-only box can produce text slowly.
  • 14B–32B at 4-bit: usually needs 16GB–24GB VRAM.
  • 70B class: needs 48GB+ VRAM, or multi-GPU.

Pick the model to fit the hardware, not the other way around.

2. Ollama: fastest single-machine path

```bash

# after install, pull a quantized model

ollama pull qwen2.5:14b-instruct-q4_K_M

ollama run qwen2.5:14b-instruct-q4_K_M

```

Ollama exposes an OpenAI-like /v1/chat/completions endpoint on port 11434, which most SDKs can point at:

```bash

export OPENAI_BASE_URL=http://localhost:11434/v1

export OPENAI_API_KEY=ollama

```

Use for: solo development, prototyping, offline environments. Limit: weak concurrency, not suitable for multiple simultaneous users.

3. vLLM: turn the box into a servable endpoint

vLLM's PagedAttention and continuous batching raise throughput substantially, making it suitable for team sharing.

```bash

pip install vllm

vllm serve Qwen/Qwen2.5-14B-Instruct \

--quantization awq \

--max-model-len 8192 \

--gpu-memory-utilization 0.90 \

--port 8000

```

It also exposes an OpenAI-compatible endpoint:

```bash

export OPENAI_BASE_URL=http://localhost:8000/v1

export OPENAI_API_KEY=dummy

```

Key flags: --max-model-len caps context, --gpu-memory-utilization controls VRAM use, --max-num-seqs controls batch concurrency.

4. Add a gateway for rate limiting and metering

Wrap the local endpoint with LiteLLM or an Nginx reverse proxy:

  • Assign quotas per API key so one script can't saturate VRAM;
  • Log per-user token usage for internal chargeback;
  • Set timeouts and retries so long requests don't stall the service.

5. Monitoring and stability

  • Watch VRAM with nvidia-smi or nvtop;
  • vLLM exposes Prometheus metrics you can wire into Grafana;
  • Use systemd or Docker restart policies to auto-recover crashed processes.

Caveats

  • Quantization loses accuracy; 4-bit degrades noticeably on code and math tasks — use 8-bit or full precision for critical work.
  • Local models have no vendor content-safety layer; add your own filtering.
  • Electricity and hardware depreciation are real costs; sustained high concurrency is not always cheaper than an API.
  • Model licenses differ; check the weights agreement before commercial use.

When to use

  • Compliance scenarios where data cannot leave the intranet;
  • High-frequency, low-value batch jobs (classification, summarization, formatting);
  • Offline or weak-network environments;
  • Teams that need a fixed cost budget.

When not to use

  • Complex tasks that need the strongest reasoning — still call frontier closed models;
  • No GPU and latency-sensitive scenarios.