Why VRAM Budget Comes Before Model Choice

Local inference is "free" in the sense of zero marginal cost, but the cost does not disappear — it moves upfront into VRAM. On a given card, how much the weights occupy and how much KV cache is left determines your usable context length and concurrency, which in turn determines real tokens/s. So the first step is not picking a model but budgeting memory:

  • Weight footprint ≈ parameter count × bytes per parameter (FP16 ~2 bytes, Q8 ~1 byte, Q4 ~0.5 bytes, plus small overhead).
  • KV cache grows linearly with context length and concurrency; for long-context workloads it often exceeds the weights.
  • Leave roughly 1–2GB for the framework itself and CUDA context.

Based on this, the guide splits into three tiers, each with copy-pasteable configuration.

Steps

Tier 1: 8GB VRAM — everyday Q&A and light agents

Applies to: a single 8GB consumer card (e.g. RTX 3060 Ti / 4060), or a thin laptop with 16GB unified memory. Goal: run 7B–8B quantized models reliably.

  1. Pull a quantized build with Ollama; default tags are usually already Q4-class:

```bash

ollama pull qwen3:8b

ollama pull llama3.1:8b

```

  1. Adjust context and concurrency. Defaults are conservative; long-document work needs explicit settings:

```bash

# via environment variables

OLLAMA_NUM_PARALLEL=2 OLLAMA_CONTEXT_LENGTH=8192 ollama serve

```

  1. Freeze parameters in a Modelfile instead of retyping flags:

```text

FROM qwen3:8b

PARAMETER num_ctx 8192

PARAMETER num_predict 1024

PARAMETER temperature 0.3

```

```bash

ollama create qwen3-8b-ctx8k -f Modelfile

```

  1. Measure throughput: ollama run qwen3-8b-ctx8k --verbose and watch the eval rate. A 7B Q4 model on an 8GB card typically lands somewhere in the mid-teens to low-thirties tokens/s depending on card and context length.

Limits: this tier is not suited to long-context retrieval or high concurrency. Pushing context past 16K squeezes weights with KV cache and may trigger CPU offload, collapsing speed.

Tier 2: 24GB VRAM — multi-user sharing with vLLM

Applies to: a single 24GB card (RTX 4090 / L4 / A10) serving several clients at once, or running 14B–32B quantized models.

  1. Install vLLM in a dedicated virtualenv:

```bash

python -m venv venv && source venv/bin/activate

pip install vllm

```

  1. Launch in OpenAI-compatible mode, making memory utilization and concurrency limits explicit:

```bash

vllm serve Qwen/Qwen3-14B-AWQ \

--quantization awq \

--max-model-len 16384 \

--gpu-memory-utilization 0.90 \

--max-num-seqs 16 \

--port 8000

```

  1. Call it from clients using the OpenAI protocol with a local base_url:

```python

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="local")

```

  1. Tuning order: fix --max-model-len first, then adjust --max-num-seqs. Higher concurrency raises aggregate throughput but lengthens per-request latency; interactive workloads favor latency, batch workloads favor throughput.

Limits: AWQ / GPTQ require matching quantized weights. With FP16 weights only, 14B fits on 24GB but 32B is very tight. vLLM has GPU architecture requirements; older cards may lack support for some kernels.

Tier 3: Multi-GPU or large unified memory — long context and batch

Applies to: two or more 24GB cards, or a machine with 64GB+ unified memory (Mac Studio class), for 32B–70B quantized models or very long context.

  1. Multi-GPU tensor parallelism:

```bash

vllm serve Qwen/Qwen3-32B-AWQ \

--quantization awq \

--tensor-parallel-size 2 \

--max-model-len 32768 \

--gpu-memory-utilization 0.92

```

  1. On unified-memory machines, Ollama is simpler; control parallelism and residency via environment variables:

```bash

OLLAMA_NUM_PARALLEL=4 OLLAMA_MAX_LOADED_MODELS=1 ollama serve

```

  1. For long-context work, run a stress test first to confirm KV cache is not evicting weights. Send fixed-length inputs repeatedly and watch whether latency grows out of control with request count.

Limits: tensor parallelism adds communication overhead; poor interconnects scale badly. Unified memory bandwidth is far below dedicated VRAM, so generation speed at long context will be noticeably lower than a discrete-GPU setup of the same parameter count.

Caveats

  • Smaller quantization is not automatically better. Q4 loss is acceptable on most tasks, but code generation and math reasoning are more sensitive — use Q6/Q8 or FP16 for those.
  • Context length is a hidden cost. Raising num_ctx from 8K to 32K can multiply KV cache several-fold; set it to actual need.
  • Local deployment saves API fees, not electricity or hardware depreciation. Factor both in for sustained high load.
  • Framework defaults differ widely: Ollama is conservative, vLLM is throughput-oriented. Do not carry concurrency assumptions across when migrating.
  • Model licenses vary; check weight license terms before commercial use.

When to use which

  • 8GB tier: personal Q&A, light agents, draft generation in offline environments.
  • 24GB tier: a small team sharing one local endpoint, batch document processing, inference where private data must not leave the intranet.
  • Multi-GPU / large-memory tier: long-document analysis and summarization, complex tasks needing 32B+ quality, offline batch pipelines.

In one sentence: compute the slack left after weights + KV cache + framework overhead for your VRAM, then pick quantization tier and framework — that is how this local token supply line stays stable.