Why "Weights First" Instead of "Engine Split"

Most people stall at step one when running open models locally: where do the model files come from, where do they live, and can they survive a machine change? Treat weights as fixed assets that land first; the inference engine is merely a consumer. That gives you a token supply line independent of the network.

Core idea:

  1. Pull models with Ollama, but do not rely on its default hidden directory — explicitly set OLLAMA_MODELS to a path you control.
  2. Use ollama show --modelfile and ollama create to export already-downloaded weights into standard GGUF / safetensors references.
  3. Point vLLM directly at local quantized weights (AWQ / GPTQ / FP8), bypassing online download.
  4. Package the whole directory as a portable "weight vault" that boots a service on any machine or network.

Steps

Step 1: Pin Ollama's model store

```bash

export OLLAMA_MODELS=/data/ai-weights/ollama

mkdir -p "$OLLAMA_MODELS"

ollama pull qwen2.5:7b-instruct

ollama pull llama3.1:8b-instruct-q4_K_M

```

Write OLLAMA_MODELS into your shell profile or the systemd unit's Environment=. Otherwise the path falls back to the default after a restart and your downloads are wasted.

Step 2: Export weights as reusable files

```bash

ollama show --modelfile qwen2.5:7b-instruct > qwen.modelfile

ollama create my-qwen -f qwen.modelfile

```

ollama show --modelfile prints the blob path that FROM points to — that is the actual weight file. Copy it out:

```bash

mkdir -p /data/ai-weights/blobs

cp "$OLLAMA_MODELS/blobs/sha256-*" /data/ai-weights/blobs/

```

After this, even if you delete Ollama's index, the weight body remains and can be referenced by other tools.

Step 3: Load local quantized weights in vLLM

vLLM accepts a local directory as long as it contains config.json, tokenizer* and weight shards. Hugging Face's snapshot_download can pull the repo locally first (one network trip only):

```bash

huggingface-cli download Qwen/Qwen2.5-7B-Instruct-AWQ \

--local-dir /data/ai-weights/vllm/qwen2.5-7b-awq

```

Launch the server:

```bash

python -m vllm.entrypoints.openai.api_server \

--model /data/ai-weights/vllm/qwen2.5-7b-awq \

--served-model-name qwen-local \

--quantization awq \

--gpu-memory-utilization 0.85 \

--max-model-len 8192 \

--port 8000

```

Key point: give --model an absolute local path and vLLM will not touch the network.

Step 4: VRAM math (decides how big you can go)

Rules of thumb (7B-class models):

  • FP16: about 14 GB weights + KV cache
  • AWQ / GPTQ 4bit: about 4–5 GB weights + KV cache
  • FP8: about 7–8 GB weights + KV cache

--gpu-memory-utilization 0.85 leaves headroom for KV cache. A 24 GB card running 7B-AWQ can usually open 8K context; a 16 GB card should use --max-model-len 4096.

Step 5: Wire into existing workflows

vLLM's OpenAI-compatible endpoint means any client supporting a custom base_url can connect directly:

```bash

curl http://localhost:8000/v1/chat/completions \

-H "Content-Type: application/json" \

-d '{"model":"qwen-local","messages":[{"role":"user","content":"hello"}]}'

```

On the Ollama side, use http://localhost:11434/v1 as the compatible entry. Request formats match, so clients only change base_url and model name.

Caveats

  • Quantization format must match: vLLM's --quantization must equal the actual weight format (awq / gptq / fp8); a mismatch errors out or silently degrades.
  • Tokenizer must be same-source: the tokenizer in the local directory must come from the same repo version as the weights, or output turns to garbage.
  • Disk budget: one 7B FP16 is about 14 GB, quantized about 4–5 GB; a few models easily exceed 100 GB, so mount a dedicated disk.
  • Ollama's default dir gets overwritten: without an explicit OLLAMA_MODELS, upgrades or reinstalls may wipe weights.
  • vLLM vs CUDA: vLLM is sensitive to CUDA and PyTorch versions; validate in an isolated environment before upgrading.
  • Not every model has ready-made quantized weights: niche models need self-quantization, and the toolchain (AutoAWQ, llm-compressor) has a learning curve.

When to use

  • Network-restricted or metered environments (intranet, edge devices, offline labs).
  • Long-running, cost-predictable batch inference (data cleaning, bulk summarization, embedding generation).
  • Multiple machines sharing one weight vault, avoiding repeated downloads.
  • Compliance scenarios where data must not leave the premises.

Limitations

"Zero marginal cost" from local deployment has preconditions: a GPU, power, and ops effort. Capability is bounded by the open weights you download and still trails closed flagship models. Context length is constrained by VRAM, so long-document tasks need chunking.