Why this angle

Most local-deployment guides obsess over "4-bit or 8-bit". Quantization tiers only answer "does it fit"; they do not answer "is the model resident when the request arrives". In real use the most expensive cost is not VRAM but the tens of seconds to minutes of cold-start loading — during which users get no tokens, and the team drifts back to cloud APIs.

So the goal here is not squeezing VRAM but keeping a hot model always online and letting a cold model appear on demand, pushing first-token latency into an acceptable range.

Steps

Step 1: Keep a "daily-tier" small model resident with Ollama

Ollama's keep_alive parameter controls how long a model stays loaded after the last request. The default is 5 minutes, which is short for daily chat.

```bash

# Keep a 7B/8B-class model resident to avoid repeated loading

curl http://localhost:11434/api/generate -d '{

"model": "qwen2.5:7b-instruct",

"prompt": "ping",

"keep_alive": -1

}'

```

keep_alive: -1 means never unload proactively. Use it for small models that fit in VRAM and are called frequently. If VRAM is tight, use a time value like "keep_alive": "30m" so it yields after idle time.

Key point: pick a "good enough" size for the resident model. Its job is to absorb ~80% of everyday requests (summarization, classification, rewriting, simple Q&A) and leave the large model for tasks that truly need it.

Step 2: Manage the "on-demand tier" vLLM with systemd

Starting a 32B-class model in vLLM usually takes tens of seconds to a couple of minutes (depending on weight-loading speed and whether --enforce-eager is used). Do not run python -m vllm.entrypoints.openai.api_server bare in a script; use a systemd unit so the router can wake it on demand and reclaim it when idle.

```ini

# /etc/systemd/system/vllm-large.service

[Unit]

Description=vLLM large model (on-demand)

After=network.target

[Service]

Type=simple

ExecStart=/opt/venv/bin/python -m vllm.entrypoints.openai.api_server \

--model /models/Qwen2.5-32B-Instruct-AWQ \

--served-model-name large \

--port 8000 \

--gpu-memory-utilization 0.85 \

--max-model-len 8192

Restart=on-failure

[Install]

WantedBy=multi-user.target

```

Key points:

  • Fix --served-model-name to large so the router does not care about the weight path;
  • Leave headroom in --gpu-memory-utilization to avoid OOM when competing with the resident small model;
  • Use Restart=on-failure to catch load failures.

Step 3: Write a "wake + wait-until-ready" hook

Before forwarding to vLLM, the router probes /v1/models. If unreachable, run systemctl start and poll until ready.

```python

import subprocess, time, requests

def ensure_large(timeout=180):

url = "http://127.0.0.1:8000/v1/models"

try:

requests.get(url, timeout=2)

return True

except Exception:

pass

subprocess.run(["systemctl", "start", "vllm-large"], check=False)

deadline = time.time() + timeout

while time.time() < deadline:

try:

requests.get(url, timeout=2)

return True

except Exception:

time.sleep(2)

return False

```

Note: the first wake-up request waits a long time. Give it its own longer timeout at the router layer, and do not let every request trigger a wake-up — only requests classified as needing the large model take this path.

Step 4: Add an idle reclaim timer

The on-demand tier must not hold VRAM forever, or the resident small model gets squeezed out. Use a systemd timer or a simple daemon script to systemctl stop vllm-large after N minutes without requests.

```bash

# Rough example: check last request time, stop if exceeded

systemctl stop vllm-large

```

The reclaim threshold depends on your usage rhythm. Teams are busy by day and quiet at night — stop at night, keep alive by day.

Step 5: Two-tier routing

A minimal routing rule:

  1. model: small or unspecified → hit Ollama's OpenAI-compatible endpoint (http://localhost:11434/v1) directly;
  2. model: large or length/complexity above a threshold → call ensure_large() first, then hit vLLM's /v1/chat/completions;
  3. Large-model wake-up fails → downgrade to the small model and log it, so supply never drops.

Clients then only need one base_url; behind it are two supply lines: resident + on-demand.

Caveats

  • VRAM is shared: compute total occupancy for resident and on-demand models together. In practice, resident small model + large model with --gpu-memory-utilization 0.85 is more stable on multi-GPU machines; on a single GPU, shrink the small model or switch to a time-based keep_alive.
  • Cold-start time cannot be eliminated: weight loading is heavily disk-bound. Put models on NVMe or use --load-format-related options to shorten wake-up noticeably.
  • Do not count wake-up time as user experience: async jobs (batch processing, offline summarization) suit the on-demand tier; interactive chat should stay on the resident tier.
  • Keep Ollama and vLLM ports separate and configure them explicitly in the router to avoid default-port conflicts.
  • Log everything: wake-up count, wake-up duration, downgrade count — these three metrics decide whether the setup is worth keeping.
  • Model names, versions, and licenses follow each project's official repository; confirm weight licenses before commercial use.

When it fits

  • The team has many lightweight daily requests (classification, extraction, rewriting) and occasionally needs a large model for hard reasoning;
  • Only one or two GPUs, not enough VRAM to keep two models resident;
  • Sensitive to first-token latency but tolerant of "a complex task can wait a bit";
  • Want to cut cloud API calls and turn token cost into fixed electricity and hardware depreciation.

How it differs from quantization-tier plans

Quantization-tier plans answer "how large a model fits on one card"; this article answers "when should a model be in VRAM". They are not in conflict: choose your quantization tier by VRAM budget first, then use the resident/on-demand layering here to decide who stays online and who appears on demand.