Why Pooling, Not One Big Model on One Big Card

A common 2026 mistake in local deployment is setting the goal as "run a 70B on a single card." In reality most people have one to three cards with uneven VRAM (say 24GB, 12GB, 8GB). No single card can host a large model, but together they can serve several small and mid-size models at once, and split a large model across two cards with tensor parallelism.

The point of this setup is not "free" — it is turning idle capacity on hardware you already own into measurable Token throughput, where the only marginal cost is electricity. Whether it is worth doing depends on whether your call volume is steady; see "When This Fits" below.

Steps

Step 1: Inventory VRAM and draw an instance map

  1. Use nvidia-smi or rocm-smi to record VRAM and compute generation for each card.
  2. Estimate per-model footprint as "weights × quantization bits ÷ 8 + KV cache headroom." Rules of thumb: a 7B at Q4 is roughly 4–5GB of weights, Q8 roughly 7–8GB; a 14B at Q4 roughly 8–9GB; a 32B at Q4 roughly 18–20GB. KV cache grows linearly with context length; for long-context workloads reserve an extra 30%–50% of the weight size.
  3. Produce an instance map, for example:
  • Card A (24GB): vLLM, single-card serving of a 14B or 32B quantized model;
  • Cards B + C (12GB + 8GB): Ollama, serving 7B-class small models and embedding models for high-frequency light tasks such as routing, classification and summarization.
  1. Note: vLLM tensor parallelism wants cards of similar VRAM and model. Mixed heterogeneous cards often fail at startup or collapse in throughput. With heterogeneous cards, prefer one instance per card rather than forcing TP.

Step 2: Stand up the always-on small-model layer with Ollama

  • Use OLLAMA_HOST and OLLAMA_MODELS to set the listen address and model directory so multiple instances can coexist.
  • Use OLLAMA_NUM_PARALLEL to cap per-instance concurrency, and OLLAMA_KEEP_ALIVE to keep hot small models resident while letting cold ones expire so large models can claim VRAM.
  • The small-model layer does three jobs: request classification and routing decisions, embeddings and reranking, and fallback answers while the large model is queued.

Step 3: Stand up the large-model layer with vLLM

  • Four flags matter most: --tensor-parallel-size (equal to the number of participating cards), --gpu-memory-utilization (0.85–0.9 is a safe range, leaving system headroom), --max-model-len (set to your real workload context, do not max it out blindly), and --max-num-seqs (concurrency cap, which directly sets peak VRAM).
  • If official quantized weights exist (AWQ / GPTQ / FP8), prefer them over self-converted ones; they are more stable.
  • vLLM's continuous batching is the main source of throughput here: as concurrency rises, compute cost per Token drops noticeably.

Step 4: Add a reverse proxy for tiered scheduling

Put a lightweight proxy (Nginx, LiteLLM Proxy, or a small FastAPI service) in front of Ollama and vLLM to do three things:

  1. Route by model name: map the request's model field to a specific instance, exposing one unified endpoint.
  2. Degrade on queue depth: poll vLLM's metrics endpoint; when the queue exceeds a threshold, divert non-critical requests to the Ollama small-model layer.
  3. Rate-limit per caller: assign each internal caller a requests-per-minute and concurrency cap so one script cannot saturate the pool.

A workable degradation sketch:

```python

if vllm_queue_depth > THRESHOLD and not request.critical:

route_to("ollama/small-model")

else:

route_to("vllm/large-model")

```

Step 5: Keep books on capacity

  • At the proxy layer, log input/output Token counts, latency and the instance hit, and aggregate daily.
  • Only with this ledger can you judge whether pooling actually saves money: if total daily output is only tens of thousands of Tokens, depreciation and ops cost will likely exceed simply buying API access.

Cautions

  • Do not force tensor parallelism on heterogeneous cards. When model, VRAM and compute generation differ, prefer one instance per card and split models at the proxy.
  • Context length is the VRAM killer. Maxing --max-model-len lets KV cache eat all headroom and OOMs as soon as concurrency rises. Set it to your real workload length.
  • Consumer cards lack ECC, so error rates under sustained load are higher than datacenter cards. Keep an external API fallback for critical tasks.
  • Power and cooling are real costs. Multi-card always-on inference draws noticeable wattage; use OLLAMA_KEEP_ALIVE and instance sleep policies to avoid idle burn.
  • Read the model license. Some open weights carry extra commercial terms; internal use and serving to third parties are treated differently.
  • Never expose a bare port. Add authentication and rate limiting, or your pool becomes someone else's free API.

When This Fits

Fits: batch summarization, classification, embeddings and reranking in internal toolchains; latency-insensitive background jobs; scenarios where data cannot leave the intranet; teams with steady, sizable daily volume.

Does not fit: very low daily volume, a single low-end card, or workloads that genuinely need frontier-model capability — in those cases a cloud API is simpler and cheaper.

A Pragmatic Decision Line

Run it for two weeks, then look at the Token ledger from the proxy. If the local pool steadily produces hundreds of thousands of output Tokens per day and the work is mostly batch tasks that mid-size models handle well, pooling pays off. Otherwise, treat it as a compliance backup for "data stays on-prem" rather than a cost-saving measure.