Why a Shared Token Supply Pool?

A single-machine local model deployment can only serve one user and cannot fully utilize hardware resources. By combining Ollama and vLLM into a multi-user shared pool, teams can centrally manage models, allocate quotas, and achieve zero-marginal-cost token supply.

Architecture Overview

  • Inference Backends: Ollama for lightweight models and rapid prototyping, vLLM for high-throughput production models.
  • API Gateway: Use LiteLLM or One API as a unified proxy to provide OpenAI-compatible endpoints, authentication, and rate limiting.
  • Resource Quotas: Set requests per minute (RPM) and token quotas per user or team via the gateway.
  • Monitoring & Metering: Use Prometheus + Grafana for metrics, or leverage gateway built-in logs.

Steps

1. Deploy Inference Backends

Ollama Installation & Configuration (Linux):

```bash

curl -fsSL https://ollama.com/install.sh | sh

# Modify service config to allow LAN access

sudo systemctl edit ollama.service

# Add: Environment="OLLAMA_HOST=0.0.0.0"

sudo systemctl restart ollama

```

Pull models:

```bash

ollama pull llama3.2:3b

ollama pull qwen2.5:7b

```

vLLM Installation & Startup:

```bash

pip install vllm

# Start OpenAI-compatible server

python -m vllm.entrypoints.openai.api_server \

--model Qwen/Qwen2.5-7B-Instruct \

--served-model-name qwen2.5-7b \

--host 0.0.0.0 \

--port 8000

```

2. Deploy API Gateway (LiteLLM as Example)

```bash

pip install litellm[proxy]

```

Create config file config.yaml:

```yaml

model_list:

  • model_name: ollama-llama3.2

litellm_params:

model: ollama/llama3.2:3b

api_base: http://localhost:11434

  • model_name: vllm-qwen2.5

litellm_params:

model: openai/qwen2.5-7b

api_base: http://localhost:8000/v1

api_key: dummy

general_settings:

master_key: sk-1234

```

Start gateway:

```bash

litellm --config config.yaml --port 4000

```

3. Configure Multi-User & Quotas

Create users and keys in LiteLLM with budget and rate limits:

```bash

curl -X POST http://localhost:4000/user/new \

-H "Authorization: Bearer sk-1234" \

-H "Content-Type: application/json" \

-d '{"user_id": "team-a", "max_budget": 10, "rpm_limit": 60}'

```

The returned key can be distributed to team members to access the unified endpoint http://<server-ip>:4000.

4. Monitoring & Metering

Enable LiteLLM logging and Prometheus metrics:

```yaml

# Add to config.yaml

litellm_settings:

drop_params: true

success_callback: ["prometheus"]

```

Access http://localhost:4000/metrics for metrics and configure Grafana dashboards.

Precautions

  • Hardware Requirements: 7B model inference requires at least 16GB GPU memory; vLLM needs CUDA environment.
  • Network & Security: Production environment should configure HTTPS and firewall to avoid public exposure.
  • Model Licenses: Ensure selected open-source models allow commercial use (e.g., Apache 2.0, MIT).
  • Concurrency Limits: Adjust vLLM's --max-num-seqs and --gpu-memory-utilization based on GPU capability.
  • Quota Management: LiteLLM budget unit is USD, adjust according to actual cost (zero here); rate limits should be set based on hardware performance.

Use Cases

  • Small to medium teams for internal use, wanting unified model access and cost control.
  • Development and testing environments needing quick switching between open-source models.
  • Educational or research institutions providing shared inference resources for multiple projects.

Limitations

  • Local token supply is limited by hardware performance and cannot scale indefinitely.
  • Requires server maintenance and model updates, with some operational overhead.
  • Not suitable for extremely latency-sensitive scenarios or very large models (e.g., 70B+) unless adequate hardware is available.