Why Skip Hosted Inference
Hosted inference (Inference API / Inference Endpoints) free tiers depend on account status, model size, and queue load. Quotas change, rate limits appear, requests queue up. The "self-hosted reserve" idea is to combine free GPU windows with open weights so your tokens come from your own process, not from someone else's quota pool.
Three components:
- HF Datasets: store your corpora, eval sets, and prompt templates. Versioned and diffable.
- HF Models: store weight references and your own inference script (
inference.pyand friends). - Notebooks / Colab: the runtime that actually produces tokens.
Steps
Step 1: Define what kind of "token supply" you need
- Batch generation: run a few thousand summaries/translations/classifications once and stop.
- Low-concurrency always-on: dozens to hundreds of calls per day, available at any time.
Batch work fits the Notebook approach best; always-on fits local Ollama/vLLM (covered elsewhere). This article focuses on batch and semi-always-on.
Step 2: Create an input repo on HF Datasets
Suggested layout:
```
inputs/
batch_001.jsonl
batch_002.jsonl
prompts/
summarize.txt
evals/
golden_50.jsonl
```
Read it with the datasets library instead of hand-rolling strings:
```python
from datasets import load_dataset
ds = load_dataset("your-name/token-inputs", data_files="inputs/*.jsonl", split="train")
```
The payoff is traceability: if one batch produces bad output, you can point directly to the source file.
Step 3: Put the inference script in an HF Models repo
Keep the script and weight reference together, and document model name, quantization, and batch size:
```python
# inference.py
from transformers import AutoModelForCausalLM, AutoTokenizer
MODEL = "Qwen/Qwen2.5-7B-Instruct" # swap for whatever small model is current
def load():
tok = AutoTokenizer.from_pretrained(MODEL)
mdl = AutoModelForCausalLM.from_pretrained(MODEL, device_map="auto", load_in_4bit=True)
return tok, mdl
```
Key rule: the script goes in the repo, the environment does not. Moving machines means reinstalling dependencies, not rewriting logic.
Step 4: Run it in a Notebook (free Colab tier)
Create a Colab notebook and mount your HF repos:
```python
!pip -q install transformers datasets accelerate bitsandbytes
from huggingface_hub import login
login() # use a read-only token, never a write-scoped one here
```
Then loop over batches and push results straight back to a dataset repo:
```python
from huggingface_hub import HfApi
api = HfApi()
api.upload_file(path_or_fileobj="out.jsonl",
path_in_repo="outputs/out.jsonl",
repo_id="your-name/token-outputs", repo_type="dataset")
```
Key free-tier Colab constraints (accept these first):
- GPU type is not guaranteed; you may get a T4, or no GPU at all on a given day.
- Sessions disconnect on idle; long jobs must be written to resume from a cursor. Upload results and record progress after each batch.
- VRAM is limited; a 7B-class model usually needs 4-bit quantization, and long contexts must be trimmed.
Step 5: Make it resumable
```python
import json, os
CURSOR = "cursor.json"
start = json.load(open(CURSOR))["i"] if os.path.exists(CURSOR) else 0
for i, row in enumerate(ds):
if i < start: continue
out = run(row)
append(out, "out.jsonl")
if i % 50 == 0:
upload("out.jsonl"); json.dump({"i": i}, open(CURSOR, "w"))
```
If Colab drops the session, you reconnect and continue from the cursor instead of starting over.
Step 6: Turn the supply into something callable daily
Once batch outputs are stored, wrap a thin FastAPI or Gradio layer and deploy it on a free HF Spaces instance as a "lookup + return on hit" service. That layer does no inference, only retrieval, so a free instance is usually enough and rarely fails from queueing.
Caveats
- Do not treat free GPUs as a production SLA: Colab's free tier can change policy at any time. Keep a local or paid fallback for critical jobs.
- Minimize token scope: use read tokens inside notebooks; use a separate token scoped only to the target repo when writing results back.
- Do not put sensitive data in public repos: HF public repos are visible by default. Use private repos and confirm your organization's compliance rules.
- Check model licenses: open-weight licenses differ on commercial use and redistribution. Read the model card before committing.
- Do not hardcode quota expectations: free resource policies at Colab and HF change. All capability statements here are "usually / approximately" and promise no specific hours or counts.
When This Fits
- Individual developers doing data labeling, batch summarization, offline translation, or eval-set generation.
- Small teams prototyping internal tools without buying API quota first.
- Teaching and research where reproducible input/output records matter.
- Not suitable for: low-latency online chat, high-concurrency production services, or anything needing committed QPS.
Cost Comparison
| Item | Self-hosted reserve | Hosted API |
|---|---|---|
| Cash cost | 0 (free GPU windows) | Per-token billing |
| Time cost | ~30-60 min initial setup | Sign up and go |
| Stability | Medium-low (depends on free windows) | High |
| Reproducibility | High (scripts + data in repos) | Depends on provider version |
Bottom line: treat the self-hosted reserve as a supplement and fallback to free quotas, not a replacement. Route batch, offline, delay-tolerant work here; keep hosted APIs for online, low-latency work.