Why Colab + Spaces Instead of Signing Up for Yet Another Platform
The real bottleneck for free tokens was never the quota number — it is whether you can keep calling. Official free tiers rate-limit, expire, and sometimes require a card. Colab and Hugging Face Spaces do not give you tokens; they give you compute hosting. You bring your own open-weight model, and the number of calls you can make is bounded by the platform's usage policy and session limits, not by some vendor's promotional table.
The value of this route:
- No dependence on any single vendor's quota policy — the weights are yours.
- Colab offers free GPU sessions, Spaces offers free CPU hosting (GPU available under some plans).
- Both can expose HTTP via Gradio, and with a thin OpenAI-compatible wrapper they plug into any client that accepts a
base_url.
Below are two tracks: a temporary Colab endpoint and a persistent Spaces endpoint.
Steps
Track A: A Temporary OpenAI-Compatible Endpoint on Colab
Best for: you cannot run large models locally and need short-lived, higher-performance inference.
Step 1: Pick a runtime. In Colab, go to Runtime → Change runtime type and select the free GPU tier. The free tier is typically T4-class; the exact model and session length cap are adjusted by Google over time — check what the UI tells you when you open it.
Step 2: Install dependencies. In a notebook cell:
```python
!pip -q install fastapi uvicorn pyngrok transformers accelerate
```
Step 3: Load a model and serve an OpenAI-compatible API. Use transformers directly, or vllm (note: the free Colab environment is harsh on vLLM's VRAM and compilation requirements; it may not start cleanly on a T4, so transformers is safer):
```python
from fastapi import FastAPI
from pydantic import BaseModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch, uvicorn, threading
MODEL = "Qwen/Qwen2.5-1.5B-Instruct" # swap in your model
tok = AutoTokenizer.from_pretrained(MODEL)
mdl = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.float16, device_map="auto")
app = FastAPI()
class ChatReq(BaseModel):
model: str = MODEL
messages: list
@app.post("/v1/chat/completions")
def chat(req: ChatReq):
text = tok.apply_chat_template(req.messages, tokenize=False, add_generation_prompt=True)
ids = tok(text, return_tensors="pt").to(mdl.device)
out = mdl.generate(**ids, max_new_tokens=512)
reply = tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True)
return {"choices": [{"message": {"role": "assistant", "content": reply}}]}
threading.Thread(target=lambda: uvicorn.run(app, host="0.0.0.0", port=8000), daemon=True).start()
```
Step 4: Expose the port. Colab cannot serve a public port directly, so use a tunnel. pyngrok needs your own ngrok token (free signup), or you can use Cloudflare Tunnel's cloudflared:
```python
from pyngrok import ngrok
ngrok.set_auth_token("YOUR_NGROK_TOKEN")
print(ngrok.connect(8000).public_url)
```
The printed https://xxxx.ngrok-free.app is your base_url; in the client use base_url + "/v1".
Track B: A Persistent Gradio Endpoint on Hugging Face Spaces
Best for: a 24/7 lightweight endpoint that does not depend on your own machine.
Step 1: Create a Space. On huggingface.co click New Space, choose the Gradio SDK, and start with the free CPU hardware.
Step 2: Write app.py. Gradio automatically exposes your function as an HTTP API (every Space has a /run/predict or newer /gradio_api/call/... endpoint):
```python
import gradio as gr
from transformers import pipeline
pipe = pipeline("text-generation", model="Qwen/Qwen2.5-0.5B-Instruct")
def generate(prompt, max_new_tokens=256):
out = pipe(prompt, max_new_tokens=max_new_tokens, do_sample=True)
return out[0]["generated_text"]
gr.Interface(fn=generate, inputs=["text", "number"], outputs="text").launch()
```
Step 3: Call it from a client. Use gradio_client and skip writing HTTP yourself:
```python
from gradio_client import Client
c = Client("your-username/your-space")
print(c.predict("Write a line about autumn", 128))
```
Step 4 (optional): Add an OpenAI-compatible proxy. If you want to use the OpenAI SDK, run a thin proxy locally or on another box that translates /v1/chat/completions into gradio_client.predict calls. Then your client code does not change at all.
Track C: Combine Them into a Primary/Backup Path
- Route everyday requests to Spaces (always on, stable, free CPU is enough).
- When you need more compute, spin up a Colab session and put its ngrok URL in your config as a backup
base_url. - Add simple failover in the client: Spaces timeout → switch to the Colab URL.
Caveats
- Colab sessions get reclaimed. Free sessions disconnect after idle time and have a runtime cap (Google changes the exact number; trust the UI). That means Track A's endpoint is temporary — do not hardcode it in production code.
- Free CPU Spaces sleep. After a period without traffic they go to sleep, and the first request pays a cold-start delay, typically seconds to tens of seconds.
- Do not try to run huge models. On free CPU, 0.5B–3B class models are realistic; larger ones either fail to load or are unusably slow.
- Follow the terms of service. Colab explicitly prohibits using free compute for mining, proxy forwarding, or circumventing service limits; Spaces requires public Space code to be viewable. High-frequency commercial use of these endpoints may trigger rate limits or bans.
- Tunnel quotas are a separate limit. ngrok's free tier caps connections and concurrency, and Cloudflare Tunnel has its own policy — do not blame Colab for those.
- Do not send private data to a public Space. Inputs may be logged; for sensitive work use a private Space or self-host.
When to Use It
| Scenario | Suitable? | Notes |
|---|---|---|
| Personal learning, prompt debugging | Yes | Free tiers are plenty |
| Occasional calls from small tools/scripts | Yes | Cold-start delay is tolerable |
| A 24/7 product backend | No | Session reclaim and sleep make it unreliable |
| Large batch data processing | No | Limited throughput, may violate terms |
| Sensitive/private data | Caution | Prefer a private Space or local deployment |
One-Line Summary
Colab and Spaces do not hand you tokens; they hand you a hosting slot for your own model. Wire it into an OpenAI-compatible endpoint and you get a call path independent of any vendor's promo policy — at the cost of instability and no SLA, fit only for learning and light personal use.