Open Source & Self-Hosting

The intuition

Calling a hosted model API is like taking a taxi: you pay per trip, someone else owns the car, and when demand spikes you wait or pay surge. Self-hosting is buying (or renting long-term) the car: you pay whether it sits in the garage or runs all day, you control who rides and which roads you take, and you are responsible when the engine fails at 2 a.m.

The taxi wins for most trips — especially short, bursty ones. The owned car wins when you drive the same route every hour of every day at high volume, when the cargo cannot leave the property, or when you need a custom vehicle the taxi company will never offer. Open-weight models are the engines you can install in either picture: via a cheap third-party API (taxi fleet running open engines) or on GPUs you operate (your garage).

Why it exists

Hosted APIs are the default for good reasons, but four hard constraints keep pushing teams toward open weights and self-hosting:

1. Data residency and privacy. Prompts and document chunks that must never leave a VPC or a national boundary make public APIs a non-starter unless the provider offers a region-pinned, zero-retention, contractually bounded deployment — and sometimes even that is not enough.
2. Unit economics at sustained volume. A GPU bills per hour whether utilized or idle; an API bills per token with zero ops. At high, steady token rates against frontier API prices, owned capacity can win. Against budget open-weight APIs, break-even often collapses.
3. Latency and tenancy control. Shared multi-tenant inference has queueing variance you cannot tune. Co-locating the model with the data path and owning batch size / max concurrent sequences lets you trade TTFT for throughput deliberately.
4. Customization and independence. Custom fine-tunes, multi-LoRA adapters, constrained decoding, exotic sampling, no provider rate limits, no forced deprecations — capabilities that require holding the weights.

The alternatives lose in specific regimes. API-only forever fails regulated "prompts never leave the boundary" mandates. Self-host everything for cost fails when traffic is bursty or when a $0.15–0.50/MTok open-weight API already undercuts your fully-loaded GPU. Laptop Ollama in production fails under concurrency — continuous batching and PagedAttention exist for a reason. This page exists so the decision is an explicit TCO-plus-constraints analysis, not a fashion choice about "running local AI."

The core idea

Self-hosting is a fixed-cost-vs-marginal-cost decision plus a set of non-cost constraints. An API bills per token with zero ops; a GPU bills per hour whether you use it or not. A single H100 rents for roughly $2–4/hour (~$1.5–3K/month sustained) and serves an open-weight model at high throughput with vLLM — so break-even is purely a function of volume and which API you're comparing against. Against frontier APIs ($3–10 per million input tokens), sustained tens of millions of tokens per month starts to justify a GPU. Against budget open-weight APIs serving the same models (DeepSeek, Together, DeepInfra at ~$0.15–0.50/MTok), break-even runs into billions of tokens — self-hosting almost never wins on cost alone there. The honest default: APIs win for the large majority of production workloads.

So the real reasons to self-host are usually not cost: data residency and privacy (prompts never leave your VPC — sometimes a hard legal requirement), latency control (no shared-tenancy variance, model co-located with your data), customization (custom fine-tunes, LoRA adapters, constrained decoding, exotic sampling), and independence (no rate limits, no deprecations, no provider outage taking you down). The price is ops burden: GPU capacity planning, serving-stack upgrades, model updates, monitoring, and being your own on-call.

Stack in one line: Ollama for laptop/dev, vLLM for production GPU serving, Hugging Face as the model source and fine-tuning ecosystem.

How it actually works

The decision tree and the serving stack are two layers: first whether to self-host, then how to serve if you do.

The serving stack

• Ollama (dev/local) — one-command model runner (ollama run gemma4) built on llama.cpp; pulls quantized GGUF weights, exposes a REST API, runs on CPU or consumer GPUs across macOS/Linux/Windows. Optimized for single-user simplicity: little concurrency, no continuous batching, minimal metrics. Use it to prototype and to sanity-check that an open model can do the task at all.
• vLLM (production) — high-throughput inference server (Apache-2.0, out of UC Berkeley) supporting 200+ Hugging Face architectures, tensor parallelism across GPUs, quantized inference (FP8/INT4), speculative decoding, and an OpenAI-compatible API so app code doesn't change. Its two core mechanisms:
  • Continuous batching: new requests are inserted into the running GPU batch the moment a token slot frees up, instead of waiting for a fixed batch to drain — keeping the GPU saturated under concurrent load.
  • PagedAttention: manages the KV cache like OS virtual memory — fixed-size blocks instead of contiguous per-request allocations — cutting KV-cache memory waste from 60–80% (naive allocation) to under ~4%, which directly translates into more concurrent sequences per GPU.
  • The measured consequence: at a single user, Ollama and vLLM land in the same throughput band; at ~64 concurrent users, vLLM delivers roughly 8–9x the aggregate tokens/sec. That's the whole "dev vs prod" argument in one number.
• Hugging Face — the distribution and tooling layer: the Hub hosts weights (hundreds of thousands of models, with quantized variants), transformers loads them, peft handles LoRA adapters, and TGI is HF's own production server (vLLM-comparable role). "Where do you get the model and its fine-tune tooling" — the answer is Hugging Face. Adapter training depth: fine-tuning.

Quantization for inference

Quantization stores weights at lower precision to shrink memory and boost throughput, at a small quality cost.

• Rule of thumb for memory: FP16 ≈ 2 bytes/param (a 70B model needs ~140GB — multiple GPUs), INT8 ≈ 1 byte, 4-bit ≈ 0.5 bytes (~35GB — a single high-memory GPU). This is why 4-bit is the default for local inference.
• GGUF is the quantized single-file format used by llama.cpp/Ollama, with quality tiers (e.g. Q4_K_M, Q8_0) trading size against fidelity; designed for CPU+GPU mixed inference.
• Server-side (vLLM): FP8 and INT4/AWQ/GPTQ checkpoints for GPU-native quantized serving.
• Quality: 8-bit is near-lossless; modern 4-bit schemes typically lose a few points on hard benchmarks — acceptable for most product tasks, but eval it; degradation concentrates in math/code/long-context.

Open-weight landscape (July 2026 — verify before quoting)

The open-weight frontier is now dominated by MoE (mixture-of-experts) models from Chinese labs, most under genuinely permissive licenses:

The gap to proprietary frontier has narrowed to months, not years — but "open-weight quality X" is only relevant if X clears your eval bar at the size you can afford to serve. This table ages as fast as the proprietary one on choosing models — re-verify licenses, sizes, and scores before any commit.

The flows

A worked example

Consider the governed enterprise platform. Internal document Q&A and PII scrubbing cannot send raw prompts to a public multi-tenant API. Hard reasoning on already-redacted tickets might still use a region-pinned hosted frontier. Walk the economics and the stack choice for the high-volume classification / scrubbing route.

Volume assumptions (illustrative, typical scale): ~10–15k daily users; scrubbing + intent classification on the order of ~50M input-heavy tokens/month on that route alone (short prompts, tiny outputs).

Cost comparison for that route:

Against frontier pricing, 50M tokens/month is interesting but still small versus a full GPU bill once utilization and engineer time enter TCO. Against $0.20/MTok open-weight APIs, pure cost does not win — break-even is in the billions of tokens. The firm still self-hosts the scrubber because prompts never leave the boundary is a hard constraint, not a cost optimization.

Stack choice:

1. Shortlist open-weight models under MIT/Apache-class licenses that pass an internal eval set for classification + redaction quality (license first — Llama-class usage terms need a legal read).
2. Pick the smallest size that clears the bar (every extra billion parameters is permanent GPU memory).
3. Serve 4-bit or FP8 with vLLM behind the platform gateway (OpenAI-compatible URL so app code matches other routes).
4. Prototype the same weights on Ollama for offline developer laptops; never point production traffic at Ollama.
5. Cap max_model_len, alert on KV memory and queue depth, keep N+1 GPU capacity for failover.
6. Router (choosing models): scrub + classify → self-hosted small model; multi-step agent with tools → managed frontier inside the tenancy.

Memory sketch for a 70B-class candidate: FP16 ~140GB (multi-GPU); 4-bit ~35GB (single high-memory GPU). If evals pass at 8B–14B 4-bit, prefer that forever — throughput and cost scale with size more painfully than leaderboard ego admits.

Common drill-downs

When would you self-host instead of using an API? Hard privacy/residency requirements, sustained high volume against frontier-API pricing, need for custom fine-tunes or decoding control, or independence from provider limits. Not for cost at low volume — below tens of millions of tokens/month the API wins, and against budget open-weight APIs it almost always wins.

Walk me through the break-even math. GPU: ~$2–4/hr rented ≈ $1.5–3K/month sustained per H100. Compare monthly token volume × API price vs GPU cost ÷ realistic utilization. Against a $3/MTok API, ~50M+ tokens/month starts to be interesting; against a $0.20/MTok open-weight API, break-even is in the billions. Then add MLOps headcount to the self-host side.

What is continuous batching and why does it matter? Requests join and leave the GPU batch at token granularity instead of waiting for a fixed batch to finish — keeping the GPU saturated under concurrency. It's the main reason vLLM sustains ~8–9x Ollama's aggregate throughput at high concurrency while being no faster for a single user.

Explain PagedAttention in one minute. The KV cache (per-token attention state) is the GPU-memory bottleneck in serving. Naive engines allocate contiguous max-length buffers per request, wasting 60–80% of memory. PagedAttention allocates the cache in fixed-size blocks with an indirection table — like OS virtual-memory paging — cutting waste to under ~4% and enabling cache sharing across requests (e.g. common prefixes). More resident sequences per GPU = more throughput.

What does 4-bit quantization cost you? ~4x memory reduction vs FP16 (70B: ~140GB → ~35GB) and higher throughput, for typically a few points of quality on hard tasks — concentrated in math, code, and long-context recall. Near-lossless at 8-bit. Always re-run your evals on the quantized artifact, not the paper numbers.

Ollama vs vLLM — when each? Ollama: local dev, prototyping, single-user tools, CPU/consumer-GPU environments — simplicity wins. vLLM: production concurrency, SLAs, metrics, multi-GPU, OpenAI-compatible serving. Common pattern: prototype on Ollama, deploy on vLLM, same model weights.

How do you pick an open-weight model? License first (Apache 2.0/MIT — Qwen, DeepSeek, GLM — vs restricted), then the smallest size that clears your eval bar (bigger = more GPU forever), then ecosystem maturity (GGUF/AWQ availability, vLLM support, fine-tune tooling). Shortlist from current leaderboards, decide on your own evals.

Production concerns

• Utilization is the whole cost story. A GPU at 10% utilization costs 10x per token what the calculator promised. Bursty traffic favors APIs; steady sustained load favors owned capacity. Autoscaling GPU inference is hard — cold model loads take minutes. See cost.
• Total cost is more than the GPU-hour: engineer time for the serving stack, monitoring, upgrades (vLLM releases move fast), redundancy (N+1 GPUs for failover), and eval infrastructure for every model/quantization change. A realistic TCO comparison prices your MLOps time.
• Latency profile: self-hosting removes provider queueing variance and network hops, and lets you tune for TTFT vs throughput (batch size, max concurrent sequences, chunked prefill). But one hot request behind a saturated batch still queues — capacity planning replaces rate limits. See latency.
• Failure modes: OOM from KV-cache growth under long contexts (cap max_model_len, watch memory headroom), throughput collapse from a few very-long-context requests dominating the batch, silent quality regressions after quantization or engine upgrades (re-run evals), and GPU hardware faults (ECC errors, thermal throttling). Wire signals into observability and reliability.
• Privacy/residency is often the deciding constraint: healthcare/finance/government workloads where prompts can't leave the boundary make the cost math irrelevant — self-host (or a VPC-deployed provider offering) is the only option. Related: security.
• Hybrid is the common end-state: self-hosted small model for high-volume/low-stakes routes (classification, extraction, PII scrubbing), API frontier model for hard reasoning — the router (see choosing-models.md) spans both.

Test yourself

Go deeper

• vLLM — GitHub — the production serving engine; docs cover deployment, quantization, and parallelism.
• Ollama — GitHub — local model runner; README doubles as a snapshot of which open models people actually run.
• Efficient Memory Management for LLM Serving with PagedAttention (arXiv 2309.06180) — the vLLM paper; read §1–3 for the KV-cache argument.
• Unsloth docs — running and fine-tuning open models locally; practical GGUF/LoRA workflows.
• host ALL your AI locally — NetworkChuck — widely-watched end-to-end local hosting walkthrough (Ollama + web UI).

Where this connects

• Choosing models — self-hosted small models are another rung on the same routing ladder as cheap/fast API tiers.
• Fine-tuning — LoRA/QLoRA adapters are a primary reason to hold weights; vLLM multi-LoRA serving is how many adapters share one base.
• Cost — utilization, batch, and tiering decide whether owned GPUs ever beat tokens-as-a-service.
• Design an LLM gateway — one gateway fronting both VPC-hosted engines and managed model APIs with unified audit and quotas.