Open Source & Self-Hosting

The intuition

Calling a hosted model API is like taking a taxi: you pay per trip, someone else owns the car, and when demand spikes you wait or pay surge. Self-hosting is buying (or renting long-term) the car: you pay whether it sits idle or runs all day, you control who rides, and you own the 2 a.m. failure.

The taxi wins for most trips — especially short, bursty ones. The owned car wins at high steady volume, when the cargo cannot leave the property, or when you need a custom vehicle no taxi offers. Open-weight models — weights you can download and run yourself — fit either picture: a third-party API can serve them, or you load them on GPUs you operate.

Why it exists

Hosted APIs are the default. Four constraints still force open weights into the design: residency (prompts that must never leave a VPC or national boundary), volume economics (steady load against frontier API prices can favor owned GPUs; against budget open-weight APIs, break-even often lands in the billions of tokens), latency control (own the batcher so you can trade TTFT for throughput without multi-tenant queue variance), and customization (multi-LoRA, constrained decoding, no provider deprecations — all need the weights).

The alternatives fail narrowly. API-only forever fails hard residency. Self-host everything for cost fails on bursty traffic or when a cheap open-weight API undercuts your fully-loaded GPU. Laptop Ollama in production fails under concurrency. This page is the TCO-plus-constraints analysis that stops "local AI" from being a fashion choice.

The core idea

An API bills per token with zero ops; a GPU bills per hour whether you use it or not. A single H100 rents roughly $2–4/hour sustained (illustrative mid-2026 cloud range — re-check), about $1.5–3K/month, and serves open weights at high throughput with vLLM. Break-even depends on volume, utilization, and which API you compare against. Against frontier prices, sustained tens of millions of tokens/month can look interesting. Against budget open-weight APIs (often well under $1/MTok), pure cost almost never wins. APIs win for most production workloads.

Teams that self-host anyway usually need residency, latency control, custom adapters, or independence — not "open source is free." The price is ops: capacity planning, upgrades, monitoring, and being your own on-call. Stack in one line: Ollama for laptop/dev, vLLM for production GPU serving, Hugging Face as the model source and fine-tune ecosystem.

How it actually works

First decide whether to self-host; then choose how to serve if you do.

The serving stack

Ollama is a one-command local runner on llama.cpp. It pulls quantized GGUF weights (a single-file format for CPU and consumer-GPU inference), exposes a REST API, and optimizes for single-user simplicity. Use it to prototype. Do not put concurrent production traffic on it.

vLLM is a high-throughput inference server (Apache-2.0, originated at UC Berkeley) with multi-GPU tensor parallelism, quantized serving (FP8/INT4 and related schemes), multi-LoRA, and an OpenAI-compatible API. Two mechanisms explain the production gap:

Continuous batching (iteration-level scheduling) inserts new requests into the running GPU batch as soon as a token slot frees, instead of waiting for a fixed batch to drain. At one user, Ollama and vLLM often look similar; under dozens of concurrent users, published benchmarks commonly show vLLM several times — sometimes an order of magnitude — higher aggregate tokens/sec, while Ollama plateaus and queues grow.

PagedAttention manages the KV cache — the per-token key/value attention state that dominates GPU memory in serving — like OS virtual memory. Naive engines pre-allocate contiguous max-length buffers per request and waste most of that space when outputs are short. PagedAttention uses fixed-size blocks and an indirection table. The original paper reports cutting KV waste from the 60–80% range to near-zero, so more sequences fit per GPU. Prefix caching reuses blocks for shared prompt prefixes.

Hugging Face is the distribution layer: Hub weights, transformers, peft for LoRA, TGI as HF's production server. Adapter training: fine-tuning.

Quantization for inference

Quantization stores weights at lower precision to shrink memory and raise throughput. Rule of thumb: FP16 ≈ 2 bytes/param (70B ≈ 140GB — multi-GPU), 4-bit ≈ 0.5 bytes (~35GB — often one high-memory GPU). That is why 4-bit is the local default.

GGUF is the llama.cpp/Ollama format (tiers like Q4_K_M, Q8_0). On vLLM you typically serve FP8 or INT4 checkpoints from AWQ or GPTQ. Eight-bit is near-lossless; modern 4-bit usually costs a few benchmark points, concentrated in math, code, and long-context. Always re-run your evals on the artifact you ship.

Open-weight landscape (mid-2026 — verify before quoting)

Names age in weeks. Process does not: license first, smallest size that clears your evals, then ecosystem maturity (GGUF/AWQ, vLLM, fine-tune tooling).

Only models that clear your eval bar at a size you can serve matter. Re-check before any commit; same rot clock as choosing models.

The flows

A worked example

On the governed enterprise platform, PII scrubbing cannot send raw prompts to a public multi-tenant API. Hard reasoning on already-redacted tickets can still use a region-pinned frontier. Consider the high-volume scrub/classify route.

Volume (illustrative): ~10–15k daily users; ~50M input-heavy tokens/month on that route.

Against frontier pricing, 50M tokens/month is small once utilization and eng time enter TCO. Against $0.20/MTok open-weight APIs, pure cost does not win. The firm self-hosts because prompts never leave the boundary.

Stack. License first (MIT/Apache-class; Llama terms need legal). Smallest model that clears an internal classification/redaction eval. Serve 4-bit or FP8 on vLLM behind the gateway (OpenAI-compatible URL). Prototype on Ollama for laptops — never production. Cap max_model_len, alert on KV memory and queue depth, keep N+1 GPUs. Router (choosing models): scrub/classify → self-hosted small model; tool agents → managed frontier inside the tenancy. If 8B–14B 4-bit clears evals, stay there (70B FP16 ≈ 140GB; 4-bit ≈ 35GB).

Production concerns

Utilization is the cost story. At 10% util a GPU costs 10× per token vs the slide. Bursty traffic favors APIs; steady load favors owned capacity. Cold model loads take minutes, so aggressive scale-to-zero burns SLOs on the way up. Fold eng time, N+1, and eval infrastructure into TCO. See cost.

Latency changes shape. You drop provider queue variance and can tune TTFT vs throughput (batch size, max concurrent sequences, chunked prefill). You still queue when the batch is full — capacity planning replaces rate limits. A few long-context requests can starve the rest of the batch. See latency.

Failure modes under traffic. OOM from KV growth (cap max_model_len). Silent quality drops after quant or engine upgrades (re-eval the exact artifact). GPU faults (ECC, thermal). Serving-stack churn — pin vLLM and canary. Wire into observability and reliability. When prompts cannot leave the boundary, cost is secondary — self-host or a VPC-deployed offering (security). Hybrid is the usual end-state: small self-hosted model for high-volume low-stakes routes; API frontier for hard reasoning (choosing models).

Common drill-downs

When does break-even favor a GPU? Against frontier prices, with steady traffic, high real utilization, and eng time counted. Against budget open-weight APIs on the same family, almost never — residency or customization must carry the case.

What do you show when Ollama "feels as fast" as vLLM? Aggregate tokens/sec and success rate at your concurrency, plus p99. Single-user TTFT misleads; Ollama often plateaus while vLLM scales until the GPU saturates.

How do you pick an open-weight model to serve? License → smallest size that clears your evals → ecosystem (GGUF/AWQ, vLLM, fine-tune tooling). Leaderboards shortlist only.

VPC private endpoint vs self-host — still hold weights? Not always. A region-pinned, contractually zero-retention endpoint can meet residency without your GPUs. Self-host when policy forbids third-party inference, you need custom adapters the vendor will not host, or independence is the requirement.

Test yourself

Go deeper

• vLLM — GitHub — production serving; deployment, quantization, parallelism.
• Ollama — GitHub — local runner; which open models people actually run.
• PagedAttention paper (arXiv 2309.06180) — KV-cache argument; start §1–3.
• Anatomy of a High-Throughput LLM Inference System — continuous batching, PagedAttention, prefix caching, serving stack.
• Unsloth docs — local run/fine-tune; practical GGUF/LoRA workflows.

Where this connects

• Choosing models — self-hosted small models as another rung on the routing ladder.
• Fine-tuning — LoRA/QLoRA is a primary reason to hold weights; vLLM multi-LoRA serves many adapters on one base.
• Cost — utilization and tiering decide whether owned GPUs beat tokens-as-a-service.
• Design an LLM gateway — one gateway for VPC engines and managed APIs with unified audit and quotas.