Orchestration

The intuition

A hand-rolled while loop calling an LLM works until the process dies at step 7 of 10, a human must approve a spend, or two parallel branches both append to chat history. Then you discover you needed a state machine with a save game.

Orchestration frameworks (LangGraph is the common reference) treat the app as a graph: rooms (nodes) do work, doors (edges) choose where to go next, and a backpack (typed state) accumulates results. Checkpointing is the save slot after every room. Interrupts are pausing the game until a human returns. Memory is what stays in the backpack this session versus what goes into a filing cabinet for next month.

Why it exists

Raw agent loops hit operational walls quickly.

1. Crash and retry. Mid-run failures re-execute side effects unless state is snapshotted after each node. Long multi-tool runs are expensive to lose.
2. Human time ≠ compute time. Approvals can take hours. You need pause/resume without holding a server connection open — durable interrupts on a thread id.
3. Branching and fan-out. "If classified as X, go here; if Y, go there" and map-reduce over N subtasks need explicit, testable control flow and safe merges of concurrent state updates (reducers).
4. Context vs knowledge. The conversation must fit the window (short-term), while user facts must survive across sessions (long-term). Those are different stores with different write/read paths.

Alternatives — pure while-loops, or ad hoc Redis keys — reinvent half of this poorly. Full frameworks too early obscure prompts. The sweet spot: adopt graph orchestration when you concretely need checkpointing, branching, or interrupts.

The core idea

Once an LLM app has more than one step, you need orchestration: something that owns the control flow, the shared state, and the ability to pause and resume. The reference framework is LangGraph (LangChain's agent runtime, 1.0+): you model the application as a graph — nodes are units of work (an LLM call, a tool execution, plain code), edges define what runs next, and a typed state object flows through and accumulates results. Three properties make this the production choice over a hand-rolled while-loop: conditional edges give you explicit, inspectable branching (route on the model's output instead of burying if-statements in a loop); checkpointing persists state after every node, so a crashed or interrupted run resumes from the last step instead of restarting; and built-in interrupts pause the graph mid-run for human approval and resume later — which requires durable state, exactly what checkpointing provides.

The second half of orchestration is memory. Short-term memory is the conversation itself — thread-scoped, held in the checkpointer, bounded by the context window, so it needs active management: trimming, summarizing older turns, compaction. Long-term memory is cross-session — user preferences and facts in a persistent store, written by the agent and retrieved into context on later runs (retrieval-backed memory, mechanically the same trick as RAG). The one-liner: short-term memory is the transcript you resend; long-term memory is a database you query.

How it actually works

State schema: a TypedDict or Pydantic model. Each key can declare a reducer — the function that merges a node's returned update into existing state. Default is overwrite; add_messages appends (the standard for chat history). Reducers are what make parallel branches safe: two nodes updating the same key concurrently merge deterministically instead of clobbering.

class State(TypedDict):
    messages: Annotated[list, add_messages]  # append, don't replace
    draft: str                               # overwrite

builder = StateGraph(State)
builder.add_node("agent", call_model)
builder.add_node("tools", run_tools)
builder.add_edge(START, "agent")
builder.add_conditional_edges("agent", route)   # route(state) -> "tools" | END
builder.add_edge("tools", "agent")              # the agent loop, as a cycle
graph = builder.compile(checkpointer=PostgresSaver(...))

• Nodes are plain functions: take state, return a partial update. A node can contain an LLM call or no LLM at all.
• Edges: fixed (add_edge), conditional (a routing function inspects state and names the next node — this is where "the model decides" becomes explicit code), plus START/END. A cycle back to the agent node is literally the agent loop; a graph without cycles is a workflow. Two dynamic mechanisms: the Send API fans out N parallel copies of a node over a list (map-reduce, worker-per-subtask), and Command lets a node return both a state update and the routing decision in one object (the primitive behind handoffs).
• Checkpointing: the compiled graph with a checkpointer writes a snapshot of state after every node (a "checkpoint" on a "thread"). Invoke with a thread_id and you get: multi-turn conversation persistence (same thread continues), fault tolerance (resume from last checkpoint), time travel (rewind to any checkpoint, edit state, replay), and human-in-the-loop. MemorySaver for dev, Postgres/Redis/SQLite savers for prod.
• Human-in-the-loop: a node calls interrupt(payload) — the graph stops, state is checkpointed, your API returns the payload to a reviewer. Hours later, Command(resume=decision) on the same thread continues from that exact node. Approve/edit/reject tool calls before execution is the canonical use.

Short-term memory mechanics: the messages list grows every turn; you bound it by (a) trimming to the last N messages/tokens, (b) rolling summarization — replace older turns with an LLM-written summary, keep recent turns verbatim (what Claude Code's compaction does: preserve decisions and unresolved issues, discard stale tool outputs), (c) offloading — write large tool results to files/store, keep references in context.

Long-term memory mechanics: a store (LangGraph BaseStore) keyed by namespace (e.g., user id) holding JSON documents, optionally with embedding-based semantic search over them. Write paths: explicitly by the agent through a save_memory tool ("hot path"), or by a background job that extracts facts from finished conversations. Read path: fetch/search relevant memories at turn start and inject into the system prompt. Checkpointer = one thread's full state; store = durable facts across all threads.

On the governed enterprise platform, orchestration is what makes agent mode governable: every tool-using turn checkpoints under a thread id; write tools interrupt() for approvers; long-term store holds user preferences without stuffing them into every prompt forever.

The flows

A worked example

Employee chat on the governed enterprise platform: "Draft a facilities ticket to fix the badge reader on floor 12, then submit it."

State (simplified):

Trace:

Wall-clock human wait: hours. Compute held: none — thread sleeps in Postgres. Without checkpointing, step 4 cannot exist safely.

Context at turn 40 of the same thread: messages may exceed the window. Compaction replaces turns 1–30 with a ~500-token summary ("user wants badge reader fixed; ticket FAC-4412 submitted") and keeps 31–40 verbatim. Critical fact ticket_id is also written to long-term store under the user namespace so a new thread next week can retrieve it.

Common drill-downs

Questions that appear once you move past "we call the model in a loop" into durable multi-step systems.

Why a graph instead of a while-loop calling the LLM? The loop works until you need: resumability after a crash (checkpointing), pausing for human approval (interrupts), explicit branching you can test and visualize (conditional edges), parallel fan-out with safe state merging (Send + reducers), and streaming of intermediate steps. A graph makes control flow declarative and state transitions inspectable; a while-loop hides both.

What is a reducer and why does it matter? The merge function for a state key. Node returns {"messages": [new_msg]}; the add_messages reducer appends instead of overwriting. Matters most with parallel branches: concurrent updates to one key merge deterministically via the reducer rather than racing.

How does human-in-the-loop actually work in LangGraph? A node calls interrupt(payload); execution stops and state persists via the checkpointer under the thread id. The client surfaces the payload for review; on decision, invoking the graph with Command(resume=value) re-enters at the interrupted node with the human's value. Requires a checkpointer — no persistence, no pause.

Short-term vs long-term memory — where does each live? Short-term: the message history in graph state, thread-scoped, persisted by the checkpointer, bounded by context window — managed with trimming/summarization. Long-term: a cross-thread store (JSON docs, often with semantic search), written by the agent or a background extractor, retrieved into the prompt on later sessions.

Your conversation exceeds the context window at turn 60. Options? Trim (keep last N tokens — loses old info entirely), rolling summarization (summary of turns 1–45 + verbatim 46–60 — cheap, lossy), retrieval over full history (embed all turns, retrieve relevant ones per query — precise, more infra), or hybrid: summary + retrieval + pinned facts in long-term memory. State the tradeoff triangle: fidelity vs tokens vs complexity.

How do you resume an agent that died mid-run at step 7 of 10? With a checkpointer, invoke the graph on the same thread_id with input None: it loads the checkpoint after step 7 and continues. Caveat: node 7's side effects may have partially applied — nodes wrapping external mutations should be idempotent.

When is checkpointing overkill? Single-shot, sub-second, stateless pipelines (classify → extract → respond) with no HITL and no multi-turn state — a plain function chain is simpler and faster. Checkpointing earns its write overhead when runs are long, interruptible, or conversational.

Production concerns

• Checkpointer choice is a real ops decision: every node transition is a DB write. Postgres saver is the default at scale; measure write latency on hot paths — chatty graphs with tiny nodes amplify checkpoint overhead. In-memory savers lose everything on restart. See reliability.
• Unbounded state = unbounded cost. Message history grows per turn; without trimming/summarization a long-lived thread hits the context limit and per-turn cost climbs linearly until then. Summarize asynchronously where possible — inline summarization adds an LLM call to user-facing latency. See cost and tokens and context.
• Summarization is lossy: each compaction can drop a detail the user references later ("as I said earlier…"). Mitigate: keep recent turns verbatim, summarize only the tail, pin critical facts to long-term memory instead of relying on the summary.
• Time travel doubles as debugging and as replay-for-eval: rewind a bad production trajectory, edit state, replay to test a fix against the real failure. See evals and testing.
• Interrupt-based HITL holds state, not connections: the pattern is async by design — the thread sleeps in the checkpointer, so approvals can take days without holding compute. Your product layer owns notifying the human and mapping their decision to Command(resume=...). See agent reliability.
• Memory-store hygiene: stale or wrong memories poison every future conversation for that user. Add TTLs, dedupe on write, let users view/delete memories, and score retrieval relevance rather than dumping the whole namespace into context. Related: RAG production freshness and ACL patterns.
• Framework tradeoff: LangGraph buys persistence, HITL, and streaming for free but adds an abstraction layer between you and the prompts. Anthropic's advice stands — start with direct API calls; adopt a graph framework when you concretely need checkpointing, branching, or interrupts, and keep prompts inspectable.

Test yourself

Go deeper

• LangGraph Graph API — LangChain docs — StateGraph, nodes, edges, reducers, Send, Command.
• LangGraph persistence — LangChain docs — checkpointers, threads, stores, time travel, memory types.
• LangGraph interrupts — LangChain docs — interrupt()/Command(resume=...) and the approval patterns.
• Effective context engineering for AI agents — Anthropic — compaction, structured note-taking, and sub-agent context isolation as memory strategies.
• LangGraph Complete Course for Beginners — freeCodeCamp.org — 3-hour hands-on build of graphs and agents from scratch.

Where this connects

• Agent foundations — the logical loop that graphs reify as cycles and checkpoints.
• Harness vs orchestration — what this layer does not own, and why a graph never removes the need for tool design, truncation, and a context policy.
• Agent reliability — HITL gates, sandboxing, and why durable interrupts belong at the tool boundary.
• Multi-agent systems — Send/Command and supervisor patterns as multi-node graphs.
• Observability — tracing node transitions and thread ids so a multi-step run is reconstructable.