Orchestration

The intuition

A hand-rolled while loop calling an LLM works until the process dies at step 7 of 10, a human must approve a spend, or two parallel branches both append to chat history. Then you discover you needed a state machine with a save game.

Orchestration frameworks (LangGraph is the common reference) treat the app as a graph: rooms (nodes) do work, doors (edges) choose where to go next, and a backpack (typed state) accumulates results. Checkpointing is the save slot after every room. Interrupts pause until a human returns. Memory is what stays in the backpack this session versus what goes into a filing cabinet for next month.

Why it exists

Raw agent loops hit operational walls. Mid-run crashes re-execute side effects unless state is snapshotted after each node. Approvals can take hours, so you need pause/resume without holding a server connection open. Branching and map-reduce need explicit control flow and safe merges of concurrent state updates. And the conversation must fit the window (short-term) while user facts must survive across sessions (long-term) — different stores, different paths.

Alternatives — pure while-loops, or ad hoc Redis keys — reinvent half of this poorly. Full frameworks too early obscure prompts. Adopt graph orchestration when you concretely need checkpointing, branching, or interrupts.

The core idea

Once an LLM app has more than one step, something must own control flow, shared state, and pause/resume. LangGraph models that as a graph: nodes are units of work (LLM call, tool run, or plain code), edges define what runs next, and a typed state object accumulates results.

Three properties beat a hand-rolled loop. Conditional edges make branching inspectable code. Checkpointing persists state after every super-step so a crashed or interrupted run resumes from the last step. Interrupts pause mid-run for human approval and resume later — which only works because the checkpointer held the thread. The other half is memory: short-term is the transcript you resend (thread-scoped, in the checkpointer); long-term is a database you query (cross-session store, same retrieval idea as RAG).

How it actually works

State and reducers. State is a TypedDict or Pydantic model. Each key can declare a reducer — the function that merges a node's partial update into existing state. Default is overwrite. add_messages appends (standard for chat history) and updates messages by id when you edit them. Reducers make parallel branches safe: concurrent updates merge deterministically.

class State(TypedDict):
    messages: Annotated[list, add_messages]  # append, don't replace
    draft: str                               # overwrite

builder = StateGraph(State)
builder.add_node("agent", call_model)
builder.add_node("tools", run_tools)
builder.add_edge(START, "agent")
builder.add_conditional_edges("agent", route)   # route(state) -> "tools" | END
builder.add_edge("tools", "agent")              # the agent loop, as a cycle
graph = builder.compile(checkpointer=PostgresSaver(...))

Nodes and edges. Nodes take state and return a partial update — with or without an LLM. Fixed edges always go A→B; conditional edges run a routing function that names the next node. A cycle back to the agent is the agent loop; no cycles is a workflow (same framework, different ownership of control flow — see agents vs workflows). Two dynamic primitives matter: the Send API fans out N parallel workers over a list (map-reduce), and Command returns a state update plus the next route — also the shape of Command(resume=...) after an interrupt.

Checkpointing and interrupts. With a checkpointer, the runtime snapshots state after every super-step on a thread. Pass a thread_id and you get multi-turn continuity, crash resume, time travel (rewind, edit, replay), and HITL. Use InMemorySaver for local dev; Postgres or another durable saver in production. A node calls interrupt(payload) to pause; hours later Command(resume=decision) on the same thread continues. Critical rule: on resume, the whole node restarts from the top. Side effects before interrupt() run again, so they must be idempotent (or live after the interrupt / in a later node).

Memory mechanics. Bound short-term history by trimming last-N tokens, rolling summarization (summary of the tail + recent turns verbatim — keep decisions and open issues, drop stale tool outputs), or offloading large tool results and keeping references. Long-term uses a store (BaseStore) keyed by namespace (e.g. user id), optionally with embedding search. Write on the hot path via a tool, or in a background extractor after the turn; read at turn start into the system prompt.

On the governed enterprise platform, every tool-using turn checkpoints under a thread id; write tools interrupt() for approvers; long-term store holds preferences without stuffing them into every prompt forever.

The flows

A worked example

Employee chat on the governed enterprise platform: "Draft a facilities ticket to fix the badge reader on floor 12, then submit it."

State: messages (add_messages), draft_ticket (overwrite), approved (overwrite).

Human wait: hours. Compute held: none — the thread sleeps in Postgres. Without checkpointing, step 4 cannot exist safely. Because resume re-enters the approve node from the start, any write before interrupt() must not create duplicates. At turn 40, compaction replaces turns 1–30 with a short summary ("badge reader fix; ticket FAC-4412 submitted") and keeps 31–40 verbatim; ticket_id is also written to long-term store so a new thread next week can retrieve it.

Production concerns

Checkpointer choice is an ops decision. Every super-step is a durable write. Chatty graphs with tiny nodes amplify DB round-trips — coarser nodes on hot paths and a tuned Postgres saver matter. In-memory savers lose everything on restart. Checkpoints also accumulate unboundedly on long threads; set retention or prune. See reliability.

Unbounded state is unbounded cost. Without trim/summarize, per-turn cost climbs with history. Prefer async summarization when you can — inline compaction adds an LLM call to user-facing latency. See cost and tokens and context.

Resume re-runs the node. Design every mutating node as if it can run twice: idempotency keys, upserts, or put side effects after the interrupt. Time travel (rewind, edit, replay) is the same mechanism for debugging and for eval fixtures — see evals and testing. HITL holds state, not connections: your product layer owns notifying the human and mapping their decision to Command(resume=...) — see agent reliability.

Memory-store hygiene. Stale memories poison every future conversation for that user. Add TTLs, dedupe on write, let users view/delete memories, and retrieve by relevance rather than dumping the namespace. Same freshness instincts as RAG production. Single-shot, stateless pipelines with no HITL should stay a function chain — adopt a graph when you need checkpointing, branching, or interrupts.

Common drill-downs

Why does resume re-execute code before interrupt()? Checkpoints land at super-step boundaries, not mid-function. On resume the runtime reloads state and re-enters the interrupted node from line one. Put non-idempotent side effects after the interrupt or in a later node.

Send vs Command — when each? Send is fan-out: a list of (node, state_slice) pairs so N workers run in parallel. Command is "update state and go here" (or Command(resume=...) as input after HITL). Do not mix static add_edge routing and dynamic Command(goto=...) from the same node — both can fire.

Conversation exceeds the window at turn 60. Options? Trim (last N), rolling summarization (older summary + recent verbatim), retrieval over full history, or hybrid: summary + retrieval + pinned facts in long-term memory. Tradeoff: fidelity vs tokens vs complexity.

Hot-path vs background memory writes? Hot path (save_memory during the turn): immediate, but adds latency. Background extractor after the turn: no user latency, but other threads may miss the fact until the job finishes. Often write critical preferences hot and bulk extraction cold.

Test yourself

Go deeper

• LangGraph Graph API — LangChain docs — StateGraph, nodes, edges, reducers, Send, Command.
• LangGraph persistence — LangChain docs — checkpointers vs stores, threads, time travel.
• LangGraph interrupts — LangChain docs — interrupt() / Command(resume=...), node re-entry, idempotency rules.
• Memory overview — LangChain docs — short-term vs long-term, hot-path vs background writes.
• Effective context engineering for AI agents — Anthropic — compaction, structured note-taking, sub-agent isolation.

Where this connects

• Agent foundations — the logical loop that graphs reify as cycles and checkpoints.
• Harness vs orchestration — what this layer does not own (tool design, truncation, context policy).
• Agent reliability — HITL gates and why durable interrupts belong at the tool boundary.
• Multi-agent systems — Send/Command and supervisor patterns as multi-node graphs.
• Observability — tracing node transitions and thread ids.