The problem: Agent runs invoke LLM calls, tools, and memory operations that take time — and any of these processes can crash mid-run. Workers or endpoints can restart, your subscriber process can disconnect, and prompts get lost. Re-running the entire agent from scratch wastes tokens and time, and loses partially completed work.
The solution: The durable runtime records every step of the run as durable history. If the executing process crashes, completed steps are not re-run — the run resumes exactly where it left off. If your client process crashes mid-run, call
GetAgentRun (non-stream) or GetAgentStream + WithOffset (stream) to reconnect without restarting the agent. The public reconnect API is the same on Temporal and Restate.
These guarantees are independent:
Keep the run alive across disconnects
The durable runtime already records history. Your call-site context decides whether the live run is still there when you reconnect:- Pass a long-lived ctx to
Run/Stream(context.Background(), process-lifetime, orcontext.WithoutCancel(req.Context())) so disconnect or process shutdown does not cancel the durable run. - Cancel only
Get/Eventswhen the HTTP client leaves or the API is shutting down — that drops the subscriber, not the run. - Bound run length with
WithTimeout(or a long-lived deadline on the Stream/Run ctx). - Avoid passing a request/shutdown ctx into
Run/Streamif it will be cancelled when the handler exits — that stops the run before reconnect.
Server-side durability
The durable runtime records every step of the agent run — LLM calls, tool executions, approvals, memory operations — in durable history (Temporal workflow history, or Restate journaled steps). When the executing process crashes or restarts:- In-flight steps are automatically retried on a recovered or different executor
- Completed steps are not re-executed — the run fast-forwards through recorded history
- The final result is identical regardless of how many interruptions occurred
Step retry and streaming
When an LLM step retries (after a transient error or process crash), the LLM call restarts on the new attempt. The SDK detects this and signals any connected subscribers to discard token events from the failed attempt. Subscribers transparently receive clean token delivery from the new attempt — no duplicate or garbled output.Experiencing it
The Durable Agent (Temporal) example is a hands-on Temporal lab for this behavior (splitNewAgentWorker topology). Scenarios 3, 4, and 6 deliberately crash or stop workers mid-run and show:
- Runs completing after the worker that started them is killed
- Graceful and crash restarts with no data loss
- Multiple workers sharing a task queue
Client-side stream recovery
If YOUR process — the one consuming the event stream — crashes or disconnects, the run continues on the server side but your subscriber loses the live connection. UseGetAgentStream to resubscribe and resume from exactly where you left off.
The five-step protocol
1. PersistrunID before consuming events.
Offset() returns (offset int64, ok bool). When ok is false the event was emitted client-side by the SDK (e.g. RUN_STARTED) and has no stream position — do not use it as a resume point.
3. On restart, call GetAgentStream then Events with the saved offset.
savedOffset. Skip events at or below that point before resuming normal handling.
RUN_FINISHED or RUN_ERROR.
Reconnect requires a live run
GetAgentStream only works while the durable run is still executing. Once it completes, fails, or times out, the stream log is no longer available for replay. Calling GetAgentStream on a finished run returns ErrRunAlreadyCompleted — clear your saved state and continue from conversation/memory or start a new run.
ErrRunAlreadyCompleted does not mean the agent failed. The durable runtime completed the run — the LLM responded, tools ran, everything finished. What is lost is only the streaming view of those events. If you have WithConversation configured, the final response is already stored in conversation history and a follow-up turn will have full context.
Practical timing: reconnect only works while the run is still live. For short queries that complete in seconds, it may finish before you reconnect. After GetAgentRun / GetAgentStream, WithTimeout (if set) starts fresh — not the remaining time from the original Run / Stream. See Timeouts & Modes.
Approval events on reconnect
If an approval was already resolved while your subscriber was disconnected, the SDK filters it out automatically on reconnect — you will not be re-prompted for an approval that was already actioned.Client-side run recovery
If YOUR process started a non-streamRun and then crashed or exited before Get returned, the durable run continues on the server. Persist runID from AgentRun.ID() immediately after Run, then call GetAgentRun on restart to wait for the result:
GetAgentRun only works while the durable run is still executing. Once it completes, fails, or times out, you get ErrRunAlreadyCompleted — clear saved state and continue from conversation/memory. Cancelling the GetAgentRun / Get context does not cancel the run — use AgentRun.Cancel. After reconnect, WithTimeout starts fresh. See Run and Timeouts & Modes.
API reference
ErrRunAlreadyCompleted — returned when the durable run is no longer live. Clear saved state and start a new turn/run.
ErrRunNotFound — returned when runID is unknown or the runtime cannot reconnect (e.g. LocalRuntime after a crash).
ErrStreamOffsetNotSupported — returned by LocalRuntime for non-zero offsets (no durable stream log).
Runtime support
Examples
Durable Agent (Temporal)
Temporal lab: crash workers, kill processes, observe durability
Durable Agent (Restate)
Restate lab: single process, kill mid-stream, reconnect
Reconnect
Temporal demo of the shared reconnect API (same calls work on Restate)
Related
Distributed Execution
Split agent client and Temporal worker into separate processes
Run
GetAgentRun — reconnect a non-stream Run
Approvals
Human-in-the-loop tool approval and reconnect interaction
Streaming
RunID, event types, and the streaming API
Temporal Runtime
Temporal durability architecture and event delivery