> ## Documentation Index
> Fetch the complete documentation index at: https://docs.agenticenv.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Durable Execution

> How agent runs survive crashes — automatic runtime resilience and client-side Run / Stream recovery

<Note>
  **Durable runtimes only (Temporal or Restate).** Durable execution — including step retries and `GetAgentStream` / `GetAgentRun` — requires the [Temporal](/runtimes/temporal) or [Restate](/runtimes/restate) runtime. The in-process `LocalRuntime` does not maintain a durable event log; non-zero offsets return `ErrStreamOffsetNotSupported`.
</Note>

**The problem:** Agent runs invoke LLM calls, tools, and memory operations that take time — and any of these processes can crash mid-run. Workers or endpoints can restart, your subscriber process can disconnect, and prompts get lost. Re-running the entire agent from scratch wastes tokens and time, and loses partially completed work.

**The solution:** The durable runtime records every step of the run as durable history. If the executing process crashes, completed steps are not re-run — the run resumes exactly where it left off. If your client process crashes mid-run, call `GetAgentRun` (non-stream) or `GetAgentStream` + `WithOffset` (stream) to reconnect without restarting the agent. The public reconnect API is the same on Temporal and Restate.

These guarantees are independent:

| Guarantee                       | What it means                                                                                                  | Who handles it                                |
| ------------------------------- | -------------------------------------------------------------------------------------------------------------- | --------------------------------------------- |
| **Server-side durability**      | The run and all its steps survive process crashes and restarts                                                 | Runtime — automatic, no SDK code needed       |
| **Client-side stream recovery** | Your subscriber process can reconnect after a crash and resume the event stream from exactly where it left off | Your code — `GetAgentStream` + `WithOffset`   |
| **Client-side run recovery**    | Your process can reconnect to a still-running non-stream `Run` and wait for the result                         | Your code — `GetAgentRun` then `Get` / `Done` |

## Keep the run alive across disconnects

The durable runtime already records history. Your call-site context decides whether the **live run** is still there when you reconnect:

* Pass a long-lived ctx to `Run` / `Stream` (`context.Background()`, process-lifetime, or `context.WithoutCancel(req.Context())`) so disconnect or process shutdown does not cancel the durable run.
* Cancel only `Get` / `Events` when the HTTP client leaves or the API is shutting down — that drops the subscriber, not the run.
* Bound run length with `WithTimeout` (or a long-lived deadline on the Stream/Run ctx).
* Avoid passing a request/shutdown ctx into `Run` / `Stream` if it will be cancelled when the handler exits — that stops the run before reconnect.

Full matrix: [Timeouts & Modes — What each context does](/advanced/timeouts-and-modes#what-each-context-does).

```go theme={null}
streamCtx := context.WithoutCancel(r.Context()) // run outlives the HTTP request
eventsCtx, cancelEvents := context.WithCancel(r.Context())
defer cancelEvents() // disconnect stops subscriber only

s, err := a.Stream(streamCtx, prompt, nil)
ch, err := s.Events(eventsCtx)
```

## Server-side durability

The durable runtime records every step of the agent run — LLM calls, tool executions, approvals, memory operations — in durable history (Temporal workflow history, or Restate journaled steps). When the executing process crashes or restarts:

* **In-flight steps** are automatically retried on a recovered or different executor
* **Completed steps** are not re-executed — the run fast-forwards through recorded history
* **The final result is identical** regardless of how many interruptions occurred

This is automatic. No SDK changes are needed beyond selecting Temporal or Restate.

| Runtime      | How steps are recorded             | Who executes after a crash             |
| ------------ | ---------------------------------- | -------------------------------------- |
| **Temporal** | Workflow history + activities      | Any worker polling the task queue      |
| **Restate**  | Durable Restate steps / awakeables | Any registered SDK endpoint deployment |

### Step retry and streaming

When an LLM step retries (after a transient error or process crash), the LLM call restarts on the new attempt. The SDK detects this and signals any connected subscribers to discard token events from the failed attempt. Subscribers transparently receive clean token delivery from the new attempt — no duplicate or garbled output.

### Experiencing it

The [Durable Agent (Temporal)](/examples/durable-agent) example is a hands-on Temporal lab for this behavior (split `NewAgentWorker` topology). Scenarios 3, 4, and 6 deliberately crash or stop workers mid-run and show:

* Runs completing after the worker that started them is killed
* Graceful and crash restarts with no data loss
* Multiple workers sharing a task queue

Restate provides the same durability guarantees through journaled steps and an embedded endpoint — see [Durable Agent (Restate)](/examples/durable-agent-restate) and [Restate runtime](/runtimes/restate). Restart the agent process (or another registered deployment) and Restate resumes from the journal.

## Client-side stream recovery

If YOUR process — the one consuming the event stream — crashes or disconnects, the run continues on the server side but your subscriber loses the live connection. Use `GetAgentStream` to resubscribe and resume from exactly where you left off.

<Warning>
  **Persist `runID` from the handle immediately.** Both `Stream` and `Run` return a handle whose `ID()` is available before events or `Get` complete. Persist that ID (and stream offsets) before consuming the channel so a mid-run crash can reconnect.
</Warning>

### The five-step protocol

**1. Persist `runID` before consuming events.**

```go theme={null}
// Stream ctx owns the run — cancel it only when you want to stop the agent.
// Use a separate Events ctx so a subscriber disconnect does not CancelWorkflow.
agentStream, err := a.Stream(ctx, prompt, nil)
if err != nil {
    return err
}
runID := agentStream.ID()
// Persist runID → DB, file, or cache — before reading events.
// It must survive a process crash.
saveRunID(sessionID, runID)

eventCh, err := agentStream.Events(eventsCtx)
if err != nil {
    return err
}
```

**2. Track the offset of each received event.**

Events from a durable stream carry a monotonic offset. Persist the last seen offset before processing each event — this is your resume point.

```go theme={null}
for ev := range eventCh {
    if ev == nil {
        continue
    }
    if ob, ok := ev.(interface{ Offset() (int64, bool) }); ok {
        if off, has := ob.Offset(); has {
            saveOffset(sessionID, off) // persist before processing
        }
    }
    handleEvent(ev)
}
```

`Offset()` returns `(offset int64, ok bool)`. When `ok` is `false` the event was emitted client-side by the SDK (e.g. `RUN_STARTED`) and has no stream position — do not use it as a resume point.

**3. On restart, call `GetAgentStream` then `Events` with the saved offset.**

```go theme={null}
savedRunID, savedOffset := loadSavedRun(sessionID)
agentStream, err := a.GetAgentStream(ctx, savedRunID)
if err != nil {
    if errors.Is(err, agent.ErrRunAlreadyCompleted) {
        // Workflow finished while disconnected — see below.
        clearSavedRun(sessionID)
        return
    }
    return err
}
resumeCh, err := agentStream.Events(ctx, agent.WithOffset(savedOffset))
if err != nil {
    return err
}
```

**4. Skip already-processed events, then resume normally.**

The stream may redeliver from `savedOffset`. Skip events at or below that point before resuming normal handling.

```go theme={null}
for ev := range resumeCh {
    if ev == nil {
        continue
    }
    if ob, ok := ev.(interface{ Offset() (int64, bool) }); ok {
        if off, has := ob.Offset(); has && off <= savedOffset {
            continue // already processed before the crash
        }
    }
    handleEvent(ev)
}
```

**5. Clear saved state on `RUN_FINISHED` or `RUN_ERROR`.**

```go theme={null}
case agent.AgentEventTypeRunFinished, agent.AgentEventTypeRunError:
    clearSavedRun(sessionID)
```

### Reconnect requires a live run

`GetAgentStream` only works while the durable run is still executing. Once it completes, fails, or times out, the stream log is no longer available for replay. Calling `GetAgentStream` on a finished run returns `ErrRunAlreadyCompleted` — clear your saved state and continue from conversation/memory or start a new run.

```go theme={null}
agentStream, err := a.GetAgentStream(ctx, savedRunID)
if errors.Is(err, agent.ErrRunAlreadyCompleted) {
    // The run completed successfully while you were disconnected.
    // The work was done — only the streaming view is lost.
    // If WithConversation is configured, the response is in history; start a new turn.
    // Otherwise, start a new run to get a fresh response.
    clearSavedRun(sessionID)
    return
}
```

**Important:** `ErrRunAlreadyCompleted` does not mean the agent failed. The durable runtime completed the run — the LLM responded, tools ran, everything finished. What is lost is only the **streaming view** of those events. If you have `WithConversation` configured, the final response is already stored in conversation history and a follow-up turn will have full context.

**Practical timing:** reconnect only works while the run is still live. For short queries that complete in seconds, it may finish before you reconnect. After `GetAgentRun` / `GetAgentStream`, `WithTimeout` (if set) starts **fresh** — not the remaining time from the original `Run` / `Stream`. See [Timeouts & Modes](/advanced/timeouts-and-modes#timeouts-after-reconnect-temporal).

### Approval events on reconnect

If an approval was already resolved while your subscriber was disconnected, the SDK filters it out automatically on reconnect — you will not be re-prompted for an approval that was already actioned.

## Client-side run recovery

If YOUR process started a non-stream `Run` and then crashed or exited before `Get` returned, the durable run continues on the server. Persist `runID` from `AgentRun.ID()` immediately after `Run`, then call `GetAgentRun` on restart to wait for the result:

```go theme={null}
agentRun, err := a.Run(ctx, prompt, nil)
if err != nil {
    return err
}
saveRunID(sessionID, agentRun.ID()) // before Get / Done

// ... process crash ...

savedRunID := loadSavedRunID(sessionID)
run, err := a.GetAgentRun(ctx, savedRunID)
if err != nil {
    if errors.Is(err, agent.ErrRunAlreadyCompleted) {
        // Run finished while disconnected — outcome is not on this handle.
        // Load from conversation/memory, or start a new run.
        clearSavedRun(sessionID)
        return nil
    }
    return err // including ErrRunNotFound
}
result, err := run.Get(ctx) // or <-run.Done() then Get
if err != nil {
    return err
}
clearSavedRun(sessionID)
fmt.Println(result.Content)
```

Same rule as stream reconnect: `GetAgentRun` only works while the durable run is still executing. Once it completes, fails, or times out, you get `ErrRunAlreadyCompleted` — clear saved state and continue from conversation/memory. Cancelling the `GetAgentRun` / `Get` context does not cancel the run — use `AgentRun.Cancel`. After reconnect, `WithTimeout` starts fresh. See [Run](/getting-started/run#reconnect-with-getagentrun) and [Timeouts & Modes](/advanced/timeouts-and-modes#what-each-context-does).

## API reference

```go theme={null}
// GetAgentStream returns a handle for a prior stream run (must still be live).
func (a *Agent) GetAgentStream(ctx context.Context, runID string) (AgentStream, error)

// Events resumes delivery; WithOffset skips already-consumed durable log entries.
func (s AgentStream) Events(ctx context.Context, opts ...AgentStreamOption) (<-chan AgentEvent, error)

func WithOffset(offset int64) AgentStreamOption

// GetAgentRun resumes a still-running non-stream Run by ID.
func (a *Agent) GetAgentRun(ctx context.Context, runID string) (AgentRun, error)
```

`ErrRunAlreadyCompleted` — returned when the durable run is no longer live. Clear saved state and start a new turn/run.

`ErrRunNotFound` — returned when `runID` is unknown or the runtime cannot reconnect (e.g. `LocalRuntime` after a crash).

`ErrStreamOffsetNotSupported` — returned by `LocalRuntime` for non-zero offsets (no durable stream log).

## Runtime support

| Runtime  | Server-side durability                    | Client-side reconnect                                              |
| -------- | ----------------------------------------- | ------------------------------------------------------------------ |
| Temporal | Automatic — workflow history + activities | `GetAgentStream` + `WithOffset`; `GetAgentRun` (while run is live) |
| Restate  | Automatic — journaled steps + awakeables  | Same public API as Temporal                                        |
| Local    | No                                        | Same-process handle only — no crash reconnect                      |

## Examples

<CardGroup cols={2}>
  <Card title="Durable Agent (Temporal)" icon="shield" href="/examples/durable-agent" horizontal>
    Temporal lab: crash workers, kill processes, observe durability
  </Card>

  <Card title="Durable Agent (Restate)" icon="cube" href="/examples/durable-agent-restate" horizontal>
    Restate lab: single process, kill mid-stream, reconnect
  </Card>

  <Card title="Reconnect" icon="rotate" href="/examples/reconnect" horizontal>
    Temporal demo of the shared reconnect API (same calls work on Restate)
  </Card>
</CardGroup>

## Related

<CardGroup cols={2}>
  <Card title="Distributed Execution" icon="gears" href="/advanced/distributed-execution" horizontal>
    Split agent client and Temporal worker into separate processes
  </Card>

  <Card title="Run" icon="play" href="/getting-started/run#reconnect-with-getagentrun" horizontal>
    GetAgentRun — reconnect a non-stream Run
  </Card>

  <Card title="Approvals" icon="circle-check" href="/features/approvals" horizontal>
    Human-in-the-loop tool approval and reconnect interaction
  </Card>

  <Card title="Streaming" icon="wave-pulse" href="/getting-started/streaming" horizontal>
    RunID, event types, and the streaming API
  </Card>

  <Card title="Temporal Runtime" icon="server" href="/runtimes/temporal" horizontal>
    Temporal durability architecture and event delivery
  </Card>
</CardGroup>