Recover an unresolved execution

Forgeplane persists an execution attempt before dispatch and records a signed terminal outcome before acknowledging its queue message. A retry of the same authenticated outcome reuses the durable receipt; it does not run the tool again.

If a worker crossed the execution fence but no authoritative terminal outcome is available, the attempt becomes unresolved. This means the effect might have happened. Forgeplane keeps the Run running and does not automatically dispatch a replacement.

Recover safely

1. Record the Run, attempt, and occurrence IDs, timestamps, state, and safe reason code.
2. Inspect the target system and worker logs without copying credentials, raw tool output, or secret values into recovery notes.
3. Allow a late signed outcome to complete the original unresolved attempt when it is still authoritative.
4. If the outcome cannot be established, use the existing authorized cancel action. A run whose attempt is still executing becomes canceling until its worker reports; once the attempt is unresolved, cancel ends the run at once.
5. Decide outside Forgeplane whether repeating the external operation is safe.
6. Only then use the existing requeue action.

Requeue closes the unresolved attempt without erasing its time, reason, actor, or historical scope. It creates a separately admitted attempt attributed to the operator who requeued it. A stale result from the closed attempt cannot complete the new one.

Cancel and requeue are separate decisions. Requeue can repeat an external effect.

Operate the recovery path

• Retain worker public keys while attempts signed by those keys remain recoverable.
• Keep PostgreSQL execution attempts and receipts beyond audit-history cleanup. They are recovery state, not audit-history rows.
• A live worker retries the same observed terminal outcome until either JetStream or the coordinator accepts it. It does not run the tool again.
• A redelivered job for an executing attempt remains unacknowledged until durable evidence exists. A completed attempt with its receipt is safe to acknowledge.
• Treat outcome_persistence_unavailable as a retryable coordinator persistence failure. JetStream redelivers the same signed outcome until its receipt is committed.
• Do not remove fields from an oversized outcome envelope. Configure every NATS server with max_payload >= 4194304.

See Run operations, Workers, and Audit logging.