Designing observable recovery paths

Recovery is a measured transition

A watchdog, retry or reset can restore service, but it can also erase the state needed to understand the incident. Record the trigger and recent relevant state before taking a disruptive action. Afterward, verify a defined health condition rather than treating “restart complete” as success.

Choose the smallest useful action

Classification can separate a missing application response, a parser error, a transport failure and a peer that is unavailable. The first recovery step should target the observed boundary. A full device reboot may be necessary in some systems, but a bounded retry or channel reinitialization may preserve more context when the evidence supports it.

Set limits on retries and record each attempt, its reason and outcome. If the same action repeatedly fails, stop cycling blindly and expose an error state for diagnostics. The exact thresholds must come from the product’s timing requirements and measurements, not from a generic diagram.

Generic recovery sketch — illustrative, not production code:

// Illustrative pseudocode, not deployed firmware
on_deadline(operation):
    save_event(operation.id, observed_state)
    boundary = classify_from_evidence()
    if retry_budget_available(boundary):
        recover_smallest_boundary(boundary)
        if verify_service_health(): mark_ready()
        else: report_error_with_context()
    else:
        report_error_with_context()

Design the evidence before the incident

• Capture a monotonic timestamp, operation ID, last known state and error class.
• Keep a small durable summary if power loss or reboot would erase the only trace.
• Distinguish “action attempted” from “service restored” in counters and logs.
• Test both successful recovery and repeated failure paths on representative hardware.

ESP-IDF documents task and interrupt watchdogs as mechanisms for detecting tasks or interrupts that fail to make progress; their behavior and configuration are platform-specific. See the ESP-IDF watchdog guide. Queue blocking behavior is documented by FreeRTOS.

Limits

The state machine above is illustrative. It is not the deployed firmware logic for any project, and it does not specify OTA confirmation deadlines, modem commands or safe retry counts. Those require implementation evidence and product-specific validation.