Start with a path, not a label
“Communication failure” can mean that an application never queued a message, a driver never transmitted bytes, a peer did not answer, or a parser rejected a valid reply. A timeout reports a missed deadline; it does not identify which event was absent. Draw the actual path and record evidence at each boundary.
For a queued network sender, useful checkpoints are enqueue, dequeue, send attempt, socket result and peer observation. FreeRTOS queues can block when empty or full; that behavior makes queue wait time a separate question from network time. FreeRTOS queue documentation describes these semantics.
Keep layers distinct
- Transport: Did bytes leave the local interface, and did the connection remain usable? TCP retransmission is a transport mechanism, not by itself an explanation for an application delay. See RFC 9293.
- Protocol: Did received bytes form the expected response? A UART receive event, an AT response and an unsolicited result code are different observations. ESP-IDF exposes UART data and error events through its driver; see the ESP-IDF UART guide and the 3GPP AT command specification.
- Recovery: What did the system do after the deadline, and did service actually return? A retry that hides the original state can make the next incident harder to classify.
Record enough context to compare events
A compact event record can include a monotonic timestamp, operation identifier, queue state, connection or UART state, deadline, result class and recovery action. Keep raw payloads out of routine telemetry when they might contain private data. Correlate local events with peer-side logs using a clearly documented time basis; unsynchronized clocks can otherwise make an apparent ordering misleading.
Generic C-shaped event record — illustrative, not production code:
// Illustrative example, not production firmware
#include <stdint.h>
typedef enum { QUEUED, SENT, PEER_SEEN, TIMED_OUT } event_kind_t;
typedef struct {
uint32_t operation_id;
uint64_t monotonic_ms;
event_kind_t kind;
} diagnostic_event_t;The shared operation ID helps compare local stages; a peer observation still needs its own evidence and clock context.
Limits
These checkpoints are a diagnostic design, not a claim that a particular system exposes every counter. The meaning of a project-specific field must come from its code or an approved definition. No single counter, duplicate ACK or timeout proves a root cause without surrounding evidence.