Overview
Individual TCP messages could be delayed, and several could arrive together after a pause. Intermittent ping failures were also observed. Network traces contained duplicate ACKs and spurious retransmissions. Those observations described the symptoms; they did not identify one definitive cause.
My role
I developed and modified ESP32 firmware in ESP-IDF for a LAN8720 Ethernet interface over RMII. My work included diagnostic firmware, a safe OTA update and rollback workflow, and analysis of communication among the ESP32, Linux/Jetson, network equipment and the receiving service.
Constraints
The fault was intermittent, and physical access to the remote system was not always available. Diagnostics and firmware updates had to be suitable for remote operation. Short captures alone were insufficient, so long-term telemetry and statistics mattered.
Architecture
Simplified architecture
Explanatory illustration · not live telemetry. Explore the stage roles and limits below.
All stage roles and limits
- 1 · Application
- Start at message creation. Compare this boundary with later observations; an application timestamp alone does not prove delivery.
- 2 · Firmware queue
- Separate the firmware handoff from subsequent transport activity. This conceptual boundary does not specify the actual queue implementation.
- 3 · TCP/IP stack
- Correlate connection state and timing with network traces. Duplicate ACKs or retransmissions do not establish a single root cause.
- 4 · MAC / RMII
- Keep the ESP32 Ethernet interface boundary distinct from the application and network. This diagram does not specify pin mappings or driver behavior.
- 5 · LAN8720 PHY
- The project used LAN8720 over RMII. Physical-link symptoms need surrounding evidence; intermittent ping failures alone do not isolate this layer.
- 6 · Network & service
- Compare ESP32 diagnostics with Linux-side logs and traces. Receiving-service observations help separate failure domains without proving that every delay was fixed.
Investigation
I added diagnostic firmware to collect network-related metrics and fields for connection state and time since recent receive activity. Instrumentation supported analysis of round-trip timing, errors, delays and network state. Diagnostic commands and test scenarios allowed comparison across firmware versions.
ESP32 telemetry was considered alongside Linux-side logs and network traces. Duplicate ACKs, retransmissions and delayed delivery were treated as evidence to correlate with timestamps and state, not proof that any single layer was at fault.
Engineering decisions
- Collect state and timing at more than one boundary so a delay is not automatically attributed to firmware or the physical link.
- Use safe OTA confirmation and rollback capability for remote diagnostic iterations.
- Compare observations across firmware versions under planned test scenarios.
Illustrative state machine
- Ready
known baseline - Operation
request active - Timeout
record evidence - Classify
application/queue · socket/TCP · MAC/RMII/PHY · network · peer/receiving service - Bounded recovery
least disruptive step - Verify
ready or error
Validation / current status
Diagnostic information was collected on both the ESP32 and Linux sides. The resulting logs made it possible to examine timing and communication over time and to separate plausible firmware, Ethernet, network and receiving-service failure domains. The investigation improved observability and safer diagnostic operation; it has not established a single cause for every delay or retransmission, and a complete fix is not claimed.
Lessons learned
When symptoms span several interfaces, useful telemetry needs timestamps, connection state and context at each boundary. A retransmission in a trace is a starting point for correlation, not a diagnosis by itself.