NE egorovnm.dev
Menu

Selected work / Case study

Ethernet & TCP diagnostics

Making intermittent delivery symptoms observable across an ESP32, its Ethernet interface, the network and the receiving service.

Overview

Individual TCP messages could be delayed, and several could arrive together after a pause. Intermittent ping failures were also observed. Network traces contained duplicate ACKs and spurious retransmissions. Those observations described the symptoms; they did not identify one definitive cause.

My role

I developed and modified ESP32 firmware in ESP-IDF for a LAN8720 Ethernet interface over RMII. My work included diagnostic firmware, a safe OTA update and rollback workflow, and analysis of communication among the ESP32, Linux/Jetson, network equipment and the receiving service.

Constraints

The fault was intermittent, and physical access to the remote system was not always available. Diagnostics and firmware updates had to be suitable for remote operation. Short captures alone were insufficient, so long-term telemetry and statistics mattered.

Architecture

Simplified architecture

  1. Application
    message creation
  2. Firmware queue
    handoff
  3. TCP/IP stack
    socket state
  4. MAC / RMII
    Ethernet interface
  5. LAN8720 PHY
    physical link
  6. Network & service
    receiver
Conceptual ESP32-to-service path, not a wiring diagram of the deployed system. Timestamps and state at several boundaries help localize a delay; one trace alone does not establish its cause.

Investigation

I added diagnostic firmware to collect network-related metrics and fields for connection state and time since recent receive activity. Instrumentation supported analysis of round-trip timing, errors, delays and network state. Diagnostic commands and test scenarios allowed comparison across firmware versions.

ESP32 telemetry was considered alongside Linux-side logs and network traces. Duplicate ACKs, retransmissions and delayed delivery were treated as evidence to correlate with timestamps and state, not proof that any single layer was at fault.

Engineering decisions

  • Collect state and timing at more than one boundary so a delay is not automatically attributed to firmware or the physical link.
  • Use safe OTA confirmation and rollback capability for remote diagnostic iterations.
  • Compare observations across firmware versions under planned test scenarios.

Illustrative state machine

  1. Ready
    known baseline
  2. Operation
    request active
  3. Timeout
    record evidence
  4. Classify
    application/queue · socket/TCP · MAC/RMII/PHY · network · peer/receiving service
  5. Bounded recovery
    least disruptive step
  6. Verify
    ready or error
Verified → ReadyNot restored → Error
Follow the numbered transitions in order. A successful verification returns to Ready; otherwise the model ends in Error. This is not a claim about exact deployed transitions.

Validation / current status

Diagnostic information was collected on both the ESP32 and Linux sides. The resulting logs made it possible to examine timing and communication over time and to separate plausible firmware, Ethernet, network and receiving-service failure domains. The investigation improved observability and safer diagnostic operation; it has not established a single cause for every delay or retransmission, and a complete fix is not claimed.

Lessons learned

When symptoms span several interfaces, useful telemetry needs timestamps, connection state and context at each boundary. A retransmission in a trace is a starting point for correlation, not a diagnosis by itself.

← All selected work