NE egorovnm.dev
Menu

Selected work / Case study

LTE modem & AT command diagnostics

Separating a silent modem from a command path that did not deliver bytes to the modem.

Overview

During migration from an EC25EU to an SLM770A modem, intermittent AT-command stalls and long MIP/TLS/MQTT operations required deeper diagnostics. ESP32 and Jetson/Linux could access the same AT channel, so contention had to be considered. Some failures appeared only after minutes or hours of operation.

My role

I worked on ESP32 firmware for LTE integration, UART/AT transport, MQTT/TLS communication, timeout and recovery logic, diagnostic builds and analysis of Linux-side logs. The work included testing several AT-handler approaches.

Constraints

Concurrent channel access made command ownership important. Some TLS and modem operations took longer than simple AT exchanges. Heavy logging could reduce CLI responsiveness, so diagnostics needed to remain useful without overwhelming the system.

Architecture

Illustrative AT exchange

  1. 1 · Command ownerESP32 queues one AT request and records its start time.
  2. 2 · UART transportBytes are sent; echo, when enabled, can independently show whether the modem saw them.
  3. 3 · Receive streamA response may arrive while an unrelated unsolicited result code (URC) is also reported.
  4. 4 · ParserClassify response and URC separately, then match the final result to the active request.
  5. 5 · DeadlineIf the expected result does not arrive, record the timeout and observed RX/TX state before recovery.
Conceptual sequence only. It does not encode modem-specific commands, exact time budgets, or the deployed firmware state machine.

Investigation

I added UART RX/TX diagnostics, AT execution timing, timeout and deadline-overrun counters, drain-failure counters, URC logging and persistent runtime diagnostics. With ATE1 echo enabled, I could check whether a command appeared on the modem side. In some failure states it did not, while Jetson could still communicate with the modem. ESP32-side receive activity could continue even when its transmit path was effectively non-functional.

Linux logs also showed USB re-enumeration and ModemManager timeouts. These were additional observations, not proof that they caused the ESP32 transmit stall. Rebooting ESP32 restored communication in the confirmed stalled condition.

Engineering decisions

  • A transparent AT handler was tested and later reverted because of duplication and contention risks.
  • UART operation returned to 115200 baud as the stable working configuration.
  • Timeout budgets for long modem and TLS operations were reviewed and extended.
  • Recovery logic was developed to avoid unnecessary AT-channel resets; receive-only diagnostic builds could observe RX and URC traffic without sending commands.

Illustrative state machine

  1. Ready
    known baseline
  2. Operation
    request active
  3. Timeout
    record evidence
  4. Classify
    TX · RX · parser · modem
  5. Bounded recovery
    least disruptive step
  6. Verify
    ready or error
Verified → ReadyNot restored → Error
Follow the numbered transitions in order. A successful verification returns to Ready; otherwise the model ends in Error. This is not a claim about exact deployed transitions.

Validation / current status

Echo, logs and counters distinguished “the modem did not respond” from “the ESP32 did not deliver the command.” Long-running diagnostics separated several failure classes, and a more robust timeout and recovery strategy was prepared. Observability and safer recovery improved, but the root cause of the ESP32 UART transmit stall remains under investigation. No uptime or stability improvement is claimed.

Lessons learned

For a shared command channel, observing both receive activity and the actual transmit path is essential. A response timeout alone cannot distinguish a modem failure, a parser issue, contention or a command that never arrived.

← All selected work