Overview
During migration from an EC25EU to an SLM770A modem, intermittent AT-command stalls and long MIP/TLS/MQTT operations required deeper diagnostics. ESP32 and Jetson/Linux could access the same AT channel, so contention had to be considered. Some failures appeared only after minutes or hours of operation.
My role
I worked on ESP32 firmware for LTE integration, UART/AT transport, MQTT/TLS communication, timeout and recovery logic, diagnostic builds and analysis of Linux-side logs. The work included testing several AT-handler approaches.
Constraints
Concurrent channel access made command ownership important. Some TLS and modem operations took longer than simple AT exchanges. Heavy logging could reduce CLI responsiveness, so diagnostics needed to remain useful without overwhelming the system.
Architecture
Illustrative AT exchange
Explanatory illustration · not live telemetry. Explore the stage roles and limits below.
All stage roles and limits
- 1 · Command owner
- Identify the active request and its start. ESP32 and Jetson could access the same AT channel; this illustration is not a claim that deployed arbitration prevents contention.
- 2 · UART transport
- ATE1 echo helped check whether ESP32 commands reached the modem. Receive activity can still exist during a transmit stall; silence alone does not prove a modem failure.
- 3 · Receive stream
- Observe responses and URCs as different evidence. Receive-only diagnostic builds could inspect this traffic without transmitting commands.
- 4 · Parser
- This is an illustrative classification step. It does not reconstruct the production parser or imply a specific AT-handler implementation.
- 5 · Deadline
- Review timeout evidence alongside RX/TX state. Long modem and TLS operations needed reviewed budgets; this diagram provides no exact timing or automatic recovery guarantee.
Investigation
I added UART RX/TX diagnostics, AT execution timing, timeout and deadline-overrun counters, drain-failure counters, URC logging and persistent runtime diagnostics. With ATE1 echo enabled, I could check whether a command appeared on the modem side. In some failure states it did not, while Jetson could still communicate with the modem. ESP32-side receive activity could continue even when its transmit path was effectively non-functional.
Linux logs also showed USB re-enumeration and ModemManager timeouts. These were additional observations, not proof that they caused the ESP32 transmit stall. Rebooting ESP32 restored communication in the confirmed stalled condition.
Engineering decisions
- A transparent AT handler was tested and later reverted because of duplication and contention risks.
- UART operation returned to 115200 baud as the stable working configuration.
- Timeout budgets for long modem and TLS operations were reviewed and extended.
- Recovery logic was developed to avoid unnecessary AT-channel resets; receive-only diagnostic builds could observe RX and URC traffic without sending commands.
Illustrative state machine
- Ready
known baseline - Operation
request active - Timeout
record evidence - Classify
TX · RX · parser · modem - Bounded recovery
least disruptive step - Verify
ready or error
Validation / current status
Echo, logs and counters distinguished “the modem did not respond” from “the ESP32 did not deliver the command.” Long-running diagnostics separated several failure classes, and a more robust timeout and recovery strategy was prepared. Observability and safer recovery improved, but the root cause of the ESP32 UART transmit stall remains under investigation. No uptime or stability improvement is claimed.
Lessons learned
For a shared command channel, observing both receive activity and the actual transmit path is essential. A response timeout alone cannot distinguish a modem failure, a parser issue, contention or a command that never arrived.