Skip to content

Release v1.9.0: multi-pilot C5, Ethernet, RotorHazard plugin 2, race director - #22

Draft
LouisHitchcock wants to merge 115 commits into
mainfrom
V1.9.0-alpha
Draft

LouisHitchcock wants to merge 115 commits into
mainfrom
V1.9.0-alpha

Conversation

@LouisHitchcock

@LouisHitchcock LouisHitchcock commented Oct 25, 2026 •

Copy link
Copy Markdown
Owner

Brings the 1.9.0 multi-pilot line into main. The source is labelled 1.9.0-rc.5. Qualification remains incomplete; this PR stays draft.

Implementation completed

  • ESP32-C5 multi-pilot: up to eight pilots with individual channels, names, colours and calibration; standings, announcements, saved history, RSSI recording and per-pilot Visual Marshal editing. C5 updates run through the gate; gate-managed max-lap/heat-time stopping and saved history work without an open browser.
  • Hardware and networking: Waveshare ESP32-S3-ETH Ethernet and C5MK AIO support; maintained AsyncTCP/WebServer libraries, receive/disconnect and SSE serialization fixes, retained HTTP bytes, responsive history loading and correct history download routing. Flash packages include the OTA boot selector.
  • RotorHazard plugin 2: per-pilot passes, 20 Hz/full-resolution RSSI, receiver settings and peak-sample pass timing. The gate preserves queued passes when the link refreshes after the caller sampled its clock, and reports sent/lost-pass counters.
  • Race director and controls: Event Mode pilots, groups, formats, schedules, channel assignment and seeding; ELRS Backpack start/stop follows binding identity and server-side countdown settings. Race screen analysis boxes and charts, browser start/stop announcements and serialized buzzer playback.
  • Community features: Beep Every Lap and Say Lap Time Too in browser announcements; Merge in the typed lap editor, preserving total race time.
  • Crash diagnostics: a thread-safe 16 KB log ring retained across software resets/crashes; optional SD logging off by default; C5 Receiver Checks, Save Log to SD Card and Download log.txt in Diagnostics.
  • Ethernet restart fix: a newer link timestamp could underflow the DHCP timeout calculation and move a running gate to 192.168.8.1 almost immediately. Signed elapsed time fixes the race. The faulty calculation predates rc.5; recovered SD logs show the affected boot continued running for 21 minutes. Investigation and validation.
  • SD warm-reset recovery: backported Espressif's two-CMD0 reset handshake with settling delays into a build-local Waveshare SD driver copy. The installed SDK is untouched; no platform upgrade or formatting. The card's invalid reset responses were captured with temporary diagnostics, excluded from the production app. Three new tests cover recovery, failure refusal, source guards and SDK isolation.
  • Validation tooling: interrupted acceptance runs wait for each lane to restore baseline state; evidence renames tolerate transient file locks; a network probe distinguishes fresh-connection failures from broader outages.

Validation

  • 37 Python tests and 166 Jest tests passed. Patched WaveshareS3ETH firmware built and installed by app-only OTA; configuration, history, Event and C5 receiver 1.2.5 preserved.
  • DHCP regression fails against the old unsigned calculation and passes against the fix, including stale clock readings, timeout boundaries and clock wrap.
  • SD reset-handshake backport built and tested: 40 normal-image restarts passed (20 logging on, 20 off), preserving DHCP and all saved data. All logging-on boots mounted first try; three logging-off boots needed retries, worst ready time 51.41 s. Original failures remain recorded in the investigation; these results do not replace soak qualification.
  • AIO rc.5 completed ten consecutive restarts, confirmed by boot counters, at about 74 seconds each.
  • Complete the eight-hour dual-lane soak. Original run failed after 3 h 49 min at Ethernet heat 373 (/timer/start ConnectTimeout). Retry rehearsal was STOPPED at 21:57 GMT on AIO ping failures and heapLow 2748 bytes; both baselines restored, overnight NOT launched. Fail-fast correction 3f233ae now records/stops on harness, network-monitor or existing 40000-byte heap-floor failures, permits restoration, and blocks phase promotion. Eleven stop/restore tests pass. No test is currently running.
  • Previous physical ELRS validation covered the correct binding ID, ten switch flips without missing/duplicate commands, Event review refusal, idle stop handling, configured countdown, one saved history record and browser starting/stopping TTS.
  • A forced RotorHazard link-refresh regression lost zero of 2,633 passes after the fix. The original rehearsal's missing crossing remains unproven.

Scope and completed receiver work

C5MK 1.2.5 is published and locked; clean board images, asset hashes and physical OTA/failed-trial rollback reporting are recorded. Final gate/plugin combination acceptance remains separate. The new C5 status LED targets after 1.2.5 and is outside this release. Preserve the accepted timing baseline; advanced camera/multipath/default-timing research must not silently become a new release blocker. Physical ELRS edge cases, LEDs/webhooks/audio, browser USB flashing and real-card SD writer/repair checks still need evidence.

Race-control retry candidate

The plugin now pools race-control connections and retries only failures before a connection is established. Read timeouts and response errors are not replayed, avoiding duplicate commands. Four transport tests and eleven tuning tests pass. Recovered connection retries are logged. This is a local companion-plugin candidate, not a published release; unchanged gate networking and heap remain under independent monitoring. The retry records AIO heap minima alongside the workload.

Before merging

  • Incorporate current main and preserve the PR branch's README update.
  • Decide scope of HDZero goggle OSD via ELRS backpack (ESP-NOW): last lap, best 3 conse… #21 (HDZero OSD).
  • Diagnose and patch the recorded premature Ethernet DHCP fallback.
  • Finish qualification of SD warm-reset recovery. Backport 0877102 passed 40 restart checks; recovered mount retries remain explicit. Complete the active endurance run and retain/document startup limits. Physical no-DHCP fallback validation is still pending.
  • Complete functional RF/storage recovery checks and physical smoke for the agreed board scope: four-board RC delivery versus the six-board stable roadmap. Advanced camera/strict timing work remains follow-up under the 29 October accepted-baseline decision unless C5MK 2.0 is adopted.
  • Disposition the older Ethernet fresh-connection stall/reset separately from the fixed boot DHCP race.
  • Resolve or constrain self-test interference with C5 telemetry, and disposition the earlier F05 Event start/advance timeouts.
  • Fix or explicitly accept the minor requestsNow/activeFiles diagnostic counter drift.
  • Investigate AIO USB interruptions/low heap. AIO SD is currently working; the earlier detection issue was not established to have a firmware cause.
  • Close or document remaining Event/non-C5 browser dependencies and the original missing-crossing finding.
  • Final version, changelog, matching six-board packages and release notes per docs/RELEASE_PROCESS.md.

Source is pushed to both V1.9.0-alpha and release/v1.9.0-rc.4. Local generated candidate bundles and private bench backups are not published. No release or flasher deployment is included.

The receiver choice now saves: fromJson rejected receiverRadio 2, so the
page fell back to RX5808 after a refresh and the C5 link never started.

The Calibration tab gets one C5 panel: link status, a band picker that
fills the 8 pilots, the RF gain, a chart of every pilot with live labels,
and a card per pilot with its frequency and its own Enter/Exit values.
The selected pilot's thresholds can be dragged on the chart. Edits save
themselves. Auto-calibration is kept.

Firmware: the pilot list is saved as all 8 slots so clearing one sticks;
the link UART starts even with no pilots and skips empty slots instead of
stalling on them; the C5 fields are range-checked on load (older configs
migrated with them uninitialised); the web stream sends all 8 slots, who
is in the gate and the link state, 10 times a second. Default C5 gain 30.
The web stream runs in a task on the other core and takes its own
millis() first; a line stamped by the main loop a moment later made the
unsigned 'time since last line' wrap, so the C5 showed as offline whenever
a line landed in between, which at 1000 lines/s was most of the time.
Compare signed instead.

The lap queue is filled by the main loop and drained by that same web
task, so guard it with a spinlock.
With C5 firmware 2 the S3 sends the slot list once and parses the C5's
binary scan records next to the status lines, mapping the C5's dump
timestamps onto micros(). The link is serviced by c5Task (core 1, above
loop) with bulk UART reads; per-byte reads had starved it. Measured:
~1030 Hz per pilot for eight pilots, no gaps or drops (was 116 Hz in
16 ms bursts). Older C5 firmware keeps the slot cycle.

Passes are timed at the filtered RSSI peak. The slot on the
Configuration pilot frequency is the race pilot: its crossings go to
LapTimer::recordCrossing(), so Gate 1, minimum lap, the lap table,
announcer, RotorHazard and race history work as with the RX5808.

The Calibration tab shows live link figures (mode, per-pilot rate,
gaps, drops, bad records, S3 poll gap) and marks the race pilot. The
validation tool reports the per-pilot rate.
Each C5 slot gets a pilot name, spoken name, colour and Race switch on
its Calibration card (config v24; the EEPROM grows from 832 to 1280
bytes, which the EEPROM library expands in place). Migration keeps
every setting and puts Race on the slot on the main frequency, so an
existing single-quad setup races as before.

With one racer the race still runs through LapTimer. With two or more,
C5MultiPilot times each racer with LapTimer's rules (Gate 1 from the
start, pass to pass, minimum lap, timed at the RSSI peak) and sends a
c5Lap event per lap; GET /api/c5/race rebuilds a race for pages opened
mid-race. The Race tab shows a column per pilot, announces "name, time",
lights the gate in the lapping pilot's colour, finishes each pilot at
max laps and saves every pilot to race history.

The RSSI debug popout now works with the C5: while it is open the S3
sends c5RssiFast (25 a second, each slot's peak since the last frame)
and the page draws a line per pilot with its thresholds.

The 5 s heap report reads the heap's total size instead of walking it.
docs/C5_MULTI_PILOT.md described the old 20 ms slot protocol. It is now
a user guide: wiring, setup, supported channels (5180-5917 MHz), the
Calibration tab and its link figures, pilot cards and thresholds,
single- and multi-pilot races, the RSSI debug window and
troubleshooting. The internal handoff (docs/C5_HANDOFF.md) moves to the
private C5 firmware repository with the rest of the receiver internals.
Link the C5MK firmware from the user guide, fix the c5link.h reference
that still named FPVGateC5RX, and drop the em dashes from the guide.
Powered from the S3's 3V3 pin the C5 never finishes booting, because the
S3's regulator can't supply the C5 radio's start-up current. Wire 5V to the
C5's 5V (VUSB) pin instead.
Firmware: C5 multi-pilot races record every slot's RSSI to the race trace
and store per-pilot laps, slot, frequency and thresholds in race history,
so the editor can correct each pilot afterwards. A C5 receiver reports
sync mode 3 (multi) and rejects the conventional sync roles.

Web UI:
- Calibration is one card: band, fill, gain and auto-calibrate in the
  header, the live chart with an editor for the selected pilot, and a
  compact list of every pilot with live level, thresholds and Race toggle.
- The Race tab's pilot cards become a standings table at the top of Race
  Analysis (position, laps, last, best, total, gap, status).
- The race editor picks pilots from a calibration-style list whose dots
  show or hide each trace; Race Details collapse above the graph.
- The debug RSSI overlay draws the 8-pilot chart when the C5 is selected,
  since the C5 sends no single-channel RSSI stream.
- Multi-pilot styling follows the existing calibration cards: no coloured
  side stripes, the theme's buttons, sliders and badges.
FastLED is not thread-safe. The web server (a flash on every page load,
LED settings), the lap timer and the LED task all called FastLED.show(),
and two at once deadlock inside its RMT driver: the web server's task
hung until the task watchdog rebooted the gate. Core dump from a USB
soak test: async_tcp in handleRoot -> RgbLed::flashGreen and the LED
task in updateAnimation, both waiting on the same RMT semaphore.

Every show() now takes a mutex; a caller that can't get it in 50 ms
skips the frame and the LED task draws it on its next pass.
Arduino-ESP32 2.0.17 links Espressif's own build of TinyUSB's
dcd_esp32sx.c (esp32-arduino-lib-builder release/v4.4), not upstream's.
Two of its bugs stop USB networking under sustained load, both
reproduced on the bench and read back over Wi-Fi:

- DIEPEMPMSK, shared by every IN endpoint, is updated with unlocked
  read-modify-writes from tasks on either core and from the ISR. A lost
  bit leaves an endpoint with data queued and no TX-FIFO-empty
  interrupt: the network send pipe stalls for good.
- On the undocumented DIEPINT bit 15 it logs "Unknown Condition" and
  calls bus_reset(), a local reset that clears the device address while
  the host has reset nothing. Networking and serial die together until
  a power cycle.

The driver is now vendored in lib/USBNET/tinyusb/dcd_esp32sx.c, which
replaces the archive's copy at link time, with a spinlock on
DIEPEMPMSK, no reset on bit 15, and the ISR's final GINTSTS write
clearing only the flags it ignores (its `|=` cleared every pending
flag). usbnet_txfe.c keeps watchdogs for both failures as a backup and
counts events; GET /api/usbnet/status and /api/usbnet/regs report them
over Wi-Fi.

Soak test on a Seeed XIAO ESP32S3 at full load (event stream, page
loads, clock polls and serial at once): previously dead within 20 s to
2 min; now 30 min with no failure, 56 MB transferred, bit 15 seen 124
times in the first 6 minutes with no effect.
The RotorHazard link only ever reported one node, and on the C5 only
the single race pilot reached it. Plugin 2 now drives the gate as
RotorHazard timing hardware:

- GET /api/rh/info: firmware, receiver, frequency range and each node's
  frequency and enter/exit levels (one node on RX5808, eight on the C5).
- GET /api/rh/clock: the gate's clock in microseconds. The plugin polls
  it to sync, and while it does the gate streams every pass and stops
  posting laps the version 1 way.
- POST /api/rh/node: set one node's frequency and levels.
- GET /api/rh/events: rhPass (each pass, timed at its RSSI peak, from
  every C5 slot whether or not a race runs) and rhRssi (all nodes, 10 Hz).
  Its own stream, so the plugin doesn't also carry the browser's.

Also: /api/coredump/summary, /api/coredump and POST /api/coredump/erase
read the last crash's core dump over Wi-Fi, and /api/system/info
reports the Ethernet link on boards that have one.

Tested with RotorHazard 4.4.0 over USB networking: 20 back-to-back races
with no disconnect.
New WaveshareS3ETH target (16MB flash, 8MB octal PSRAM, onboard microSD
and WS2812). lib/ETHNET drives its W5500 through ESP-IDF's esp_eth on
its own SPI controller: DHCP, falling back to 192.168.8.1 when no
server answers (a cable straight to a laptop). The web UI, API and
RotorHazard plugin work over it unchanged; Wi-Fi and USB networking
stay available.

Pins the user wires are all on the left header (RX5808, buzzer,
battery sense, C5 link); see docs/WAVESHARE_S3_ETH.md. Builds, not yet
run on hardware.
Second Multi-Pilot alpha: USB networking that stays up under load,
firmware support for RotorHazard plugin 2, and an UNTESTED build for
the Waveshare ESP32-S3-ETH with wired Ethernet.

Tested on a Seeed XIAO ESP32S3 with an ESP32-C5 over USB networking:
30 minutes at full load and 28 RotorHazard races with no failure. The
other board builds share the code but were not run on hardware.
The RotorHazard plugin 2 connects to the gate through /api/rh/*, which is
always on, so the old mode where the gate posted laps to a RotorHazard
server is no longer needed. Remove it: lib/ROTORHAZARD (RHManager) and its
calls, the /api/rh/status and /api/rh/syncClock endpoints, and the Sync
"RotorHazard" role with its RH Host IP, seat, status and clock sync panel
and race banner. The config stops reading and writing rhEnabled, rhHostIP
and rhNodeIndex but keeps the fields, so the EEPROM layout is unchanged.
Plugin 2 still accepts laps from gates on older firmware.

C5 gates now also send RSSI and pass peaks at full resolution (0-1023):
"raw" in rhRssi, "rawPeak" in rhPass and "rawMax" in /api/rh/info, for the
plugin's Full-resolution RSSI option. The existing 0-255 fields are
unchanged, so older plugins are unaffected.

Rewrite docs/ROTORHAZARD_INTEGRATION.md for plugin 2.
lib_deps took AsyncTCP and ESPAsyncWebServer from the head of their GitHub
repositories, and FastLED, ElegantOTA, arduinoWebSockets, ESP8266Audio and
the LCD libraries by version range, so each fresh checkout built against
whatever was newest that day. Alpha-2 boards already differed: the S3-ETH
picked up FastLED 3.10.6 and is 380 KB bigger than the XIAO build.

Pin every board to what the tested XIAO Alpha-2 build linked: AsyncTCP
3.3.2 (ef448a8) and ESPAsyncWebServer 3.6.0 (ad3741d) from the me-no-dev
repositories, FastLED 3.10.3, ElegantOTA 3.1.7, arduinoWebSockets e3edca4,
ESP8266Audio 1.9.9, and on the LCD boards GFX 1.6.5, LVGL 8.4.0 and
TFT_eSPI 2.5.43. Rebuilding v1.9.0-Multi-Alpha-2 with these pins gives the
released XIAO firmware's exact size; only the build timestamp and image
hash differ.
From Hazard's night test on an S3-ETH: a flood of crossings that took
RotorHazard down at race start, unusable Marshal data, and an odd address
on his home network.

A C5 pass now ends only once the RSSI has stayed below Exit for 300 ms
(PASS_MERGE_MS). The light per-pilot filter let multipath dips during one
fly-through cross Exit, and each dip ended a pass. One fly-through is now
one pass, timed at its highest peak.

The RSSI sent to RotorHazard goes from 10 to 20 times a second, each value
the highest since the last (takeRhPeak), so the history Marshal works from
has every pass's peak.

Ethernet: a new link asks for DHCP again, even after falling back to the
fixed 192.168.8.1, so moving the gate to another network works without a
restart. The DHCP wait goes from 15 to 45 s for switches that hold a new
port for ~30 s.
@LouisHitchcock

Copy link
Copy Markdown
Owner Author

Update pushed to V1.9.0-alpha without merging.

Latest logical commits:

  • f210e7c — ELRS Backpack starts now follow the configured server-side race countdown.
  • 4040785 — browser race announcements and start/stop audio handling restored.
  • f7e3a6b — SD initialization retries once at 1 MHz after the normal 10 MHz attempt.

The SD retry was also tested against the C5MK AIO; the card still failed under both the Alpha-6 firmware and the v1.8.3 Novacore AIO firmware, so the hardware investigation remains open.

rhLinked() subtracted the link stamp from the caller's clock reading as
unsigned. /api/rh/clock stamps from the HTTP task, so the stamp can be newer
than a reading parallelTask took earlier in its loop. The difference wrapped,
the link read as lost for one tick, and sendRhUpdates() cleared passes the gate
had already acknowledged.

Forced on the bench with clock polling load, rc.3 lost 504 of 1051 passes and
the fixed build lost none. This is not proven to be the loss seen in rehearsal
heat 35.
Prepare rc.4. Nothing recorded where the rehearsal's missing pass went, because
every loss path on the gate was silent. /api/system/info now reports rhPasses:
passes sent to the stream, refused by a client's queue, sent with no client,
refused by a full queue, cleared by a lost link, and link losses.

The acceptance harness snapshots these around each RotorHazard heat, so a
future shortfall can be placed on or beyond the gate.
An 8-hour run stopped at 17 minutes when Windows refused os.replace on an input
journal it rewrites many times per heat. Every completed heat had reconciled, so
the gate was not at fault. The retry covers the local file write only and never
resends a crossing; a refusal lasting 5 s still fails the run.
On a requested Ctrl-C the main thread's timed join returned early, so the
process exited while the AIO lane was mid-heat. Its restoration never ran:
config stayed at test values and its RotorHazard server kept running.

The main thread now sets the stop flag and waits on each lane's liveness
against a deadline, ignoring further interrupts. A lane still busy after 15
minutes is recorded as not terminated and its RH server is stopped.
Interrupted mid-heat on both bench gates, each lane restored its gate before
the run ended.
Samples a new connection, a held connection and ping once a second, so a
gap where only new connections fail shows the gate is up but not accepting.
Burst mode repeats the harness's page-load shape; monitor mode avoids the
RotorHazard link endpoints so it can run beside an acceptance run.
Two community requests from Discord that roadmap item 6 kept open.

Announcer (spova): a beep for each lap in the consecutive modes, and
combinations such as "beep every lap + 3-lap time" or "lap time + 3-lap time".
Two settings sit under the announcer type. Beep Every Lap adds the beep to any
mode; Say Lap Time Too speaks each lap time with the 2- and 3-lap totals, and on
its own until a total exists. The gate accepts only announcer types 0-4, so
these are kept per browser like the beep volume, leaving the firmware alone.
Single-pilot and C5 multi-pilot announcements both use them.

Lap editor: a false or double pass splits one lap in two, and deleting either
half drops its time. Each row in the typed lap editor now has Merge, which adds
the lap's time to the next lap (or the previous one, for the last lap) and keeps
the race total. Delete still drops the lap, and its confirmation now says so.
The logger kept 100 lines in a std::vector that every task appended to and
erased from without a lock, a data race, and it held about 26 KB of internal
RAM. Lines leading up to a crash were lost with the reboot.

The log is now an 8 KB ring in RAM that a crash, watchdog or software reset
leaves alone, sealed with a checksum so a ring from the previous boot is told
from power-on contents. Writers hold a short critical section only to copy
bytes. Each boot logs its number and reset reason.

Saving to the SD card is off by default. With /logs/enabled on the card, a
priority-1 task appends the ring to /logs/log.txt every 5 s in 2 KB chunks
through Storage's SPI lock, rotating to log.1.txt at 256 KB. It never writes
without a card, so it cannot wear the board's flash. New endpoints report the
state, switch saving and download the files; /api/debuglog adds a seq that
orders lines across boots.
C5 Receiver Checks watches the c5Rssi telemetry the gate already streams for
six seconds and reports connection, firmware version against the paired 1.2.5,
tuning, racing slots, arriving samples and new link errors. It runs in the
browser, needs no firmware support and refuses while a race runs.

Download log.txt writes one file for bug reports: system and receiver status,
the last crash dump summary, the latest self-test and C5 checks, then the log
(from the SD card when saving is on, otherwise the recent lines in memory).
The Save Log to SD Card switch sits beside it, off by default. The System
Monitor now orders lines by seq so lines kept from before a restart do not
hide newer ones.
The SD card log changes the firmware, so the soaked rc.4 images no longer
match. Label the new build rc.5 and have the acceptance harness expect it.
Storage logged five progress lines for every file it read or wrote, so
loading race history at boot filled the 8 KB ring in under a minute and
pushed the boot line out. Drop those progress lines, keeping failures, and
size the ring at 16 KB, still less than the 26 KB the old logger used.
/api/debuglog returns the newest 6 KB because the System Monitor polls it
twice a second, so Download log.txt without an SD card missed older lines,
including the boot line. /api/log/recent streams the whole ring as text from
one copy, in PSRAM where the board has it, and the download uses it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci dependencies documentation Improvements or additions to documentation firmware web-ui

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant