trail-mate/docs/engineering/native-navigation.md

Native saved-route navigation

Trail Mate consumes the released TMAP SDK 0.2.0 and stores calculated foot routes as self-contained .tmroute files. Tracker creates and previews them; Map reads the visible geometry, follows fresh GPS fixes and owns the navigation session. The existing KML following path remains supported. This document distinguishes host acceptance, target compilation and physical-device measurements.

Required lifecycle

Select current/searched endpoints; calculate with an explicit external-memory budget; write legs directly to an exclusive SD spool; release search storage; export and finalize a self-contained .tmroute; sync and publish; list, preview and activate it; read only the current viewport; match fresh GPS fixes; show the next turn and distance; replan after confirmed deviation; finish or cancel the session and release resources. Loading errors, missing GPS/SD, ambiguous matches, memory/storage budgets, cancellation and failed replanning need visible UI states.

Implementation and acceptance matrix

Requirement	Implementation and current evidence	Verification limit
Sink-based search and phased export	Actual production worker, ArchiveCalculation, SD map/archive adapters; restricted-turn fixture and Center London 20.779 km pass	Host SD/RTOS ports do not establish device latency
Independent route reads	Map's TmRouteMap uses the renderer's projection and viewport; fixed snapshot; LVGL draw callback performs no file I/O	Host display adapter, no physical screen capture
External-memory policy	Explicit external owners, RequireExternal and PSRAM-only file handles; refusal and complete release tests pass	Whole-firmware concurrent RAM/PSRAM peaks remain unmeasured
Temporary file ownership	Exclusive creation in Arduino and IDF; partial transfer/collision/media-session tests; cancellation survives a denied UI mutex	Sudden power-loss durability is not established
Named saved routes	Actual worker save/list/reopen/full validation/remove test; Tracker native thumbnail and Map readers	UI creation dialog uses host platform fakes in LVGL tests
Progress and turn lookahead	1,170 real London segment endpoints match; 299 geometry-turn hints; final cue is published atomically	Geometry hints have no street names or intersection semantics
Ambiguity, stale fixes and deviation	Stale GPS, ambiguity fixture, off-route dwell and SD-loss cleanup tested; distinct visible Map states	Synthetic fixes, not a recorded physical GNSS walk
Replanning	Automatic/manual policy, cooldown, cancel/retry and failed-candidate retention; new archive validated before config/path swap	Session policy and actual worker tested separately on host
Arrival and end	Prior progress, endpoint proximity and five-second fresh dwell; session releases its GPS lease; config is cleared and Finish remains visible	Other GPS consumers may retain their own leases
UI controls	Actual LVGL dialog tests: dynamic Cancel/Retry/Finish, keyboard focus recovery and fixed touch Stop footer	No claim of physical input acceptance
Interactive route preview	Actual Tracker model/gesture/control/key handlers tested with LVGL: drag, zoom, Fit, retained view and image-pin selection; native detail uses TmRouteMap	Tile loading and physical input are separate acceptance steps
Reader contention and dense views	Actual Arduino LVGL filesystem callbacks and archive reader tolerate 11 injected Busy events; a 10,000-point dense view recovers after zoom without increasing reader storage	Host fault injection does not reproduce physical SD timing

Using saved navigation

1. Install Center's map and font output using its SD directory structure. A detailed OSM package must contain a foot-routing graph; raster coverage by itself cannot calculate a route.
2. Open Tracker → Menu → Nav → New. Choose a searched destination and either the current fresh GNSS position or a searched origin. Calculate, then choose Save and preview. The saved file lives in /routes/navigation. Drag the preview map to move it; use +, − and Fit to zoom or restore the route overview. Keyboard devices including Pager use W/A/S/D to pan, Q/E to zoom out/in, and F to fit. Arrow and + / − aliases remain available. Press H for Preview Help; W/S scrolls the help. When the image strip is shown, W/S selects images; hide it with P to pan in all directions. G downloads images and V toggles the profile. Changing map layers or image selection preserves the chosen view.
3. Load the selected saved plan as the active route and open Map. The filename identifies the saved plan; the route is reopened after leaving/reentering Map or restarting the application. A preview alone does not activate following.
4. Map → Menu → Navigation offers Replan from here, Reload route, Automatic replan and Stop navigation. During a job the replan action becomes Cancel replan; failure offers Retry replan. A candidate is opened successfully before replacing the active path. Cancellation/failure leaves the previous saved plan available.
5. After confirmed arrival, active-route configuration is cleared, the completed view remains available and the fixed footer becomes Finish navigation. Stop/Finish closes the session without deleting saved routes. Delete is a separate Tracker file action.

The current navigation session runs while Map is open. Leaving Map releases its navigation reader/session and power lease; the saved active path resumes on reentry. Background spoken guidance or guidance from other pages is not provided. Planning is offered on PSRAM-capable ESP Arduino and ESP-IDF ports. Other shared UI targets can read saved archives but do not advertise an available planner.

Resource policy

ArchiveCalculation defaults to 65,536 search labels and a 2 MiB array budget, 200,000 expansions, 65,536 maximum spooled legs and an 8 MiB archive budget. The arrays are constructed in bounded slices. ALT is selected when landmark sections are present; otherwise bounded heap Dijkstra is used without allocating unused landmark storage. Invalid landmark metadata is not silently ignored. Search arrays/cursor are released before the archive workspace/exporter are created. The application owns the files and their lifetime throughout each job.

The new SavedRoute has no map/graph dependency. One reader/page workspace is shared by two small cursors, so a viewport query can interleave with matching and lookahead. No complete geometry or route-sized RAM index is retained. Queries have work/node/segment bounds; budget exhaustion must not be shown as a complete route. Geometry turns are directional hints, not street-name or surveyed-junction instructions. Repeated/stale fixes do not establish arrival or deviation dwell.

Map keeps 128 clipped screen segments (1,024 B) rather than all route geometry. Old-camera geometry is cleared immediately on pan/zoom; a complete query is published together. If more than 128 visible segments are needed, the UI asks the user to zoom in instead of showing a silently truncated line. Tracker's 96-point overview provides initial bounds and summary metrics; native preview detail uses another instance of the same bounded reader. Each preview retains 20 B of scalar camera state and one fixed snapshot rather than a full polyline. The preview reader and its archive workspace require external memory on ESP; there is no fallback to internal RAM for those owners. Closing the preview deletes its poll timer and releases its reader. LVGL controls, map tiles and allocator overhead are additional costs, not part of the reader payload. Queries run in bounded poll slices; a single synchronous SD call can still take longer than the nominal 4 ms poll budget on actual hardware.

Fresh fixes are at most five seconds old. Core matching accepts reported accuracy up to 30 m; the current application GPS port supplies no accuracy estimate, so Map uses the default zero uncertainty and still enforces age. Reliable following is within 15 m, deviation requires more than 35 m for five seconds, and arrival requires being within 10 m of the geometric endpoint with at most 10 m remaining for five seconds after earlier progress. Endpoints are snapped within the producer's 50 m bound. Original POI names/requested coordinates and semantic maneuvers are not stored by the archive; the saved geometric end is the replan destination. Automatic replan is limited to one attempt per continuous deviation and a minimum 30-second interval; manual retry remains available.

SdRouteFile uses SdRuntimeMemoryPolicy::PsramOnly, checks the medium session, and retries partial reads/writes from the original offset. New files use exclusive creation. Cleanup must only remove files created by the current job. Existing saved routes are not overwritten during calculation or unsuccessful replanning.

Current real London pipeline evidence

Input: Center's maps/tmap/osm/london-osm-z13-17.tmap, from the SD delivery manifest, SHA-256 122b58d9ae7eb124f5f67d23dba204deb53506fd3ede17cdafb2022a814e75a1. Coordinate request: (51.5009000, -0.1245000) to (51.4036000, -0.3377000), foot mode, 50 m snapping, ALT and default pipeline limits.

Metric	Measured 32-bit MinGW host result
Geometry length	20,778.62 m
Clipped legs / geometry points	550 / 1,171
Provider peak during search	2,108,244 B
Provider live bytes after search release	10,024 B
Provider peak during export	11,800 B
Live allocations after pipeline completion	0 B
Saved route file	43,776 B
Search logical map reads	2,229,930,944 B

Normal and every-seventh-operation Busy runs produce identical complete files. The current machine-readable pipeline evidence records both runs, the saved file hash and the compiler frame maximum. Each run closes the map, spool and output before reopening and fully validating the archive. The fixture also checks allocation refusal and cancellation. This is the actual application pipeline, not just the upstream SDK probe. Additional production-worker and actual LVGL reader tests are recorded below. None is a physical PSRAM/SD-latency measurement. Provider bookkeeping, owner controls, task stack, display and driver costs are additional. The large logical search-read volume remains a hardware acceptance concern even though display memory is independent of total route length.

cmake --build .codex-build/tmap-dependency
ctest --test-dir .codex-build/tmap-dependency --output-on-failure
.codex-build/tmap-dependency/navigation/navigation_archive_pipeline \
  /data/sd/maps/tmap/osm/london-osm-z13-17.tmap /data/test/london-20km.tmroute --london

The current native suite has 15 passing tests. A real ESP32-S3 cross-compiler probe in tests/navigation/compile_embedded.cpp compiles the production inline archive/match/view paths without exceptions/RTTI and passes the 240 B individual stack-frame gate. Compiler frames do not establish full task-stack consumption.

Production worker and LVGL replay evidence

The production-worker host target compiles the actual platform_ui_navigation_runtime.cpp, SdTmapStorage and SdRouteFile. Only the RTOS, physical SD and capability heap are replaced by deterministic host ports. It tests calculate → pending hidden from list → save → list → release all owners → reopen → full archive validation → delete. A second job injects cancellation while the cancellation mutex admission fails, and verifies cleanup with the previous saved file preserved. A PSRAM refusal prevents a new job.

Metric	32-bit host measurement
Production worker external allocation payload peak	2,143,140 B
Worker map and spool read volume through SD host port	1,542,653,456 B
Actual UI Map owner, including 128-segment snapshot	1,576 B
Reader/page/two-cursor external payload while following	2,160 B
Replayed geometry endpoints	1,170 matched / 0 ambiguous
Non-straight geometry cues during replay	299
View queries across start/middle/end, z12/14/16/18	12
View read total / maximum per tested view	66,016 / 20,064 B
Matching read total for replay and final arrival dwell	3,938,176 B
Dense overview views / maximum published screen segments	4 / 72
Pixel-identical full-geometry oracle comparisons for fitting views	8 / 8
Live route allocations and open files after close	0 / 0

All tested z16/z18 views fit the fixed screen snapshot. The eight fitting views produce exactly the same LVGL pixels as the full-geometry host oracle, and draw callbacks perform zero route reads. Coarser dense views show the zoom-in state. UI replay injects a partial Busy read every seventh operation; the file remains unchanged. It also checks stale fixes, sustained deviation, rejoin, final arrival and cold reopening without a false arrival. The host oracle loads full geometry only in test code; production code retains bounded owners. See machine-readable application evidence.

tests/navigation_ui contains the actual LVGL reader, creation dialog, navigation controls and session-policy tests. The runtime's config/active-route swap wiring is compiled in the firmware; the host tests do not claim to drive the entire GPS-page global state through physical input. Target build results are recorded in the application evidence alongside the precise tested scope.

The complete Wio Tracker L2 PlatformIO build finished successfully with exit 0 in 795.892 seconds. Its ELF contains the production navigation submit/save and native Map poll symbols. PlatformIO reports 96,756 B static RAM and 4,913,237 B Flash in the 6,291,456 B application partition. The generated 4,913,648 B binary has SHA-256 163bf3086e3dbf2623d4578de3006ec5d6a8251cc8476e4c9de82d93a79b7500. Static RAM is not runtime high-water. ESP-IDF's producer, SD adapter and both UI consumer sources also pass the real P4 compiler's syntax checks; a complete P4 firmware build and physical acceptance are not asserted by those checks.

The 2,143,140 B measurement covers allocations routed through the host capability heap, including the worker state and map pages. It excludes allocator metadata, real SD-handle/driver allocations, the 4,096-byte task-stack request, RTOS control blocks, LVGL/display buffers, other application features and concurrent readers. During candidate activation both old/new readers may coexist briefly. These figures are admission evidence, not a claim that a device with 2 MiB free PSRAM can run the whole firmware. Internal RAM, real PSRAM high-water, SD wall-clock latency and energy still require physical-board measurements. Search read volume is substantial; do not derive an MCU calculation time from the host runtime.

cmake --build .codex-build/navigation-ui
ctest --test-dir .codex-build/navigation-ui --output-on-failure
.codex-build/navigation-ui/navigation/navigation_production_worker \
  /data/sd/maps/tmap/osm/london-osm-z13-17.tmap /data/test/london-worker --london
.codex-build/navigation-ui/tmroute_ui_contract \
  /data/test/london-20km.tmroute --london

The production-worker host target uses Windows 64-bit file-position APIs and is enabled for MinGW. The other native pipeline/UI tests are portable. No physical device flash or route walk is implied by these commands.

Diagnosing Calculate route failures

Navigation emits bounded ASCII serial diagnostics with the [Navigation] prefix. Trace schema 1 records UI Calculate/validation, each backend admission attempt, worker creation, selected map paths, phase changes, export, save, cancellation and terminal outcomes. Each backend attempt has a request ID. Long-running work emits a progress line at most once per five seconds; it does not log every expansion or geometry point. Logging uses no route-sized buffer and no SD log file. Arduino USB CDC forwards lines as HostLink EV_LOG frames after HELLO, using the existing bounded single-writer queue. Formatting retains one 384-byte scratch allocation for the firmware lifetime, allocated in PSRAM only when a ready USB client first requests diagnostics; allocation failure keeps UART output without falling back to internal RAM. Concurrent formatting or a full event queue can drop USB lines. Endpoint names, exact coordinates and configuration fields are omitted. Memory values are capability-heap snapshots in bytes, not allocation peaks or a claim that the whole firmware fits them.

Capture serial output before pressing Calculate route, through the error and a few seconds afterwards. At 115200 baud, retain all [Navigation] lines and any neighbouring SD, USB, allocation or reset errors. Capture before the action: a failed admission has no worker log and cannot be diagnosed by waiting only for a worker-completion message. Use firmware containing trace=1; previous images cannot gain these diagnostics by replacing SD files.

The optional tools/hostlink/capture_navigation.py helper reads native USB CDC at 115200 baud for 1–60 seconds, validates log-frame CRCs and writes a local timestamped log. Its only transmitted command is HELLO, which enables the existing HostLink event session. It waits for HELLO_ACK before declaring capture ready, changes no configuration and performs no bootloader/reset sequence. It uses the same native CDC DTR/RTS connection signals as the existing HostLink transport. Close other applications holding the selected serial port before capture. Use --uart for passive ASCII capture from a physical UART instead. Ordinary UART printf output is not automatically mirrored to the Arduino native USB CDC port.

python tools/hostlink/capture_navigation.py --port COMxx --seconds 45 \
  --output .codex-build/navigation-ui/device-navigation.log

Trace event / reason	Interpretation
[UI] event=calculate without a submit	Endpoint selection or fresh-current-GPS validation stopped the action; inspect the next UI message/fix fields
psram_unavailable	No external memory is available to this runtime
invalid_origin, invalid_destination	Coordinate/name validation failed before map reads
sd_not_ready	SD runtime was unavailable before creating a task
mutex_allocation_failed, mutex_busy	Backend admission could not obtain its state lock
previous_job_retained	Another job or its terminal state still exists; preceding status/running/cancel fields identify it
state_allocation_failed	External allocation of worker state failed
task_create_failed	FreeRTOS rejected task creation; compare internal free/largest block and the 4096-byte stack request
[allocation] rejected	A search/export provider request failed; requested bytes/alignment and external heap capacity are recorded
mkdir_failed, pending_open_failed, spool_open_failed	Working-file setup failed before graph search; inspect SD readiness, medium session and USB ownership
[map] event=candidate	Exact installed package considered by the worker
[search] event=failed result=...	SDK returned a search/open/budget/coverage result; unsupported packages can be skipped before trying another candidate
[worker] event=finish status=...	Final application outcome after route/map owners were released

Numeric phase IDs follow ArchiveCalculation::Phase: Idle 0, Opening 1, Allocating 2, Beginning 3, Searching 4, Ready 5, ExportBeginning 6, Exporting 7, Complete 8, Failed 9, Cancelled 10. usb_owner=1 means USB holds SD ownership at that observation. A card exposed over USB can pass desktop file checks while application SD access is unavailable. USB ownership observed during a later inspection does not prove it was active during an earlier failed calculation.

tests/navigation/verify_worker_trace.py fault-injects admission failures through the actual production worker target and checks distinct rejection reasons, memory fields, lifecycle logs and omission of private endpoint data. The host test does not reproduce the physical device's exact heap fragmentation or USB transition timing. A new reproduction with the instrumented firmware is still needed to establish which rejection occurred on a particular device.

The 2026-10-16 Wio investigation also found a deterministic work-directory bug: the worker unconditionally created /routes, but SdFat returns false when the final mkdir component already exists. The card already had that directory. The worker now checks sd_is_directory before creating either working directory; a regular file at either path still fails. The host worker port now reproduces SdFat's exclusive final-component mkdir behavior. The unpatched worker failed with mkdir_failed and storage_unavailable under that regression; the patched worker passes existing-parent/existing-work-directory and file-collision cases. This establishes the source defect and host regression, not successful physical route calculation after the fix; on-device acceptance remains separate.

The corrected Wio image built successfully in 215.072 seconds (exit 0): 90,096 B static RAM and 5,029,329 B linked Flash. Relative to the first diagnostic image in the same incident, static RAM increased by 8 B and Flash by 288 B. The 5,029,744-byte binary has SHA-256 e03bca86d81190888c346bc45a4fa3960fc86ea0d61ad001819c9c6604107ce8. Flashing that image to the confirmed Wio target completed with hash verification in 67.686 seconds (exit 0). The application CDC port cannot automatically enter download mode with reboot disabled; manual BOOT/RESET was used for this flash. The corrected production-worker host regression also completed the Center London 20,778.6 m case with 1,171 points, 43,776 archive bytes, 2,143,140 B peak provider payload and zero live allocations after close. That host payload does not include the Arduino USB scratch/queued frames or physical SD/RTOS overhead. Successful flashing and host routing do not yet establish a successful route calculation or runtime memory peak on the Wio itself.

The subsequent Wio CDC reproduction did reach Searching, using the pre-existing /maps/navigation/navigation.tmap, then returned io_error after 20,730 ms. At the failure snapshot SD was ready, USB did not own the medium, the medium session matched, PSRAM free was 6,656,699 B and internal free was 183,084 B. These are observations at log points, not runtime allocation peaks. No route was published. The trace rules out admission failure for that reproduction; it does not establish map corruption or a complete physical-card diagnosis.

This exposed another desktop/embedded filesystem difference: exFAT seekSet rejects positions beyond the current data length. Reverse leg-spool output can first write a later record, and archive geometry begins at an aligned offset. The desktop file ports had allowed sparse seeks. Tightening both host SD ports to reject beyond-EOF seeks reproduced the same Searching → io_error failure in the actual producer, and the new gap-write adapter case failed as expected. The adapter now pads at most 256 bytes per call and returns More before retrying the target write, preserving partial Busy progress. Its padding is a read-only Flash constant, with no additional route-sized RAM owner. Tests check the per-call growth limit, zero padding, partial Busy retry and final archive CRCs. Current integrated map directories are searched before older standalone graph directories. Fatal map and route adapter I/O now report the operation, offset and requested byte count without endpoint names or coordinates. These changes still require their own physical-board acceptance.

The combined directory/gap/selection correction built for Wio in 226.850 seconds (exit 0), with 90,096 B static RAM and 5,030,413 B linked Flash. The ELF places the 256-byte padding symbol at Flash-mapped address 0x3c4545ac; it is not a task stack or mutable RAM array. The 5,030,832-byte binary has SHA-256 58430ab89931c1fa337353b1e43e8eb99adf0f987b8851979d86340614f0730d. Upload and hash verification completed in 61.342 seconds (exit 0). With strict exFAT seek semantics, the Center London host run still produces 20,778.6 m, 1,171 points and 43,776 archive bytes, with 2,143,148 B external provider payload, 1,542,653,456 logical read bytes and zero live owners after close. The additional 8 B relative to the prior host run is the map-read observer. Physical route calculation was subsequently reported successful by the Wio user. No terminal trace, route length or physical allocation peak was captured for that success. The user then reported an unavailable/damaged message when opening Map; this is a separate saved-route read/display issue.

The saved-route investigation found that Arduino's LVGL callbacks discarded SdRuntimeFile::read_busy() on seek/tell and on short reads. A transient bus admission failure could therefore become an I/O error or a false zero position. The callbacks now propagate LV_FS_RES_BUSY, and position() records failed bus admission like size() and seek(). A short open retry also handles LVGL's generic open error; real read errors still fail visibly. Terminal reader logs record the stage, operation, byte offset and result without private coordinates. Exceeding a viewport query budget leaves the archive open and asks the user to zoom in. It no longer classifies density as damage.

The reader/preview stage passed all 15 tests then available, including the production-callback Busy/read-error regression and a 10,000-point dense-view/zoom recovery case. The latter keeps the same reader allocation before and after zoom, with zero open file handles after destruction. Interactive preview tests use the actual Tracker camera, gestures, controls and key handlers with deterministic map/app ports. ESP stack hygiene and four ESP-IDF navigation syntax checks also pass. The subsequent user report confirmed normal operation on the physical Wio. This is user-reported acceptance, without a captured trace or a measured whole-firmware memory peak.

The reader/interactive-preview Wio image built successfully in 239.846 seconds (exit 0), using 90,124 B static RAM and 5,035,053 B linked Flash. Compared with the preceding spool-correction image, these increased by 28 B and 4,640 B. This excludes dynamic LVGL/map/reader allocations and does not measure runtime heap peaks. The 5,035,472-byte binary has SHA-256 993725458a27f0131b7af7a527f4bda36b570c998b905b7da865d4086a0071c3. Upload to the confirmed Wio target and Flash hash verification completed in 61.459 seconds (exit 0). The Wio user subsequently tested the updated image and reported normal operation. A later USB capture attempt could not open COM69 despite Windows listing it, so no diagnostic capture or additional device resource measurement is claimed for that acceptance.

Pager keyboard follow-up adds letter-based pan and zoom matching Map, with Fit and an explicit scrollable Preview Help list. It changes preview downloads from D to G and the profile from E to V to avoid collisions. The actual Tracker key handler regression checks both letter cases for all four pan directions, Q/E zoom and F reset. Pager physical-key acceptance remains separate from the earlier Wio touch verification.