meshcore-sar/docs/voice-mode-technical.md

Voice Mode Technical Design

1. Overview

Voice mode uses a two-plane architecture with optional swarm-assisted recovery:

• Control plane (text messages):
  • VE3: voice envelope announces voice availability in chat.
• Control plane (raw binary request):
  • Binary voice fetch request (same raw route as voice packets).
• Data plane (raw binary packets):
  • VoicePacket payload streamed via cmdSendRawData and received through pushRawData.

This design avoids broadcasting full voice payloads to channels/rooms. Chat carries only metadata; audio is fetched on demand when user presses play.

Swarm fallback is documented in Swarm Mode Technical Design.

2. Key Modules

• lib/utils/voice_message_parser.dart
  • VoicePacket (binary direct-packet format)
  • VoiceEnvelope (VE3)
  • VoiceFetchRequest (binary)
• lib/screens/messages_tab.dart
  • Capture/encode voice, cache encoded packets, send envelope only
• lib/providers/voice_provider.dart
  • Reassembly/playback sessions
  • Outgoing session cache + deferred serving
• lib/providers/app_provider.dart
  • Incoming routing for VE3, binary voice fetch requests, and raw swarm control payloads
  • Handles raw packet ingestion
• lib/widgets/messages/voice_message_bubble.dart
  • Play behavior (immediate play if complete, otherwise fetch + auto-play)
• lib/providers/messages_provider.dart
  • Message-level voice detection (VE3)
• lib/services/message_storage_service.dart
  • Persists isVoice and voiceId

3. Wire Formats

3.1 Voice Envelope (VE3)

Prefix: VE3: + colon-delimited compact payload (base36 numeric fields)

Fields:

• sid (string): base36 token for 32-bit session ID
• mode (base36): codec mode ID (VoicePacketMode.id)
• total (base36): packet count (1..255)
• durS (base36): estimated duration in seconds

sid is base36 on wire and expands to 8-hex internally.

Compact format:

VE3:{sid}:{mode}:{total}:{durS}

Example:

VE3:a:1:4:4

3.2 Voice Fetch Request (binary)

Binary payload format:

[magic=0x72][sid:4B][flags:1B][requesterKey6:6B][missingCount:1B][missingIndices...]

3.3 Raw Voice Packet (data plane)

Binary payload structure:

• Byte 0: magic 0x56 ('V')
• Bytes 1..4: session ID (4 bytes)
• Byte 5: packet index
• Bytes 6..N: codec2 data

Header is 6 bytes. Mode and total packet count come from the VE3 envelope.

4. Outgoing Flow (Send)

1. Recorder captures PCM chunks.
2. Each chunk is codec2-encoded into VoicePacket objects.
3. Packets are cached in VoiceProvider outgoing cache (TTL 15 min).
4. Sender inserts local voice placeholder message (isVoice=true, voiceId=sessionId).
5. Sender sends one envelope (VE3) through normal message path:
   • channel/room: sendChannelMessage
   • direct: sendTextMessage
6. No raw audio packets are sent during initial send.

5. Incoming Routing

5.1 VE3 envelope received

AppProvider records sender identity from message metadata, registers the voice session envelope, marks the message as voice (isVoice, voiceId), and adds it to chat.

5.2 Binary voice fetch request received

AppProvider treats it as control-plane only:

• request is not added to chat
• resolves requester contact via key prefix
• calls voiceProvider.serveSessionTo(...)

5.3 Swarm control messages received

AppProvider also handles raw swarm control payloads:

• binary swarm requests advertise which voice fragments are still missing
• binary swarm availability responses advertise which fragments a peer can relay
• swarm control payloads arrive via pushRawData and are not added to chat history

Swarm semantics are shared with image mode and documented in Swarm Mode Technical Design.

5.4 Raw packet received (pushRawData)

AppProvider.onRawDataReceived parses VoicePacket binary and appends to session in VoiceProvider.

6. Play / Fetch Behavior

In VoiceMessageBubble:

• If session already complete: play immediately.
• If incomplete/missing:
  1. Resolve sender contact from message sender metadata
  2. Prefer a direct fetch from the original sender if its raw route is healthy
  3. If the sender path does not respond, fan out a raw swarm request with the exact missing packet indices to reachable peers
  4. Wait up to 10 seconds for raw peer availability responses
  5. Send a direct binary fetch request to the best alternate peer for the missing subset it advertised
  6. Show requesting state in UI
  7. Auto-play when session becomes complete

If sender cannot be resolved or request cannot be sent, bubble remains and shows: "Voice unavailable right now".

7. Outgoing Cache Details

VoiceProvider outgoing cache:

• key: sessionId
• value: encoded packet list + cached timestamp
• TTL: 15 minutes (_outgoingSessionTtl)
• eviction: lazy (on cache access/add/serve)

Serving prerequisites:

• session exists in outgoing cache or already-received session state
• sendRawPacketCallback configured
• requester has direct path (outPathLen >= 0)

Received partial sessions can therefore act as relay sources during swarm recovery.

8. Persistence

MessageStorageService now stores and restores:

• isVoice
• voiceId

This ensures envelope messages remain voice bubbles across app restart.

9. Validation and Safety

Parser validation enforces:

• strict hex lengths for IDs and key prefixes
• valid mode range
• valid packet counts and duration bounds
• compact base36 numeric fields in envelope/request
• Binary request flags specify all or missing indices.
• Request payload includes requesterKey6 to resolve return route.

10. Transmit Time Estimate (UI)

Voice bubbles and Message Technical Details show an estimated transmit time (~... tx).

The estimate is airtime-based (LoRa packet model), not file-duration-only:

• Source inputs:
  • packetCount and durationMs from VE3 envelope, or
  • numeric envelope values decoded from base36
  • actual received VoicePacket.codec2Data.length bytes when local session packets exist
  • pathLen from message metadata
  • current radio params from deviceInfo: radioBw, radioSf, radioCr
• Per-packet payload model:
  • meshHeader(2) + pathLen + voiceHeader(6) + codec2Bytes
• LoRa airtime:
  • standard symbol-time formula (preamble + payload symbols)
• Mesh pacing/hops:
  • multiplied by (1 + airtimeBudgetFactor) where default factor is 1.0
  • multiplied by hop count (pathLen + 1)
• Total estimate:
  • sum over all packets

BW handling:

• If radioBw is index 0..9, app maps it to Hz (7.8k .. 500k)
• If radioBw > 1000, it is treated as Hz directly

Fallback defaults are used when radio params are unavailable: SF10, BW250kHz, CR5.

11. Operational Constraints

• No firmware changes required.
• On-demand fetch prefers the original sender, but a partial session can also be completed from alternate peers that already hold packets.
• Raw return path needs a currently valid direct route to requester.
• Voice capture is available on iOS and Android (Platform.isIOS || Platform.isAndroid).
• Swarm discovery uses the same cmdSendRawData / pushRawData path as voice fetch and packet delivery.

11.1 Raw Binary Routing Semantics

• Voice payload packets use companion command CMD_SEND_RAW_DATA (25 / 0x19).
• Companion push back to the app is PUSH_CODE_RAW_DATA (0x84).
• Over-the-air packet type for this flow is PAYLOAD_TYPE_RAW_CUSTOM (0x0F).
• This flow is direct-route only, not flood/broadcast:
  • it is sent to one destination path;
  • only nodes on that path relay it;
  • it is not received by everyone in the mesh.

12. High-Level Sequence

sequenceDiagram
  participant A as Sender App
  participant P as Peer With Packets
  participant N as Reachable Peers
  participant B as Receiver App

  A->>A: Record + encode voice packets
  A->>A: Cache session packets (TTL 15m)
  A->>M: Send VE3 envelope
  M->>B: Deliver VE3
  B->>B: Render voice bubble (metadata only)
  B->>A: Send binary fetch request on Play
  A->>B: Stream raw VoicePacket packets (partial)
  Note over A,B: Sender path stops responding
  B->>N: Raw swarm requests with missing voice packet indices
  P->>B: Raw swarm availability response
  B->>P: Direct binary fetch request for missing subset
  P->>B: Stream remaining raw VoicePacket packets
  B->>B: Reassemble session
  B->>B: Auto-play when complete