meshcore-sar/docs/image-mode-technical.md

Image Mode Technical Design

1. Overview

Image mode mirrors the voice on-demand architecture exactly, including swarm-assisted recovery for stalled partial transfers:

• Control plane (text messages):
  • IE4: image envelope announces image availability in chat.
• Control plane (raw binary request):
  • Binary image fetch request (same raw route as image fragments).
• Data plane (raw binary packets):
  • ImagePacket binary payload streamed via cmdSendRawData / pushRawData.

Images are never broadcast in full to channels. Chat carries only metadata; pixels are fetched on demand when the user taps the image bubble.

Shared swarm fallback is documented in Swarm Mode Technical Design.

2. Key Modules

• lib/utils/image_message_parser.dart
  • ImagePacket (binary fragment format)
  • ImageEnvelope (IE4)
  • ImageFetchRequest (binary)
  • fragmentImage() — split compressed bytes into packets
  • reassembleImage() — join received fragments into bytes
• lib/screens/messages_tab.dart
  • Pick image from gallery or camera, compress with user settings, cache, send envelope
• lib/providers/image_provider.dart
  • Reassembly sessions, outgoing cache, deferred serving
  • Outgoing sessions also registered as complete incoming sessions for immediate local display
  • Received partial sessions can be re-served during swarm recovery
• lib/providers/app_provider.dart
  • Incoming routing for IE4, binary image fetch requests, binary 0x49 packets, and raw swarm control payloads
• lib/widgets/messages/image_message_bubble.dart
  • Square cover thumbnail (up to 256 px); tap-to-load for received images; progress ring during fetch; full-screen InteractiveViewer on tap
• lib/services/image_codec_service.dart
  • ImageCodecService.compress() — aspect-ratio-preserving resize + optional grayscale + AVIF encode via flutter_avif
• lib/services/image_preferences.dart
  • ImagePreferences — persists max size, compression level, grayscale toggle

3. Wire Formats

3.1 Image Envelope (IE4)

Prefix: IE4: + colon-delimited compact payload (base36 numeric fields)

Fields:

Field	Type	Description
sid	string	base36 token for 32-bit session ID
fmt	base36	ImageFormat.id (0 = AVIF, 1 = JPEG)
total	base36	Fragment count (1..255)
w	base36	Actual image width after compression (pixels)
h	base36	Actual image height after compression (pixels)
bytes	base36	Total compressed size in bytes
Compact format:

IE4:{sid}:{fmt}:{total}:{w}:{h}:{bytes}

Example (256×171 landscape image, 14 fragments):

IE4:a:0:e:74:4r:1mc

Note: sid is base36 on wire and expands to 8-hex internally. w and h reflect the actual post-compression dimensions, which preserve the source aspect ratio (contain within the configured max size).

3.2 Image Fetch Request (IR4 + binary)

Text format:

IR4:{sid}:{want}:{requesterKey6}

Binary payload format:

[magic=0x69][sid:4B][flags:1B][requesterKey6:6B][missingCount:1B][missingIndices...]

Field	Value
flags	bit0=1 => request missing indices, else all
requesterKey6	6-byte requester key prefix

3.3 Raw Image Packet (data plane)

Binary payload structure:

• Byte 0: magic 0x49 ('I')
• Bytes 1..4: session ID (4 bytes)
• Byte 5: fragment index (0-based)
• Bytes 6..N: image data (max 152 bytes per fragment)

Header is 6 bytes. Image format and total fragment count come from the IE4 envelope.

4. Compression Pipeline

ImageCodecService.compress(rawBytes, {maxDimension, compression, grayscale}):

1. Probe — decode source image at original resolution to read srcW × srcH.
2. Contain — compute dstW × dstH that fits within maxDimension × maxDimension while preserving aspect ratio. Images smaller than the cap are not upscaled.
3. Decode — dart:ui.instantiateImageCodec with targetWidth: dstW, targetHeight: dstH.
4. RGBA export — image.toByteData(format: rawRgba).
5. Grayscale (optional, default on) — luminance 0.299R + 0.587G + 0.114B applied in-place; if disabled, colour pixels are passed through unchanged.
6. PNG re-encode — ui.ImageDescriptor.raw → toByteData(format: png). Required because encodeAvif() takes an encoded image, not raw RGBA.
7. AVIF encode — flutter_avif.encodeAvif(pngBytes, maxQuantizer, minQuantizer, speed: 8). compression (10–90) maps to libavif CQ scale: maxQ = round(compression/100 × 63), minQ = round(maxQ × 0.65).
8. Fragment — fragmentImage() at 152 bytes/packet (≤255 packets).

Returns ({Uint8List bytes, int width, int height}) with the actual compressed dimensions.

User-configurable settings (ImagePreferences)

Setting	Key	Default	Range / Options
Max size	image_max_size	256	64 / 128 / 256
Compression	image_quality	90	10–90
Grayscale	image_grayscale	true	true / false

Expected sizes (grayscale AVIF, compression 90)

Max size	Typical size	Fragments
64×64	100–400 B	1–3
128×128	300–900 B	2–6
256×256	700–3000 B	5–20

Non-square images produce fewer fragments than the square equivalent because only the shorter axis is padded — no cropping occurs.

5. Outgoing Flow (Send)

1. User picks image from gallery or camera (image_picker).
2. ImagePreferences is read: max size, compression, grayscale.
3. ImageCodecService.compress() returns (bytes, width, height).
4. fragmentImage() splits bytes into ImagePacket list (≤255 fragments).
5. ImageProvider.cacheOutgoingSession() stores fragments in both the outgoing cache (for serving to peers) and the incoming session map (so the local bubble renders the image immediately without a fetch round-trip).
6. ImageEnvelope built with actual width/height from step 3.
7. Envelope sent via normal message path:
   • Channel: sendChannelMessage
   • Direct: sendTextMessage
8. Local placeholder message added (IE4: text, deliveryStatus.sending).

6. Incoming Flow (Receive)

6.1 IE4 envelope received

AppProvider calls imageProvider.registerEnvelope() and adds the message to chat. The bubble shows a grey square placeholder with a download icon.

6.2 Binary image fetch request received

AppProvider treats it as control-plane only (not added to chat):

• Resolves requester contact.
• Calls imageProvider.serveSessionTo() which streams all cached fragments.

6.3 Swarm control messages received

AppProvider also handles shared raw swarm discovery payloads:

• binary swarm requests advertise which image fragments are still missing
• binary swarm availability responses advertise which fragments another peer can relay
• swarm payloads arrive via pushRawData and are intercepted instead of being added to chat

Shared discovery and responder semantics are documented in Swarm Mode Technical Design.

6.4 Raw packet received (pushRawData, magic 0x49)

AppProvider.onRawDataReceived parses ImagePacket binary and calls imageProvider.addFragment(). When the session becomes complete, the bubble automatically rebuilds with the full image.

7. Outgoing Cache Details

ImageProvider outgoing cache:

• key: sessionId
• value: encoded fragment list + cached envelope + timestamp
• TTL: 15 minutes
• eviction: lazy on access

When cacheOutgoingSession() is called it also writes all fragments into _sessions[sessionId], so the sender sees the image immediately in the bubble (no tap-to-load required for own messages).

If a peer later receives only part of an image session, those received fragments can also be served onward to another requester during swarm recovery.

8. Display

ImageMessageBubble (max width 256 px):

• Complete session: AspectRatio(1.0) → AvifImage.memory(fit: cover) square thumbnail; tap → full-screen InteractiveViewer with fade transition.
• Incomplete/missing: grey square placeholder with download icon; tap → sends binary fetch request, with raw swarm fallback if the original sender path stalls.
• Loading: circular progress indicator showing received/total count.
• Error: broken-image icon.

Status line below thumbnail:

• Sender: 🖼️ {w}×{h} AVIF · {n} seg
• Receiver (complete): 🖼️ {w}×{h} AVIF
• Receiver (pending): 🖼️ Tap to load · {w}×{h}
• Loading: 📥 Loading… {received}/{total}

9. Transmit Time Estimate (UI)

Image bubbles and Message Technical Details show an estimated transmit time (~... tx).

The estimate is airtime-based (LoRa packet model), not just compressed image size:

• Source inputs:
  • total fragments and bytes from IE4 envelope
  • all numeric envelope values are decoded from base36
  • pathLen from message metadata
  • current radio params from deviceInfo: radioBw, radioSf, radioCr
• Per-fragment payload model:
  • meshHeader(2) + pathLen + imageHeader(6) + fragmentBytes
• LoRa airtime:
  • standard symbol-time formula (preamble + payload symbols)
• Mesh pacing/hops:
  • multiplied by (1 + airtimeBudgetFactor) where default factor is 1.0
  • multiplied by hop count (pathLen + 1)
• Total estimate:
  • sum over all fragments

BW handling:

• If radioBw is index 0..9, app maps it to Hz (7.8k .. 500k)
• If radioBw > 1000, it is treated as Hz directly

Fallback defaults are used when radio params are unavailable: SF10, BW250kHz, CR5.

10. Persistence

ImageProvider stores sessions in SharedPreferences under key stored_image_sessions_v1:

• Incoming: fragment list serialized as base64 binary packets.
• Outgoing: fragment list + envelope text + cachedAt timestamp.
• Expired outgoing sessions (> 15 min) are not restored on startup.

11. Operational Constraints

• No firmware changes required (reuses cmdSendRawData / pushRawData).
• On-demand fetch prefers the original sender, but a partial image can also be completed from alternate peers that already hold matching fragments.
• Raw return path requires a valid direct route to requester.
• Available on iOS and Android (image_picker + flutter_avif).
• Swarm discovery uses the same cmdSendRawData / pushRawData path as image fetch and fragment delivery.

11.1 Raw Binary Routing Semantics

• Image fragment payloads use companion command CMD_SEND_RAW_DATA (25 / 0x19).
• Companion push back to the app is PUSH_CODE_RAW_DATA (0x84).
• Over-the-air packet type for this flow is PAYLOAD_TYPE_RAW_CUSTOM (0x0F).
• This flow is direct-route only, not flood/broadcast:
  • it is sent to one destination path;
  • only nodes on that path relay it;
  • it is not received by everyone in the mesh.

12. Swarm Fallback Sequence

sequenceDiagram
  participant A as Original Sender
  participant P as Peer With Fragments
  participant N as Reachable Peers
  participant B as Receiver App

  A->>M: Send IE4 envelope
  M->>B: Deliver IE4
  B->>A: Direct binary image fetch request
  A->>B: Stream raw ImagePacket fragments (partial)
  Note over A,B: Sender path stops responding
  B->>N: Raw swarm requests with missing image fragment indices
  P->>B: Raw swarm availability response
  B->>P: Direct binary fetch request for missing subset
  P->>B: Stream remaining raw ImagePacket fragments
  B->>B: Reassemble completed image

13. High-Level Sequence

sequenceDiagram
  participant A as Sender App
  participant M as Mesh Chat
  participant B as Receiver App

  A->>A: Pick image (gallery or camera)
  A->>A: Compress: contain resize → grayscale → PNG → AVIF
  A->>A: Fragment into ≤152B packets
  A->>A: Cache outgoing + populate local session (immediate display)
  A->>M: Send IE4 envelope (actual w×h, fragment count)
  M->>B: Deliver IE4
  B->>B: Render grey placeholder bubble
  B->>A: Tap → send binary fetch request
  A->>B: Stream binary ImagePackets
  B->>B: Reassemble fragments
  B->>B: Display AVIF image (cover thumbnail)