diff --git a/AUDIO-WORKBENCH.md b/AUDIO-WORKBENCH.md new file mode 100644 index 0000000..18cff4c --- /dev/null +++ b/AUDIO-WORKBENCH.md @@ -0,0 +1,159 @@ +# AUDIO-WORKBENCH.md — Sound Shaping & Debug Interface + +## Problem + +Live packets arrive randomly and animate too fast to understand what's happening musically. You hear sound, but can't connect it to what the data is doing — which bytes become which notes, why this packet sounds different from that one. + +## Milestone 1: Packet Jukebox + +A standalone page (`#/audio-lab`) that lets you trigger packets manually and understand the data→sound mapping. + +### Packet Buckets + +Pre-load representative packets from the database, bucketed by type: + +| Type ID | Name | Typical Size | Notes | +|---------|------|-------------|-------| +| 0x04 | ADVERT | 109-177 bytes | Node advertisements, most musical (long payload) | +| 0x05 | GRP_TXT | 18-173 bytes | Group messages, wide size range | +| 0x01 | TXT_MSG | 22-118 bytes | Direct messages | +| 0x02 | ACK/REQ | 22-57 bytes | Short acknowledgments | +| 0x09 | TRACE | 11-13 bytes | Very short, sparse | +| 0x00 | RAW | 22-33 bytes | Raw packets | + +For each type, pull 5-10 representative packets spanning the size range (smallest, median, largest) and observation count range (1 obs, 10+ obs, 50+ obs). + +### API + +New endpoint: `GET /api/audio-lab/buckets` + +Returns pre-selected packets grouped by type with decoded data and raw_hex. Server picks representatives so the client doesn't need to sift through hundreds. + +### UI Layout + +``` +┌─────────────────────────────────────────────────────┐ +│ 🎵 Audio Lab │ +├──────────┬──────────────────────────────────────────┤ +│ │ │ +│ ADVERT │ [▶ Play] [🔁 Loop] [⏱ Slow 0.5x] │ +│ ▸ #1 │ │ +│ ▸ #2 │ ┌─ Packet Data ──────────────────────┐ │ +│ ▸ #3 │ │ Type: ADVERT │ │ +│ │ │ Size: 141 bytes (payload: 138) │ │ +│ GRP_TXT │ │ Hops: 3 Observations: 12 │ │ +│ ▸ #1 │ │ Raw: 04 8b 33 87 e9 c5 cd ea ... │ │ +│ ▸ #2 │ └────────────────────────────────────┘ │ +│ │ │ +│ TXT_MSG │ ┌─ Sound Mapping ────────────────────┐ │ +│ ▸ #1 │ │ Instrument: Bell (triangle) │ │ +│ │ │ Scale: C major pentatonic │ │ +│ TRACE │ │ Notes: 12 (√138 ≈ 11.7) │ │ +│ ▸ #1 │ │ Filter: 4200 Hz (3 hops) │ │ +│ │ │ Volume: 0.48 (12 obs) │ │ +│ │ │ Voices: 4 (12 obs, capped) │ │ +│ │ │ Pan: -0.3 (lon: -105.2) │ │ +│ │ └────────────────────────────────────┘ │ +│ │ │ +│ │ ┌─ Note Sequence ────────────────────┐ │ +│ │ │ #1: byte 0x8B → C4 (880Hz) 310ms │ │ +│ │ │ gap: 82ms (Δ=0x58) │ │ +│ │ │ #2: byte 0x33 → G3 (392Hz) 120ms │ │ +│ │ │ gap: 210ms (Δ=0xB4) │ │ +│ │ │ ... │ │ +│ │ └────────────────────────────────────┘ │ +│ │ │ +│ │ ┌─ Byte Visualizer ──────────────────┐ │ +│ │ │ ████░░██████░░░████████░░██░░░░████ │ │ +│ │ │ ↑ ↑ ↑ ↑ │ │ +│ │ │ sampled bytes highlighted in payload │ │ +│ │ └────────────────────────────────────┘ │ +├──────────┴──────────────────────────────────────────┤ +│ BPM [====●========] 120 Vol [==●===========] 30 │ +│ Voice: [constellation ▾] │ +└─────────────────────────────────────────────────────┘ +``` + +### Key Features + +1. **Play button** — triggers `sonifyPacket()` with the selected packet +2. **Loop** — retrigger every N seconds (configurable) +3. **Slow mode** — 0.25x / 0.5x / 1x / 2x tempo override (separate from BPM, multiplies it) +4. **Note sequence breakdown** — shows every sampled byte, its MIDI note, frequency, duration, gap to next. Highlights each note in real-time as it plays. +5. **Byte visualizer** — hex dump of payload with sampled bytes highlighted. Shows which bytes the voice module chose and what they became. +6. **Sound mapping panel** — shows computed parameters (instrument, scale, filter, pan, volume, voice count) so you can see exactly why it sounds the way it does. + +### Playback Highlighting + +As each note plays, highlight: +- The corresponding byte in the hex dump +- The note row in the sequence table +- A playhead marker on the byte visualizer bar + +This connects the visual and auditory — you SEE which byte is playing RIGHT NOW. + +--- + +## Milestone 2: Parameter Overrides + +Once you can hear individual packets clearly, add override sliders to shape the sound: + +- **Scale override** — force any scale regardless of packet type +- **Instrument override** — force oscillator type (sine/triangle/square/sawtooth) +- **ADSR sliders** — attack, decay, sustain, release +- **Filter cutoff** — manual override of hop-based cutoff +- **Voice count** — force 1-4 voices regardless of observation count +- **Detune amount** — how much chord spread per voice +- **Pan lock** — force stereo position + +Each override has a "lock" toggle — locked = your value, unlocked = data-driven. + +The voice module's `play()` would accept an `overrides` object from the workbench. + +--- + +## Milestone 3: A/B Voice Comparison + +- Split-screen: two voice modules side by side +- Same packet, different voices +- "Play Both" button with configurable delay between them +- Good for iterating on v2/v3 voices against v1 constellation + +--- + +## Milestone 4: Sequence Editor + +- Drag packets into a timeline to create a sequence +- Adjust timing between packets manually +- Play the sequence as a composition +- Export as audio (MediaRecorder API → WAV/WebM) +- Useful for demoing "this is what the mesh sounds like" without waiting for live traffic + +--- + +## Milestone 5: Live Annotation Mode + +- Toggle on live map that shows the sound mapping panel for each packet as it plays +- Small floating card near the animated path showing: type, notes, instrument +- Fades out after the notes finish +- Connects the live visualization with the audio in real-time + +--- + +## Architecture Notes + +- Audio Lab is a new SPA page like packets/nodes/analytics +- Reuses existing `MeshAudio.sonifyPacket()` and voice modules +- Voice modules need a small extension: `play()` should return a `NoteSequence` object describing what it will play, not just play it. This enables the visualizer. +- Or: add a `describe(parsed, opts)` method that returns the mapping without playing +- BPM/volume/voice selection shared with live map via `MeshAudio.*` + +## Implementation Order + +1. API endpoint for bucketed representative packets +2. Basic page layout with packet list and play button +3. Sound mapping panel (computed parameters display) +4. Note sequence breakdown +5. Playback highlighting +6. Byte visualizer +7. Override sliders (M2)