From 3f8b8aec79765037170f3b3a48adbb3fa16bd97b Mon Sep 17 00:00:00 2001 From: you Date: Sun, 22 Mar 2026 09:15:14 +0000 Subject: [PATCH] =?UTF-8?q?Audio=20plan:=20final=20=E2=80=94=20byte-driven?= =?UTF-8?q?=20timing,=20BPM=20tempo=20control,=20no=20fixed=20spacing?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- AUDIO-PLAN.md | 178 +++++++++++++++++++++----------------------------- 1 file changed, 73 insertions(+), 105 deletions(-) diff --git a/AUDIO-PLAN.md b/AUDIO-PLAN.md index 4cd0569..86e129a 100644 --- a/AUDIO-PLAN.md +++ b/AUDIO-PLAN.md @@ -2,136 +2,104 @@ *Turn raw packet bytes into generative music.* -## Available Data Per Packet - -| Field | Range | Musical Mapping | -|-------|-------|----------------| -| `raw_hex` | 20-60+ bytes | Melody — each byte = a note | -| `payload_type` | ADVERT, GRP_TXT, TXT_MSG, TRACE | Instrument/scale selection | -| `hop_count` | 1-12+ | Note duration | -| `path` | node sequence | Arpeggio pattern | -| `SNR` | dB value | ~~mostly unavailable~~ — skip | -| `RSSI` | dBm value | ~~mostly unavailable~~ — skip | -| `hash_size` | 1-4 bytes | — | -| `observation_count` | 1-40+ | Chord voicing / polyphony | -| `channel_hash` | 0-255 | Only on GRP_TXT — skip as primary | -| `arrival_rate` | packets/min | Tempo (BPM) | - ## What Every Packet Has (guaranteed) - `raw_hex` — melody source -- `hop_count` — duration + filter -- `observation_count` — volume/polyphony +- `hop_count` — note duration + filter cutoff +- `observation_count` — volume + chord voicing - `payload_type` — instrument + scale + root key -- `packet_hash` — unique identifier +- `node_lat/lon` — stereo pan - `timestamp` — arrival timing -| `node_lat/lon` | coordinates | Stereo pan | -## Mapping Design +## Final Mapping -### Melody from Raw Bytes -- **Header bytes configure the voice** — payload type selects instrument/scale/key (see below), flags and transport codes select articulation and envelope shape. Header bytes are NOT played as notes. -- **Payload bytes are the melody** — evenly sampled and quantized to a scale: - - Note count = `sqrt(payload_length)`, so proportional to packet size: - - 16-byte payload → 4 notes - - 36-byte payload → 6 notes - - 64-byte payload → 8 notes - - Bytes selected via even spacing across the full payload (not just first N) — represents the whole packet's "shape" - - Each byte (0-255) quantized to the scale selected by payload type, across 2-3 octaves - - NOT chromatic (too chaotic) — pentatonic or modal scales - - Note timing: ~150-200ms per note, so 4-8 notes = 0.6-1.6 seconds per packet -- Different payload types get different scales: - - **ADVERT** → major pentatonic (bright, announcing, beacon-like) - - **GRP_TXT** → minor pentatonic (conversational, human) - - **TXT_MSG** → natural minor (direct, intentional) - - **TRACE** → whole tone (mysterious, searching, probing) +| Data | Musical Role | +|------|-------------| +| **payload_type** | Instrument + scale + root key | +| **payload bytes** (evenly sampled, sqrt(len) count) | Melody notes (pitch) | +| **byte value** | Note length (higher = longer sustain, lower = staccato) | +| **byte-to-byte delta** | Note spacing (big jump = longer gap, small = rapid) | +| **hop_count** | Low-pass filter cutoff (more hops = more muffled) | +| **observation_count** | Volume + chord voicing (more observers = louder + stacked detuned voices) | +| **node longitude** | Stereo pan (west = left, east = right) | +| **BPM tempo** (user control) | Master time multiplier on all durations | -### Rhythm from Network Activity -- Packet arrival rate sets effective BPM - - Busy network = fast tempo, quiet = ambient drone -- Hop count = note duration: - - 1 hop = sixteenth note (short staccato) - - 3 hops = quarter note - - 8+ hops = whole note (sustained, traveled far) +## Instruments & Scales by Type -### Timbre from Payload Type -- **ADVERT** = soft pad or bell (constant background beacons) -- **GRP_TXT** = plucked string or marimba (conversation, human-initiated) -- **TXT_MSG** = piano (direct, personal) -- **TRACE** = reversed cymbal or ethereal whoosh (probing) +| Type | Instrument | Scale | Root | +|------|-----------|-------|------| +| ADVERT | Bell / pad | C major pentatonic | C | +| GRP_TXT | Marimba / pluck | A minor pentatonic | A | +| TXT_MSG | Piano | E natural minor | E | +| TRACE | Ethereal synth | D whole tone | D | -### Spatial from Geography -- Node longitude → stereo pan (west = left, east = right) -- Or: packet origin's X position on current map viewport = pan -- Distance between hops → reverb amount (long hops = more reverb, signal traveled far) +## How a Packet Plays -### Dynamics from Observations + Hops -- **Observation count → velocity/volume**: more observers heard it = louder, more present -- **Hop count → filter cutoff**: few hops = bright and clear (nearby), many hops = muffled/filtered (traveled far, degraded) +1. **Header configures the voice** — payload type selects instrument, scale, root key. Flags/transport codes select envelope shape. Header bytes are NOT played as notes. +2. **Sample payload bytes** — pick `sqrt(payload_length)` bytes, evenly spaced across payload: + - 16-byte payload → 4 notes + - 36-byte payload → 6 notes + - 64-byte payload → 8 notes +3. **Each sampled byte → a note:** + - **Pitch**: byte value (0-255) quantized to selected scale across 2-3 octaves + - **Length**: byte value maps to sustain duration (low byte = short staccato ~50ms, high byte = sustained ~400ms) + - **Spacing**: delta between current and next sampled byte determines gap to next note (small delta = rapid fire, large delta = pause). Scaled by BPM tempo multiplier. +4. **Filter**: low-pass cutoff from hop_count — few hops = bright/clear, many hops = muffled (signal traveled far) +5. **Volume**: observation_count — more observers = louder +6. **Chord voicing**: if observations > 1, stack slightly detuned voices (±5-15 cents per voice, chorus effect) +7. **Pan**: origin node longitude mapped to stereo field +8. **All timings scaled by BPM tempo control** -### Harmony from Observations -- 1 observation = single note -- Multiple observers = chord voicing (each adds a voice) -- Each observer slightly detuned (chorus effect) — more observers = richer, more "present" -- Alternative: observation count triggers arpeggiation speed +## UI Controls -### Root Key from Payload Type -- Each payload type gets a fixed root note — always available, clean separation: - - **ADVERT** → C (beacons, foundation) - - **GRP_TXT** → A (conversation, warm) - - **TXT_MSG** → E (direct, bright) - - **TRACE** → D (searching, modal) -- This replaces channel hash (only available on GRP_TXT) - -### Ambient Layer from ADVERTs -- ADVERTs are constant background heartbeats → generative ambient drone -- Active node count modulates drone complexity (more nodes = richer harmonics) -- Could use ADVERT battery level to modulate brightness of the drone +- **Audio toggle** — on/off (next to Matrix / Rain) +- **BPM tempo slider** — master time multiplier (slow = ambient, fast = techno) +- **Volume slider** — master gain +- **Mute button** — pause audio without losing toggle state ## Implementation -### Library: Tone.js -- Built on Web Audio API, ~150KB +### Library: Tone.js (~150KB) - `Tone.Synth` / `Tone.PolySynth` for melody + chords -- `Tone.Sampler` if we want realistic instrument sounds -- `Tone.Transport` for tempo-synced playback tied to packet rate -- `Tone.Reverb`, `Tone.Filter`, `Tone.Chorus` for signal-quality effects -- `Tone.Panner` for geographic stereo positioning +- `Tone.Sampler` for realistic instruments +- `Tone.Filter` for hop-based cutoff +- `Tone.Chorus` for observation detuning +- `Tone.Panner` for geographic stereo +- `Tone.Reverb` for spatial depth -### Integration Points -- New **"Audio"** toggle on live map controls (next to Matrix / Rain) +### Integration - `animatePacket(pkt)` also calls `sonifyPacket(pkt)` -- Optional: **"Sonify"** button on packet detail page — hear one packet's melody -- Master volume slider in controls -- Mute button (separate from toggle — lets you pause audio without losing state) +- Optional "Sonify" button on packet detail page +- Web Audio runs on separate thread — won't block UI/animations +- Polyphony capped at 8-12 voices to prevent mudding +- Voice stealing when busy -### Core Function Sketch +### Core Function ``` sonifyPacket(pkt): 1. Extract raw_hex → byte array - 2. Select scale based on payload_type - 3. Map first 8-16 bytes to notes in scale (quantized) - 4. Set note duration from hop_count - 5. Set velocity from observation_count (more observers = louder) - 6. Set filter cutoff from hop_count (more hops = more muffled) - 7. Set pan from origin longitude - 9. If observation_count > 1: - - Play as chord (stack observation_count voices) - - Each voice slightly detuned (+/- cents) - 10. Trigger synth.triggerAttackRelease() + 2. Separate header (first ~3 bytes) from payload + 3. Header → select instrument, scale, root key, envelope + 4. Sample sqrt(payload.length) bytes evenly across payload + 5. For each sampled byte: + - pitch = quantize(byte, scale, rootKey) + - duration = map(byte, 50ms, 400ms) × tempoMultiplier + - gap to next = map(abs(nextByte - byte), 30ms, 300ms) × tempoMultiplier + 6. Set filter cutoff from hop_count + 7. Set gain from observation_count + 8. Set pan from origin longitude + 9. If observation_count > 1: detune +/- cents per voice + 10. Schedule note sequence via Tone.js ``` -### Performance Considerations -- Web Audio runs on separate thread — won't block UI/animations -- Limit polyphony (max 8-12 simultaneous voices) to prevent audio mudding -- Use note pooling / voice stealing when busy -- ADVERT drone should be a single oscillator modulated, not per-packet +## The Full Experience -### The Full Experience -Matrix mode + Rain + Audio = watching green hex bytes flow across the map, columns of raw data raining down, and each packet plays its own melody from its actual bytes. Quiet periods are sparse atmospheric ambience; traffic bursts become dense polyrhythmic cascades. +Matrix mode + Rain + Audio: green hex bytes flow across the map, columns of raw data rain down, and each packet plays its own unique melody derived from its actual bytes. Quiet periods are sparse atmospheric ambience; traffic bursts become dense polyrhythmic cascades. Crank the BPM for techno, slow it down for ambient. -### Future Ideas -- "Record" button → export as MIDI or WAV -- Packet type mute toggles (silence ADVERTs, only hear messages) +## Future Ideas + +- "Record" button → export MIDI or WAV +- Per-type mute toggles (silence ADVERTs, only hear messages) - "DJ mode" — crossfade between regions - Historical playback at accelerated speed = mesh network symphony -- Different "presets" (ambient, techno, classical mapping) +- Presets (ambient, techno, classical, minimal) +- ADVERT ambient drone layer (single modulated oscillator, not per-packet)