Hold verified one-key DMs until contact acceptance

docs/companion_protocol.md

@@ -639,7 +639,7 @@ eight-peer session cache is replaced.
 `dm.one_key` is **off by default**, including when older preferences are loaded.
 With it off, the receiver verifies the introduction, emits a synthetic
 `PUSH_CODE_NEW_ADVERT` (`0x8A`) contact record for the app's advert list, and
-does not add a contact or decrypt the following DM. The user can add that
+does not add a contact or deliver the following DM. The user can add that
 record with the ordinary `CMD_ADD_UPDATE_CONTACT` (`0x09`) flow. It is an
 app-facing advert record, not a signed broadcast advert packet; it cannot be
 used with `CMD_IMPORT_CONTACT` (`0x12`). A signed, encrypted refusal also tells
@@ -647,7 +647,14 @@ the sender to stop including introductions for that peer on retries. The sender
 emits `PUSH_CODE_ONE_KEY_DM_REJECTED` (`0x91`, followed by the recipient's full
 32-byte public key) for apps that want to show the refusal. The refusal cache is
 limited to eight peers and resets on reboot or eviction. If the user later adds
-the contact, an ordinary retry can be delivered and acknowledged.
+the contact, already-received DMs can be delivered from the pending queue
+without the sender retransmitting them. The receiver holds up to **15 verified,
+decryptable text DMs** across pending senders in RAM. It drops the oldest held
+DM when a sixteenth arrives, ignores duplicate retries, and waits for space in
+the normal offline queue before releasing accepted DMs. Held DMs are lost on
+radio reboot; packets that never reached the receiver still require a retry.
+Packets received before a valid `DMK1` introduction cannot enter this queue.
+Delivery after acceptance is acknowledged to the sender when possible.
 Temporary anonymous-request entries do not grant text-DM permission.
 The pending advert notification requires a connected app; it is not saved as a
 contact on the radio. Existing apps must handle `0x91` to show a visible
@@ -665,9 +672,10 @@ uses an additional radio packet and its reported timeout includes the delay.
 For a two-radio hardware check, run `python3 tools/hil/one_key_dm.py --sender
 /dev/ttyACM0 --recipient /dev/ttyACM1 --reset-contact` with two nearby
 Companion radios on the same profile. The default test checks a pending advert,
-refusal, manual contact addition, then message notification and ACK without
-draining the recipient's message queue. Add `--auto-accept` to test the opt-in
-automatic path (restart the sender after a refusal to clear its session cache).
+refusal, manual contact addition, then delivery and ACK of the original DM
+without a retry or draining the recipient's message queue. Add `--auto-accept`
+to test the opt-in automatic path (restart the sender after a refusal to clear
+its session cache).
 Add `--invalid-signature-first` to check signature rejection.
 
 ---

docs/payloads.md

@@ -237,8 +237,10 @@ without a broadcast advert. The ciphertext contains:
 The signed bytes are the recipient's full public key followed by the tag,
 marker, and NUL-terminated name. The recipient verifies that signature against
 the 32-byte sender key in the anonymous request header. By default it exposes
-the sender as a pending, app-facing advert without storing a contact. If
-`comp.one_key_dm` is enabled, it stores the sender as a chat contact. A receiver
+the sender as a pending, app-facing advert without storing a contact. Up to 15
+verified, decryptable text packets from pending senders can be held in RAM and
+delivered after the contact is added, oldest first. If `comp.one_key_dm` is
+enabled, it stores the sender as a chat contact. A receiver
 that does not implement `DMK1` ignores the introduction; the following normal
 text packet still works when it already has the sender's key. The sender's full
 public key remains visible in the radio packet header, as with other anonymous

examples/companion_radio/MyMesh.cpp

@@ -578,8 +578,9 @@ void MyMesh::onAnonDataRecv(mesh::Packet* packet, const uint8_t* secret,
   contact.out_path_len = OUT_PATH_UNKNOWN;
   contact.lastmod = getRTCClock()->getCurrentTime();
   if (_prefs.one_key_dm_enabled != 1) {
+    rememberVerifiedPendingSender(sender.pub_key);
     // Mirror the manual-advert flow: notify the app with a complete, addable
-    // contact record, but do not store it or decrypt the following DM. This is
+    // contact record, but do not store it or deliver the following DM. This is
     // a synthetic advert notification, not a forged signed radio advert.
     onDiscoveredContact(contact, true, packet->path_len, packet->path);
 
@@ -611,6 +612,143 @@ void MyMesh::onAnonDataRecv(mesh::Packet* packet, const uint8_t* secret,
   onDiscoveredContact(*added, true, OUT_PATH_UNKNOWN, NULL);
 }
 
+void MyMesh::rememberVerifiedPendingSender(const uint8_t* pub_key) {
+  for (uint8_t i = 0; i < verified_pending_count; ++i) {
+    if (memcmp(verified_pending_keys[i], pub_key, PUB_KEY_SIZE) == 0) return;
+  }
+  if (verified_pending_count == MAX_HELD_ONE_KEY_DMS) {
+    for (uint8_t i = 1; i < verified_pending_count; ++i) {
+      memcpy(verified_pending_keys[i - 1], verified_pending_keys[i], PUB_KEY_SIZE);
+    }
+    --verified_pending_count;
+  }
+  memcpy(verified_pending_keys[verified_pending_count++], pub_key, PUB_KEY_SIZE);
+}
+
+void MyMesh::forgetVerifiedPendingSender(const uint8_t* pub_key) {
+  for (uint8_t i = 0; i < verified_pending_count; ++i) {
+    if (memcmp(verified_pending_keys[i], pub_key, PUB_KEY_SIZE) == 0) {
+      for (uint8_t j = i + 1; j < verified_pending_count; ++j) {
+        memcpy(verified_pending_keys[j - 1], verified_pending_keys[j], PUB_KEY_SIZE);
+      }
+      --verified_pending_count;
+      return;
+    }
+  }
+}
+
+void MyMesh::makeOneKeyDMId(uint8_t id[ONE_KEY_DM_ID_SIZE], uint32_t timestamp,
+                            const char* text) {
+  mesh::Utils::sha256(id, ONE_KEY_DM_ID_SIZE,
+                      reinterpret_cast<const uint8_t*>(&timestamp), sizeof(timestamp),
+                      reinterpret_cast<const uint8_t*>(text), strlen(text));
+}
+
+bool MyMesh::wasDeliveredOneKeyDM(const uint8_t* pub_key,
+                                  const uint8_t id[ONE_KEY_DM_ID_SIZE]) const {
+  for (uint8_t i = 0; i < delivered_dm_count; ++i) {
+    if (memcmp(delivered_dms[i].sender_key, pub_key, PUB_KEY_SIZE) == 0 &&
+        memcmp(delivered_dms[i].id, id, ONE_KEY_DM_ID_SIZE) == 0) return true;
+  }
+  return false;
+}
+
+void MyMesh::rememberDeliveredOneKeyDM(const uint8_t* pub_key,
+                                       const uint8_t id[ONE_KEY_DM_ID_SIZE]) {
+  DeliveredOneKeyDM& entry = delivered_dms[delivered_dm_next];
+  memcpy(entry.sender_key, pub_key, PUB_KEY_SIZE);
+  memcpy(entry.id, id, ONE_KEY_DM_ID_SIZE);
+  if (delivered_dm_count < MAX_HELD_ONE_KEY_DMS) ++delivered_dm_count;
+  delivered_dm_next = (delivered_dm_next + 1) % MAX_HELD_ONE_KEY_DMS;
+}
+
+bool MyMesh::onAddressedTextPacket(mesh::Packet* packet, uint8_t src_hash,
+                                   const uint8_t* mac_and_data, size_t len) {
+  for (uint8_t i = 0; i < verified_pending_count; ++i) {
+    const uint8_t* pub_key = verified_pending_keys[i];
+    if (pub_key[0] != src_hash) continue;
+    ContactInfo* existing = lookupContactByPubKey(pub_key, PUB_KEY_SIZE);
+    if (existing != NULL && existing->type != ADV_TYPE_NONE) continue;
+
+    uint8_t secret[PUB_KEY_SIZE];
+    self_id.calcSharedSecret(secret, pub_key);
+    uint8_t data[MAX_PACKET_PAYLOAD + 1];
+    const int decoded_len = mesh::Utils::MACThenDecrypt(secret, data, mac_and_data, len);
+    if (decoded_len <= 5 || decoded_len > MAX_PACKET_PAYLOAD) continue;
+    data[decoded_len] = 0;
+    if ((data[4] >> 2) != TXT_TYPE_PLAIN) continue;
+    const char* text = reinterpret_cast<const char*>(&data[5]);
+    if (strlen(text) > MAX_TEXT_LEN) continue;
+
+    uint32_t timestamp;
+    memcpy(&timestamp, data, sizeof(timestamp));
+    uint8_t id[ONE_KEY_DM_ID_SIZE];
+    makeOneKeyDMId(id, timestamp, text);
+    for (uint8_t j = 0; j < held_dm_count; ++j) {
+      if (memcmp(held_dms[j].sender_key, pub_key, PUB_KEY_SIZE) == 0 &&
+          memcmp(held_dms[j].id, id, ONE_KEY_DM_ID_SIZE) == 0) return true;
+    }
+
+    // All retained messages have passed MAC/decryption and sender-key checks.
+    // Keep one global FIFO of 15; the oldest decodable DM rolls off first.
+    if (held_dm_count == MAX_HELD_ONE_KEY_DMS) {
+      for (uint8_t j = 1; j < held_dm_count; ++j) held_dms[j - 1] = held_dms[j];
+      --held_dm_count;
+    }
+    HeldOneKeyDM& entry = held_dms[held_dm_count++];
+    memcpy(entry.sender_key, pub_key, PUB_KEY_SIZE);
+    memcpy(entry.id, id, ONE_KEY_DM_ID_SIZE);
+    entry.packet = *packet;
+    return true;
+  }
+  return false;
+}
+
+void MyMesh::releaseHeldOneKeyDMs() {
+  for (uint8_t i = 0; i < verified_pending_count; ) {
+    ContactInfo* contact = lookupContactByPubKey(verified_pending_keys[i], PUB_KEY_SIZE);
+    if (contact != NULL && contact->type != ADV_TYPE_NONE) {
+      uint8_t key[PUB_KEY_SIZE];
+      memcpy(key, verified_pending_keys[i], PUB_KEY_SIZE);
+      forgetVerifiedPendingSender(key);
+    } else {
+      ++i;
+    }
+  }
+
+  for (uint8_t i = 0; i < held_dm_count; ) {
+    HeldOneKeyDM& entry = held_dms[i];
+    ContactInfo* contact = lookupContactByPubKey(entry.sender_key, PUB_KEY_SIZE);
+    if (contact == NULL || contact->type == ADV_TYPE_NONE) { ++i; continue; }
+    if (offline_queue_len >= OFFLINE_QUEUE_SIZE) break;
+
+    uint8_t data[MAX_PACKET_PAYLOAD + 1];
+    const int decoded_len = mesh::Utils::MACThenDecrypt(
+        contact->getSharedSecret(self_id), data, &entry.packet.payload[2],
+        entry.packet.payload_len - 2);
+    if (decoded_len > 5 && decoded_len <= MAX_PACKET_PAYLOAD) {
+      data[decoded_len] = 0;
+      if ((data[4] >> 2) == TXT_TYPE_PLAIN) {
+        const char* text = reinterpret_cast<const char*>(&data[5]);
+        uint32_t timestamp;
+        memcpy(&timestamp, data, sizeof(timestamp));
+        if (strlen(text) <= MAX_TEXT_LEN) {
+          if (!wasDeliveredOneKeyDM(entry.sender_key, entry.id)) {
+            onMessageRecv(*contact, &entry.packet, timestamp, text);
+            rememberDeliveredOneKeyDM(entry.sender_key, entry.id);
+          }
+          uint8_t ack_hash[4];
+          mesh::Utils::sha256(ack_hash, sizeof(ack_hash), data, 5 + strlen(text),
+                              contact->id.pub_key, PUB_KEY_SIZE);
+          sendAckTo(*contact, ack_hash, sizeof(ack_hash));
+        }
+      }
+    }
+    for (uint8_t j = i + 1; j < held_dm_count; ++j) held_dms[j - 1] = held_dms[j];
+    --held_dm_count;
+  }
+}
+
 void MyMesh::queueMessage(const ContactInfo &from, uint8_t txt_type, mesh::Packet *pkt,
                           uint32_t sender_timestamp, const uint8_t *extra, int extra_len, const char *text) {
   int i = 0;
@@ -700,6 +838,9 @@ void MyMesh::sendFloodScoped(const mesh::GroupChannel& channel, mesh::Packet* pk
 
 void MyMesh::onMessageRecv(const ContactInfo &from, mesh::Packet *pkt, uint32_t sender_timestamp,
                            const char *text) {
+  uint8_t id[ONE_KEY_DM_ID_SIZE];
+  makeOneKeyDMId(id, sender_timestamp, text);
+  if (wasDeliveredOneKeyDM(from.id.pub_key, id)) return;
   markConnectionActive(from); // in case this is from a server, and we have a connection
   queueMessage(from, TXT_TYPE_PLAIN, pkt, sender_timestamp, NULL, 0, text);
 }
@@ -1530,7 +1671,22 @@ void MyMesh::handleCmdFrame(size_t len) {
     uint8_t *pub_key = &cmd_frame[1];
     ContactInfo *recipient = lookupContactByPubKey(pub_key, PUB_KEY_SIZE);
     uint32_t last_mod = getRTCClock()->getCurrentTime();  // fallback value if not present in cmd_frame
-    if (recipient) {
+    if (recipient != NULL && recipient->type == ADV_TYPE_NONE) {
+      // A prior anonymous request can leave this key in a transient slot.
+      // Accepting an advert must create a durable contact, not merely change
+      // that slot's type (transient slots are omitted from saved contacts).
+      ContactInfo contact;
+      updateContactFromFrame(contact, last_mod, cmd_frame, len);
+      contact.lastmod = last_mod;
+      contact.sync_since = 0;
+      if (addContact(contact)) {
+        memset(recipient, 0, sizeof(*recipient));
+        dirty_contacts_expiry = futureMillis(LAZY_CONTACTS_WRITE_DELAY);
+        writeOKFrame();
+      } else {
+        writeErrFrame(ERR_CODE_TABLE_FULL);
+      }
+    } else if (recipient) {
       updateContactFromFrame(*recipient, last_mod, cmd_frame, len);
       recipient->lastmod = last_mod;
       dirty_contacts_expiry = futureMillis(LAZY_CONTACTS_WRITE_DELAY);
@@ -2643,6 +2799,10 @@ void MyMesh::loop() {
 #endif
   }
 
+  // Contact acceptance can arrive through the app or a signed advert. Hold
+  // accepted DMs until the ordinary offline queue has room to deliver them.
+  if (held_dm_count > 0 || verified_pending_count > 0) releaseHeldOneKeyDMs();
+
   // is there are pending dirty contacts write needed?
   if (dirty_contacts_expiry && millisHasNowPassed(dirty_contacts_expiry)) {
     saveContacts();

examples/companion_radio/MyMesh.h

@@ -158,6 +158,8 @@ protected:
   void onAnonDataRecv(mesh::Packet* packet, const uint8_t* secret,
                       const mesh::Identity& sender, uint8_t* data,
                       size_t len) override;
+  bool onAddressedTextPacket(mesh::Packet* packet, uint8_t src_hash,
+                             const uint8_t* mac_and_data, size_t len) override;
   void queueMessage(const ContactInfo &from, uint8_t txt_type, mesh::Packet *pkt, uint32_t sender_timestamp,
                     const uint8_t *extra, int extra_len, const char *text);
 
@@ -305,6 +307,34 @@ private:
   uint32_t sendOneKeyIntroduction(const ContactInfo& contact);
   int next_ack_idx;
 
+  static constexpr uint8_t MAX_HELD_ONE_KEY_DMS = 15;
+  static constexpr uint8_t ONE_KEY_DM_ID_SIZE = 8;
+  uint8_t verified_pending_keys[MAX_HELD_ONE_KEY_DMS][PUB_KEY_SIZE] = {};
+  uint8_t verified_pending_count = 0;
+  struct HeldOneKeyDM {
+    uint8_t sender_key[PUB_KEY_SIZE];
+    uint8_t id[ONE_KEY_DM_ID_SIZE];
+    mesh::Packet packet;
+  };
+  HeldOneKeyDM held_dms[MAX_HELD_ONE_KEY_DMS];
+  uint8_t held_dm_count = 0;
+  struct DeliveredOneKeyDM {
+    uint8_t sender_key[PUB_KEY_SIZE];
+    uint8_t id[ONE_KEY_DM_ID_SIZE];
+  };
+  DeliveredOneKeyDM delivered_dms[MAX_HELD_ONE_KEY_DMS] = {};
+  uint8_t delivered_dm_count = 0;
+  uint8_t delivered_dm_next = 0;
+  void rememberVerifiedPendingSender(const uint8_t* pub_key);
+  void forgetVerifiedPendingSender(const uint8_t* pub_key);
+  static void makeOneKeyDMId(uint8_t id[ONE_KEY_DM_ID_SIZE], uint32_t timestamp,
+                             const char* text);
+  bool wasDeliveredOneKeyDM(const uint8_t* pub_key,
+                            const uint8_t id[ONE_KEY_DM_ID_SIZE]) const;
+  void rememberDeliveredOneKeyDM(const uint8_t* pub_key,
+                                 const uint8_t id[ONE_KEY_DM_ID_SIZE]);
+  void releaseHeldOneKeyDMs();
+
   #define ADVERT_PATH_TABLE_SIZE   16
   AdvertPath advert_paths[ADVERT_PATH_TABLE_SIZE]; // circular table
 };

src/Mesh.cpp

@@ -184,6 +184,10 @@ DispatcherAction Mesh::onRecvPacket(Packet* pkt) {
               break;
             }
           }
+          if (pkt->getPayloadType() == PAYLOAD_TYPE_TXT_MSG &&
+              onAddressedTextPacket(pkt, src_hash, macAndData, pkt->payload_len - i)) {
+            found = true;
+          }
           if (found) {
             pkt->markDoNotRetransmit();  // packet was for this node, so don't retransmit
           } else {
@@ -738,4 +742,4 @@ void Mesh::sendZeroHop(Packet* packet, uint16_t* transport_codes, uint32_t delay
   sendPacket(packet, 0, delay_millis);
 }
 
-}
+}

src/Mesh.h

@@ -94,6 +94,12 @@ protected:
   */
   virtual void onPeerDataRecv(Packet* packet, uint8_t type, int sender_idx, const uint8_t* secret, uint8_t* data, size_t len) { }
 
+  // Called for addressed text packets after contact lookup. Receivers can
+  // recognize and retain a packet from a verified, not-yet-added sender.
+  // Return true when the packet was handled and should not be retransmitted.
+  virtual bool onAddressedTextPacket(Packet* packet, uint8_t src_hash,
+                                     const uint8_t* mac_and_data, size_t len) { return false; }
+
   /**
    * \brief  A TRACE packet has been received. (and has reached the end of its given path)
    *         NOTE: this may have been initiated by another node.

src/helpers/BaseChatMesh.h

@@ -75,9 +75,10 @@ class BaseChatMesh : public mesh::Mesh {
   ConnectionInfo connections[MAX_CONNECTIONS];
 
   mesh::Packet* composeMsgPacket(const ContactInfo& recipient, uint32_t timestamp, uint8_t attempt, const char *text, uint32_t& expected_ack);
-  void sendAckTo(const ContactInfo& dest, const uint8_t* ack_hash, uint8_t ack_len=4);
 
 protected:
+  void sendAckTo(const ContactInfo& dest, const uint8_t* ack_hash, uint8_t ack_len=4);
+
   BaseChatMesh(mesh::Radio& radio, mesh::MillisecondClock& ms, mesh::RNG& rng, mesh::RTCClock& rtc, mesh::PacketManager& mgr, mesh::MeshTables& tables)
       : mesh::Mesh(radio, ms, rng, rtc, mgr, tables)
   {

tools/hil/one_key_dm.py

@@ -230,10 +230,8 @@ def run(sender_port: str, recipient_port: str, reset_contact: bool,
             recipient.request(bytes([9]) + advertised[1:], (0,))
             if recipient.contact(sender_key) is None:
                 raise AssertionError("manual acceptance did not add the sender")
-            retry = bytes([2, 0, 1]) + struct.pack("<I", timestamp) + recipient_key[:6] + text
-            sent = sender.request(retry, (6,), seconds=20)
-            expected_ack = sent[2:6]
-            timeout_ms = struct.unpack("<I", sent[6:10])[0]
+            # No retry: the already-transmitted DM must emerge from the
+            # receiver's bounded pending queue after the contact is added.
             until = time.monotonic() + min(timeout_ms / 1000 + 5, 120)
             while time.monotonic() < until:
                 for link in (sender, recipient):
@@ -256,7 +254,7 @@ def run(sender_port: str, recipient_port: str, reset_contact: bool,
             "ack_confirmed": confirmed,
             "recipient_message_waiting": waiting,
             "recipient_learned_sender": learned,
-            "mode": "auto_accept" if auto_accept else "manual_accept_after_refusal",
+            "mode": "auto_accept" if auto_accept else "buffered_accept_after_refusal",
             "pending_advert_received": advertised is not None,
             "sender_notified_of_refusal": rejected,
             "timeout_ms": timeout_ms,