ukmesh/docs/spam-detection.md

# Message-Spam Detection (Spam Watch)

Detects people spamming the MeshCore mesh by finding **repeated, near-duplicate
messages** sent in short time windows, groups them into incidents, and estimates
a **coarse** origin area. Powers the public `ukmesh.com/spam` dashboard.

This is distinct from the older advert/identity-spoof detector
(`backend/src/mqtt/spamDetector.ts`), which scores repeater/device adverts. This
system is about **chat-message** abuse.

## How it works

```
packets (type 5, decoded channel msgs)
        │  loadRecentMessages()       backend/src/spam/repository.ts
        ▼
MessageRecord[]  (one per packet_hash, with observers)
        │  clusterMessages()          backend/src/spam/cluster.ts
        ▼
Incident[]       (near-duplicate text, close in time)
        │  estimateOrigin()           backend/src/spam/origin.ts
        │  sanitizeIncident()         backend/src/spam/sanitize.ts
        ▼
spam_message_incidents / spam_message_members   (public_json is pre-sanitized)
        │  getPublic*()               backend/src/spam/repository.ts
        ▼
/api/spam/messages/*  →  Spam Watch dashboard (frontend SpamPage)
```

Only decoded channel messages are used (MeshCore public/known-key channels where
`payload.decrypted.message` exists). Direct messages and unknown-key channels stay
encrypted and are never inspected.

### Pipeline stages

1. **Normalize** (`normalize.ts`) — lowercase, trim, collapse whitespace,
   normalize punctuation, collapse stretched repeats (`heyyyy`→`heyy`), and
   strip/canonicalize URLs (drop scheme/`www.`/query so tracking variants
   collapse). `ukmesh.com/spam` is recognised as a marker — supporting evidence,
   never sufficient on its own.
2. **Fuzzy match** (`similarity.ts`) — message similarity blends trigram Dice,
   token-set Jaccard and Levenshtein ratio; a shared canonical URL is a strong
   signal. Username similarity adds prefix/substring affinity so `John`,
   `John2`, `John_UK` read as related.
3. **Cluster** (`cluster.ts`) — time-ordered greedy clustering inside a join
   window. Similar usernames lower the text-similarity bar but never create a
   cluster alone. A qualification gate then separates abuse from normal chatter
   (see below).
4. **Estimate origin** — two methods, path-based preferred:

   **a. Relay-path adjacency chain-walk** (`spamResolver.ts`, used first for
   public incidents). Every reception carries its ordered relay path; the **first
   repeater is the source of truth** (the spammer transmitted into it). We take
   the **closest receivers** (lowest hop count) and **walk each path backward from
   the observer**, hop by hop: at each step we pick the candidate node that has a
   confirmed RF link (`node_links`) to the previously-resolved node, else the
   nearest candidate within range. Because a consecutive `(AA,BB)` hash pair is
   only realisable where two such repeaters sit within range, the adjacency chain
   disambiguates the otherwise-ambiguous 1-byte hashes — and the **random
   deviations in the middle of the paths** give many independent chains that
   converge on the same source. The chain's first repeaters are then combined into
   a geographic consensus. Confidence is **tempered by the closest observer's hop
   count**: a 2-hop reception is trusted (short chain); a source only heard 5–7
   hops out is honestly less certain (long chain, compounding per-hop error) even
   when the chains agree. (`spamResolver.ts` is an independent copy of the lazy
   path resolver so the production resolver is never touched.)

   **b. Observer-signal fallback** (`origin.ts`) — when paths can't be resolved
   (no path hashes, or no confident consensus), fall back to the observer
   estimate below. A flood radiates outward from its source,
   so an observer that heard it at a **low hop count** sits close to the source
   while distant relays only heard it after it crossed the network. The estimate
   is therefore **anchored on the closest-receiver cohort** (the lowest hop
   counts available — `min-hop + originNearHopSlack`, at least everyone within
   `originNearHopMax` hops); their location gives the centroid, and the radius is
   their spread plus a per-hop allowance. Confidence comes from how close the
   nearest reception was — both absolutely and *relative* to how far the flood
   otherwise travelled (a 2-hop reception of something that otherwise relayed 14
   hops is strong localisation). This works UK-wide: dense areas yield tight,
   high-confidence pins; sparse areas (where the closest reception is several
   hops out) honestly produce a broad radius and lower confidence rather than a
   false-precise point. Snaps to a coarse named UK region.

   **Coverage note:** localisation quality tracks observer density — strong
   across England, broader/lower-confidence in the far north of Scotland and the
   islands. With fewer than `originMinObservers` geolocated receivers the origin
   is reported as `insufficient` (no map zone) rather than guessed.
5. **Sanitize** (`sanitize.ts`) — the single gate to public output (see Privacy).
6. **Persist** (`repository.ts`) — upsert incidents; `first_seen` preserved via
   `LEAST`; in-window incidents that no longer cluster are removed; older history
   is frozen.

## What counts as spam (and what doesn't)

A cluster only becomes an incident when it is genuine abuse, not normal traffic:

- **Requires** `minTransmissions` near-duplicates with `minBurst` inside one
  burst window.
- **Requires substantive content** — at least `minContentChars`, *or* a URL,
  *or* the spam marker. Trivial/connectivity messages (`test`, `ack`, `morning`,
  `gm`, …) never qualify on their own.
- **Requires a suspicious sender pattern** — a single repeating sender, one
  dominant sender, or **rotating similar usernames**. Many *dissimilar* people
  sending the same thing is treated as broad legitimate chatter, not one abuser.
- The **test channel is excluded by default** (it carries repeated test traffic
  by design). Configurable via `SPAM_MESSAGE_EXCLUDE_CHANNELS`. A flood that
  *also* hits another channel (e.g. Public) is still caught via those copies.
- Content is dampened only when it is **dominated by** connectivity/test tokens
  — i.e. once benign words, bare numbers and 1–2-char fragments are removed,
  nothing substantive is left (`test`, `rx ack 73`, `test 123`). A templated
  message that merely *contains* the word `test` (e.g. an automated
  `ANDY KIRBY MESHCORE(TM) 00:16:4: !test` flood) keeps real words and is **not**
  treated as testing — so this camouflage no longer hides spam.
- A **sustained high-volume flood** (≥ `floodMinMessages` near-duplicates) is a
  strong automated-abuse signal on its own and is scored up — but a benign
  test/connectivity flood is never rewarded, so it stays below the public floor
  no matter how large it grows.

## Incident lifecycle

- **active / ongoing** while the last matching message is within
  `SPAM_MESSAGE_ONGOING_WINDOW_MIN`.
- **closed** after that cooldown elapses with no new matching messages.

The periodic analyzer recomputes recent incidents every
`SPAM_MESSAGE_INTERVAL_MS` and re-evaluates active→closed transitions each run.

## Configuration

All knobs live in `backend/src/spam/config.ts` and read from the environment.

| Env var | Default | Meaning |
| --- | --- | --- |
| `SPAM_MESSAGE_ANALYZER_ENABLED` | `true` | Run the in-process periodic analyzer |
| `SPAM_MESSAGE_INTERVAL_MS` | `300000` | Analyzer interval (5 min) |
| `SPAM_MESSAGE_WINDOW_HOURS` | `24` | Rolling window the analyzer (re)clusters |
| `SPAM_MESSAGE_TEXT_SIM` | `0.82` | Min text similarity to join a cluster |
| `SPAM_MESSAGE_TEXT_SIM_NAME` | `0.6` | Lower text bar when usernames are similar |
| `SPAM_MESSAGE_NAME_SIM` | `0.7` | Min username similarity (evidence) |
| `SPAM_MESSAGE_JOIN_WINDOW_MIN` | `30` | Max gap to join the same incident |
| `SPAM_MESSAGE_MIN_TRANSMISSIONS` | `8` | Min distinct messages for an incident |
| `SPAM_MESSAGE_BURST_WINDOW_MIN` | `10` | Burst window |
| `SPAM_MESSAGE_MIN_BURST` | `3` | Min messages inside one burst window |
| `SPAM_MESSAGE_MIN_CONTENT_CHARS` | `12` | Min canonical length (unless URL/marker) |
| `SPAM_MESSAGE_MAX_INDEPENDENT_SENDERS` | `4` | Above this (dissimilar) → broad chatter |
| `SPAM_MESSAGE_DOMINANT_SENDER_SHARE` | `0.6` | One sender at this share qualifies |
| `SPAM_MESSAGE_FLOOD_MIN_MESSAGES` | `25` | ≥ this many near-duplicates → sustained-flood boost |
| `SPAM_MESSAGE_EXCLUDE_CHANNELS` | `test` | Channel labels skipped (comma list) |
| `SPAM_MESSAGE_PUBLIC_MIN_SCORE` | `0.5` | Confidence floor for public list / status |
| `SPAM_MESSAGE_ONGOING_WINDOW_MIN` | `30` | How long an incident stays "ongoing" |
| `SPAM_MESSAGE_ORIGIN_USE_PATHS` | `true` | Anchor origin on the relay-path first repeater |
| `SPAM_MESSAGE_ORIGIN_PATH_MAX_PACKETS` | `40` | Max closest-receiver transmissions resolved |
| `SPAM_MESSAGE_ORIGIN_PATH_MIN_VOTES` | `3` | Min resolved paths before a path anchor is trusted |
| `SPAM_MESSAGE_ORIGIN_PATH_CLUSTER_KM` | `45` | Consensus cluster radius (outliers beyond it dropped) |
| `SPAM_MESSAGE_ORIGIN_PATH_AMBIGUOUS_WEIGHT` | `0.4` | Weight of an ambiguous vs certain repeater resolution |
| `SPAM_MESSAGE_ORIGIN_MIN_OBSERVERS` | `2` | Min observers before estimating origin (fallback) |
| `SPAM_MESSAGE_ORIGIN_NEAR_HOP_MAX` | `2` | Receivers within this many hops count as "near source" |
| `SPAM_MESSAGE_ORIGIN_NEAR_HOP_SLACK` | `1` | Width (hops) of the closest-receiver cohort that anchors it |
| `SPAM_MESSAGE_ORIGIN_PER_HOP_KM` | `12` | Radius allowance per hop (≈ LoRa link range) |
| `SPAM_MESSAGE_ORIGIN_MIN_RADIUS_KM` | `8` | Floor on reported radius (privacy) |
| `SPAM_MESSAGE_COARSEN_STEP_DEG` | `0.1` | Public coordinate coarsening (~11 km) |
| `SPAM_MESSAGE_REGION_SNAP_KM` | `70` | Max distance to snap to a named region |

**Tuning sensitivity:** lower `SPAM_MESSAGE_PUBLIC_MIN_SCORE` to surface more
(noisier) clusters; raise `SPAM_MESSAGE_MIN_TRANSMISSIONS` / `MIN_CONTENT_CHARS`
to be stricter.

## Confidence scores

Two separate confidences are shown:

- **Detection confidence** (incident `confidence`, 0–1) — how strongly the
  pattern resembles spam. Built from volume, burst density, sustained-flood
  size, repeated links, the spam marker, and rotating sender names; content
  dominated by connectivity/test tokens is dampened. `High ≥ 0.66`,
  `Medium ≥ 0.4`, else `Low`. Only incidents at/above
  `SPAM_MESSAGE_PUBLIC_MIN_SCORE` appear publicly by default.
- **Origin confidence** (origin `confidence` + `level`) — how trustworthy the
  reported *zone* is (region + radius), driven mainly by how close the nearest
  reception was (low hop count) and how much it stands out from the flood's
  overall reach, plus corroboration from other near-source receivers.
  `insufficient` means too few geolocated observers to triangulate; the map
  zone is then omitted. Note that confidence is confidence in the *zone* — a
  larger radius absorbs the uncertainty, so "high confidence" means the source
  is very likely inside that (coarse) area, not that it is pinpointed.

A single low-hop receiver gives a confident *region* but a coarse radius; more
near-source receivers tighten the radius and raise confidence further. On the
dashboard, click **Estimated origin** (or "View area on map") to see the coarse
confidence circle on a map.

## Privacy model

Abuse mitigation only. The public API serves the pre-computed `public_json`
column, which is produced solely by `sanitizeIncident()`:

- **Sender names** are redacted to a short non-identifying hint (`John_UK`→`Jo…`)
  that still conveys variant similarity. Raw names stay in local-only columns.
- **Message samples** strip URLs (`[link]`, marker → `[spam-link]`), `@mentions`
  (`[mention]`) and long hex/number runs (`[id]`), and are length-bounded.
- **Locations** are coarse only: a region label, a confidence level, and an
  optional broad heat zone (centre coarsened to a ~0.1° grid, radius floored at
  `ORIGIN_MIN_RADIUS_KM` and bucketed to 5 km). Exact coordinates, observer ids
  and node metadata are never published.
- Clusters are labelled **"suspected spam cluster"** — neutral, not an
  accusation of any named individual.
- No IPs, credentials, public keys or channel keys are exposed.

## API

| Endpoint | Description |
| --- | --- |
| `GET /api/spam/messages/status` | Current status: ongoing?, counts, last activity |
| `GET /api/spam/messages/incidents?status=&minConfidence=&limit=&offset=` | Sanitized incident list |
| `GET /api/spam/messages/incidents/:id` | One incident + sanitized timeline |

`minConfidence` defaults to `SPAM_MESSAGE_PUBLIC_MIN_SCORE`; pass `0` to include
lower-confidence clusters (the dashboard's "show lower-confidence" toggle).

## Deployment

The derived tables are created by migration
`backend/src/db/migrations/005_spam_message_incidents.sql` (additive, no
destructive changes). The one-shot `db-migrate` service applies unrecorded
migrations before the backend starts. Do not apply the SQL file or insert its
migration-ledger row manually.

For a normal deployment, use:

```bash
docker compose up -d --build
```

To inspect or apply pending migrations manually after building the image:

```bash
docker compose build db-migrate
docker compose run --rm db-migrate
```

The in-process analyzer starts automatically. To backfill historical incidents
or run a one-off pass:

```bash
# dry run over 30 days (no writes)
docker compose exec backend node dist/tools/recomputeSpamMessages.js --hours 720
# apply
docker compose exec backend node dist/tools/recomputeSpamMessages.js --hours 720 --apply
```

## Tests

Pure logic (normalization, fuzzy matching, username similarity, clustering,
ongoing/closed status, origin scoring, sanitization) is covered by unit tests:

```bash
cd backend && npm ci && npm test
```