beacon-server/docs/historical-stats.md

# Historical stats

## The rule

Historical stats never scan `packet_observations` per request or per refresh. They read
hourly rollup tables filled once per closed hour by one background task. Raw rows are read
only for per-entity drill-downs (a packet, a trace, a route), observer comparison and
sub-hour observer activity.

## Rollup contract

- **Hours.** `hour` is the UTC `date_trunc('hour', heard_at)` and leads every rollup table's
  primary key. Rollup tables have no foreign keys: history outlives deleted observers, nodes
  and IATAs.
- **Eligibility.** Hour `[H, H+1h)` can be rolled once `now() >= H + 95 min`. Ingest clamps
  `heard_at` to ±30 minutes of server time, so nothing can still land in it.
- **Registration.** Every eligible hour since the earliest retained observation gets a row
  in `analytics_rollup_hours`, including hours with no traffic. Status is `missing`,
  `complete` or `partial`.
- **Rolling.** One process holds the advisory lock (`analytics_rollup` task, every 5
  minutes, plus a catch-up at boot). Each hour is rolled in one REPEATABLE READ transaction:
  copy its observations (joined to packet dimensions) into a temp table, delete and reinsert
  every family, hash the result, mark the hour complete. A serialization failure skips the
  hour until the next pass.
- **Revision.** `analytics_state.revision` increases when an hour's content hash changes or
  an hour turns partial; `analytics_raw_state.coverage` increases when retention deletes
  rollup hours. Cache keys for rollup-backed endpoints include their sum (each counter only
  grows), read through a 10 s memo.
- **Raw deletion.** Cleanup deletes packets older than `packets.retention`, but holds the
  cutoff 35 minutes behind the oldest `missing` hour, for at most 24 hours (it logs a
  warning past 6). Each batch advances `analytics_raw_state.raw_deleted_before` to the
  newest deleted `last_heard_at`. Cleanup and the rollup never write the same row.
- **Partial hours.** A not-yet-complete hour starting at or before `raw_deleted_before + 30 min`
  lost raw rows before it was rolled. It is marked `partial` and never rolled; readers
  report it without values.
- **Re-rolls.** Hours are not re-rolled periodically. A message stored by channel backfill (at
  startup, and when keys are imported at runtime) queues every hour its packet was heard in
  `analytics_dirty_hours`, in the same statement as the message insert. Each pass re-rolls a
  few complete ones while their raw rows remain. A queued hour still waiting for its first roll
  keeps its entry unless that roll's snapshot already included it.
- **Retention.** Rollups are kept for `analytics.rollup_retention` (default 90 days), which
  is also the longest window `/stats/series` accepts.

## Combining hours

| Metric | Across hours and IATAs |
| --- | --- |
| Observations, receptions, payload counts | Sum |
| Unique packets, adverts, messages, scoped packets | Sum of hourly IATA-set counts: each is counted once per hour it was heard, so one heard across an hour boundary counts twice (measured +1.1%) |
| Active observers, IATAs, scopes | Distinct across the window |
| Active scope observers and nodes (per scope, per hour) | Distinct within the hour, across the requested IATAs |
| Max path entries | Max |
| SNR / RSSI averages | Σ sum / Σ samples, never an average of averages |

Windows snap to UTC hours on the server, and responses report the effective window.
`/stats/overview` spans the 24 newest eligible hours; its `windowHours` counts the complete
ones its totals cover.

## IATA sets

A packet heard in YVR and YYZ appears in both IATAs' per-IATA rows, so summing per-IATA
distinct counts overcounts (2.26× on production data). The `*_sets` tables instead group
each distinct packet, advert or message under the exact sorted set of IATAs that heard it
that hour (`iatas char(3)[]`, kept canonical by a CHECK). For any requested set S, the rows
where `iatas && S` count every item heard in S exactly once. This is exact for one IATA, a
region or all IATAs.

## Raw or current, by design

- Packet, trace and route detail, the packet list, observer comparison, and observer
  activity below one hour. Hourly observer activity reads raw rows for the hours after the
  newest complete one (at most the last 3 hours), so the unrolled tail isn't shown as zero.
- Current memberships: scope observer and node counts, node types, clock drift, radio presets.
- `/traces` reads `trace_tags`, a per-tag summary kept at ingest, not a rollup. A new packet
  adds its count, type and payload in the statement that stores it; hearings set the times.

## Adding a metric

1. Add a family table to `db/migrations/001_baseline.sql` (on 2.x, a new migration): `hour`
   first in the primary key, no foreign keys, and a set table with
   `CHECK (iata_set_is_canonical(iatas))` plus a GIN index if it needs distinct counts.
2. Add a delete in `DeleteRollupHour`, a `Roll*` insert over `rollup_obs` in
   `db/queries/rollup.sql`, a line in `RollupContentHash` and `DeleteOldRollups`, and the
   call in `rollHour` (`db/rollup.go`).
3. Read it with an hour-bounded query, using the UNION-ALL IATA guard (see `series.sql`) for
   per-IATA tables or `iatas && @iatas` for set tables.
4. Cache it with `cachedRollup` so its key carries the revision.
5. Add a raw-SQL oracle case to `db/rollup_integration_test.go`.
6. Run `sqlc generate`, regenerate mocks and swagger.