beacon-server/docs/historical-stats.md

Historical stats

The rule

Historical stats read hourly rollup tables filled once per closed hour by one background task. Raw rows are read for per-entity drill-downs (a packet, a trace, a route), observer comparison, sub-hour observer activity, and bounded observer-directory slices that cannot use a completed whole-hour rollup.

Rollup contract

• Hours. hour is the UTC date_trunc('hour', heard_at) and leads every rollup table's primary key. Rollup tables have no foreign keys: history outlives deleted observers, nodes and IATAs.
• Eligibility. Hour [H, H+1h) can be rolled once now() >= H + 95 min. Ingest clamps heard_at to ±30 minutes of server time, so nothing can still land in it.
• Registration. Every eligible hour since the earliest retained observation gets a row in analytics_rollup_hours, including hours with no traffic. Status is missing, complete or partial.
• Rolling. One process holds the advisory lock (analytics_rollup task, every 5 minutes, plus a catch-up at boot). Each hour is rolled in one REPEATABLE READ transaction: copy its observations (joined to packet dimensions) into a temp table, delete and reinsert every family, hash the result, mark the hour complete. A serialization failure skips the hour until the next pass.
• Revision. analytics_state.revision increases when an hour's content hash changes or an hour turns partial; analytics_raw_state.coverage increases when retention deletes rollup hours. Cache keys for rollup-backed endpoints include their sum (each counter only grows), read through a 10 s memo.
• Raw deletion. Cleanup deletes packets older than packets.retention, but holds the cutoff 35 minutes behind the oldest missing hour, for at most 24 hours (it logs a warning past 6). Each batch advances analytics_raw_state.raw_deleted_before to the newest deleted last_heard_at. Cleanup and the rollup never write the same row.
• Partial hours. A not-yet-complete hour starting at or before raw_deleted_before + 30 min lost raw rows before it was rolled. It is marked partial and never rolled; readers report it without values, except the observer directory, which counts any retained raw observations in the requested slice.
• Re-rolls. Hours are not re-rolled periodically. A message stored by channel backfill (at startup, and when keys are imported at runtime) queues every hour its packet was heard in analytics_dirty_hours, in the same statement as the message insert. Each pass re-rolls a few complete ones while their raw rows remain. A queued hour still waiting for its first roll keeps its entry unless that roll's snapshot already included it.
• Retention. Rollups are kept for analytics.rollup_retention (default 90 days), which is also the longest window /stats/series accepts.

Combining hours

Metric	Across hours and IATAs
Observations, receptions, payload counts	Sum
Unique packets, adverts, messages, scoped packets	Sum of hourly IATA-set counts: each is counted once per hour it was heard, so one heard across an hour boundary counts twice (measured +1.1%)
Active observers, IATAs, scopes	Distinct across the window
Active scope observers and nodes (per scope, per hour)	Distinct within the hour, across the requested IATAs
Max path entries	Max
SNR / RSSI averages	Σ sum / Σ samples, never an average of averages

Historical stats windows snap to UTC hours on the server, and responses report the effective window. The observer directory preserves exact millisecond bounds instead. /stats/overview spans the 24 newest eligible hours; its windowHours counts the complete ones its totals cover.

Observer directory

GET /api/v1/observers/directory defaults to [request time - 7 days, request time). Explicit since and until are epoch milliseconds and remain exact; windows must be positive and at most 31 days. If only until is supplied, since defaults to seven days before it. If only since is supplied, until defaults to request time.

Counts combine complete rollups for whole UTC hours fully inside the window with retained raw observations for every other hour and the exact boundary slices. These sources never overlap. Completed whole-hour history survives raw retention; a sliced boundary whose raw rows have expired cannot be reconstructed from its whole-hour total.

Every observer receives a numeric observationCount, including zero. maxObservationCount is the maximum across all matching observers, independent of pagination, and is zero when none match. Incomplete history never suppresses these counts or changes effectiveSort: traffic sorting uses available counts, while explicit name sorting remains supported. Both sorts break name ties by observer ID.

coverage is conservative and informational. expectedHours counts intersecting UTC hours, including boundary slices. completeHours counts the whole hours read from complete rollups. Other hours are partialHours when there is a partial/boundary rollup or matching retained raw traffic; otherwise they are missingHours. Raw-only traffic does not establish complete history. Status is complete only when all intersecting hours are complete, partial when any complete or partial hour exists, and unavailable otherwise. Zero counts with incomplete coverage mean no available observations, not proof that no traffic occurred.

The response uses Cache-Control: no-store. For continuation requests, send nextCursor, the same filters and sort, and the exact returned windowStart/windowEnd as since/until. These bounds fix the time window, not a database snapshot; late arrivals and metadata changes can still move results. Refresh without bounds to obtain a new rolling window and include new traffic.

IATA sets

A packet heard in YVR and YYZ appears in both IATAs' per-IATA rows, so summing per-IATA distinct counts overcounts (2.26× on production data). The *_sets tables instead group each distinct packet, advert or message under the exact sorted set of IATAs that heard it that hour (iatas char(3)[], kept canonical by a CHECK). For any requested set S, the rows where iatas && S count every item heard in S exactly once. This is exact for one IATA, a region or all IATAs.

Raw or current, by design

• Packet, trace and route detail, the packet list, observer comparison, and observer activity below one hour. Hourly observer activity reads raw rows for the hours after the newest complete one (at most the last 24 hours, the raw holdback), so the unrolled tail isn't shown as zero. rolledUntil and rawFrom in the response mark any hours still uncovered.
• Current memberships: scope observer and node counts, node types, clock drift, radio presets.
• /traces reads trace_tags, a per-tag summary kept at ingest, not a rollup. A new packet adds its count, type and payload in the statement that stores it; hearings set the times.

Adding a metric

1. Add a family table in a new numbered migration (001_baseline.sql is frozen from 2.0.0): hour first in the primary key, no foreign keys, and a set table with CHECK (iata_set_is_canonical(iatas)) plus a GIN index if it needs distinct counts.
2. Add a delete in DeleteRollupHour, a Roll* insert over rollup_obs in db/queries/rollup.sql, a line in RollupContentHash and DeleteOldRollups, and the call in rollHour (db/rollup.go).
3. Read it with an hour-bounded query, using the UNION-ALL IATA guard (see series.sql) for per-IATA tables or iatas && @iatas for set tables.
4. Cache it with cachedRollup so its key carries the revision.
5. Add a raw-SQL oracle case to db/rollup_integration_test.go.
6. Run sqlc generate, regenerate mocks and swagger.