mirror of
https://github.com/agessaman/meshcore-bot.git
synced 2026-10-22 14:01:14 +00:00
feat(web-viewer): rebuild dashboard on a background snapshot
The landing page re-ran ~50 aggregate queries five times per load, then repeated the whole sequence every 30 seconds forever — including in backgrounded tabs. Against the live 1.44 GB database that was roughly 20 seconds of SQLite work per page load. Move the work off the request path. A refresher thread in the viewer process (which already runs migrations, so it works for a split-DB install) writes two tables: daily_rollup, one row per local date, and dashboard_snapshot, a single JSON row. A page load now reads one row. Measured on the live database: first paint 6 requests -> 2, /api/dashboard/summary p50 1.1 ms (304 in 0.8 ms), /api/stats 130 ms, and a 0.32 s refresh once a minute in the background. Make the numbers mean what they say: - Window selectors are built from each source's retention. The page offered "30d" and "All" against tables pruned at 7 days, so three of four choices returned the same figure under a label that denied it. - The incoming-packet chart reports its measured window instead of claiming 7 days for a table pruned at 3 — it sat beside a genuine 7-day contacts chart inviting an invalid comparison. - Days with no source data store NULL and render as gaps. Writing 0 would put a fake cliff at every retention boundary. - Signal metrics are stored as sums and counts, never means, so any window re-aggregates correctly. - Delta chips compare the last two complete calendar days and say so; the headline above them is a rolling 24 hours. - Unmapped role ordinals (type0..type15) bucket into "Unknown". - SNR comes from message_stats, where it is populated on every row, not from complete_contact_tracking, where it is populated on 7%. Kill the json_extract scans: packet_stream gains denormalized route_type_name, payload_type_name, path_len and bytes_per_hop, written at capture time. Aggregating those from JSON cost 3-6 s per query. Existing rows convert a bounded batch per tick rather than in one migration that would rewrite ~180 MB into the WAL and stall bot startup. A partial index serves as the backfill worklist — without it the "any rows left?" probe is a full scan costing 4.6 s per tick, and it costs that after the backfill finishes, because finding nothing still means reading everything. Also: replace the per-contact hop-prefix scan with the existing bucketed matcher and memoize the 7-day chunk set (264 ms -> 35 ms on a synthetic 100k-row database, regression-locked by a test); move the dashboard's JS and CSS to static files, which removes the CSP nonce requirement for the bulk of the page; and give cleanup_old_stats a future-timestamp guard, without which rows dated 2103 are never older than the cutoff and so live forever. Deletes the orphaned /stats page, unreachable from the nav and rendering stub charts that never populated. /api/stats stays as a shim with every key name intact plus Deprecation and Sunset headers. All schema changes are additive, so a downgraded codebase can read the data; it would however need the new schema_version rows removed, since MigrationRunner rejects versions it does not know.
This commit is contained in:
agessaman committed
1 parent
9007080c48
commit
7e3ab04b53
23 files changed
+4722
-1952
No files matched your search
@@ -4,6 +4,73 @@ All notable changes to this project are documented here. The format loosely foll
|
||||
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project adheres to
|
||||
semantic versioning.
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
|
||||
- Rebuilt web-viewer dashboard, served from a background snapshot instead of
|
||||
recomputing statistics on every request. A refresher thread in the viewer
|
||||
process writes `daily_rollup` (one row per local date) and
|
||||
`dashboard_snapshot` (a single JSON row), so a page load reads one row rather
|
||||
than running ~50 aggregate queries five times over.
|
||||
- Daily trends that outlive retention: `message_stats` and friends are pruned at
|
||||
7 days and `packet_stream` at 3, so a 30-day chart had nothing to draw from
|
||||
until now. Signal metrics are stored as sums and counts, never means, so any
|
||||
window re-aggregates correctly.
|
||||
- New `/api/dashboard/{summary,series,top,windows,refresh}` endpoints. `summary`
|
||||
sends a strong `ETag`, so the page's 30-second poll is normally a bodyless
|
||||
`304`, and polling stops entirely while the tab is hidden.
|
||||
- New dashboard tiles and charts: routing mix (flood vs direct), hop-count and
|
||||
path-length histograms, an SNR/RSSI distribution, a 30-day multibyte adoption
|
||||
trend, busiest repeaters, and a role/device mix.
|
||||
- New `[Web_Viewer]` settings: `dashboard_snapshot_enabled`,
|
||||
`dashboard_snapshot_interval_seconds`, `dashboard_snapshot_history_days`, and
|
||||
`dashboard_packet_backfill_rows`.
|
||||
- Partial index `idx_packet_stream_undimensioned` as the backfill worklist.
|
||||
Probing for remaining un-dimensioned rows was otherwise a full table scan,
|
||||
and it cost the same 4.6 seconds *after* the backfill finished as during it,
|
||||
because finding nothing still meant reading everything.
|
||||
|
||||
### Changed
|
||||
|
||||
- Time-window selectors are now built from each source's configured retention.
|
||||
The dashboard previously offered "30d" and "All" against tables pruned at 7
|
||||
days, so three of the four choices returned the same number under a label
|
||||
that claimed otherwise.
|
||||
- The incoming-packet chart no longer claims to show 7 days. `packet_stream` is
|
||||
pruned at 3 days, so the covered window is now measured and labelled from the
|
||||
data — and shown beside a genuinely 7-day contacts chart it can be compared
|
||||
to.
|
||||
- `packet_stream` gained denormalized `route_type_name`, `payload_type_name`,
|
||||
`path_len`, and `bytes_per_hop` columns, written at capture time. Aggregating
|
||||
these via `json_extract` cost 3–6 seconds per query on a large database.
|
||||
Existing rows are converted a bounded batch per refresh rather than in one
|
||||
table-rewriting migration.
|
||||
- Dashboard JavaScript and CSS moved to static files, removing the CSP nonce
|
||||
requirement for the bulk of the page's code.
|
||||
- `cleanup_old_stats` now also deletes rows dated implausibly far in the future.
|
||||
Such rows are never older than the retention cutoff, so they were immortal —
|
||||
observed in the wild dated 2103.
|
||||
|
||||
### Removed
|
||||
|
||||
- The orphaned `/stats` page, which was unreachable from the navigation and
|
||||
rendered stub charts that never populated.
|
||||
|
||||
### Deprecated
|
||||
|
||||
- `GET /api/stats`. Every key name is preserved and the response now carries
|
||||
`Deprecation` and `Sunset` headers; use `/api/dashboard/*` instead. It will be
|
||||
removed at the next major version.
|
||||
|
||||
### Notes for downgrades
|
||||
|
||||
All schema changes are additive — two new tables, four new nullable columns, and
|
||||
new indexes — so the data itself is safe to read with older code. However,
|
||||
`MigrationRunner` fails startup on encountering an applied migration version it
|
||||
does not know about, so downgrading below this release requires deleting the
|
||||
corresponding `schema_version` rows.
|
||||
|
||||
## [1.0.0] — 2026-07-28
|
||||
|
||||
v1.0.0 marks the first stable release. It adds transport recovery, location and rain
|
||||
|
||||
@@ -104,6 +104,8 @@ by running `python scripts/update_todos.py` (see [Auto-Update](#auto-update)).
|
||||
- [x] (2026-03-15) **Configuration tab** — `/config` page; SMTP + nightly email toggle; log rotation; DB backup; stored in `bot_metadata`
|
||||
- [x] (2026-03-15) **Real-time log viewer** — `/logs` page; SocketIO `subscribe_logs`/`log_line`; level-based coloring; pause/clear/filter; log tail thread; "Logs" nav link
|
||||
- [ ] **Mobile-responsive improvements** — optimize layout for small screens
|
||||
- [x] (2026-07-29) **Dashboard modernization** — landing page rebuilt on a background snapshot instead of live aggregation. Migrations 0018 (`daily_rollup`, `dashboard_snapshot`) and 0019 (`packet_stream` denormalized `route_type_name`/`payload_type_name`/`path_len`/`bytes_per_hop`, backfilled a bounded batch per tick rather than in one table-rewriting UPDATE); new `modules/web_viewer/dashboard_stats.py`; refresher thread in the viewer process with a `bot_metadata` lease; new `/api/dashboard/{summary,series,top,windows,refresh}`; JS/CSS extracted to `static/` (no CSP nonce needed); `/stats` page and `stats.html` deleted; `/api/stats` kept as a deprecated shim. Measured on the live 1.44 GB DB: first paint 6 requests → 2, `/api/dashboard/summary` p50 **1.1 ms** (304s in 0.8 ms), `/api/stats` 130 ms, and a 0.32 s refresh once a minute off the request path. Window labels and the packet chart's window are now derived from retention instead of asserted. 37 tests added
|
||||
- [ ] **Dashboard follow-ups** — expand-chart interaction wired to `/api/dashboard/series` (endpoint exists, no UI yet); consider the deferred `observed_paths(last_seen, path_length, bytes_per_hop, public_key)` covering index if refresher timing shows the path-length histogram dominating
|
||||
- [x] (2026-07-11) **Radio page: Node Settings card** — path hash size (response path hash bytes, mode 0–2), name/location/advert-policy, multi-ACKs/telemetry permissions, RX-delay/airtime tuning, zero-hop/flood advert buttons; extended `/api/radio/params`, new `/api/radio/advert`, scheduler write ops; config-managed settings are not writable from the panel (new-contact mode owned by `auto_manage_contacts`; name locked when `bot_name` + `auto_update_device_name`); loop.detect dropped from firmware endpoints (repeater CLI config, not a companion setting); 23 tests added
|
||||
- [x] (TASK-01 2026-03-15) **Remove firmware config + reboot UI** — radio.html: Firmware Configuration card and Reboot Radio button removed; JS handlers removed; 4 tests added
|
||||
- [x] (TASK-02 2026-03-15) **Fix realtime stream blank on load** — added 50-row history replay to `subscribe_commands`; fixed `last_timestamp = 0` → `time.time() - 300` in polling thread; 5 tests added (BUG-023 fixed)
|
||||
|
||||
@@ -1605,6 +1605,31 @@ auto_start = false
|
||||
# true: expose page/nav/API endpoint
|
||||
multibyte_monitor_enabled = false
|
||||
|
||||
# Dashboard snapshot refresher
|
||||
# A background thread in the viewer process recomputes the landing page's
|
||||
# statistics on an interval and stores them in two tables (daily_rollup and
|
||||
# dashboard_snapshot), so page loads read one row instead of running ~50
|
||||
# aggregate queries. It also accumulates daily trends that outlive the raw
|
||||
# tables' retention — message/command/path stats are pruned at 7 days and
|
||||
# packet_stream at 3, so without this there is no history to chart.
|
||||
# Disabling it leaves /api/dashboard/summary returning 503 and the dashboard
|
||||
# unable to render.
|
||||
# dashboard_snapshot_enabled = true
|
||||
#
|
||||
# How often to recompute. Each refresh recomputes today plus a 3-day trailing
|
||||
# window; older days are frozen.
|
||||
# dashboard_snapshot_interval_seconds = 60
|
||||
#
|
||||
# How many days of daily_rollup history to retain (~1 row/day, so 400 days is
|
||||
# well under a megabyte).
|
||||
# dashboard_snapshot_history_days = 400
|
||||
#
|
||||
# packet_stream rows written before this feature existed keep their route type
|
||||
# and hop encoding only inside a JSON blob. Each refresh converts this many of
|
||||
# them to real columns rather than rewriting the whole table at once, which on
|
||||
# an SD-card Pi would stall the bot for minutes. Set to 0 to disable.
|
||||
# dashboard_packet_backfill_rows = 2000
|
||||
|
||||
# Optional timeout overrides for web viewer integration (seconds)
|
||||
# edge_post_timeout_sec = 1.0
|
||||
# node_post_timeout_sec = 0.5
|
||||
|
||||
@@ -28,8 +28,22 @@ Stats tables (message_stats, command_stats, path_stats) use **`[Stats_Command]`*
|
||||
| **purging_log** | Audit trail for repeater purges | 90 days |
|
||||
| **mesh_connections** | Path graph edges (in-memory + DB); should be ≥ Path_Command `graph_edge_expiration_days` | 7 days |
|
||||
| **message_stats, command_stats, path_stats** | Stats command data | 7 days (`[Stats_Command]` `data_retention_days`) |
|
||||
| **daily_rollup** | Per-day dashboard rollups, so trends outlive the tables above | 400 days (`[Web_Viewer]` `dashboard_snapshot_history_days`) |
|
||||
| **dashboard_snapshot** | Single-row current-state payload for the dashboard | Overwritten in place |
|
||||
| **geocoding_cache, generic_cache** | Expired entries removed by scheduler | By expiry time |
|
||||
|
||||
`daily_rollup` exists precisely because the retention above is short: the
|
||||
dashboard charts 30 days of message and packet activity from tables pruned at 7
|
||||
and 3 days respectively. One row per day is roughly 200 bytes, so a year of
|
||||
history costs well under a megabyte. Pruning it is handled by the web viewer's
|
||||
snapshot refresher rather than the bot scheduler, because the viewer may point
|
||||
at a different database file.
|
||||
|
||||
Note that cleanup also removes rows dated implausibly far in the **future**.
|
||||
A node with a bad clock can write a timestamp years ahead; because such a row is
|
||||
never older than the cutoff, a lower-bound-only delete would keep it forever and
|
||||
stretch every chart axis to match.
|
||||
|
||||
Shorter retention (e.g. 2–3 days for `packet_stream`) is enough for the web viewer and transmission_tracker; longer retention is only needed if you want more history.
|
||||
|
||||
## How cleanup runs
|
||||
|
||||
+58
-5
@@ -137,9 +137,45 @@ proxy_set_header X-Forwarded-Proto $scheme;
|
||||
## Pages Overview
|
||||
|
||||
### Dashboard
|
||||
- Database status and statistics
|
||||
- Contact counts and cache information
|
||||
- Quick navigation to other sections
|
||||
- Health strip: bot status, database size, uptime, connected clients, radio state,
|
||||
and snapshot age with a manual refresh control
|
||||
- **Mesh**: nodes heard, adverts, new nodes, nodes gone quiet, tracked/known
|
||||
contacts, and geographic coverage — each with a 30-day sparkline and a
|
||||
change chip
|
||||
- Routing mix (flood vs direct), hop-count and path-length histograms, and an
|
||||
SNR/RSSI distribution
|
||||
- Path encoding: multibyte share among contacts and among incoming packets,
|
||||
plus a 30-day multibyte adoption trend
|
||||
- Busiest repeaters and the role/device mix
|
||||
- **Bot**: messages, commands, reply rate, and unique users, plus the top
|
||||
commands/users/channels and longest paths
|
||||
- Live activity feed
|
||||
|
||||
The dashboard is served from a **snapshot**, not from live aggregation. A
|
||||
background thread in the viewer process recomputes it every
|
||||
`dashboard_snapshot_interval_seconds` (default 60) and writes two tables:
|
||||
|
||||
| Table | Contents |
|
||||
|-------|----------|
|
||||
| `daily_rollup` | One row per local date. Retains counts whose raw sources are pruned long before the dashboard's 30-day window. Signal metrics are stored as sums and counts, never means, so any window re-aggregates correctly. |
|
||||
| `dashboard_snapshot` | A single JSON row describing current state. |
|
||||
|
||||
Two consequences worth knowing:
|
||||
|
||||
- **Gaps are not zeros.** A day with no source data stores NULL and renders as a
|
||||
break in the line. Days seeded when the feature was first enabled have real
|
||||
advert counts but no message, command, path, or packet figures — those raw
|
||||
rows were already pruned and cannot be recovered.
|
||||
- **Window labels come from retention.** The time-window selectors are built
|
||||
from each source's configured retention, so the list cannot offer "30 days"
|
||||
against a table pruned at 7.
|
||||
|
||||
The **multibyte share** trend is accumulated forward, not derived. `observed_paths`
|
||||
is deduplicated with a lifetime `observation_count` and a `last_seen` that is
|
||||
bumped on every re-observation, so historical per-day shares cannot be
|
||||
reconstructed from it — nearly half the observation volume would be attributed
|
||||
to the wrong day. Each refresh recomputes today plus a three-day trailing
|
||||
window; older days stay frozen at the value recorded then.
|
||||
|
||||
### Repeater Contacts
|
||||
- Active repeater contacts
|
||||
@@ -192,7 +228,24 @@ proxy_set_header X-Forwarded-Proto $scheme;
|
||||
|
||||
The viewer also provides JSON API endpoints:
|
||||
|
||||
- `GET /api/stats` - Database statistics
|
||||
- `GET /api/dashboard/summary` - Snapshot-backed dashboard payload, including
|
||||
30-day sparkline series and change figures. Sends a strong `ETag`; poll with
|
||||
`If-None-Match` to get a bodyless `304` while the snapshot is unchanged.
|
||||
- `GET /api/dashboard/series?metric=<m>&days=<n>` - Full-history points for one
|
||||
metric. `metric` is one of `messages`, `commands`, `adverts`, `nodes`,
|
||||
`new_nodes`, `packets`, `multibyte_share`.
|
||||
- `GET /api/dashboard/top?kind=<k>&window=<w>&limit=<n>` - One leaderboard.
|
||||
`kind` is one of `users`, `commands`, `channels`, `paths`, `repeaters`. The
|
||||
response carries `window_label`, `retention_days`, and
|
||||
`truncated_by_retention`.
|
||||
- `GET /api/dashboard/windows` - Selector options derived from each source's
|
||||
retention.
|
||||
- `POST /api/dashboard/refresh` - Force a snapshot recomputation.
|
||||
- `GET /api/stats` - **Deprecated.** The whole-database statistics payload the
|
||||
dashboard used to call five times per page load. Every key name is preserved
|
||||
for external consumers, and the response carries `Deprecation` and `Sunset`
|
||||
headers. Use the `/api/dashboard/*` endpoints instead; this one is removed at
|
||||
the next major version.
|
||||
- `GET /api/contacts` - Repeater contacts data. The contacts page uses optional
|
||||
`page`, `page_size` (maximum 200), `search`, `sort`, and `direction` parameters;
|
||||
callers that omit pagination retain the legacy full-list response.
|
||||
@@ -200,7 +253,7 @@ The viewer also provides JSON API endpoints:
|
||||
|
||||
Example usage:
|
||||
```bash
|
||||
curl http://localhost:5000/api/stats
|
||||
curl http://localhost:5000/api/dashboard/summary
|
||||
```
|
||||
|
||||
## Database Requirements
|
||||
|
||||
@@ -789,35 +789,47 @@ class StatsCommand(BaseCommand):
|
||||
self.logger.error(f"Error getting adverts leaderboard: {e}")
|
||||
return self.translate('commands.stats.error_adverts', error=str(e))
|
||||
|
||||
# Rows timestamped beyond this margin ahead of now came from a bad clock on
|
||||
# the sending node, not from the future. Without an upper bound they are
|
||||
# never older than the cutoff, so retention never removes them: observed in
|
||||
# the wild dated 2103-08-15, which also stretches any chart's axis to match.
|
||||
FUTURE_TIMESTAMP_GRACE_SECONDS = 24 * 60 * 60
|
||||
|
||||
def cleanup_old_stats(self, days_to_keep: int = 7) -> None:
|
||||
"""Clean up old stats data to prevent database bloat.
|
||||
|
||||
Deletes rows older than the retention cutoff *and* rows dated
|
||||
implausibly far in the future.
|
||||
|
||||
Args:
|
||||
days_to_keep: Number of days of data to retain.
|
||||
"""
|
||||
try:
|
||||
cutoff_time = int(time.time()) - (days_to_keep * 24 * 60 * 60)
|
||||
now = int(time.time())
|
||||
cutoff_time = now - (days_to_keep * 24 * 60 * 60)
|
||||
future_cutoff = now + self.FUTURE_TIMESTAMP_GRACE_SECONDS
|
||||
|
||||
with self.bot.db_manager.connection() as conn:
|
||||
cursor = conn.cursor()
|
||||
|
||||
# Clean up old message stats
|
||||
cursor.execute('DELETE FROM message_stats WHERE timestamp < ?', (cutoff_time,))
|
||||
messages_deleted = cursor.rowcount
|
||||
|
||||
# Clean up old command stats
|
||||
cursor.execute('DELETE FROM command_stats WHERE timestamp < ?', (cutoff_time,))
|
||||
commands_deleted = cursor.rowcount
|
||||
|
||||
# Clean up old path stats
|
||||
cursor.execute('DELETE FROM path_stats WHERE timestamp < ?', (cutoff_time,))
|
||||
paths_deleted = cursor.rowcount
|
||||
deleted = {}
|
||||
for table in ('message_stats', 'command_stats', 'path_stats'):
|
||||
cursor.execute(
|
||||
f'DELETE FROM {table} WHERE timestamp < ? OR timestamp > ?', # noqa: S608 - fixed table names
|
||||
(cutoff_time, future_cutoff),
|
||||
)
|
||||
deleted[table] = cursor.rowcount
|
||||
|
||||
conn.commit()
|
||||
|
||||
total_deleted = messages_deleted + commands_deleted + paths_deleted
|
||||
total_deleted = sum(deleted.values())
|
||||
if total_deleted > 0:
|
||||
self.logger.info(f"Cleaned up {total_deleted} old stats entries ({messages_deleted} messages, {commands_deleted} commands, {paths_deleted} paths)")
|
||||
self.logger.info(
|
||||
f"Cleaned up {total_deleted} old stats entries "
|
||||
f"({deleted['message_stats']} messages, "
|
||||
f"{deleted['command_stats']} commands, "
|
||||
f"{deleted['path_stats']} paths)"
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
self.logger.error(f"Error cleaning up old stats: {e}")
|
||||
|
||||
@@ -201,6 +201,13 @@ SECTIONS: dict[str, SectionMeta] = {
|
||||
"sqlite_busy_timeout_ms": KeyMeta(type="int", default="60000"),
|
||||
"sqlite_foreign_keys": KeyMeta(type="bool", default="true"),
|
||||
"sqlite_journal_mode": KeyMeta(default="WAL"),
|
||||
# Dashboard snapshot refresher — moves the landing page's aggregate
|
||||
# queries off the request path and accumulates trends that outlive the
|
||||
# raw tables' retention.
|
||||
"dashboard_snapshot_enabled": KeyMeta(type="bool", default="true"),
|
||||
"dashboard_snapshot_interval_seconds": KeyMeta(type="int", default="60"),
|
||||
"dashboard_snapshot_history_days": KeyMeta(type="int", default="400"),
|
||||
"dashboard_packet_backfill_rows": KeyMeta(type="int", default="2000"),
|
||||
}),
|
||||
"Localization": SectionMeta(keys={
|
||||
"language": KeyMeta(default="en"),
|
||||
|
||||
@@ -586,6 +586,98 @@ def _m0017_feed_queue_item_uniqueness(cursor: sqlite3.Cursor) -> None:
|
||||
)
|
||||
|
||||
|
||||
def _m0018_dashboard_rollup_tables(cursor: sqlite3.Cursor) -> None:
|
||||
"""Durable dashboard storage: per-day rollups plus a current-state snapshot.
|
||||
|
||||
``daily_rollup`` holds one row per local date for the metrics whose raw
|
||||
sources are pruned long before the dashboard's 30-day window (message_stats
|
||||
and friends at 7 days, packet_stream at 3). Signal metrics are stored as
|
||||
sums and counts rather than means so an arbitrary window re-aggregates
|
||||
correctly — averaging daily averages weights a quiet day the same as a busy
|
||||
one. Every numeric column is nullable on purpose: NULL means "no source
|
||||
data for this day", which the UI must render as a gap, while 0 means "the
|
||||
source was present and the count really was zero".
|
||||
|
||||
``dashboard_snapshot`` is a single row of JSON. The current-state half of
|
||||
the dashboard is ~40 heterogeneous scalars and small arrays that are read
|
||||
together and never queried by field, so columnizing it would cost a
|
||||
migration per new tile for no query benefit.
|
||||
"""
|
||||
cursor.executescript(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS daily_rollup (
|
||||
date TEXT PRIMARY KEY,
|
||||
messages_total INTEGER, messages_dm INTEGER, messages_channel INTEGER,
|
||||
unique_senders INTEGER, unique_channels INTEGER,
|
||||
commands_total INTEGER, commands_replied INTEGER, unique_command_users INTEGER,
|
||||
path_obs_total INTEGER, path_len_max INTEGER, path_len_sum INTEGER,
|
||||
snr_sum REAL, snr_count INTEGER, rssi_sum REAL, rssi_count INTEGER,
|
||||
hops_sum INTEGER, hops_count INTEGER,
|
||||
packets_total INTEGER, packets_flood INTEGER, packets_direct INTEGER,
|
||||
packets_multibyte INTEGER,
|
||||
adverts_total INTEGER, nodes_active INTEGER, nodes_new INTEGER,
|
||||
adverts_from_multibyte INTEGER, adverts_from_singlebyte INTEGER,
|
||||
contacts_known INTEGER, contacts_tracked INTEGER,
|
||||
sources_present INTEGER NOT NULL DEFAULT 0,
|
||||
is_backfilled INTEGER NOT NULL DEFAULT 0,
|
||||
is_final INTEGER NOT NULL DEFAULT 0,
|
||||
computed_at TEXT
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_daily_rollup_computed ON daily_rollup(computed_at);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS dashboard_snapshot (
|
||||
id INTEGER PRIMARY KEY CHECK (id = 1),
|
||||
generated_at REAL NOT NULL,
|
||||
duration_ms INTEGER,
|
||||
schema_rev INTEGER NOT NULL DEFAULT 1,
|
||||
payload TEXT NOT NULL
|
||||
);
|
||||
"""
|
||||
)
|
||||
|
||||
|
||||
def _m0019_packet_stream_denorm_dims(cursor: sqlite3.Cursor) -> None:
|
||||
"""Denormalize the four packet dimensions the dashboard aggregates on.
|
||||
|
||||
Counting packets by bytes_per_hop or route type via
|
||||
``json_extract(data, '$.…')`` is a full scan that parses every row's JSON:
|
||||
~3s for bytes_per_hop and ~6s for route_type_name on a 178k-row table.
|
||||
The writer already holds these values as parsed fields, so storing them as
|
||||
columns removes the parse entirely.
|
||||
|
||||
Deliberately no bulk UPDATE here. packet_stream rows average ~1KB, so
|
||||
widening 178k of them rewrites ~180MB into the WAL inside MigrationRunner's
|
||||
transaction — a multi-minute stall blocking bot startup on an SD-card Pi.
|
||||
The dashboard refresher backfills a bounded batch per tick instead, and the
|
||||
table's 3-day retention makes the gap self-healing regardless.
|
||||
"""
|
||||
if not _table_exists(cursor, "packet_stream"):
|
||||
return
|
||||
_add_column(cursor, "packet_stream", "route_type_name", "TEXT")
|
||||
_add_column(cursor, "packet_stream", "payload_type_name", "TEXT")
|
||||
_add_column(cursor, "packet_stream", "path_len", "INTEGER")
|
||||
_add_column(cursor, "packet_stream", "bytes_per_hop", "INTEGER")
|
||||
cursor.executescript(
|
||||
"""
|
||||
CREATE INDEX IF NOT EXISTS idx_packet_stream_dims
|
||||
ON packet_stream(timestamp, bytes_per_hop, route_type_name, payload_type_name)
|
||||
WHERE type = 'packet';
|
||||
|
||||
-- Worklist for the refresher's incremental backfill. Without it, the
|
||||
-- "any rows left to dimension?" probe is a full table scan that costs
|
||||
-- ~4.6s on a 180MB packet_stream — and it costs that on every tick
|
||||
-- *after* the backlog is empty, because finding nothing still means
|
||||
-- reading everything. The index shrinks toward empty as rows are
|
||||
-- dimensioned, so the probe converges to a no-op. It must carry
|
||||
-- ``type`` as well: non-packet rows keep a NULL route type forever and
|
||||
-- would otherwise leave the index permanently non-empty.
|
||||
CREATE INDEX IF NOT EXISTS idx_packet_stream_undimensioned
|
||||
ON packet_stream(type, id)
|
||||
WHERE route_type_name IS NULL;
|
||||
"""
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Migration registry — append new entries here, never remove or reorder.
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -610,6 +702,8 @@ MIGRATIONS: list[MigrationEntry] = [
|
||||
(15, "channel_operations: claimed_at", _m0015_channel_operations_claimed_at),
|
||||
(16, "channel_operations: claim owner identity", _m0016_channel_operations_claim_owner),
|
||||
(17, "feed_message_queue: unique identifiable items", _m0017_feed_queue_item_uniqueness),
|
||||
(18, "dashboard rollup and snapshot tables", _m0018_dashboard_rollup_tables),
|
||||
(19, "packet_stream: denormalized packet dimensions", _m0019_packet_stream_denorm_dims),
|
||||
]
|
||||
|
||||
|
||||
|
||||
+365
-80
@@ -65,6 +65,15 @@ from modules.settings_schema import (
|
||||
)
|
||||
from modules.settings_store import get_settings_store
|
||||
from modules.version_info import resolve_runtime_version
|
||||
from modules.web_viewer.dashboard_stats import (
|
||||
SERIES_METRICS,
|
||||
TOP_KINDS,
|
||||
DashboardStatsService,
|
||||
humanize_span,
|
||||
)
|
||||
|
||||
# RFC 8594 Sunset date advertised on the deprecated /api/stats endpoint.
|
||||
STATS_ENDPOINT_SUNSET = "Fri, 01 Jan 2027 00:00:00 GMT"
|
||||
|
||||
|
||||
def _validate_dynamic_key(key: str) -> "str | None":
|
||||
@@ -213,6 +222,8 @@ class BotDataViewer:
|
||||
'path_stats',
|
||||
'schema_version',
|
||||
'greeter_rollout',
|
||||
'daily_rollup',
|
||||
'dashboard_snapshot',
|
||||
}
|
||||
|
||||
def __init__(self, db_path="meshcore_bot.db", repeater_db_path=None, config_path="config.ini"):
|
||||
@@ -287,9 +298,10 @@ class BotDataViewer:
|
||||
# The contacts list needs all-time multibyte hop-prefix evidence for its
|
||||
# capability badge. Cache that derived set between requests and invalidate
|
||||
# it when the underlying multibyte-path population changes.
|
||||
# Keyed by the recent_days window (None = all time) so the contacts list
|
||||
# and the dashboard's 7-day view do not evict each other.
|
||||
self._contacts_badge_cache_lock = threading.Lock()
|
||||
self._contacts_badge_cache_signature = None
|
||||
self._contacts_badge_cache_chunks: set[str] = set()
|
||||
self._contacts_badge_cache: dict[int | None, tuple[tuple, set[str]]] = {}
|
||||
|
||||
# Use [Bot] db_path when [Web_Viewer] db_path is unset
|
||||
bot_db = self.config.get('Bot', 'db_path', fallback='meshcore_bot.db')
|
||||
@@ -319,6 +331,8 @@ class BotDataViewer:
|
||||
except (configparser.NoSectionError, configparser.NoOptionError, ValueError, TypeError):
|
||||
self.multibyte_monitor_enabled = False
|
||||
|
||||
self._init_dashboard_service()
|
||||
|
||||
# Configure CORS for SocketIO — default to same-origin (no cross-origin)
|
||||
cors_raw = self.config.get('Web_Viewer', 'cors_allowed_origins', fallback='').strip()
|
||||
if cors_raw:
|
||||
@@ -349,6 +363,10 @@ class BotDataViewer:
|
||||
# Start periodic cleanup
|
||||
self._start_cleanup_scheduler()
|
||||
|
||||
# Start the dashboard snapshot refresher (moves the landing page's
|
||||
# aggregate queries off the request path)
|
||||
self._start_dashboard_refresher()
|
||||
|
||||
self.logger.info("BotDataViewer initialized with Flask-SocketIO 5.x best practices")
|
||||
|
||||
def _setup_logging(self):
|
||||
@@ -792,7 +810,6 @@ class BotDataViewer:
|
||||
'radio',
|
||||
'realtime',
|
||||
'contacts',
|
||||
'stats',
|
||||
'plugins_page',
|
||||
'greeter',
|
||||
'logs',
|
||||
@@ -859,8 +876,23 @@ class BotDataViewer:
|
||||
|
||||
@self.app.route('/')
|
||||
def index():
|
||||
"""Main dashboard"""
|
||||
return render_template('index.html')
|
||||
"""Main dashboard.
|
||||
|
||||
Server-side config reaches the page as a JSON data block rather than
|
||||
an inline script, so the dashboard's JavaScript can live in a static
|
||||
file that ``script-src 'self'`` already covers.
|
||||
"""
|
||||
database_size = None
|
||||
with suppress(OSError):
|
||||
database_size = os.path.getsize(self.db_path)
|
||||
return render_template(
|
||||
'index.html',
|
||||
dashboard_boot={
|
||||
'database_size': database_size,
|
||||
'snapshot_interval_seconds': self.dashboard_snapshot_interval,
|
||||
'snapshot_enabled': self.dashboard_snapshot_enabled,
|
||||
},
|
||||
)
|
||||
|
||||
@self.app.route('/realtime')
|
||||
def realtime():
|
||||
@@ -883,11 +915,6 @@ class BotDataViewer:
|
||||
return redirect('/config#database')
|
||||
|
||||
|
||||
@self.app.route('/stats')
|
||||
def stats():
|
||||
"""Statistics page"""
|
||||
return render_template('stats.html')
|
||||
|
||||
@self.app.route('/multibyte-rollout')
|
||||
def multibyte_rollout():
|
||||
"""Multibyte hash rollout analytics page"""
|
||||
@@ -2094,7 +2121,13 @@ class BotDataViewer:
|
||||
|
||||
@self.app.route('/api/stats')
|
||||
def api_stats():
|
||||
"""Get comprehensive database statistics for dashboard"""
|
||||
"""Deprecated: whole-database statistics in one payload.
|
||||
|
||||
Superseded by /api/dashboard/summary (snapshot-backed) and
|
||||
/api/dashboard/top (one narrow query per leaderboard). Retained
|
||||
with every key name intact for external consumers, and scheduled for
|
||||
removal at the next major version.
|
||||
"""
|
||||
try:
|
||||
# Get optional time window parameters for analytics
|
||||
top_users_window = request.args.get('top_users_window', 'all')
|
||||
@@ -2107,13 +2140,124 @@ class BotDataViewer:
|
||||
top_paths_window=top_paths_window,
|
||||
top_channels_window=top_channels_window
|
||||
)
|
||||
return jsonify(stats)
|
||||
stats['deprecated'] = True
|
||||
response = jsonify(stats)
|
||||
response.headers['Deprecation'] = 'true'
|
||||
response.headers['Sunset'] = STATS_ENDPOINT_SUNSET
|
||||
response.headers['Link'] = '</api/dashboard/summary>; rel="successor-version"'
|
||||
return response
|
||||
except Exception as e:
|
||||
self.logger.error(f"Error getting stats: {e}")
|
||||
return jsonify({'error': str(e)}), 500
|
||||
|
||||
|
||||
|
||||
@self.app.route('/api/dashboard/summary')
|
||||
def api_dashboard_summary():
|
||||
"""Snapshot-backed dashboard payload — one row read, no aggregation.
|
||||
|
||||
Sparkline series are folded in so first paint costs two requests
|
||||
(/api/health plus this) instead of the six the old page made.
|
||||
"""
|
||||
try:
|
||||
with self._with_db_connection() as conn:
|
||||
payload = self.dashboard_stats.read_summary(conn)
|
||||
if payload is None:
|
||||
# No snapshot yet: the refresher runs a couple of seconds
|
||||
# after startup, so tell the client to retry rather than
|
||||
# recomputing everything on the request path.
|
||||
response = jsonify({
|
||||
'error': 'Dashboard snapshot not generated yet',
|
||||
'pending': True,
|
||||
})
|
||||
response.status_code = 503
|
||||
response.headers['Retry-After'] = '5'
|
||||
return response
|
||||
|
||||
# ETags.contains() wants the bare tag; the quotes belong only in
|
||||
# the header itself.
|
||||
tag = str(payload['generated_at'])
|
||||
if request.if_none_match.contains(tag):
|
||||
response = self.app.response_class(status=304)
|
||||
else:
|
||||
response = jsonify(payload)
|
||||
response.headers['ETag'] = f'"{tag}"'
|
||||
response.headers['Cache-Control'] = 'private, max-age=15'
|
||||
return response
|
||||
except Exception as e:
|
||||
self.logger.error(f"Error reading dashboard summary: {e}")
|
||||
return jsonify({'error': str(e)}), 500
|
||||
|
||||
@self.app.route('/api/dashboard/series')
|
||||
def api_dashboard_series():
|
||||
"""Full-history points for one metric, for the expand-chart interaction."""
|
||||
metric = request.args.get('metric', 'messages')
|
||||
if metric not in SERIES_METRICS:
|
||||
return jsonify({'error': f'Unknown metric: {metric}'}), 400
|
||||
try:
|
||||
days = int(request.args.get('days', 30))
|
||||
except (TypeError, ValueError):
|
||||
days = 30
|
||||
try:
|
||||
with self._with_db_connection() as conn:
|
||||
payload = self.dashboard_stats.read_series(conn, metric, days)
|
||||
tag = f'{metric}:{days}:{payload.pop("etag_source", None)}'
|
||||
if request.if_none_match.contains(tag):
|
||||
response = self.app.response_class(status=304)
|
||||
else:
|
||||
response = jsonify(payload)
|
||||
response.headers['ETag'] = f'"{tag}"'
|
||||
response.headers['Cache-Control'] = 'private, max-age=300'
|
||||
return response
|
||||
except Exception as e:
|
||||
self.logger.error(f"Error reading dashboard series: {e}")
|
||||
return jsonify({'error': str(e)}), 500
|
||||
|
||||
@self.app.route('/api/dashboard/top')
|
||||
def api_dashboard_top():
|
||||
"""One leaderboard, one narrow query — replaces four whole-payload reads."""
|
||||
kind = request.args.get('kind', 'users')
|
||||
if kind not in TOP_KINDS:
|
||||
return jsonify({'error': f'Unknown kind: {kind}'}), 400
|
||||
window = request.args.get('window', 'all')
|
||||
if window not in ('24h', '7d', '30d', '90d', 'all'):
|
||||
window = 'all'
|
||||
try:
|
||||
limit = int(request.args.get('limit', 15))
|
||||
except (TypeError, ValueError):
|
||||
limit = 15
|
||||
try:
|
||||
with self._with_db_connection() as conn:
|
||||
payload = self.dashboard_stats.read_top(conn, kind, window, limit)
|
||||
response = jsonify(payload)
|
||||
response.headers['Cache-Control'] = 'private, max-age=30'
|
||||
return response
|
||||
except Exception as e:
|
||||
self.logger.error(f"Error reading dashboard top {kind}: {e}")
|
||||
return jsonify({'error': str(e)}), 500
|
||||
|
||||
@self.app.route('/api/dashboard/windows')
|
||||
def api_dashboard_windows():
|
||||
"""Selector options derived from retention, so no label overclaims."""
|
||||
try:
|
||||
response = jsonify(self.dashboard_stats.derive_windows())
|
||||
response.headers['Cache-Control'] = 'private, max-age=300'
|
||||
return response
|
||||
except Exception as e:
|
||||
self.logger.error(f"Error deriving dashboard windows: {e}")
|
||||
return jsonify({'error': str(e)}), 500
|
||||
|
||||
@self.app.route('/api/dashboard/refresh', methods=['POST'])
|
||||
def api_dashboard_refresh():
|
||||
"""Force a snapshot refresh. Used by the health strip's manual control."""
|
||||
try:
|
||||
with closing(self._dashboard_connection()) as conn:
|
||||
result = self.dashboard_stats.refresh(conn)
|
||||
return jsonify({'success': True, **result})
|
||||
except Exception as e:
|
||||
self.logger.error(f"Error refreshing dashboard snapshot: {e}")
|
||||
return jsonify({'success': False, 'error': str(e)}), 500
|
||||
|
||||
@self.app.route('/api/stats/rate_limiters')
|
||||
def api_rate_limiter_stats():
|
||||
"""Return current rate limiter statistics from the running bot."""
|
||||
@@ -4457,6 +4601,159 @@ class BotDataViewer:
|
||||
polling_thread.start()
|
||||
self.logger.info("Database polling started")
|
||||
|
||||
def _config_int(self, section: str, option: str, fallback: int) -> int:
|
||||
"""Read an int config value, falling back on a missing or malformed entry."""
|
||||
try:
|
||||
return self.config.getint(section, option, fallback=fallback)
|
||||
except (configparser.Error, ValueError, TypeError):
|
||||
return fallback
|
||||
|
||||
def _init_dashboard_service(self):
|
||||
"""Build the dashboard rollup/snapshot service from config."""
|
||||
try:
|
||||
self.dashboard_snapshot_enabled = self.config.getboolean(
|
||||
'Web_Viewer', 'dashboard_snapshot_enabled', fallback=True
|
||||
)
|
||||
except (configparser.Error, ValueError, TypeError):
|
||||
self.dashboard_snapshot_enabled = True
|
||||
|
||||
self.dashboard_snapshot_interval = max(
|
||||
15, self._config_int('Web_Viewer', 'dashboard_snapshot_interval_seconds', 60)
|
||||
)
|
||||
self.dashboard_stats = DashboardStatsService(
|
||||
self.logger,
|
||||
history_days=self._config_int('Web_Viewer', 'dashboard_snapshot_history_days', 400),
|
||||
packet_backfill_rows=self._config_int('Web_Viewer', 'dashboard_packet_backfill_rows', 2000),
|
||||
interval_seconds=self.dashboard_snapshot_interval,
|
||||
# Retention drives which window labels the UI is allowed to offer,
|
||||
# so read the same keys the cleanup jobs enforce.
|
||||
stats_retention_days=self._config_int('Stats_Command', 'data_retention_days', 7),
|
||||
packet_retention_days=self._config_int('Data_Retention', 'packet_stream_retention_days', 3),
|
||||
adverts_retention_days=self._config_int('Data_Retention', 'daily_stats_retention_days', 90),
|
||||
multibyte_contacts_fn=self._count_contacts_7d_multibyte,
|
||||
)
|
||||
|
||||
def _count_contacts_7d_multibyte(self, cursor) -> tuple[int, int] | None:
|
||||
"""(multibyte, total) contacts heard in the last 7 days, or None if unavailable.
|
||||
|
||||
Injected into DashboardStatsService so the snapshot reuses the viewer's
|
||||
memoized hop-prefix evidence instead of rebuilding it.
|
||||
"""
|
||||
try:
|
||||
cursor.execute(
|
||||
"""
|
||||
SELECT COUNT(*) FROM complete_contact_tracking
|
||||
WHERE last_heard > datetime('now', 'localtime', '-7 days')
|
||||
"""
|
||||
)
|
||||
total = cursor.fetchone()[0] or 0
|
||||
except sqlite3.Error as e:
|
||||
self.logger.debug(f"Could not count 7d contacts: {e}")
|
||||
return None
|
||||
|
||||
chunk_buckets = self._bucket_hop_chunks(
|
||||
self._get_cached_contact_multibyte_hop_chunks(cursor, recent_days=7)
|
||||
)
|
||||
mb_advert_pks: set[str] = set()
|
||||
try:
|
||||
cursor.execute(
|
||||
"""
|
||||
SELECT DISTINCT public_key FROM observed_paths
|
||||
WHERE packet_type = 'advert' AND public_key IS NOT NULL
|
||||
AND bytes_per_hop IN (2, 3)
|
||||
AND date(last_seen) >= date('now', 'localtime', '-7 days')
|
||||
"""
|
||||
)
|
||||
mb_advert_pks = {row[0] for row in cursor.fetchall() if row[0]}
|
||||
except sqlite3.Error as e:
|
||||
self.logger.debug(f"Could not load 7d multibyte advert keys: {e}")
|
||||
|
||||
try:
|
||||
cursor.execute(
|
||||
"""
|
||||
SELECT public_key, role, out_bytes_per_hop
|
||||
FROM complete_contact_tracking
|
||||
WHERE last_heard > datetime('now', 'localtime', '-7 days')
|
||||
"""
|
||||
)
|
||||
multibyte = sum(
|
||||
1
|
||||
for row in cursor.fetchall()
|
||||
if self._contact_has_multibyte_path_evidence(
|
||||
row[0], row[1], row[2], mb_advert_pks, chunk_buckets
|
||||
)
|
||||
)
|
||||
except sqlite3.Error as e:
|
||||
self.logger.debug(f"Could not compute contacts_7d_multibyte_path: {e}")
|
||||
return None
|
||||
return multibyte, total
|
||||
|
||||
def _dashboard_connection(self):
|
||||
"""Connection for the refresher: autocommit, so BEGIN IMMEDIATE is ours.
|
||||
|
||||
The reader connections leave transaction control to pysqlite, but the
|
||||
refresher deliberately brackets its own write phase and must not have an
|
||||
implicit transaction opened underneath it.
|
||||
"""
|
||||
conn = sqlite3.connect(self.db_path, timeout=60, isolation_level=None)
|
||||
conn.row_factory = sqlite3.Row
|
||||
with suppress(sqlite3.OperationalError):
|
||||
conn.execute(f"PRAGMA busy_timeout={self._config_int('Web_Viewer', 'sqlite_busy_timeout_ms', 60000)}")
|
||||
conn.execute("PRAGMA journal_mode=WAL")
|
||||
return conn
|
||||
|
||||
def _start_dashboard_refresher(self):
|
||||
"""Recompute the dashboard snapshot on an interval, in this process.
|
||||
|
||||
The refresher lives in the viewer rather than the bot for two reasons:
|
||||
the viewer may point at a different database ([Web_Viewer] db_path), and
|
||||
it already runs migrations itself, so the rollup tables exist in
|
||||
whichever file this process opens. A bot-side scheduler job would write
|
||||
to the wrong file in a split-DB install and would not run at all for a
|
||||
standalone viewer.
|
||||
"""
|
||||
if not self.dashboard_snapshot_enabled:
|
||||
self.logger.info("Dashboard snapshot refresher disabled by config")
|
||||
return
|
||||
|
||||
def refresher():
|
||||
consecutive_errors = 0
|
||||
# Refresh once at startup so the first page load has a snapshot.
|
||||
delay = 2.0
|
||||
while True:
|
||||
time.sleep(delay)
|
||||
delay = self.dashboard_snapshot_interval
|
||||
try:
|
||||
with closing(self._dashboard_connection()) as conn:
|
||||
if not self.dashboard_stats.try_claim_lease(conn):
|
||||
self.logger.debug(
|
||||
"Another viewer holds the dashboard snapshot lease; skipping tick"
|
||||
)
|
||||
continue
|
||||
result = self.dashboard_stats.refresh(conn)
|
||||
consecutive_errors = 0
|
||||
self.logger.debug(
|
||||
"Dashboard snapshot refreshed in %sms (%s days, %s packet rows backfilled)",
|
||||
result['duration_ms'],
|
||||
result['days'],
|
||||
result['backfilled_packet_rows'],
|
||||
)
|
||||
except Exception as e:
|
||||
consecutive_errors += 1
|
||||
if consecutive_errors == 1:
|
||||
self.logger.error(f"Dashboard snapshot refresh failed: {e}", exc_info=True)
|
||||
else:
|
||||
self.logger.warning(
|
||||
f"Dashboard snapshot refresh failed ({consecutive_errors}): {e}"
|
||||
)
|
||||
delay = min(600, self.dashboard_snapshot_interval * (2 ** min(consecutive_errors, 5)))
|
||||
|
||||
thread = threading.Thread(target=refresher, name="dashboard-snapshot", daemon=True)
|
||||
thread.start()
|
||||
self.logger.info(
|
||||
f"Dashboard snapshot refresher started (every {self.dashboard_snapshot_interval}s)"
|
||||
)
|
||||
|
||||
def _start_cleanup_scheduler(self):
|
||||
"""Start background thread for periodic database cleanup"""
|
||||
import threading
|
||||
@@ -4600,12 +4897,14 @@ class BotDataViewer:
|
||||
# the pie chart matches "last 7 days" (lifetime paths + stale out_bytes_per_hop
|
||||
# otherwise inflated the percentage).
|
||||
stats['contacts_7d_multibyte_path'] = 0
|
||||
multibyte_chunks: set[str] = set()
|
||||
chunk_buckets = self._bucket_hop_chunks(set())
|
||||
mb_advert_pks: set[str] = set()
|
||||
if 'observed_paths' in tables:
|
||||
try:
|
||||
multibyte_chunks = self._collect_multibyte_hop_chunks(
|
||||
cursor, recent_days=7
|
||||
chunk_buckets = self._bucket_hop_chunks(
|
||||
self._get_cached_contact_multibyte_hop_chunks(
|
||||
cursor, recent_days=7
|
||||
)
|
||||
)
|
||||
# Use date() — julianday(iso8601) often returns NULL for Python isoformat() strings
|
||||
cursor.execute(
|
||||
@@ -4636,7 +4935,7 @@ class BotDataViewer:
|
||||
row["role"],
|
||||
row["out_bytes_per_hop"],
|
||||
mb_advert_pks,
|
||||
multibyte_chunks,
|
||||
chunk_buckets,
|
||||
):
|
||||
mb_7d += 1
|
||||
stats['contacts_7d_multibyte_path'] = mb_7d
|
||||
@@ -4674,7 +4973,10 @@ class BotDataViewer:
|
||||
""")
|
||||
stats['unique_device_types'] = cursor.fetchone()[0]
|
||||
|
||||
# Incoming packets (packet_stream): multibyte path share, last 7 days (decoded bytes_per_hop)
|
||||
# Incoming packets: multibyte path share over whatever packet_stream
|
||||
# actually retains. The key names still say 7d for compatibility,
|
||||
# but the window is reported honestly alongside them —
|
||||
# packet_stream is pruned at 3 days, so the old label was never true.
|
||||
stats['incoming_packets_7d'] = 0
|
||||
stats['incoming_packets_7d_multibyte_path'] = 0
|
||||
if 'packet_stream' in tables:
|
||||
@@ -4682,28 +4984,23 @@ class BotDataViewer:
|
||||
cutoff_ts = time.time() - 7 * 86400
|
||||
cursor.execute(
|
||||
"""
|
||||
SELECT COUNT(*) FROM packet_stream
|
||||
WHERE type = ? AND timestamp > ?
|
||||
SELECT COUNT(*),
|
||||
SUM(CASE WHEN bytes_per_hop IN (2, 3) THEN 1 ELSE 0 END),
|
||||
MIN(timestamp)
|
||||
FROM packet_stream
|
||||
WHERE type = ? AND timestamp > ? AND route_type_name IS NOT NULL
|
||||
""",
|
||||
("packet", cutoff_ts),
|
||||
)
|
||||
stats['incoming_packets_7d'] = cursor.fetchone()[0] or 0
|
||||
mb_pk = 0
|
||||
try:
|
||||
cursor.execute(
|
||||
"""
|
||||
SELECT COUNT(*) FROM packet_stream
|
||||
WHERE type = ? AND timestamp > ?
|
||||
AND CAST(json_extract(data, '$.bytes_per_hop') AS INTEGER) IN (2, 3)
|
||||
""",
|
||||
("packet", cutoff_ts),
|
||||
)
|
||||
mb_pk = cursor.fetchone()[0] or 0
|
||||
except sqlite3.OperationalError:
|
||||
mb_pk = self._count_multibyte_packets_from_stream_json(cursor, cutoff_ts)
|
||||
stats['incoming_packets_7d_multibyte_path'] = mb_pk
|
||||
row = cursor.fetchone()
|
||||
stats['incoming_packets_7d'] = row[0] or 0
|
||||
stats['incoming_packets_7d_multibyte_path'] = row[1] or 0
|
||||
stats['incoming_packets_from'] = row[2]
|
||||
stats['incoming_packets_window_label'] = humanize_span(
|
||||
time.time() - row[2] if row[2] else None
|
||||
)
|
||||
except Exception as e:
|
||||
self.logger.debug(f"Could not compute incoming_packets_7d multibyte stats: {e}")
|
||||
self.logger.debug(f"Could not compute incoming packet multibyte stats: {e}")
|
||||
|
||||
# Advertisement statistics using daily tracking table
|
||||
if 'daily_stats' in tables:
|
||||
@@ -5251,38 +5548,6 @@ class BotDataViewer:
|
||||
out.append(seg.lower())
|
||||
return out
|
||||
|
||||
def _count_multibyte_packets_from_stream_json(self, cursor, cutoff_ts: float) -> int:
|
||||
"""Count packet_stream rows (type=packet) since cutoff with bytes_per_hop in (2, 3). JSON parse fallback."""
|
||||
import json
|
||||
|
||||
n = 0
|
||||
try:
|
||||
cursor.execute(
|
||||
"""
|
||||
SELECT data FROM packet_stream
|
||||
WHERE type = ? AND timestamp > ?
|
||||
""",
|
||||
("packet", cutoff_ts),
|
||||
)
|
||||
for row in cursor.fetchall():
|
||||
raw = row["data"]
|
||||
if not raw:
|
||||
continue
|
||||
try:
|
||||
d = json.loads(raw)
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
continue
|
||||
bph = d.get("bytes_per_hop")
|
||||
try:
|
||||
bph_i = int(bph) if bph is not None else None
|
||||
except (TypeError, ValueError):
|
||||
bph_i = None
|
||||
if bph_i in (2, 3):
|
||||
n += 1
|
||||
except Exception as e:
|
||||
self.logger.debug(f"packet_stream JSON scan for multibyte: {e}")
|
||||
return n
|
||||
|
||||
def _collect_multibyte_hop_chunks(
|
||||
self, cursor, recent_days: int | None = None
|
||||
) -> set[str]:
|
||||
@@ -5322,8 +5587,15 @@ class BotDataViewer:
|
||||
self.logger.debug(f"Could not load multibyte hop chunks: {e}")
|
||||
return chunks
|
||||
|
||||
def _get_cached_contact_multibyte_hop_chunks(self, cursor) -> set[str]:
|
||||
"""Return all-time contact badge evidence without rebuilding it per page request."""
|
||||
def _get_cached_contact_multibyte_hop_chunks(
|
||||
self, cursor, recent_days: int | None = None
|
||||
) -> set[str]:
|
||||
"""Return contact badge evidence without rebuilding it per page request.
|
||||
|
||||
Memoized per ``recent_days`` window: the contacts list wants all-time
|
||||
evidence (None) while the dashboard wants the 7-day set, and both are
|
||||
rebuilt often enough that sharing one slot would thrash the cache.
|
||||
"""
|
||||
try:
|
||||
# last_seen changes when an existing path is observed again; count changes on
|
||||
# inserts and retention deletes. Together they cheaply invalidate the cache while
|
||||
@@ -5336,19 +5608,27 @@ class BotDataViewer:
|
||||
)
|
||||
signature_row = cursor.fetchone()
|
||||
signature = tuple(signature_row) if signature_row else (None, 0)
|
||||
if recent_days is not None:
|
||||
# A windowed set also changes when the window itself slides, which
|
||||
# leaves the row fingerprint untouched — pin it to the local date.
|
||||
cursor.execute("SELECT date('now', 'localtime')")
|
||||
signature = (*signature, cursor.fetchone()[0])
|
||||
except Exception as e:
|
||||
self.logger.debug(f"Could not fingerprint multibyte hop chunks: {e}")
|
||||
return self._collect_multibyte_hop_chunks(cursor)
|
||||
return self._collect_multibyte_hop_chunks(cursor, recent_days=recent_days)
|
||||
|
||||
lock = getattr(self, '_contacts_badge_cache_lock', None)
|
||||
if lock is None:
|
||||
lock = self._contacts_badge_cache_lock = threading.Lock()
|
||||
with lock:
|
||||
if getattr(self, '_contacts_badge_cache_signature', None) == signature:
|
||||
return self._contacts_badge_cache_chunks
|
||||
chunks = self._collect_multibyte_hop_chunks(cursor)
|
||||
self._contacts_badge_cache_signature = signature
|
||||
self._contacts_badge_cache_chunks = chunks
|
||||
cache = getattr(self, '_contacts_badge_cache', None)
|
||||
if cache is None:
|
||||
cache = self._contacts_badge_cache = {}
|
||||
cached = cache.get(recent_days)
|
||||
if cached is not None and cached[0] == signature:
|
||||
return cached[1]
|
||||
chunks = self._collect_multibyte_hop_chunks(cursor, recent_days=recent_days)
|
||||
cache[recent_days] = (signature, chunks)
|
||||
return chunks
|
||||
|
||||
@staticmethod
|
||||
@@ -5437,9 +5717,13 @@ class BotDataViewer:
|
||||
role: str | None,
|
||||
out_bytes_per_hop: Any,
|
||||
multibyte_advert_public_keys: set[str],
|
||||
multibyte_hop_chunks: set[str],
|
||||
chunk_buckets: dict[int, set[str]],
|
||||
) -> bool:
|
||||
"""Multibyte detection for dashboard 7d stats (observed_paths scoped by date in SQL)."""
|
||||
"""Multibyte detection for dashboard 7d stats (observed_paths scoped by date in SQL).
|
||||
|
||||
``chunk_buckets`` is the length-bucketed form of the multibyte hop chunks
|
||||
(see _bucket_hop_chunks) — build it once per request, not per contact.
|
||||
"""
|
||||
pk = public_key or ""
|
||||
role_l = (role or "").lower()
|
||||
obph: int | None
|
||||
@@ -5455,10 +5739,11 @@ class BotDataViewer:
|
||||
if pk and pk in multibyte_advert_public_keys:
|
||||
return True
|
||||
if role_l in ("repeater", "roomserver") and pk:
|
||||
# Chunks are fixed-length (4 or 6), so startswith reduces to prefix-equality
|
||||
# lookups — O(1) per contact instead of a scan of the whole chunk set.
|
||||
pk_low = pk.lower()
|
||||
for chunk in multibyte_hop_chunks:
|
||||
if pk_low.startswith(chunk):
|
||||
return True
|
||||
if pk_low[:4] in chunk_buckets[4] or pk_low[:6] in chunk_buckets[6]:
|
||||
return True
|
||||
return False
|
||||
|
||||
def _compute_single_byte_relay_metrics(
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -211,11 +211,21 @@ class BotIntegration:
|
||||
return resolve_path(raw, base_dir)
|
||||
return str(Path(self.bot.db_manager.db_path).resolve())
|
||||
|
||||
# Denormalized packet dimensions written alongside the JSON blob. Order
|
||||
# matters: it is the INSERT column order and the queue tuple order.
|
||||
PACKET_DIM_COLUMNS = ("route_type_name", "payload_type_name", "path_len", "bytes_per_hop")
|
||||
|
||||
def _init_packet_stream_table(self):
|
||||
"""Backward-compatible initializer (now handled by migrations).
|
||||
|
||||
Kept for older call sites and tests that patch this method. Safe to call
|
||||
multiple times and safe to ignore failures.
|
||||
|
||||
The table-exists path still runs the ALTERs: this process writes to
|
||||
``[Web_Viewer] db_path``, which in a split-DB install is a file the bot's
|
||||
own MigrationRunner never touches. If the viewer has not started yet
|
||||
that DB has an old three-column packet_stream, and skipping the ALTERs
|
||||
would leave every insert failing on unknown columns.
|
||||
"""
|
||||
try:
|
||||
import sqlite3
|
||||
@@ -226,6 +236,8 @@ class BotIntegration:
|
||||
"SELECT name FROM sqlite_master WHERE type='table' AND name='packet_stream'"
|
||||
)
|
||||
if cur.fetchone() is not None:
|
||||
self._add_missing_packet_dim_columns(conn)
|
||||
conn.commit()
|
||||
return
|
||||
try:
|
||||
foreign_keys = self.bot.config.getboolean("Web_Viewer", "sqlite_foreign_keys", fallback=True)
|
||||
@@ -247,7 +259,11 @@ class BotIntegration:
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
timestamp REAL NOT NULL,
|
||||
data TEXT NOT NULL,
|
||||
type TEXT NOT NULL
|
||||
type TEXT NOT NULL,
|
||||
route_type_name TEXT,
|
||||
payload_type_name TEXT,
|
||||
path_len INTEGER,
|
||||
bytes_per_hop INTEGER
|
||||
)
|
||||
"""
|
||||
)
|
||||
@@ -257,10 +273,62 @@ class BotIntegration:
|
||||
conn.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_packet_stream_type ON packet_stream(type)"
|
||||
)
|
||||
conn.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_packet_stream_dims "
|
||||
"ON packet_stream(timestamp, bytes_per_hop, route_type_name, payload_type_name) "
|
||||
"WHERE type = 'packet'"
|
||||
)
|
||||
conn.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_packet_stream_undimensioned "
|
||||
"ON packet_stream(type, id) WHERE route_type_name IS NULL"
|
||||
)
|
||||
conn.commit()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def _add_missing_packet_dim_columns(self, conn) -> None:
|
||||
"""ALTER in any denormalized dimension column packet_stream is missing."""
|
||||
existing = {row[1] for row in conn.execute("PRAGMA table_info(packet_stream)")}
|
||||
types = {"path_len": "INTEGER", "bytes_per_hop": "INTEGER"}
|
||||
for column in self.PACKET_DIM_COLUMNS:
|
||||
if column not in existing:
|
||||
conn.execute(
|
||||
f"ALTER TABLE packet_stream ADD COLUMN {column} {types.get(column, 'TEXT')}"
|
||||
)
|
||||
if self.PACKET_DIM_COLUMNS[0] not in existing:
|
||||
conn.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_packet_stream_dims "
|
||||
"ON packet_stream(timestamp, bytes_per_hop, route_type_name, payload_type_name) "
|
||||
"WHERE type = 'packet'"
|
||||
)
|
||||
conn.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_packet_stream_undimensioned "
|
||||
"ON packet_stream(type, id) WHERE route_type_name IS NULL"
|
||||
)
|
||||
|
||||
def _packet_stream_has_dim_columns(self, conn) -> bool:
|
||||
"""Probe once per process whether the target DB carries the dimension columns.
|
||||
|
||||
The viewer DB may be a separate file this process never migrates, so the
|
||||
writer cannot assume the wide schema — it falls back to the three-column
|
||||
INSERT rather than failing every flush.
|
||||
"""
|
||||
cached = getattr(self, '_packet_dims_supported', None)
|
||||
if cached is not None:
|
||||
return cached
|
||||
try:
|
||||
existing = {row[1] for row in conn.execute("PRAGMA table_info(packet_stream)")}
|
||||
supported = all(c in existing for c in self.PACKET_DIM_COLUMNS)
|
||||
except Exception:
|
||||
supported = False
|
||||
self._packet_dims_supported = supported
|
||||
if not supported:
|
||||
self.bot.logger.info(
|
||||
"packet_stream lacks denormalized dimension columns; "
|
||||
"writing legacy rows (dashboard packet tiles will backfill from JSON)"
|
||||
)
|
||||
return supported
|
||||
|
||||
def _start_drain_thread(self) -> None:
|
||||
"""Start the background thread that flushes the write queue every DRAIN_INTERVAL seconds."""
|
||||
self._drain_thread = threading.Thread(
|
||||
@@ -276,7 +344,7 @@ class BotIntegration:
|
||||
self._drain_stop.wait(timeout=self.DRAIN_INTERVAL)
|
||||
self._flush_write_queue()
|
||||
|
||||
def _requeue_rows(self, rows: list[tuple[float, str, str]]) -> None:
|
||||
def _requeue_rows(self, rows: list[tuple]) -> None:
|
||||
"""Restore rows to the queue after a failed flush (FIFO). Logs if the queue stays full."""
|
||||
for i, row in enumerate(rows):
|
||||
try:
|
||||
@@ -296,7 +364,7 @@ class BotIntegration:
|
||||
with self._flush_lock:
|
||||
if self._write_queue.empty():
|
||||
return
|
||||
rows: list[tuple[float, str, str]] = []
|
||||
rows: list[tuple] = []
|
||||
while not self._write_queue.empty():
|
||||
try:
|
||||
rows.append(self._write_queue.get_nowait())
|
||||
@@ -309,10 +377,18 @@ class BotIntegration:
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
with closing(sqlite3.connect(str(db_path), timeout=self.sqlite_connect_timeout_sec)) as conn:
|
||||
conn.executemany(
|
||||
'INSERT INTO packet_stream (timestamp, data, type) VALUES (?, ?, ?)',
|
||||
rows,
|
||||
)
|
||||
if self._packet_stream_has_dim_columns(conn):
|
||||
conn.executemany(
|
||||
'INSERT INTO packet_stream '
|
||||
'(timestamp, data, type, route_type_name, payload_type_name, '
|
||||
'path_len, bytes_per_hop) VALUES (?, ?, ?, ?, ?, ?, ?)',
|
||||
rows,
|
||||
)
|
||||
else:
|
||||
conn.executemany(
|
||||
'INSERT INTO packet_stream (timestamp, data, type) VALUES (?, ?, ?)',
|
||||
[row[:3] for row in rows],
|
||||
)
|
||||
conn.commit()
|
||||
return
|
||||
except sqlite3.OperationalError as e:
|
||||
@@ -331,9 +407,20 @@ class BotIntegration:
|
||||
self._requeue_rows(rows)
|
||||
return
|
||||
|
||||
def _insert_packet_stream_row(self, data_json: str, row_type: str, log_prefix: str = "packet data"):
|
||||
"""Queue one row for batched insertion into packet_stream by the drain thread."""
|
||||
item = (time.time(), data_json, row_type)
|
||||
def _insert_packet_stream_row(
|
||||
self,
|
||||
data_json: str,
|
||||
row_type: str,
|
||||
log_prefix: str = "packet data",
|
||||
dims: tuple | None = None,
|
||||
):
|
||||
"""Queue one row for batched insertion into packet_stream by the drain thread.
|
||||
|
||||
``dims`` carries the denormalized (route_type_name, payload_type_name,
|
||||
path_len, bytes_per_hop) values for packet rows; command/message/other
|
||||
row types pass None and store NULLs.
|
||||
"""
|
||||
item = (time.time(), data_json, row_type, *(dims or (None, None, None, None)))
|
||||
try:
|
||||
self._write_queue.put(item, timeout=self._WRITE_QUEUE_PUT_TIMEOUT_SEC)
|
||||
except queue.Full:
|
||||
@@ -382,11 +469,50 @@ class BotIntegration:
|
||||
serializable_data = self._make_json_serializable(packet_data)
|
||||
|
||||
# Store in database for web viewer to read (retries on database is locked)
|
||||
self._insert_packet_stream_row(json.dumps(serializable_data), 'packet', "packet data")
|
||||
self._insert_packet_stream_row(
|
||||
json.dumps(serializable_data),
|
||||
'packet',
|
||||
"packet data",
|
||||
dims=self._packet_dims_from_data(serializable_data),
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
self.bot.logger.warning(f"Error storing packet data for web viewer: {e}")
|
||||
|
||||
@staticmethod
|
||||
def _packet_dims_from_data(packet_data: dict) -> tuple:
|
||||
"""Pull the denormalized dimension values out of an already-parsed packet.
|
||||
|
||||
Returns (route_type_name, payload_type_name, path_len, bytes_per_hop),
|
||||
with None for anything absent or non-coercible — the dashboard reads
|
||||
these as "unknown" rather than mis-bucketing the packet.
|
||||
"""
|
||||
def _text(key: str) -> str | None:
|
||||
value = packet_data.get(key)
|
||||
if value is None or isinstance(value, (dict, list)):
|
||||
return None
|
||||
text = str(value).strip()
|
||||
return text[:32] or None
|
||||
|
||||
def _number(key: str) -> int | None:
|
||||
value = packet_data.get(key)
|
||||
if value is None or isinstance(value, bool):
|
||||
return None
|
||||
try:
|
||||
return int(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
return (
|
||||
# Never NULL: route_type_name is the dashboard's "this row has been
|
||||
# dimensioned" marker, and a NULL would put the row back in the
|
||||
# refresher's backfill queue on every tick.
|
||||
_text('route_type_name') or 'UNKNOWN_ROUTE',
|
||||
_text('payload_type_name'),
|
||||
_number('path_len'),
|
||||
_number('bytes_per_hop'),
|
||||
)
|
||||
|
||||
def capture_command(self, message, command_name, response, success, command_id=None):
|
||||
"""Capture command data and store in database for web viewer"""
|
||||
try:
|
||||
|
||||
@@ -0,0 +1,394 @@
|
||||
/* Dashboard (landing page) styles.
|
||||
*
|
||||
* Only the theme tokens declared in base.html are used (--bg-*, --text-*,
|
||||
* --border-color, --card-*, --shadow-color), and every rule that hard-codes a
|
||||
* colour has a matching [data-theme="dark"] override — Bootstrap 5.1.3 has no
|
||||
* native dark theme here, so nothing inherits one for free.
|
||||
*/
|
||||
|
||||
/* ── Health strip ──────────────────────────────────────────────────────── */
|
||||
|
||||
.health-strip {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
align-items: center;
|
||||
gap: 0.25rem 1.5rem;
|
||||
min-height: 56px;
|
||||
padding: 0.5rem 1rem;
|
||||
margin-bottom: 1.25rem;
|
||||
background-color: var(--card-bg);
|
||||
border: 1px solid var(--border-color);
|
||||
border-radius: 10px;
|
||||
box-shadow: 0 1px 3px var(--shadow-color);
|
||||
}
|
||||
|
||||
.health-strip__item {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 0.45rem;
|
||||
font-size: 0.875rem;
|
||||
color: var(--text-color);
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.health-strip__label {
|
||||
color: var(--text-muted);
|
||||
text-transform: uppercase;
|
||||
font-size: 0.7rem;
|
||||
letter-spacing: 0.04em;
|
||||
}
|
||||
|
||||
.health-strip__spacer {
|
||||
flex: 1 1 auto;
|
||||
}
|
||||
|
||||
.health-strip .btn-refresh {
|
||||
padding: 0.1rem 0.4rem;
|
||||
line-height: 1;
|
||||
border: none;
|
||||
background: transparent;
|
||||
color: var(--text-muted);
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.health-strip .btn-refresh:hover,
|
||||
.health-strip .btn-refresh:focus {
|
||||
color: #0d6efd;
|
||||
}
|
||||
|
||||
.health-strip .btn-refresh[disabled] {
|
||||
opacity: 0.5;
|
||||
cursor: progress;
|
||||
}
|
||||
|
||||
/* ── Section headings ──────────────────────────────────────────────────── */
|
||||
|
||||
.dashboard-section-title {
|
||||
display: flex;
|
||||
align-items: baseline;
|
||||
gap: 0.6rem;
|
||||
margin: 1.75rem 0 0.75rem;
|
||||
font-size: 0.8rem;
|
||||
font-weight: 700;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.08em;
|
||||
color: var(--text-muted);
|
||||
}
|
||||
|
||||
.dashboard-section-title::after {
|
||||
content: "";
|
||||
flex: 1 1 auto;
|
||||
height: 1px;
|
||||
background-color: var(--border-color);
|
||||
}
|
||||
|
||||
/* ── Stat tiles ────────────────────────────────────────────────────────── */
|
||||
|
||||
.stat-tile {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
height: 100%;
|
||||
padding: 0.75rem 0.9rem 0.5rem;
|
||||
background-color: var(--card-bg);
|
||||
border: 1px solid var(--border-color);
|
||||
border-radius: 10px;
|
||||
box-shadow: 0 1px 3px var(--shadow-color);
|
||||
}
|
||||
|
||||
.stat-tile__label {
|
||||
font-size: 0.7rem;
|
||||
font-weight: 600;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.04em;
|
||||
color: var(--text-muted);
|
||||
margin-bottom: 0.2rem;
|
||||
}
|
||||
|
||||
.stat-tile__row {
|
||||
display: flex;
|
||||
align-items: baseline;
|
||||
gap: 0.5rem;
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
|
||||
.stat-tile__value {
|
||||
font-size: 1.6rem;
|
||||
font-weight: 700;
|
||||
line-height: 1.1;
|
||||
color: var(--text-color);
|
||||
font-variant-numeric: tabular-nums;
|
||||
}
|
||||
|
||||
.stat-tile__value--muted {
|
||||
color: var(--text-muted);
|
||||
}
|
||||
|
||||
.stat-tile__sub {
|
||||
font-size: 0.75rem;
|
||||
color: var(--text-muted);
|
||||
}
|
||||
|
||||
.stat-tile__spark {
|
||||
position: relative;
|
||||
height: 48px;
|
||||
margin-top: auto;
|
||||
}
|
||||
|
||||
/* Sparkline canvases are sized by their container, not by their attributes. */
|
||||
.stat-tile__spark canvas {
|
||||
width: 100% !important;
|
||||
height: 48px !important;
|
||||
}
|
||||
|
||||
/* ── Delta chips ───────────────────────────────────────────────────────── */
|
||||
|
||||
.delta-chip {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 0.15rem;
|
||||
padding: 0.05rem 0.35rem;
|
||||
border-radius: 999px;
|
||||
font-size: 0.72rem;
|
||||
font-weight: 600;
|
||||
font-variant-numeric: tabular-nums;
|
||||
background-color: var(--bg-tertiary);
|
||||
color: var(--text-muted);
|
||||
}
|
||||
|
||||
.delta-chip--up {
|
||||
background-color: rgba(25, 135, 84, 0.12);
|
||||
color: #146c43;
|
||||
}
|
||||
|
||||
.delta-chip--down {
|
||||
background-color: rgba(220, 53, 69, 0.12);
|
||||
color: #b02a37;
|
||||
}
|
||||
|
||||
[data-theme="dark"] .delta-chip--up {
|
||||
background-color: rgba(25, 135, 84, 0.22);
|
||||
color: #75b798;
|
||||
}
|
||||
|
||||
[data-theme="dark"] .delta-chip--down {
|
||||
background-color: rgba(220, 53, 69, 0.22);
|
||||
color: #ea868f;
|
||||
}
|
||||
|
||||
/* ── Mini bar (tracked vs known) ───────────────────────────────────────── */
|
||||
|
||||
.mini-bar {
|
||||
height: 6px;
|
||||
width: 100%;
|
||||
margin-top: 0.4rem;
|
||||
border-radius: 3px;
|
||||
background-color: var(--bg-tertiary);
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.mini-bar__fill {
|
||||
height: 100%;
|
||||
background-color: #0d6efd;
|
||||
border-radius: 3px;
|
||||
}
|
||||
|
||||
/* ── Stacked route-mix bar ─────────────────────────────────────────────── */
|
||||
|
||||
.stacked-bar {
|
||||
display: flex;
|
||||
height: 26px;
|
||||
width: 100%;
|
||||
border-radius: 6px;
|
||||
overflow: hidden;
|
||||
background-color: var(--bg-tertiary);
|
||||
}
|
||||
|
||||
.stacked-bar__segment {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
font-size: 0.72rem;
|
||||
font-weight: 600;
|
||||
color: #fff;
|
||||
white-space: nowrap;
|
||||
overflow: hidden;
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
.stacked-bar__legend {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 0.25rem 1rem;
|
||||
margin-top: 0.5rem;
|
||||
font-size: 0.78rem;
|
||||
color: var(--text-muted);
|
||||
}
|
||||
|
||||
.stacked-bar__swatch {
|
||||
display: inline-block;
|
||||
width: 10px;
|
||||
height: 10px;
|
||||
border-radius: 2px;
|
||||
margin-right: 0.35rem;
|
||||
vertical-align: baseline;
|
||||
}
|
||||
|
||||
/* ── Horizontal mix rows (role / device) ───────────────────────────────── */
|
||||
|
||||
.mix-row {
|
||||
display: grid;
|
||||
grid-template-columns: minmax(5.5rem, 8rem) 1fr auto;
|
||||
align-items: center;
|
||||
gap: 0.6rem;
|
||||
margin-bottom: 0.35rem;
|
||||
font-size: 0.82rem;
|
||||
}
|
||||
|
||||
.mix-row__name {
|
||||
color: var(--text-color);
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.mix-row__track {
|
||||
height: 10px;
|
||||
border-radius: 5px;
|
||||
background-color: var(--bg-tertiary);
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.mix-row__fill {
|
||||
height: 100%;
|
||||
border-radius: 5px;
|
||||
}
|
||||
|
||||
.mix-row__count {
|
||||
color: var(--text-muted);
|
||||
font-variant-numeric: tabular-nums;
|
||||
}
|
||||
|
||||
/* ── Signal percentile readout ─────────────────────────────────────────── */
|
||||
|
||||
.signal-grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(3, 1fr);
|
||||
gap: 0.5rem;
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
.signal-grid__value {
|
||||
font-size: 1.1rem;
|
||||
font-weight: 700;
|
||||
color: var(--text-color);
|
||||
font-variant-numeric: tabular-nums;
|
||||
}
|
||||
|
||||
.signal-grid__label {
|
||||
font-size: 0.68rem;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.04em;
|
||||
color: var(--text-muted);
|
||||
}
|
||||
|
||||
/* ── Leaderboard rows ──────────────────────────────────────────────────── */
|
||||
|
||||
.top-list-row {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
gap: 0.5rem;
|
||||
padding: 0.25rem 0;
|
||||
}
|
||||
|
||||
.top-list-row + .top-list-row {
|
||||
border-top: 1px solid var(--border-color);
|
||||
}
|
||||
|
||||
.top-list-row__name {
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
/* ── Chart wrappers ────────────────────────────────────────────────────── */
|
||||
|
||||
.chart-box {
|
||||
position: relative;
|
||||
width: 100%;
|
||||
height: 200px;
|
||||
}
|
||||
|
||||
.chart-box--tall {
|
||||
height: 240px;
|
||||
}
|
||||
|
||||
.doughnut-box {
|
||||
position: relative;
|
||||
max-width: 190px;
|
||||
width: 100%;
|
||||
margin: 0 auto;
|
||||
}
|
||||
|
||||
.dashboard-note {
|
||||
font-size: 0.75rem;
|
||||
color: var(--text-muted);
|
||||
}
|
||||
|
||||
.dashboard-empty {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
height: 100%;
|
||||
min-height: 100px;
|
||||
color: var(--text-muted);
|
||||
font-size: 0.85rem;
|
||||
font-style: italic;
|
||||
}
|
||||
|
||||
/* Info affordance shared by tooltip triggers in card headers. */
|
||||
.dashboard-info-btn {
|
||||
padding: 0;
|
||||
border: 0;
|
||||
background: none;
|
||||
color: var(--text-muted);
|
||||
line-height: 1;
|
||||
}
|
||||
|
||||
.dashboard-info-btn:hover,
|
||||
.dashboard-info-btn:focus {
|
||||
color: #0dcaf0;
|
||||
text-decoration: none;
|
||||
}
|
||||
|
||||
.tooltip.dashboard-hint .tooltip-inner {
|
||||
max-width: min(22rem, 92vw);
|
||||
text-align: left;
|
||||
}
|
||||
|
||||
/* ── Live activity feed ────────────────────────────────────────────────── */
|
||||
|
||||
.live-feed {
|
||||
height: 260px;
|
||||
overflow-y: auto;
|
||||
background-color: var(--bg-secondary);
|
||||
padding: 0.5rem;
|
||||
}
|
||||
|
||||
.live-feed__entry {
|
||||
padding: 3px 8px;
|
||||
margin-bottom: 3px;
|
||||
font-size: 0.82rem;
|
||||
border-radius: 2px;
|
||||
border-left: 3px solid var(--border-color);
|
||||
}
|
||||
|
||||
@media (max-width: 575.98px) {
|
||||
.stat-tile__value {
|
||||
font-size: 1.35rem;
|
||||
}
|
||||
|
||||
.health-strip {
|
||||
gap: 0.25rem 1rem;
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -58,13 +58,20 @@
|
||||
("GET", "/api/health", "Bot health: radio state, uptime, client count"),
|
||||
("GET", "/api/banner-status", "Banner state: zombie, offline, initializing flags"),
|
||||
("GET", "/api/system-health", "Extended system health: DB size, queue depths, scheduler"),
|
||||
("GET", "/api/stats", "Bot-wide statistics: messages, commands, contacts"),
|
||||
("GET", "/api/stats", "DEPRECATED — whole-database statistics; use /api/dashboard/*"),
|
||||
("GET", "/api/stats/rate_limiters", "Per-sender rate limiter stats"),
|
||||
("GET", "/api/connected_clients","SocketIO connected client count"),
|
||||
("GET", "/api/cache", "In-memory cache snapshot"),
|
||||
("GET", "/api/database", "Database metadata: path, size, table row counts"),
|
||||
("POST", "/api/optimize-database","Run VACUUM + ANALYZE on the SQLite database"),
|
||||
]),
|
||||
("Dashboard", "fa-tachometer-alt", [
|
||||
("GET", "/api/dashboard/summary", "Snapshot-backed dashboard payload with 30-day sparkline series"),
|
||||
("GET", "/api/dashboard/series", "Points for one metric (metric, days); period totals and change"),
|
||||
("GET", "/api/dashboard/top", "One leaderboard (kind, window, limit) with retention labelling"),
|
||||
("GET", "/api/dashboard/windows", "Selector options derived from each source's retention"),
|
||||
("POST", "/api/dashboard/refresh", "Force a snapshot recomputation now"),
|
||||
]),
|
||||
("Contacts", "fa-users", [
|
||||
("GET", "/api/contacts", "List contacts; supports paging, search, and sorting"),
|
||||
("GET", "/api/export/contacts", "Export contacts as CSV"),
|
||||
|
||||
+400
-1453
File diff suppressed because it is too large.
Load diff
@@ -1,355 +0,0 @@
|
||||
{% extends "base.html" %}
|
||||
|
||||
{% block title %}Statistics - MeshCore Bot Data Viewer{% endblock %}
|
||||
|
||||
{% block content %}
|
||||
<div class="row">
|
||||
<div class="col-12">
|
||||
<h1 class="mb-4">
|
||||
<i class="fas fa-chart-bar"></i> Statistics
|
||||
</h1>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Key Metrics -->
|
||||
<div class="row mb-4">
|
||||
<div class="col-md-3">
|
||||
<div class="card">
|
||||
<div class="card-header">
|
||||
<i class="fas fa-users"></i> Total Contacts
|
||||
</div>
|
||||
<div class="card-body">
|
||||
<h3 id="total-contacts">0</h3>
|
||||
<small class="text-muted">All time</small>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="col-md-3">
|
||||
<div class="card">
|
||||
<div class="card-header">
|
||||
<i class="fas fa-comments"></i> Total Adverts
|
||||
</div>
|
||||
<div class="card-body">
|
||||
<h3 id="total-adverts">0</h3>
|
||||
<small class="text-muted">All time</small>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="col-md-3">
|
||||
<div class="card">
|
||||
<div class="card-header">
|
||||
<i class="fas fa-clock"></i> Recent Activity
|
||||
</div>
|
||||
<div class="card-body">
|
||||
<h3 id="recent-activity">0</h3>
|
||||
<small class="text-muted">Last 24 hours</small>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="col-md-3">
|
||||
<div class="card">
|
||||
<div class="card-header">
|
||||
<i class="fas fa-chart-line"></i> Growth Rate
|
||||
</div>
|
||||
<div class="card-body">
|
||||
<h3 id="growth-rate">0%</h3>
|
||||
<small class="text-muted">Daily growth</small>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Activity Charts -->
|
||||
<div class="row mb-4">
|
||||
<div class="col-md-6">
|
||||
<div class="card">
|
||||
<div class="card-header">
|
||||
<i class="fas fa-chart-pie"></i> Activity Distribution
|
||||
</div>
|
||||
<div class="card-body">
|
||||
<div id="activity-chart">
|
||||
<div class="loading">Loading activity chart...</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="col-md-6">
|
||||
<div class="card">
|
||||
<div class="card-header">
|
||||
<i class="fas fa-chart-line"></i> Usage Trends
|
||||
</div>
|
||||
<div class="card-body">
|
||||
<div id="trends-chart">
|
||||
<div class="loading">Loading trends chart...</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Detailed Statistics -->
|
||||
<div class="row mb-4">
|
||||
<div class="col-12">
|
||||
<div class="card">
|
||||
<div class="card-header">
|
||||
<i class="fas fa-table"></i> Detailed Statistics
|
||||
</div>
|
||||
<div class="card-body">
|
||||
<div class="row">
|
||||
<div class="col-md-6">
|
||||
<h6>Contact Statistics</h6>
|
||||
<ul class="list-unstyled" id="contact-stats">
|
||||
<li class="loading">Loading contact statistics...</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="col-md-6">
|
||||
<h6>Advert Statistics</h6>
|
||||
<ul class="list-unstyled" id="advert-stats">
|
||||
<li class="loading">Loading advert statistics...</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- System Performance -->
|
||||
<div class="row">
|
||||
<div class="col-md-6">
|
||||
<div class="card">
|
||||
<div class="card-header">
|
||||
<i class="fas fa-server"></i> System Performance
|
||||
</div>
|
||||
<div class="card-body">
|
||||
<div id="system-performance">
|
||||
<div class="loading">Loading system performance...</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="col-md-6">
|
||||
<div class="card">
|
||||
<div class="card-header">
|
||||
<i class="fas fa-cog"></i> Configuration
|
||||
</div>
|
||||
<div class="card-body">
|
||||
<div id="system-config">
|
||||
<div class="loading">Loading system configuration...</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
{% endblock %}
|
||||
|
||||
{% block extra_js %}
|
||||
<script nonce="{{ g.csp_nonce }}">
|
||||
class ModernStatsManager {
|
||||
constructor() {
|
||||
this.statsData = {};
|
||||
this.initializeStats();
|
||||
}
|
||||
|
||||
async initializeStats() {
|
||||
await this.loadStatsData();
|
||||
this.setupEventHandlers();
|
||||
this.updateStatistics();
|
||||
this.renderCharts();
|
||||
this.renderDetailedStats();
|
||||
this.renderSystemInfo();
|
||||
}
|
||||
|
||||
async loadStatsData() {
|
||||
try {
|
||||
const response = await fetch('/api/stats');
|
||||
const data = await response.json();
|
||||
|
||||
if (data.error) {
|
||||
this.showError('Failed to load statistics: ' + data.error);
|
||||
return;
|
||||
}
|
||||
|
||||
this.statsData = data;
|
||||
|
||||
} catch (error) {
|
||||
console.error('Error loading statistics:', error);
|
||||
this.showError('Failed to load statistics: ' + error.message);
|
||||
}
|
||||
}
|
||||
|
||||
setupEventHandlers() {
|
||||
// Auto-refresh every 30 seconds
|
||||
setInterval(() => {
|
||||
this.loadStatsData().then(() => {
|
||||
this.updateStatistics();
|
||||
this.renderCharts();
|
||||
this.renderDetailedStats();
|
||||
this.renderSystemInfo();
|
||||
});
|
||||
}, 30000);
|
||||
}
|
||||
|
||||
updateStatistics() {
|
||||
document.getElementById('total-contacts').textContent = this.statsData.contact_count || 0;
|
||||
document.getElementById('total-adverts').textContent = this.statsData.total_advertisements || 0;
|
||||
document.getElementById('recent-activity').textContent = this.statsData.recent_contacts_24h || 0;
|
||||
|
||||
// Calculate growth rate
|
||||
const recentContacts = this.statsData.recent_contacts_24h || 0;
|
||||
const totalContacts = this.statsData.contact_count || 0;
|
||||
const growthRate = totalContacts > 0 ? Math.round((recentContacts / totalContacts) * 100) : 0;
|
||||
document.getElementById('growth-rate').textContent = growthRate + '%';
|
||||
}
|
||||
|
||||
renderCharts() {
|
||||
// Activity distribution chart
|
||||
const activityChart = document.getElementById('activity-chart');
|
||||
const recentContacts = this.statsData.recent_contacts_24h || 0;
|
||||
const recentAdverts = this.statsData.advertisements_24h || 0;
|
||||
const total = recentContacts + recentAdverts;
|
||||
|
||||
if (total > 0) {
|
||||
const contactsPercent = Math.round((recentContacts / total) * 100);
|
||||
const advertsPercent = Math.round((recentAdverts / total) * 100);
|
||||
|
||||
const row = document.createElement('div');
|
||||
row.className = 'row';
|
||||
row.append(
|
||||
this.makeMetricColumn(`${contactsPercent}%`, 'Contacts', 'text-primary'),
|
||||
this.makeMetricColumn(`${advertsPercent}%`, 'Adverts', 'text-warning'),
|
||||
);
|
||||
activityChart.replaceChildren(row);
|
||||
} else {
|
||||
activityChart.replaceChildren(
|
||||
this.makeTextElement('div', 'No recent activity', 'text-muted text-center'),
|
||||
);
|
||||
}
|
||||
|
||||
// Trends chart
|
||||
const trendsChart = document.getElementById('trends-chart');
|
||||
const growthRate = this.statsData.contact_count > 0 ?
|
||||
Math.round(((this.statsData.recent_contacts_24h || 0) / this.statsData.contact_count) * 100) : 0;
|
||||
|
||||
const trend = document.createElement('div');
|
||||
trend.className = 'text-center';
|
||||
trend.append(
|
||||
this.makeTextElement('h4', `${growthRate}%`, `text-${growthRate > 0 ? 'success' : 'secondary'}`),
|
||||
this.makeTextElement('small', 'Daily growth rate', 'text-muted'),
|
||||
);
|
||||
trendsChart.replaceChildren(trend);
|
||||
}
|
||||
|
||||
renderDetailedStats() {
|
||||
// Contact statistics
|
||||
const contactStats = document.getElementById('contact-stats');
|
||||
this.renderLabeledList(contactStats, [
|
||||
['Total Contacts', this.statsData.contact_count || 0],
|
||||
['Recent (24h)', this.statsData.recent_contacts_24h || 0],
|
||||
['Growth Rate', `${this.calculateGrowthRate()}%`],
|
||||
['Last Updated', new Date().toLocaleString()],
|
||||
], false);
|
||||
|
||||
// Advert statistics
|
||||
const advertStats = document.getElementById('advert-stats');
|
||||
this.renderLabeledList(advertStats, [
|
||||
['Total Adverts', this.statsData.total_advertisements || 0],
|
||||
['Recent (24h)', this.statsData.advertisements_24h || 0],
|
||||
['Active Users', this.statsData.active_users_24h || 0],
|
||||
['Last Updated', new Date().toLocaleString()],
|
||||
], false);
|
||||
}
|
||||
|
||||
renderSystemInfo() {
|
||||
// System performance
|
||||
const systemPerformance = document.getElementById('system-performance');
|
||||
this.renderLabeledList(systemPerformance, [
|
||||
['Connected Clients', this.statsData.connected_clients || 0],
|
||||
['Database Status', 'Healthy'],
|
||||
['Last Activity', new Date().toLocaleString()],
|
||||
['Uptime', this.calculateUptime()],
|
||||
]);
|
||||
|
||||
// System configuration
|
||||
const systemConfig = document.getElementById('system-config');
|
||||
this.renderLabeledList(systemConfig, [
|
||||
['Version', 'Modern v2.0'],
|
||||
['Flask-SocketIO', '5.x'],
|
||||
['Database', 'SQLite'],
|
||||
['Real-time', 'Enabled'],
|
||||
]);
|
||||
}
|
||||
|
||||
makeTextElement(tagName, value, className = '') {
|
||||
const element = document.createElement(tagName);
|
||||
element.className = className;
|
||||
element.textContent = String(value ?? '');
|
||||
return element;
|
||||
}
|
||||
|
||||
makeMetricColumn(value, label, valueClass) {
|
||||
const column = document.createElement('div');
|
||||
column.className = 'col-6';
|
||||
const content = document.createElement('div');
|
||||
content.className = 'text-center';
|
||||
content.append(
|
||||
this.makeTextElement('h4', value, valueClass),
|
||||
this.makeTextElement('small', label, 'text-muted'),
|
||||
);
|
||||
column.appendChild(content);
|
||||
return column;
|
||||
}
|
||||
|
||||
renderLabeledList(container, rows, wrap = true) {
|
||||
const list = document.createElement(wrap ? 'ul' : 'div');
|
||||
if (wrap) list.className = 'list-unstyled';
|
||||
rows.forEach(([label, value]) => {
|
||||
const item = document.createElement('li');
|
||||
const strong = document.createElement('strong');
|
||||
strong.textContent = `${label}:`;
|
||||
item.append(strong, document.createTextNode(` ${String(value ?? '')}`));
|
||||
list.appendChild(item);
|
||||
});
|
||||
container.replaceChildren(...(wrap ? [list] : Array.from(list.children)));
|
||||
}
|
||||
|
||||
calculateGrowthRate() {
|
||||
const recentContacts = this.statsData.recent_contacts_24h || 0;
|
||||
const totalContacts = this.statsData.contact_count || 0;
|
||||
return totalContacts > 0 ? Math.round((recentContacts / totalContacts) * 100) : 0;
|
||||
}
|
||||
|
||||
calculateUptime() {
|
||||
// This would be calculated based on actual system uptime
|
||||
return 'Unknown';
|
||||
}
|
||||
|
||||
showError(message) {
|
||||
const errorDiv = document.createElement('div');
|
||||
errorDiv.className = 'error';
|
||||
errorDiv.textContent = message;
|
||||
|
||||
const content = document.querySelector('.container-fluid');
|
||||
if (content) {
|
||||
content.insertBefore(errorDiv, content.firstChild);
|
||||
|
||||
setTimeout(() => {
|
||||
if (errorDiv.parentNode) {
|
||||
errorDiv.parentNode.removeChild(errorDiv);
|
||||
}
|
||||
}, 5000);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Initialize stats manager when page loads
|
||||
document.addEventListener('DOMContentLoaded', () => {
|
||||
window.statsManager = new ModernStatsManager();
|
||||
});
|
||||
</script>
|
||||
{% endblock %}
|
||||
+1
-1
@@ -5,7 +5,7 @@
|
||||
"description": "Frontend linting for meshcore-bot web viewer templates",
|
||||
"scripts": {
|
||||
"lint:html": "htmlhint \"modules/web_viewer/templates/**/*.html\"",
|
||||
"lint:js": "eslint \"modules/web_viewer/templates/**/*.html\"",
|
||||
"lint:js": "eslint \"modules/web_viewer/templates/**/*.html\" modules/web_viewer/static/js/dashboard.js modules/web_viewer/static/js/channel_operations.js",
|
||||
"lint:frontend": "npm run lint:html && npm run lint:js",
|
||||
"test:contacts": "node --test tests/js/contacts_manager.test.mjs"
|
||||
},
|
||||
|
||||
@@ -0,0 +1,717 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Tests for the web viewer dashboard: rollup service, snapshot API, and the
|
||||
performance floor of the legacy /api/stats payload.
|
||||
|
||||
The perf test seeds a synthetic database roughly the shape of a live install
|
||||
(~100k rows across the tables the dashboard reads) so the request-path cost is
|
||||
regression-locked rather than eyeballed.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import sqlite3
|
||||
import time
|
||||
from configparser import ConfigParser
|
||||
from contextlib import closing
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
|
||||
from modules.web_viewer.app import BotDataViewer
|
||||
from modules.web_viewer.dashboard_stats import (
|
||||
LEASE_KEY,
|
||||
TOP_KINDS,
|
||||
day_bounds,
|
||||
local_date_str,
|
||||
normalize_role,
|
||||
)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Synthetic database
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
CONTACTS = 2000
|
||||
OBSERVED_PATHS = 20000
|
||||
PACKETS = 40000
|
||||
MESSAGES = 20000
|
||||
COMMANDS = 5000
|
||||
PATHS = 5000
|
||||
DAILY_STATS_DAYS = 90
|
||||
DAILY_STATS_NODES = 200
|
||||
|
||||
# Only the first ADVERTISING_CONTACTS contacts appear in observed_paths, so the
|
||||
# rest fall through the cheap public-key check into the hop-prefix matcher —
|
||||
# the quadratic step this suite exists to keep out.
|
||||
ADVERTISING_CONTACTS = 500
|
||||
|
||||
|
||||
def _pk(i: int) -> str:
|
||||
"""Deterministic 64-hex-char public key for row *i*."""
|
||||
return f"{i:064x}"
|
||||
|
||||
|
||||
def _create_stats_tables(conn: sqlite3.Connection) -> None:
|
||||
"""Create the three tables stats_command owns (migrations do not)."""
|
||||
conn.executescript(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS message_stats (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
timestamp INTEGER NOT NULL,
|
||||
sender_id TEXT NOT NULL,
|
||||
channel TEXT,
|
||||
content TEXT NOT NULL,
|
||||
is_dm BOOLEAN NOT NULL,
|
||||
hops INTEGER,
|
||||
snr REAL,
|
||||
rssi INTEGER,
|
||||
path TEXT,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS command_stats (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
timestamp INTEGER NOT NULL,
|
||||
sender_id TEXT NOT NULL,
|
||||
command_name TEXT NOT NULL,
|
||||
channel TEXT,
|
||||
is_dm BOOLEAN NOT NULL,
|
||||
response_sent BOOLEAN NOT NULL,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS path_stats (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
timestamp INTEGER NOT NULL,
|
||||
sender_id TEXT NOT NULL,
|
||||
channel TEXT,
|
||||
path_length INTEGER NOT NULL,
|
||||
path_string TEXT NOT NULL,
|
||||
hops INTEGER,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_message_timestamp ON message_stats(timestamp);
|
||||
CREATE INDEX IF NOT EXISTS idx_command_timestamp ON command_stats(timestamp);
|
||||
"""
|
||||
)
|
||||
|
||||
|
||||
def seed_database(db_path: str, now: float | None = None) -> None:
|
||||
"""Populate *db_path* with a synthetic mesh roughly the shape of a live install."""
|
||||
now = time.time() if now is None else now
|
||||
roles = ["repeater", "roomserver", "companion", "sensor", "type3"]
|
||||
devices = ["Repeater", "Companion", "RoomServer", "Sensor"]
|
||||
|
||||
with sqlite3.connect(db_path, timeout=60) as conn:
|
||||
_create_stats_tables(conn)
|
||||
|
||||
conn.executemany(
|
||||
"""
|
||||
INSERT INTO complete_contact_tracking
|
||||
(public_key, name, role, device_type, first_heard, last_heard,
|
||||
advert_count, snr, signal_strength, hop_count, is_currently_tracked,
|
||||
out_bytes_per_hop, city, state, country)
|
||||
VALUES (?, ?, ?, ?, datetime('now','localtime','-40 days'),
|
||||
datetime('now','localtime', ?), ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
""",
|
||||
[
|
||||
(
|
||||
_pk(i),
|
||||
f"node-{i}",
|
||||
roles[i % len(roles)],
|
||||
devices[i % len(devices)],
|
||||
f"-{i % 10} days",
|
||||
1 + (i % 50),
|
||||
-10.0 + (i % 30),
|
||||
-120.0 + (i % 60),
|
||||
i % 6,
|
||||
1 if i % 3 == 0 else 0,
|
||||
3 if i % 4 == 0 else 1,
|
||||
f"city-{i % 90}",
|
||||
f"state-{i % 14}",
|
||||
"United States" if i % 4 else "Canada",
|
||||
)
|
||||
for i in range(CONTACTS)
|
||||
],
|
||||
)
|
||||
|
||||
conn.executemany(
|
||||
"""
|
||||
INSERT INTO observed_paths
|
||||
(public_key, from_prefix, to_prefix, path_hex, path_length,
|
||||
bytes_per_hop, packet_type, first_seen, last_seen, observation_count)
|
||||
VALUES (?, ?, ?, ?, ?, ?, 'advert',
|
||||
datetime('now','localtime','-20 days'),
|
||||
datetime('now','localtime', ?), ?)
|
||||
""",
|
||||
[
|
||||
(
|
||||
_pk(i % ADVERTISING_CONTACTS),
|
||||
f"{i % 65536:04x}",
|
||||
f"{(i * 7) % 65536:04x}",
|
||||
f"{i % 65536:04x}{(i * 7) % 65536:04x}",
|
||||
2,
|
||||
2 if i % 5 else 1,
|
||||
f"-{i % 12} days",
|
||||
1 + (i % 9),
|
||||
)
|
||||
for i in range(OBSERVED_PATHS)
|
||||
],
|
||||
)
|
||||
|
||||
conn.executemany(
|
||||
"INSERT INTO packet_stream (timestamp, data, type) VALUES (?, ?, 'packet')",
|
||||
[
|
||||
(
|
||||
now - (i % (3 * 86400)),
|
||||
json.dumps(
|
||||
{
|
||||
"bytes_per_hop": 2 if i % 4 else 1,
|
||||
"route_type_name": "FLOOD" if i % 3 else "DIRECT",
|
||||
"payload_type_name": "ADVERT" if i % 2 else "TXT_MSG",
|
||||
"path_len": i % 5,
|
||||
"payload_hex": "ab" * 40,
|
||||
}
|
||||
),
|
||||
)
|
||||
for i in range(PACKETS)
|
||||
],
|
||||
)
|
||||
|
||||
conn.executemany(
|
||||
"""
|
||||
INSERT INTO message_stats
|
||||
(timestamp, sender_id, channel, content, is_dm, hops, snr, rssi)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
|
||||
""",
|
||||
[
|
||||
(
|
||||
int(now - (i % (7 * 86400))),
|
||||
f"user-{i % 120}",
|
||||
None if i % 5 == 0 else f"channel-{i % 4}",
|
||||
f"message {i}",
|
||||
1 if i % 5 == 0 else 0,
|
||||
i % 5,
|
||||
-5.0 + (i % 20),
|
||||
-110 + (i % 50),
|
||||
)
|
||||
for i in range(MESSAGES)
|
||||
],
|
||||
)
|
||||
|
||||
conn.executemany(
|
||||
"""
|
||||
INSERT INTO command_stats
|
||||
(timestamp, sender_id, command_name, channel, is_dm, response_sent)
|
||||
VALUES (?, ?, ?, ?, ?, ?)
|
||||
""",
|
||||
[
|
||||
(
|
||||
int(now - (i % (7 * 86400))),
|
||||
f"user-{i % 60}",
|
||||
f"cmd{i % 12}",
|
||||
f"channel-{i % 4}",
|
||||
i % 2,
|
||||
0 if i % 20 == 0 else 1,
|
||||
)
|
||||
for i in range(COMMANDS)
|
||||
],
|
||||
)
|
||||
|
||||
conn.executemany(
|
||||
"""
|
||||
INSERT INTO path_stats
|
||||
(timestamp, sender_id, channel, path_length, path_string, hops)
|
||||
VALUES (?, ?, ?, ?, ?, ?)
|
||||
""",
|
||||
[
|
||||
(
|
||||
int(now - (i % (7 * 86400))),
|
||||
f"user-{i % 60}",
|
||||
f"channel-{i % 4}",
|
||||
i % 8,
|
||||
",".join(f"{(i + h) % 256:02x}" for h in range(i % 8)),
|
||||
i % 8,
|
||||
)
|
||||
for i in range(PATHS)
|
||||
],
|
||||
)
|
||||
|
||||
conn.executemany(
|
||||
"""
|
||||
INSERT INTO daily_stats (date, public_key, advert_count)
|
||||
VALUES (date('now','localtime', ?), ?, ?)
|
||||
""",
|
||||
[
|
||||
(f"-{day} days", _pk(node), 1 + ((day * node) % 20))
|
||||
for day in range(DAILY_STATS_DAYS)
|
||||
for node in range(DAILY_STATS_NODES)
|
||||
],
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
|
||||
def _fake_setup_logging(self: BotDataViewer) -> None:
|
||||
self.logger = logging.getLogger("test_dashboard_stats")
|
||||
self.logger.setLevel(logging.DEBUG)
|
||||
if not self.logger.handlers:
|
||||
self.logger.addHandler(logging.NullHandler())
|
||||
self.logger.propagate = False
|
||||
|
||||
|
||||
def build_viewer(tmp_path, extra_web_viewer: dict[str, str] | None = None) -> BotDataViewer:
|
||||
"""Create a BotDataViewer over a fresh temp DB with all background threads off."""
|
||||
db_path = str(tmp_path / "meshcore_bot.db")
|
||||
config_path = str(tmp_path / "config.ini")
|
||||
|
||||
config = ConfigParser()
|
||||
config.add_section("Bot")
|
||||
config.set("Bot", "db_path", db_path)
|
||||
config.add_section("Web_Viewer")
|
||||
config.set("Web_Viewer", "enabled", "false")
|
||||
config.set("Web_Viewer", "web_viewer_password", "")
|
||||
for key, value in (extra_web_viewer or {}).items():
|
||||
config.set("Web_Viewer", key, value)
|
||||
with open(config_path, "w") as handle:
|
||||
config.write(handle)
|
||||
|
||||
with (
|
||||
patch.object(BotDataViewer, "_setup_logging", _fake_setup_logging),
|
||||
patch.object(BotDataViewer, "_start_database_polling", lambda self: None),
|
||||
patch.object(BotDataViewer, "_start_log_tailing", lambda self: None),
|
||||
patch.object(BotDataViewer, "_start_cleanup_scheduler", lambda self: None),
|
||||
patch.object(BotDataViewer, "_start_dashboard_refresher", lambda self: None),
|
||||
patch.object(BotDataViewer, "_setup_socketio_handlers", lambda self: None),
|
||||
patch("modules.web_viewer.app.RepeaterManager"),
|
||||
):
|
||||
viewer = BotDataViewer(db_path=db_path, config_path=config_path)
|
||||
|
||||
viewer.app.config["TESTING"] = True
|
||||
return viewer
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def seeded_viewer(tmp_path_factory):
|
||||
"""Module-scoped viewer over a ~100k-row synthetic database."""
|
||||
tmp_path = tmp_path_factory.mktemp("dashboard_perf")
|
||||
viewer = build_viewer(tmp_path)
|
||||
seed_database(viewer.db_path)
|
||||
with sqlite3.connect(viewer.db_path, timeout=60) as conn:
|
||||
conn.execute("ANALYZE")
|
||||
yield viewer
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Performance floor
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestStatsPerformance:
|
||||
"""Regression lock on the request-path cost of the dashboard payload.
|
||||
|
||||
Thresholds are deliberately loose relative to measured times (~10x headroom)
|
||||
so they fail on an algorithmic regression, not on a slow CI runner.
|
||||
"""
|
||||
|
||||
def test_api_stats_under_threshold(self, seeded_viewer):
|
||||
"""~35ms after the rework, against ~264ms for the linear hop-prefix scan."""
|
||||
with seeded_viewer.app.test_client() as client:
|
||||
client.get("/api/stats") # warm the page cache
|
||||
started = time.perf_counter()
|
||||
response = client.get("/api/stats")
|
||||
elapsed = time.perf_counter() - started
|
||||
|
||||
assert response.status_code == 200
|
||||
payload = response.get_json()
|
||||
assert "error" not in payload
|
||||
assert payload["total_contacts"] == CONTACTS
|
||||
assert elapsed < 0.2, f"/api/stats took {elapsed:.2f}s on a synthetic 100k-row DB"
|
||||
|
||||
def test_dashboard_summary_is_a_single_row_read(self, seeded_viewer):
|
||||
with closing(seeded_viewer._dashboard_connection()) as conn:
|
||||
seeded_viewer.dashboard_stats.refresh(conn)
|
||||
with seeded_viewer.app.test_client() as client:
|
||||
client.get("/api/dashboard/summary")
|
||||
started = time.perf_counter()
|
||||
response = client.get("/api/dashboard/summary")
|
||||
elapsed = time.perf_counter() - started
|
||||
|
||||
assert response.status_code == 200
|
||||
assert elapsed < 0.05, f"/api/dashboard/summary took {elapsed * 1000:.0f}ms"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Rollup correctness
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def viewer(tmp_path):
|
||||
"""Function-scoped viewer over an empty migrated database."""
|
||||
return build_viewer(tmp_path)
|
||||
|
||||
|
||||
def _refresh(viewer, **kwargs) -> dict:
|
||||
with closing(viewer._dashboard_connection()) as conn:
|
||||
return viewer.dashboard_stats.refresh(conn, **kwargs)
|
||||
|
||||
|
||||
def _rollup(viewer, date_str: str) -> dict | None:
|
||||
with closing(viewer._dashboard_connection()) as conn:
|
||||
row = conn.execute("SELECT * FROM daily_rollup WHERE date = ?", (date_str,)).fetchone()
|
||||
return dict(row) if row else None
|
||||
|
||||
|
||||
class TestRollupCorrectness:
|
||||
|
||||
def test_backfilled_days_are_null_not_zero(self, viewer):
|
||||
"""A pruned source must read as a gap; zero would look like an outage."""
|
||||
seed_database(viewer.db_path)
|
||||
_refresh(viewer)
|
||||
|
||||
old_date = (datetime.now() - timedelta(days=60)).strftime("%Y-%m-%d")
|
||||
row = _rollup(viewer, old_date)
|
||||
assert row is not None
|
||||
assert row["is_backfilled"] == 1
|
||||
# daily_stats survives 90 days, so adverts are real...
|
||||
assert row["adverts_total"] > 0
|
||||
# ...while the 7-day and 3-day sources are unrecoverable, not empty.
|
||||
for column in ("messages_total", "commands_total", "path_obs_total", "packets_total"):
|
||||
assert row[column] is None, f"{column} should be NULL for a backfilled day"
|
||||
|
||||
def test_future_timestamps_do_not_create_rollup_rows(self, viewer):
|
||||
"""Rows dated 2103 exist in the wild; an unclamped bound charts them."""
|
||||
seed_database(viewer.db_path)
|
||||
with sqlite3.connect(viewer.db_path) as conn:
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO message_stats (timestamp, sender_id, channel, content, is_dm)
|
||||
VALUES (?, 'ghost', 'general', 'from the future', 0)
|
||||
""",
|
||||
(int(datetime(2103, 8, 15).timestamp()),),
|
||||
)
|
||||
_refresh(viewer)
|
||||
|
||||
with closing(viewer._dashboard_connection()) as conn:
|
||||
latest = conn.execute("SELECT MAX(date) FROM daily_rollup").fetchone()[0]
|
||||
assert latest <= local_date_str()
|
||||
|
||||
def test_today_excludes_future_rows_from_its_count(self, viewer):
|
||||
seed_database(viewer.db_path)
|
||||
today = local_date_str()
|
||||
before = _refresh(viewer) and _rollup(viewer, today)["messages_total"]
|
||||
|
||||
with sqlite3.connect(viewer.db_path) as conn:
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO message_stats (timestamp, sender_id, channel, content, is_dm)
|
||||
VALUES (?, 'ghost', 'general', 'later today but not yet', 0)
|
||||
""",
|
||||
(int(time.time()) + 3600,),
|
||||
)
|
||||
_refresh(viewer)
|
||||
assert _rollup(viewer, today)["messages_total"] == before
|
||||
|
||||
def test_refresh_is_idempotent(self, viewer):
|
||||
seed_database(viewer.db_path)
|
||||
# Clear the packet backfill backlog first: while it drains, successive
|
||||
# refreshes legitimately see more dimensioned rows than the last.
|
||||
viewer.dashboard_stats.packet_backfill_rows = PACKETS
|
||||
_refresh(viewer)
|
||||
first = _rollup(viewer, local_date_str())
|
||||
_refresh(viewer)
|
||||
second = _rollup(viewer, local_date_str())
|
||||
for column, value in first.items():
|
||||
if column == "computed_at":
|
||||
continue
|
||||
assert second[column] == value, column
|
||||
|
||||
def test_signal_stored_as_sums_and_counts(self, viewer):
|
||||
"""Means cannot be re-aggregated across a window; sums and counts can."""
|
||||
seed_database(viewer.db_path)
|
||||
_refresh(viewer)
|
||||
row = _rollup(viewer, local_date_str())
|
||||
assert row["snr_count"] == MESSAGES
|
||||
assert row["rssi_count"] == MESSAGES
|
||||
with sqlite3.connect(viewer.db_path) as conn:
|
||||
expected = conn.execute("SELECT SUM(snr) FROM message_stats").fetchone()[0]
|
||||
assert row["snr_sum"] == pytest.approx(expected)
|
||||
|
||||
def test_missing_stats_tables_degrade_to_null(self, viewer):
|
||||
"""`collect_stats = false` leaves the tables absent entirely."""
|
||||
seed_database(viewer.db_path)
|
||||
with sqlite3.connect(viewer.db_path) as conn:
|
||||
for table in ("message_stats", "command_stats", "path_stats"):
|
||||
conn.execute(f"DROP TABLE {table}")
|
||||
_refresh(viewer)
|
||||
|
||||
row = _rollup(viewer, local_date_str())
|
||||
assert row["messages_total"] is None
|
||||
assert row["commands_total"] is None
|
||||
assert row["adverts_total"] > 0 # unaffected sources still populate
|
||||
|
||||
with viewer.app.test_client() as client:
|
||||
payload = client.get("/api/dashboard/summary").get_json()
|
||||
assert "message_stats" not in payload["coverage"]["sources_present"]
|
||||
assert payload["bot"] == {}
|
||||
|
||||
def test_local_date_attribution_under_non_utc_timezone(self, viewer, monkeypatch):
|
||||
"""The rollup keys on local dates while message_stats stores UTC epochs.
|
||||
|
||||
Get this wrong and the message series is offset from the advert series
|
||||
by several hours of traffic — silently, and for months.
|
||||
"""
|
||||
monkeypatch.setenv("TZ", "America/Los_Angeles")
|
||||
time.tzset()
|
||||
try:
|
||||
# 03:00 UTC is 20:00 the previous day in Los Angeles.
|
||||
utc_moment = datetime(2026, 7, 29, 3, 0, tzinfo=UTC)
|
||||
local_date = utc_moment.astimezone().strftime("%Y-%m-%d")
|
||||
assert local_date == "2026-07-28", "fixture assumes a UTC-7/8 offset"
|
||||
|
||||
start, end = day_bounds(local_date)
|
||||
assert start <= utc_moment.timestamp() < end
|
||||
|
||||
with sqlite3.connect(viewer.db_path) as conn:
|
||||
_create_stats_tables(conn)
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO message_stats (timestamp, sender_id, channel, content, is_dm)
|
||||
VALUES (?, 'tz-probe', 'general', 'evening message', 0)
|
||||
""",
|
||||
(int(utc_moment.timestamp()),),
|
||||
)
|
||||
|
||||
with closing(viewer._dashboard_connection()) as conn:
|
||||
computed = viewer.dashboard_stats.compute_day(
|
||||
conn,
|
||||
local_date,
|
||||
sources=viewer.dashboard_stats.detect_sources(conn),
|
||||
today=local_date,
|
||||
first_seen={},
|
||||
multibyte_keys=None,
|
||||
)
|
||||
assert computed["values"]["messages_total"] == 1
|
||||
finally:
|
||||
monkeypatch.undo()
|
||||
time.tzset()
|
||||
|
||||
|
||||
class TestPacketDimensions:
|
||||
|
||||
def test_column_counts_match_json_extract(self, viewer):
|
||||
"""The denormalized columns must agree with the JSON they replace."""
|
||||
seed_database(viewer.db_path)
|
||||
viewer.dashboard_stats.packet_backfill_rows = PACKETS
|
||||
_refresh(viewer)
|
||||
|
||||
with sqlite3.connect(viewer.db_path) as conn:
|
||||
from_column = conn.execute(
|
||||
"SELECT COUNT(*) FROM packet_stream "
|
||||
"WHERE type='packet' AND bytes_per_hop IN (2, 3)"
|
||||
).fetchone()[0]
|
||||
from_json = conn.execute(
|
||||
"SELECT COUNT(*) FROM packet_stream WHERE type='packet' "
|
||||
"AND CAST(json_extract(data, '$.bytes_per_hop') AS INTEGER) IN (2, 3)"
|
||||
).fetchone()[0]
|
||||
flood_column = conn.execute(
|
||||
"SELECT COUNT(*) FROM packet_stream "
|
||||
"WHERE type='packet' AND route_type_name LIKE '%FLOOD'"
|
||||
).fetchone()[0]
|
||||
flood_json = conn.execute(
|
||||
"SELECT COUNT(*) FROM packet_stream WHERE type='packet' "
|
||||
"AND json_extract(data, '$.route_type_name') LIKE '%FLOOD'"
|
||||
).fetchone()[0]
|
||||
|
||||
assert from_column == from_json
|
||||
assert flood_column == flood_json
|
||||
|
||||
def test_backfill_advances_and_terminates(self, viewer):
|
||||
"""Rows whose JSON has no route type must still be marked processed."""
|
||||
with sqlite3.connect(viewer.db_path) as conn:
|
||||
conn.executemany(
|
||||
"INSERT INTO packet_stream (timestamp, data, type) VALUES (?, ?, 'packet')",
|
||||
[(time.time(), json.dumps({"hops": 1}), ) for _ in range(10)],
|
||||
)
|
||||
viewer.dashboard_stats.packet_backfill_rows = 100
|
||||
with closing(viewer._dashboard_connection()) as conn:
|
||||
first = viewer.dashboard_stats.backfill_packet_dims(conn)
|
||||
second = viewer.dashboard_stats.backfill_packet_dims(conn)
|
||||
assert first == 10
|
||||
assert second == 0, "rows without a route type were reselected forever"
|
||||
|
||||
def test_backlog_probe_uses_the_worklist_index(self, viewer):
|
||||
"""The 'anything left to dimension?' probe must not scan the table.
|
||||
|
||||
Finding nothing still means reading everything without this index, and
|
||||
that cost is paid on every tick *after* the backfill completes — 4.6s
|
||||
per minute forever on a 180MB packet_stream.
|
||||
"""
|
||||
with sqlite3.connect(viewer.db_path) as conn:
|
||||
plan = conn.execute(
|
||||
"EXPLAIN QUERY PLAN "
|
||||
"SELECT id FROM packet_stream "
|
||||
"WHERE type = 'packet' AND route_type_name IS NULL "
|
||||
"ORDER BY id DESC LIMIT 2000"
|
||||
).fetchall()
|
||||
assert any("idx_packet_stream_undimensioned" in str(step) for step in plan), plan
|
||||
|
||||
def test_coverage_reports_the_measured_window(self, viewer):
|
||||
"""The old page labelled a 3-day table 'last 7 days'."""
|
||||
now = time.time()
|
||||
with sqlite3.connect(viewer.db_path) as conn:
|
||||
conn.executemany(
|
||||
"INSERT INTO packet_stream (timestamp, data, type, route_type_name) "
|
||||
"VALUES (?, '{}', 'packet', 'FLOOD')",
|
||||
[(now - 2.25 * 86400,), (now - 60,)],
|
||||
)
|
||||
with closing(viewer._dashboard_connection()) as conn:
|
||||
coverage = viewer.dashboard_stats.packet_coverage(conn)
|
||||
assert coverage["packets_window_label"] == "last 2d 6h"
|
||||
assert coverage["packets_with_dims"] == 2
|
||||
|
||||
|
||||
class TestDerivedWindows:
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("retention", "expected"),
|
||||
[
|
||||
(7, ["24h", "7d", "all"]),
|
||||
(30, ["24h", "7d", "30d", "all"]),
|
||||
(1, ["24h", "all"]),
|
||||
],
|
||||
)
|
||||
def test_options_never_outrun_retention(self, tmp_path, retention, expected):
|
||||
viewer = build_viewer(tmp_path)
|
||||
viewer.dashboard_stats.stats_retention_days = retention
|
||||
windows = viewer.dashboard_stats.derive_windows()
|
||||
assert [o["value"] for o in windows["windows"]["users"]] == expected
|
||||
assert windows["windows"]["users"][-1]["label"] == f"All retained ({retention}d)"
|
||||
|
||||
|
||||
class TestRoleBucketing:
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("raw", "expected"),
|
||||
[
|
||||
("repeater", "repeater"),
|
||||
("RoomServer", "roomserver"),
|
||||
("type0", "Unknown"),
|
||||
("type15", "Unknown"),
|
||||
("", "Unknown"),
|
||||
(None, "Unknown"),
|
||||
("typewriter", "typewriter"),
|
||||
],
|
||||
)
|
||||
def test_unmapped_ordinals_fold_into_unknown(self, raw, expected):
|
||||
assert normalize_role(raw) == expected
|
||||
|
||||
|
||||
class TestSnapshotLease:
|
||||
|
||||
def test_second_holder_is_refused_until_expiry(self, viewer):
|
||||
with closing(viewer._dashboard_connection()) as conn:
|
||||
assert viewer.dashboard_stats.try_claim_lease(conn) is True
|
||||
# Same process reclaims its own lease; a different owner does not.
|
||||
assert viewer.dashboard_stats.try_claim_lease(conn) is True
|
||||
conn.execute(
|
||||
"UPDATE bot_metadata SET value = ? WHERE key = ?",
|
||||
(f"999999:other-host:{time.time() + 600:.0f}", LEASE_KEY),
|
||||
)
|
||||
assert viewer.dashboard_stats.try_claim_lease(conn) is False
|
||||
|
||||
def test_expired_lease_is_reclaimed(self, viewer):
|
||||
with closing(viewer._dashboard_connection()) as conn:
|
||||
viewer.dashboard_stats.try_claim_lease(conn)
|
||||
conn.execute(
|
||||
"UPDATE bot_metadata SET value = ? WHERE key = ?",
|
||||
(f"999999:other-host:{time.time() - 10:.0f}", LEASE_KEY),
|
||||
)
|
||||
assert viewer.dashboard_stats.try_claim_lease(conn) is True
|
||||
|
||||
def test_concurrent_refreshes_agree(self, viewer):
|
||||
"""A double refresh wastes CPU but must not corrupt anything."""
|
||||
seed_database(viewer.db_path)
|
||||
viewer.dashboard_stats.packet_backfill_rows = PACKETS
|
||||
_refresh(viewer)
|
||||
first = _rollup(viewer, local_date_str())
|
||||
_refresh(viewer) # simulates a second viewer that won the race
|
||||
second = _rollup(viewer, local_date_str())
|
||||
assert {k: v for k, v in first.items() if k != "computed_at"} == {
|
||||
k: v for k, v in second.items() if k != "computed_at"
|
||||
}
|
||||
|
||||
|
||||
class TestDashboardApi:
|
||||
|
||||
def test_summary_returns_503_before_first_refresh(self, viewer):
|
||||
with viewer.app.test_client() as client:
|
||||
response = client.get("/api/dashboard/summary")
|
||||
assert response.status_code == 503
|
||||
assert response.get_json()["pending"] is True
|
||||
|
||||
def test_summary_etag_yields_304(self, viewer):
|
||||
seed_database(viewer.db_path)
|
||||
_refresh(viewer)
|
||||
with viewer.app.test_client() as client:
|
||||
first = client.get("/api/dashboard/summary")
|
||||
second = client.get(
|
||||
"/api/dashboard/summary",
|
||||
headers={"If-None-Match": first.headers["ETag"]},
|
||||
)
|
||||
assert first.status_code == 200
|
||||
assert second.status_code == 304
|
||||
assert second.data == b""
|
||||
|
||||
def test_series_gaps_survive_as_null(self, viewer):
|
||||
seed_database(viewer.db_path)
|
||||
_refresh(viewer)
|
||||
with viewer.app.test_client() as client:
|
||||
payload = client.get("/api/dashboard/series?metric=messages&days=60").get_json()
|
||||
values = [point["value"] for point in payload["points"]]
|
||||
assert None in values, "backfilled days must stay null, not become zero"
|
||||
assert any(v is not None for v in values)
|
||||
|
||||
def test_ratio_series_averages_rather_than_sums(self, viewer):
|
||||
seed_database(viewer.db_path)
|
||||
_refresh(viewer)
|
||||
with viewer.app.test_client() as client:
|
||||
payload = client.get(
|
||||
"/api/dashboard/series?metric=multibyte_share&days=30"
|
||||
).get_json()
|
||||
assert payload["is_ratio"] is True
|
||||
if payload["current_period_total"] is not None:
|
||||
assert 0 <= payload["current_period_total"] <= 100
|
||||
|
||||
def test_unknown_metric_and_kind_rejected(self, viewer):
|
||||
with viewer.app.test_client() as client:
|
||||
assert client.get("/api/dashboard/series?metric=drop%20table").status_code == 400
|
||||
assert client.get("/api/dashboard/top?kind=drop%20table").status_code == 400
|
||||
|
||||
@pytest.mark.parametrize("kind", list(TOP_KINDS))
|
||||
def test_top_endpoints_return_items(self, viewer, kind):
|
||||
seed_database(viewer.db_path)
|
||||
with viewer.app.test_client() as client:
|
||||
payload = client.get(f"/api/dashboard/top?kind={kind}&window=7d&limit=5").get_json()
|
||||
assert payload["kind"] == kind
|
||||
assert len(payload["items"]) <= 5
|
||||
assert payload["window_label"] == "Last 7 days"
|
||||
|
||||
def test_manual_refresh_advances_generated_at(self, viewer):
|
||||
seed_database(viewer.db_path)
|
||||
_refresh(viewer)
|
||||
with viewer.app.test_client() as client:
|
||||
before = client.get("/api/dashboard/summary").get_json()["generated_at"]
|
||||
posted = client.post(
|
||||
"/api/dashboard/refresh",
|
||||
headers={"X-Requested-With": "XMLHttpRequest"},
|
||||
)
|
||||
after = client.get("/api/dashboard/summary").get_json()["generated_at"]
|
||||
assert posted.status_code == 200
|
||||
assert after > before
|
||||
|
||||
def test_stats_shim_is_marked_deprecated(self, viewer):
|
||||
with viewer.app.test_client() as client:
|
||||
response = client.get("/api/stats")
|
||||
assert response.status_code == 200
|
||||
assert response.headers["Deprecation"] == "true"
|
||||
assert "Sunset" in response.headers
|
||||
assert response.get_json()["deprecated"] is True
|
||||
@@ -305,6 +305,7 @@ def viewer(tmp_path):
|
||||
with patch.object(BotDataViewer, "_start_database_polling"), \
|
||||
patch.object(BotDataViewer, "_start_log_tailing"), \
|
||||
patch.object(BotDataViewer, "_start_cleanup_scheduler"), \
|
||||
patch.object(BotDataViewer, "_start_dashboard_refresher"), \
|
||||
patch.object(BotDataViewer, "_setup_socketio_handlers"), \
|
||||
patch("modules.web_viewer.app.RepeaterManager"):
|
||||
v = BotDataViewer(db_path=str(tmp_path / "meshcore_bot.db"), config_path=config_path)
|
||||
|
||||
@@ -65,6 +65,7 @@ def viewer(tmp_path_factory):
|
||||
patch.object(BotDataViewer, "_start_database_polling", lambda self: None),
|
||||
patch.object(BotDataViewer, "_start_log_tailing", lambda self: None),
|
||||
patch.object(BotDataViewer, "_start_cleanup_scheduler", lambda self: None),
|
||||
patch.object(BotDataViewer, "_start_dashboard_refresher", lambda self: None),
|
||||
):
|
||||
v = BotDataViewer(db_path=db_path, config_path=config_path)
|
||||
|
||||
@@ -103,6 +104,7 @@ def auth_viewer(tmp_path_factory):
|
||||
patch.object(BotDataViewer, "_start_database_polling", lambda self: None),
|
||||
patch.object(BotDataViewer, "_start_log_tailing", lambda self: None),
|
||||
patch.object(BotDataViewer, "_start_cleanup_scheduler", lambda self: None),
|
||||
patch.object(BotDataViewer, "_start_dashboard_refresher", lambda self: None),
|
||||
):
|
||||
v = BotDataViewer(db_path=db_path, config_path=config_path)
|
||||
|
||||
@@ -171,14 +173,24 @@ class TestPageRoutes:
|
||||
# Scroll buttons
|
||||
assert 'id="live-scroll-top"' in html
|
||||
assert 'id="live-scroll-bottom"' in html
|
||||
assert 'scrollLiveFeed' in html
|
||||
# Filter checkboxes with data-type attributes
|
||||
assert 'data-type="packet"' in html
|
||||
assert 'data-type="command"' in html
|
||||
assert 'data-type="message"' in html
|
||||
assert 'live-filter-cb' in html
|
||||
# [#channel] prefix logic present in JS
|
||||
assert 'applyFilters' in html
|
||||
# Behaviour lives in the external, nonce-free dashboard script
|
||||
assert 'js/dashboard.js' in html
|
||||
|
||||
def test_dashboard_script_wires_live_feed_controls(self):
|
||||
"""The extracted dashboard script still implements the feed behaviour."""
|
||||
script = (
|
||||
Path(__file__).resolve().parents[1]
|
||||
/ "modules" / "web_viewer" / "static" / "js" / "dashboard.js"
|
||||
).read_text(encoding="utf-8")
|
||||
assert "applyFilters" in script
|
||||
assert "live-scroll-top" in script
|
||||
assert "live-scroll-bottom" in script
|
||||
assert "initLiveFeed" in script
|
||||
|
||||
def test_realtime(self, client):
|
||||
resp = client.get("/realtime")
|
||||
@@ -223,9 +235,10 @@ class TestPageRoutes:
|
||||
assert resp.status_code == 302
|
||||
assert resp.headers["Location"].endswith("/config#database")
|
||||
|
||||
def test_stats(self, client):
|
||||
def test_stats_page_removed(self, client):
|
||||
"""The orphaned /stats stub page was folded into the dashboard."""
|
||||
resp = client.get("/stats")
|
||||
assert resp.status_code == 200
|
||||
assert resp.status_code == 404
|
||||
|
||||
def test_greeter(self, client):
|
||||
resp = client.get("/greeter")
|
||||
@@ -2111,6 +2124,7 @@ def socketio_viewer(tmp_path_factory):
|
||||
_patch.object(BotDataViewer, "_start_database_polling", lambda self: None),
|
||||
_patch.object(BotDataViewer, "_start_log_tailing", lambda self: None),
|
||||
_patch.object(BotDataViewer, "_start_cleanup_scheduler", lambda self: None),
|
||||
_patch.object(BotDataViewer, "_start_dashboard_refresher", lambda self: None),
|
||||
):
|
||||
v = BotDataViewer(db_path=db_path, config_path=config_path)
|
||||
|
||||
@@ -2325,7 +2339,7 @@ class TestConnectedClientsApi:
|
||||
html = resp.data.decode()
|
||||
assert "connectedClientsModal" in html
|
||||
assert "connected-clients-table" in html
|
||||
assert "loadConnectedClients" in html
|
||||
assert 'id="health-clients-link"' in html
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -3491,6 +3505,7 @@ class TestDbPathResolutionFromConfigDir:
|
||||
patch.object(BotDataViewer, "_start_database_polling", lambda self: None),
|
||||
patch.object(BotDataViewer, "_start_log_tailing", lambda self: None),
|
||||
patch.object(BotDataViewer, "_start_cleanup_scheduler", lambda self: None),
|
||||
patch.object(BotDataViewer, "_start_dashboard_refresher", lambda self: None),
|
||||
):
|
||||
return BotDataViewer(config_path=config_path)
|
||||
|
||||
@@ -3573,6 +3588,7 @@ class TestWebViewerLoggingRespectsLogFile:
|
||||
patch.object(BotDataViewer, "_start_database_polling", lambda self: None),
|
||||
patch.object(BotDataViewer, "_start_log_tailing", lambda self: None),
|
||||
patch.object(BotDataViewer, "_start_cleanup_scheduler", lambda self: None),
|
||||
patch.object(BotDataViewer, "_start_dashboard_refresher", lambda self: None),
|
||||
):
|
||||
return BotDataViewer(config_path=config_path)
|
||||
|
||||
|
||||
@@ -69,6 +69,7 @@ def viewer_with_db(tmp_path):
|
||||
with patch.object(BotDataViewer, "_start_database_polling"), \
|
||||
patch.object(BotDataViewer, "_start_log_tailing"), \
|
||||
patch.object(BotDataViewer, "_start_cleanup_scheduler"), \
|
||||
patch.object(BotDataViewer, "_start_dashboard_refresher"), \
|
||||
patch.object(BotDataViewer, "_setup_socketio_handlers"), \
|
||||
patch("modules.web_viewer.app.RepeaterManager"):
|
||||
viewer = BotDataViewer(db_path=db_path, config_path=config_path)
|
||||
@@ -123,6 +124,7 @@ def mock_viewer(tmp_path):
|
||||
with patch.object(BotDataViewer, "_start_database_polling"), \
|
||||
patch.object(BotDataViewer, "_start_log_tailing"), \
|
||||
patch.object(BotDataViewer, "_start_cleanup_scheduler"), \
|
||||
patch.object(BotDataViewer, "_start_dashboard_refresher"), \
|
||||
patch.object(BotDataViewer, "_setup_socketio_handlers"), \
|
||||
patch("modules.web_viewer.app.RepeaterManager"):
|
||||
viewer = BotDataViewer(db_path=db_path, config_path=config_path)
|
||||
@@ -150,6 +152,7 @@ class TestAllowedTables:
|
||||
'greeted_users', 'feed_subscriptions', 'feed_activity', 'feed_errors',
|
||||
'path_stats', 'unique_advert_packets', 'schema_version',
|
||||
'channel_operations', 'channels', 'feed_message_queue',
|
||||
'daily_rollup', 'dashboard_snapshot',
|
||||
}
|
||||
assert expected_tables == BotDataViewer.ALLOWED_TABLES
|
||||
|
||||
|
||||
@@ -205,7 +205,7 @@ class TestInsertPacketStreamRow:
|
||||
bi = _make_bot_integration()
|
||||
bi._insert_packet_stream_row('{"x": 1}', "packet")
|
||||
assert not bi._write_queue.empty()
|
||||
ts, data, row_type = bi._write_queue.get_nowait()
|
||||
ts, data, row_type, *_dims = bi._write_queue.get_nowait()
|
||||
assert data == '{"x": 1}'
|
||||
assert row_type == "packet"
|
||||
|
||||
@@ -228,7 +228,7 @@ class TestCaptureFullPacketData:
|
||||
bi = _make_bot_integration()
|
||||
bi.capture_full_packet_data({"snr": -5.0, "path_len": 2})
|
||||
assert not bi._write_queue.empty()
|
||||
ts, data, row_type = bi._write_queue.get_nowait()
|
||||
ts, data, row_type, *_dims = bi._write_queue.get_nowait()
|
||||
parsed = json.loads(data)
|
||||
assert row_type == "packet"
|
||||
assert parsed["hops"] == 2
|
||||
@@ -236,21 +236,21 @@ class TestCaptureFullPacketData:
|
||||
def test_no_path_len_defaults_hops_to_0(self):
|
||||
bi = _make_bot_integration()
|
||||
bi.capture_full_packet_data({"snr": -5.0})
|
||||
ts, data, row_type = bi._write_queue.get_nowait()
|
||||
ts, data, row_type, *_dims = bi._write_queue.get_nowait()
|
||||
parsed = json.loads(data)
|
||||
assert parsed["hops"] == 0
|
||||
|
||||
def test_existing_hops_not_overwritten(self):
|
||||
bi = _make_bot_integration()
|
||||
bi.capture_full_packet_data({"hops": 3, "path_len": 2})
|
||||
ts, data, row_type = bi._write_queue.get_nowait()
|
||||
ts, data, row_type, *_dims = bi._write_queue.get_nowait()
|
||||
parsed = json.loads(data)
|
||||
assert parsed["hops"] == 3
|
||||
|
||||
def test_datetime_added(self):
|
||||
bi = _make_bot_integration()
|
||||
bi.capture_full_packet_data({"snr": 0})
|
||||
ts, data, _ = bi._write_queue.get_nowait()
|
||||
ts, data, _, *_dims = bi._write_queue.get_nowait()
|
||||
parsed = json.loads(data)
|
||||
assert "datetime" in parsed
|
||||
|
||||
@@ -276,7 +276,7 @@ class TestCaptureCommand:
|
||||
msg.content = "ping"
|
||||
bi.capture_command(msg, "ping", "Pong!", True)
|
||||
assert not bi._write_queue.empty()
|
||||
ts, data, row_type = bi._write_queue.get_nowait()
|
||||
ts, data, row_type, *_dims = bi._write_queue.get_nowait()
|
||||
assert row_type == "command"
|
||||
parsed = json.loads(data)
|
||||
assert parsed["command"] == "ping"
|
||||
@@ -290,7 +290,7 @@ class TestCaptureCommand:
|
||||
msg.channel = "ch"
|
||||
msg.content = "cmd"
|
||||
bi.capture_command(msg, "cmd", "resp", True)
|
||||
ts, data, _ = bi._write_queue.get_nowait()
|
||||
ts, data, _, *_dims = bi._write_queue.get_nowait()
|
||||
parsed = json.loads(data)
|
||||
assert parsed["repeat_count"] == 0
|
||||
|
||||
@@ -313,7 +313,7 @@ class TestCaptureChannelMessage:
|
||||
msg.is_dm = False
|
||||
bi.capture_channel_message(msg)
|
||||
assert not bi._write_queue.empty()
|
||||
ts, data, row_type = bi._write_queue.get_nowait()
|
||||
ts, data, row_type, *_dims = bi._write_queue.get_nowait()
|
||||
assert row_type == "message"
|
||||
parsed = json.loads(data)
|
||||
assert parsed["type"] == "message"
|
||||
@@ -330,7 +330,7 @@ class TestCaptureChannelMessage:
|
||||
msg.path = ""
|
||||
msg.is_dm = True
|
||||
bi.capture_channel_message(msg)
|
||||
ts, data, _ = bi._write_queue.get_nowait()
|
||||
ts, data, _, *_dims = bi._write_queue.get_nowait()
|
||||
parsed = json.loads(data)
|
||||
assert parsed["is_dm"] is True
|
||||
|
||||
@@ -345,7 +345,7 @@ class TestCapturePacketRouting:
|
||||
bi = _make_bot_integration()
|
||||
bi.capture_packet_routing({"path_nodes": ["aa", "bb"]})
|
||||
assert not bi._write_queue.empty()
|
||||
ts, data, row_type = bi._write_queue.get_nowait()
|
||||
ts, data, row_type, *_dims = bi._write_queue.get_nowait()
|
||||
assert row_type == "routing"
|
||||
|
||||
|
||||
|
||||
@@ -10,13 +10,19 @@ import pytest
|
||||
|
||||
from modules.web_viewer.app import BotDataViewer
|
||||
|
||||
TEMPLATES = Path(__file__).resolve().parents[1] / "modules" / "web_viewer" / "templates"
|
||||
_WEB_VIEWER = Path(__file__).resolve().parents[1] / "modules" / "web_viewer"
|
||||
TEMPLATES = _WEB_VIEWER / "templates"
|
||||
STATIC = _WEB_VIEWER / "static"
|
||||
|
||||
|
||||
def _source(name: str) -> str:
|
||||
return (TEMPLATES / name).read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def _static_source(name: str) -> str:
|
||||
return (STATIC / name).read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def _fake_setup_logging(viewer: BotDataViewer) -> None:
|
||||
viewer.logger = logging.getLogger("test_web_viewer_xss_hardening")
|
||||
viewer.logger.addHandler(logging.NullHandler())
|
||||
@@ -47,6 +53,7 @@ def client(tmp_path_factory):
|
||||
patch.object(BotDataViewer, "_start_database_polling", lambda self: None),
|
||||
patch.object(BotDataViewer, "_start_log_tailing", lambda self: None),
|
||||
patch.object(BotDataViewer, "_start_cleanup_scheduler", lambda self: None),
|
||||
patch.object(BotDataViewer, "_start_dashboard_refresher", lambda self: None),
|
||||
):
|
||||
viewer = BotDataViewer(db_path=str(db_path), config_path=str(config_path))
|
||||
viewer.app.config["TESTING"] = True
|
||||
@@ -55,19 +62,24 @@ def client(tmp_path_factory):
|
||||
|
||||
|
||||
def test_dashboard_has_no_html_parser_sink_for_api_or_mesh_values() -> None:
|
||||
source = _source("index.html")
|
||||
assert ".innerHTML" not in source
|
||||
# The dashboard's script now lives in a static file (no CSP nonce needed),
|
||||
# so the trust boundary to police moved with it.
|
||||
template = _source("index.html")
|
||||
script = _static_source("js/dashboard.js")
|
||||
assert ".innerHTML" not in template
|
||||
assert ".innerHTML" not in script
|
||||
# Representative attacker-controlled sender, channel/title, path, client,
|
||||
# and error values all terminate in textContent-backed DOM construction.
|
||||
# and role values all terminate in textContent-backed DOM construction.
|
||||
for safe_sink in (
|
||||
"entryLabel.textContent = String(label ?? '')",
|
||||
"entryText.textContent = ` ${String(text ?? '')}`",
|
||||
"channelName.title = String(channel.channel ?? '')",
|
||||
"pathCode = this.makeTextElement('code', path.path_string",
|
||||
"clientCell.appendChild(this.makeTextElement('code', c.client_id))",
|
||||
"errorText.textContent = String(message ?? '')",
|
||||
"node.textContent = String(value ?? '');",
|
||||
"makeTextElement('strong', label)",
|
||||
"makeTextElement('span', ' ' + String(text ?? ''), 'text-muted')",
|
||||
"label.title = String(name ?? '');",
|
||||
"makeTextElement('code', item.path_string, 'small text-break')",
|
||||
"idCell.appendChild(makeTextElement('code', client.client_id))",
|
||||
"makeTextElement('div', name, 'mix-row__name')",
|
||||
):
|
||||
assert safe_sink in source
|
||||
assert safe_sink in script, safe_sink
|
||||
|
||||
|
||||
def test_feed_metadata_preview_details_and_alerts_use_text_nodes() -> None:
|
||||
@@ -162,9 +174,6 @@ def test_legacy_admin_pages_encode_or_text_render_untrusted_values() -> None:
|
||||
assert "header.innerHTML = `" not in plugins
|
||||
assert "description.textContent = String(plugin.description ?? '')" in plugins
|
||||
|
||||
stats = _source("stats.html")
|
||||
assert ".innerHTML" not in stats
|
||||
|
||||
|
||||
def test_nonce_hardened_templates_have_no_inline_event_attributes() -> None:
|
||||
event_attribute = re.compile(r"<[^>]*\son[a-z]+\s*=", re.IGNORECASE)
|
||||
@@ -176,7 +185,6 @@ def test_nonce_hardened_templates_have_no_inline_event_attributes() -> None:
|
||||
"radio.html",
|
||||
"realtime.html",
|
||||
"contacts.html",
|
||||
"stats.html",
|
||||
"plugins.html",
|
||||
"greeter.html",
|
||||
"logs.html",
|
||||
@@ -196,7 +204,6 @@ def test_nonce_hardened_templates_have_no_inline_event_attributes() -> None:
|
||||
"/radio",
|
||||
"/realtime",
|
||||
"/contacts",
|
||||
"/stats",
|
||||
"/plugins",
|
||||
"/greeter",
|
||||
"/logs",
|
||||
@@ -220,10 +227,14 @@ def test_csp_nonce_matches_every_rendered_inline_script(client, path: str) -> No
|
||||
expected_nonce = nonce_match.group(1)
|
||||
|
||||
html = response.get_data(as_text=True)
|
||||
# A block with a non-JavaScript type is a data island, not code: the browser
|
||||
# never executes it, so CSP does not gate it and it needs no nonce.
|
||||
data_block = re.compile(r'\btype\s*=\s*"(?!(?:text/javascript|module)\b)[^"]+"', re.IGNORECASE)
|
||||
inline_scripts = [
|
||||
attributes
|
||||
for attributes in re.findall(r"<script\b([^>]*)>", html, re.IGNORECASE)
|
||||
if not re.search(r"\bsrc\s*=", attributes, re.IGNORECASE)
|
||||
and not data_block.search(attributes)
|
||||
]
|
||||
assert inline_scripts
|
||||
for attributes in inline_scripts:
|
||||
|
||||
Reference in new issue
Block a user