agessaman
7e3ab04b53
The landing page re-ran ~50 aggregate queries five times per load, then repeated the whole sequence every 30 seconds forever — including in backgrounded tabs. Against the live 1.44 GB database that was roughly 20 seconds of SQLite work per page load. Move the work off the request path. A refresher thread in the viewer process (which already runs migrations, so it works for a split-DB install) writes two tables: daily_rollup, one row per local date, and dashboard_snapshot, a single JSON row. A page load now reads one row. Measured on the live database: first paint 6 requests -> 2, /api/dashboard/summary p50 1.1 ms (304 in 0.8 ms), /api/stats 130 ms, and a 0.32 s refresh once a minute in the background. Make the numbers mean what they say: - Window selectors are built from each source's retention. The page offered "30d" and "All" against tables pruned at 7 days, so three of four choices returned the same figure under a label that denied it. - The incoming-packet chart reports its measured window instead of claiming 7 days for a table pruned at 3 — it sat beside a genuine 7-day contacts chart inviting an invalid comparison. - Days with no source data store NULL and render as gaps. Writing 0 would put a fake cliff at every retention boundary. - Signal metrics are stored as sums and counts, never means, so any window re-aggregates correctly. - Delta chips compare the last two complete calendar days and say so; the headline above them is a rolling 24 hours. - Unmapped role ordinals (type0..type15) bucket into "Unknown". - SNR comes from message_stats, where it is populated on every row, not from complete_contact_tracking, where it is populated on 7%. Kill the json_extract scans: packet_stream gains denormalized route_type_name, payload_type_name, path_len and bytes_per_hop, written at capture time. Aggregating those from JSON cost 3-6 s per query. Existing rows convert a bounded batch per tick rather than in one migration that would rewrite ~180 MB into the WAL and stall bot startup. A partial index serves as the backfill worklist — without it the "any rows left?" probe is a full scan costing 4.6 s per tick, and it costs that after the backfill finishes, because finding nothing still means reading everything. Also: replace the per-contact hop-prefix scan with the existing bucketed matcher and memoize the 7-day chunk set (264 ms -> 35 ms on a synthetic 100k-row database, regression-locked by a test); move the dashboard's JS and CSS to static files, which removes the CSP nonce requirement for the bulk of the page; and give cleanup_old_stats a future-timestamp guard, without which rows dated 2103 are never older than the cutoff and so live forever. Deletes the orphaned /stats page, unreachable from the nav and rendering stub charts that never populated. /api/stats stays as a shim with every key name intact plus Deprecation and Sunset headers. All schema changes are additive, so a downgraded codebase can read the data; it would however need the new schema_version rows removed, since MigrationRunner rejects versions it does not know.
4.1 KiB
Data retention
The bot stores data in a SQLite database for the web viewer, stats, repeater management, and path routing. To limit database size, data retention controls how long rows are kept. Cleanup runs daily from the bot’s scheduler, so retention is enforced even when the standalone web viewer is not running.
Configuration
All retention options live in the [Data_Retention] section of config.ini. Example (see config.ini.example for full comments):
[Data_Retention]
packet_stream_retention_days = 3
daily_stats_retention_days = 90
observed_paths_retention_days = 90
purging_log_retention_days = 90
mesh_connections_retention_days = 7
Stats tables (message_stats, command_stats, path_stats) use [Stats_Command] data_retention_days (default 7); the scheduler runs that cleanup daily as well. Stats are collected by default with collect_stats = true under [Stats_Command], even if the user-facing stats chat command is disabled with enabled = false. Set collect_stats = false only if you want to stop writing those dashboard stats tables.
Tables and defaults
| Table / data | Purpose | Default retention |
|---|---|---|
| packet_stream | Real-time packets, commands, routing in the web viewer; transmission_tracker repeat counts | 3 days |
| daily_stats | Daily repeater/advert stats | 90 days |
| unique_advert_packets | Unique packet hashes for advert stats | 90 days (same as daily_stats) |
| observed_paths | Path strings from adverts and messages | 90 days |
| purging_log | Audit trail for repeater purges | 90 days |
| mesh_connections | Path graph edges (in-memory + DB); should be ≥ Path_Command graph_edge_expiration_days |
7 days |
| message_stats, command_stats, path_stats | Stats command data | 7 days ([Stats_Command] data_retention_days) |
| daily_rollup | Per-day dashboard rollups, so trends outlive the tables above | 400 days ([Web_Viewer] dashboard_snapshot_history_days) |
| dashboard_snapshot | Single-row current-state payload for the dashboard | Overwritten in place |
| geocoding_cache, generic_cache | Expired entries removed by scheduler | By expiry time |
daily_rollup exists precisely because the retention above is short: the
dashboard charts 30 days of message and packet activity from tables pruned at 7
and 3 days respectively. One row per day is roughly 200 bytes, so a year of
history costs well under a megabyte. Pruning it is handled by the web viewer's
snapshot refresher rather than the bot scheduler, because the viewer may point
at a different database file.
Note that cleanup also removes rows dated implausibly far in the future. A node with a bad clock can write a timestamp years ahead; because such a row is never older than the cutoff, a lower-bound-only delete would keep it forever and stretch every chart axis to match.
Shorter retention (e.g. 2–3 days for packet_stream) is enough for the web viewer and transmission_tracker; longer retention is only needed if you want more history.
How cleanup runs
- The scheduler (in the main bot process) runs a single data-retention task on a 24-hour interval after startup (the first run is not immediate on boot; it aligns with the nightly maintenance email cadence).
- That task:
- Cleans packet_stream (via web viewer integration when enabled).
- Cleans purging_log, daily_stats, unique_advert_packets, and observed_paths (repeater manager).
- Cleans message_stats, command_stats, path_stats (stats command’s
cleanup_old_stats). - Removes expired rows from geocoding_cache and generic_cache (DB manager).
- Deletes old rows from mesh_connections (mesh graph).
So as long as the bot is running, the database is pruned on a schedule regardless of whether you run the standalone web viewer or the stats command.
Log files ([Logging] log_file, e.g. meshcore_bot.log) use rotating file logging: the bot rotates at 5 MB and keeps up to 3 backup files (same policy as the web viewer), so log disk use stays bounded.