mirror of
https://github.com/agessaman/meshcore-bot.git
synced 2026-10-11 16:57:22 +00:00
agessaman
7e3ab04b53
The landing page re-ran ~50 aggregate queries five times per load, then repeated the whole sequence every 30 seconds forever — including in backgrounded tabs. Against the live 1.44 GB database that was roughly 20 seconds of SQLite work per page load. Move the work off the request path. A refresher thread in the viewer process (which already runs migrations, so it works for a split-DB install) writes two tables: daily_rollup, one row per local date, and dashboard_snapshot, a single JSON row. A page load now reads one row. Measured on the live database: first paint 6 requests -> 2, /api/dashboard/summary p50 1.1 ms (304 in 0.8 ms), /api/stats 130 ms, and a 0.32 s refresh once a minute in the background. Make the numbers mean what they say: - Window selectors are built from each source's retention. The page offered "30d" and "All" against tables pruned at 7 days, so three of four choices returned the same figure under a label that denied it. - The incoming-packet chart reports its measured window instead of claiming 7 days for a table pruned at 3 — it sat beside a genuine 7-day contacts chart inviting an invalid comparison. - Days with no source data store NULL and render as gaps. Writing 0 would put a fake cliff at every retention boundary. - Signal metrics are stored as sums and counts, never means, so any window re-aggregates correctly. - Delta chips compare the last two complete calendar days and say so; the headline above them is a rolling 24 hours. - Unmapped role ordinals (type0..type15) bucket into "Unknown". - SNR comes from message_stats, where it is populated on every row, not from complete_contact_tracking, where it is populated on 7%. Kill the json_extract scans: packet_stream gains denormalized route_type_name, payload_type_name, path_len and bytes_per_hop, written at capture time. Aggregating those from JSON cost 3-6 s per query. Existing rows convert a bounded batch per tick rather than in one migration that would rewrite ~180 MB into the WAL and stall bot startup. A partial index serves as the backfill worklist — without it the "any rows left?" probe is a full scan costing 4.6 s per tick, and it costs that after the backfill finishes, because finding nothing still means reading everything. Also: replace the per-contact hop-prefix scan with the existing bucketed matcher and memoize the 7-day chunk set (264 ms -> 35 ms on a synthetic 100k-row database, regression-locked by a test); move the dashboard's JS and CSS to static files, which removes the CSP nonce requirement for the bulk of the page; and give cleanup_old_stats a future-timestamp guard, without which rows dated 2103 are never older than the cutoff and so live forever. Deletes the orphaned /stats page, unreachable from the nav and rendering stub charts that never populated. /api/stats stays as a shim with every key name intact plus Deprecation and Sunset headers. All schema changes are additive, so a downgraded codebase can read the data; it would however need the new schema_version rows removed, since MigrationRunner rejects versions it does not know.
62 lines
4.1 KiB
Markdown
62 lines
4.1 KiB
Markdown
# Data retention
|
||
|
||
The bot stores data in a SQLite database for the web viewer, stats, repeater management, and path routing. To limit database size, **data retention** controls how long rows are kept. Cleanup runs **daily** from the bot’s scheduler, so retention is enforced even when the standalone web viewer is not running.
|
||
|
||
## Configuration
|
||
|
||
All retention options live in the **`[Data_Retention]`** section of `config.ini`. Example (see `config.ini.example` for full comments):
|
||
|
||
```ini
|
||
[Data_Retention]
|
||
packet_stream_retention_days = 3
|
||
daily_stats_retention_days = 90
|
||
observed_paths_retention_days = 90
|
||
purging_log_retention_days = 90
|
||
mesh_connections_retention_days = 7
|
||
```
|
||
|
||
Stats tables (message_stats, command_stats, path_stats) use **`[Stats_Command]`** `data_retention_days` (default 7); the scheduler runs that cleanup daily as well. Stats are **collected** by default with **`collect_stats = true`** under `[Stats_Command]`, even if the user-facing `stats` chat command is disabled with `enabled = false`. Set **`collect_stats = false`** only if you want to stop writing those dashboard stats tables.
|
||
|
||
## Tables and defaults
|
||
|
||
| Table / data | Purpose | Default retention |
|
||
|--------------|---------|--------------------|
|
||
| **packet_stream** | Real-time packets, commands, routing in the web viewer; transmission_tracker repeat counts | 3 days |
|
||
| **daily_stats** | Daily repeater/advert stats | 90 days |
|
||
| **unique_advert_packets** | Unique packet hashes for advert stats | 90 days (same as daily_stats) |
|
||
| **observed_paths** | Path strings from adverts and messages | 90 days |
|
||
| **purging_log** | Audit trail for repeater purges | 90 days |
|
||
| **mesh_connections** | Path graph edges (in-memory + DB); should be ≥ Path_Command `graph_edge_expiration_days` | 7 days |
|
||
| **message_stats, command_stats, path_stats** | Stats command data | 7 days (`[Stats_Command]` `data_retention_days`) |
|
||
| **daily_rollup** | Per-day dashboard rollups, so trends outlive the tables above | 400 days (`[Web_Viewer]` `dashboard_snapshot_history_days`) |
|
||
| **dashboard_snapshot** | Single-row current-state payload for the dashboard | Overwritten in place |
|
||
| **geocoding_cache, generic_cache** | Expired entries removed by scheduler | By expiry time |
|
||
|
||
`daily_rollup` exists precisely because the retention above is short: the
|
||
dashboard charts 30 days of message and packet activity from tables pruned at 7
|
||
and 3 days respectively. One row per day is roughly 200 bytes, so a year of
|
||
history costs well under a megabyte. Pruning it is handled by the web viewer's
|
||
snapshot refresher rather than the bot scheduler, because the viewer may point
|
||
at a different database file.
|
||
|
||
Note that cleanup also removes rows dated implausibly far in the **future**.
|
||
A node with a bad clock can write a timestamp years ahead; because such a row is
|
||
never older than the cutoff, a lower-bound-only delete would keep it forever and
|
||
stretch every chart axis to match.
|
||
|
||
Shorter retention (e.g. 2–3 days for `packet_stream`) is enough for the web viewer and transmission_tracker; longer retention is only needed if you want more history.
|
||
|
||
## How cleanup runs
|
||
|
||
1. The **scheduler** (in the main bot process) runs a single data-retention task on a **24-hour interval** after startup (the first run is not immediate on boot; it aligns with the nightly maintenance email cadence).
|
||
2. That task:
|
||
- Cleans **packet_stream** (via web viewer integration when enabled).
|
||
- Cleans **purging_log**, **daily_stats**, **unique_advert_packets**, and **observed_paths** (repeater manager).
|
||
- Cleans **message_stats**, **command_stats**, **path_stats** (stats command’s `cleanup_old_stats`).
|
||
- Removes expired rows from **geocoding_cache** and **generic_cache** (DB manager).
|
||
- Deletes old rows from **mesh_connections** (mesh graph).
|
||
|
||
So as long as the bot is running, the database is pruned on a schedule regardless of whether you run the standalone web viewer or the stats command.
|
||
|
||
**Log files** (`[Logging] log_file`, e.g. `meshcore_bot.log`) use rotating file logging: the bot rotates at 5 MB and keeps up to 3 backup files (same policy as the web viewer), so log disk use stays bounded.
|