meshcore-bot/docs/data-retention.md

Data retention

The bot stores data in a SQLite database for the web viewer, stats, repeater management, and path routing. To limit database size, data retention controls how long rows are kept. Cleanup runs daily from the bot’s APScheduler-based maintenance loop, so retention is enforced even when the standalone web viewer is not running.

Configuration

All retention options live in the [Data_Retention] section of config.ini. Example (see config.ini.example for full comments):

[Data_Retention]
packet_stream_retention_days = 3
daily_stats_retention_days = 90
observed_paths_retention_days = 90
purging_log_retention_days = 90
mesh_connections_retention_days = 7

Stats tables (message_stats, command_stats, path_stats) use [Stats_Command] data_retention_days (default 7); the scheduler runs that cleanup daily as well. Stats are collected by default with collect_stats = true under [Stats_Command], even if the user-facing stats chat command is disabled with enabled = false. Set collect_stats = false only if you want to stop writing those dashboard stats tables.

Tables and defaults

Table / data	Purpose	Default retention
packet_stream	Real-time packets, commands, routing in the web viewer; transmission_tracker repeat counts	3 days
daily_stats	Daily repeater/advert stats	90 days
unique_advert_packets	Unique packet hashes for advert stats	90 days (same as daily_stats)
observed_paths	Path strings from adverts and messages	90 days
purging_log	Audit trail for repeater purges	90 days
mesh_connections	Path graph edges (in-memory + DB); should be ≥ Path_Command graph_edge_expiration_days	7 days
message_stats, command_stats, path_stats	Stats command data	7 days ([Stats_Command] data_retention_days)
daily_rollup	Per-day dashboard rollups, so trends outlive the tables above	400 days ([Web_Viewer] dashboard_snapshot_history_days)
dashboard_snapshot	Single-row current-state payload for the dashboard	Overwritten in place
geocoding_cache, generic_cache	Expired entries removed by scheduler	By expiry time

daily_rollup exists precisely because the retention above is short: the dashboard charts 30 days of message and packet activity from tables pruned at 7 and 3 days respectively. One row per day is roughly 200 bytes, so a year of history costs well under a megabyte. Pruning it is handled by the web viewer's snapshot refresher rather than the bot scheduler, because the viewer may point at a different database file.

Note that cleanup also removes rows dated implausibly far in the future. A node with a bad clock can write a timestamp years ahead; because such a row is never older than the cutoff, a lower-bound-only delete would keep it forever and stretch every chart axis to match.

Shorter retention (e.g. 2–3 days for packet_stream) is enough for the web viewer and transmission_tracker; longer retention is only needed if you want more history.

How cleanup runs

1. The scheduler (in the main bot process) runs a single data-retention task on a 24-hour interval after startup (the first run is not immediate on boot; it aligns with the nightly maintenance email cadence).
2. That task:
   • Cleans packet_stream (via web viewer integration when enabled).
   • Cleans purging_log, daily_stats, unique_advert_packets, and observed_paths (repeater manager).
   • Cleans message_stats, command_stats, path_stats (stats command’s cleanup_old_stats).
   • Removes expired rows from geocoding_cache and generic_cache (DB manager).
   • Deletes old rows from mesh_connections (mesh graph).

So as long as the bot is running, the database is pruned on a schedule regardless of whether you run the standalone web viewer or the stats command.

Log files ([Logging] log_file, e.g. meshcore_bot.log) use rotating file logging: the bot rotates at 5 MB and keeps up to 3 backup files (same policy as the web viewer), so log disk use stays bounded.