6b4e6351c4
- Added support for rotating log files in `core.py` using `RotatingFileHandler`, with a maximum size of 5 MB and up to 3 backup files. - Updated `config.ini.example`, `config.ini.minimal-example`, and `config.ini.quickstart` to include descriptions of the new log rotation feature. - Enhanced `data-retention.md` to clarify log file management and retention policies.
3.0 KiB
Data retention
The bot stores data in a SQLite database for the web viewer, stats, repeater management, and path routing. To limit database size, data retention controls how long rows are kept. Cleanup runs daily from the bot’s scheduler, so retention is enforced even when the standalone web viewer is not running.
Configuration
All retention options live in the [Data_Retention] section of config.ini. Example (see config.ini.example for full comments):
[Data_Retention]
packet_stream_retention_days = 3
daily_stats_retention_days = 90
observed_paths_retention_days = 90
purging_log_retention_days = 90
mesh_connections_retention_days = 7
Stats tables (message_stats, command_stats, path_stats) use [Stats_Command] data_retention_days (default 7); the scheduler runs that cleanup daily as well. Stats are collected when the stats command is enabled, or when the optional collect_stats = true is set under [Stats_Command] (so the web viewer dashboard can show message/command/path stats even if the stats chat command is disabled).
Tables and defaults
| Table / data | Purpose | Default retention |
|---|---|---|
| packet_stream | Real-time packets, commands, routing in the web viewer; transmission_tracker repeat counts | 3 days |
| daily_stats | Daily repeater/advert stats | 90 days |
| unique_advert_packets | Unique packet hashes for advert stats | 90 days (same as daily_stats) |
| observed_paths | Path strings from adverts and messages | 90 days |
| purging_log | Audit trail for repeater purges | 90 days |
| mesh_connections | Path graph edges (in-memory + DB); should be ≥ Path_Command graph_edge_expiration_days |
7 days |
| message_stats, command_stats, path_stats | Stats command data | 7 days ([Stats_Command] data_retention_days) |
| geocoding_cache, generic_cache | Expired entries removed by scheduler | By expiry time |
Shorter retention (e.g. 2–3 days for packet_stream) is enough for the web viewer and transmission_tracker; longer retention is only needed if you want more history.
How cleanup runs
- The scheduler (in the main bot process) runs a single data-retention task once every 24 hours.
- That task:
- Cleans packet_stream (via web viewer integration when enabled).
- Cleans purging_log, daily_stats, unique_advert_packets, and observed_paths (repeater manager).
- Cleans message_stats, command_stats, path_stats (stats command’s
cleanup_old_stats). - Removes expired rows from geocoding_cache and generic_cache (DB manager).
- Deletes old rows from mesh_connections (mesh graph).
So as long as the bot is running, the database is pruned on a schedule regardless of whether you run the standalone web viewer or the stats command.
Log files ([Logging] log_file, e.g. meshcore_bot.log) use rotating file logging: the bot rotates at 5 MB and keeps up to 3 backup files (same policy as the web viewer), so log disk use stays bounded.