The landing page re-ran ~50 aggregate queries five times per load, then
repeated the whole sequence every 30 seconds forever — including in
backgrounded tabs. Against the live 1.44 GB database that was roughly 20
seconds of SQLite work per page load.
Move the work off the request path. A refresher thread in the viewer
process (which already runs migrations, so it works for a split-DB
install) writes two tables: daily_rollup, one row per local date, and
dashboard_snapshot, a single JSON row. A page load now reads one row.
Measured on the live database: first paint 6 requests -> 2,
/api/dashboard/summary p50 1.1 ms (304 in 0.8 ms), /api/stats 130 ms,
and a 0.32 s refresh once a minute in the background.
Make the numbers mean what they say:
- Window selectors are built from each source's retention. The page
offered "30d" and "All" against tables pruned at 7 days, so three of
four choices returned the same figure under a label that denied it.
- The incoming-packet chart reports its measured window instead of
claiming 7 days for a table pruned at 3 — it sat beside a genuine
7-day contacts chart inviting an invalid comparison.
- Days with no source data store NULL and render as gaps. Writing 0
would put a fake cliff at every retention boundary.
- Signal metrics are stored as sums and counts, never means, so any
window re-aggregates correctly.
- Delta chips compare the last two complete calendar days and say so;
the headline above them is a rolling 24 hours.
- Unmapped role ordinals (type0..type15) bucket into "Unknown".
- SNR comes from message_stats, where it is populated on every row, not
from complete_contact_tracking, where it is populated on 7%.
Kill the json_extract scans: packet_stream gains denormalized
route_type_name, payload_type_name, path_len and bytes_per_hop, written
at capture time. Aggregating those from JSON cost 3-6 s per query.
Existing rows convert a bounded batch per tick rather than in one
migration that would rewrite ~180 MB into the WAL and stall bot startup.
A partial index serves as the backfill worklist — without it the "any
rows left?" probe is a full scan costing 4.6 s per tick, and it costs
that after the backfill finishes, because finding nothing still means
reading everything.
Also: replace the per-contact hop-prefix scan with the existing bucketed
matcher and memoize the 7-day chunk set (264 ms -> 35 ms on a synthetic
100k-row database, regression-locked by a test); move the dashboard's JS
and CSS to static files, which removes the CSP nonce requirement for the
bulk of the page; and give cleanup_old_stats a future-timestamp guard,
without which rows dated 2103 are never older than the cutoff and so
live forever.
Deletes the orphaned /stats page, unreachable from the nav and rendering
stub charts that never populated. /api/stats stays as a shim with every
key name intact plus Deprecation and Sunset headers.
All schema changes are additive, so a downgraded codebase can read the
data; it would however need the new schema_version rows removed, since
MigrationRunner rejects versions it does not know.
Changed the default setting for `auto_manage_contacts` from `false` to `device` across configuration files and updated related documentation. This ensures that the device handles auto-addition of contacts while the bot manages capacity. Adjusted comments and documentation to reflect this change for clarity and consistency.
- Added detailed diagnostics for database initialization failures in `DBManager`, improving error logging with filesystem checks.
- Implemented a new method `_resolve_viewer_db_path` in `WebViewerIntegration` to determine the database path for the viewer subprocess.
- Introduced `_preflight_database_path` to validate the viewer database path, ensuring the parent directory exists, is writable, and is a directory.
- Added unit tests to verify logging behavior for missing or invalid database paths in both `DBManager` and `WebViewerIntegration`.
Introduced optional timeout settings in the configuration for various web viewer operations, including edge and node post timeouts, SQLite connection timeout, and requeue timeout. Updated the web viewer integration to utilize these settings, enhancing flexibility and reliability. Added commands to inspect the resolved configuration with sensitive keys redacted, and updated documentation accordingly.
- Updated `config.ini.example` to clarify password requirements for web viewer authentication, emphasizing the need for a password when binding to non-loopback interfaces.
- Introduced a new function `normalized_web_viewer_password` in `integration.py` to standardize password retrieval and validation, handling various empty and null placeholder cases.
- Enhanced error logging in `core.py` to use `logger.error` for web viewer integration failures, improving visibility of issues.
- Modified `app.py` to utilize the new password normalization function, ensuring consistent password handling across the application.
- Added tests in `test_web_viewer_integration.py` to validate password normalization and error logging behavior when the web viewer is configured without a password.
- Updated the `resolve_path` function in `utils.py` to clarify behavior regarding absolute paths.
- Changed type hints in `discord_bridge_service.py` for better clarity and consistency.
- Removed unused imports and unnecessary comments in various test files to improve code cleanliness and readability.
agessaman