- Implemented caching for multi-byte mesh aggregation, allowing concurrent requests to share computation and reducing CPU load.
- Updated the web viewer to coalesce live edge events and refresh the mesh graph every 30 seconds, improving responsiveness on busy networks.
- Adjusted logging behavior in the web viewer to respect the configured log level, ensuring efficient logging without excessive duplicate entries.
- Added configuration options for mesh graph caching duration, enhancing user control over performance settings.
- Updated mesh graph path splitting and aggregation to run in SQLite, improving performance by avoiding Python materialization.
- Defaulted graph persistence to batched writes for new installations, reducing WAL churn and SD-card writes.
- Enhanced data retention to execute shortly after startup, independent of the nightly maintenance schedule.
- Added a table-specific index for `mesh_connections` to support window and retention queries, ensuring efficient data access.
The landing page re-ran ~50 aggregate queries five times per load, then
repeated the whole sequence every 30 seconds forever — including in
backgrounded tabs. Against the live 1.44 GB database that was roughly 20
seconds of SQLite work per page load.
Move the work off the request path. A refresher thread in the viewer
process (which already runs migrations, so it works for a split-DB
install) writes two tables: daily_rollup, one row per local date, and
dashboard_snapshot, a single JSON row. A page load now reads one row.
Measured on the live database: first paint 6 requests -> 2,
/api/dashboard/summary p50 1.1 ms (304 in 0.8 ms), /api/stats 130 ms,
and a 0.32 s refresh once a minute in the background.
Make the numbers mean what they say:
- Window selectors are built from each source's retention. The page
offered "30d" and "All" against tables pruned at 7 days, so three of
four choices returned the same figure under a label that denied it.
- The incoming-packet chart reports its measured window instead of
claiming 7 days for a table pruned at 3 — it sat beside a genuine
7-day contacts chart inviting an invalid comparison.
- Days with no source data store NULL and render as gaps. Writing 0
would put a fake cliff at every retention boundary.
- Signal metrics are stored as sums and counts, never means, so any
window re-aggregates correctly.
- Delta chips compare the last two complete calendar days and say so;
the headline above them is a rolling 24 hours.
- Unmapped role ordinals (type0..type15) bucket into "Unknown".
- SNR comes from message_stats, where it is populated on every row, not
from complete_contact_tracking, where it is populated on 7%.
Kill the json_extract scans: packet_stream gains denormalized
route_type_name, payload_type_name, path_len and bytes_per_hop, written
at capture time. Aggregating those from JSON cost 3-6 s per query.
Existing rows convert a bounded batch per tick rather than in one
migration that would rewrite ~180 MB into the WAL and stall bot startup.
A partial index serves as the backfill worklist — without it the "any
rows left?" probe is a full scan costing 4.6 s per tick, and it costs
that after the backfill finishes, because finding nothing still means
reading everything.
Also: replace the per-contact hop-prefix scan with the existing bucketed
matcher and memoize the 7-day chunk set (264 ms -> 35 ms on a synthetic
100k-row database, regression-locked by a test); move the dashboard's JS
and CSS to static files, which removes the CSP nonce requirement for the
bulk of the page; and give cleanup_old_stats a future-timestamp guard,
without which rows dated 2103 are never older than the cutoff and so
live forever.
Deletes the orphaned /stats page, unreachable from the nav and rendering
stub charts that never populated. /api/stats stays as a shim with every
key name intact plus Deprecation and Sunset headers.
All schema changes are additive, so a downgraded codebase can read the
data; it would however need the new schema_version rows removed, since
MigrationRunner rejects versions it does not know.
- Enhanced the `/api/contacts` endpoint to support pagination, allowing users to specify `page` and `page_size` parameters.
- Added search functionality to filter contacts based on a search term.
- Implemented sorting options for the contacts list, enabling users to sort by various fields.
- Updated the frontend to reflect these changes, including a new pagination UI and improved loading states.
- Introduced caching for multibyte hop chunks to optimize performance when retrieving contact badge evidence.
- Added tests to ensure the new features work as expected and maintain existing functionality.
Single-byte repeater identity guessing has been persistently unreliable,
and mesh_connections flattens away evidence provenance (confirmed_2byte
never persists). observed_paths retains it: bytes_per_hop marks which
paths carry unambiguous multi-byte hops.
- /api/mesh/edges?evidence=multibyte derives edges purely from unique
multi-byte path observations (consecutive hop pairs, aggregated with a
distinct-path count); 2-byte edges coalesce into a 3-byte edge only on
a unique prefix match, mirroring MeshGraph.add_edge
- default mode tags each edge evidence=multibyte|singlebyte by key length
- Evidence filter on the mesh page (persisted with the other filters);
single-byte edges render dashed in map and graph views, with an
evidence line in tooltips and a legend entry
- client-side haversine fallback for edges without a stored distance
- prefix-compatible edge/node matching so degree sizing and node details
work when edge and node prefixes differ in resolution
- /api/mesh/stats reports multibyte_edges; shown on the Edges stat card
Adds device-level companion settings to /radio, headlined by the response
path hash size (mode 0-2 = 1-3 bytes per hop) wired to the existing
firmware config endpoints. New sections: Identity & Adverts (name, advert
lat/lon, location policy, zero-hop/flood advert buttons), Mesh Behavior
(extra ACKs, telemetry permissions), and write-only Advanced Tuning
(RX delay base, airtime factor, sent x1000 per the wire format).
Backend: GET /api/radio/params now returns the full SELF_INFO node fields;
POST accepts the new fields with validation; new POST /api/radio/advert;
scheduler ops handle the writes (other-params group as one read-modify-write
frame so partial updates never clobber device state).
Config-managed settings are not writable from the panel: new-contact mode
is owned by [Bot] auto_manage_contacts and shown read-only; the node name
is locked while bot_name + auto_update_device_name manage it. loop.detect
is removed from the firmware endpoints entirely - it is repeater/room-server
CLI config, and companion custom vars map to sensor settings.
Also: dark-mode styling for disabled form fields (base.html), TASK-01 guard
test updated for the deliberate reintroduction, 23 endpoint/template tests.
- Introduced a new configuration option `announce_disallowed` to allow notifications for goals overturned by VAR.
- Updated `WorldCupLiveService` to handle disallowed goals, including formatting for previously announced scorers.
- Enhanced `ESPNClient` to ensure accurate goal tracking and state management.
- Added unit tests to validate the new disallowed goal functionality and ensure correct behavior during matches.
- Introduced a new configuration option `multibyte_monitor_enabled` in `config.ini.example` to control the visibility of the multibyte monitor page and API endpoints.
- Implemented the `/multibyte-rollout` and `/api/multibyte-rollout` routes in the web viewer, which are accessible only when the multibyte monitor feature is enabled.
- Updated documentation in `web-viewer.md` to reflect the new feature and its configuration.
- Added tests to ensure the multibyte routes are disabled by default and accessible when enabled.
- Implemented validation for the `channel_name` field to ensure it is not empty when updating feeds, raising a ValueError if validation fails.
- Added test cases to verify that updates to `channel_name` persist correctly and that attempts to set an empty `channel_name` are rejected with an appropriate error response.
Introduced optional timeout settings in the configuration for various web viewer operations, including edge and node post timeouts, SQLite connection timeout, and requeue timeout. Updated the web viewer integration to utilize these settings, enhancing flexibility and reliability. Added commands to inspect the resolved configuration with sensitive keys redacted, and updated documentation accordingly.
Modified the config item retrieval in BotDataViewer to use raw=True, ensuring that literal '%' characters in configuration values are not subject to interpolation. Additionally, removed the breadcrumb navigation from the admin configuration template for a cleaner UI. Added a test to verify that '%' in values does not cause server errors.
Introduced a new test class, TestAdminConfig, to verify the loading of the admin configuration page and ensure sensitive information like passwords is redacted in the response. This enhances test coverage for the admin configuration functionality.
- Add allow_private param to validate_external_url (alias for
allow_localhost) to unblock web viewer SSRF guard using allow_private=
- Block non-globally-routable IPs (RFC 6598 100.64.0.0/10 CGN) on
Python 3.10 which does not classify them as private or reserved
- Remove tests for greeter DB tables and admin_config template that
depend on features not present on this branch
- USE-05: Add /api-explorer page listing all ~65 API endpoints in 9
categories (System, Contacts, Mesh, Channels, Feeds, Radio, Admin,
Maintenance, Config, Greeter) with method badges, descriptions, and
curl example modal. Filter bar and collapse per section. Nav item
added to base.html.
- USE-06: Three targeted error-message improvements:
1. 500 handler now returns user-friendly HTML page (error.html) for
browser requests and sanitized JSON for API/JSON requests instead
of a bare string.
2. Feed processed-items query failures promoted from logger.debug to
logger.warning so operators see them in normal log output.
3. Global JS fetch interceptor in base.html redirects to /login?next=
on any 401 response, handling session expiry mid-page.
- Fix pre-existing test bug: test_reload_endpoint_success mock return
value did not match actual code message from reload_config.
agessaman