Removes 26 console.log calls, including contacts.html dumping whole data
structures and sample contact records on every render. Variables and
callback parameters that existed only to feed those logs go with them
(deviceTypes, anyNewDevices, an availableEdges debug block, and a
realtime status handler whose body was nothing but a log), so no new
no-unused-vars warnings are introduced — the count drops 60 to 59.
console.warn and console.error are kept. One console.log survives in
realtime.html: the decoder key-count line is a one-time startup message
with real diagnostic value, and removing it would mean deleting two
counters and their increments inside a loop.
Also normalizes the mesh page's user-visible "Neighbours" strings to
match the project's American spelling. The option value stays
"neighbors", so evidence-mode filtering is untouched.
login.html loads DM Sans and Outfit from fonts.googleapis.com with the
font files on fonts.gstatic.com, but neither host was in style-src or
font-src, so the browser blocked both and the login page silently fell
back to system fonts with CSP violations in the console.
Only consider observed_paths rows with bytes_per_hop in {1,2,3} so NULL
or invalid encodings fall back to out_bytes_per_hop instead of classifying
as 1-byte.
Four review findings on this branch, all confirmed against the code and the
installed meshcore 2.3.8:
Single-flight the discovery cycle. The command guarded only its own task, so
the scheduler's independent call could overlap a manual cycle and each round
would collect into the other's discover window. run_neighbors_cycle is now a
guard around the cycle body, refusing whichever trigger arrives second.
Make the 15-minute cooldown per node rather than per sender. The base class
rations per user, but the cost here is mesh airtime: users could take turns and
keep the radio discovering continuously. Measured from the last cycle that
produced a result, so the scheduler's cycles count and a cycle that bailed out
without transmitting does not start the clock.
Window the neighbours evidence label. The combined viewer applied `days` only
to mesh_connections while reading every lifetime row from neighbor_links, which
is deliberately never pruned — so a link last heard years ago kept claiming a
recent path-derived edge was a current direct neighbour.
Match that label on full public keys too. MeshGraph.add_edge does not promote a
1-byte edge that has no public key, but still fills in the keys discovery
supplied, so the 3-byte prefix comparison alone left confirmed neighbours
labelled singlebyte. Truncating our keys to 2 chars instead would relabel every
other node sharing that byte.
Restore a contact's flood path after an interrupted scope request. send_anon_req
pins a path-less contact to zero-hop and restores it after the send with no
try/finally, so our own budget cancelling the request left the contact pinned
and every later message to it sent direct-only.
The meshcore>=2.3.8 pin needed no change: PyPI publishes 2.3.8 now.
Port the observer firmware's neighbours feature into the bot's packet capture
service, by way of meshcore-packet-capture (upstream PRs #42/#43). On a long
interval the bot asks which repeaters it hears directly and records each
confirmed link with its measured SNR.
This is the strongest link evidence the bot collects: a first-party RF
measurement between two full 32-byte public keys. Path inference works from
1-3 byte prefixes with no keys, and complete_contact_tracking.hop_count
over-claims zero-hop (800 claimed vs 68 corroborated on the live database).
modules/neighbors_discovery.py keeps upstream's public names so its fixes and
tests stay portable. Two deliberate divergences:
- No command_lock plumbing. _SerializedCommands in modules/core.py already
serialises and paces every radio command, strictly more than upstream's
reentrant lock did.
- neighbors_collect_scopes defaults off. Upstream's zero-hop scope probe
relies on a neighbour not being a known contact; this bot tracks contacts,
and for a repeater with no stored path the library reaches zero-hop by
calling change_contact_path() then reset_path() -- mutating the device's
contact table per neighbour. Scope requests also hold the radio lock for
their whole round trip (~25s), stalling bot replies. The default cycle is
one command plus a passive listen window, during which the bot stays
responsive.
Evidence lands in neighbor_links and neighbor_observations (migration 22)
rather than mesh_connections, which cannot persist provenance. The viewer
exposes it as evidence=neighbors on /api/mesh/edges and a Neighbours Only
mode on the mesh page, with populated public keys and real SNR; confirmed
neighbours also relabel edges in the combined view and count as
provenance-trusted when framing the initial map.
neighbors_enabled is the single switch. Every enabled broker publishes once
it is on (mqttN_neighbors defaults true; set false to hold one back). The
topic derives from each broker's packets topic with the last segment swapped,
so a templated broker gets meshcore/{IATA}/{PUBLIC_KEY}/neighbors -- the
topic the firmware uses -- instead of an unrelated flat one. A derived
location-routed topic is skipped with a warning when no iata is set, rather
than publishing into meshcore/XYZ/... on a shared namespace. Snapshots are
non-retained: heard_secs_ago is relative to publish time, so a retained copy
would read as current days later.
Also adds a DM-gated `neighbors` command (the 12h interval floor makes
waiting for the scheduler impractical), which acks immediately and reports in
a second message once the window closes.
Requires meshcore >= 2.3.8 for send_node_discover_req / req_regions_sync.
- Replaced inline timestamp formatting logic with a call to `format_relative_timestamp` for improved readability and maintainability.
- Modularized feed formatting functions by moving them to `feed_format`, enhancing code organization and reusability.
- Updated references in `feed_manager` and `web_viewer` to utilize the new modular functions, ensuring consistent behavior across modules.
- Adjusted tests to reflect changes in function imports and ensure proper functionality of the new structure.
- Introduced a locking mechanism to ensure journal mode is initialized once per configuration section, preventing redundant database operations.
- Added a validation function to check for required repeater tables, raising an error with actionable messages if any are missing.
- Updated the DBManager to handle journal modes more efficiently, ensuring that only persistent modes are applied across connections.
- Refactored the RepeaterManager to utilize the new validation function, improving error handling during initialization.
- Enhanced tests to cover new journal mode behaviors and validate repeater table existence, ensuring robust database management.
- Added a new feature to track and display the multibyte share of packets by payload type in the dashboard.
- Introduced a new database migration to store per-payload-type multibyte encoding data in the daily rollup.
- Updated the dashboard to visualize the multibyte share as a stacked bar chart, reflecting the share of each day's packets that took a multibyte path.
- Enhanced the API to provide raw counts for each payload type, ensuring accurate representation in the dashboard.
- Adjusted the frontend to maintain consistent color coding for payload types and improve the overall user experience.
- Updated tests to validate the new multibyte share functionality and ensure data integrity.
- Implemented caching for multi-byte mesh aggregation, allowing concurrent requests to share computation and reducing CPU load.
- Updated the web viewer to coalesce live edge events and refresh the mesh graph every 30 seconds, improving responsiveness on busy networks.
- Adjusted logging behavior in the web viewer to respect the configured log level, ensuring efficient logging without excessive duplicate entries.
- Added configuration options for mesh graph caching duration, enhancing user control over performance settings.
- Implemented chunked deletion for data retention, allowing for smoother cleanup processes without monopolizing SQLite's writer lock.
- Configured retention settings to delete in batches with pauses, improving performance on SD-card installations.
- Updated Linux service installers to allocate 1GB of memory and 200% CPU, providing better resource management for Raspberry Pi workloads.
- Added detailed configuration examples for Raspberry Pi in documentation to guide users on optimal settings.
- Updated mesh graph path splitting and aggregation to run in SQLite, improving performance by avoiding Python materialization.
- Defaulted graph persistence to batched writes for new installations, reducing WAL churn and SD-card writes.
- Enhanced data retention to execute shortly after startup, independent of the nightly maintenance schedule.
- Added a table-specific index for `mesh_connections` to support window and retention queries, ensuring efficient data access.
The flood tail decays over roughly twenty hops in bars under a pixel
tall. Buckets holding less than 0.1% of the series are no longer drawn,
which on the live mesh takes the axis from 64 buckets to 44.
What is left out is reported, not dropped: a line under the chart reads
"1,467 further flood packets (0.9%) sit in 20 hop buckets below 0.1%
each, and are not drawn." An unannounced cut would be the same lie as a
silently truncated category list.
Two details that matter for honesty:
Percentages still divide by the whole series, never by the drawn subset,
so removing the tail cannot inflate the bars that remain. The smallest
surviving bar on live data is 0.118%, which is what it was before.
A withheld bucket inside the axis is null rather than zero. Zero would
claim no packets travelled that far, which is a different and false
statement; null draws nothing and says nothing. Padding gaps stay zero,
because there it is true.
The threshold applies to the flood series only — the node series is
small enough to draw in full — and the underlying computation still
covers the whole 64-hop protocol range.
Three layout and cost changes.
Hops away and Roles now grow into their cards instead of leaving dead
space under a fixed-height canvas. Chart.js needs a positioned parent
with a real height, so the body becomes a flex column and the chart takes
the slack via flex-basis 0 — height: 100% would resolve against an
auto-height parent and collapse.
The hop axis now ends at the last hop that carries an observation rather
than at the last non-empty bucket of the padded union. On a quiet window
that collapses 64 buckets to 13; on a full one it changes nothing,
because the flood series really does have packets at every hop out to 63
— 14 of them at hop 63, and 14.1% of all flood traffic beyond hop 20.
That tail is real data, so it is drawn rather than truncated.
Dropped the Live Activity card. It opened three SocketIO subscriptions
and re-rendered on every packet to duplicate /realtime, which is a page
that already does it better. The dashboard now costs one snapshot read
per poll and holds no streaming subscriptions. A test asserts it stays
that way rather than merely that the markup is gone.
MeshCore carries up to a 64-byte path, so a route can be 64 hops long at
one byte per hop — and proportionally fewer with 2- or 3-byte hashes,
which is where the observed ceilings of 32 and 21 come from.
The chart capped at 32. That number was inherited from the old
dashboard's `BETWEEN 0 AND 32` filters and carried forward without
checking it against the protocol. On the live database it silently
discarded 5,654 flood packets arriving from as far as 63 hops.
Worse for the node series: the cap runs after the per-node MIN(), so a
node whose *closest* route exceeded 32 hops disappeared from the chart
altogether instead of appearing at the far end. The live history holds
one such node, reachable only by a 48-hop path.
The axis can now span 64 grouped categories, so bars drop their rounded
corners and padding above 24 buckets and the x ticks auto-skip. 64 is
kept as a hard ceiling: beyond it the path field could not have held the
route, so the value is corrupt rather than distant.
Keeps the advert series — nodes by their closest observed path — and
adds arriving flood packets by how far they had already travelled.
The two answer different questions and, on the live mesh, disagree
usefully: nodes peak at 2-3 hops and fall away quickly, while flood
traffic peaks at 5 and holds a long tail past 16. A close-in
neighbourhood absorbing flood from well beyond it.
One series counts nodes (2.8k) and the other packets (156k), so raw
counts on a shared axis would flatten the node series into the baseline.
Both are drawn as a share of their own total, with absolute counts in
the tooltip, and padded onto one contiguous hop range so the bars line
up. They also cover different spans — 7 days of adverts against
whatever packet_stream retains — so each is labelled with its own
window instead of being presented as one period.
Watch the units. observed_paths.path_length is a BYTE count, so hops are
path_length / bytes_per_hop. packet_stream.path_len is already a HOP
count, with the byte length carried separately as path_byte_length. A
17-hop 3-byte path is path_length 51 in one table and path_len 17 in the
other. Applying either rule to the other silently rescales the axis and
the only symptom is a chart that looks a bit off, so both conventions
are now pinned by tests against the shapes real rows take.
Flood packets carry no sender identity and observed_paths holds only
adverts, so the flood series cannot be reduced to a shortest path per
node the way the advert series is. It is a per-packet distribution, and
the tooltip says so.
Moving the hop histogram out into its own card left the routing mix as a
26px bar alone in a col-lg-5, next to a full-height chart. Fill it with
something that belongs there: what the traffic is, beside how it is
routed, over exactly the same packets — the totals agree because both
read the dimensioned rows.
This also gives payload_type_name a reason to exist. Migration 0019
added the column, integration.py writes it at capture time, the
refresher backfills it and an index covers it, and until now nothing
read it.
Category lists now roll their tail into "Other" instead of being
truncated at eight rows in the client. Silently dropping the tail left
bars that no longer summed to the total printed beside them; on the live
database that would have hidden 1,015 packets across four payload types.
307 direct neighbours was not plausible, and it was not real.
complete_contact_tracking.hop_count claims 800 zero-hop contacts. Only
68 of them have any one-hop path in observed_paths to corroborate that.
Their stored SNR piles up in a 1.5 dB band — 655 of 800 between 11.25
and 12.75 dB — and their RSSI clusters at -39..-48 dBm. Hundreds of
radios at different distances and terrain cannot land in a 10 dB window.
That is the signature of one strong local link being recorded against
every node whose traffic happened to arrive through it. Their return
paths agree: these "direct" contacts have out_path_len of 3 to 11.
The writer's intent is sound — repeater_manager only stores RSSI/SNR
when signal_info reports hops == 0 — so the field being fed to it does
not mean what the surrounding code assumes. Left as is; this change
stops the dashboard depending on it.
Neighbour membership now comes from path evidence: an advert whose
path_length equals its bytes_per_hop travelled exactly one hop. That
yields 38 nodes in 24h and 124 in 7d, with a plausible spread. Signal is
shown only where the path evidence and the stored hop count agree, which
is 5 and 12 nodes respectively; the rest read "no signal reading" rather
than borrowing a measurement taken on somebody else's link. A 24h/7d
selector bounds the window, capped well under observed_paths' 90-day
retention because a month-old link says nothing about today.
Separately, this fixes a bug I introduced. path_length is a BYTE count,
and with 2- or 3-byte hop encoding a 3-hop path is 6 or 9 bytes long. The
path-length histogram plotted that raw value on an axis readers would
take as hops, overstating distance two- to threefold on a mesh that is
~95% multibyte. It is replaced by a single hops-away chart computed as
path_length / bytes_per_hop, which also retires the histogram built on
the untrustworthy stored hop count. The result is unimodal, peaking at 3
hops and decaying — the shape a mesh should have, and not the bimodal
one the old chart drew.
Three adjustments from review.
Role and device type are the same field twice. Measured on the live
database they disagree on 16 of 11,028 contacts (ten roomservers and a
handful of bots and gateways reporting device Companion); every other
row is repeater/Repeater, companion/Companion, type11/Type11 and so on.
Charting both filled half a card with a copy of the other half. Keep the
role mix, which also carries the type0..type15 bucketing, and move it
into the routing row where the old signal card was.
Drop the tracked-contacts tile. is_currently_tracked does not describe
anything a reader can act on, and node activity is already covered by
nodes-heard and gone-quiet. The known-contacts total moves onto the
coverage tile, which leaves five tiles splitting the row evenly.
Rebuild the signal panel around zero-hop neighbours. Percentiles over
every received message answered no question anyone has: SNR on a relayed
packet measures the last hop into this radio, not the link to the node
that sent it, so averaging across hop counts describes nothing in
particular.
The panel now shows the nodes heard with no repeater in between — how
many, their SNR distribution, and the weakest links named, worst first,
on a fixed -12..+14 dB scale so bars mean the same thing between
refreshes. On the live database that is 307 neighbours, median 12.0 dB,
with eight marginal links surfaced from -9.0 dB down.
This also corrects the source. The plan rejected
complete_contact_tracking.snr as a badly biased 7% sample; in fact it is
populated on exactly the 800 hop_count=0 contacts and NULL on all 10,228
others. It is not a sample of the network, it is a complete census of
the neighbours — which is precisely the population the metric applies
to. message_stats.hops=0 covers only 34 senders by comparison.
The landing page re-ran ~50 aggregate queries five times per load, then
repeated the whole sequence every 30 seconds forever — including in
backgrounded tabs. Against the live 1.44 GB database that was roughly 20
seconds of SQLite work per page load.
Move the work off the request path. A refresher thread in the viewer
process (which already runs migrations, so it works for a split-DB
install) writes two tables: daily_rollup, one row per local date, and
dashboard_snapshot, a single JSON row. A page load now reads one row.
Measured on the live database: first paint 6 requests -> 2,
/api/dashboard/summary p50 1.1 ms (304 in 0.8 ms), /api/stats 130 ms,
and a 0.32 s refresh once a minute in the background.
Make the numbers mean what they say:
- Window selectors are built from each source's retention. The page
offered "30d" and "All" against tables pruned at 7 days, so three of
four choices returned the same figure under a label that denied it.
- The incoming-packet chart reports its measured window instead of
claiming 7 days for a table pruned at 3 — it sat beside a genuine
7-day contacts chart inviting an invalid comparison.
- Days with no source data store NULL and render as gaps. Writing 0
would put a fake cliff at every retention boundary.
- Signal metrics are stored as sums and counts, never means, so any
window re-aggregates correctly.
- Delta chips compare the last two complete calendar days and say so;
the headline above them is a rolling 24 hours.
- Unmapped role ordinals (type0..type15) bucket into "Unknown".
- SNR comes from message_stats, where it is populated on every row, not
from complete_contact_tracking, where it is populated on 7%.
Kill the json_extract scans: packet_stream gains denormalized
route_type_name, payload_type_name, path_len and bytes_per_hop, written
at capture time. Aggregating those from JSON cost 3-6 s per query.
Existing rows convert a bounded batch per tick rather than in one
migration that would rewrite ~180 MB into the WAL and stall bot startup.
A partial index serves as the backfill worklist — without it the "any
rows left?" probe is a full scan costing 4.6 s per tick, and it costs
that after the backfill finishes, because finding nothing still means
reading everything.
Also: replace the per-contact hop-prefix scan with the existing bucketed
matcher and memoize the 7-day chunk set (264 ms -> 35 ms on a synthetic
100k-row database, regression-locked by a test); move the dashboard's JS
and CSS to static files, which removes the CSP nonce requirement for the
bulk of the page; and give cleanup_old_stats a future-timestamp guard,
without which rows dated 2103 are never older than the cutoff and so
live forever.
Deletes the orphaned /stats page, unreachable from the nav and rendering
stub charts that never populated. /api/stats stays as a shim with every
key name intact plus Deprecation and Sunset headers.
All schema changes are additive, so a downgraded codebase can read the
data; it would however need the new schema_version rows removed, since
MigrationRunner rejects versions it does not know.
Changed the default setting for `auto_manage_contacts` from `false` to `device` across configuration files and updated related documentation. This ensures that the device handles auto-addition of contacts while the bot manages capacity. Adjusted comments and documentation to reflect this change for clarity and consistency.
Remaining findings from the same triage pass. None are security
relevant; each produces a wrong user-visible result.
- greeter: treat rollout_started_at as UTC. It is written by SQLite
CURRENT_TIMESTAMP, but datetime.timestamp() read the naive value as
local time, shifting the backfill cutoff by the host's offset (7h on
PT). Users who posted inside that window were never marked as already
greeted and could be sent a welcome they should not have received.
- greeter: write greeted_at in SQLite's own format. The rollout backfill
used isoformat() while every other path used CURRENT_TIMESTAMP. "T"
sorts above a space, so ORDER BY greeted_at interleaved the two
formats wrongly, corrupting duplicate cleanup and the web viewer's
recently-greeted list.
- greeter: let an empty `channels =` disable the command. BaseCommand
reads an empty value as disabled-on-channels, but the greeter
collapsed "key absent" and "key present but empty" into one fallback
and kept greeting via monitor_channels.
- greeter: stop comma-splitting greeting text. channel_greetings split
entries on ",", so "Public:Welcome to the mesh, {sender}!" was stored
as "Welcome to the mesh" with the placeholder silently dropped. A
fragment now starts a new entry only when the text before its first
colon looks like a channel name, which keeps commas, URLs and clock
times attached to the greeting they belong to.
- wxsim_parser: match condition abbreviations longest-first. Substring
matching in dict order let RAIN shadow CHNC. RAIN, so five conditions
lost their "chance" qualifier (rain, snow, drizzle, t-storm, and
FAIR-P.C.).
- wxsim_parser: re-anchor "now" on each parse. current_date and
current_year were fixed at construction and wx_command builds one
parser at startup, so after a few days forecast dates rolled back a
year and staleness checks read permanently true.
- thesportsdb_client: hold the rate-limit read/sleep/write under a lock.
Concurrent callers read the same last_request_time, slept the same
interval and fired together, bursting past the 2.1s throttle. This
exposes an unrelated request fan-out problem in fetch_league_scores;
filed in TODO.md rather than fixed here.
- alert_command: stop duplicating the first incident. The tail treated
any single-line buffer as "header only" and appended incidents[0],
but messages after the first carry no header, so a final chunk holding
one incident had incident 0 pasted onto it. The same confusion inside
the loop also let a second incident be appended past the 130-character
limit.
- feed: reject non-positive poll intervals. -1 is truthy and was stored,
and the poller's `now - last_check >= interval` then treats the feed
as permanently due and re-fetches the URL every cycle; 0 was ignored
while still reporting success. The web viewer, which is the primary
editor, had no validation at all, and a JSON null there raised a
TypeError that aborted the poll cycle for every feed rather than one.
feed_manager falls back to the default for rows written before this.
- transmission_tracker: age out confirmed transmissions that have
repeats. Cleanup removed only repeat_count == 0, so repeated records
accumulated for the lifetime of the process, which matters on a Pi
Zero. Repeat counts are already persisted to packet_stream, so the
longer 30-minute retention loses nothing.
- mesh_graph: weighted-merge avg_hop_position when promoting an edge.
Promotion overwrote the average with the single new observation, so an
edge averaging 2.0 over 4 observations became 9.0 instead of 3.4 and
skewed subsequent path scoring.
- multitest_command: show the path that ends exactly at the display LCP.
The suffix helpers are correct in isolation; the loss happens in the
cluster formatters, where _shrink_display_lcp refuses to shrink a
single-token LCP and the trunk then rendered as "96 ┐", meaning
"everything continues past here" and dropping the bare 96 route while
the header still counted it. Now uses the file's existing "├ common"
marker. Empty suffixes are also no longer handed to the nested
renderer, which mapped them onto the same [] as a route ending at the
inner LCP and drew a row for a route that did not exist.
- sports_mappings: stop shadowing seven unique team nicknames. In one
flat dict a repeated key silently drops the earlier team, so hawks
resolved to the NBA Hawks rather than the Seahawks alias it was added
as, blazers to Kamloops rather than Portland, and rockets to Kelowna
rather than Houston. First definition now wins for hawks, giants,
jets, rangers, kings, blazers and rockets; every shadowed team keeps
its unambiguous full-name alias. The 55 city and abbreviation
collisions (chicago, sf, la, ...) are genuinely ambiguous and stay
last-wins, now pinned by a test so a new collision fails loudly
instead of passing unnoticed.
Four defects in how configuration is written and displayed. All are
reachable without authentication when web_viewer_password is unset,
which is an explicitly supported setup.
- ini_writer: reject sections, keys and values that cannot survive the
round-trip. update_ini_values() wrote free text straight into the
file, so a newline in any value ended the key's line and everything
after it was re-parsed as INI on the next load. Saving a greeting of
"Welcome!\n[Injected]\npwned = true" through the plugin settings
endpoint created a real new section; a repeated section name bricked
startup with DuplicateSectionError. The check lives at the writer
because three separate callers reach it, and raises IniValueError so a
bad payload is a 400 instead of a half-written file. DEFAULT is
refused as well: its keys apply to every section, and
ConfigParser.add_section('DEFAULT') raises.
- settings_store, web_viewer: persist to disk before mirroring into the
in-memory config. The mirror ran first, so a value rejected by the
writer stayed live in the running process despite never reaching
disk. Both routes taking free-form input (the zombie and offline
alert emails) had the same ordering. The zombie route also caught
only OSError, so an IniValueError there escaped as a 500 rather than
the intended 400.
- config_snapshot: redact Discord webhook URLs. Neither
discord_webhook_urls nor the [DiscordBridge] bridge.<channel> keys
matched any redaction rule, so --show-config and /admin/config
printed live webhook secrets in full; anyone holding one can post to
the channel. Telegram's api_token was already redacted. The
bridge. prefix needs its own rule because the varying part is the
channel name, leaving no fixed stem for the substring match.
- mqtt_weather, packet_capture: verify TLS certificates by default.
Both called tls_set(cert_reqs=ssl.CERT_NONE) unconditionally, so the
broker username and password sent immediately afterwards were
readable by anyone able to intercept the connection.
BREAKING: brokers presenting self-signed certificates now fail to
connect until tls_insecure (mqtt_weather) or mqttN_tls_insecure
(packet_capture) is set to true. Both log a warning while enabled.
- Added a tracked `LICENSE` file and updated `pyproject.toml` to include license metadata.
- Enhanced `CHANGELOG.md` with recent changes and clarifications.
- Updated `config.ini.example` and related documentation to reflect clamping behavior for numeric limits in `[Feed_Manager]`.
- Improved startup validation to suggest corrections for unknown and misspelled keys.
- Refactored geocoding and HTTP request handling to run off-thread, preventing event loop stalls.
- Added thread safety to cache management in geocoding functions to avoid race conditions.
- Enhanced the `/api/contacts` endpoint to support pagination, allowing users to specify `page` and `page_size` parameters.
- Added search functionality to filter contacts based on a search term.
- Implemented sorting options for the contacts list, enabling users to sort by various fields.
- Updated the frontend to reflect these changes, including a new pagination UI and improved loading states.
- Introduced caching for multibyte hop chunks to optimize performance when retrieving contact badge evidence.
- Added tests to ensure the new features work as expected and maintain existing functionality.
Resolve the feeds.html conflict by keeping the branch's XSS-safe DOM
construction of the feed-details view and re-adding dev's per-feed and
reset-all error buttons via addEventListener instead of inline onclick,
since /feeds is a nonce-CSP page where inline handlers are blocked.
Update tests/test_feed_manager_extended.py::TestPollFeedPosting to patch
the SafeUrlPolicy.validate_async path (this branch's SSRF refactor) rather
than the removed module-level validate_external_url symbol, matching the
idiom already used throughout that test file.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Added max_posts_per_check to limit the number of items posted per check, defaulting to max_items_per_check for backward compatibility.
- Updated logic in FeedManager to process new items, ensuring that only the specified number of posts are made while examining a larger set of items.
- Enhanced emoji selection to prioritize per-item emojis from the API, falling back to heuristics based on feed names when absent.
- Introduced new truncation and substring functions for formatting messages, improving text handling in feed outputs.
- Added API endpoints to reset feed error counts globally or for individual feeds, enhancing error management capabilities in the web viewer.
Three independent wins, measured against a 1.4 GB production database
(714k observed_paths, 38.7k mesh_connections):
- Migration 0014: partial covering index on observed_paths for
bytes_per_hop >= 2. The multibyte evidence query was a full table
scan (2.2 s); with the index it reads only the index (77 ms). The
optional days filter is covered too via last_seen in slot 2.
- Compress responses with flask-compress when the client supports it.
/api/mesh/edges alone is 16.3 MB of JSON uncompressed, 3.9 MB
gzipped. Falls back gracefully (with a warning) if the package is
not installed.
- Fetch stats, edges, and nodes concurrently on the mesh page instead
of three serialized awaits. Node prefixes are now computed client-side
from public_key at the edge prefix length, which removes the
edges-before-nodes ordering dependency. Rendering still waits for
stats so initial framing can use the bot location.
Also call map.invalidateSize() before the initial fitBounds: if the
map was created while its container had no layout (e.g. a background
tab), Leaflet's cached zero width made fitBounds zoom to max.
Nodes occasionally carry wrong GPS, which stretched the initial
fit-to-all-nodes view across continents. /api/mesh/stats now exposes
the bot's configured position ([Bot] bot_latitude/bot_longitude), and
the mesh page BFSes the multi-byte-evidence subgraph from repeaters
within 30 km of the bot (nearest one as fallback), framing the initial
map view to that connected component. Falls back to the old fit when
the bot position is unset or the component is trivially small; other
mesh islands stay on the map, just outside the initial frame.
- Hot connections mode now also colors repeater dots with viridis by
their total observation count across touching edges (log-normalized,
same dark-mode ramp compression as edges); node popup gains a Path
Observations row and the heat legend a clarifying note
- Density pass: smaller dot base/cap (5..13 vs 6..16), edge widths
capped at 5px, and both dots and lines scale down to 55% as the map
zooms out (full size from zoom 12 in), rescaled live on zoomend
- Node degree and observation totals are now precomputed once per
filter pass (computeNodeStats) instead of per-node scans over all
edges, using the same endpoint resolution as the renderers
- Multi-byte evidence is now the default mode for the mesh graph
- Mesh display theme: follows the site theme until toggled on the page,
then remembered independently (localStorage); dark CARTO basemap,
dark Leaflet popups/controls/legends, dark graph canvas
- Hot connections: optional viridis edge coloring by log-scaled
observation count with a gradient legend; the near-black cold end of
the ramp is compressed away in dark mode so cold edges stay visible
- Single-row header (title + stat chips + icon controls) replaces the
title and stats-card rows; footer hidden on this page to maximize the
visualization viewport
- Mobile: page scrolls naturally instead of starving the fixed-height
flex container (which stranded the absolutely-positioned legends)
The old slider defaulted to floor(avg observations) — on heavy-tailed data
the average sits above the 75th percentile, so the first visit hid most of
the graph. Worse, the default (e.g. 98) exceeded the slider's max of 50:
the browser clamped the control while the label kept showing the unclamped
value, so the label lied about the active filter.
- slider position now maps logarithmically onto 1..max(observation_count),
rescaled whenever edge data loads, giving fine control at the low end
and full reach into the tail
- first-visit default is the median observation count of the loaded edges
- the persisted value is the threshold itself, so it stays meaningful when
the scale changes (e.g. switching evidence modes)
- label shows the threshold plus live 'shown/total edges' feedback
Single-byte repeater identity guessing has been persistently unreliable,
and mesh_connections flattens away evidence provenance (confirmed_2byte
never persists). observed_paths retains it: bytes_per_hop marks which
paths carry unambiguous multi-byte hops.
- /api/mesh/edges?evidence=multibyte derives edges purely from unique
multi-byte path observations (consecutive hop pairs, aggregated with a
distinct-path count); 2-byte edges coalesce into a 3-byte edge only on
a unique prefix match, mirroring MeshGraph.add_edge
- default mode tags each edge evidence=multibyte|singlebyte by key length
- Evidence filter on the mesh page (persisted with the other filters);
single-byte edges render dashed in map and graph views, with an
evidence line in tooltips and a legend entry
- client-side haversine fallback for edges without a stored distance
- prefix-compatible edge/node matching so degree sizing and node details
work when edge and node prefixes differ in resolution
- /api/mesh/stats reports multibyte_edges; shown on the Edges stat card
Adds device-level companion settings to /radio, headlined by the response
path hash size (mode 0-2 = 1-3 bytes per hop) wired to the existing
firmware config endpoints. New sections: Identity & Adverts (name, advert
lat/lon, location policy, zero-hop/flood advert buttons), Mesh Behavior
(extra ACKs, telemetry permissions), and write-only Advanced Tuning
(RX delay base, airtime factor, sent x1000 per the wire format).
Backend: GET /api/radio/params now returns the full SELF_INFO node fields;
POST accepts the new fields with validation; new POST /api/radio/advert;
scheduler ops handle the writes (other-params group as one read-modify-write
frame so partial updates never clobber device state).
Config-managed settings are not writable from the panel: new-contact mode
is owned by [Bot] auto_manage_contacts and shown read-only; the node name
is locked while bot_name + auto_update_device_name manage it. loop.detect
is removed from the firmware endpoints entirely - it is repeater/room-server
CLI config, and companion custom vars map to sensor settings.
Also: dark-mode styling for disabled form fields (base.html), TASK-01 guard
test updated for the deliberate reintroduction, 23 endpoint/template tests.
- reload_config: clear all sections before re-reading so keys deleted from
config.ini (e.g. rows removed in the settings UI) don't survive in memory
(ConfigParser.read merges instead of replacing)
- reload_config: re-instantiate command plugins so settings cached in
__init__ (including 'enabled') take effect on reload
- /api/plugins save: treat dynamic_sections / repeating_blocks absent from
the payload as 'don't touch' instead of 'delete all managed keys'
- settings view: respect per-plugin settings_enabled_default so opt-in
commands (Announcements, Greeter) display as off when unconfigured
- ruff: drop unused import, organize command_manager imports
- add tests for ini_writer, settings_schema, settings_store and the
/api/plugins endpoints (38 cases) plus reload regression tests
- Updated the `[Rain_Command]` section in `config.ini.example` to include support for snow alongside rain, improving the command's functionality.
- Enhanced documentation for the rain command to reflect the new snow alias and clarify response behavior based on the selected keyword.
- Added a new `collect_stats` option in the `[Stats_Command]` section, allowing stats collection to be enabled independently of the user-facing command, with updated documentation to explain its behavior.
- Improved the web viewer documentation to clarify how stats are collected and displayed, ensuring users understand the configuration options.
- Added detailed diagnostics for database initialization failures in `DBManager`, improving error logging with filesystem checks.
- Implemented a new method `_resolve_viewer_db_path` in `WebViewerIntegration` to determine the database path for the viewer subprocess.
- Introduced `_preflight_database_path` to validate the viewer database path, ensuring the parent directory exists, is writable, and is a directory.
- Added unit tests to verify logging behavior for missing or invalid database paths in both `DBManager` and `WebViewerIntegration`.
- Introduced a new configuration option `announce_disallowed` to allow notifications for goals overturned by VAR.
- Updated `WorldCupLiveService` to handle disallowed goals, including formatting for previously announced scorers.
- Enhanced `ESPNClient` to ensure accurate goal tracking and state management.
- Added unit tests to validate the new disallowed goal functionality and ensure correct behavior during matches.
- Updated `contacts.html` to escape HTML for contact usernames and advertisement data, preventing potential XSS vulnerabilities.
- Modified `mesh.html` to escape node names and prefixes in tooltips, ensuring safe rendering in the vis-network.
- Removed unnecessary parameters from the `showDeleteConfirmation` method to derive the username directly from the contact data.
- Introduced a new utility function `public_key_has_prefix` to check if a public key starts with a specified prefix in a case-insensitive manner.
- Updated the `PathCommand` class to utilize the new utility function for public key prefix matching, enhancing code readability and maintainability.
- Refactored logic in `PathCommand` and `BotDataViewer` to streamline the handling of recent repeaters based on recency scores, improving clarity in the filtering process.
- Introduced a new configuration option `multibyte_monitor_enabled` in `config.ini.example` to control the visibility of the multibyte monitor page and API endpoints.
- Implemented the `/multibyte-rollout` and `/api/multibyte-rollout` routes in the web viewer, which are accessible only when the multibyte monitor feature is enabled.
- Updated documentation in `web-viewer.md` to reflect the new feature and its configuration.
- Added tests to ensure the multibyte routes are disabled by default and accessible when enabled.
- Implemented validation for the `channel_name` field to ensure it is not empty when updating feeds, raising a ValueError if validation fails.
- Added test cases to verify that updates to `channel_name` persist correctly and that attempts to set an empty `channel_name` are rejected with an appropriate error response.
Prevent exportView runtime failure by restoring the download anchor setup and move radio param imports to module scope so Ruff passes on the PR changes.
Made-with: Cursor
agessaman