Commit Graph
172 Commits
Author SHA1 Message Date
gadgethd 2fa80fa10d fix(ws): release client state when initial loading fails (#90) 2026-09-23 11:52:15 +01:00
gadgethd 05803a5e74 perf(health): combine repeated packet diagnostic scans (#88) 2026-09-23 11:52:11 +01:00
gadgethd 822fb66af0 perf(cache): skip expiry scans while cached entries are fresh (#86) 2026-09-23 11:52:07 +01:00
hermes-gadgetandgadgethd 2c2b9d9de4 fix(owner): make grant reconciliation rollout-safe (#79)
Co-authored-by: gadgethd <111318106+gadgethd@users.noreply.github.com>
2026-08-31 16:18:05 +01:00
dependabot[bot] e2ee775fae chore(deps-dev): bump @types/pg (#74)
Bumps the backend-dependencies group in /backend with 1 update: [@types/pg](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/pg).


Updates `@types/pg` from 8.21.0 to 8.23.1
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/pg)

---
updated-dependencies:
- dependency-name: "@types/pg"
  dependency-version: 8.23.1
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: backend-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-29 11:22:21 +01:00
gadgethd aee892cc82 fix(ws): improve initial-state size estimator accuracy and drop duplicate viable_pairs 2026-08-27 23:59:16 +00:00
gadgethd d864d0a15b infra: migrate preconnects, CSP, and service worker from CARTO to OpenFreeMap 2026-08-27 23:48:01 +00:00
gadgethd 44f52fd7e4 fix(api): coerce COUNT(*) string to numeric adverts_in_window 2026-08-27 23:48:01 +00:00
gadgethd 79e5e4ad0b feat(repeaters): top 10 adverting repeaters (24h) in repeater search empty state 2026-08-27 23:48:01 +00:00
gadgethd bf7d7ad85e fix: anchor confirmed paths and stabilize observer dots 2026-08-27 23:48:01 +00:00
gadgethd b7bd60f629 fix(owner): preserve authorization inventory across reset 2026-08-27 23:48:01 +00:00
gadgethd 26781dc7d9 fix(db): make reset schema and lifecycle reproducible 2026-08-27 23:48:01 +00:00
dependabot[bot] 5fbc45be2b chore(deps): bump the backend-dependencies group (#66)
Bumps the backend-dependencies group in /backend with 2 updates: [pg](https://github.com/brianc/node-postgres/tree/HEAD/packages/pg) and [tsx](https://github.com/privatenumber/tsx).


Updates `pg` from 8.22.0 to 8.23.0
- [Changelog](https://github.com/brianc/node-postgres/blob/master/CHANGELOG.md)
- [Commits](https://github.com/brianc/node-postgres/commits/pg@8.23.0/packages/pg)

Updates `tsx` from 4.23.11 to 4.23.12
- [Release notes](https://github.com/privatenumber/tsx/releases)
- [Changelog](https://github.com/privatenumber/tsx/blob/master/release.config.cjs)
- [Commits](https://github.com/privatenumber/tsx/compare/v4.23.11...v4.23.12)

---
updated-dependencies:
- dependency-name: pg
  dependency-version: 8.23.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: backend-dependencies
- dependency-name: tsx
  dependency-version: 4.23.12
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: backend-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-27 09:41:37 +01:00
gadgethd 181a41e891 test(dataLifecycle): packets retention now 30 days per 2026-08-12 policy 2026-08-12 20:25:30 +00:00
gadgethd b8628a78bf feat(telemetry+gdpr): shared client error capture, console.error hook, privacy page, retention jobs, packet_paths store
- clientErrors.ts: window.onerror/unhandledrejection for BOTH domains via main.tsx
  (was app-only in App.tsx); resource-failure classification; telemetry=off gate;
  console.error hook (kind warning) with terrain-tiles noise filter; page field
  now pathname-only (no query strings)
- PrivacyPage at /privacy linked from footer + cookie banner; docs/privacy.md LIA
- retention: packet_decryptions added to row-table retention targets; packets
  180d->30d in lifecycle policies; migration 050 schedules timescaledb retention
  jobs (packets 30d, node_status 180d, node_neighbor 7d) that never existed
- migration 050: packet_paths hypertable (content-stripped path store, no
  retention) + backfill of 2.56M path-bearing packets + ingest dual-write
2026-08-12 20:23:13 +00:00
gadgethd fb787d3f9d fix: guard packet-privacy rewrite to privacy transitions only (049); emit maplibre-gl worker asset; serve .mjs as JS 2026-08-12 11:25:42 +00:00
hermes-gadgetandhermes-gadget e069f6b102 fix: close Wave 3 medium findings (owner auth, deploy, newuser, alerts, gitleaks, viewshed) (#64)
* fix(owner): Wave 3 mediums — password handling, deploy rollback, newuser races, session revocation

- BUG-017: stop trimming MQTT passwords (frontend + backend). A valid
  broker password may begin/end with whitespace; trimming made it
  permanently unauthenticatable. Username trimming unchanged.
- BUG-013: deploy-website.sh now set -Eeuo pipefail with an EXIT rollback
  trap armed only after the pin mutates; container-down and bundle-mismatch
  paths fail loudly and restore the old pin + verify the restored service.
- BUG-012: newuser.sh installs an EXIT trap (ERR does not fire on explicit
  exit 1 from die()) so every post-mutation failure rolls back; rollback is
  idempotent and a no-op before any mutation flag. Disarm points clear EXIT.
- BUG-011: newuser.sh re-reads + re-validates OWNER_MQTT_USERNAME_MAP under
  the re-acquired lock after the long unlocked discovery window, then merges
  the new grant into the current map — concurrent provisioning runs can no
  longer be overwritten by a stale snapshot.
- BUG-010: owner sessions are now v3 cookies carrying a credential
  generation (Redis-backed). When the broker rejects a previously-valid
  password (revocation detected), the generation bumps and every older
  session is rejected on the next request. TTL shortened 30d -> 7d.

Verified: tsc clean, 310/310 backend tests pass, bash -n on both scripts.

* fix(alert-receiver): durable alert delivery with bounded retry + dead-letter (BUG-014)

Forwarding was fire-and-forget: failures logged asynchronously after HTTP 202,
/healthz stayed green, and alerts could be archived to the local JSONL while
operators never saw them. Now:
- Every receipt is enqueued for delivery with bounded exponential backoff
  (ALERT_FORWARD_MAX_ATTEMPTS=5, base 1s, cap 60s) and dead-lettered to
  receipts.jsonl.dead after exhausting attempts.
- /healthz keeps returning 200 (compose wget healthcheck must not restart
  the container) but the body reports degraded status + detail; new /readyz
  returns 503 when ALERT_FORWARD_URL is unset (archive-only), no successful
  forward since startup, or alerts stuck undelivered >5min.

Verified: tsc clean, 310/310 tests pass.

* fix(ci): classify reviewed public channel keys + fixtures in gitleaks (BUG-009)

The nightly full-history secret scan flagged 42 findings spanning the
documented community channel keys (VALIDATED_CHANNELS — intentionally
public, each verified to decrypt real UK Mesh group text), fake test
fixtures, and a deployed-commit SHA in the website live manifests. That
made a genuine credential easy to dismiss among expected hits.

- Rule-scoped allowlists with exact fingerprints (regexTarget: secret,
  anchored full-value matches) for the Public channel key, test fixtures
  (0123456789... / abcdef0123...), and the manifest commit SHA.
- Structure-exact line allowlist for channelRegistry.ts VALIDATED_CHANNELS.
- Verified locally with gitleaks 8.24.3 full-history scan: 42 -> 0 findings.

* fix(viewshed): side-effect completion markers prevent skipped link jobs (BUG-006)

store_coverage() commits on an autocommit connection, then the worker queues
physical-link jobs and publishes Redis notifications. A Redis failure after
the DB commit NACKed the job; on retry already_calculated() saw the coverage
row and returned early — link work and frontend notifications were skipped
forever.

- Record a Redis completion marker (viewshed:side-effects:<node>) only after
  EVERY side effect succeeds.
- On the already_calculated early return, a missing marker triggers an
  idempotent replay: link jobs are re-enqueued from the stored node position
  (admission is idempotent) and coverage_update/node_upsert notifications are
  re-published from the stored coverage row.
- Redis read failures are treated as incomplete (replay attempt re-raises and
  NACKs rather than silently skipping).
- Added tests/test_side_effect_markers.py (5 tests) with a conftest that stubs
  osgeo/psycopg2 so pure-logic worker tests run without GDAL (CI keeps real
  GDAL via setdefault). Verified: 5/5 new tests pass; full suite 34 passed,
  2 GDAL-required terrain tests fail only in stub env (pass in CI image).

---------

Co-authored-by: hermes-gadget <hermes-gadget@users.noreply.github.com>
2026-08-11 22:50:16 +01:00
06b62a3d21 chore(deps): bump the backend-dependencies group across 1 directory with 9 updates (#39)
* chore(deps): bump the backend-dependencies group across 1 directory with 9 updates

Bumps the backend-dependencies group with 9 updates in the /backend directory:

| Package | From | To |
| --- | --- | --- |
| [express](https://github.com/expressjs/express) | `4.22.2` | `5.2.1` |
| [@types/express](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/express) | `4.17.25` | `5.0.6` |
| [express-rate-limit](https://github.com/express-rate-limit/express-rate-limit) | `7.5.1` | `8.6.2` |
| [ioredis](https://github.com/redis/ioredis) | `5.11.1` | `6.0.0` |
| [ws](https://github.com/websockets/ws) | `8.21.0` | `8.21.3` |
| [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) | `20.19.43` | `26.2.0` |
| [@types/pg](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/pg) | `8.20.0` | `8.21.0` |
| [tsx](https://github.com/privatenumber/tsx) | `4.23.0` | `4.23.11` |
| [typescript](https://github.com/microsoft/TypeScript) | `5.9.3` | `7.0.2` |



Updates `express` from 4.22.2 to 5.2.1
- [Release notes](https://github.com/expressjs/express/releases)
- [Changelog](https://github.com/expressjs/express/blob/master/History.md)
- [Commits](https://github.com/expressjs/express/compare/v4.22.2...v5.2.1)

Updates `@types/express` from 4.17.25 to 5.0.6
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/express)

Updates `express-rate-limit` from 7.5.1 to 8.6.2
- [Release notes](https://github.com/express-rate-limit/express-rate-limit/releases)
- [Commits](https://github.com/express-rate-limit/express-rate-limit/compare/v7.5.1...v8.6.2)

Updates `ioredis` from 5.11.1 to 6.0.0
- [Release notes](https://github.com/redis/ioredis/releases)
- [Changelog](https://github.com/redis/ioredis/blob/main/CHANGELOG.md)
- [Commits](https://github.com/redis/ioredis/compare/v5.11.1...v6.0.0)

Updates `ws` from 8.21.0 to 8.21.3
- [Release notes](https://github.com/websockets/ws/releases)
- [Commits](https://github.com/websockets/ws/compare/8.21.0...8.21.3)

Updates `@types/express` from 4.17.25 to 5.0.6
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/express)

Updates `@types/node` from 20.19.43 to 26.2.0
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

Updates `@types/pg` from 8.20.0 to 8.21.0
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/pg)

Updates `tsx` from 4.23.0 to 4.23.11
- [Release notes](https://github.com/privatenumber/tsx/releases)
- [Changelog](https://github.com/privatenumber/tsx/blob/master/release.config.cjs)
- [Commits](https://github.com/privatenumber/tsx/compare/v4.23.0...v4.23.11)

Updates `typescript` from 5.9.3 to 7.0.2
- [Release notes](https://github.com/microsoft/TypeScript/releases)
- [Commits](https://github.com/microsoft/TypeScript/commits)

---
updated-dependencies:
- dependency-name: "@types/express"
  dependency-version: 5.0.6
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: backend-dependencies
- dependency-name: "@types/express"
  dependency-version: 5.0.6
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: backend-dependencies
- dependency-name: "@types/node"
  dependency-version: 26.1.2
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: backend-dependencies
- dependency-name: "@types/pg"
  dependency-version: 8.20.3
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: backend-dependencies
- dependency-name: express
  dependency-version: 5.2.1
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: backend-dependencies
- dependency-name: express-rate-limit
  dependency-version: 8.6.1
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: backend-dependencies
- dependency-name: ioredis
  dependency-version: 6.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: backend-dependencies
- dependency-name: tsx
  dependency-version: 4.23.5
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: backend-dependencies
- dependency-name: typescript
  dependency-version: 7.0.2
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: backend-dependencies
- dependency-name: ws
  dependency-version: 8.21.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: backend-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix(backend): Express 5 / TS 7 type migrations for dependency bump

- req.params values are string|string[] in Express 5 types: normalize
  array params before validation in exports.ts and spam.ts routes
- ioredis v6 zrange overloads require string stop: pass '0'
- workerPool error handler: narrow unknown err before use

Verified: tsc --noEmit clean, 309/309 backend tests pass.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: hermes-gadget <whatinthe3duk@gmail.com>
Co-authored-by: hermes-gadget <hermes-gadget@users.noreply.github.com>
2026-08-11 21:42:30 +01:00
hermes-gadgetandgadgethd ab6372b8a9 fix(ci): pin gitleaks-action to valid commit after upstream force-push (#63)
* fix(ci): pin gitleaks-action to valid commit after upstream force-push

Upstream deleted dcedce43 (force-push), so every ci.yml run fails at
workflow-parse with 0 jobs. The dependabot actions-group bump (#30)
contains this fix but cannot merge whole: its docker/* actions require
node24, which GitHub runners do not support yet. Pin gitleaks alone
(ff98106e is node20).

* fix(ci): downgrade v7 actions to node20-compatible versions

actions/checkout@v7, setup-node@v7, setup-python@v7 require node24,
which GitHub-hosted runners do not support yet — every ci.yml run has
failed at workflow-parse (0 jobs) since the Aug 3 v7 bump. Pin to the
latest node20 versions (checkout v4, setup-node v4, setup-python v5).
Also pins gitleaks-action to a valid commit (upstream force-pushed away
dcedce43).

* ci: noop retrigger

* fix(ci): escape literal ${{ in bash string so GitHub parser accepts workflow

The 'Validate tracked build and Compose inventory' step matched literal
${{ inside run-block strings, which GitHub's expression parser reads as
the start of an expression (${{'* is invalid) — the workflow fails at
parse time with 0 jobs. Split the literal as '$''{{' (bash concatenates
adjacent quoted strings at runtime; the source no longer contains a
contiguous ${{).

* fix(ci): resolve all three pre-existing ukmesh CI failures

Backend (packetBatchWriteCoalesce):
- 045: restore nodes_public_visibility_generation trigger — 015 is
  superseded by 044 on fresh DBs so the trigger was never created,
  leaving generation/visibility_generation out of sync and public
  packet reads returning empty.
- 046: restore packet privacy classification — the 042 fence design
  assumes sync_private_node_prefixes rewrites packets on privacy
  change, but 026's version only maintains prefixes. Node flips to
  private left old packets visible; direct SQL inserts were never
  classified. Restore the packet rewrite in the sync trigger and add
  a BEFORE INSERT trigger mirroring the batch path's is_private /
  visibility_ok computation.
- packetBatch.integration.test.ts: exclude the prefix-cache refresh
  query from statementCount (matches unit-test convention).

Frontend e2e:
- public.spec.ts: assert the TopologyMap component's actual labels
  ('Geographic repeater topology map', '2 mapped repeaters · 2
  observed relationships') instead of the pre-map SVG graph labels.

Workers (pip install):
- viewshed-worker: bump numpy 1.26.4→2.3.5, scipy 1.13.1→1.16.3,
  psycopg2-binary 2.9.10→2.9.11 (cp314 wheels); the base gdal image
  ships Python 3.14 so the old pins had no wheels. Inherit shapely
  from the image's python3-shapely apt package (no cp314 wheel
  exists; source builds are GEOS 3.14-incompatible).

* fix(ci): restore data-plane services for compose validation and smoke test

Commit 8c5e1c8 split the long-lived data-plane services (timescaledb,
mosquitto, redis, mosquitto-reloader) into the external meshcore-infra
project on the live host, but the Workers-and-Compose CI job still
asserted their presence in the app compose project (max_worker_processes
check, inventory check, smoke-test execs). CI had been broken since
Aug 3 so the mismatch was never caught.

Add docker-compose.ci.yml — a CI-only overlay restoring the four
data-plane services from their pre-split definitions — and point every
compose invocation in the Workers-and-Compose job at
'-f docker-compose.yml -f docker-compose.ci.yml'.

Validated locally: merged config parses, the jq assertions and the full
inventory (db-migrate backend app-ukmesh website-ukmesh website-dev
mesh-health-check mosquitto-reloader link-worker link-backfill-worker
hopreach timescaledb mosquitto) all pass.

* fix(ci): include dev profile in compose inventory check

website-dev is a dev-profile service; the inventory gate compared the
base-config service list, so the merged stack never matched. This gate
has been red since the workflow was first written (CI parse-broken from
Aug 3 until the gitleaks pin landed).

* fix(ci): db-migrate must wait for timescaledb health

The Aug 9 data-plane split removed db-migrate's depends_on along with
the timescaledb service definition, so the migration runner raced the
fresh database init in the empty-volume smoke test and failed on the
not-yet-created base schema. Restore the pre-split condition:
timescaledb service_healthy.

* fix(ci): restore internal TCP MQTT listener for smoke test + exporter

Commit 40e843f dropped the 1883 TCP listener, breaking the CI smoke
(mosquitto_pub targets 1883) and the Prometheus mosquitto exporter.
The listener is internal-only: compose never publishes 1883 on the
host, and the password/ACL policy still applies.

---------

Co-authored-by: gadgethd <111318106+gadgethd@users.noreply.github.com>
2026-08-11 19:55:29 +01:00
hermes-gadgetandgadgethd 2fa099ff8b fix: unblock fresh-DB migrations + retire removed CI components + restore frontend CI (#59)
* fix: unblock fresh database migrations (#44)

* fix: remove retired CI components (#48)

* fix: restore frontend CI and RF control (#49)

---------

Co-authored-by: gadgethd <111318106+gadgethd@users.noreply.github.com>
2026-08-11 03:38:57 +01:00
gadgethd 1d6b9ae1ff perf: retire raw multibyte chart decoding 2026-08-10 00:30:00 +00:00
gadgethd 6657ddccb9 fix: count newly written multibyte facts 2026-08-10 00:23:44 +00:00
gadgethd d91d688f4a perf: decode multibyte backfill set-wise 2026-08-10 00:07:19 +00:00
gadgethd 042b343187 fix: bound multibyte fact backfill statements 2026-08-09 23:42:55 +00:00
gadgethd feb582be57 chore: bump ip-address + nanoid to clear npm audit findings 2026-08-09 23:03:00 +00:00
gadgethd 9bb7a85552 fix: allow full WebSocket canary completion 2026-08-09 20:58:05 +00:00
gadgethd e523224e4b fix: defer full synthetic canary until ready 2026-08-09 19:48:21 +00:00
gadgethd 549aebff91 fix: prune retired worker health status 2026-08-09 19:44:33 +00:00
gadgethd 4d9872768a perf: reduce monitoring load and retire path history 2026-08-09 19:37:59 +00:00
gadgethd d0fca1ea88 fix: format alert forwards for Discord 2026-08-09 18:30:47 +00:00
gadgethd a1c8016397 perf: remove retired ML path learner (score read, dashboard, compose, schema) 2026-08-09 17:57:04 +00:00
gadgethd ca7a23748c perf: index staged path model keys 2026-08-09 17:02:15 +00:00
gadgethd c8099f8932 fix: publish path models on analytics pool 2026-08-09 16:53:48 +00:00
gadgethd 1a996f82ab fix: bound multibyte backfill to explicit chunks 2026-08-09 16:46:47 +00:00
gadgethd 13fb33bb51 perf: serialize heavy database workloads 2026-08-09 16:23:41 +00:00
gadgethd 6e07077e60 perf: make link worker change driven 2026-08-09 16:16:43 +00:00
gadgethd 6b0aa340ed perf: eliminate identity and ingest row probes 2026-08-09 16:11:27 +00:00
gadgethd 91999dab0b fix: keep public visibility generation stable 2026-08-09 15:54:34 +00:00
gadgethd de4664d123 fix: stabilize chart visibility refresh 2026-08-09 15:44:29 +00:00
gadgethd 046113a6c9 fix: make chart regeneration observable 2026-08-09 15:37:07 +00:00
gadgethd dd61ccb8ef fix: grant bounded broker uptime reads 2026-08-09 15:30:46 +00:00
gadgethd 75b6a607bd perf: skip unchanged identity alias rewrites 2026-08-09 15:30:37 +00:00
gadgethd f9c02dfbf0 perf: combine and cache topology 2026-08-09 15:25:27 +00:00
gadgethd 92f2f11dfd fix: honor chart analytics timeout 2026-08-09 15:20:51 +00:00
gadgethd dbe1d304ff perf: publish path learning deltas 2026-08-09 15:09:18 +00:00
gadgethd 4933928cf3 fix: serve charts during generation refresh 2026-08-09 14:53:09 +00:00
gadgethd 61d3466a4a fix: bound multibyte backfill by chunk 2026-08-09 14:30:47 +00:00
gadgethd b5e329eb66 perf: persist multibyte chart facts 2026-08-09 14:20:09 +00:00
gadgethd 2a4dbe586e fix: prevent chart regen on warm restart 2026-08-09 13:35:58 +00:00
gadgethd 2e76daf37a perf: batch health and synthetic persistence 2026-08-09 13:20:16 +00:00