Files
ukmesh/viewshed-worker/tests/test_side_effect_markers.py
T
hermes-gadgetandhermes-gadget e069f6b102 fix: close Wave 3 medium findings (owner auth, deploy, newuser, alerts, gitleaks, viewshed) (#64)
* fix(owner): Wave 3 mediums — password handling, deploy rollback, newuser races, session revocation

- BUG-017: stop trimming MQTT passwords (frontend + backend). A valid
  broker password may begin/end with whitespace; trimming made it
  permanently unauthenticatable. Username trimming unchanged.
- BUG-013: deploy-website.sh now set -Eeuo pipefail with an EXIT rollback
  trap armed only after the pin mutates; container-down and bundle-mismatch
  paths fail loudly and restore the old pin + verify the restored service.
- BUG-012: newuser.sh installs an EXIT trap (ERR does not fire on explicit
  exit 1 from die()) so every post-mutation failure rolls back; rollback is
  idempotent and a no-op before any mutation flag. Disarm points clear EXIT.
- BUG-011: newuser.sh re-reads + re-validates OWNER_MQTT_USERNAME_MAP under
  the re-acquired lock after the long unlocked discovery window, then merges
  the new grant into the current map — concurrent provisioning runs can no
  longer be overwritten by a stale snapshot.
- BUG-010: owner sessions are now v3 cookies carrying a credential
  generation (Redis-backed). When the broker rejects a previously-valid
  password (revocation detected), the generation bumps and every older
  session is rejected on the next request. TTL shortened 30d -> 7d.

Verified: tsc clean, 310/310 backend tests pass, bash -n on both scripts.

* fix(alert-receiver): durable alert delivery with bounded retry + dead-letter (BUG-014)

Forwarding was fire-and-forget: failures logged asynchronously after HTTP 202,
/healthz stayed green, and alerts could be archived to the local JSONL while
operators never saw them. Now:
- Every receipt is enqueued for delivery with bounded exponential backoff
  (ALERT_FORWARD_MAX_ATTEMPTS=5, base 1s, cap 60s) and dead-lettered to
  receipts.jsonl.dead after exhausting attempts.
- /healthz keeps returning 200 (compose wget healthcheck must not restart
  the container) but the body reports degraded status + detail; new /readyz
  returns 503 when ALERT_FORWARD_URL is unset (archive-only), no successful
  forward since startup, or alerts stuck undelivered >5min.

Verified: tsc clean, 310/310 tests pass.

* fix(ci): classify reviewed public channel keys + fixtures in gitleaks (BUG-009)

The nightly full-history secret scan flagged 42 findings spanning the
documented community channel keys (VALIDATED_CHANNELS — intentionally
public, each verified to decrypt real UK Mesh group text), fake test
fixtures, and a deployed-commit SHA in the website live manifests. That
made a genuine credential easy to dismiss among expected hits.

- Rule-scoped allowlists with exact fingerprints (regexTarget: secret,
  anchored full-value matches) for the Public channel key, test fixtures
  (0123456789... / abcdef0123...), and the manifest commit SHA.
- Structure-exact line allowlist for channelRegistry.ts VALIDATED_CHANNELS.
- Verified locally with gitleaks 8.24.3 full-history scan: 42 -> 0 findings.

* fix(viewshed): side-effect completion markers prevent skipped link jobs (BUG-006)

store_coverage() commits on an autocommit connection, then the worker queues
physical-link jobs and publishes Redis notifications. A Redis failure after
the DB commit NACKed the job; on retry already_calculated() saw the coverage
row and returned early — link work and frontend notifications were skipped
forever.

- Record a Redis completion marker (viewshed:side-effects:<node>) only after
  EVERY side effect succeeds.
- On the already_calculated early return, a missing marker triggers an
  idempotent replay: link jobs are re-enqueued from the stored node position
  (admission is idempotent) and coverage_update/node_upsert notifications are
  re-published from the stored coverage row.
- Redis read failures are treated as incomplete (replay attempt re-raises and
  NACKs rather than silently skipping).
- Added tests/test_side_effect_markers.py (5 tests) with a conftest that stubs
  osgeo/psycopg2 so pure-logic worker tests run without GDAL (CI keeps real
  GDAL via setdefault). Verified: 5/5 new tests pass; full suite 34 passed,
  2 GDAL-required terrain tests fail only in stub env (pass in CI image).

---------

Co-authored-by: hermes-gadget <hermes-gadget@users.noreply.github.com>
2026-08-11 22:50:16 +01:00

126 lines
4.1 KiB
Python

"""Focused unit test for BUG-006 side-effect marker + replay logic.
Runs without GDAL by importing only the pure helper functions via a stub
module shim. Covers:
- side_effects_complete / mark_side_effects_complete round-trip
- replay path publishes both notifications and re-enqueues link jobs
- success path marks completion
"""
import json
import sys
import types
import unittest
from unittest import mock
# ---- Stub the heavy third-party modules so we can import worker.py's helpers ----
for mod_name in ('osgeo', 'osgeo.gdal', 'osgeo.ogr', 'osgeo.osr'):
sys.modules.setdefault(mod_name, types.ModuleType(mod_name))
sys.modules.setdefault('psycopg2', types.ModuleType('psycopg2'))
sys.modules.setdefault('psycopg2.extras', types.ModuleType('psycopg2.extras'))
import worker # noqa: E402
class FakeRedis:
def __init__(self):
self.data = {}
self.published = []
def exists(self, key):
return 1 if key in self.data else 0
def set(self, key, value, ex=None):
self.data[key] = value
def publish(self, channel, message):
self.published.append((channel, json.loads(message)))
class FakeCursor:
def __init__(self, rows):
self._rows = rows
self._idx = 0
self.sql = None
def __enter__(self):
return self
def __exit__(self, *args):
return False
def execute(self, sql, params=None):
self.sql = sql
def fetchone(self):
if self._idx < len(self._rows):
row = self._rows[self._idx]
self._idx += 1
return row
return None
class FakeDb:
def __init__(self, rows):
self.rows = rows
def cursor(self):
return FakeCursor(self.rows)
class SideEffectMarkerTest(unittest.TestCase):
def test_marker_round_trip(self):
r = FakeRedis()
node = 'A' * 64
self.assertFalse(worker.side_effects_complete(r, node))
worker.mark_side_effects_complete(r, node)
self.assertTrue(worker.side_effects_complete(r, node))
def test_marker_read_failure_treated_as_incomplete(self):
r = FakeRedis()
r.exists = mock.Mock(side_effect=RuntimeError('redis down'))
self.assertFalse(worker.side_effects_complete(r, 'B' * 64))
def test_replay_publishes_notifications_and_marks_complete(self):
r = FakeRedis()
node = 'C' * 64
geom = {'type': 'Polygon', 'coordinates': []}
strength = {'s1': {'type': 'Polygon', 'coordinates': []}}
db = FakeDb([
(geom, strength, 5000.0, 12.0), # coverage row
(54.0, -1.5), # node position for link replay
])
with mock.patch.object(worker, 'enqueue_physical_link_jobs_for_node', return_value=2) as enqueue:
worker.replay_coverage_side_effects(db, r, node)
channels = [ch for ch, _ in r.published]
self.assertEqual(channels, [worker.LIVE_CHANNEL, worker.LIVE_CHANNEL])
types_ = [m['type'] for _, m in r.published]
self.assertIn('coverage_update', types_)
self.assertIn('node_upsert', types_)
self.assertTrue(worker.side_effects_complete(r, node))
enqueue.assert_called_once()
def test_replay_without_position_skips_links_but_notifies(self):
r = FakeRedis()
node = 'D' * 64
geom = {'type': 'Polygon', 'coordinates': []}
db = FakeDb([
(geom, None, None, 5.0), # coverage row, NULL strength/radius
(None, None), # node position missing
])
with mock.patch.object(worker, 'enqueue_physical_link_jobs_for_node') as enqueue:
worker.replay_coverage_side_effects(db, r, node)
self.assertEqual(len(r.published), 2)
self.assertTrue(worker.side_effects_complete(r, node))
enqueue.assert_not_called()
def test_replay_missing_coverage_row_is_noop(self):
r = FakeRedis()
db = FakeDb([None])
worker.replay_coverage_side_effects(db, r, 'E' * 64)
self.assertEqual(r.published, [])
self.assertFalse(worker.side_effects_complete(r, 'E' * 64))
if __name__ == '__main__':
unittest.main()