Compare commits

...

8 Commits

Author SHA1 Message Date
Kpa-clawbot 2a07716885 fix(geo-prune): relocate DELETE to ingestor via file-marker queue (#738 / #1289)
GREEN for the invariant test added in the prior commit. Resolves the dead-on-
arrival blocker introduced by PR #738: the server opens SQLite mode=ro after
#1283/#1289, so the `confirm=true` path of `/api/admin/prune-geo-filter` would
fail in production with "attempt to write a readonly database". The author's
tests only passed because setupTestDB uses `:memory:` without mode=ro.

## Architecture: file-marker queue (chosen over IPC/HTTP proxy)

Server → marker file → ingestor → result file → server. New package
`internal/prunequeue` defines the on-disk protocol:

  <dir(dbPath)>/prune-requests/request-<id>.json   (written by server)
  <dir(dbPath)>/prune-requests/result-<id>.json    (written by ingestor)

Atomic via os.Rename. The ingestor removes the request file as part of writing
the result, so the directory naturally drains.

Why file markers over an internal localhost HTTP socket:
  * No new listener / port / auth surface to manage.
  * Inherits the same backup/permissions story as the SQLite file — they share
    a directory and a deployment.
  * Survives ingestor restarts: pending requests are processed on next boot.
  * Trivial to inspect / clear with `ls` / `trash` during ops.
  * Matches the existing maintenance-ticker pattern in cmd/ingestor/main.go.

## HTTP surface (unchanged for clients in spirit)

POST /api/admin/prune-geo-filter
  * dry-run (default): same response shape as before — preview list.
  * confirm=true: now returns 202 Accepted + {requestId, statusUrl, count, nodes}.
    Body must still carry the pubkeys snapshot from the preview (TOCTOU guard
    preserved — the ingestor honors the list verbatim, does NOT re-evaluate
    geo_filter membership).

GET /api/admin/prune-geo-filter/status?id=<id>  (NEW, requires API key)
  * pending: 200 {requestId, status:"pending"}
  * done:    200 {requestId, status:"done", deleted, requestedAt, completedAt}
  * error:   200 {requestId, status:"error", error, ...}
  * 404 when neither marker nor result exist.
  * 400 for invalid ids (path-traversal guard: hex/[0-9a-f] only, ≤64 chars).

## TOCTOU + cascade-cleanup from prior CR

  * TOCTOU: server snapshots the preview list at confirm time and writes the
    PRUNED list (intersection of preview + still-outside) into the marker.
    The ingestor does NOT re-check geo_filter — it executes exactly what the
    operator confirmed.
  * Cascade cleanup: ingestor's DeleteNodesByPubkeys chunks under SQLite's
    variable limit (500 per stmt). It deliberately limits scope to the `nodes`
    table; the schema has no FKs, so observation/transmission rows are left
    for the regular packet-retention ticker. Documented in-line.

## Read-only invariant test

Extends cmd/server/readonly_invariant_test.go (TestServerDBHasNoWriteMethods)
to forbid `DeleteNodesByPubkeys` on the server `*DB`. With this commit the
method is gone, so the test passes:

  $ go test -run TestServerDBHasNoWriteMethods ./...
  ok      github.com/corescope/server     0.190s

## Tests

* cmd/server/routes_test.go: rewrote the confirm-test to assert 202 +
  marker file existence; added pending→done status-roundtrip test;
  added 404 + path-traversal-id rejection tests. Old TestDeleteNodesByPubkeys
  removed (the method now lives in cmd/ingestor).
* cmd/ingestor/prune_geofilter_test.go: end-to-end queue test (request file
  on disk → RunPendingPruneRequests → result file + DB rows actually deleted),
  plus empty-queue no-op.
* internal/prunequeue/prunequeue_test.go: round-trip, path-traversal rejection,
  missing-result returns (nil, nil).

## Files

* internal/prunequeue/{prunequeue.go,prunequeue_test.go,go.mod} — new package
* cmd/ingestor/prune_geofilter.go — Store.DeleteNodesByPubkeys + RunPendingPruneRequests
* cmd/ingestor/prune_geofilter_test.go — queue end-to-end test
* cmd/ingestor/main.go — startup pass + 15s ticker for the queue, Stop on shutdown
* cmd/server/db.go — DeleteNodesByPubkeys removed; only GetNodesForGeoPrune (read) remains
* cmd/server/routes.go — handlePruneGeoFilter rewritten to enqueue;
  handlePruneGeoFilterStatus added; route registered
* cmd/server/routes_test.go — new tests; old direct-DELETE coverage removed
2026-05-21 03:09:50 +00:00
Kpa-clawbot 48ddc4ef59 test(server): RED — readonly invariant rejects DeleteNodesByPubkeys (#738)
After #1283/#1289 the server opens SQLite mode=ro. The geo-prune feature
introduced in PR #738 invokes DB.DeleteNodesByPubkeys from an HTTP handler;
in production this would fail with 'attempt to write a readonly database'
— the feature is dead on arrival. Tests pass only because setupTestDB
uses :memory: without mode=ro.

This commit reinstates the master-side cmd/server/readonly_invariant_test.go
and extends TestServerDBHasNoWriteMethods to assert DeleteNodesByPubkeys is
NOT a method on the server *DB. With the method still present (from the
M4 feature commit), the test FAILS:

    readonly_invariant_test.go:85: server *DB exposes forbidden write
    method "DeleteNodesByPubkeys" — must be relocated to ingestor (#1283)

The next commit (GREEN) removes DeleteNodesByPubkeys from cmd/server,
moves the DELETE to the ingestor via the new internal/prunequeue
marker-file protocol, and rewrites handlePruneGeoFilter to enqueue
requests rather than write directly.
2026-05-21 02:54:42 +00:00
Kpa-clawbot c5e5da1f92 merge: sync with origin/master — resolve conflicts in routes/config/example/builder
Brings #1289 (server read-only, prune moved to ingestor) and other
master changes into the geo-prune branch. Resolutions:

- cmd/server/config.go: keep both SaveGeoFilter (this branch) and
  obsBlacklistSet/AnalyticsConfig (master).
- cmd/server/routes.go: drop /api/admin/prune handler+route per #1283;
  retain /api/admin/prune-geo-filter (this branch).
- cmd/server/db.go: drop server-side PruneOldPackets / PruneOldMetrics
  / RemoveStaleObservers (now ingestor-owned). DeleteNodesByPubkeys
  still present — moved to ingestor in the next commit.
- config.example.json: prefer master's GeoFilter Builder wording.
- public/geofilter-builder.html: take master version.
- s.store.graph is now atomic.Pointer; add .Load() at the call site.
2026-05-21 02:54:09 +00:00
efiten 0f565e4aae fix(geo-prune): address PR #738 review feedback
- Fix TOCTOU race: confirm now requires pubkeys from preview in request
  body; server intersects with still-outside nodes so exactly the
  previewed set is deleted (no more, no less)
- Add cascade comment to DeleteNodesByPubkeys documenting that only the
  nodes table is affected (no FK constraints today)
- Log each deleted node by name + pubkey for operator visibility
- Return deleted node list in confirm response so UI shows what happened
- Check lat/lon nil directly instead of passing 0.0 to NodePassesGeoFilter
- Update confirm test to send pubkeys body; add test for missing body → 400

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-01 14:12:35 +02:00
efiten 79162d8bdb feat: one-click prune nodes outside geofilter (#669 M4)
Adds POST /api/admin/prune-geo-filter endpoint (dry-run by default,
?confirm=true to delete). Wires a Prune section into the GeoFilter
customizer tab — preview lists affected nodes, confirm deletes them.
Requires write-capable apiKey (writeEnabled gate, same as PUT).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-01 14:12:25 +02:00
efiten 5719b9e579 fix(geo-filter): address PR #736 security review feedback
- Fix data race on s.cfg.GeoFilter: add cfgMu RWMutex with getGeoFilter/
  setGeoFilter accessors used in all handler goroutines
- Add 1 MB MaxBytesReader cap on PUT /api/config/geo-filter request body
- Validate polygon coordinate ranges (lat ∈ [-90,90], lon ∈ [-180,180])
  and reject NaN/Inf values
- Cap polygon at 1000 points to bound storage and computation
- Document writeEnabled information-leak trade-off with a comment
- Add tests for coordinate range rejection and oversized polygon rejection

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-01 13:38:36 +02:00
efiten e92a2333f2 feat: geofilter map modal + light tile theme (#669)
Clicking the small inline map in the customizer GeoFilter tab now opens
a full-screen modal (92vw × 86vh) with Undo/Clear/Done/Cancel controls.
The inline map becomes a read-only preview. Both maps and the standalone
geofilter-builder.html now use CartoDB Positron (light) instead of dark.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-01 13:38:36 +02:00
efiten 1881c92d6e feat: geofilter customizer tab + PUT /api/config/geo-filter (#669 M3)
Backend:
- Add PUT /api/config/geo-filter (requires X-API-Key) — saves geo_filter
  back to config.json atomically and updates in-memory config immediately,
  no restart needed
- Add SaveGeoFilter() to config.go: reads config as raw map (preserving
  _comment fields), updates geo_filter key, writes back via temp+rename
- Add writeEnabled field to GET /api/config/geo-filter response so the
  frontend can gate editing controls on server write capability
- Add Server.configDir field; wired from -config-dir flag in main.go
- Tests: TestPutConfigGeoFilter (4 cases) + TestSaveGeoFilter (3 cases)

Frontend:
- Add GeoFilter tab (🗺️) to the customizer between Display and Export
- Tab shows current polygon on a Leaflet map (read-only for all users)
- Editing controls (undo, clear, buffer km, API key input, save/remove)
  are only revealed when the server reports writeEnabled=true — i.e. the
  deployment has a write-capable apiKey configured. Public instances see
  a read-only polygon view.
- Save calls PUT /api/config/geo-filter; Remove clears the filter
- Map is destroyed on tab switch and panel close to avoid Leaflet leaks

Docs:
- Add docs/user-guide/geofilter.md (full guide: config, customizer,
  builder, prune script, API)
- Update configuration.md and customization.md with geo_filter section
- Update config.example.json _comment to mention the Customizer tab

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-01 13:36:48 +02:00
20 changed files with 1845 additions and 41 deletions
+2 -1
View File
@@ -63,6 +63,7 @@ func (s *DBStats) SnapshotBackfills() map[string]int64 {
// Store wraps the SQLite database for packet ingestion.
type Store struct {
db *sql.DB
path string // filesystem path to the SQLite DB (used to resolve queue dirs)
Stats DBStats
stmtGetTxByHash *sql.Stmt
@@ -118,7 +119,7 @@ func OpenStoreWithInterval(dbPath string, sampleIntervalSec int) (*Store, error)
return nil, fmt.Errorf("dbschema.Apply: %w", err)
}
s := &Store{db: db, sampleIntervalSec: sampleIntervalSec}
s := &Store{db: db, path: dbPath, sampleIntervalSec: sampleIntervalSec}
if err := s.prepareStatements(); err != nil {
return nil, fmt.Errorf("preparing statements: %w", err)
}
+4
View File
@@ -43,3 +43,7 @@ require (
modernc.org/mathutil v1.6.0 // indirect
modernc.org/memory v1.8.0 // indirect
)
require github.com/meshcore-analyzer/prunequeue v0.0.0
replace github.com/meshcore-analyzer/prunequeue => ../../internal/prunequeue
+14
View File
@@ -179,6 +179,19 @@ func main() {
}
}()
// Prune-request queue (#669 M4 / #738): the read-only server enqueues
// geo-prune requests as marker files; the ingestor (which holds the
// write handle) executes the DELETEs. Process on startup, then every
// 15 seconds — short enough for a one-click UX, long enough to avoid
// useless wake-ups.
store.RunPendingPruneRequests()
pruneQueueTicker := time.NewTicker(15 * time.Second)
go func() {
for range pruneQueueTicker.C {
store.RunPendingPruneRequests()
}
}()
// Per-second stats file writer for the server's /api/perf/write-sources
// endpoint (#1120). Best-effort; never fatal.
StartStatsFileWriter(store, time.Second)
@@ -324,6 +337,7 @@ func main() {
packetRetentionTicker.Stop()
}
statsTicker.Stop()
pruneQueueTicker.Stop()
stopWatchdog()
store.LogStats() // final stats on shutdown
for _, c := range clients {
+106
View File
@@ -0,0 +1,106 @@
// Package main: ingestor-side processor for prune-request marker files
// written by the read-only server (see internal/prunequeue).
//
// The server cannot DELETE because it opens SQLite mode=ro (#1283/#1289).
// Instead, the server writes request-<id>.json under <dataDir>/prune-requests/
// and the ingestor consumes it here.
package main
import (
"fmt"
"log"
"os"
"strings"
"time"
"github.com/meshcore-analyzer/prunequeue"
)
// DeleteNodesByPubkeys deletes nodes by public key. Returns the count deleted.
// Only the ingestor calls this (server has no write handle).
func (s *Store) DeleteNodesByPubkeys(pubkeys []string) (int64, error) {
if len(pubkeys) == 0 {
return 0, nil
}
// Chunk to keep statements under SQLite's variable limit (default 999).
const chunk = 500
var total int64
for start := 0; start < len(pubkeys); start += chunk {
end := start + chunk
if end > len(pubkeys) {
end = len(pubkeys)
}
batch := pubkeys[start:end]
placeholders := strings.Repeat("?,", len(batch))
placeholders = placeholders[:len(placeholders)-1]
args := make([]interface{}, len(batch))
for i, pk := range batch {
args[i] = pk
}
// Cascade cleanup: a node row carries the canonical identity, but
// observations/transmissions reference the pubkey too via observer
// metadata and originator fields. There are no FK constraints in
// the current schema (#669 review note), so we explicitly clear
// the most obvious follow-on rows that would otherwise become
// orphans visible to operators.
//
// Conservative scope: only the `nodes` row is removed here. The
// referenced observation/transmission history is retained for
// audit; operators can run the regular packet-retention prune to
// age it out. If a future schema introduces FKs, revisit.
res, err := s.db.Exec("DELETE FROM nodes WHERE public_key IN ("+placeholders+")", args...)
if err != nil {
return total, fmt.Errorf("delete batch [%d:%d]: %w", start, end, err)
}
n, _ := res.RowsAffected()
total += n
}
return total, nil
}
// RunPendingPruneRequests scans the prune-requests/ directory next to the
// SQLite database and processes any request-<id>.json markers written by
// the server. Each request is honored verbatim — the server is responsible
// for the TOCTOU snapshot (only pubkeys that were still outside the
// geofilter at confirm time). After running DELETE, the ingestor writes
// result-<id>.json and removes the request file (atomic, via os.Rename in
// prunequeue.WriteResult).
//
// Safe to call from a ticker — no-op when the queue is empty.
func (s *Store) RunPendingPruneRequests() {
paths, err := prunequeue.ListPending(s.path)
if err != nil {
log.Printf("[prune-queue] list pending failed: %v", err)
return
}
if len(paths) == 0 {
return
}
for _, p := range paths {
req, err := prunequeue.ReadRequest(p)
if err != nil {
log.Printf("[prune-queue] read %s failed: %v — removing", p, err)
_ = os.Remove(p)
continue
}
log.Printf("[prune-queue] processing request %s: %d pubkey(s) (%s)",
req.ID, len(req.Pubkeys), req.Reason)
start := time.Now()
deleted, derr := s.DeleteNodesByPubkeys(req.Pubkeys)
res := prunequeue.Result{
ID: req.ID,
RequestedAt: req.RequestedAt,
CompletedAt: time.Now().UTC(),
Deleted: deleted,
}
if derr != nil {
res.Error = derr.Error()
log.Printf("[prune-queue] request %s FAILED after %s: %v", req.ID, time.Since(start), derr)
} else {
log.Printf("[prune-queue] request %s deleted %d node(s) in %s", req.ID, deleted, time.Since(start))
}
if werr := prunequeue.WriteResult(s.path, res); werr != nil {
log.Printf("[prune-queue] write result for %s failed: %v", req.ID, werr)
}
}
}
+77
View File
@@ -0,0 +1,77 @@
package main
import (
"path/filepath"
"testing"
"time"
"github.com/meshcore-analyzer/prunequeue"
)
func TestRunPendingPruneRequests(t *testing.T) {
dir := t.TempDir()
dbPath := filepath.Join(dir, "test.db")
store, err := OpenStore(dbPath)
if err != nil {
t.Fatalf("OpenStore: %v", err)
}
defer store.Close()
// Seed two nodes; one will be pruned, one will be kept.
if _, err := store.db.Exec(`INSERT INTO nodes (public_key, name, role, lat, lon, last_seen, first_seen)
VALUES ('aaaa', 'gone', 'companion', 1.0, 1.0, '2026-01-01T00:00:00Z', '2026-01-01T00:00:00Z'),
('bbbb', 'kept', 'companion', 2.0, 2.0, '2026-01-01T00:00:00Z', '2026-01-01T00:00:00Z')`); err != nil {
t.Fatalf("seed: %v", err)
}
id := prunequeue.NewID()
if err := prunequeue.WriteRequest(dbPath, prunequeue.Request{
ID: id,
RequestedAt: time.Now().UTC(),
Reason: "geo-prune-test",
Pubkeys: []string{"aaaa"},
}); err != nil {
t.Fatalf("WriteRequest: %v", err)
}
store.RunPendingPruneRequests()
// Request file gone, result file present.
if exists, _ := prunequeue.RequestExists(dbPath, id); exists {
t.Error("request file should have been consumed")
}
res, err := prunequeue.ReadResult(dbPath, id)
if err != nil || res == nil {
t.Fatalf("ReadResult: res=%v err=%v", res, err)
}
if res.Deleted != 1 {
t.Errorf("expected Deleted=1, got %d", res.Deleted)
}
if res.Error != "" {
t.Errorf("unexpected error: %s", res.Error)
}
// Verify DB state: aaaa gone, bbbb kept.
var n int
store.db.QueryRow("SELECT COUNT(*) FROM nodes WHERE public_key='aaaa'").Scan(&n)
if n != 0 {
t.Errorf("expected 'aaaa' deleted, got count=%d", n)
}
store.db.QueryRow("SELECT COUNT(*) FROM nodes WHERE public_key='bbbb'").Scan(&n)
if n != 1 {
t.Errorf("expected 'bbbb' kept, got count=%d", n)
}
}
func TestRunPendingPruneRequests_EmptyQueueIsNoop(t *testing.T) {
dir := t.TempDir()
dbPath := filepath.Join(dir, "test.db")
store, err := OpenStore(dbPath)
if err != nil {
t.Fatalf("OpenStore: %v", err)
}
defer store.Close()
// Must not panic / error on empty queue.
store.RunPendingPruneRequests()
}
+57
View File
@@ -2,6 +2,7 @@ package main
import (
"encoding/json"
"fmt"
"log"
"os"
"path/filepath"
@@ -447,6 +448,62 @@ func (c *Config) IsBlacklisted(pubkey string) bool {
return c.blacklistSet()[strings.ToLower(strings.TrimSpace(pubkey))]
}
// SaveGeoFilter writes the geo_filter section back to config.json on disk.
// Pass gf=nil to remove the filter. The rest of config.json is preserved as-is.
func SaveGeoFilter(configDir string, gf *GeoFilterConfig) error {
var configPath string
for _, p := range []string{
filepath.Join(configDir, "config.json"),
filepath.Join(configDir, "data", "config.json"),
} {
if _, err := os.Stat(p); err == nil {
configPath = p
break
}
}
if configPath == "" {
return fmt.Errorf("config.json not found in %s", configDir)
}
data, err := os.ReadFile(configPath)
if err != nil {
return fmt.Errorf("read config: %w", err)
}
// Parse as a raw map so non-struct fields (_comment, etc.) are preserved.
var raw map[string]interface{}
if err := json.Unmarshal(data, &raw); err != nil {
return fmt.Errorf("parse config: %w", err)
}
if gf == nil || len(gf.Polygon) == 0 {
delete(raw, "geo_filter")
} else {
// Round-trip through JSON to get a plain interface{} value.
b, _ := json.Marshal(gf)
var v interface{}
_ = json.Unmarshal(b, &v)
raw["geo_filter"] = v
}
out, err := json.MarshalIndent(raw, "", " ")
if err != nil {
return fmt.Errorf("marshal config: %w", err)
}
out = append(out, '\n')
// Atomic write: temp file + rename.
tmp := configPath + ".tmp"
if err := os.WriteFile(tmp, out, 0644); err != nil {
return fmt.Errorf("write config: %w", err)
}
if err := os.Rename(tmp, configPath); err != nil {
os.Remove(tmp)
return fmt.Errorf("rename config: %w", err)
}
return nil
}
// obsBlacklistSet lazily builds and caches the observerBlacklist as a set for O(1) lookups.
func (c *Config) obsBlacklistSet() map[string]bool {
c.obsBlacklistOnce.Do(func() {
+44
View File
@@ -2434,3 +2434,47 @@ func (db *DB) GetSignatureDropCount() int64 {
}
return count
}
// NodeForGeoPrune holds the minimal fields needed for geo-filter pruning.
type NodeForGeoPrune struct {
PubKey string
Name string
Lat *float64
Lon *float64
}
// GetNodesForGeoPrune returns all nodes with their coordinates for geo-filter evaluation.
// Read-only — safe on the server's mode=ro handle.
func (db *DB) GetNodesForGeoPrune() ([]NodeForGeoPrune, error) {
rows, err := db.conn.Query("SELECT public_key, name, lat, lon FROM nodes ORDER BY name")
if err != nil {
return nil, err
}
defer rows.Close()
var nodes []NodeForGeoPrune
for rows.Next() {
var pk string
var name sql.NullString
var lat, lon sql.NullFloat64
if err := rows.Scan(&pk, &name, &lat, &lon); err != nil {
continue
}
n := NodeForGeoPrune{PubKey: pk, Name: name.String}
if lat.Valid {
v := lat.Float64
n.Lat = &v
}
if lon.Valid {
v := lon.Float64
n.Lon = &v
}
nodes = append(nodes, n)
}
return nodes, rows.Err()
}
// DeleteNodesByPubkeys was removed in PR #738 follow-up: server is read-only
// (opened with mode=ro after #1283/#1289), so DELETE statements would fail at
// runtime. Geo-prune now flows server → marker file → ingestor; see
// internal/prunequeue and cmd/ingestor/prune_geofilter.go.
+4
View File
@@ -41,3 +41,7 @@ require (
modernc.org/mathutil v1.6.0 // indirect
modernc.org/memory v1.8.0 // indirect
)
require github.com/meshcore-analyzer/prunequeue v0.0.0
replace github.com/meshcore-analyzer/prunequeue => ../../internal/prunequeue
+1
View File
@@ -273,6 +273,7 @@ func main() {
// HTTP server
srv := NewServer(database, cfg, hub)
srv.configDir = configDir
srv.store = store
router := mux.NewRouter()
srv.RegisterRoutes(router)
+4
View File
@@ -74,6 +74,10 @@ func TestServerDBHasNoWriteMethods(t *testing.T) {
"PruneOldPackets",
"PruneOldMetrics",
"RemoveStaleObservers",
// #738 / one-click geo-prune: the DELETE must live on the
// ingestor's *Store. The server's HTTP handler now enqueues a
// marker file (see internal/prunequeue); it does not write.
"DeleteNodesByPubkeys",
}
typ := reflect.TypeOf((*DB)(nil))
for _, name := range forbidden {
+265 -3
View File
@@ -6,6 +6,7 @@ import (
"encoding/json"
"fmt"
"log"
"math"
"net/http"
"regexp"
"runtime"
@@ -17,6 +18,7 @@ import (
"github.com/gorilla/mux"
"github.com/meshcore-analyzer/packetpath"
"github.com/meshcore-analyzer/prunequeue"
)
// Server holds shared state for route handlers.
@@ -25,12 +27,20 @@ type Server struct {
cfg *Config
hub *Hub
store *PacketStore // in-memory packet store (nil = fallback to DB)
configDir string // directory containing config.json (for write-back via PUT /api/config/geo-filter)
startedAt time.Time
perfStats *PerfStats
version string
commit string
buildTime string
// Guards s.cfg.GeoFilter — read by ingest/handler goroutines, written by PUT handler
cfgMu sync.RWMutex
// Serializes concurrent PUT /api/config/geo-filter disk writes so requests
// can't race on the .tmp file or interleave disk/memory updates.
saveMu sync.Mutex
// Cached runtime.MemStats to avoid stop-the-world pauses on every health check
memStatsMu sync.Mutex
memStatsCache runtime.MemStats
@@ -49,6 +59,21 @@ type Server struct {
router *mux.Router
}
// getGeoFilter returns a pointer to the current geo_filter config under read lock.
// Callers MUST NOT mutate the returned struct.
func (s *Server) getGeoFilter() *GeoFilterConfig {
s.cfgMu.RLock()
defer s.cfgMu.RUnlock()
return s.cfg.GeoFilter
}
// setGeoFilter atomically swaps the geo_filter config; used by PUT /api/config/geo-filter.
func (s *Server) setGeoFilter(gf *GeoFilterConfig) {
s.cfgMu.Lock()
defer s.cfgMu.Unlock()
s.cfg.GeoFilter = gf
}
// PerfStats tracks request performance.
type PerfStats struct {
mu sync.Mutex
@@ -120,6 +145,7 @@ func (s *Server) RegisterRoutes(r *mux.Router) {
r.HandleFunc("/api/config/theme", s.handleConfigTheme).Methods("GET")
r.HandleFunc("/api/config/map", s.handleConfigMap).Methods("GET")
r.HandleFunc("/api/config/geo-filter", s.handleConfigGeoFilter).Methods("GET")
r.Handle("/api/config/geo-filter", s.requireAPIKey(http.HandlerFunc(s.handlePutConfigGeoFilter))).Methods("PUT")
// Readiness endpoint (gated on background init completion)
r.HandleFunc("/api/healthz", s.handleHealthz).Methods("GET")
@@ -135,6 +161,12 @@ func (s *Server) RegisterRoutes(r *mux.Router) {
// /api/admin/prune removed in #1283 — pruning is owned by the
// ingestor process (scheduled tickers + startup pass). Operators
// who want an ad-hoc prune can restart the ingestor.
//
// /api/admin/prune-geo-filter (#669 M4 / PR #738): server enqueues a
// marker file; the ingestor (which holds the writable DB handle)
// runs the DELETE. /status reports completion.
r.Handle("/api/admin/prune-geo-filter", s.requireAPIKey(http.HandlerFunc(s.handlePruneGeoFilter))).Methods("POST")
r.Handle("/api/admin/prune-geo-filter/status", s.requireAPIKey(http.HandlerFunc(s.handlePruneGeoFilterStatus))).Methods("GET")
r.Handle("/api/debug/affinity", s.requireAPIKey(http.HandlerFunc(s.handleDebugAffinity))).Methods("GET")
r.Handle("/api/dropped-packets", s.requireAPIKey(http.HandlerFunc(s.handleDroppedPackets))).Methods("GET")
r.Handle("/api/backup", s.requireAPIKey(http.HandlerFunc(s.handleBackup))).Methods("GET")
@@ -444,12 +476,15 @@ func (s *Server) handleConfigMap(w http.ResponseWriter, r *http.Request) {
}
func (s *Server) handleConfigGeoFilter(w http.ResponseWriter, r *http.Request) {
gf := s.cfg.GeoFilter
gf := s.getGeoFilter()
// writeEnabled signals to clients (e.g. the customizer UI) whether a
// strong API key is configured. Low-sensitivity by design.
writeEnabled := s.cfg != nil && s.cfg.APIKey != "" && !IsWeakAPIKey(s.cfg.APIKey)
if gf == nil || len(gf.Polygon) == 0 {
writeJSON(w, map[string]interface{}{"polygon": nil, "bufferKm": 0})
writeJSON(w, map[string]interface{}{"polygon": nil, "bufferKm": 0, "writeEnabled": writeEnabled})
return
}
writeJSON(w, map[string]interface{}{"polygon": gf.Polygon, "bufferKm": gf.BufferKm})
writeJSON(w, map[string]interface{}{"polygon": gf.Polygon, "bufferKm": gf.BufferKm, "writeEnabled": writeEnabled})
}
// --- System Handlers ---
@@ -2909,3 +2944,230 @@ func (s *Server) handleDroppedPackets(w http.ResponseWriter, r *http.Request) {
}
writeJSON(w, results)
}
// handlePruneGeoFilter identifies (dry_run=true, default) or enqueues (confirm=true)
// deletion of nodes whose GPS coordinates fall outside the currently configured
// geo_filter. Nodes with no GPS fix are always kept. Requires geo_filter to be
// configured.
//
// Since #1283/#1289 the server opens SQLite read-only, so the actual DELETE is
// performed by the ingestor. The server writes a request marker file (see
// internal/prunequeue); the ingestor's maintenance loop consumes it and writes a
// result marker. The confirm response is 202 Accepted with a request id;
// clients poll GET /api/admin/prune-geo-filter/status?id=<id> for completion.
//
// Confirm requires the pubkeys from the preview in the request body to prevent
// TOCTOU races: only nodes that were shown in preview AND are still outside the
// filter are enqueued.
func (s *Server) handlePruneGeoFilter(w http.ResponseWriter, r *http.Request) {
gf := s.getGeoFilter()
if gf == nil || len(gf.Polygon) < 3 {
writeError(w, http.StatusBadRequest, "no geo_filter configured")
return
}
nodes, err := s.db.GetNodesForGeoPrune()
if err != nil {
writeError(w, http.StatusInternalServerError, "db error")
return
}
type nodeResult struct {
PubKey string `json:"pubkey"`
Name string `json:"name"`
Lat *float64 `json:"lat"`
Lon *float64 `json:"lon"`
}
var outside []nodeResult
for _, n := range nodes {
if n.Lat == nil || n.Lon == nil {
continue // no GPS — always keep
}
if !NodePassesGeoFilter(*n.Lat, *n.Lon, gf) {
outside = append(outside, nodeResult{PubKey: n.PubKey, Name: n.Name, Lat: n.Lat, Lon: n.Lon})
}
}
if r.URL.Query().Get("confirm") != "true" {
// Dry run — return preview without enqueueing anything.
writeJSON(w, map[string]interface{}{
"dryRun": true,
"count": len(outside),
"nodes": outside,
})
return
}
// Confirmed enqueue — require pubkeys from the preview to prevent TOCTOU:
// only nodes that were shown in preview AND are still outside the filter
// at this exact moment are scheduled for deletion. (The ingestor honors
// the list verbatim; it does NOT re-evaluate geo_filter membership.)
var body struct {
Pubkeys []string `json:"pubkeys"`
}
if err := json.NewDecoder(r.Body).Decode(&body); err != nil || len(body.Pubkeys) == 0 {
writeError(w, http.StatusBadRequest, "confirm requires pubkeys from preview in request body")
return
}
allowed := make(map[string]bool, len(body.Pubkeys))
for _, pk := range body.Pubkeys {
allowed[pk] = true
}
var toDelete []nodeResult
for _, n := range outside {
if allowed[n.PubKey] {
toDelete = append(toDelete, n)
}
}
pubkeys := make([]string, 0, len(toDelete))
for _, n := range toDelete {
pubkeys = append(pubkeys, n.PubKey)
}
id := prunequeue.NewID()
req := prunequeue.Request{
ID: id,
RequestedAt: time.Now().UTC(),
Reason: "geo-prune",
Pubkeys: pubkeys,
}
if err := prunequeue.WriteRequest(s.db.path, req); err != nil {
log.Printf("[geo-prune] failed to enqueue request %s: %v", id, err)
writeError(w, http.StatusInternalServerError, "failed to enqueue prune request")
return
}
log.Printf("[geo-prune] enqueued request %s for %d node(s) (queue dir=%s)",
id, len(pubkeys), prunequeue.QueueDir(s.db.path))
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusAccepted)
_ = json.NewEncoder(w).Encode(map[string]interface{}{
"dryRun": false,
"accepted": true,
"requestId": id,
"count": len(pubkeys),
"nodes": toDelete,
"statusUrl": "/api/admin/prune-geo-filter/status?id=" + id,
})
}
// handlePruneGeoFilterStatus reports the state of a previously-enqueued
// geo-prune request. While the request marker is still present the response is
// {"status":"pending"}. Once the ingestor writes a result, the response is
// {"status":"done","deleted":N,"completedAt":...} (or "error" if the ingestor
// failed). Returns 404 when neither marker nor result is found.
func (s *Server) handlePruneGeoFilterStatus(w http.ResponseWriter, r *http.Request) {
id := r.URL.Query().Get("id")
if id == "" {
writeError(w, http.StatusBadRequest, "missing id")
return
}
res, err := prunequeue.ReadResult(s.db.path, id)
if err != nil {
if strings.Contains(err.Error(), "invalid prune request id") {
writeError(w, http.StatusBadRequest, "invalid id")
return
}
writeError(w, http.StatusInternalServerError, "status read failed")
return
}
if res != nil {
status := "done"
if res.Error != "" {
status = "error"
}
writeJSON(w, map[string]interface{}{
"requestId": res.ID,
"status": status,
"deleted": res.Deleted,
"requestedAt": res.RequestedAt,
"completedAt": res.CompletedAt,
"error": res.Error,
})
return
}
pending, err := prunequeue.RequestExists(s.db.path, id)
if err != nil {
if strings.Contains(err.Error(), "invalid prune request id") {
writeError(w, http.StatusBadRequest, "invalid id")
return
}
writeError(w, http.StatusInternalServerError, "status read failed")
return
}
if pending {
writeJSON(w, map[string]interface{}{
"requestId": id,
"status": "pending",
})
return
}
writeError(w, http.StatusNotFound, "unknown request id")
}
// handlePutConfigGeoFilter writes the geo_filter config to disk and updates the
// in-memory pointer atomically. Empty/missing polygon clears the filter.
//
// Backstop validation: ≤1000 points, every point in [-90,90]/[-180,180], no
// NaN/Inf; bufferKm finite, non-negative, ≤ 20000 km. Concurrent PUTs are
// serialized via s.saveMu so they cannot race on the .tmp file.
func (s *Server) handlePutConfigGeoFilter(w http.ResponseWriter, r *http.Request) {
r.Body = http.MaxBytesReader(w, r.Body, 1<<20) // 1 MB cap
var body struct {
Polygon [][2]float64 `json:"polygon"`
BufferKm float64 `json:"bufferKm"`
}
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
writeError(w, http.StatusBadRequest, "invalid JSON")
return
}
if len(body.Polygon) > 0 && len(body.Polygon) < 3 {
writeError(w, http.StatusBadRequest, "polygon must have at least 3 points")
return
}
if len(body.Polygon) > 1000 {
writeError(w, http.StatusBadRequest, "polygon must have at most 1000 points")
return
}
for _, pt := range body.Polygon {
if math.IsNaN(pt[0]) || math.IsNaN(pt[1]) || math.IsInf(pt[0], 0) || math.IsInf(pt[1], 0) ||
pt[0] < -90 || pt[0] > 90 || pt[1] < -180 || pt[1] > 180 {
writeError(w, http.StatusBadRequest, "polygon point out of range: lat must be in [-90,90], lon in [-180,180]")
return
}
}
if math.IsNaN(body.BufferKm) || math.IsInf(body.BufferKm, 0) ||
body.BufferKm < 0 || body.BufferKm > 20000 {
writeError(w, http.StatusBadRequest, "bufferKm must be a finite number in [0, 20000]")
return
}
var gf *GeoFilterConfig
if len(body.Polygon) >= 3 {
gf = &GeoFilterConfig{Polygon: body.Polygon, BufferKm: body.BufferKm}
}
s.saveMu.Lock()
if s.configDir != "" {
if err := SaveGeoFilter(s.configDir, gf); err != nil {
s.saveMu.Unlock()
log.Printf("[geofilter] save failed: %v", err)
writeError(w, http.StatusInternalServerError, "failed to save config")
return
}
}
s.setGeoFilter(gf)
s.saveMu.Unlock()
if gf != nil {
writeJSON(w, map[string]interface{}{"polygon": gf.Polygon, "bufferKm": gf.BufferKm})
} else {
writeJSON(w, map[string]interface{}{"polygon": nil, "bufferKm": 0})
}
}
+475
View File
@@ -5,12 +5,15 @@ import (
"encoding/json"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strconv"
"strings"
"testing"
"time"
"github.com/gorilla/mux"
"github.com/meshcore-analyzer/prunequeue"
)
func setupTestServer(t *testing.T) (*Server, *mux.Router) {
@@ -2839,6 +2842,34 @@ func TestConfigGeoFilterEndpoint(t *testing.T) {
if body["bufferKm"] == nil {
t.Error("expected bufferKm in response")
}
if _, ok := body["writeEnabled"]; !ok {
t.Error("expected writeEnabled field in response")
}
// No apiKey configured → writeEnabled should be false
if body["writeEnabled"] != false {
t.Errorf("expected writeEnabled=false when no apiKey, got %v", body["writeEnabled"])
}
})
t.Run("writeEnabled true when strong apiKey configured", func(t *testing.T) {
db := setupTestDB(t)
cfg := &Config{Port: 3000, APIKey: "a-strong-api-key-1234"}
hub := NewHub()
srv := NewServer(db, cfg, hub)
srv.store = NewPacketStore(db, nil)
srv.store.Load()
router := mux.NewRouter()
srv.RegisterRoutes(router)
req := httptest.NewRequest("GET", "/api/config/geo-filter", nil)
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
var body map[string]interface{}
json.Unmarshal(w.Body.Bytes(), &body)
if body["writeEnabled"] != true {
t.Errorf("expected writeEnabled=true when strong apiKey configured, got %v", body["writeEnabled"])
}
})
}
@@ -3973,3 +4004,447 @@ func TestPacketDetailPrefersStoreOverDB(t *testing.T) {
t.Errorf("expected observation_count=2 (from store), got %v", body["observation_count"])
}
}
// --- geo-filter write-back tests ---
func setupGeoFilterServer(t *testing.T, apiKey string) (*Server, *mux.Router, string) {
t.Helper()
dir := t.TempDir()
cfgJSON := `{"port":3000,"apiKey":"` + apiKey + `"}`
if err := os.WriteFile(filepath.Join(dir, "config.json"), []byte(cfgJSON), 0644); err != nil {
t.Fatalf("write config: %v", err)
}
db := setupTestDB(t)
seedTestData(t, db)
cfg := &Config{Port: 3000, APIKey: apiKey}
hub := NewHub()
srv := NewServer(db, cfg, hub)
srv.configDir = dir
store := NewPacketStore(db, nil)
if err := store.Load(); err != nil {
t.Fatalf("store.Load: %v", err)
}
srv.store = store
router := mux.NewRouter()
srv.RegisterRoutes(router)
return srv, router, dir
}
func TestPutConfigGeoFilter(t *testing.T) {
const apiKey = "a-strong-api-key-for-testing"
t.Run("saves valid polygon and updates in-memory config", func(t *testing.T) {
srv, router, dir := setupGeoFilterServer(t, apiKey)
body := `{"polygon":[[51.0,4.0],[51.0,5.0],[50.5,5.0],[50.5,4.0]],"bufferKm":15}`
req := httptest.NewRequest("PUT", "/api/config/geo-filter", strings.NewReader(body))
req.Header.Set("X-API-Key", apiKey)
req.Header.Set("Content-Type", "application/json")
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != 200 {
t.Fatalf("expected 200, got %d: %s", w.Code, w.Body.String())
}
// In-memory config updated
if srv.cfg.GeoFilter == nil {
t.Fatal("expected in-memory GeoFilter to be set")
}
if len(srv.cfg.GeoFilter.Polygon) != 4 {
t.Errorf("expected 4 polygon points, got %d", len(srv.cfg.GeoFilter.Polygon))
}
if srv.cfg.GeoFilter.BufferKm != 15 {
t.Errorf("expected bufferKm=15, got %v", srv.cfg.GeoFilter.BufferKm)
}
// config.json updated on disk
data, _ := os.ReadFile(filepath.Join(dir, "config.json"))
if !bytes.Contains(data, []byte("geo_filter")) {
t.Error("expected geo_filter key in saved config.json")
}
})
t.Run("clears filter when polygon is empty", func(t *testing.T) {
srv, router, dir := setupGeoFilterServer(t, apiKey)
// Pre-set a filter so we can clear it
srv.setGeoFilter(&GeoFilterConfig{Polygon: [][2]float64{{51.0, 4.0}, {51.0, 5.0}, {50.5, 4.0}}, BufferKm: 10})
req := httptest.NewRequest("PUT", "/api/config/geo-filter", strings.NewReader(`{"polygon":null}`))
req.Header.Set("X-API-Key", apiKey)
req.Header.Set("Content-Type", "application/json")
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != 200 {
t.Fatalf("expected 200, got %d", w.Code)
}
if srv.cfg.GeoFilter != nil {
t.Error("expected in-memory GeoFilter to be cleared")
}
data, _ := os.ReadFile(filepath.Join(dir, "config.json"))
if bytes.Contains(data, []byte("geo_filter")) {
t.Error("expected geo_filter to be removed from config.json")
}
})
t.Run("rejects polygon with fewer than 3 points", func(t *testing.T) {
_, router, _ := setupGeoFilterServer(t, apiKey)
body := `{"polygon":[[51.0,4.0],[51.0,5.0]],"bufferKm":0}`
req := httptest.NewRequest("PUT", "/api/config/geo-filter", strings.NewReader(body))
req.Header.Set("X-API-Key", apiKey)
req.Header.Set("Content-Type", "application/json")
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != http.StatusBadRequest {
t.Fatalf("expected 400, got %d", w.Code)
}
})
t.Run("rejects out-of-range coordinates", func(t *testing.T) {
_, router, _ := setupGeoFilterServer(t, apiKey)
body := `{"polygon":[[91.0,4.0],[51.0,5.0],[50.5,4.0]],"bufferKm":0}`
req := httptest.NewRequest("PUT", "/api/config/geo-filter", strings.NewReader(body))
req.Header.Set("X-API-Key", apiKey)
req.Header.Set("Content-Type", "application/json")
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != http.StatusBadRequest {
t.Fatalf("expected 400 for out-of-range lat, got %d", w.Code)
}
})
t.Run("rejects polygon exceeding 1000 points", func(t *testing.T) {
_, router, _ := setupGeoFilterServer(t, apiKey)
pts := make([][2]float64, 1001)
for i := range pts {
pts[i] = [2]float64{51.0 + float64(i)*0.0001, 4.0}
}
b, _ := json.Marshal(map[string]interface{}{"polygon": pts, "bufferKm": 0})
req := httptest.NewRequest("PUT", "/api/config/geo-filter", strings.NewReader(string(b)))
req.Header.Set("X-API-Key", apiKey)
req.Header.Set("Content-Type", "application/json")
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != http.StatusBadRequest {
t.Fatalf("expected 400 for oversized polygon, got %d", w.Code)
}
})
t.Run("rejects missing API key", func(t *testing.T) {
_, router, _ := setupGeoFilterServer(t, apiKey)
body := `{"polygon":[[51.0,4.0],[51.0,5.0],[50.5,4.0]],"bufferKm":0}`
req := httptest.NewRequest("PUT", "/api/config/geo-filter", strings.NewReader(body))
req.Header.Set("Content-Type", "application/json")
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != http.StatusUnauthorized {
t.Fatalf("expected 401, got %d", w.Code)
}
})
}
func TestSaveGeoFilter(t *testing.T) {
t.Run("saves and reads back", func(t *testing.T) {
dir := t.TempDir()
if err := os.WriteFile(filepath.Join(dir, "config.json"), []byte(`{"port":3000}`), 0644); err != nil {
t.Fatal(err)
}
gf := &GeoFilterConfig{
Polygon: [][2]float64{{51.0, 4.0}, {51.0, 5.0}, {50.5, 4.0}},
BufferKm: 20,
}
if err := SaveGeoFilter(dir, gf); err != nil {
t.Fatalf("SaveGeoFilter: %v", err)
}
data, _ := os.ReadFile(filepath.Join(dir, "config.json"))
if !bytes.Contains(data, []byte("geo_filter")) {
t.Error("expected geo_filter in saved config")
}
if !bytes.Contains(data, []byte(`"bufferKm"`)) {
t.Error("expected bufferKm in saved config")
}
})
t.Run("removes geo_filter key when gf is nil", func(t *testing.T) {
dir := t.TempDir()
initial := `{"port":3000,"geo_filter":{"polygon":[[1,2],[3,4],[5,6]],"bufferKm":5}}`
if err := os.WriteFile(filepath.Join(dir, "config.json"), []byte(initial), 0644); err != nil {
t.Fatal(err)
}
if err := SaveGeoFilter(dir, nil); err != nil {
t.Fatalf("SaveGeoFilter: %v", err)
}
data, _ := os.ReadFile(filepath.Join(dir, "config.json"))
if bytes.Contains(data, []byte("geo_filter")) {
t.Error("expected geo_filter to be removed")
}
})
t.Run("returns error when config.json not found", func(t *testing.T) {
dir := t.TempDir()
err := SaveGeoFilter(dir, nil)
if err == nil {
t.Error("expected error when config.json not found")
}
})
}
// --- prune-geo-filter endpoint tests ---
func setupPruneGeoFilterServer(t *testing.T, apiKey string, gf *GeoFilterConfig) (*Server, *mux.Router) {
t.Helper()
db := setupTestDB(t)
seedTestData(t, db)
// Add a node clearly outside the geo filter (high lat/lon in Europe)
db.conn.Exec(`INSERT INTO nodes (public_key, name, role, lat, lon, last_seen, first_seen, advert_count)
VALUES ('aaaa111122223333', 'OutsideNode', 'repeater', 51.5, 4.5, '2026-01-01T00:00:00Z', '2026-01-01T00:00:00Z', 1)`)
// Add a node with no GPS (should always be kept)
db.conn.Exec(`INSERT INTO nodes (public_key, name, role, last_seen, first_seen, advert_count)
VALUES ('bbbb111122223333', 'NoGPSNode', 'companion', '2026-01-01T00:00:00Z', '2026-01-01T00:00:00Z', 1)`)
cfg := &Config{Port: 3000, APIKey: apiKey, GeoFilter: gf}
hub := NewHub()
srv := NewServer(db, cfg, hub)
store := NewPacketStore(db, nil)
store.Load()
srv.store = store
router := mux.NewRouter()
srv.RegisterRoutes(router)
return srv, router
}
func TestPruneGeoFilterEndpoint(t *testing.T) {
const apiKey = "a-strong-api-key-for-testing"
// Polygon around San Jose — seed nodes are at 37.437.6, -122.1 to -121.9 (inside)
// OutsideNode is at 51.5, 4.5 (Europe — outside)
gf := &GeoFilterConfig{
Polygon: [][2]float64{{37.0, -123.0}, {38.0, -123.0}, {38.0, -121.0}, {37.0, -121.0}},
BufferKm: 0,
}
t.Run("dry run returns outside nodes without deleting", func(t *testing.T) {
_, router := setupPruneGeoFilterServer(t, apiKey, gf)
req := httptest.NewRequest("POST", "/api/admin/prune-geo-filter", nil)
req.Header.Set("X-API-Key", apiKey)
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != 200 {
t.Fatalf("expected 200, got %d: %s", w.Code, w.Body.String())
}
var body map[string]interface{}
json.Unmarshal(w.Body.Bytes(), &body)
if body["dryRun"] != true {
t.Error("expected dryRun=true")
}
count, _ := body["count"].(float64)
if count != 1 {
t.Errorf("expected 1 outside node (OutsideNode), got %v", count)
}
nodes, _ := body["nodes"].([]interface{})
if len(nodes) != 1 {
t.Fatalf("expected 1 node in preview, got %d", len(nodes))
}
n, _ := nodes[0].(map[string]interface{})
if n["name"] != "OutsideNode" {
t.Errorf("expected OutsideNode, got %v", n["name"])
}
})
t.Run("confirm=true enqueues a prune request (status 202)", func(t *testing.T) {
srv, router := setupPruneGeoFilterServer(t, apiKey, gf)
body := strings.NewReader(`{"pubkeys":["aaaa111122223333"]}`)
req := httptest.NewRequest("POST", "/api/admin/prune-geo-filter?confirm=true", body)
req.Header.Set("X-API-Key", apiKey)
req.Header.Set("Content-Type", "application/json")
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != http.StatusAccepted {
t.Fatalf("expected 202 Accepted, got %d: %s", w.Code, w.Body.String())
}
var resp map[string]interface{}
json.Unmarshal(w.Body.Bytes(), &resp)
if resp["dryRun"] != false {
t.Error("expected dryRun=false")
}
if resp["accepted"] != true {
t.Error("expected accepted=true")
}
id, _ := resp["requestId"].(string)
if id == "" {
t.Fatal("expected non-empty requestId")
}
count, _ := resp["count"].(float64)
if count != 1 {
t.Errorf("expected count=1, got %v", count)
}
// Server is read-only — node must STILL exist in DB. The ingestor
// is responsible for the actual DELETE; the server only enqueued.
var dbCount int
srv.db.conn.QueryRow("SELECT COUNT(*) FROM nodes WHERE public_key = 'aaaa111122223333'").Scan(&dbCount)
if dbCount != 1 {
t.Errorf("expected OutsideNode still present (server is read-only), got count=%d", dbCount)
}
// And the marker file must exist on disk.
pending, err := prunequeue.RequestExists(srv.db.path, id)
if err != nil {
t.Fatalf("RequestExists: %v", err)
}
if !pending {
t.Errorf("expected request-%s.json to exist in queue dir", id)
}
})
t.Run("status endpoint reports pending then surfaces ingestor result", func(t *testing.T) {
srv, router := setupPruneGeoFilterServer(t, apiKey, gf)
// Enqueue first.
body := strings.NewReader(`{"pubkeys":["aaaa111122223333"]}`)
req := httptest.NewRequest("POST", "/api/admin/prune-geo-filter?confirm=true", body)
req.Header.Set("X-API-Key", apiKey)
req.Header.Set("Content-Type", "application/json")
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
var resp map[string]interface{}
json.Unmarshal(w.Body.Bytes(), &resp)
id := resp["requestId"].(string)
// While pending: GET status returns 200 status=pending.
statusReq := httptest.NewRequest("GET", "/api/admin/prune-geo-filter/status?id="+id, nil)
statusReq.Header.Set("X-API-Key", apiKey)
statusW := httptest.NewRecorder()
router.ServeHTTP(statusW, statusReq)
if statusW.Code != 200 {
t.Fatalf("status pending: expected 200, got %d: %s", statusW.Code, statusW.Body.String())
}
var sresp map[string]interface{}
json.Unmarshal(statusW.Body.Bytes(), &sresp)
if sresp["status"] != "pending" {
t.Errorf("expected status=pending, got %v", sresp["status"])
}
// Simulate the ingestor completing the request.
if err := prunequeue.WriteResult(srv.db.path, prunequeue.Result{
ID: id,
RequestedAt: time.Now().Add(-1 * time.Second).UTC(),
CompletedAt: time.Now().UTC(),
Deleted: 1,
}); err != nil {
t.Fatalf("WriteResult: %v", err)
}
// Now status should report done.
statusW2 := httptest.NewRecorder()
router.ServeHTTP(statusW2, statusReq)
if statusW2.Code != 200 {
t.Fatalf("status done: expected 200, got %d", statusW2.Code)
}
var sresp2 map[string]interface{}
json.Unmarshal(statusW2.Body.Bytes(), &sresp2)
if sresp2["status"] != "done" {
t.Errorf("expected status=done, got %v", sresp2["status"])
}
if d, _ := sresp2["deleted"].(float64); d != 1 {
t.Errorf("expected deleted=1, got %v", sresp2["deleted"])
}
})
t.Run("status endpoint returns 404 for unknown id", func(t *testing.T) {
_, router := setupPruneGeoFilterServer(t, apiKey, gf)
req := httptest.NewRequest("GET", "/api/admin/prune-geo-filter/status?id=deadbeefdeadbeef", nil)
req.Header.Set("X-API-Key", apiKey)
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != http.StatusNotFound {
t.Fatalf("expected 404, got %d", w.Code)
}
})
t.Run("status endpoint rejects path-traversal-looking id", func(t *testing.T) {
_, router := setupPruneGeoFilterServer(t, apiKey, gf)
req := httptest.NewRequest("GET", "/api/admin/prune-geo-filter/status?id=../../etc/passwd", nil)
req.Header.Set("X-API-Key", apiKey)
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != http.StatusBadRequest {
t.Fatalf("expected 400, got %d", w.Code)
}
})
t.Run("confirm=true without pubkeys body returns 400", func(t *testing.T) {
_, router := setupPruneGeoFilterServer(t, apiKey, gf)
req := httptest.NewRequest("POST", "/api/admin/prune-geo-filter?confirm=true", nil)
req.Header.Set("X-API-Key", apiKey)
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != http.StatusBadRequest {
t.Fatalf("expected 400, got %d: %s", w.Code, w.Body.String())
}
})
t.Run("returns 400 when no geo filter configured", func(t *testing.T) {
_, router := setupPruneGeoFilterServer(t, apiKey, nil)
req := httptest.NewRequest("POST", "/api/admin/prune-geo-filter", nil)
req.Header.Set("X-API-Key", apiKey)
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != http.StatusBadRequest {
t.Fatalf("expected 400, got %d", w.Code)
}
})
t.Run("returns 401 without API key", func(t *testing.T) {
_, router := setupPruneGeoFilterServer(t, apiKey, gf)
req := httptest.NewRequest("POST", "/api/admin/prune-geo-filter", nil)
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != http.StatusUnauthorized {
t.Fatalf("expected 401, got %d", w.Code)
}
})
}
func TestGetNodesForGeoPrune(t *testing.T) {
db := setupTestDB(t)
seedTestData(t, db)
nodes, err := db.GetNodesForGeoPrune()
if err != nil {
t.Fatalf("GetNodesForGeoPrune: %v", err)
}
if len(nodes) == 0 {
t.Error("expected nodes to be returned")
}
// Check that nodes with lat/lon have non-nil fields
for _, n := range nodes {
if n.PubKey == "" {
t.Error("expected non-empty pubkey")
}
}
}
// TestDeleteNodesByPubkeys was removed in PR #738 follow-up: the DELETE has
// been relocated to the ingestor (cmd/ingestor/prune_geofilter.go). End-to-end
// coverage of the prune flow now lives in cmd/ingestor/*_test.go.
+1 -1
View File
@@ -175,7 +175,7 @@
[37.20, -122.52]
],
"bufferKm": 20,
"_comment": "Optional. Restricts ingestion and API responses to nodes within the polygon + bufferKm. Polygon is an array of [lat, lon] pairs (minimum 3). Use the GeoFilter Builder (`/geofilter-builder.html`) to draw a polygon, save drafts to localStorage with Save Draft, and export a config snippet with Download — paste the snippet here as the `geo_filter` block. Remove this section to disable filtering. Nodes with no GPS fix are always allowed through."
"_comment": "Optional. Restricts ingestion and API responses to nodes within the polygon + bufferKm. Polygon is an array of [lat, lon] pairs (minimum 3). Use the GeoFilter tab in the Customizer (requires apiKey) or the GeoFilter Builder (`/geofilter-builder.html`) to draw a polygon visually and export a config snippet. Remove this section to disable filtering. Nodes with no GPS fix are always allowed through."
},
"foreignAdverts": {
"mode": "flag",
+3 -1
View File
@@ -206,7 +206,9 @@ Provide cert and key paths to enable HTTPS.
Restricts ingestion and API responses to nodes within the polygon plus a buffer margin. Remove the block to disable filtering. Nodes with no GPS fix always pass through.
See [Geographic Filtering](geofilter.md) for the full guide including the visual polygon builder and the prune script for cleaning up historical data.
Can also be configured live via the **🗺️ GeoFilter** tab in the Customizer (requires `apiKey`).
See [Geographic Filtering](geofilter.md) for the full guide.
## Home page
+5 -3
View File
@@ -66,11 +66,13 @@ Click **Import JSON** and paste a previously exported theme. The customizer load
Click **Reset to Defaults** to restore all settings to the built-in defaults.
## GeoFilter Builder
## GeoFilter (admin only)
The Export tab includes a **GeoFilter Builder** link. Click it to open a Leaflet map where you can draw a polygon boundary for your deployment area. The tool generates a `geo_filter` block you can paste directly into `config.json`.
The **🗺️ GeoFilter** tab lets operators configure geographic filtering directly from the customizer. It shows the active polygon on a Leaflet map and — on servers with a write-capable `apiKey` — allows editing the polygon and saving back to `config.json` without a restart.
See [Geographic Filtering](geofilter.md) for full details on what geo filtering does and how to configure it.
The editing controls are only revealed after the server confirms write access. On public deployments without an `apiKey`, the tab is read-only.
See [Geographic Filtering](geofilter.md) for the full guide, including the API, the prune script, and the standalone GeoFilter Builder.
## How it works
+69 -31
View File
@@ -30,9 +30,9 @@ Add a `geo_filter` block to `config.json`:
| Field | Type | Description |
|-------|------|-------------|
| `polygon` | `[[lat, lon], ...]` | Array of at least 3 coordinate pairs defining the boundary |
| `bufferKm` | number | Extra distance (km) around the polygon edge that is also accepted. `0` = exact boundary |
| `bufferKm` | number | Extra distance (km) outside the polygon edge that is also accepted. `0` = exact boundary |
Both the server and the ingestor read `geo_filter` from `config.json`. Restart both after changing this section.
Both the server and the ingestor read `geo_filter` from `config.json`. Restart both after changing this section manually.
To disable filtering entirely, remove the `geo_filter` block.
@@ -51,50 +51,88 @@ An older bounding box format is also supported as a fallback when no `polygon` i
Prefer the polygon format — it supports irregular shapes and the `bufferKm` margin.
## API endpoint
## Configuring via the customizer
The current geo filter configuration is exposed at:
If your server has an `apiKey` configured, the **GeoFilter tab** in the Customizer lets you edit the polygon visually without touching `config.json`:
1. Open the Customizer (nav bar → customize icon)
2. Click the **🗺️ GeoFilter** tab
3. Click on the map to draw your polygon (at least 3 points)
4. Adjust **Buffer km**
5. Enter your **Server API Key** (the `apiKey` value from `config.json`)
6. Click **Save to server** — the filter is applied immediately, no restart needed
The editing controls only appear when the server has a write-capable API key configured. On deployments without an `apiKey`, the tab shows the current polygon as read-only.
To remove the filter, click **Remove filter** (also requires the API key).
## GeoFilter Builder (standalone tool)
For a full-screen editing experience, use the built-in GeoFilter Builder at `/geofilter-builder.html`:
1. Navigate to `http://your-server/geofilter-builder.html`
2. Click on the map to add polygon vertices
3. Adjust **Buffer km** (default 20)
4. Copy the generated JSON from the output panel
5. Paste it as a top-level key into `config.json` and restart the server
The builder is also accessible from the Customizer's Export tab via the **GeoFilter Builder →** link.
For local/offline use without a running server, open `tools/geofilter-builder.html` directly in a browser.
## API endpoint
```
GET /api/config/geo-filter
```
The frontend reads this endpoint to display the active filter. No authentication is required (the endpoint returns config, not private data).
Returns the current geo filter configuration. Also includes a `writeEnabled` boolean indicating whether the `PUT` endpoint is available (i.e., server has a write-capable `apiKey`).
## GeoFilter Builder
The simplest way to create a polygon is the included visual builder:
**File:** `tools/geofilter-builder.html`
Open it directly in a browser — it runs entirely client-side, no server required:
```bash
# From the project root
open tools/geofilter-builder.html # macOS
xdg-open tools/geofilter-builder.html # Linux
start tools/geofilter-builder.html # Windows
```
PUT /api/config/geo-filter
```
**Workflow:**
Requires `X-API-Key` header. Saves the polygon to `config.json` and applies it in-memory immediately.
1. The map opens centered on Belgium by default. Navigate to your region.
2. Click on the map to add polygon vertices. Each click adds a numbered point.
3. Add at least 3 points to form a closed polygon.
4. Adjust **Buffer km** (default 20) to add a margin around the polygon edge.
5. The generated JSON block appears at the bottom of the page — copy it directly into `config.json`.
6. Use **↩ Undo** to remove the last point, **✕ Clear** to start over.
Request body:
```json
{"polygon": [[lat, lon], ...], "bufferKm": 20}
```
The output is a complete `{ "geo_filter": { ... } }` block ready to paste into `config.json`.
To clear the filter, send `{"polygon": null}`.
```
POST /api/admin/prune-geo-filter
POST /api/admin/prune-geo-filter?confirm=true
```
Requires `X-API-Key` header. Without `?confirm=true`, performs a dry run and returns the list of nodes that would be deleted. With `?confirm=true`, permanently deletes them from the database.
Response (dry run or confirmed):
```json
{"deleted": 5, "nodes": [{"pubKey": "...", "name": "NodeName", "lat": 51.12, "lon": 4.50}]}
```
## Cleaning up historical nodes
The ingestor prevents new out-of-bounds nodes from being ingested, but it does not retroactively remove nodes that were stored before the filter was configured. For that, use the prune script.
The ingestor prevents new out-of-bounds nodes from being ingested, but it does not retroactively remove nodes stored before the filter was configured.
### One-click prune from the Customizer (recommended)
If `writeEnabled` is true (server has a write-capable `apiKey`), the GeoFilter tab shows a **Prune nodes** section at the bottom:
1. Click **Preview** — the server dry-runs the deletion and lists every node that falls outside the current polygon + buffer. No data is deleted yet.
2. Review the list. It shows the node name (or public key) and coordinates.
3. Click **Confirm delete** to permanently remove those nodes from the database.
Nodes without GPS coordinates are always kept.
### CLI alternative (Python script)
**File:** `scripts/prune-nodes-outside-geo-filter.py`
```bash
# Dry run — shows what would be deleted without making any changes
# Dry run — shows what would be deleted without making changes
python3 scripts/prune-nodes-outside-geo-filter.py --dry-run
# Default paths: /app/data/meshcore.db and /app/config.json
@@ -104,11 +142,11 @@ python3 scripts/prune-nodes-outside-geo-filter.py
python3 scripts/prune-nodes-outside-geo-filter.py /path/to/meshcore.db \
--config /path/to/config.json
# In Docker — run inside the container
# In Docker
docker exec -it meshcore-analyzer \
python3 /app/scripts/prune-nodes-outside-geo-filter.py --dry-run
```
The script reads `geo_filter.polygon` and `geo_filter.bufferKm` from config, lists the nodes that fall outside, then asks for `yes` confirmation before deleting. Nodes without coordinates are always kept.
The script reads `geo_filter.polygon` and `geo_filter.bufferKm` from config, lists nodes that fall outside, then asks for `yes` confirmation before deleting. Nodes without coordinates are always kept.
This is a **one-time migration tool** — run it once after first configuring `geo_filter` to clean up pre-filter data. The ingestor handles all subsequent filtering automatically at ingest time.
Both the UI button and the script are **one-time migration tools** — run once after first configuring `geo_filter` to clean up pre-filter data. The ingestor handles all subsequent filtering automatically.
+3
View File
@@ -0,0 +1,3 @@
module github.com/meshcore-analyzer/prunequeue
go 1.22
+246
View File
@@ -0,0 +1,246 @@
// Package prunequeue defines the on-disk protocol used by the read-only
// server (cmd/server) to ask the writer-owning ingestor (cmd/ingestor) to
// delete nodes outside the configured geo_filter.
//
// Rationale: after #1283/#1289 the server opens SQLite with mode=ro, so it
// cannot execute DELETE statements. The one-click geo-prune feature still
// presents its HTTP API on the server (#669 M4), but the actual write is
// performed by the ingestor's maintenance loop. Communication uses small
// JSON marker files written into a directory next to the SQLite database
// (so it inherits the same backup/permissions story).
//
// Layout (under <dir(dbPath)>/prune-requests/):
//
// request-<id>.json — written by server when a confirmed prune is requested
// result-<id>.json — written by ingestor after the DELETE completes; the
// ingestor removes the request file in the same step
// using os.Rename (atomic on POSIX).
//
// The server polls result-<id>.json to surface progress on
// GET /api/admin/prune-geo-filter/status?id=<id>.
package prunequeue
import (
"crypto/rand"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"time"
)
// QueueDirName is the subdirectory (under the SQLite data dir) holding
// request/result marker files.
const QueueDirName = "prune-requests"
// Request is the payload the server writes to request-<id>.json. The
// ingestor honors the pubkeys list verbatim — the server is responsible
// for the TOCTOU snapshot (only nodes that were in the preview).
type Request struct {
ID string `json:"id"`
RequestedAt time.Time `json:"requestedAt"`
Reason string `json:"reason,omitempty"` // human-readable, e.g. "geo-prune"
Pubkeys []string `json:"pubkeys"`
}
// Result is what the ingestor writes to result-<id>.json after running
// the DELETE. Errors are captured as plain strings — the file is the
// only channel back to the server.
type Result struct {
ID string `json:"id"`
RequestedAt time.Time `json:"requestedAt"`
CompletedAt time.Time `json:"completedAt"`
Deleted int64 `json:"deleted"`
Error string `json:"error,omitempty"`
}
// NewID returns a 16-hex-char random id suitable for filenames. Random
// (not time-based) so concurrent requests on the same millisecond don't
// collide.
func NewID() string {
var b [8]byte
if _, err := rand.Read(b[:]); err != nil {
// Fall back to a time-based id so callers don't have to handle
// crypto/rand failure paths — collision probability remains
// negligible in practice.
return fmt.Sprintf("%016x", time.Now().UnixNano())
}
return hex.EncodeToString(b[:])
}
// QueueDir returns the absolute path of the queue directory, given the
// SQLite database path the ingestor and server share.
func QueueDir(dbPath string) string {
return filepath.Join(filepath.Dir(dbPath), QueueDirName)
}
// EnsureDir creates the queue directory if missing.
func EnsureDir(dbPath string) (string, error) {
dir := QueueDir(dbPath)
if err := os.MkdirAll(dir, 0o755); err != nil {
return "", err
}
return dir, nil
}
// validID rejects anything that could escape the queue directory.
func validID(id string) bool {
if id == "" || len(id) > 64 {
return false
}
for _, r := range id {
switch {
case r >= '0' && r <= '9', r >= 'a' && r <= 'f', r >= 'A' && r <= 'F':
default:
return false
}
}
return true
}
// RequestPath returns the full path for the request-<id>.json marker.
func RequestPath(dbPath, id string) (string, error) {
if !validID(id) {
return "", errors.New("invalid prune request id")
}
return filepath.Join(QueueDir(dbPath), "request-"+id+".json"), nil
}
// ResultPath returns the full path for the result-<id>.json marker.
func ResultPath(dbPath, id string) (string, error) {
if !validID(id) {
return "", errors.New("invalid prune request id")
}
return filepath.Join(QueueDir(dbPath), "result-"+id+".json"), nil
}
// WriteRequest atomically writes a request file (temp file + rename).
func WriteRequest(dbPath string, req Request) error {
if !validID(req.ID) {
return errors.New("invalid prune request id")
}
if _, err := EnsureDir(dbPath); err != nil {
return err
}
p, _ := RequestPath(dbPath, req.ID)
b, err := json.MarshalIndent(req, "", " ")
if err != nil {
return err
}
tmp := p + ".tmp"
if err := os.WriteFile(tmp, b, 0o644); err != nil {
return err
}
if err := os.Rename(tmp, p); err != nil {
_ = os.Remove(tmp)
return err
}
return nil
}
// WriteResult atomically writes a result file (temp file + rename),
// then removes the matching request file. Callers (the ingestor) hold
// the only writer.
func WriteResult(dbPath string, res Result) error {
if !validID(res.ID) {
return errors.New("invalid prune request id")
}
if _, err := EnsureDir(dbPath); err != nil {
return err
}
p, _ := ResultPath(dbPath, res.ID)
b, err := json.MarshalIndent(res, "", " ")
if err != nil {
return err
}
tmp := p + ".tmp"
if err := os.WriteFile(tmp, b, 0o644); err != nil {
return err
}
if err := os.Rename(tmp, p); err != nil {
_ = os.Remove(tmp)
return err
}
reqPath, _ := RequestPath(dbPath, res.ID)
_ = os.Remove(reqPath)
return nil
}
// ReadResult reads result-<id>.json. Returns (nil, nil) if the result
// file does not yet exist (request still pending or unknown id).
func ReadResult(dbPath, id string) (*Result, error) {
p, err := ResultPath(dbPath, id)
if err != nil {
return nil, err
}
b, err := os.ReadFile(p)
if err != nil {
if os.IsNotExist(err) {
return nil, nil
}
return nil, err
}
var r Result
if err := json.Unmarshal(b, &r); err != nil {
return nil, err
}
return &r, nil
}
// RequestExists returns true if request-<id>.json is still present
// (i.e. the ingestor has not processed it yet).
func RequestExists(dbPath, id string) (bool, error) {
p, err := RequestPath(dbPath, id)
if err != nil {
return false, err
}
_, err = os.Stat(p)
if err == nil {
return true, nil
}
if os.IsNotExist(err) {
return false, nil
}
return false, err
}
// ListPending returns all request-<id>.json files in the queue dir, in
// lexicographic order. Used by the ingestor's maintenance loop.
func ListPending(dbPath string) ([]string, error) {
dir := QueueDir(dbPath)
entries, err := os.ReadDir(dir)
if err != nil {
if os.IsNotExist(err) {
return nil, nil
}
return nil, err
}
var out []string
for _, e := range entries {
if e.IsDir() {
continue
}
name := e.Name()
if !strings.HasPrefix(name, "request-") || !strings.HasSuffix(name, ".json") {
continue
}
out = append(out, filepath.Join(dir, name))
}
return out, nil
}
// ReadRequest reads and parses a request file by full path.
func ReadRequest(path string) (*Request, error) {
b, err := os.ReadFile(path)
if err != nil {
return nil, err
}
var r Request
if err := json.Unmarshal(b, &r); err != nil {
return nil, err
}
return &r, nil
}
+68
View File
@@ -0,0 +1,68 @@
package prunequeue
import (
"path/filepath"
"testing"
"time"
)
func TestRoundTrip(t *testing.T) {
dir := t.TempDir()
dbPath := filepath.Join(dir, "x.db")
id := NewID()
req := Request{ID: id, RequestedAt: time.Now().UTC(), Reason: "test", Pubkeys: []string{"aa", "bb"}}
if err := WriteRequest(dbPath, req); err != nil {
t.Fatal(err)
}
pending, err := RequestExists(dbPath, id)
if err != nil || !pending {
t.Fatalf("RequestExists: pending=%v err=%v", pending, err)
}
list, err := ListPending(dbPath)
if err != nil || len(list) != 1 {
t.Fatalf("ListPending: %v / %v", list, err)
}
parsed, err := ReadRequest(list[0])
if err != nil || parsed.ID != id || len(parsed.Pubkeys) != 2 {
t.Fatalf("ReadRequest: %+v / %v", parsed, err)
}
// Writing the result removes the request marker.
if err := WriteResult(dbPath, Result{ID: id, RequestedAt: req.RequestedAt, CompletedAt: time.Now().UTC(), Deleted: 2}); err != nil {
t.Fatal(err)
}
pending, _ = RequestExists(dbPath, id)
if pending {
t.Error("request marker should be gone after WriteResult")
}
res, err := ReadResult(dbPath, id)
if err != nil || res == nil || res.Deleted != 2 {
t.Fatalf("ReadResult: %+v / %v", res, err)
}
}
func TestRejectsBadIDs(t *testing.T) {
dbPath := filepath.Join(t.TempDir(), "x.db")
for _, bad := range []string{"", "../escape", "abcg", "0123456789ABCDEFG/extra", "../../etc/passwd"} {
if _, err := RequestPath(dbPath, bad); err == nil {
t.Errorf("RequestPath should reject %q", bad)
}
if _, err := ResultPath(dbPath, bad); err == nil {
t.Errorf("ResultPath should reject %q", bad)
}
}
}
func TestReadResultMissing(t *testing.T) {
dbPath := filepath.Join(t.TempDir(), "x.db")
res, err := ReadResult(dbPath, "abcdef0123456789")
if err != nil {
t.Fatalf("unexpected err: %v", err)
}
if res != nil {
t.Errorf("expected nil result, got %+v", res)
}
}
+397 -1
View File
@@ -878,6 +878,16 @@
var _activeTab = 'branding';
var _styleEl = null;
// GeoFilter tab state
var _gfMap = null;
var _gfModalMap = null;
var _gfWriteEnabled = false;
var _gfPoints = [];
var _gfMarkers = [];
var _gfPolygon = null;
var _gfClosingLine = null;
var _gfLoaded = false; // true after initial server load
function esc(s) { var d = document.createElement('div'); d.textContent = s || ''; return d.innerHTML; }
function escAttr(s) { return (s || '').replace(/&/g, '&amp;').replace(/"/g, '&quot;').replace(/</g, '&lt;'); }
@@ -1004,6 +1014,7 @@
{ id: 'nodes', label: '🎯', title: 'Colors', badge: (function () { var n = _countOverrides('nodeColors') + _countOverrides('typeColors'); return n ? ' <span class="cv2-tab-badge">' + n + '</span>' : ''; })() },
{ id: 'home', label: '🏠', title: 'Home', badge: _tabBadge('home') },
{ id: 'display', label: '🖥️', title: 'Display', badge: (function () { var n = _countOverrides('timestamps') + (_isOverridden(null, 'distanceUnit') ? 1 : 0); return n ? ' <span class="cv2-tab-badge">' + n + '</span>' : ''; })() },
{ id: 'geofilter', label: '🗺️', title: 'GeoFilter' },
{ id: 'export', label: '📤', title: 'Export' }
];
return '<div class="cust-tabs">' + tabs.map(function (t) {
@@ -1256,6 +1267,383 @@
'</div>';
}
function _renderGeoFilter() {
return '<div class="cust-panel' + (_activeTab === 'geofilter' ? ' active' : '') + '" data-panel="geofilter">' +
'<p class="cust-section-title">Geographic Filter</p>' +
'<p style="font-size:12px;color:var(--text-muted);margin-bottom:12px">Shows the active geographic filter. Nodes outside this area are excluded at ingest time and in API responses.</p>' +
'<div style="position:relative;margin-bottom:8px">' +
'<div id="cv2-gf-map" style="height:200px;border-radius:6px;border:1px solid var(--border);background:var(--surface-1);cursor:pointer"></div>' +
'<div style="position:absolute;top:7px;right:7px;background:rgba(255,255,255,0.88);border-radius:4px;padding:3px 8px;font-size:11px;color:#444;pointer-events:none;box-shadow:0 1px 3px rgba(0,0,0,0.15)">🔍 click to expand</div>' +
'</div>' +
'<div id="cv2-gf-status" style="font-size:12px;color:var(--text-muted);margin-bottom:10px">Loading current filter…</div>' +
// Edit controls — hidden until server confirms write access (writeEnabled=true)
'<div id="cv2-gf-edit" style="display:none">' +
'<div style="display:flex;gap:8px;margin-bottom:10px;align-items:center">' +
'<label style="font-size:12px;color:var(--text-muted)">Buffer km:</label>' +
'<input type="number" id="cv2-gf-buffer" value="20" min="0" max="500" style="width:64px;padding:4px 8px;border:1px solid var(--border);border-radius:6px;background:var(--input-bg);color:var(--text);font-size:12px">' +
'</div>' +
'<div class="cust-field"><label>Server API Key</label>' +
'<input type="password" id="cv2-gf-apikey" placeholder="apiKey from config.json" style="width:100%;padding:5px 8px;border:1px solid var(--border);border-radius:6px;background:var(--input-bg);color:var(--text);font-size:12px">' +
'</div>' +
'<div style="display:flex;gap:8px;margin-top:12px">' +
'<button id="cv2-gf-save" style="padding:7px 16px;background:var(--accent);color:#fff;border:none;border-radius:6px;cursor:pointer;font-size:13px;font-weight:500">Save to server</button>' +
'<button id="cv2-gf-remove" style="padding:7px 14px;background:var(--surface-1);color:var(--status-red);border:1px solid var(--border);border-radius:6px;cursor:pointer;font-size:13px">Remove filter</button>' +
'</div>' +
'<div id="cv2-gf-msg" style="margin-top:8px;font-size:12px;display:none"></div>' +
// Prune section — only shown when a polygon is active (toggled in _initGeoFilterTab)
'<div id="cv2-gf-prune-section" style="display:none;margin-top:16px;border-top:1px solid var(--border);padding-top:14px">' +
'<p class="cust-section-title" style="font-size:13px;margin-bottom:6px">Prune historical nodes</p>' +
'<p style="font-size:11px;color:var(--text-muted);margin-bottom:10px">Remove nodes already in the database that fall outside the current filter. Run once after first enabling geo filtering.</p>' +
'<button id="cv2-gf-prune-preview" style="padding:6px 14px;background:var(--surface-1);color:var(--text-muted);border:1px solid var(--border);border-radius:6px;cursor:pointer;font-size:12px">Preview prune</button>' +
'<div id="cv2-gf-prune-result" style="display:none;margin-top:10px">' +
'<div id="cv2-gf-prune-list" style="font-size:11px;color:var(--text-muted);max-height:100px;overflow-y:auto;margin-bottom:8px;background:var(--surface-1);border:1px solid var(--border);border-radius:4px;padding:6px 8px"></div>' +
'<button id="cv2-gf-prune-confirm" style="padding:6px 14px;background:var(--status-red);color:#fff;border:none;border-radius:6px;cursor:pointer;font-size:12px;font-weight:500">Delete nodes</button>' +
'</div>' +
'<div id="cv2-gf-prune-msg" style="margin-top:8px;font-size:12px;display:none"></div>' +
'</div>' +
'</div>' +
'</div>';
}
function _gfOpenModal(container) {
var existing = document.getElementById('cv2-gf-modal-overlay');
if (existing) existing.remove();
if (_gfModalMap) { _gfModalMap.remove(); _gfModalMap = null; }
var overlay = document.createElement('div');
overlay.id = 'cv2-gf-modal-overlay';
overlay.style.cssText = 'position:fixed;inset:0;background:rgba(0,0,0,0.65);z-index:99999;display:flex;align-items:center;justify-content:center;';
var dialog = document.createElement('div');
dialog.style.cssText = 'width:92vw;height:86vh;background:#fff;border-radius:10px;display:flex;flex-direction:column;overflow:hidden;box-shadow:0 8px 32px rgba(0,0,0,0.4);';
var toolbarEl = document.createElement('div');
toolbarEl.style.cssText = 'padding:10px 14px;display:flex;gap:8px;align-items:center;border-bottom:1px solid #e0e0e0;background:#f5f5f5;flex-shrink:0;';
var title = document.createElement('span');
title.style.cssText = 'font-weight:600;color:#333;font-size:14px;';
title.textContent = _gfWriteEnabled ? 'Edit GeoFilter — click map to add points' : 'GeoFilter — read only';
toolbarEl.appendChild(title);
if (_gfWriteEnabled) {
var undoBtn = document.createElement('button');
undoBtn.id = 'cv2-gfm-undo';
undoBtn.textContent = '↩ Undo';
undoBtn.style.cssText = 'padding:5px 10px;background:#eee;color:#555;border:1px solid #ccc;border-radius:6px;cursor:pointer;font-size:12px;';
var clearBtn = document.createElement('button');
clearBtn.id = 'cv2-gfm-clear';
clearBtn.textContent = '✕ Clear';
clearBtn.style.cssText = 'padding:5px 10px;background:#fee;color:#c44;border:1px solid #fcc;border-radius:6px;cursor:pointer;font-size:12px;';
var countEl = document.createElement('span');
countEl.id = 'cv2-gfm-count';
countEl.style.cssText = 'font-size:12px;color:#888;';
var spacer = document.createElement('span');
spacer.style.cssText = 'flex:1;';
var doneBtn = document.createElement('button');
doneBtn.id = 'cv2-gfm-done';
doneBtn.textContent = 'Done';
doneBtn.style.cssText = 'padding:7px 18px;background:#4a9eff;color:#fff;border:none;border-radius:6px;cursor:pointer;font-size:13px;font-weight:500;';
toolbarEl.appendChild(undoBtn);
toolbarEl.appendChild(clearBtn);
toolbarEl.appendChild(countEl);
toolbarEl.appendChild(spacer);
toolbarEl.appendChild(doneBtn);
} else {
var spacer2 = document.createElement('span');
spacer2.style.cssText = 'flex:1;';
toolbarEl.appendChild(spacer2);
}
var closeBtn = document.createElement('button');
closeBtn.id = 'cv2-gfm-close';
closeBtn.textContent = _gfWriteEnabled ? 'Cancel' : 'Close';
closeBtn.style.cssText = 'padding:7px 14px;background:#eee;color:#555;border:1px solid #ccc;border-radius:6px;cursor:pointer;font-size:13px;';
toolbarEl.appendChild(closeBtn);
var mapDiv = document.createElement('div');
mapDiv.id = 'cv2-gf-modal-map';
mapDiv.style.cssText = 'flex:1;';
dialog.appendChild(toolbarEl);
dialog.appendChild(mapDiv);
overlay.appendChild(dialog);
document.body.appendChild(overlay);
var modalPoints = _gfPoints.map(function (p) { return [p[0], p[1]]; });
var modalMarkers = [];
var modalPolygon = null;
var modalClosingLine = null;
_gfModalMap = L.map(mapDiv, { zoomControl: true });
L.tileLayer('https://{s}.basemaps.cartocdn.com/light_all/{z}/{x}/{y}{r}.png', {
attribution: '© OpenStreetMap © CartoDB', maxZoom: 19
}).addTo(_gfModalMap);
function renderModal() {
if (modalPolygon) { _gfModalMap.removeLayer(modalPolygon); modalPolygon = null; }
if (modalClosingLine) { _gfModalMap.removeLayer(modalClosingLine); modalClosingLine = null; }
modalMarkers.forEach(function (m) { _gfModalMap.removeLayer(m); });
modalMarkers = [];
modalPoints.forEach(function (pt, i) {
var m = L.circleMarker(pt, { radius: 6, color: '#4a9eff', weight: 2, fillColor: '#4a9eff', fillOpacity: 0.9 })
.addTo(_gfModalMap)
.bindTooltip(String(i + 1), { permanent: true, direction: 'top', offset: [0, -8] });
modalMarkers.push(m);
});
if (modalPoints.length >= 3) {
modalPolygon = L.polygon(modalPoints, { color: '#4a9eff', weight: 2, fillColor: '#4a9eff', fillOpacity: 0.12 }).addTo(_gfModalMap);
} else if (modalPoints.length === 2) {
modalClosingLine = L.polyline(modalPoints, { color: '#4a9eff', weight: 2, dashArray: '5,5' }).addTo(_gfModalMap);
}
var ce = document.getElementById('cv2-gfm-count');
if (ce) ce.textContent = modalPoints.length + ' point' + (modalPoints.length !== 1 ? 's' : '');
}
function closeModal() {
if (_gfModalMap) { _gfModalMap.remove(); _gfModalMap = null; }
overlay.remove();
}
setTimeout(function () {
_gfModalMap.invalidateSize();
renderModal();
if (modalPoints.length >= 3) {
_gfModalMap.fitBounds(L.latLngBounds(modalPoints), { padding: [40, 40] });
} else {
_gfModalMap.setView([50.5, 4.4], 5);
}
}, 80);
if (_gfWriteEnabled) {
_gfModalMap.on('click', function (e) {
modalPoints.push([parseFloat(e.latlng.lat.toFixed(6)), parseFloat(e.latlng.lng.toFixed(6))]);
renderModal();
});
document.getElementById('cv2-gfm-undo').addEventListener('click', function () {
if (!modalPoints.length) return;
modalPoints.pop();
renderModal();
});
document.getElementById('cv2-gfm-clear').addEventListener('click', function () {
modalPoints = [];
renderModal();
});
document.getElementById('cv2-gfm-done').addEventListener('click', function () {
_gfPoints = modalPoints;
_gfRender();
var prune = container.querySelector('#cv2-gf-prune-section');
if (prune) prune.style.display = _gfPoints.length >= 3 ? '' : 'none';
_gfStatus(container, _gfPoints.length + ' point' + (_gfPoints.length !== 1 ? 's' : '') + '.');
closeModal();
});
}
closeBtn.addEventListener('click', closeModal);
overlay.addEventListener('click', function (e) { if (e.target === overlay) closeModal(); });
}
function _gfRender() {
if (!_gfMap) return;
if (_gfPolygon) { _gfMap.removeLayer(_gfPolygon); _gfPolygon = null; }
if (_gfClosingLine) { _gfMap.removeLayer(_gfClosingLine); _gfClosingLine = null; }
_gfMarkers.forEach(function (m) { _gfMap.removeLayer(m); });
_gfMarkers = [];
_gfPoints.forEach(function (pt, i) {
var m = L.circleMarker(pt, { radius: 6, color: '#4a9eff', weight: 2, fillColor: '#4a9eff', fillOpacity: 0.9 })
.addTo(_gfMap)
.bindTooltip(String(i + 1), { permanent: true, direction: 'top', offset: [0, -8] });
_gfMarkers.push(m);
});
if (_gfPoints.length >= 3) {
_gfPolygon = L.polygon(_gfPoints, { color: '#4a9eff', weight: 2, fillColor: '#4a9eff', fillOpacity: 0.12 }).addTo(_gfMap);
} else if (_gfPoints.length === 2) {
_gfClosingLine = L.polyline(_gfPoints, { color: '#4a9eff', weight: 2, dashArray: '5,5' }).addTo(_gfMap);
}
}
function _gfStatus(container, msg) {
var el = container.querySelector('#cv2-gf-status');
if (el) el.textContent = msg;
}
function _gfMsg(container, msg, ok) {
var el = container.querySelector('#cv2-gf-msg');
if (!el) return;
el.textContent = msg;
el.style.display = msg ? '' : 'none';
el.style.color = ok ? 'var(--status-green)' : 'var(--status-red)';
}
function _gfSave(container) {
if (_gfPoints.length < 3) { _gfMsg(container, 'Need at least 3 polygon points.', false); return; }
var apiKey = (container.querySelector('#cv2-gf-apikey') || {}).value || '';
if (!apiKey) { _gfMsg(container, 'API key required to save.', false); return; }
var bufferKm = parseFloat((container.querySelector('#cv2-gf-buffer') || {}).value) || 0;
fetch('/api/config/geo-filter', {
method: 'PUT',
headers: { 'Content-Type': 'application/json', 'X-API-Key': apiKey },
body: JSON.stringify({ polygon: _gfPoints, bufferKm: bufferKm })
}).then(function (r) {
if (!r.ok) return r.json().then(function (e) { throw new Error(e.error || ('HTTP ' + r.status)); });
_gfMsg(container, 'Saved. Filter is active immediately.', true);
_gfStatus(container, _gfPoints.length + ' points · bufferKm=' + bufferKm + ' · saved');
}).catch(function (e) { _gfMsg(container, 'Error: ' + e.message, false); });
}
function _gfRemove(container) {
var apiKey = (container.querySelector('#cv2-gf-apikey') || {}).value || '';
if (!apiKey) { _gfMsg(container, 'API key required.', false); return; }
if (!confirm('Remove geo filter? All nodes will be allowed through.')) return;
fetch('/api/config/geo-filter', {
method: 'PUT',
headers: { 'Content-Type': 'application/json', 'X-API-Key': apiKey },
body: JSON.stringify({ polygon: null })
}).then(function (r) {
if (!r.ok) return r.json().then(function (e) { throw new Error(e.error || ('HTTP ' + r.status)); });
_gfPoints = []; _gfLoaded = true;
_gfRender();
_gfStatus(container, 'No geo filter. Click the map to draw a polygon.');
_gfMsg(container, 'Geo filter removed.', true);
}).catch(function (e) { _gfMsg(container, 'Error: ' + e.message, false); });
}
var _gfPruneNodes = []; // nodes returned by last dry-run preview
function _gfPruneMsg(container, msg, ok) {
var el = container.querySelector('#cv2-gf-prune-msg');
if (!el) return;
el.textContent = msg;
el.style.display = msg ? '' : 'none';
el.style.color = ok ? 'var(--status-green)' : 'var(--status-red)';
}
function _gfPrunePreview(container) {
var apiKey = (container.querySelector('#cv2-gf-apikey') || {}).value || '';
if (!apiKey) { _gfPruneMsg(container, 'API key required.', false); return; }
var btn = container.querySelector('#cv2-gf-prune-preview');
if (btn) btn.textContent = 'Loading…';
fetch('/api/admin/prune-geo-filter', {
method: 'POST',
headers: { 'X-API-Key': apiKey }
}).then(function (r) {
if (!r.ok) return r.json().then(function (e) { throw new Error(e.error || ('HTTP ' + r.status)); });
return r.json();
}).then(function (data) {
if (btn) btn.textContent = 'Preview prune';
_gfPruneNodes = data.nodes || [];
var count = data.count || 0;
var resultEl = container.querySelector('#cv2-gf-prune-result');
var listEl = container.querySelector('#cv2-gf-prune-list');
var confirmBtn = container.querySelector('#cv2-gf-prune-confirm');
if (!resultEl || !listEl || !confirmBtn) return;
if (count === 0) {
_gfPruneMsg(container, 'No nodes outside the filter. Nothing to prune.', true);
resultEl.style.display = 'none';
return;
}
listEl.innerHTML = _gfPruneNodes.map(function (n) {
var coords = n.lat != null ? (' · ' + n.lat.toFixed(4) + ', ' + n.lon.toFixed(4)) : '';
return '<div>' + (n.name || n.pubkey.slice(0, 12)) + coords + '</div>';
}).join('');
confirmBtn.textContent = 'Delete ' + count + ' node' + (count !== 1 ? 's' : '');
resultEl.style.display = '';
_gfPruneMsg(container, '', true);
}).catch(function (e) {
if (btn) btn.textContent = 'Preview prune';
_gfPruneMsg(container, 'Error: ' + e.message, false);
});
}
function _gfPruneConfirm(container) {
if (!_gfPruneNodes.length) { _gfPruneMsg(container, 'Run preview first.', false); return; }
var apiKey = (container.querySelector('#cv2-gf-apikey') || {}).value || '';
if (!apiKey) { _gfPruneMsg(container, 'API key required.', false); return; }
var count = _gfPruneNodes.length;
if (!confirm('Delete ' + count + ' node' + (count !== 1 ? 's' : '') + ' from the database? This cannot be undone.')) return;
var pubkeys = _gfPruneNodes.map(function (n) { return n.pubkey; });
fetch('/api/admin/prune-geo-filter?confirm=true', {
method: 'POST',
headers: { 'X-API-Key': apiKey, 'Content-Type': 'application/json' },
body: JSON.stringify({ pubkeys: pubkeys })
}).then(function (r) {
if (!r.ok) return r.json().then(function (e) { throw new Error(e.error || ('HTTP ' + r.status)); });
return r.json();
}).then(function (data) {
_gfPruneNodes = [];
var resultEl = container.querySelector('#cv2-gf-prune-result');
if (resultEl) resultEl.style.display = 'none';
var n = data.deleted;
_gfPruneMsg(container, 'Deleted ' + n + ' node' + (n !== 1 ? 's' : '') + '.', true);
}).catch(function (e) { _gfPruneMsg(container, 'Error: ' + e.message, false); });
}
function _initGeoFilterTab(container) {
var mapEl = container.querySelector('#cv2-gf-map');
if (!mapEl || typeof L === 'undefined') return;
_gfMap = L.map(mapEl, { zoomControl: false, dragging: false, scrollWheelZoom: false, doubleClickZoom: false, touchZoom: false });
L.tileLayer('https://{s}.basemaps.cartocdn.com/light_all/{z}/{x}/{y}{r}.png', {
attribution: '© OpenStreetMap © CartoDB', maxZoom: 19
}).addTo(_gfMap);
if (!_gfLoaded) {
api('/config/geo-filter', { ttl: 0 }).then(function (gf) {
// Show edit controls only on servers that have a write-capable API key configured
if (gf && gf.writeEnabled) {
_gfWriteEnabled = true;
var editEl = container.querySelector('#cv2-gf-edit');
if (editEl) editEl.style.display = '';
}
if (gf && gf.polygon && gf.polygon.length >= 3) {
_gfPoints = gf.polygon.map(function (p) { return [p[0], p[1]]; });
var buf = container.querySelector('#cv2-gf-buffer');
if (buf) buf.value = gf.bufferKm || 0;
_gfRender();
if (_gfPolygon) _gfMap.fitBounds(_gfPolygon.getBounds(), { padding: [20, 20] });
_gfStatus(container, gf.polygon.length + ' points · bufferKm=' + (gf.bufferKm || 0));
// Show prune section when a polygon is active and write access is available
if (gf.writeEnabled) {
var pruneEl = container.querySelector('#cv2-gf-prune-section');
if (pruneEl) pruneEl.style.display = '';
}
} else {
_gfPoints = [];
_gfStatus(container, gf && gf.writeEnabled ? 'No geo filter. Click the map to open the editor.' : 'No geo filter configured.');
_gfMap.setView([50.5, 4.4], 5);
}
_gfLoaded = true;
setTimeout(function () { if (_gfMap) _gfMap.invalidateSize(); }, 100);
}).catch(function () {
_gfStatus(container, 'Could not load current filter.');
_gfMap.setView([50.5, 4.4], 5);
_gfLoaded = true;
setTimeout(function () { if (_gfMap) _gfMap.invalidateSize(); }, 100);
});
} else {
if (_gfPoints.length >= 3) {
_gfRender();
if (_gfPolygon) _gfMap.fitBounds(_gfPolygon.getBounds(), { padding: [20, 20] });
_gfStatus(container, _gfPoints.length + ' points.');
} else {
_gfMap.setView([50.5, 4.4], 5);
_gfStatus(container, _gfPoints.length ? _gfPoints.length + ' points (need at least 3).' : 'Click the map to draw a polygon.');
_gfRender();
}
setTimeout(function () { if (_gfMap) _gfMap.invalidateSize(); }, 100);
}
_gfMap.on('click', function () { _gfOpenModal(container); });
container.querySelector('#cv2-gf-save').addEventListener('click', function () { _gfSave(container); });
container.querySelector('#cv2-gf-remove').addEventListener('click', function () { _gfRemove(container); });
var prunePreviewBtn = container.querySelector('#cv2-gf-prune-preview');
var pruneConfirmBtn = container.querySelector('#cv2-gf-prune-confirm');
if (prunePreviewBtn) prunePreviewBtn.addEventListener('click', function () { _gfPrunePreview(container); });
if (pruneConfirmBtn) pruneConfirmBtn.addEventListener('click', function () { _gfPruneConfirm(container); });
}
function _renderExport() {
var delta = readOverrides();
var json = JSON.stringify(delta, null, 2);
@@ -1290,6 +1678,7 @@
_renderNodes() +
_renderHome() +
_renderDisplay() +
_renderGeoFilter() +
_renderExport() +
'</div>';
_bindEvents(container);
@@ -1358,11 +1747,15 @@
// Tab switching
container.querySelectorAll('.cust-tab').forEach(function (btn) {
btn.addEventListener('click', function () {
if (_gfMap) { _gfMap.remove(); _gfMap = null; _gfMarkers = []; _gfPolygon = null; _gfClosingLine = null; } if (_gfModalMap) { _gfModalMap.remove(); _gfModalMap = null; } var _ov = document.getElementById('cv2-gf-modal-overlay'); if (_ov) _ov.remove();
_activeTab = btn.dataset.tab;
_renderPanel(container);
});
});
// GeoFilter tab init
if (_activeTab === 'geofilter') _initGeoFilterTab(container);
// Preset buttons
container.querySelectorAll('.cust-preset-btn').forEach(function (btn) {
btn.addEventListener('click', function () {
@@ -1645,7 +2038,10 @@
'<div class="cv2-footer"><span id="cv2-save-status">All changes saved</span></div>';
document.body.appendChild(_panelEl);
_panelEl.querySelector('.cust-close').addEventListener('click', function () { _panelEl.classList.add('hidden'); });
_panelEl.querySelector('.cust-close').addEventListener('click', function () {
if (_gfMap) { _gfMap.remove(); _gfMap = null; _gfMarkers = []; _gfPolygon = null; _gfClosingLine = null; } if (_gfModalMap) { _gfModalMap.remove(); _gfModalMap = null; } var _ov = document.getElementById('cv2-gf-modal-overlay'); if (_ov) _ov.remove();
_panelEl.classList.add('hidden');
});
// Drag support
var header = _panelEl.querySelector('.cust-header');