mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-04-23 15:16:00 +00:00
## Summary
Observers that stop actively sending data now get removed after a
configurable retention period (default 14 days).
Previously, observers remained in the `observers` table forever. This
meant nodes that were once observers for an instance but are no longer
connected (even if still active in the mesh elsewhere) would continue
appearing in the observer list indefinitely.
## Key Design Decisions
- **Active data requirement**: `last_seen` is only updated when the
observer itself sends packets (via `stmtUpdateObserverLastSeen`). Being
seen by another node does NOT update this field. So an observer must
actively send data to stay listed.
- **Default: 14 days** — observers not seen in 14 days are removed
- **`-1` = keep forever** — for users who want observers to never be
removed
- **`0` = use default (14 days)** — same as not setting the field
- **Runs on startup + daily ticker** — staggered 3 minutes after metrics
prune to avoid DB contention
## Changes
| File | Change |
|------|--------|
| `cmd/ingestor/config.go` | Add `ObserverDays` to `RetentionConfig`,
add `ObserverDaysOrDefault()` |
| `cmd/ingestor/db.go` | Add `RemoveStaleObservers()` — deletes
observers with `last_seen` before cutoff |
| `cmd/ingestor/main.go` | Wire up startup + daily ticker for observer
retention |
| `cmd/server/config.go` | Add `ObserverDays` to `RetentionConfig`, add
`ObserverDaysOrDefault()` |
| `cmd/server/db.go` | Add `RemoveStaleObservers()` (server-side, uses
read-write connection) |
| `cmd/server/main.go` | Wire up startup + daily ticker, shutdown
cleanup |
| `cmd/server/routes.go` | Admin prune API now also removes stale
observers |
| `config.example.json` | Add `observerDays: 14` with documentation |
| `cmd/ingestor/coverage_boost_test.go` | 4 tests: basic removal, empty
store, keep forever (-1), default (0→14) |
| `cmd/server/config_test.go` | 4 tests: `ObserverDaysOrDefault` edge
cases |
## Config Example
```json
{
"retention": {
"nodeDays": 7,
"observerDays": 14,
"packetDays": 30,
"_comment": "observerDays: -1 = keep forever, 0 = use default (14)"
}
}
```
## Admin API
The `/api/admin/prune` endpoint now also removes stale observers (using
`observerDays` from config) and reports `observers_removed` in the
response alongside `packets_deleted`.
## Test Plan
- [x] `TestRemoveStaleObservers` — old observer removed, recent observer
kept
- [x] `TestRemoveStaleObserversNone` — empty store, no errors
- [x] `TestRemoveStaleObserversKeepForever` — `-1` keeps even year-old
observers
- [x] `TestRemoveStaleObserversDefault` — `0` defaults to 14 days
- [x] `TestObserverDaysOrDefault` (ingestor) —
nil/zero/positive/keep-forever
- [x] `TestObserverDaysOrDefault` (server) —
nil/zero/positive/keep-forever
- [x] Both binaries compile cleanly (`go build`)
- [ ] Manual: verify observer count decreases after retention period on
a live instance
226 lines
7.4 KiB
JSON
226 lines
7.4 KiB
JSON
{
|
|
"port": 3000,
|
|
"apiKey": "your-secret-api-key-here",
|
|
"retention": {
|
|
"nodeDays": 7,
|
|
"observerDays": 14,
|
|
"packetDays": 30,
|
|
"_comment": "nodeDays: nodes not seen in N days moved to inactive_nodes (default 7). observerDays: observers not sending data in N days are removed (-1 = keep forever, default 14). packetDays: transmissions older than N days are deleted (0 = disabled)."
|
|
},
|
|
"https": {
|
|
"cert": "/path/to/cert.pem",
|
|
"key": "/path/to/key.pem",
|
|
"_comment": "TLS cert/key paths for direct HTTPS. Most deployments use Caddy (included in Docker) for auto-TLS instead."
|
|
},
|
|
"branding": {
|
|
"siteName": "CoreScope",
|
|
"tagline": "Real-time MeshCore LoRa mesh network analyzer",
|
|
"logoUrl": null,
|
|
"faviconUrl": null,
|
|
"_comment": "Customize site name, tagline, logo, and favicon. logoUrl/faviconUrl can be absolute URLs or relative paths."
|
|
},
|
|
"theme": {
|
|
"accent": "#4a9eff",
|
|
"accentHover": "#6db3ff",
|
|
"navBg": "#0f0f23",
|
|
"navBg2": "#1a1a2e",
|
|
"statusGreen": "#45644c",
|
|
"statusYellow": "#b08b2d",
|
|
"statusRed": "#b54a4a",
|
|
"_comment": "CSS color overrides. Use the in-app Theme Customizer for live preview, then export values here."
|
|
},
|
|
"nodeColors": {
|
|
"repeater": "#dc2626",
|
|
"companion": "#2563eb",
|
|
"room": "#16a34a",
|
|
"sensor": "#d97706",
|
|
"observer": "#8b5cf6",
|
|
"_comment": "Marker/badge colors per node role. Used on map, nodes list, and live feed."
|
|
},
|
|
"home": {
|
|
"heroTitle": "CoreScope",
|
|
"heroSubtitle": "Find your nodes to start monitoring them.",
|
|
"steps": [
|
|
{
|
|
"emoji": "\ud83d\udce1",
|
|
"title": "Connect",
|
|
"description": "Link your node to the mesh"
|
|
},
|
|
{
|
|
"emoji": "\ud83d\udd0d",
|
|
"title": "Monitor",
|
|
"description": "Watch packets flow in real-time"
|
|
},
|
|
{
|
|
"emoji": "\ud83d\udcca",
|
|
"title": "Analyze",
|
|
"description": "Understand your network's health"
|
|
}
|
|
],
|
|
"checklist": [
|
|
{
|
|
"question": "How do I add my node?",
|
|
"answer": "Search for your node name or paste your public key."
|
|
},
|
|
{
|
|
"question": "What regions are covered?",
|
|
"answer": "Check the map page to see active observers and nodes."
|
|
}
|
|
],
|
|
"footerLinks": [
|
|
{
|
|
"label": "\ud83d\udce6 Packets",
|
|
"url": "#/packets"
|
|
},
|
|
{
|
|
"label": "\ud83d\uddfa\ufe0f Network Map",
|
|
"url": "#/map"
|
|
},
|
|
{
|
|
"label": "\ud83d\udd34 Live",
|
|
"url": "#/live"
|
|
},
|
|
{
|
|
"label": "\ud83d\udce1 All Nodes",
|
|
"url": "#/nodes"
|
|
},
|
|
{
|
|
"label": "\ud83d\udcac Channels",
|
|
"url": "#/channels"
|
|
}
|
|
],
|
|
"_comment": "Customize the landing page hero, onboarding steps, FAQ, and footer links."
|
|
},
|
|
"mqtt": {
|
|
"broker": "mqtt://localhost:1883",
|
|
"topic": "meshcore/+/+/packets",
|
|
"_comment": "Legacy single-broker config. Prefer mqttSources[] for multiple brokers."
|
|
},
|
|
"mqttSources": [
|
|
{
|
|
"name": "local",
|
|
"broker": "mqtt://localhost:1883",
|
|
"topics": [
|
|
"meshcore/+/+/packets",
|
|
"meshcore/#"
|
|
]
|
|
},
|
|
{
|
|
"name": "lincomatic",
|
|
"broker": "mqtts://mqtt.lincomatic.com:8883",
|
|
"username": "your-username",
|
|
"password": "your-password",
|
|
"rejectUnauthorized": false,
|
|
"topics": [
|
|
"meshcore/SJC/#",
|
|
"meshcore/SFO/#",
|
|
"meshcore/OAK/#",
|
|
"meshcore/MRY/#"
|
|
],
|
|
"iataFilter": [
|
|
"SJC",
|
|
"SFO",
|
|
"OAK",
|
|
"MRY"
|
|
]
|
|
}
|
|
],
|
|
"channelKeys": {
|
|
"Public": "8b3387e9c5cdea6ac9e5edbaa115cd72"
|
|
},
|
|
"hashChannels": [
|
|
"#LongFast",
|
|
"#test",
|
|
"#sf",
|
|
"#wardrive",
|
|
"#yo",
|
|
"#bot",
|
|
"#queer",
|
|
"#bookclub",
|
|
"#shtf"
|
|
],
|
|
"healthThresholds": {
|
|
"infraDegradedHours": 24,
|
|
"infraSilentHours": 72,
|
|
"nodeDegradedHours": 1,
|
|
"nodeSilentHours": 24,
|
|
"_comment": "How long (hours) before nodes show as degraded/silent. 'infra' = repeaters & rooms, 'node' = companions & others."
|
|
},
|
|
"defaultRegion": "SJC",
|
|
"mapDefaults": {
|
|
"center": [
|
|
37.45,
|
|
-122.0
|
|
],
|
|
"zoom": 9
|
|
},
|
|
"geo_filter": {
|
|
"polygon": [
|
|
[37.80, -122.52],
|
|
[37.80, -121.80],
|
|
[37.20, -121.80],
|
|
[37.20, -122.52]
|
|
],
|
|
"bufferKm": 20,
|
|
"_comment": "Optional. Restricts ingestion and API responses to nodes within the polygon + bufferKm. Polygon is an array of [lat, lon] pairs (minimum 3). Use tools/geofilter-builder.html to draw a polygon visually. Remove this section to disable filtering. Nodes with no GPS fix are always allowed through."
|
|
},
|
|
"regions": {
|
|
"SJC": "San Jose, US",
|
|
"SFO": "San Francisco, US",
|
|
"OAK": "Oakland, US",
|
|
"MRY": "Monterey, US"
|
|
},
|
|
"cacheTTL": {
|
|
"stats": 10,
|
|
"nodeDetail": 300,
|
|
"nodeHealth": 300,
|
|
"nodeList": 90,
|
|
"bulkHealth": 600,
|
|
"networkStatus": 600,
|
|
"observers": 300,
|
|
"channels": 15,
|
|
"channelMessages": 10,
|
|
"analyticsRF": 1800,
|
|
"analyticsTopology": 1800,
|
|
"analyticsChannels": 1800,
|
|
"analyticsHashSizes": 3600,
|
|
"analyticsSubpaths": 3600,
|
|
"analyticsSubpathDetail": 3600,
|
|
"nodeAnalytics": 60,
|
|
"nodeSearch": 10,
|
|
"invalidationDebounce": 30,
|
|
"_comment": "All values in seconds. Server uses these directly. Client fetches via /api/config/cache."
|
|
},
|
|
"liveMap": {
|
|
"propagationBufferMs": 5000,
|
|
"_comment": "How long (ms) to buffer incoming observations of the same packet before animating. Mesh packets propagate through multiple paths and arrive at different observers over several seconds. This window collects all observations of a single transmission so the live map can animate them simultaneously as one realistic propagation event. Set higher for wide meshes with many observers, lower for snappier animations. 5000ms captures ~95% of observations for a typical mesh."
|
|
},
|
|
"timestamps": {
|
|
"defaultMode": "ago",
|
|
"timezone": "local",
|
|
"formatPreset": "iso",
|
|
"customFormat": "",
|
|
"allowCustomFormat": false,
|
|
"_comment": "defaultMode: ago|local|iso. timezone: local|utc. formatPreset: iso|us|eu. customFormat: strftime-style (requires allowCustomFormat: true)."
|
|
},
|
|
"packetStore": {
|
|
"maxMemoryMB": 1024,
|
|
"estimatedPacketBytes": 450,
|
|
"_comment": "In-memory packet store. maxMemoryMB caps RAM usage. All packets loaded on startup, served from RAM."
|
|
},
|
|
"resolvedPath": {
|
|
"backfillHours": 24,
|
|
"_comment": "How far back (hours) the async backfill scans for observations with NULL resolved_path. Default: 24. Set higher to backfill older data, lower to speed up startup."
|
|
},
|
|
"neighborGraph": {
|
|
"maxAgeDays": 5,
|
|
"_comment": "Neighbor edges older than this many days are pruned on startup and daily. Default: 5."
|
|
},
|
|
"_comment_mqttSources": "Each source connects to an MQTT broker. topics: what to subscribe to. iataFilter: only ingest packets from these regions (optional).",
|
|
"_comment_channelKeys": "Hex keys for decrypting channel messages. Key name = channel display name. public channel key is well-known.",
|
|
"_comment_hashChannels": "Channel names whose keys are derived via SHA256. Key = SHA256(name)[:16]. Listed here so the ingestor can auto-derive keys.",
|
|
"_comment_defaultRegion": "IATA code shown by default in region filters.",
|
|
"_comment_mapDefaults": "Initial map center [lat, lon] and zoom level.",
|
|
"_comment_regions": "IATA code to display name mapping. Packets are tagged with region codes by MQTT topic structure."
|
|
}
|