Closes#1552.
## What
Make observer `Online` / `Stale` / `Offline` thresholds
operator-configurable via `config.json`'s existing `healthThresholds`
block — and **raise the defaults** from 10 min / 60 min to **60 min /
1440 min (1 h / 24 h)** so they match the node thresholds and stop
producing flap out of the box.
⚠️ **This is a default behavior change.** Operators who want the old
aggressive 10-min Online threshold must opt in via:
```json
"healthThresholds": { "observerOnlineMinutes": 10 }
```
## Why
Per #1552: the `600000` / `3600000` constants in `public/observers.js`
were not tunable, *and* 10 min is wrong as a default. Wide-geo,
low-traffic meshes legitimately see observers go quiet for >10 min
between reports, and operators behind a CDN (#1551) get cached
`last_seen` values that can push the observer 15+ min behind reality —
guaranteeing flap at the 10-min threshold. The meshat.se operator (43
observers, v3.8.3) reports exactly this pattern.
Defaults raised from 10 / 60 minutes to 60 / 1440 minutes (1 h / 24 h)
to match the node thresholds for consistency and eliminate flap on
low-traffic / CDN-fronted instances. Operators wanting the old 10-min
Online behavior can set `observerOnlineMinutes: 10` in config.
## Changes
Backend (`cmd/server/config.go`):
- `HealthThresholds` gains `ObserverOnlineMinutes` /
`ObserverStaleMinutes` (int).
- `GetHealthThresholds()` defaults to **60 / 1440** when zero/absent.
- `ToClientMs()` emits `observerOnlineMs` / `observerStaleMs`, picked up
by the existing `/api/config-public` → `roles.js`
`Object.assign(HEALTH_THRESHOLDS, …)` pipeline.
`config.example.json`: new `observerOnlineMinutes` /
`observerStaleMinutes` keys (60 / 1440) + `_comment_observerThresholds`
explaining the rationale and opt-out.
Frontend:
- `public/observers.js` `healthStatus()` — reads from
`window.HEALTH_THRESHOLDS.observerOnlineMs / observerStaleMs`, falls
back to **3600000 / 86400000** (matching the new Go defaults for the
pre-`/api/config-public` window).
- `public/observer-detail.js` — same refactor (was previously hardcoded
`600000` + misusing `nodeDegradedMs` for the Stale boundary).
## Backward compat
- API shape: unchanged — only adds two optional keys.
- Config: unchanged keys / no renames.
- Default behavior: **changed** — operators relying on the implicit
10/60 must opt in (one config line).
## TDD
- RED 1 (`ee19058f`): assertions on the new fields + `ToClientMs` keys +
`healthStatus` reading from `window.HEALTH_THRESHOLDS`. CI:
[failure](https://github.com/Kpa-clawbot/CoreScope/actions/runs/26945264822).
- GREEN 1 (`30cfbf7a`): configurability landed (defaults still old
10/60). CI:
[success](https://github.com/Kpa-clawbot/CoreScope/actions/runs/26945220598).
- RED 2 (`2649cf35`): pin new 60/1440 defaults — empty-config Go path +
JS `healthStatus` with no `HEALTH_THRESHOLDS`. CI must fail.
- GREEN 2 (`5ef85bca`): bump Go defaults to 60/1440, JS fallbacks to
3600000/86400000, `config.example.json` updated. CI must pass.
## Preflight
Clean (exit 0). `cross-stack` ack in commit messages — single feature
spans Go + JSON + JS readers.
## Not in scope
- Customizer UI for editing the thresholds (config-only per issue).
- Node/infra thresholds (unchanged).
- The deeper observer-flap root cause (#1551 cache-control is a separate
PR in flight).
---------
Co-authored-by: corescope-bot <bot@corescope>
Co-authored-by: mc-bot <bot@meshcore.local>
## Summary
Observers that stop actively sending data now get removed after a
configurable retention period (default 14 days).
Previously, observers remained in the `observers` table forever. This
meant nodes that were once observers for an instance but are no longer
connected (even if still active in the mesh elsewhere) would continue
appearing in the observer list indefinitely.
## Key Design Decisions
- **Active data requirement**: `last_seen` is only updated when the
observer itself sends packets (via `stmtUpdateObserverLastSeen`). Being
seen by another node does NOT update this field. So an observer must
actively send data to stay listed.
- **Default: 14 days** — observers not seen in 14 days are removed
- **`-1` = keep forever** — for users who want observers to never be
removed
- **`0` = use default (14 days)** — same as not setting the field
- **Runs on startup + daily ticker** — staggered 3 minutes after metrics
prune to avoid DB contention
## Changes
| File | Change |
|------|--------|
| `cmd/ingestor/config.go` | Add `ObserverDays` to `RetentionConfig`,
add `ObserverDaysOrDefault()` |
| `cmd/ingestor/db.go` | Add `RemoveStaleObservers()` — deletes
observers with `last_seen` before cutoff |
| `cmd/ingestor/main.go` | Wire up startup + daily ticker for observer
retention |
| `cmd/server/config.go` | Add `ObserverDays` to `RetentionConfig`, add
`ObserverDaysOrDefault()` |
| `cmd/server/db.go` | Add `RemoveStaleObservers()` (server-side, uses
read-write connection) |
| `cmd/server/main.go` | Wire up startup + daily ticker, shutdown
cleanup |
| `cmd/server/routes.go` | Admin prune API now also removes stale
observers |
| `config.example.json` | Add `observerDays: 14` with documentation |
| `cmd/ingestor/coverage_boost_test.go` | 4 tests: basic removal, empty
store, keep forever (-1), default (0→14) |
| `cmd/server/config_test.go` | 4 tests: `ObserverDaysOrDefault` edge
cases |
## Config Example
```json
{
"retention": {
"nodeDays": 7,
"observerDays": 14,
"packetDays": 30,
"_comment": "observerDays: -1 = keep forever, 0 = use default (14)"
}
}
```
## Admin API
The `/api/admin/prune` endpoint now also removes stale observers (using
`observerDays` from config) and reports `observers_removed` in the
response alongside `packets_deleted`.
## Test Plan
- [x] `TestRemoveStaleObservers` — old observer removed, recent observer
kept
- [x] `TestRemoveStaleObserversNone` — empty store, no errors
- [x] `TestRemoveStaleObserversKeepForever` — `-1` keeps even year-old
observers
- [x] `TestRemoveStaleObserversDefault` — `0` defaults to 14 days
- [x] `TestObserverDaysOrDefault` (ingestor) —
nil/zero/positive/keep-forever
- [x] `TestObserverDaysOrDefault` (server) —
nil/zero/positive/keep-forever
- [x] Both binaries compile cleanly (`go build`)
- [ ] Manual: verify observer count decreases after retention period on
a live instance
Change healthThresholds config from milliseconds to hours for readability.
Config keys: infraDegradedHours, infraSilentHours, nodeDegradedHours, nodeSilentHours.
Defaults: infra degraded 24h, silent 72h; node degraded 1h, silent 24h.
- Config stored in hours, converted to ms at comparison time
- /api/config/client sends ms to frontend (backward compatible)
- Frontend tooltips use dynamic thresholds instead of hardcoded strings
- Added healthThresholds section to config.example.json
- Updated Go and Node.js servers, tests
