This PR replaces the strict, hardcoded limits on API list endpoints
(introduced in the recent security patch) with a new
operator-configurable `listLimits` block. This change is needed as issue
1540's implementation introduced a 500max node limit on the live map or
any other function that leverages the api/nodes backend.
Previously, we attempted to bypass public caps for internal UI requests
using a heuristic based on browser headers (`Sec-Fetch-Site`). Following
review, we decided to drop that heuristic entirely to eliminate any
security-by-browser-convention surface area.
Instead, `queryLimit()` returns to its original, mathematically simple
bounds-checking shape, and the absolute maximums are now drawn from
`config.json`. This provides equal DoS protection against all callers
while allowing server operators to tune the ceilings based on the size
of their mesh (e.g. embedded devices can tighten the knobs, regional
hubs can raise them).
### Changes Made:
- **`config.go`**: Introduced a `ListLimits` config struct containing
`PacketsMax`, `NodesMax`, `AnalyticsMax`, and `ChannelMessagesMax`.
Added safe initialization to ensure default caps (10000, 2000, 200, 500
respectively) apply even if the block is omitted from the config.
- **`clamp_limit.go`**: Deleted `isInternalUIRequest` entirely and
restored `queryLimit` to its original signature (`r, def, max`).
- **`routes.go`**: Replaced all hardcoded integer ceilings on list
endpoints (`/api/packets`, `/api/nodes`, etc.) with
`s.cfg.ListLimits.*`.
- **`config.example.json`**: Added the `listLimits` block with
documentation to guide new operators.
- **`clamp_limit_test.go`**: Purged all header-heuristic testing.
### Verification:
- All 611 backend unit tests pass (`npm run test:unit`).
- Bounds-checking math continues to enforce hard DoS clipping exactly at
the operator's specified configuration limit.
---------
Co-authored-by: mc-bot <bot@openclaw.local>
Co-authored-by: openclaw-bot <bot@openclaw>
Closes#1552.
## What
Make observer `Online` / `Stale` / `Offline` thresholds
operator-configurable via `config.json`'s existing `healthThresholds`
block — and **raise the defaults** from 10 min / 60 min to **60 min /
1440 min (1 h / 24 h)** so they match the node thresholds and stop
producing flap out of the box.
⚠️ **This is a default behavior change.** Operators who want the old
aggressive 10-min Online threshold must opt in via:
```json
"healthThresholds": { "observerOnlineMinutes": 10 }
```
## Why
Per #1552: the `600000` / `3600000` constants in `public/observers.js`
were not tunable, *and* 10 min is wrong as a default. Wide-geo,
low-traffic meshes legitimately see observers go quiet for >10 min
between reports, and operators behind a CDN (#1551) get cached
`last_seen` values that can push the observer 15+ min behind reality —
guaranteeing flap at the 10-min threshold. The meshat.se operator (43
observers, v3.8.3) reports exactly this pattern.
Defaults raised from 10 / 60 minutes to 60 / 1440 minutes (1 h / 24 h)
to match the node thresholds for consistency and eliminate flap on
low-traffic / CDN-fronted instances. Operators wanting the old 10-min
Online behavior can set `observerOnlineMinutes: 10` in config.
## Changes
Backend (`cmd/server/config.go`):
- `HealthThresholds` gains `ObserverOnlineMinutes` /
`ObserverStaleMinutes` (int).
- `GetHealthThresholds()` defaults to **60 / 1440** when zero/absent.
- `ToClientMs()` emits `observerOnlineMs` / `observerStaleMs`, picked up
by the existing `/api/config-public` → `roles.js`
`Object.assign(HEALTH_THRESHOLDS, …)` pipeline.
`config.example.json`: new `observerOnlineMinutes` /
`observerStaleMinutes` keys (60 / 1440) + `_comment_observerThresholds`
explaining the rationale and opt-out.
Frontend:
- `public/observers.js` `healthStatus()` — reads from
`window.HEALTH_THRESHOLDS.observerOnlineMs / observerStaleMs`, falls
back to **3600000 / 86400000** (matching the new Go defaults for the
pre-`/api/config-public` window).
- `public/observer-detail.js` — same refactor (was previously hardcoded
`600000` + misusing `nodeDegradedMs` for the Stale boundary).
## Backward compat
- API shape: unchanged — only adds two optional keys.
- Config: unchanged keys / no renames.
- Default behavior: **changed** — operators relying on the implicit
10/60 must opt in (one config line).
## TDD
- RED 1 (`ee19058f`): assertions on the new fields + `ToClientMs` keys +
`healthStatus` reading from `window.HEALTH_THRESHOLDS`. CI:
[failure](https://github.com/Kpa-clawbot/CoreScope/actions/runs/26945264822).
- GREEN 1 (`30cfbf7a`): configurability landed (defaults still old
10/60). CI:
[success](https://github.com/Kpa-clawbot/CoreScope/actions/runs/26945220598).
- RED 2 (`2649cf35`): pin new 60/1440 defaults — empty-config Go path +
JS `healthStatus` with no `HEALTH_THRESHOLDS`. CI must fail.
- GREEN 2 (`5ef85bca`): bump Go defaults to 60/1440, JS fallbacks to
3600000/86400000, `config.example.json` updated. CI must pass.
## Preflight
Clean (exit 0). `cross-stack` ack in commit messages — single feature
spans Go + JSON + JS readers.
## Not in scope
- Customizer UI for editing the thresholds (config-only per issue).
- Node/infra thresholds (unchanged).
- The deeper observer-flap root cause (#1551 cache-control is a separate
PR in flight).
---------
Co-authored-by: corescope-bot <bot@corescope>
Co-authored-by: mc-bot <bot@meshcore.local>
## Summary
Observers that stop actively sending data now get removed after a
configurable retention period (default 14 days).
Previously, observers remained in the `observers` table forever. This
meant nodes that were once observers for an instance but are no longer
connected (even if still active in the mesh elsewhere) would continue
appearing in the observer list indefinitely.
## Key Design Decisions
- **Active data requirement**: `last_seen` is only updated when the
observer itself sends packets (via `stmtUpdateObserverLastSeen`). Being
seen by another node does NOT update this field. So an observer must
actively send data to stay listed.
- **Default: 14 days** — observers not seen in 14 days are removed
- **`-1` = keep forever** — for users who want observers to never be
removed
- **`0` = use default (14 days)** — same as not setting the field
- **Runs on startup + daily ticker** — staggered 3 minutes after metrics
prune to avoid DB contention
## Changes
| File | Change |
|------|--------|
| `cmd/ingestor/config.go` | Add `ObserverDays` to `RetentionConfig`,
add `ObserverDaysOrDefault()` |
| `cmd/ingestor/db.go` | Add `RemoveStaleObservers()` — deletes
observers with `last_seen` before cutoff |
| `cmd/ingestor/main.go` | Wire up startup + daily ticker for observer
retention |
| `cmd/server/config.go` | Add `ObserverDays` to `RetentionConfig`, add
`ObserverDaysOrDefault()` |
| `cmd/server/db.go` | Add `RemoveStaleObservers()` (server-side, uses
read-write connection) |
| `cmd/server/main.go` | Wire up startup + daily ticker, shutdown
cleanup |
| `cmd/server/routes.go` | Admin prune API now also removes stale
observers |
| `config.example.json` | Add `observerDays: 14` with documentation |
| `cmd/ingestor/coverage_boost_test.go` | 4 tests: basic removal, empty
store, keep forever (-1), default (0→14) |
| `cmd/server/config_test.go` | 4 tests: `ObserverDaysOrDefault` edge
cases |
## Config Example
```json
{
"retention": {
"nodeDays": 7,
"observerDays": 14,
"packetDays": 30,
"_comment": "observerDays: -1 = keep forever, 0 = use default (14)"
}
}
```
## Admin API
The `/api/admin/prune` endpoint now also removes stale observers (using
`observerDays` from config) and reports `observers_removed` in the
response alongside `packets_deleted`.
## Test Plan
- [x] `TestRemoveStaleObservers` — old observer removed, recent observer
kept
- [x] `TestRemoveStaleObserversNone` — empty store, no errors
- [x] `TestRemoveStaleObserversKeepForever` — `-1` keeps even year-old
observers
- [x] `TestRemoveStaleObserversDefault` — `0` defaults to 14 days
- [x] `TestObserverDaysOrDefault` (ingestor) —
nil/zero/positive/keep-forever
- [x] `TestObserverDaysOrDefault` (server) —
nil/zero/positive/keep-forever
- [x] Both binaries compile cleanly (`go build`)
- [ ] Manual: verify observer count decreases after retention period on
a live instance
Change healthThresholds config from milliseconds to hours for readability.
Config keys: infraDegradedHours, infraSilentHours, nodeDegradedHours, nodeSilentHours.
Defaults: infra degraded 24h, silent 72h; node degraded 1h, silent 24h.
- Config stored in hours, converted to ms at comparison time
- /api/config/client sends ms to frontend (backward compatible)
- Frontend tooltips use dynamic thresholds instead of hardcoded strings
- Added healthThresholds section to config.example.json
- Updated Go and Node.js servers, tests
