mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-06-04 21:01:23 +00:00
13bdee57d4
## What Three of the four P0s from #1481's scale-test findings. Each cuts a distinct hot path; together they target /api/observers, /api/analytics/neighbor-graph, and /api/observers/{id}/analytics — the top three live offenders. ### P0-1: 5-min atomic-pointer cache for default neighbor-graph response - Live p95 10.8s on the most-trafficked organic endpoint. - Background recomputer (5-min cadence per operator directive) builds the default-filter (`minCount=5 minScore=0.1`, no region, no role) `NeighborGraphResponse` and stores it via `atomic.Pointer`. - `handleNeighborGraph` short-circuits on the default shape; non-default filters take the extracted `computeNeighborGraphResponse` path (identical semantics to the previous inline build). ### P0-2: cache parsed `StoreObs.Timestamp` + drop RLock window - `handleObserverAnalytics` re-parsed the RFC3339 timestamp three times per observation, for 60k+ observations per active observer, under `s.store.mu.RLock` — blocking writers for the full scan. - `StoreObs.ParsedTime()` parses once via `sync.Once` (mirrors `StoreTx.ParsedDecoded`). - Handler snapshots the `byObserver[id]` pointer slice, releases the RLock immediately, then iterates locally. ### P0-3: 30s cache for `/api/observers` + sargable `IN` + covering index - Three SQL queries on every request → ~1.7s p50 at 50-concurrent. - Atomic-pointer 30s cache for the default (no-filter) query. - `GetNodeLocationsByKeys` drops `LOWER(public_key) IN (...)` (non-sargable); callers pre-lowercase in Go and the plain `IN` matches the existing `public_key` index. - New ingestor migration `obs_observer_ts_idx_v1` adds composite index `idx_observations_observer_idx_timestamp(observer_idx, timestamp)` so `GetObserverPacketCounts` can resolve its GROUP-BY + range filter from the index without scanning the 1.9M-row observations table. ### P0-4: deferred `perfMiddleware`'s global mutex was claimed to serialize every API request. A direct test (`50 concurrent requests through the middleware, handler sleeps 20ms each`) shows total elapsed ≈ 25ms, not 1s — the lock is held only for the post-handler bookkeeping (a few µs). Real impact is below measurement noise. Skipping to avoid invasive churn on PerfStats consumers without a demonstrable win. ## Test plan Red → green per P0: - `observers_cache_test.go` — handler reads `s.observersCache` before SQL, TTL boundary, atomic.Pointer (no mutex contention). - `storeobs_parsedtime_test.go` — parses three timestamp shapes, caches result, no race under concurrent readers. - `neighbor_graph_cache_test.go` — handler serves from atomic pointer when set, bypasses cache when `?region=` (or any non-default filter) is passed. Full server + ingestor suites pass: `go test -count=1 ./...`. ## Perf proof Before/after p50/p95/p99 (50 requests × 50 concurrent) against prod (before) and staging once CI deploys (after) will be posted as a PR comment per the operator's "no merge without proof of improvement" gate. Closes #1481 ## TDD exemption — P0-1 and P0-2 (net-new surfaces, AGENTS.md) Per CoreScope `AGENTS.md` § "Exemptions": **net-new code surfaces with no prior tests to break** may land tests in the same PR without a strict test-first → impl commit split. - **P0-1 (neighbor-graph atomic-pointer cache)** — `neighborGraphCache`, `recomputeNeighborGraphCache`, `loadNeighborGraphCacheBytes`, `startNeighborGraphRecomputer` and the default-shape short-circuit in `handleNeighborGraph` were brand-new code with no pre-existing assertions covering them. There was no green test to first turn red. - **P0-2 (cached `StoreObs.Timestamp` + RLock window drop)** — `StoreObs.ParsedTime()` and the snapshot+release pattern in `handleObserverAnalytics` were new surfaces; the prior code did the parse inline per call with no behavioural test to break. P0-3 was authored properly red-then-green (commit `6e63ec6a` red, then `83ae129b` green) and does NOT use this exemption. ## Default-filter detection vs frontend reality (#1483 follow-up) The Neighbor Graph analytics tab in `public/analytics.js` fetches `/analytics/neighbor-graph?min_count=1&min_score=0` because the client-side sliders need the full edge set to filter from. That shape did NOT match the `(5, 0.1)` cached default, so the UI tab still paid the cold compute cost despite #1481 P0-1. The #1483 follow-up commit caches BOTH shapes in the same recomputer pass: - `(minCount=5, minScore=0.1, no region, no role)` — `live.js` affinity-scoring consumer. - `(minCount=1, minScore=0, no region, no role)` — analytics tab. Both are served from `atomic.Pointer` with an `X-Cache-Age-Seconds` header. The per-shape cost in the background goroutine is roughly linear in edge count; total recompute time stays well under the 5-minute cadence on prod-scale graphs. --------- Co-authored-by: openclaw-bot <bot@openclaw.dev> Co-authored-by: mc-bot <mc-bot@users.noreply.github.com>
655 lines
22 KiB
Go
655 lines
22 KiB
Go
package main
|
|
|
|
import (
|
|
"encoding/json"
|
|
"fmt"
|
|
"log"
|
|
"os"
|
|
"path/filepath"
|
|
"strings"
|
|
"sync"
|
|
"time"
|
|
|
|
"github.com/meshcore-analyzer/dbconfig"
|
|
"github.com/meshcore-analyzer/geofilter"
|
|
)
|
|
|
|
// AreaEntry defines a geographic area by polygon or bounding box.
|
|
type AreaEntry struct {
|
|
Label string `json:"label"`
|
|
Polygon [][2]float64 `json:"polygon,omitempty"`
|
|
LatMin *float64 `json:"latMin,omitempty"`
|
|
LatMax *float64 `json:"latMax,omitempty"`
|
|
LonMin *float64 `json:"lonMin,omitempty"`
|
|
LonMax *float64 `json:"lonMax,omitempty"`
|
|
}
|
|
|
|
// Config mirrors the Node.js config.json structure (read-only fields).
|
|
type Config struct {
|
|
Port int `json:"port"`
|
|
APIKey string `json:"apiKey"`
|
|
DBPath string `json:"dbPath"`
|
|
|
|
// NodeBlacklist is a list of public keys to exclude from all API responses.
|
|
// Blacklisted nodes are hidden from node lists, search, detail, map, and stats.
|
|
// Use this to filter out trolls, nodes with offensive names, or nodes
|
|
// reporting deliberately false data (e.g. wrong GPS position) that the
|
|
// operator refuses to fix.
|
|
NodeBlacklist []string `json:"nodeBlacklist"`
|
|
|
|
// blacklistSetCached is the lazily-built set version of NodeBlacklist.
|
|
blacklistSetCached map[string]bool
|
|
blacklistOnce sync.Once
|
|
|
|
Branding map[string]interface{} `json:"branding"`
|
|
Theme map[string]interface{} `json:"theme"`
|
|
ThemeDark map[string]interface{} `json:"themeDark"`
|
|
NodeColors map[string]interface{} `json:"nodeColors"`
|
|
TypeColors map[string]interface{} `json:"typeColors"`
|
|
Home map[string]interface{} `json:"home"`
|
|
|
|
MapDefaults struct {
|
|
Center []float64 `json:"center"`
|
|
Zoom int `json:"zoom"`
|
|
} `json:"mapDefaults"`
|
|
|
|
Regions map[string]string `json:"regions"`
|
|
|
|
Roles map[string]interface{} `json:"roles"`
|
|
HealthThresholds *HealthThresholds `json:"healthThresholds"`
|
|
Tiles map[string]interface{} `json:"tiles"`
|
|
SnrThresholds map[string]interface{} `json:"snrThresholds"`
|
|
DistThresholds map[string]interface{} `json:"distThresholds"`
|
|
MaxHopDist *float64 `json:"maxHopDist"`
|
|
Limits map[string]interface{} `json:"limits"`
|
|
PerfSlowMs *int `json:"perfSlowMs"`
|
|
WsReconnectMs *int `json:"wsReconnectMs"`
|
|
CacheInvalidMs *int `json:"cacheInvalidateMs"`
|
|
ExternalUrls map[string]interface{} `json:"externalUrls"`
|
|
|
|
LiveMap struct {
|
|
PropagationBufferMs int `json:"propagationBufferMs"`
|
|
} `json:"liveMap"`
|
|
|
|
CacheTTL map[string]interface{} `json:"cacheTTL"`
|
|
|
|
Retention *RetentionConfig `json:"retention,omitempty"`
|
|
|
|
DB *DBConfig `json:"db,omitempty"`
|
|
|
|
PacketStore *PacketStoreConfig `json:"packetStore,omitempty"`
|
|
|
|
GeoFilter *GeoFilterConfig `json:"geo_filter,omitempty"`
|
|
|
|
Areas map[string]AreaEntry `json:"areas,omitempty"`
|
|
|
|
Timestamps *TimestampConfig `json:"timestamps,omitempty"`
|
|
|
|
// CORSAllowedOrigins is the list of origins permitted to make cross-origin
|
|
// requests. When empty (default), no Access-Control-* headers are sent,
|
|
// so browsers enforce same-origin policy. Set to ["*"] to allow all origins.
|
|
CORSAllowedOrigins []string `json:"corsAllowedOrigins,omitempty"`
|
|
|
|
DebugAffinity bool `json:"debugAffinity,omitempty"`
|
|
|
|
// MapDarkTileProvider selects the default dark-mode basemap provider for
|
|
// new visitors. The client may override per-browser via the customizer
|
|
// (persisted to localStorage). Allowed values: "carto-dark" (default),
|
|
// "esri-darkgray-labels", "voyager-inverted", "positron-inverted". See
|
|
// public/map-tile-providers.js for the registry. #1420.
|
|
MapDarkTileProvider string `json:"mapDarkTileProvider,omitempty"`
|
|
|
|
// ObserverBlacklist is a list of observer public keys to exclude from API
|
|
// responses (defense in depth — ingestor drops at ingest, server filters
|
|
// any that slipped through from a prior unblocked window).
|
|
ObserverBlacklist []string `json:"observerBlacklist,omitempty"`
|
|
|
|
// obsBlacklistSetCached is the lazily-built set version of ObserverBlacklist.
|
|
obsBlacklistSetCached map[string]bool
|
|
obsBlacklistOnce sync.Once
|
|
|
|
Compression *CompressionConfig `json:"compression,omitempty"`
|
|
ResolvedPath *ResolvedPathConfig `json:"resolvedPath,omitempty"`
|
|
NeighborGraph *NeighborGraphConfig `json:"neighborGraph,omitempty"`
|
|
|
|
// Observers cache settings (#1481 P0-3 / #1483).
|
|
ObserversCache *ObserversCacheConfig `json:"observersCache,omitempty"`
|
|
|
|
// Analytics steady-state background recompute (issue #1240).
|
|
Analytics *AnalyticsConfig `json:"analytics,omitempty"`
|
|
|
|
// BatteryThresholds: voltage cutoffs for low/critical alerts (#663).
|
|
BatteryThresholds *BatteryThresholdsConfig `json:"batteryThresholds,omitempty"`
|
|
}
|
|
|
|
// weakAPIKeys is the blocklist of known default/example API keys that must be rejected.
|
|
var weakAPIKeys = map[string]bool{
|
|
"your-secret-api-key-here": true,
|
|
"change-me": true,
|
|
"example": true,
|
|
"test": true,
|
|
"password": true,
|
|
"admin": true,
|
|
"apikey": true,
|
|
"api-key": true,
|
|
"secret": true,
|
|
"default": true,
|
|
}
|
|
|
|
// IsWeakAPIKey returns true if the key is in the blocklist or shorter than 16 characters.
|
|
func IsWeakAPIKey(key string) bool {
|
|
if key == "" {
|
|
return false // empty is handled separately (endpoints disabled)
|
|
}
|
|
if weakAPIKeys[strings.ToLower(key)] {
|
|
return true
|
|
}
|
|
if len(key) < 16 {
|
|
return true
|
|
}
|
|
return false
|
|
}
|
|
|
|
// CompressionConfig controls HTTP gzip and WebSocket permessage-deflate compression.
|
|
// Both are disabled by default — enable only when the upstream proxy does not already compress.
|
|
type CompressionConfig struct {
|
|
GZip bool `json:"gzip"`
|
|
Websocket bool `json:"websocket"`
|
|
|
|
// Level is the gzip compression level (1=BestSpeed … 9=BestCompression).
|
|
// 0 / out-of-range means "use compress/gzip.DefaultCompression".
|
|
Level int `json:"level,omitempty"`
|
|
|
|
// MinSizeBytes is an advisory minimum response size below which gzip
|
|
// would not pay off. Currently informational — kept here so operators
|
|
// can express intent and so future small-body fast-paths can use it.
|
|
MinSizeBytes int `json:"minSizeBytes,omitempty"`
|
|
|
|
// ContentTypes overrides the default compressible-MIME allow-list. When
|
|
// empty, a conservative default (application/json, text/html, text/css,
|
|
// application/javascript, text/plain, image/svg+xml, application/xml)
|
|
// is used. Already-compressed types (image/*, video/*, application/zip,
|
|
// application/x-gzip, …) are always skipped.
|
|
ContentTypes []string `json:"contentTypes,omitempty"`
|
|
}
|
|
|
|
// GZipEnabled returns true when HTTP gzip compression is explicitly enabled.
|
|
func (c *Config) GZipEnabled() bool {
|
|
return c.Compression != nil && c.Compression.GZip
|
|
}
|
|
|
|
// WSCompressionEnabled returns true when WebSocket permessage-deflate is explicitly enabled.
|
|
func (c *Config) WSCompressionEnabled() bool {
|
|
return c.Compression != nil && c.Compression.Websocket
|
|
}
|
|
|
|
// ResolvedPathConfig controls async backfill behavior.
|
|
type ResolvedPathConfig struct {
|
|
BackfillHours int `json:"backfillHours"` // how far back (hours) to scan for NULL resolved_path (default 24)
|
|
}
|
|
|
|
// NeighborGraphConfig controls neighbor edge pruning.
|
|
type NeighborGraphConfig struct {
|
|
MaxAgeDays int `json:"maxAgeDays"` // edges older than this are pruned (default 5)
|
|
MaxEdgeKm float64 `json:"maxEdgeKm"` // geo-implausibility threshold (km); 0 = default 500; negative disables (#1228)
|
|
|
|
// CacheRecomputeIntervalSeconds: cadence for the background
|
|
// recomputer that rebuilds the default-shape neighbor-graph
|
|
// response (#1481 P0-1). 0/missing = default 300 (5 min).
|
|
// Lower = fresher data, more CPU per minute. #1483.
|
|
CacheRecomputeIntervalSeconds int `json:"cacheRecomputeIntervalSeconds,omitempty"`
|
|
}
|
|
|
|
// ObserversCacheConfig controls the /api/observers default-shape cache.
|
|
// #1481 P0-3 / #1483.
|
|
type ObserversCacheConfig struct {
|
|
// TTLSeconds: how long the cached default-shape /api/observers
|
|
// response is served before a singleflight-collapsed refill.
|
|
// 0/missing = default 30. Lower = fresher data, more SQL pressure.
|
|
TTLSeconds int `json:"ttlSeconds,omitempty"`
|
|
}
|
|
|
|
// PacketStoreConfig controls in-memory packet store limits.
|
|
type PacketStoreConfig struct {
|
|
RetentionHours float64 `json:"retentionHours"` // max age of packets in hours (0 = unlimited)
|
|
MaxMemoryMB int `json:"maxMemoryMB"` // hard memory ceiling in MB (0 = unlimited)
|
|
MaxResolvedPubkeyIndexEntries int `json:"maxResolvedPubkeyIndexEntries"` // warning threshold for index size (0 = 5M default)
|
|
HotStartupHours float64 `json:"hotStartupHours"` // load only this many hours synchronously; 0 = disabled
|
|
}
|
|
|
|
// GeoFilterConfig is an alias for the shared geofilter.Config type.
|
|
type GeoFilterConfig = geofilter.Config
|
|
|
|
type RetentionConfig struct {
|
|
NodeDays int `json:"nodeDays"`
|
|
ObserverDays int `json:"observerDays"`
|
|
PacketDays int `json:"packetDays"`
|
|
MetricsDays int `json:"metricsDays"`
|
|
}
|
|
|
|
// DBConfig is the shared SQLite vacuum/maintenance config (#919, #921).
|
|
type DBConfig = dbconfig.DBConfig
|
|
|
|
// IncrementalVacuumPages returns the configured pages per vacuum or 1024 default.
|
|
func (c *Config) IncrementalVacuumPages() int {
|
|
if c.DB != nil && c.DB.IncrementalVacuumPages > 0 {
|
|
return c.DB.IncrementalVacuumPages
|
|
}
|
|
return 1024
|
|
}
|
|
|
|
// MetricsRetentionDays returns configured metrics retention or 30 days default.
|
|
func (c *Config) MetricsRetentionDays() int {
|
|
if c.Retention != nil && c.Retention.MetricsDays > 0 {
|
|
return c.Retention.MetricsDays
|
|
}
|
|
return 30
|
|
}
|
|
|
|
// BackfillHours returns configured backfill window or 24h default.
|
|
func (c *Config) BackfillHours() int {
|
|
if c.ResolvedPath != nil && c.ResolvedPath.BackfillHours > 0 {
|
|
return c.ResolvedPath.BackfillHours
|
|
}
|
|
return 24
|
|
}
|
|
|
|
// NeighborMaxAgeDays returns configured max edge age or 30 days default.
|
|
func (c *Config) NeighborMaxAgeDays() int {
|
|
if c.NeighborGraph != nil && c.NeighborGraph.MaxAgeDays > 0 {
|
|
return c.NeighborGraph.MaxAgeDays
|
|
}
|
|
return 5
|
|
}
|
|
|
|
// NeighborMaxEdgeKm returns the geo-implausibility threshold in km.
|
|
// 0 (unset) → DefaultMaxEdgeKm (500). Negative → 0 (filter disabled).
|
|
// See issue #1228.
|
|
func (c *Config) NeighborMaxEdgeKm() float64 {
|
|
if c == nil || c.NeighborGraph == nil || c.NeighborGraph.MaxEdgeKm == 0 {
|
|
return DefaultMaxEdgeKm
|
|
}
|
|
if c.NeighborGraph.MaxEdgeKm < 0 {
|
|
return 0
|
|
}
|
|
return c.NeighborGraph.MaxEdgeKm
|
|
}
|
|
|
|
type TimestampConfig struct {
|
|
DefaultMode string `json:"defaultMode"` // "ago" | "absolute"
|
|
Timezone string `json:"timezone"` // "local" | "utc"
|
|
FormatPreset string `json:"formatPreset"` // "iso" | "iso-seconds" | "locale"
|
|
CustomFormat string `json:"customFormat"` // freeform, only used when AllowCustomFormat=true
|
|
AllowCustomFormat bool `json:"allowCustomFormat"` // admin gate
|
|
}
|
|
|
|
func defaultTimestampConfig() TimestampConfig {
|
|
return TimestampConfig{
|
|
DefaultMode: "ago",
|
|
Timezone: "local",
|
|
FormatPreset: "iso",
|
|
CustomFormat: "",
|
|
AllowCustomFormat: false,
|
|
}
|
|
}
|
|
|
|
// NodeDaysOrDefault returns the configured retention.nodeDays or 7 if not set.
|
|
func (c *Config) NodeDaysOrDefault() int {
|
|
if c.Retention != nil && c.Retention.NodeDays > 0 {
|
|
return c.Retention.NodeDays
|
|
}
|
|
return 7
|
|
}
|
|
|
|
// ObserverDaysOrDefault returns the configured retention.observerDays or 14 if not set.
|
|
// A value of -1 means observers are never removed.
|
|
func (c *Config) ObserverDaysOrDefault() int {
|
|
if c.Retention != nil && c.Retention.ObserverDays != 0 {
|
|
return c.Retention.ObserverDays
|
|
}
|
|
return 14
|
|
}
|
|
|
|
type HealthThresholds struct {
|
|
InfraDegradedHours float64 `json:"infraDegradedHours"`
|
|
InfraSilentHours float64 `json:"infraSilentHours"`
|
|
NodeDegradedHours float64 `json:"nodeDegradedHours"`
|
|
NodeSilentHours float64 `json:"nodeSilentHours"`
|
|
// RelayActiveHours: how recent a path-hop appearance must be for a
|
|
// repeater to be considered "actively relaying" vs only "alive
|
|
// (advert-only)". See issue #662. Defaults to 24h.
|
|
RelayActiveHours float64 `json:"relayActiveHours"`
|
|
}
|
|
|
|
// ThemeFile mirrors theme.json overlay.
|
|
type ThemeFile struct {
|
|
Branding map[string]interface{} `json:"branding"`
|
|
Theme map[string]interface{} `json:"theme"`
|
|
ThemeDark map[string]interface{} `json:"themeDark"`
|
|
NodeColors map[string]interface{} `json:"nodeColors"`
|
|
TypeColors map[string]interface{} `json:"typeColors"`
|
|
Home map[string]interface{} `json:"home"`
|
|
}
|
|
|
|
func LoadConfig(baseDirs ...string) (*Config, error) {
|
|
if len(baseDirs) == 0 {
|
|
baseDirs = []string{"."}
|
|
}
|
|
paths := make([]string, 0)
|
|
for _, d := range baseDirs {
|
|
paths = append(paths, filepath.Join(d, "config.json"))
|
|
paths = append(paths, filepath.Join(d, "data", "config.json"))
|
|
}
|
|
|
|
cfg := &Config{Port: 3000}
|
|
for _, p := range paths {
|
|
data, err := os.ReadFile(p)
|
|
if err != nil {
|
|
continue
|
|
}
|
|
if err := json.Unmarshal(data, cfg); err != nil {
|
|
continue
|
|
}
|
|
cfg.NormalizeTimestampConfig()
|
|
return cfg, nil
|
|
}
|
|
cfg.NormalizeTimestampConfig()
|
|
return cfg, nil // defaults
|
|
}
|
|
|
|
func LoadTheme(baseDirs ...string) *ThemeFile {
|
|
if len(baseDirs) == 0 {
|
|
baseDirs = []string{"."}
|
|
}
|
|
for _, d := range baseDirs {
|
|
for _, name := range []string{"theme.json"} {
|
|
p := filepath.Join(d, name)
|
|
data, err := os.ReadFile(p)
|
|
if err != nil {
|
|
p = filepath.Join(d, "data", name)
|
|
data, err = os.ReadFile(p)
|
|
if err != nil {
|
|
continue
|
|
}
|
|
}
|
|
var t ThemeFile
|
|
if json.Unmarshal(data, &t) == nil {
|
|
return &t
|
|
}
|
|
}
|
|
}
|
|
return &ThemeFile{}
|
|
}
|
|
|
|
func (c *Config) GetHealthThresholds() HealthThresholds {
|
|
h := HealthThresholds{
|
|
InfraDegradedHours: 24,
|
|
InfraSilentHours: 72,
|
|
NodeDegradedHours: 1,
|
|
NodeSilentHours: 24,
|
|
RelayActiveHours: 24,
|
|
}
|
|
if c.HealthThresholds != nil {
|
|
if c.HealthThresholds.InfraDegradedHours > 0 {
|
|
h.InfraDegradedHours = c.HealthThresholds.InfraDegradedHours
|
|
}
|
|
if c.HealthThresholds.InfraSilentHours > 0 {
|
|
h.InfraSilentHours = c.HealthThresholds.InfraSilentHours
|
|
}
|
|
if c.HealthThresholds.NodeDegradedHours > 0 {
|
|
h.NodeDegradedHours = c.HealthThresholds.NodeDegradedHours
|
|
}
|
|
if c.HealthThresholds.NodeSilentHours > 0 {
|
|
h.NodeSilentHours = c.HealthThresholds.NodeSilentHours
|
|
}
|
|
if c.HealthThresholds.RelayActiveHours > 0 {
|
|
h.RelayActiveHours = c.HealthThresholds.RelayActiveHours
|
|
}
|
|
}
|
|
return h
|
|
}
|
|
|
|
// GetHealthMs returns degraded/silent thresholds in ms for a given role.
|
|
func (h HealthThresholds) GetHealthMs(role string) (degradedMs, silentMs int) {
|
|
const hourMs = 3600000
|
|
if role == "repeater" || role == "room" {
|
|
return int(h.InfraDegradedHours * hourMs), int(h.InfraSilentHours * hourMs)
|
|
}
|
|
return int(h.NodeDegradedHours * hourMs), int(h.NodeSilentHours * hourMs)
|
|
}
|
|
|
|
// ToClientMs returns the thresholds as ms for the frontend.
|
|
func (h HealthThresholds) ToClientMs() map[string]int {
|
|
const hourMs = 3600000
|
|
return map[string]int{
|
|
"infraDegradedMs": int(h.InfraDegradedHours * hourMs),
|
|
"infraSilentMs": int(h.InfraSilentHours * hourMs),
|
|
"nodeDegradedMs": int(h.NodeDegradedHours * hourMs),
|
|
"nodeSilentMs": int(h.NodeSilentHours * hourMs),
|
|
}
|
|
}
|
|
|
|
func (c *Config) ResolveDBPath(baseDir string) string {
|
|
if c.DBPath != "" {
|
|
return c.DBPath
|
|
}
|
|
if v := os.Getenv("DB_PATH"); v != "" {
|
|
return v
|
|
}
|
|
return filepath.Join(baseDir, "data", "meshcore.db")
|
|
}
|
|
|
|
|
|
func (c *Config) NormalizeTimestampConfig() {
|
|
defaults := defaultTimestampConfig()
|
|
if c.Timestamps == nil {
|
|
log.Printf("[config] timestamps not configured - using defaults (ago/local/iso)")
|
|
c.Timestamps = &defaults
|
|
return
|
|
}
|
|
|
|
origMode := c.Timestamps.DefaultMode
|
|
mode := strings.ToLower(strings.TrimSpace(origMode))
|
|
switch mode {
|
|
case "ago", "absolute":
|
|
c.Timestamps.DefaultMode = mode
|
|
default:
|
|
log.Printf("[config] warning: timestamps.defaultMode=%q is invalid, using %q", origMode, defaults.DefaultMode)
|
|
c.Timestamps.DefaultMode = defaults.DefaultMode
|
|
}
|
|
|
|
origTimezone := c.Timestamps.Timezone
|
|
timezone := strings.ToLower(strings.TrimSpace(origTimezone))
|
|
switch timezone {
|
|
case "local", "utc":
|
|
c.Timestamps.Timezone = timezone
|
|
default:
|
|
log.Printf("[config] warning: timestamps.timezone=%q is invalid, using %q", origTimezone, defaults.Timezone)
|
|
c.Timestamps.Timezone = defaults.Timezone
|
|
}
|
|
|
|
origPreset := c.Timestamps.FormatPreset
|
|
formatPreset := strings.ToLower(strings.TrimSpace(origPreset))
|
|
switch formatPreset {
|
|
case "iso", "iso-seconds", "locale":
|
|
c.Timestamps.FormatPreset = formatPreset
|
|
default:
|
|
log.Printf("[config] warning: timestamps.formatPreset=%q is invalid, using %q", origPreset, defaults.FormatPreset)
|
|
c.Timestamps.FormatPreset = defaults.FormatPreset
|
|
}
|
|
}
|
|
|
|
func (c *Config) GetTimestampConfig() TimestampConfig {
|
|
if c == nil || c.Timestamps == nil {
|
|
return defaultTimestampConfig()
|
|
}
|
|
return *c.Timestamps
|
|
}
|
|
func (c *Config) PropagationBufferMs() int {
|
|
if c.LiveMap.PropagationBufferMs > 0 {
|
|
return c.LiveMap.PropagationBufferMs
|
|
}
|
|
return 5000
|
|
}
|
|
|
|
// blacklistSet lazily builds and caches the nodeBlacklist as a set for O(1) lookups.
|
|
// Uses sync.Once to eliminate the data race on first concurrent access.
|
|
func (c *Config) blacklistSet() map[string]bool {
|
|
c.blacklistOnce.Do(func() {
|
|
if len(c.NodeBlacklist) == 0 {
|
|
return
|
|
}
|
|
m := make(map[string]bool, len(c.NodeBlacklist))
|
|
for _, pk := range c.NodeBlacklist {
|
|
trimmed := strings.ToLower(strings.TrimSpace(pk))
|
|
if trimmed != "" {
|
|
m[trimmed] = true
|
|
}
|
|
}
|
|
c.blacklistSetCached = m
|
|
})
|
|
return c.blacklistSetCached
|
|
}
|
|
|
|
// IsBlacklisted returns true if the given public key is in the nodeBlacklist.
|
|
func (c *Config) IsBlacklisted(pubkey string) bool {
|
|
if c == nil || len(c.NodeBlacklist) == 0 {
|
|
return false
|
|
}
|
|
return c.blacklistSet()[strings.ToLower(strings.TrimSpace(pubkey))]
|
|
}
|
|
|
|
// SaveGeoFilter writes the geo_filter section back to config.json on disk.
|
|
// Pass gf=nil to remove the filter. The rest of config.json is preserved as-is.
|
|
func SaveGeoFilter(configDir string, gf *GeoFilterConfig) error {
|
|
var configPath string
|
|
for _, p := range []string{
|
|
filepath.Join(configDir, "config.json"),
|
|
filepath.Join(configDir, "data", "config.json"),
|
|
} {
|
|
if _, err := os.Stat(p); err == nil {
|
|
configPath = p
|
|
break
|
|
}
|
|
}
|
|
if configPath == "" {
|
|
return fmt.Errorf("config.json not found in %s", configDir)
|
|
}
|
|
|
|
data, err := os.ReadFile(configPath)
|
|
if err != nil {
|
|
return fmt.Errorf("read config: %w", err)
|
|
}
|
|
|
|
// Parse as a raw map so non-struct fields (_comment, etc.) are preserved.
|
|
var raw map[string]interface{}
|
|
if err := json.Unmarshal(data, &raw); err != nil {
|
|
return fmt.Errorf("parse config: %w", err)
|
|
}
|
|
|
|
if gf == nil || len(gf.Polygon) == 0 {
|
|
delete(raw, "geo_filter")
|
|
} else {
|
|
// Round-trip through JSON to get a plain interface{} value.
|
|
b, _ := json.Marshal(gf)
|
|
var v interface{}
|
|
_ = json.Unmarshal(b, &v)
|
|
raw["geo_filter"] = v
|
|
}
|
|
|
|
out, err := json.MarshalIndent(raw, "", " ")
|
|
if err != nil {
|
|
return fmt.Errorf("marshal config: %w", err)
|
|
}
|
|
out = append(out, '\n')
|
|
|
|
// Atomic write: temp file + rename.
|
|
tmp := configPath + ".tmp"
|
|
if err := os.WriteFile(tmp, out, 0644); err != nil {
|
|
return fmt.Errorf("write config: %w", err)
|
|
}
|
|
if err := os.Rename(tmp, configPath); err != nil {
|
|
os.Remove(tmp)
|
|
return fmt.Errorf("rename config: %w", err)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// obsBlacklistSet lazily builds and caches the observerBlacklist as a set for O(1) lookups.
|
|
func (c *Config) obsBlacklistSet() map[string]bool {
|
|
c.obsBlacklistOnce.Do(func() {
|
|
if len(c.ObserverBlacklist) == 0 {
|
|
return
|
|
}
|
|
m := make(map[string]bool, len(c.ObserverBlacklist))
|
|
for _, pk := range c.ObserverBlacklist {
|
|
trimmed := strings.ToLower(strings.TrimSpace(pk))
|
|
if trimmed != "" {
|
|
m[trimmed] = true
|
|
}
|
|
}
|
|
c.obsBlacklistSetCached = m
|
|
})
|
|
return c.obsBlacklistSetCached
|
|
}
|
|
|
|
// IsObserverBlacklisted returns true if the given observer ID is in the observerBlacklist.
|
|
func (c *Config) IsObserverBlacklisted(id string) bool {
|
|
if c == nil || len(c.ObserverBlacklist) == 0 {
|
|
return false
|
|
}
|
|
return c.obsBlacklistSet()[strings.ToLower(strings.TrimSpace(id))]
|
|
}
|
|
|
|
// AnalyticsConfig controls steady-state background recompute of
|
|
// analytics endpoints (issue #1240).
|
|
//
|
|
// DefaultIntervalSeconds applies to every endpoint that does not have
|
|
// an explicit per-endpoint override in RecomputeIntervalSeconds. The
|
|
// project default is 300 (5 minutes): the operator's guiding principle
|
|
// is "serving slightly stale data quickly is better than real-time
|
|
// data slowly." Lower values give fresher data at higher CPU cost.
|
|
//
|
|
// RecomputeIntervalSeconds keys (all optional):
|
|
// topology, rf, distance, channels, hashCollisions, hashSizes, roles, observersClockSkew, nodesClockSkew
|
|
type AnalyticsConfig struct {
|
|
DefaultIntervalSeconds int `json:"defaultIntervalSeconds,omitempty"`
|
|
RecomputeIntervalSeconds map[string]int `json:"recomputeIntervalSeconds,omitempty"`
|
|
}
|
|
|
|
// AnalyticsDefaultRecomputeInterval returns the configured default
|
|
// recompute interval, or 5 minutes if unset/invalid.
|
|
func (c *Config) AnalyticsDefaultRecomputeInterval() time.Duration {
|
|
if c != nil && c.Analytics != nil && c.Analytics.DefaultIntervalSeconds > 0 {
|
|
return time.Duration(c.Analytics.DefaultIntervalSeconds) * time.Second
|
|
}
|
|
return 5 * time.Minute
|
|
}
|
|
|
|
// AnalyticsRecomputeIntervals returns the per-endpoint override map.
|
|
// Returns the zero value (all defaults) if the analytics block is
|
|
// absent or empty.
|
|
func (c *Config) AnalyticsRecomputeIntervals() AnalyticsRecomputeIntervals {
|
|
out := AnalyticsRecomputeIntervals{}
|
|
if c == nil || c.Analytics == nil || c.Analytics.RecomputeIntervalSeconds == nil {
|
|
return out
|
|
}
|
|
get := func(key string) time.Duration {
|
|
v, ok := c.Analytics.RecomputeIntervalSeconds[key]
|
|
if !ok || v <= 0 {
|
|
return 0
|
|
}
|
|
return time.Duration(v) * time.Second
|
|
}
|
|
out.Topology = get("topology")
|
|
out.RF = get("rf")
|
|
out.Distance = get("distance")
|
|
out.Channels = get("channels")
|
|
out.HashCollisions = get("hashCollisions")
|
|
out.HashSizes = get("hashSizes")
|
|
out.Roles = get("roles")
|
|
out.ObserversClockSkew = get("observersClockSkew")
|
|
out.NodesClockSkew = get("nodesClockSkew")
|
|
return out
|
|
}
|