The flat 'deploy' concurrency group caused ALL PRs to share one queue,
so pushing to any PR would cancel CI runs on other PRs.
Changed to deploy-${{ github.event.pull_request.number || github.ref }}
so each PR gets its own concurrency group while re-pushes to the same
PR still cancel the previous run.
SQLite :memory: databases create separate databases per connection.
When the connection pool opens multiple connections (e.g. poller goroutine
vs main test goroutine), tables created on one connection are invisible
to others. Setting MaxOpenConns(1) ensures all queries use the same
in-memory database, fixing TestPollerBroadcastsMultipleObservations.
IngestNewFromDB now broadcasts one message per observation (not per
transmission). IngestNewObservations also broadcasts late arrivals.
Tests verify multi-observer packets produce multiple WS messages.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- /api/stats: 10s server-side cache — was running 5 SQLite COUNT queries
on every call, taking ~1500ms with 28 concurrent WS clients polling every 15s
- GetNodeHashSizeInfo: 15s cache — was doing a full O(n) scan + JSON unmarshal
of all advert packets in memory on every /nodes request, taking ~1200ms
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Linux Docker doesn't resolve host.docker.internal by default.
Required when MQTT sources in config.json point to the host machine.
Harmless on Docker Desktop where it already works.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Without build: directive, docker compose tries to pull corescope:latest
from Docker Hub instead of building locally.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add preflight check for 'docker compose' in manage.sh (catches plugin missing)
- Document named Caddy volumes as cert storage, not user data
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove all legacy docker run code paths. manage.sh is now a pure
docker compose wrapper with no dual-mode branching.
Removed:
- COMPOSE_MODE flag and all if/else branches
- get_docker_run_args(), get_data_mount_args(), recreate_container()
- get_required_ports(), get_current_ports(), check_port_match()
- CONTAINER_NAME, DATA_VOLUME, CADDY_VOLUME variables
- All direct docker run/stop/start/rm invocations
All commands now delegate to docker compose:
- start → docker compose up -d prod
- stop → docker compose down / docker compose stop
- restart → docker compose up -d --force-recreate
- update → docker compose build prod + up -d --force-recreate
- reset → docker compose down --rmi local
- backup/restore use bind mount path from .env (PROD_DATA_DIR)
- verify_health, mqtt-test, status all use corescope-prod
Net result: -248 lines, zero dual-mode logic, identical behavior
to running docker compose directly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Replace all hardcoded \C:\Users\KpaBap/meshcore-data with \ variable
- \ resolves from \ in .env or defaults to ~/meshcore-data
- Updated get_data_mount_args(), cmd_backup(), cmd_restore(), cmd_reset()
- Enhanced .env.example with detailed comments for each variable
- Both docker compose and manage.sh now read same .env file
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
PROBLEM:
manage.sh was using named Docker volumes (meshcore-data) as the default,
which hides the database and theme files inside Docker's internal storage.
Users couldn't find their DB on the filesystem for backups or inspection.
The function get_data_mount_args() had conditional logic that only used
bind mounts IF it detected an existing ~/meshcore-data with a DB file.
For new installs, it fell through to the named volume — silently hiding
all data in /var/lib/docker/volumes/.
FIXES:
1. get_data_mount_args() — Always use bind mount to ~/meshcore-data
- Creates the directory if it doesn't exist
- Removes all conditional logic and the named volume fallback
2. cmd_backup() — Use direct path C:\Users\KpaBap/meshcore-data/meshcore.db
- No longer tries to inspect the named volume
- Consistent with the bind mount approach
3. cmd_restore() — Use direct path for restore operations
- Ensures directory exists before restoring files
- No fallback to docker cp
4. cmd_reset() — Updated message to reflect bind mount location
- Changed from 'docker volume rm' to '~/meshcore-data (not removed)'
5. docker-compose.yml — Added documentation comment
- Clarifies that bind mounts are intentional, not named volumes
- Ensures future changes maintain this pattern
VALIDATION:
- docker-compose.yml already used bind mounts correctly (\)
- Legacy 'docker run' mode now matches compose behavior
- All backup/restore operations reference the same bind mount path
DATABASE LOCATION:
- Always: ~/meshcore-data/meshcore.db
- Never: Hidden in Docker's volume storage
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Requested-by: Kpa-clawbot
- Load() SQL: keep o.timestamp DESC (consistent with IngestNewFromDB) so
pickBestObservation tie-breaking is identical on both load paths
- GetTimestamps: scan from tail instead of head (was breaking on first item
assuming it was the newest, now correctly reads from newest end)
- QueryMultiNodePackets: apply same DESC/ASC tail-read pagination as
QueryPackets (was sorting for ASC and assuming DESC as-is)
- GetNodeHealth recentPackets: read from tail to return 20 newest items
(was reading from head = 20 oldest items)
- Remove stale "Prepend (newest first)" comments, replace with accurate
"oldest-first; new items go to tail" wording
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
s.packets and s.byPayloadType[t] were prepended on every new packet
to maintain newest-first order, copying the entire slice each time.
With 2-3M packets in memory this meant ~24MB of pointer copies per
ingest cycle, causing sustained high CPU and GC pressure.
Fix: store both slices oldest-first (append to tail). Load() SQL
changed to ASC ordering. QueryPackets DESC pagination now reads from
the tail in O(page_size) with no sort; GetChannelMessages switches
from reverse-iteration to forward-iteration.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* refactor: remove all packets_v SQL fallbacks — store handles all queries
Remove DB fallback paths from all route handlers. The in-memory
PacketStore now handles all packet/node/analytics queries. Handlers
return empty results or 404 when no store is available instead of
falling back to direct DB queries.
- Remove else-DB branches from handlePacketDetail, handleNodeHealth,
handleNodeAnalytics, handleBulkHealth, handlePacketTimestamps, etc.
- Remove unused DB methods (GetPacketByHash, GetTransmissionByID,
GetPacketByID, GetObservationsForHash, GetTimestamps, GetNodeHealth,
GetNodeAnalytics, GetBulkHealth, etc.)
- Remove packets_v VIEW creation from schema
- Update tests for new behavior (no-store returns 404/empty, not 500)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: address PR #220 review comments
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: KpaBap <kpabap@gmail.com>
- Add pull_request trigger for PRs against master
- Add 'if: github.event_name == push' to build/deploy/publish jobs
- Test jobs (go-test, node-test) now run on both push and PRs
- Build/deploy/publish only run on push to master
This fixes the chicken-and-egg problem where branch protection requires
CI checks but CI doesn't run on PRs. Now PRs get test validation before
merge while keeping production deployments only on master pushes.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The glob trick COPY .git-commi[t] only works with BuildKit.
manage.sh uses legacy docker build. Just create a default via RUN.
Commit hash comes through --build-arg ldflags anyway.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Documents what existing users need to update when the rename
from MeshCore Analyzer to CoreScope lands:
- Git remote URL update
- Docker image/container name changes
- Config branding.siteName (if customized)
- CI/CD references (if applicable)
- Confirms data dirs, MQTT, browser state unchanged
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The glob trick COPY .git-commi[t] only works with BuildKit.
manage.sh uses legacy docker build. Just create a default via RUN.
Commit hash comes through --build-arg ldflags anyway.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Server defaults to 6060, ingestor to 6061. Removed shared PPROF_PORT
env var. Bind failure logs warning instead of log.Fatal killing the process.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sensor nodes embed telemetry (battery_mv, temperature_c) in their advert
appdata after the null-terminated name. This commit adds decoding and
storage for both the Go ingestor and Node.js backend.
Changes:
- decoder.go/decoder.js: Parse telemetry bytes from advert appdata
(battery_mv as uint16 LE millivolts, temperature_c as int16 LE /100)
- db.go/db.js: Add battery_mv INTEGER and temperature_c REAL columns
to nodes and inactive_nodes tables, with migration for existing DBs
- main.go/server.js: Update node telemetry on advert processing
- server db.go: Include battery_mv/temperature_c in node API responses
- Tests: Decoder telemetry tests (positive, negative temp, no telemetry),
DB migration test, node telemetry update test, server API shape tests
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fresh Go installs failed with 'no such table: packets_v' because the
ingestor created tables but never the VIEW that the Go server queries.
Add DROP VIEW IF EXISTS + CREATE VIEW packets_v to applySchema(), using
the v3 definition (observer_idx → observers.rowid JOIN). The view is
rebuilt on every startup to stay current with any definition changes.
Add tests: verify view exists after OpenStore, and verify it returns
correct observer_id/observer_name via the LEFT JOIN.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add net/http/pprof support to both Go server (default port 6060) and
ingestor (default port 6061). Profiling is off by default — only
starts the pprof HTTP listener when ENABLE_PPROF=true.
PPROF_PORT env var overrides the default port for each binary.
Enable on staging-go in docker-compose with exposed ports 6060/6061.
Not enabled on prod.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
SQLite stores these as REAL on some instances. Go *int scan silently
fails, dropping the entire observer row (404 on detail, missing from list).
Reported for YC-Base-Repeater and YC-Work-Repeater.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Kpa-clawbot