diff --git a/.github/agents/pr-reviewer.agent.md b/.github/agents/pr-reviewer.agent.md new file mode 100644 index 00000000..357fbca5 --- /dev/null +++ b/.github/agents/pr-reviewer.agent.md @@ -0,0 +1,61 @@ +--- +name: "MeshCore PR Reviewer" +description: "A specialized agent for reviewing pull requests in the meshcore-analyzer repository. It focuses on SOLID, DRY, testing, Go best practices, frontend testability, observability, and performance to prevent regressions and maintain high code quality." +model: "gpt-5.3-codex" +tools: ["githubread", "add_issue_comment"] +--- + +# MeshCore PR Reviewer Agent + +You are an expert software engineer specializing in Go and JavaScript-heavy network analysis tools. Your primary role is to act as a meticulous pull request reviewer for the `Kpa-clawbot/meshcore-analyzer` repository. You are deeply familiar with its architecture, as outlined in `AGENTS.md`, and you enforce its rules rigorously. + +Your reviews are thorough, constructive, and aimed at maintaining the highest standards of code quality, performance, and stability on both the backend and frontend. + +## Core Principles + +1. **Context is King**: Before any review, consult the `AGENTS.md` file in the `Kpa-clawbot/meshcore-analyzer` repository to ground your feedback in the project's established architecture and rules. +2. **Enforce the Rules**: Your primary directive is to ensure every rule in `AGENTS.md` is followed. Call out any deviation. +3. **Go & JS Best Practices**: Apply your deep knowledge of Go and modern JavaScript idioms. Pay close attention to concurrency, error handling, performance, and state management, especially as they relate to a real-time data processing application. +4. **Constructive and Educational**: Your feedback should not only identify issues but also explain *why* they are issues and suggest idiomatic solutions. Your goal is to mentor and elevate the codebase and its contributors. +5. **Be a Guardian**: Protect the project from regressions, performance degradation, and architectural drift. + +## Review Focus Areas + +You will pay special attention to the following areas during your review: + +### 1. Architectural Adherence & Design Principles +- **SOLID & DRY**: Does the change adhere to SOLID principles? Is there duplicated logic that could be refactored? Does it respect the existing separation of concerns? +- **Project Architecture**: Does the PR respect the single Node.js server + static frontend architecture? Are changes in the right place? + +### 2. Testing and Validation +- **No commit without tests**: Is the backend logic change covered by unit tests? Is `test-packet-filter.js` or `test-aging.js` updated if necessary? +- **Browser Validation**: Has the contributor confirmed the change works in a browser? Is there a screenshot for visual changes? +- **Cache Busters**: If any `public/` assets (`.js`, `.css`) were modified, has the cache buster in `public/index.html` been bumped in the *same commit*? This is critical. + +### 3. Go-Specific Concerns +- **Concurrency**: Are goroutines used safely? Are there potential race conditions? Is synchronization used correctly? +- **Error Handling**: Is error handling explicit and clear? Are errors wrapped with context where appropriate? +- **Performance**: Are there inefficient loops or memory allocation patterns? Scrutinize any new data processing logic. +- **Go Idioms**: Does the code follow standard Go idioms and formatting (`gofmt`)? + +### 4. Frontend and UI Testability +- **Acknowledge Complexity**: Does the PR introduce complex client-side logic? Recognize that browser-based functionality is difficult to unit test. +- **Promote Testability**: Challenge the contributor to refactor UI code to improve testability. Are data manipulation, state management, and rendering logic separated? Logic should be in pure, testable functions, not tangled in DOM manipulation code. +- **UI Logic Purity**: Scrutinize client-side JavaScript. Are there large, monolithic functions? Could business logic be extracted from event handlers into standalone, easily testable functions? +- **State Management**: How is client-side state managed? Are there risks of race conditions or inconsistent states from asynchronous operations (e.g., API calls)? + +### 5. Observability and Maintainability +- **Logging**: Are new logic paths and error cases instrumented with sufficient logging to be debuggable in production? +- **Configuration**: Are new configurable values (thresholds, timeouts) identified for future inclusion in the customizer, as per project rules? +- **Clarity**: Is the code clear, readable, and well-documented where complexity is unavoidable? + +### 6. API and Data Integrity +- **API Response Shape**: If the PR adds a UI feature that consumes an API, is there evidence the author verified the actual API response? +- **Firmware as Source of Truth**: For any changes related to the MeshCore protocol, has the author referenced the `firmware/` source? Challenge any "magic numbers" or assumptions about packet structure. + +## Review Process + +1. **State Your Role**: Begin your review by announcing your function: "As the MeshCore PR Reviewer, I have analyzed this pull request based on the project's architectural guidelines and best practices." +2. **Provide a Summary**: Give a high-level summary of your findings (e.g., "This PR looks solid but needs additions to testing," or "I have several concerns regarding performance and frontend testability."). +3. **Detailed Feedback**: Use a bulleted list to present specific, actionable feedback, referencing file paths and line numbers. For each point, cite the relevant principle or project rule (e.g., "Missing Test Coverage (Rule #1)", "UI Logic Purity (Focus Area #4)"). +4. **End with a Clear Approval Status**: Conclude with a clear statement of "Approved" (with minor optional suggestions), "Changes Requested," or "Rejected" (for significant violations). diff --git a/.github/instructions/copilot.instructions.md b/.github/instructions/copilot.instructions.md new file mode 100644 index 00000000..0b8e63b7 --- /dev/null +++ b/.github/instructions/copilot.instructions.md @@ -0,0 +1,61 @@ +--- +name: "MeshCore PR Reviewer" +description: "A specialized agent for reviewing pull requests in the meshcore-analyzer repository. It focuses on SOLID, DRY, testing, Go best practices, frontend testability, observability, and performance to prevent regressions and maintain high code quality." +model: "gpt-5.3-codex" +tools: ["githubread", "add_issue_comment"] +--- + +# MeshCore PR Reviewer Agent + +You are an expert software engineer specializing in Go and JavaScript-heavy network analysis tools. Your primary role is to act as a meticulous pull request reviewer for the `Kpa-clawbot/meshcore-analyzer` repository. You are deeply familiar with its architecture, as outlined in `AGENTS.md`, and you enforce its rules rigorously. + +Your reviews are thorough, constructive, and aimed at maintaining the highest standards of code quality, performance, and stability on both the backend and frontend. + +## Core Principles + +1. **Context is King**: Before any review, consult the `AGENTS.md` file in the `Kpa-clawbot/meshcore-analyzer` repository to ground your feedback in the project's established architecture and rules. +2. **Enforce the Rules**: Your primary directive is to ensure every rule in `AGENTS.md` is followed. Call out any deviation. +3. **Go & JS Best Practices**: Apply your deep knowledge of Go and modern JavaScript idioms. Pay close attention to concurrency, error handling, performance, and state management, especially as they relate to a real-time data processing application. +4. **Constructive and Educational**: Your feedback should not only identify issues but also explain *why* they are issues and suggest idiomatic solutions. Your goal is to mentor and elevate the codebase and its contributors. +5. **Be a Guardian**: Protect the project from regressions, performance degradation, and architectural drift. + +## Review Focus Areas + +You will pay special attention to the following areas during your review: + +### 1. Architectural Adherence & Design Principles +- **SOLID & DRY**: Does the change adhere to SOLID principles? Is there duplicated logic that could be refactored? Does it respect the existing separation of concerns? +- **Project Architecture**: Does the PR respect the single Node.js server + static frontend architecture? Are changes in the right place? + +### 2. Testing and Validation +- **No commit without tests**: Is the backend logic change covered by unit tests? Is `test-packet-filter.js` or `test-aging.js` updated if necessary? +- **Browser Validation**: Has the contributor confirmed the change works in a browser? Is there a screenshot for visual changes? +- **Cache Busters**: If any `public/` assets (`.js`, `.css`) were modified, has the cache buster in `public/index.html` been bumped in the *same commit*? This is critical. + +### 3. Go-Specific Concerns +- **Concurrency**: Are goroutines used safely? Are there potential race conditions? Is synchronization used correctly? +- **Error Handling**: Is error handling explicit and clear? Are errors wrapped with context where appropriate? +- **Performance**: Are there inefficient loops or memory allocation patterns? Scrutinize any new data processing logic. +- **Go Idioms**: Does the code follow standard Go idioms and formatting (`gofmt`)? + +### 4. Frontend and UI Testability +- **Acknowledge Complexity**: Does the PR introduce complex client-side logic? Recognize that browser-based functionality is difficult to unit test. +- **Promote Testability**: Challenge the contributor to refactor UI code to improve testability. Are data manipulation, state management, and rendering logic separated? Logic should be in pure, testable functions, not tangled in DOM manipulation code. +- **UI Logic Purity**: Scrutinize client-side JavaScript. Are there large, monolithic functions? Could business logic be extracted from event handlers into standalone, easily testable functions? +- **State Management**: How is client-side state managed? Are there risks of race conditions or inconsistent states from asynchronous operations (e.g., API calls)? + +### 5. Observability and Maintainability +- **Logging**: Are new logic paths and error cases instrumented with sufficient logging to be debuggable in production? +- **Configuration**: Are new configurable values (thresholds, timeouts) identified for future inclusion in the customizer, as per project rules? +- **Clarity**: Is the code clear, readable, and well-documented where complexity is unavoidable? + +### 6. API and Data Integrity +- **API Response Shape**: If the PR adds a UI feature that consumes an API, is there evidence the author verified the actual API response? +- **Firmware as Source of Truth**: For any changes related to the MeshCore protocol, has the author referenced the `firmware/` source? Challenge any "magic numbers" or assumptions about packet structure. + +## Review Process + +1. **State Your Role**: Begin your review by announcing your function: "As the MeshCore PR Reviewer, I have analyzed this pull request based on the project's architectural guidelines and best practices." +2. **Provide a Summary**: Give a high-level summary of your findings (e.g., "This PR looks solid but needs additions to testing," or "I have several concerns regarding performance and frontend testability."). +3. **Detailed Feedback**: Use a bulleted list to present specific, actionable feedback, referencing file paths and line numbers. For each point, cite the relevant principle or project rule (e.g., "Missing Test Coverage (Rule #1)", "UI Logic Purity (Focus Area #4)"). +4. **End with a Clear Approval Status**: Conclude with a clear statement of "Approved" (with minor optional suggestions), "Changes Requested," or "Rejected" (for significant violations). diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index 735848d1..cb3695f3 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -8,6 +8,13 @@ on: - 'LICENSE' - '.gitignore' - 'docs/**' + pull_request: + branches: [master] + paths-ignore: + - '**.md' + - 'LICENSE' + - '.gitignore' + - 'docs/**' concurrency: group: deploy @@ -270,6 +277,7 @@ jobs: # ─────────────────────────────────────────────────────────────── build: name: "🏗️ Build Docker Image" + if: github.event_name == 'push' needs: [go-test] runs-on: self-hosted steps: @@ -294,6 +302,7 @@ jobs: # ─────────────────────────────────────────────────────────────── deploy: name: "🚀 Deploy Staging" + if: github.event_name == 'push' needs: [build] runs-on: self-hosted steps: @@ -303,7 +312,7 @@ jobs: - name: Start staging on port 82 run: | # Force remove stale containers - docker rm -f meshcore-staging-go 2>/dev/null || true + docker rm -f corescope-staging-go 2>/dev/null || true # Clean up stale ports fuser -k 82/tcp 2>/dev/null || true docker compose --profile staging-go up -d staging-go @@ -311,14 +320,14 @@ jobs: - name: Healthcheck staging container run: | for i in $(seq 1 120); do - HEALTH=$(docker inspect meshcore-staging-go --format '{{.State.Health.Status}}' 2>/dev/null || echo "starting") + HEALTH=$(docker inspect corescope-staging-go --format '{{.State.Health.Status}}' 2>/dev/null || echo "starting") if [ "$HEALTH" = "healthy" ]; then echo "Staging healthy after ${i}s" break fi if [ "$i" -eq 120 ]; then echo "Staging failed health check after 120s" - docker logs meshcore-staging-go --tail 50 + docker logs corescope-staging-go --tail 50 exit 1 fi sleep 1 @@ -338,6 +347,7 @@ jobs: # ─────────────────────────────────────────────────────────────── publish: name: "📝 Publish Badges & Summary" + if: github.event_name == 'push' needs: [deploy] runs-on: self-hosted steps: @@ -378,6 +388,6 @@ jobs: echo "To promote to production:" >> $GITHUB_STEP_SUMMARY echo "\`\`\`bash" >> $GITHUB_STEP_SUMMARY echo "ssh deploy@\$VM_HOST" >> $GITHUB_STEP_SUMMARY - echo "cd /opt/meshcore-deploy" >> $GITHUB_STEP_SUMMARY + echo "cd /opt/corescope-deploy" >> $GITHUB_STEP_SUMMARY echo "./manage.sh promote" >> $GITHUB_STEP_SUMMARY echo "\`\`\`" >> $GITHUB_STEP_SUMMARY diff --git a/.squad/agents/bishop/charter.md b/.squad/agents/bishop/charter.md index a3d37a8e..9e64613e 100644 --- a/.squad/agents/bishop/charter.md +++ b/.squad/agents/bishop/charter.md @@ -1,10 +1,10 @@ # Bishop — Tester -Unit tests, Playwright E2E, coverage gates, and quality assurance for MeshCore Analyzer. +Unit tests, Playwright E2E, coverage gates, and quality assurance for CoreScope. ## Project Context -**Project:** MeshCore Analyzer — Real-time LoRa mesh packet analyzer +**Project:** CoreScope — Real-time LoRa mesh packet analyzer **Stack:** Node.js native test runner, Playwright, c8 + nyc (coverage), supertest **User:** User diff --git a/.squad/agents/bishop/history.md b/.squad/agents/bishop/history.md index 99292a4a..5792f3cf 100644 --- a/.squad/agents/bishop/history.md +++ b/.squad/agents/bishop/history.md @@ -2,7 +2,7 @@ ## Project Context -MeshCore Analyzer has 14 test files, 4,290 lines of test code. Backend coverage 85%+, frontend 42%+. Tests use Node.js native runner, Playwright for E2E, c8/nyc for coverage, supertest for API routes. vm.createContext pattern used for testing frontend helpers in Node.js. +CoreScope has 14 test files, 4,290 lines of test code. Backend coverage 85%+, frontend 42%+. Tests use Node.js native runner, Playwright for E2E, c8/nyc for coverage, supertest for API routes. vm.createContext pattern used for testing frontend helpers in Node.js. User: User diff --git a/.squad/agents/hicks/charter.md b/.squad/agents/hicks/charter.md index 61b3dbfe..9a7d12ef 100644 --- a/.squad/agents/hicks/charter.md +++ b/.squad/agents/hicks/charter.md @@ -1,10 +1,10 @@ # Hicks — Backend Dev -Server, decoder, packet-store, SQLite, API, MQTT, WebSocket, and performance for MeshCore Analyzer. +Server, decoder, packet-store, SQLite, API, MQTT, WebSocket, and performance for CoreScope. ## Project Context -**Project:** MeshCore Analyzer — Real-time LoRa mesh packet analyzer +**Project:** CoreScope — Real-time LoRa mesh packet analyzer **Stack:** Node.js 18+, Express 5, SQLite (better-sqlite3), MQTT (mqtt), WebSocket (ws) **User:** User diff --git a/.squad/agents/hicks/history.md b/.squad/agents/hicks/history.md index 077192d4..4e521242 100644 --- a/.squad/agents/hicks/history.md +++ b/.squad/agents/hicks/history.md @@ -2,7 +2,7 @@ ## Project Context -MeshCore Analyzer is a real-time LoRa mesh packet analyzer. Node.js + Express + SQLite backend, vanilla JS SPA frontend. Custom decoder.js fixes path_length bug from upstream library. In-memory packet store provides O(1) lookups for 30K+ packets. TTL response cache achieves 7,000× speedup on bulk health endpoint. +CoreScope is a real-time LoRa mesh packet analyzer. Node.js + Express + SQLite backend, vanilla JS SPA frontend. Custom decoder.js fixes path_length bug from upstream library. In-memory packet store provides O(1) lookups for 30K+ packets. TTL response cache achieves 7,000× speedup on bulk health endpoint. User: User diff --git a/.squad/agents/kobayashi/charter.md b/.squad/agents/kobayashi/charter.md index caaa2c00..ff5aeb4d 100644 --- a/.squad/agents/kobayashi/charter.md +++ b/.squad/agents/kobayashi/charter.md @@ -1,10 +1,10 @@ # Kobayashi — Lead -Architecture, code review, and decision-making for MeshCore Analyzer. +Architecture, code review, and decision-making for CoreScope. ## Project Context -**Project:** MeshCore Analyzer — Real-time LoRa mesh packet analyzer +**Project:** CoreScope — Real-time LoRa mesh packet analyzer **Stack:** Node.js 18+, Express 5, SQLite, vanilla JS frontend, Leaflet, WebSocket, MQTT **User:** User diff --git a/.squad/agents/kobayashi/history.md b/.squad/agents/kobayashi/history.md index 18381061..b49a33ba 100644 --- a/.squad/agents/kobayashi/history.md +++ b/.squad/agents/kobayashi/history.md @@ -2,7 +2,7 @@ ## Project Context -MeshCore Analyzer is a real-time LoRa mesh packet analyzer. Node.js + Express + SQLite backend, vanilla JS SPA frontend with Leaflet maps, WebSocket live feed, MQTT ingestion. Production at v2.6.0, ~18K lines, 85%+ backend test coverage. +CoreScope is a real-time LoRa mesh packet analyzer. Node.js + Express + SQLite backend, vanilla JS SPA frontend with Leaflet maps, WebSocket live feed, MQTT ingestion. Production at v2.6.0, ~18K lines, 85%+ backend test coverage. User: User diff --git a/.squad/agents/newt/charter.md b/.squad/agents/newt/charter.md index d6152ace..f35c59e1 100644 --- a/.squad/agents/newt/charter.md +++ b/.squad/agents/newt/charter.md @@ -1,10 +1,10 @@ # Newt — Frontend Dev -Vanilla JS UI, Leaflet maps, live visualization, theming, and all public/ modules for MeshCore Analyzer. +Vanilla JS UI, Leaflet maps, live visualization, theming, and all public/ modules for CoreScope. ## Project Context -**Project:** MeshCore Analyzer — Real-time LoRa mesh packet analyzer +**Project:** CoreScope — Real-time LoRa mesh packet analyzer **Stack:** Vanilla HTML/CSS/JavaScript (ES5/6), Leaflet maps, WebSocket, Canvas animations **User:** User diff --git a/.squad/agents/newt/history.md b/.squad/agents/newt/history.md index abc42c1e..3d4d8207 100644 --- a/.squad/agents/newt/history.md +++ b/.squad/agents/newt/history.md @@ -2,7 +2,7 @@ ## Project Context -MeshCore Analyzer is a real-time LoRa mesh packet analyzer with a vanilla JS SPA frontend. 22 frontend modules, Leaflet maps, WebSocket live feed, VCR playback, Canvas animations, theme customizer with CSS variables. No build step, no framework. ES5/6 for broad browser support. +CoreScope is a real-time LoRa mesh packet analyzer with a vanilla JS SPA frontend. 22 frontend modules, Leaflet maps, WebSocket live feed, VCR playback, Canvas animations, theme customizer with CSS variables. No build step, no framework. ES5/6 for broad browser support. User: User diff --git a/.squad/agents/ralph/charter.md b/.squad/agents/ralph/charter.md index 7f7ef5b5..dd36f312 100644 --- a/.squad/agents/ralph/charter.md +++ b/.squad/agents/ralph/charter.md @@ -4,7 +4,7 @@ Tracks the work queue and keeps the team moving. Always on the roster. ## Project Context -**Project:** MeshCore Analyzer — Real-time LoRa mesh packet analyzer +**Project:** CoreScope — Real-time LoRa mesh packet analyzer **User:** User ## Responsibilities diff --git a/.squad/agents/ripley/charter.md b/.squad/agents/ripley/charter.md index c96666f4..352336bb 100644 --- a/.squad/agents/ripley/charter.md +++ b/.squad/agents/ripley/charter.md @@ -1,10 +1,10 @@ # Ripley — Support Engineer -Deep knowledge of every frontend behavior, API response, and user-facing feature in MeshCore Analyzer. Fields community questions, triages bug reports, and explains "why does X look like Y." +Deep knowledge of every frontend behavior, API response, and user-facing feature in CoreScope. Fields community questions, triages bug reports, and explains "why does X look like Y." ## Project Context -**Project:** MeshCore Analyzer — Real-time LoRa mesh packet analyzer +**Project:** CoreScope — Real-time LoRa mesh packet analyzer **Stack:** Vanilla JS frontend (public/*.js), Node.js backend, SQLite, WebSocket, MQTT **User:** Kpa-clawbot diff --git a/.squad/agents/ripley/history.md b/.squad/agents/ripley/history.md index e99a50b2..86c8611d 100644 --- a/.squad/agents/ripley/history.md +++ b/.squad/agents/ripley/history.md @@ -1,7 +1,7 @@ # Ripley — Support Engineer History ## Core Context -- Project: MeshCore Analyzer — real-time LoRa mesh packet analyzer +- Project: CoreScope — real-time LoRa mesh packet analyzer - User: Kpa-clawbot - Joined the team 2026-03-27 to handle community support and triage diff --git a/.squad/agents/scribe/charter.md b/.squad/agents/scribe/charter.md index 1bf1dcf1..2209a57b 100644 --- a/.squad/agents/scribe/charter.md +++ b/.squad/agents/scribe/charter.md @@ -1,10 +1,10 @@ # Scribe — Session Logger -Silent agent that maintains decisions, logs, and cross-agent context for MeshCore Analyzer. +Silent agent that maintains decisions, logs, and cross-agent context for CoreScope. ## Project Context -**Project:** MeshCore Analyzer — Real-time LoRa mesh packet analyzer +**Project:** CoreScope — Real-time LoRa mesh packet analyzer **User:** User ## Responsibilities diff --git a/.squad/casting/history.json b/.squad/casting/history.json index bf07c51c..1a27f6b6 100644 --- a/.squad/casting/history.json +++ b/.squad/casting/history.json @@ -5,7 +5,7 @@ "universe": "aliens", "created_at": "2026-03-26T04:22:08Z", "agents": ["Kobayashi", "Hicks", "Newt", "Bishop"], - "reason": "Initial team casting for MeshCore Analyzer project" + "reason": "Initial team casting for CoreScope project" } ] } diff --git a/.squad/team.md b/.squad/team.md index cfb7bba7..d45190e4 100644 --- a/.squad/team.md +++ b/.squad/team.md @@ -1,8 +1,8 @@ -# Squad — MeshCore Analyzer +# Squad — CoreScope ## Project Context -**Project:** MeshCore Analyzer — Real-time LoRa mesh packet analyzer +**Project:** CoreScope — Real-time LoRa mesh packet analyzer **Stack:** Node.js 18+, Express 5, SQLite (better-sqlite3), vanilla JS frontend, Leaflet maps, WebSocket (ws), MQTT (mqtt) **User:** User **Description:** Self-hosted alternative to analyzer.letsmesh.net. Ingests MeshCore mesh network packets via MQTT, decodes with custom parser (decoder.js), stores in SQLite with in-memory indexing (packet-store.js), and serves a rich SPA with live visualization, packet analysis, node analytics, channel chat, observer health, and theme customizer. ~18K lines, 14 test files, 85%+ backend coverage. Production at v2.6.0. diff --git a/AGENTS.md b/AGENTS.md index 7d5d59e3..4698c413 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,4 +1,4 @@ -# AGENTS.md — MeshCore Analyzer +# AGENTS.md — CoreScope Guide for AI agents working on this codebase. Read this before writing any code. diff --git a/Dockerfile b/Dockerfile index 9920a105..f96617f0 100644 --- a/Dockerfile +++ b/Dockerfile @@ -11,14 +11,14 @@ WORKDIR /build/server COPY cmd/server/go.mod cmd/server/go.sum ./ RUN go mod download COPY cmd/server/ ./ -RUN go build -ldflags "-X main.Version=${APP_VERSION} -X main.Commit=${GIT_COMMIT} -X main.BuildTime=${BUILD_TIME}" -o /meshcore-server . +RUN go build -ldflags "-X main.Version=${APP_VERSION} -X main.Commit=${GIT_COMMIT} -X main.BuildTime=${BUILD_TIME}" -o /corescope-server . # Build ingestor WORKDIR /build/ingestor COPY cmd/ingestor/go.mod cmd/ingestor/go.sum ./ RUN go mod download COPY cmd/ingestor/ ./ -RUN go build -o /meshcore-ingestor . +RUN go build -o /corescope-ingestor . # Runtime image FROM alpine:3.20 @@ -28,7 +28,7 @@ RUN apk add --no-cache mosquitto mosquitto-clients supervisor caddy wget WORKDIR /app # Go binaries -COPY --from=builder /meshcore-server /meshcore-ingestor /app/ +COPY --from=builder /corescope-server /corescope-ingestor /app/ # Frontend assets + config COPY public/ ./public/ diff --git a/Dockerfile.go b/Dockerfile.go index 427a5d4f..6819e579 100644 --- a/Dockerfile.go +++ b/Dockerfile.go @@ -11,14 +11,14 @@ WORKDIR /build/server COPY cmd/server/go.mod cmd/server/go.sum ./ RUN go mod download COPY cmd/server/ ./ -RUN go build -ldflags "-X main.Version=${APP_VERSION} -X main.Commit=${GIT_COMMIT} -X main.BuildTime=${BUILD_TIME}" -o /meshcore-server . +RUN go build -ldflags "-X main.Version=${APP_VERSION} -X main.Commit=${GIT_COMMIT} -X main.BuildTime=${BUILD_TIME}" -o /corescope-server . # Build ingestor WORKDIR /build/ingestor COPY cmd/ingestor/go.mod cmd/ingestor/go.sum ./ RUN go mod download COPY cmd/ingestor/ ./ -RUN go build -o /meshcore-ingestor . +RUN go build -o /corescope-ingestor . # Runtime image FROM alpine:3.20 @@ -28,7 +28,7 @@ RUN apk add --no-cache mosquitto mosquitto-clients supervisor caddy wget WORKDIR /app # Go binaries -COPY --from=builder /meshcore-server /meshcore-ingestor /app/ +COPY --from=builder /corescope-server /corescope-ingestor /app/ # Frontend assets + config COPY public/ ./public/ diff --git a/README.md b/README.md index 39c90e48..cbd0d2c9 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,10 @@ -# MeshCore Analyzer +# CoreScope -[](https://github.com/Kpa-clawbot/meshcore-analyzer/actions/workflows/deploy.yml) -[](https://github.com/Kpa-clawbot/meshcore-analyzer/actions/workflows/deploy.yml) -[](https://github.com/Kpa-clawbot/meshcore-analyzer/actions/workflows/deploy.yml) -[](https://github.com/Kpa-clawbot/meshcore-analyzer/actions/workflows/deploy.yml) -[](https://github.com/Kpa-clawbot/meshcore-analyzer/actions/workflows/deploy.yml) +[](https://github.com/Kpa-clawbot/corescope/actions/workflows/deploy.yml) +[](https://github.com/Kpa-clawbot/corescope/actions/workflows/deploy.yml) +[](https://github.com/Kpa-clawbot/corescope/actions/workflows/deploy.yml) +[](https://github.com/Kpa-clawbot/corescope/actions/workflows/deploy.yml) +[](https://github.com/Kpa-clawbot/corescope/actions/workflows/deploy.yml) > High-performance mesh network analyzer powered by Go. Sub-millisecond packet queries, ~300 MB memory for 56K+ packets, real-time WebSocket broadcast, full channel decryption. @@ -79,8 +79,8 @@ Full experience on your phone — proper touch controls, iOS safe area support, No Go installation needed — everything builds inside the container. ```bash -git clone https://github.com/Kpa-clawbot/meshcore-analyzer.git -cd meshcore-analyzer +git clone https://github.com/Kpa-clawbot/corescope.git +cd corescope ./manage.sh setup ``` @@ -171,7 +171,7 @@ Or POST raw hex packets to `POST /api/packets` for manual injection. ## Project Structure ``` -meshcore-analyzer/ +corescope/ ├── cmd/ │ ├── server/ # Go HTTP server + WebSocket + REST API │ │ ├── main.go # Entry point diff --git a/RELEASE-v3.0.0.md b/RELEASE-v3.0.0.md index aece5ab9..c249a10f 100644 --- a/RELEASE-v3.0.0.md +++ b/RELEASE-v3.0.0.md @@ -73,8 +73,8 @@ Advert counts now reflect unique transmissions, not total observations. A packet The Go backend is two binaries managed by supervisord inside Docker: -- **`meshcore-ingestor`** — connects to MQTT brokers, decodes packets, writes to SQLite, maintains the in-memory store -- **`meshcore-server`** — HTTP API, WebSocket broadcast, static file serving, analytics computation +- **`corescope-ingestor`** — connects to MQTT brokers, decodes packets, writes to SQLite, maintains the in-memory store +- **`corescope-server`** — HTTP API, WebSocket broadcast, static file serving, analytics computation Both share the same SQLite database (WAL mode). The frontend is unchanged — same vanilla JS, same `public/` directory, served by the Go HTTP server through Caddy. @@ -120,7 +120,7 @@ curl -s http://localhost/api/health | grep engine The Node.js Dockerfile is preserved as `Dockerfile.node`: ```bash -docker build -f Dockerfile.node -t meshcore-analyzer:latest . +docker build -f Dockerfile.node -t corescope:latest . docker compose up -d --force-recreate prod ``` @@ -152,7 +152,7 @@ This release wouldn't exist without the community: - **LitBomb** — issue reports from production deployments - **mibzzer15** — issue reports and edge case discovery -And to everyone running MeshCore Analyzer in the wild — your packet data, bug reports, and feature requests are what drive this project forward. The Go rewrite happened because the community outgrew what Node.js could handle. 56K packets, dozens of observers, sub-second queries. This is your tool. We just rewrote the engine. +And to everyone running CoreScope in the wild — your packet data, bug reports, and feature requests are what drive this project forward. The Go rewrite happened because the community outgrew what Node.js could handle. 56K packets, dozens of observers, sub-second queries. This is your tool. We just rewrote the engine. --- diff --git a/benchmark.js b/benchmark.js index d7ce92d1..e7b39899 100644 --- a/benchmark.js +++ b/benchmark.js @@ -148,7 +148,7 @@ async function benchmarkEndpoints(port, endpoints, nocache = false) { } async function run() { - console.log(`\nMeshCore Analyzer Benchmark — ${RUNS} runs per endpoint`); + console.log(`\nCoreScope Benchmark — ${RUNS} runs per endpoint`); console.log('Launching servers...\n'); // Launch both servers diff --git a/cmd/ingestor/README.md b/cmd/ingestor/README.md index abd46a84..24eb0718 100644 --- a/cmd/ingestor/README.md +++ b/cmd/ingestor/README.md @@ -1,6 +1,6 @@ # MeshCore MQTT Ingestor (Go) -Standalone MQTT ingestion service for MeshCore Analyzer. Connects to MQTT brokers, decodes raw MeshCore packets, and writes to the same SQLite database used by the Node.js web server. +Standalone MQTT ingestion service for CoreScope. Connects to MQTT brokers, decodes raw MeshCore packets, and writes to the same SQLite database used by the Node.js web server. This is the first step of a larger Go rewrite — separating MQTT ingestion from the web server. @@ -23,19 +23,19 @@ Requires Go 1.22+. ```bash cd cmd/ingestor -go build -o meshcore-ingestor . +go build -o corescope-ingestor . ``` Cross-compile for Linux (e.g., for the production VM): ```bash -GOOS=linux GOARCH=amd64 go build -o meshcore-ingestor . +GOOS=linux GOARCH=amd64 go build -o corescope-ingestor . ``` ## Run ```bash -./meshcore-ingestor -config /path/to/config.json +./corescope-ingestor -config /path/to/config.json ``` The config file uses the same format as the Node.js `config.json`. The ingestor reads the `mqttSources` array (or legacy `mqtt` object) and `dbPath` fields. diff --git a/cmd/ingestor/config.go b/cmd/ingestor/config.go index 850014bd..d840c32b 100644 --- a/cmd/ingestor/config.go +++ b/cmd/ingestor/config.go @@ -26,13 +26,14 @@ type MQTTLegacy struct { // Config holds the ingestor configuration, compatible with the Node.js config.json format. type Config struct { - DBPath string `json:"dbPath"` - MQTT *MQTTLegacy `json:"mqtt,omitempty"` - MQTTSources []MQTTSource `json:"mqttSources,omitempty"` - LogLevel string `json:"logLevel,omitempty"` - ChannelKeysPath string `json:"channelKeysPath,omitempty"` - ChannelKeys map[string]string `json:"channelKeys,omitempty"` - Retention *RetentionConfig `json:"retention,omitempty"` + DBPath string `json:"dbPath"` + MQTT *MQTTLegacy `json:"mqtt,omitempty"` + MQTTSources []MQTTSource `json:"mqttSources,omitempty"` + LogLevel string `json:"logLevel,omitempty"` + ChannelKeysPath string `json:"channelKeysPath,omitempty"` + ChannelKeys map[string]string `json:"channelKeys,omitempty"` + HashChannels []string `json:"hashChannels,omitempty"` + Retention *RetentionConfig `json:"retention,omitempty"` } // RetentionConfig controls how long stale nodes are kept before being moved to inactive_nodes. diff --git a/cmd/ingestor/db.go b/cmd/ingestor/db.go index 3a86f1bd..74bd69e1 100644 --- a/cmd/ingestor/db.go +++ b/cmd/ingestor/db.go @@ -35,7 +35,8 @@ type Store struct { stmtUpsertNode *sql.Stmt stmtIncrementAdvertCount *sql.Stmt stmtUpsertObserver *sql.Stmt - stmtGetObserverRowid *sql.Stmt + stmtGetObserverRowid *sql.Stmt + stmtUpdateNodeTelemetry *sql.Stmt } // OpenStore opens or creates a SQLite DB at the given path, applying the @@ -81,7 +82,9 @@ func applySchema(db *sql.DB) error { lon REAL, last_seen TEXT, first_seen TEXT, - advert_count INTEGER DEFAULT 0 + advert_count INTEGER DEFAULT 0, + battery_mv INTEGER, + temperature_c REAL ); CREATE TABLE IF NOT EXISTS observers ( @@ -111,7 +114,9 @@ func applySchema(db *sql.DB) error { lon REAL, last_seen TEXT, first_seen TEXT, - advert_count INTEGER DEFAULT 0 + advert_count INTEGER DEFAULT 0, + battery_mv INTEGER, + temperature_c REAL ); CREATE INDEX IF NOT EXISTS idx_inactive_nodes_last_seen ON inactive_nodes(last_seen); @@ -215,6 +220,65 @@ func applySchema(db *sql.DB) error { log.Println("[migration] noise_floor migration complete") } + // One-time migration: add telemetry columns to nodes and inactive_nodes tables. + row = db.QueryRow("SELECT 1 FROM _migrations WHERE name = 'node_telemetry_v1'") + if row.Scan(&migDone) != nil { + log.Println("[migration] Adding telemetry columns to nodes/inactive_nodes...") + + // checkAndAddColumn checks whether `column` already exists in `table` + // using PRAGMA table_info, and adds it if missing. All call sites pass + // hardcoded table/column/type literals so there is no SQL injection risk. + checkAndAddColumn := func(table, column, colType string) error { + rows, err := db.Query(fmt.Sprintf("PRAGMA table_info(%s)", table)) + if err != nil { + return fmt.Errorf("querying table info for %s: %w", table, err) + } + defer rows.Close() + + exists := false + for rows.Next() { + var cid int + var name, ctype string + var notnull, pk int + var dfltValue sql.NullString + if err := rows.Scan(&cid, &name, &ctype, ¬null, &dfltValue, &pk); err != nil { + return fmt.Errorf("scanning table info for %s: %w", table, err) + } + if name == column { + exists = true + break + } + } + if err := rows.Err(); err != nil { + return fmt.Errorf("iterating table info for %s: %w", table, err) + } + if exists { + return nil + } + if _, err := db.Exec(fmt.Sprintf("ALTER TABLE %s ADD COLUMN %s %s", table, column, colType)); err != nil { + return fmt.Errorf("adding column %s to %s: %w", column, table, err) + } + return nil + } + + if err := checkAndAddColumn("nodes", "battery_mv", "INTEGER"); err != nil { + return err + } + if err := checkAndAddColumn("nodes", "temperature_c", "REAL"); err != nil { + return err + } + if err := checkAndAddColumn("inactive_nodes", "battery_mv", "INTEGER"); err != nil { + return err + } + if err := checkAndAddColumn("inactive_nodes", "temperature_c", "REAL"); err != nil { + return err + } + if _, err := db.Exec(`INSERT INTO _migrations (name) VALUES ('node_telemetry_v1')`); err != nil { + return fmt.Errorf("recording node_telemetry_v1 migration: %w", err) + } + log.Println("[migration] node telemetry columns added") + } + return nil } @@ -289,6 +353,16 @@ func (s *Store) prepareStatements() error { return err } + s.stmtUpdateNodeTelemetry, err = s.db.Prepare(` + UPDATE nodes SET + battery_mv = COALESCE(?, battery_mv), + temperature_c = COALESCE(?, temperature_c) + WHERE public_key = ? + `) + if err != nil { + return err + } + return nil } @@ -393,6 +467,22 @@ func (s *Store) IncrementAdvertCount(pubKey string) error { return err } +// UpdateNodeTelemetry updates battery and temperature for a node. +func (s *Store) UpdateNodeTelemetry(pubKey string, batteryMv *int, temperatureC *float64) error { + var bv, tc interface{} + if batteryMv != nil { + bv = *batteryMv + } + if temperatureC != nil { + tc = *temperatureC + } + _, err := s.stmtUpdateNodeTelemetry.Exec(bv, tc, pubKey) + if err != nil { + s.Stats.WriteErrors.Add(1) + } + return err +} + // ObserverMeta holds optional observer hardware metadata. type ObserverMeta struct { BatteryMv *int // millivolts, always integer diff --git a/cmd/ingestor/db_test.go b/cmd/ingestor/db_test.go index a719630f..fef7241f 100644 --- a/cmd/ingestor/db_test.go +++ b/cmd/ingestor/db_test.go @@ -1151,3 +1151,75 @@ func TestLoadTestThroughput(t *testing.T) { t.Errorf("transmissions=%d, want %d", txCount, totalMessages) } } + +func TestUpdateNodeTelemetry(t *testing.T) { + s, err := OpenStore(tempDBPath(t)) + if err != nil { + t.Fatal(err) + } + defer s.Close() + + lat := 37.0 + lon := -122.0 + if err := s.UpsertNode("telem1", "TelemetryNode", "sensor", &lat, &lon, "2026-03-25T00:00:00Z"); err != nil { + t.Fatal(err) + } + + battery := 3700 + temp := 28.5 + if err := s.UpdateNodeTelemetry("telem1", &battery, &temp); err != nil { + t.Fatal(err) + } + + var bv int + var tc float64 + err = s.db.QueryRow("SELECT battery_mv, temperature_c FROM nodes WHERE public_key = 'telem1'").Scan(&bv, &tc) + if err != nil { + t.Fatal(err) + } + if bv != 3700 { + t.Errorf("battery_mv=%d, want 3700", bv) + } + if tc != 28.5 { + t.Errorf("temperature_c=%f, want 28.5", tc) + } + + newTemp := -5.0 + if err := s.UpdateNodeTelemetry("telem1", nil, &newTemp); err != nil { + t.Fatal(err) + } + err = s.db.QueryRow("SELECT battery_mv, temperature_c FROM nodes WHERE public_key = 'telem1'").Scan(&bv, &tc) + if err != nil { + t.Fatal(err) + } + if bv != 3700 { + t.Errorf("battery_mv after nil update=%d, want 3700 (preserved)", bv) + } + if tc != -5.0 { + t.Errorf("temperature_c after update=%f, want -5.0", tc) + } +} + +func TestTelemetryMigrationAddsColumns(t *testing.T) { + s, err := OpenStore(tempDBPath(t)) + if err != nil { + t.Fatal(err) + } + defer s.Close() + + _, err = s.db.Exec("SELECT battery_mv, temperature_c FROM nodes LIMIT 1") + if err != nil { + t.Errorf("nodes table should have battery_mv and temperature_c columns: %v", err) + } + + _, err = s.db.Exec("SELECT battery_mv, temperature_c FROM inactive_nodes LIMIT 1") + if err != nil { + t.Errorf("inactive_nodes table should have battery_mv and temperature_c columns: %v", err) + } + + var count int + s.db.QueryRow("SELECT COUNT(*) FROM _migrations WHERE name = 'node_telemetry_v1'").Scan(&count) + if count != 1 { + t.Errorf("migration node_telemetry_v1 should be recorded, count=%d", count) + } +} diff --git a/cmd/ingestor/decoder.go b/cmd/ingestor/decoder.go index ba1a8da1..147ec883 100644 --- a/cmd/ingestor/decoder.go +++ b/cmd/ingestor/decoder.go @@ -111,6 +111,8 @@ type Payload struct { Lat *float64 `json:"lat,omitempty"` Lon *float64 `json:"lon,omitempty"` Name string `json:"name,omitempty"` + BatteryMv *int `json:"battery_mv,omitempty"` + TemperatureC *float64 `json:"temperature_c,omitempty"` ChannelHash int `json:"channelHash,omitempty"` ChannelHashHex string `json:"channelHashHex,omitempty"` DecryptionStatus string `json:"decryptionStatus,omitempty"` @@ -251,10 +253,37 @@ func decodeAdvert(buf []byte) Payload { off += 8 } if p.Flags.HasName { - name := string(appdata[off:]) - name = strings.TrimRight(name, "\x00") + // Find null terminator to separate name from trailing telemetry bytes + nameEnd := len(appdata) + for i := off; i < len(appdata); i++ { + if appdata[i] == 0x00 { + nameEnd = i + break + } + } + name := string(appdata[off:nameEnd]) name = sanitizeName(name) p.Name = name + off = nameEnd + // Skip null terminator(s) + for off < len(appdata) && appdata[off] == 0x00 { + off++ + } + } + + // Telemetry bytes after name: battery_mv(2 LE) + temperature_c(2 LE, signed, /100) + // Only sensor nodes (advType=4) carry telemetry bytes. + if p.Flags.Sensor && off+4 <= len(appdata) { + batteryMv := int(binary.LittleEndian.Uint16(appdata[off : off+2])) + tempRaw := int16(binary.LittleEndian.Uint16(appdata[off+2 : off+4])) + tempC := float64(tempRaw) / 100.0 + if batteryMv > 0 && batteryMv <= 10000 { + p.BatteryMv = &batteryMv + } + // Raw int16 / 100 → °C; accept -50°C to 100°C (raw: -5000 to 10000) + if tempRaw >= -5000 && tempRaw <= 10000 { + p.TemperatureC = &tempC + } } } diff --git a/cmd/ingestor/decoder_test.go b/cmd/ingestor/decoder_test.go index 58108733..8c219dd0 100644 --- a/cmd/ingestor/decoder_test.go +++ b/cmd/ingestor/decoder_test.go @@ -1355,3 +1355,154 @@ func TestDecodeGrpTxtGarbageMarkedFailed(t *testing.T) { t.Errorf("type=%s, want GRP_TXT", p.Type) } } + +func TestDecodeAdvertWithTelemetry(t *testing.T) { + pubkey := strings.Repeat("AA", 32) + timestamp := "78563412" + signature := strings.Repeat("BB", 64) + flags := "94" // sensor(4) | hasLocation(0x10) | hasName(0x80) + lat := "40933402" + lon := "E0E6B8F8" + name := hex.EncodeToString([]byte("Sensor1")) + nullTerm := "00" + batteryLE := make([]byte, 2) + binary.LittleEndian.PutUint16(batteryLE, 3700) + tempLE := make([]byte, 2) + binary.LittleEndian.PutUint16(tempLE, uint16(int16(2850))) + + hexStr := "1200" + pubkey + timestamp + signature + flags + lat + lon + + name + nullTerm + + hex.EncodeToString(batteryLE) + hex.EncodeToString(tempLE) + + pkt, err := DecodePacket(hexStr, nil) + if err != nil { + t.Fatal(err) + } + + if pkt.Payload.Name != "Sensor1" { + t.Errorf("name=%s, want Sensor1", pkt.Payload.Name) + } + if pkt.Payload.BatteryMv == nil { + t.Fatal("battery_mv should not be nil") + } + if *pkt.Payload.BatteryMv != 3700 { + t.Errorf("battery_mv=%d, want 3700", *pkt.Payload.BatteryMv) + } + if pkt.Payload.TemperatureC == nil { + t.Fatal("temperature_c should not be nil") + } + if math.Abs(*pkt.Payload.TemperatureC-28.50) > 0.01 { + t.Errorf("temperature_c=%f, want 28.50", *pkt.Payload.TemperatureC) + } +} + +func TestDecodeAdvertWithTelemetryNegativeTemp(t *testing.T) { + pubkey := strings.Repeat("CC", 32) + timestamp := "00000000" + signature := strings.Repeat("DD", 64) + flags := "84" // sensor(4) | hasName(0x80), no location + name := hex.EncodeToString([]byte("Cold")) + nullTerm := "00" + batteryLE := make([]byte, 2) + binary.LittleEndian.PutUint16(batteryLE, 4200) + tempLE := make([]byte, 2) + var negTemp int16 = -550 + binary.LittleEndian.PutUint16(tempLE, uint16(negTemp)) + + hexStr := "1200" + pubkey + timestamp + signature + flags + + name + nullTerm + + hex.EncodeToString(batteryLE) + hex.EncodeToString(tempLE) + + pkt, err := DecodePacket(hexStr, nil) + if err != nil { + t.Fatal(err) + } + + if pkt.Payload.Name != "Cold" { + t.Errorf("name=%s, want Cold", pkt.Payload.Name) + } + if pkt.Payload.BatteryMv == nil || *pkt.Payload.BatteryMv != 4200 { + t.Errorf("battery_mv=%v, want 4200", pkt.Payload.BatteryMv) + } + if pkt.Payload.TemperatureC == nil { + t.Fatal("temperature_c should not be nil") + } + if math.Abs(*pkt.Payload.TemperatureC-(-5.50)) > 0.01 { + t.Errorf("temperature_c=%f, want -5.50", *pkt.Payload.TemperatureC) + } +} + +func TestDecodeAdvertWithoutTelemetry(t *testing.T) { + pubkey := strings.Repeat("EE", 32) + timestamp := "00000000" + signature := strings.Repeat("FF", 64) + flags := "82" // repeater(2) | hasName(0x80) + name := hex.EncodeToString([]byte("Node1")) + + hexStr := "1200" + pubkey + timestamp + signature + flags + name + pkt, err := DecodePacket(hexStr, nil) + if err != nil { + t.Fatal(err) + } + + if pkt.Payload.Name != "Node1" { + t.Errorf("name=%s, want Node1", pkt.Payload.Name) + } + if pkt.Payload.BatteryMv != nil { + t.Errorf("battery_mv should be nil for advert without telemetry, got %d", *pkt.Payload.BatteryMv) + } + if pkt.Payload.TemperatureC != nil { + t.Errorf("temperature_c should be nil for advert without telemetry, got %f", *pkt.Payload.TemperatureC) + } +} + +func TestDecodeAdvertNonSensorIgnoresTelemetryBytes(t *testing.T) { + // A repeater node with 4 trailing bytes after the name should NOT decode telemetry. + pubkey := strings.Repeat("AB", 32) + timestamp := "00000000" + signature := strings.Repeat("CD", 64) + flags := "82" // repeater(2) | hasName(0x80) + name := hex.EncodeToString([]byte("Rptr")) + nullTerm := "00" + extraBytes := "B40ED403" // battery-like and temp-like bytes + + hexStr := "1200" + pubkey + timestamp + signature + flags + name + nullTerm + extraBytes + pkt, err := DecodePacket(hexStr, nil) + if err != nil { + t.Fatal(err) + } + if pkt.Payload.BatteryMv != nil { + t.Errorf("battery_mv should be nil for non-sensor node, got %d", *pkt.Payload.BatteryMv) + } + if pkt.Payload.TemperatureC != nil { + t.Errorf("temperature_c should be nil for non-sensor node, got %f", *pkt.Payload.TemperatureC) + } +} + +func TestDecodeAdvertTelemetryZeroTemp(t *testing.T) { + // 0°C is a valid temperature and must be emitted. + pubkey := strings.Repeat("12", 32) + timestamp := "00000000" + signature := strings.Repeat("34", 64) + flags := "84" // sensor(4) | hasName(0x80) + name := hex.EncodeToString([]byte("FreezeSensor")) + nullTerm := "00" + batteryLE := make([]byte, 2) + binary.LittleEndian.PutUint16(batteryLE, 3600) + tempLE := make([]byte, 2) // tempRaw=0 → 0°C + + hexStr := "1200" + pubkey + timestamp + signature + flags + + name + nullTerm + + hex.EncodeToString(batteryLE) + hex.EncodeToString(tempLE) + + pkt, err := DecodePacket(hexStr, nil) + if err != nil { + t.Fatal(err) + } + if pkt.Payload.TemperatureC == nil { + t.Fatal("temperature_c should not be nil for 0°C") + } + if *pkt.Payload.TemperatureC != 0.0 { + t.Errorf("temperature_c=%f, want 0.0", *pkt.Payload.TemperatureC) + } +} diff --git a/cmd/ingestor/go.mod b/cmd/ingestor/go.mod index 7ef0f163..cc2098e7 100644 --- a/cmd/ingestor/go.mod +++ b/cmd/ingestor/go.mod @@ -1,4 +1,4 @@ -module github.com/meshcore-analyzer/ingestor +module github.com/corescope/ingestor go 1.22 diff --git a/cmd/ingestor/main.go b/cmd/ingestor/main.go index a16e2b3d..78fe3310 100644 --- a/cmd/ingestor/main.go +++ b/cmd/ingestor/main.go @@ -270,6 +270,12 @@ func handleMessage(store *Store, tag string, source MQTTSource, m mqtt.Message, log.Printf("MQTT [%s] advert count error: %v", tag, err) } } + // Update telemetry if present in advert + if decoded.Payload.BatteryMv != nil || decoded.Payload.TemperatureC != nil { + if err := store.UpdateNodeTelemetry(decoded.Payload.PubKey, decoded.Payload.BatteryMv, decoded.Payload.TemperatureC); err != nil { + log.Printf("MQTT [%s] node telemetry update error: %v", tag, err) + } + } } else { log.Printf("MQTT [%s] skipping corrupted ADVERT: %s", tag, reason) } @@ -506,34 +512,64 @@ func firstNonEmpty(vals ...string) string { return "" } +// deriveHashtagChannelKey derives an AES-128 key from a channel name. +// Same algorithm as Node.js: SHA-256(channelName) → first 32 hex chars (16 bytes). +func deriveHashtagChannelKey(channelName string) string { + h := sha256.Sum256([]byte(channelName)) + return hex.EncodeToString(h[:16]) +} + // loadChannelKeys loads channel decryption keys from config and/or a JSON file. -// Priority: CHANNEL_KEYS_PATH env var > cfg.ChannelKeysPath > channel-rainbow.json next to config. +// Merge priority: rainbow (lowest) → derived from hashChannels → explicit config (highest). func loadChannelKeys(cfg *Config, configPath string) map[string]string { keys := make(map[string]string) - // Determine file path for rainbow keys + // 1. Rainbow table keys (lowest priority) keysPath := os.Getenv("CHANNEL_KEYS_PATH") if keysPath == "" { keysPath = cfg.ChannelKeysPath } if keysPath == "" { - // Default: look for channel-rainbow.json next to config file keysPath = filepath.Join(filepath.Dir(configPath), "channel-rainbow.json") } + rainbowCount := 0 if data, err := os.ReadFile(keysPath); err == nil { var fileKeys map[string]string if err := json.Unmarshal(data, &fileKeys); err == nil { for k, v := range fileKeys { keys[k] = v } - log.Printf("Loaded %d channel keys from %s", len(fileKeys), keysPath) + rainbowCount = len(fileKeys) + log.Printf("Loaded %d channel keys from %s", rainbowCount, keysPath) } else { log.Printf("Warning: failed to parse channel keys file %s: %v", keysPath, err) } } - // Merge inline config keys (override file keys) + // 2. Derived keys from hashChannels (middle priority) + derivedCount := 0 + for _, raw := range cfg.HashChannels { + trimmed := strings.TrimSpace(raw) + if trimmed == "" { + continue + } + channelName := trimmed + if !strings.HasPrefix(channelName, "#") { + channelName = "#" + channelName + } + // Skip if explicit config already has this key + if _, exists := cfg.ChannelKeys[channelName]; exists { + continue + } + keys[channelName] = deriveHashtagChannelKey(channelName) + derivedCount++ + } + if derivedCount > 0 { + log.Printf("[channels] %d derived from hashChannels", derivedCount) + } + + // 3. Explicit config keys (highest priority — overrides rainbow + derived) for k, v := range cfg.ChannelKeys { keys[k] = v } @@ -546,7 +582,7 @@ var version = "dev" func init() { if len(os.Args) > 1 && os.Args[1] == "--version" { - fmt.Println("meshcore-ingestor", version) + fmt.Println("corescope-ingestor", version) os.Exit(0) } } diff --git a/cmd/ingestor/main_test.go b/cmd/ingestor/main_test.go index b9dab87f..e5540f15 100644 --- a/cmd/ingestor/main_test.go +++ b/cmd/ingestor/main_test.go @@ -3,6 +3,8 @@ package main import ( "encoding/json" "math" + "os" + "path/filepath" "testing" "time" ) @@ -492,3 +494,132 @@ func TestAdvertRole(t *testing.T) { }) } } + +func TestDeriveHashtagChannelKey(t *testing.T) { + // Test vectors validated against Node.js server-helpers.js + tests := []struct { + name string + want string + }{ + {"#General", "649af2cab73ed5a890890a5485a0c004"}, + {"#test", "9cd8fcf22a47333b591d96a2b848b73f"}, + {"#MeshCore", "dcf73f393fa217f6b28fcec6ffc411ad"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := deriveHashtagChannelKey(tt.name) + if got != tt.want { + t.Errorf("deriveHashtagChannelKey(%q) = %q, want %q", tt.name, got, tt.want) + } + }) + } + + // Deterministic + k1 := deriveHashtagChannelKey("#foo") + k2 := deriveHashtagChannelKey("#foo") + if k1 != k2 { + t.Error("deriveHashtagChannelKey should be deterministic") + } + + // Returns 32-char hex string (16 bytes) + if len(k1) != 32 { + t.Errorf("key length = %d, want 32", len(k1)) + } + + // Different inputs → different keys + k3 := deriveHashtagChannelKey("#bar") + if k1 == k3 { + t.Error("different inputs should produce different keys") + } +} + +func TestLoadChannelKeysMergePriority(t *testing.T) { + dir := t.TempDir() + cfgPath := filepath.Join(dir, "config.json") + + // Create a rainbow file with two keys: #rainbow (unique) and #override (to be overridden) + rainbowPath := filepath.Join(dir, "channel-rainbow.json") + t.Setenv("CHANNEL_KEYS_PATH", rainbowPath) + rainbow := map[string]string{ + "#rainbow": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "#override": "rainbow_value_should_be_overridden", + } + rainbowJSON, err := json.Marshal(rainbow) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(rainbowPath, rainbowJSON, 0o644); err != nil { + t.Fatal(err) + } + + cfg := &Config{ + HashChannels: []string{"General", "#override"}, + ChannelKeys: map[string]string{"#override": "explicit_wins"}, + } + + keys := loadChannelKeys(cfg, cfgPath) + + // Rainbow key loaded + if keys["#rainbow"] != "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" { + t.Errorf("rainbow key missing or wrong: %q", keys["#rainbow"]) + } + + // HashChannels derived #General + expected := deriveHashtagChannelKey("#General") + if keys["#General"] != expected { + t.Errorf("#General = %q, want %q (derived)", keys["#General"], expected) + } + + // Explicit config wins over both rainbow and derived + if keys["#override"] != "explicit_wins" { + t.Errorf("#override = %q, want explicit_wins", keys["#override"]) + } +} + +func TestLoadChannelKeysHashChannelsNormalization(t *testing.T) { + t.Setenv("CHANNEL_KEYS_PATH", "") + dir := t.TempDir() + cfgPath := filepath.Join(dir, "config.json") + + cfg := &Config{ + HashChannels: []string{ + "NoPound", // should become #NoPound + "#HasPound", // stays #HasPound + " Spaced ", // trimmed → #Spaced + "", // skipped + }, + } + + keys := loadChannelKeys(cfg, cfgPath) + + if _, ok := keys["#NoPound"]; !ok { + t.Error("should derive key for #NoPound (auto-prefixed)") + } + if _, ok := keys["#HasPound"]; !ok { + t.Error("should derive key for #HasPound") + } + if _, ok := keys["#Spaced"]; !ok { + t.Error("should derive key for #Spaced (trimmed)") + } + if len(keys) != 3 { + t.Errorf("expected 3 keys, got %d", len(keys)) + } +} + +func TestLoadChannelKeysSkipExplicit(t *testing.T) { + t.Setenv("CHANNEL_KEYS_PATH", "") + dir := t.TempDir() + cfgPath := filepath.Join(dir, "config.json") + + cfg := &Config{ + HashChannels: []string{"General"}, + ChannelKeys: map[string]string{"#General": "my_explicit_key"}, + } + + keys := loadChannelKeys(cfg, cfgPath) + + // Explicit key should win — hashChannels derivation should be skipped + if keys["#General"] != "my_explicit_key" { + t.Errorf("#General = %q, want my_explicit_key", keys["#General"]) + } +} diff --git a/cmd/server/coverage_test.go b/cmd/server/coverage_test.go index 700f8209..9cd0a718 100644 --- a/cmd/server/coverage_test.go +++ b/cmd/server/coverage_test.go @@ -26,7 +26,8 @@ func setupTestDBv2(t *testing.T) *DB { schema := ` CREATE TABLE nodes ( public_key TEXT PRIMARY KEY, name TEXT, role TEXT, - lat REAL, lon REAL, last_seen TEXT, first_seen TEXT, advert_count INTEGER DEFAULT 0 + lat REAL, lon REAL, last_seen TEXT, first_seen TEXT, advert_count INTEGER DEFAULT 0, + battery_mv INTEGER, temperature_c REAL ); CREATE TABLE observers ( id TEXT PRIMARY KEY, name TEXT, iata TEXT, last_seen TEXT, first_seen TEXT, @@ -45,14 +46,6 @@ func setupTestDBv2(t *testing.T) *DB { observer_id TEXT, observer_name TEXT, direction TEXT, snr REAL, rssi REAL, score INTEGER, path_json TEXT, timestamp INTEGER NOT NULL ); - CREATE VIEW packets_v AS - SELECT o.id, t.raw_hex, - strftime('%Y-%m-%dT%H:%M:%fZ', o.timestamp, 'unixepoch') AS timestamp, - o.observer_id, o.observer_name, - o.direction, o.snr, o.rssi, o.score, t.hash, t.route_type, - t.payload_type, t.payload_version, o.path_json, t.decoded_json, t.created_at - FROM observations o - JOIN transmissions t ON t.id = o.transmission_id; ` if _, err := conn.Exec(schema); err != nil { t.Fatal(err) @@ -550,8 +543,8 @@ func TestHandlePacketDetailNoStore(t *testing.T) { req := httptest.NewRequest("GET", "/api/packets/abc123def4567890", nil) w := httptest.NewRecorder() router.ServeHTTP(w, req) - if w.Code != 200 { - t.Fatalf("expected 200, got %d: %s", w.Code, w.Body.String()) + if w.Code != 404 { + t.Fatalf("expected 404 (no store), got %d: %s", w.Code, w.Body.String()) } }) @@ -559,8 +552,8 @@ func TestHandlePacketDetailNoStore(t *testing.T) { req := httptest.NewRequest("GET", "/api/packets/1", nil) w := httptest.NewRecorder() router.ServeHTTP(w, req) - if w.Code != 200 { - t.Fatalf("expected 200, got %d: %s", w.Code, w.Body.String()) + if w.Code != 404 { + t.Fatalf("expected 404 (no store), got %d: %s", w.Code, w.Body.String()) } }) @@ -1474,8 +1467,8 @@ func TestHandleObserverAnalyticsNoStore(t *testing.T) { req := httptest.NewRequest("GET", "/api/observers/obs1/analytics", nil) w := httptest.NewRecorder() router.ServeHTTP(w, req) - if w.Code != 200 { - t.Fatalf("expected 200, got %d: %s", w.Code, w.Body.String()) + if w.Code != 503 { + t.Fatalf("expected 503, got %d: %s", w.Code, w.Body.String()) } } @@ -3271,20 +3264,6 @@ func TestHandlePacketDetailWithStoreAllPaths(t *testing.T) { // --- Additional DB function coverage --- -func TestDBGetTimestamps(t *testing.T) { - db := setupTestDB(t) - defer db.Close() - seedTestData(t, db) - - ts, err := db.GetTimestamps("2000-01-01") - if err != nil { - t.Fatal(err) - } - if len(ts) < 1 { - t.Error("expected >=1 timestamps") - } -} - func TestDBGetNewTransmissionsSince(t *testing.T) { db := setupTestDB(t) defer db.Close() diff --git a/cmd/server/db.go b/cmd/server/db.go index 5be4073d..f96f41d6 100644 --- a/cmd/server/db.go +++ b/cmd/server/db.go @@ -120,14 +120,16 @@ func (db *DB) scanTransmissionRow(rows *sql.Rows) map[string]interface{} { // Node represents a row from the nodes table. type Node struct { - PublicKey string `json:"public_key"` - Name *string `json:"name"` - Role *string `json:"role"` - Lat *float64 `json:"lat"` - Lon *float64 `json:"lon"` - LastSeen *string `json:"last_seen"` - FirstSeen *string `json:"first_seen"` - AdvertCount int `json:"advert_count"` + PublicKey string `json:"public_key"` + Name *string `json:"name"` + Role *string `json:"role"` + Lat *float64 `json:"lat"` + Lon *float64 `json:"lon"` + LastSeen *string `json:"last_seen"` + FirstSeen *string `json:"first_seen"` + AdvertCount int `json:"advert_count"` + BatteryMv *int `json:"battery_mv"` + TemperatureC *float64 `json:"temperature_c"` } // Observer represents a row from the observers table. @@ -160,7 +162,7 @@ type Transmission struct { CreatedAt *string `json:"created_at"` } -// Observation (from packets_v view). +// Observation (observation-level data). type Observation struct { ID int `json:"id"` RawHex *string `json:"raw_hex"` @@ -433,7 +435,7 @@ func (db *DB) QueryGroupedPackets(q PacketQuery) (*PacketResult, error) { w = "WHERE " + strings.Join(where, " AND ") } - // Count total transmissions (fast — queries transmissions directly, not packets_v) + // Count total transmissions (fast — queries transmissions directly, not a VIEW) var total int if len(where) == 0 { db.conn.QueryRow("SELECT COUNT(*) FROM transmissions").Scan(&total) @@ -626,18 +628,6 @@ func (db *DB) resolveNodePubkey(nodeIDOrName string) string { return pk } -// GetPacketByID fetches a single packet/observation. -func (db *DB) GetPacketByID(id int) (map[string]interface{}, error) { - rows, err := db.conn.Query("SELECT id, raw_hex, timestamp, observer_id, observer_name, direction, snr, rssi, score, hash, route_type, payload_type, payload_version, path_json, decoded_json, created_at FROM packets_v WHERE id = ?", id) - if err != nil { - return nil, err - } - defer rows.Close() - if rows.Next() { - return scanPacketRow(rows), nil - } - return nil, nil -} // GetTransmissionByID fetches from transmissions table with observer data. func (db *DB) GetTransmissionByID(id int) (map[string]interface{}, error) { @@ -671,24 +661,6 @@ func (db *DB) GetPacketByHash(hash string) (map[string]interface{}, error) { return nil, nil } -// GetObservationsForHash returns all observations for a given hash. -func (db *DB) GetObservationsForHash(hash string) ([]map[string]interface{}, error) { - rows, err := db.conn.Query(`SELECT id, raw_hex, timestamp, observer_id, observer_name, direction, snr, rssi, score, hash, route_type, payload_type, payload_version, path_json, decoded_json, created_at - FROM packets_v WHERE hash = ? ORDER BY timestamp DESC`, strings.ToLower(hash)) - if err != nil { - return nil, err - } - defer rows.Close() - - result := make([]map[string]interface{}, 0) - for rows.Next() { - p := scanPacketRow(rows) - if p != nil { - result = append(result, p) - } - } - return result, nil -} // GetNodes returns filtered, paginated node list. func (db *DB) GetNodes(limit, offset int, role, search, before, lastHeard, sortBy, region string) ([]map[string]interface{}, int, map[string]int, error) { @@ -739,7 +711,7 @@ func (db *DB) GetNodes(limit, offset int, role, search, before, lastHeard, sortB var total int db.conn.QueryRow(fmt.Sprintf("SELECT COUNT(*) FROM nodes %s", w), args...).Scan(&total) - querySQL := fmt.Sprintf("SELECT public_key, name, role, lat, lon, last_seen, first_seen, advert_count FROM nodes %s ORDER BY %s LIMIT ? OFFSET ?", w, order) + querySQL := fmt.Sprintf("SELECT public_key, name, role, lat, lon, last_seen, first_seen, advert_count, battery_mv, temperature_c FROM nodes %s ORDER BY %s LIMIT ? OFFSET ?", w, order) qArgs := append(args, limit, offset) rows, err := db.conn.Query(querySQL, qArgs...) @@ -765,7 +737,7 @@ func (db *DB) SearchNodes(query string, limit int) ([]map[string]interface{}, er if limit <= 0 { limit = 10 } - rows, err := db.conn.Query(`SELECT public_key, name, role, lat, lon, last_seen, first_seen, advert_count + rows, err := db.conn.Query(`SELECT public_key, name, role, lat, lon, last_seen, first_seen, advert_count, battery_mv, temperature_c FROM nodes WHERE name LIKE ? OR public_key LIKE ? ORDER BY last_seen DESC LIMIT ?`, "%"+query+"%", query+"%", limit) if err != nil { @@ -785,7 +757,7 @@ func (db *DB) SearchNodes(query string, limit int) ([]map[string]interface{}, er // GetNodeByPubkey returns a single node. func (db *DB) GetNodeByPubkey(pubkey string) (map[string]interface{}, error) { - rows, err := db.conn.Query("SELECT public_key, name, role, lat, lon, last_seen, first_seen, advert_count FROM nodes WHERE public_key = ?", pubkey) + rows, err := db.conn.Query("SELECT public_key, name, role, lat, lon, last_seen, first_seen, advert_count, battery_mv, temperature_c FROM nodes WHERE public_key = ?", pubkey) if err != nil { return nil, err } @@ -796,30 +768,6 @@ func (db *DB) GetNodeByPubkey(pubkey string) (map[string]interface{}, error) { return nil, nil } -// GetRecentPacketsForNode returns recent packets referencing a node. -func (db *DB) GetRecentPacketsForNode(pubkey string, name string, limit int) ([]map[string]interface{}, error) { - if limit <= 0 { - limit = 20 - } - pk := "%" + pubkey + "%" - np := "%" + name + "%" - rows, err := db.conn.Query(`SELECT id, raw_hex, timestamp, observer_id, observer_name, direction, snr, rssi, score, hash, route_type, payload_type, payload_version, path_json, decoded_json, created_at - FROM packets_v WHERE decoded_json LIKE ? OR decoded_json LIKE ? - ORDER BY timestamp DESC LIMIT ?`, pk, np, limit) - if err != nil { - return nil, err - } - defer rows.Close() - - packets := make([]map[string]interface{}, 0) - for rows.Next() { - p := scanPacketRow(rows) - if p != nil { - packets = append(packets, p) - } - } - return packets, nil -} // GetRecentTransmissionsForNode returns recent transmissions referencing a node (Node.js-compatible shape). func (db *DB) GetRecentTransmissionsForNode(pubkey string, name string, limit int) ([]map[string]interface{}, error) { @@ -1043,103 +991,6 @@ func (db *DB) GetDistinctIATAs() ([]string, error) { return codes, nil } -// GetNodeHealth returns health info for a node (observers, stats, recent packets). -func (db *DB) GetNodeHealth(pubkey string) (map[string]interface{}, error) { - node, err := db.GetNodeByPubkey(pubkey) - if err != nil || node == nil { - return nil, err - } - - name := "" - if n, ok := node["name"]; ok && n != nil { - name = fmt.Sprintf("%v", n) - } - - pk := "%" + pubkey + "%" - np := "%" + name + "%" - whereClause := "decoded_json LIKE ? OR decoded_json LIKE ?" - if name == "" { - whereClause = "decoded_json LIKE ?" - np = pk - } - - todayStart := time.Now().UTC().Truncate(24 * time.Hour).Format(time.RFC3339) - - // Observers - observerSQL := fmt.Sprintf(`SELECT observer_id, observer_name, AVG(snr) as avgSnr, AVG(rssi) as avgRssi, COUNT(*) as packetCount - FROM packets_v WHERE (%s) AND observer_id IS NOT NULL GROUP BY observer_id ORDER BY packetCount DESC`, whereClause) - oRows, err := db.conn.Query(observerSQL, pk, np) - if err != nil { - return nil, err - } - defer oRows.Close() - - observers := make([]map[string]interface{}, 0) - for oRows.Next() { - var obsID, obsName sql.NullString - var avgSnr, avgRssi sql.NullFloat64 - var pktCount int - oRows.Scan(&obsID, &obsName, &avgSnr, &avgRssi, &pktCount) - observers = append(observers, map[string]interface{}{ - "observer_id": nullStr(obsID), - "observer_name": nullStr(obsName), - "avgSnr": nullFloat(avgSnr), - "avgRssi": nullFloat(avgRssi), - "packetCount": pktCount, - }) - } - - // Stats - var packetsToday, totalPackets int - db.conn.QueryRow(fmt.Sprintf("SELECT COUNT(*) FROM packets_v WHERE (%s) AND timestamp > ?", whereClause), pk, np, todayStart).Scan(&packetsToday) - db.conn.QueryRow(fmt.Sprintf("SELECT COUNT(*) FROM packets_v WHERE (%s)", whereClause), pk, np).Scan(&totalPackets) - - var avgSnr sql.NullFloat64 - db.conn.QueryRow(fmt.Sprintf("SELECT AVG(snr) FROM packets_v WHERE (%s)", whereClause), pk, np).Scan(&avgSnr) - - var lastHeard sql.NullString - db.conn.QueryRow(fmt.Sprintf("SELECT MAX(timestamp) FROM packets_v WHERE (%s)", whereClause), pk, np).Scan(&lastHeard) - - // Avg hops - hRows, _ := db.conn.Query(fmt.Sprintf("SELECT path_json FROM packets_v WHERE (%s) AND path_json IS NOT NULL", whereClause), pk, np) - totalHops, hopCount := 0, 0 - if hRows != nil { - defer hRows.Close() - for hRows.Next() { - var pj sql.NullString - hRows.Scan(&pj) - if pj.Valid { - var hops []interface{} - if json.Unmarshal([]byte(pj.String), &hops) == nil { - totalHops += len(hops) - hopCount++ - } - } - } - } - avgHops := 0 - if hopCount > 0 { - avgHops = int(math.Round(float64(totalHops) / float64(hopCount))) - } - - // Recent packets - recentPackets, _ := db.GetRecentTransmissionsForNode(pubkey, name, 20) - - return map[string]interface{}{ - "node": node, - "observers": observers, - "stats": map[string]interface{}{ - "totalTransmissions": totalPackets, - "totalObservations": totalPackets, - "totalPackets": totalPackets, - "packetsToday": packetsToday, - "avgSnr": nullFloat(avgSnr), - "avgHops": avgHops, - "lastHeard": nullStr(lastHeard), - }, - "recentPackets": recentPackets, - }, nil -} // GetNetworkStatus returns overall network health status. func (db *DB) GetNetworkStatus(healthThresholds HealthThresholds) (map[string]interface{}, error) { @@ -1188,10 +1039,28 @@ func (db *DB) GetNetworkStatus(healthThresholds HealthThresholds) (map[string]in }, nil } -// GetTraces returns observations for a hash. +// GetTraces returns observations for a hash using direct table queries. func (db *DB) GetTraces(hash string) ([]map[string]interface{}, error) { - rows, err := db.conn.Query(`SELECT observer_id, observer_name, timestamp, snr, rssi, path_json - FROM packets_v WHERE hash = ? ORDER BY timestamp ASC`, strings.ToLower(hash)) + var querySQL string + if db.isV3 { + querySQL = `SELECT obs.id AS observer_id, obs.name AS observer_name, + strftime('%Y-%m-%dT%H:%M:%fZ', o.timestamp, 'unixepoch') AS timestamp, + o.snr, o.rssi, o.path_json + FROM observations o + JOIN transmissions t ON t.id = o.transmission_id + LEFT JOIN observers obs ON obs.rowid = o.observer_idx + WHERE t.hash = ? + ORDER BY o.timestamp ASC` + } else { + querySQL = `SELECT o.observer_id, o.observer_name, + strftime('%Y-%m-%dT%H:%M:%fZ', o.timestamp, 'unixepoch') AS timestamp, + o.snr, o.rssi, o.path_json + FROM observations o + JOIN transmissions t ON t.id = o.transmission_id + WHERE t.hash = ? + ORDER BY o.timestamp ASC` + } + rows, err := db.conn.Query(querySQL, strings.ToLower(hash)) if err != nil { return nil, err } @@ -1217,7 +1086,7 @@ func (db *DB) GetTraces(hash string) ([]map[string]interface{}, error) { } // GetChannels returns channel list from GRP_TXT packets. -// Queries transmissions directly (not packets_v) to avoid observation-level +// Queries transmissions directly (not a VIEW) to avoid observation-level // duplicates that could cause stale lastMessage when an older message has // a later re-observation timestamp. func (db *DB) GetChannels() ([]map[string]interface{}, error) { @@ -1433,31 +1302,7 @@ func (db *DB) GetChannelMessages(channelHash string, limit, offset int) ([]map[s return messages, total, nil } -// GetTimestamps returns packet timestamps since a given time. -func (db *DB) GetTimestamps(since string) ([]string, error) { - rows, err := db.conn.Query("SELECT timestamp FROM packets_v WHERE timestamp > ? ORDER BY timestamp ASC", since) - if err != nil { - return nil, err - } - defer rows.Close() - var timestamps []string - for rows.Next() { - var ts string - rows.Scan(&ts) - timestamps = append(timestamps, ts) - } - if timestamps == nil { - timestamps = []string{} - } - return timestamps, nil -} -// GetNodeCountsForPacket returns observation count for a hash. -func (db *DB) GetObservationCount(hash string) int { - var count int - db.conn.QueryRow("SELECT COUNT(*) FROM packets_v WHERE hash = ?", strings.ToLower(hash)).Scan(&count) - return count -} // GetNewTransmissionsSince returns new transmissions after a given ID for WebSocket polling. func (db *DB) GetNewTransmissionsSince(lastID int, limit int) ([]map[string]interface{}, error) { @@ -1658,11 +1503,13 @@ func scanNodeRow(rows *sql.Rows) map[string]interface{} { var name, role, lastSeen, firstSeen sql.NullString var lat, lon sql.NullFloat64 var advertCount int + var batteryMv sql.NullInt64 + var temperatureC sql.NullFloat64 - if err := rows.Scan(&pk, &name, &role, &lat, &lon, &lastSeen, &firstSeen, &advertCount); err != nil { + if err := rows.Scan(&pk, &name, &role, &lat, &lon, &lastSeen, &firstSeen, &advertCount, &batteryMv, &temperatureC); err != nil { return nil } - return map[string]interface{}{ + m := map[string]interface{}{ "public_key": pk, "name": nullStr(name), "role": nullStr(role), @@ -1675,6 +1522,17 @@ func scanNodeRow(rows *sql.Rows) map[string]interface{} { "hash_size": nil, "hash_size_inconsistent": false, } + if batteryMv.Valid { + m["battery_mv"] = int(batteryMv.Int64) + } else { + m["battery_mv"] = nil + } + if temperatureC.Valid { + m["temperature_c"] = temperatureC.Float64 + } else { + m["temperature_c"] = nil + } + return m } func nullStr(ns sql.NullString) interface{} { diff --git a/cmd/server/db_test.go b/cmd/server/db_test.go index 66cb27dc..05fa7855 100644 --- a/cmd/server/db_test.go +++ b/cmd/server/db_test.go @@ -28,7 +28,9 @@ func setupTestDB(t *testing.T) *DB { lon REAL, last_seen TEXT, first_seen TEXT, - advert_count INTEGER DEFAULT 0 + advert_count INTEGER DEFAULT 0, + battery_mv INTEGER, + temperature_c REAL ); CREATE TABLE observers ( @@ -71,16 +73,6 @@ func setupTestDB(t *testing.T) *DB { timestamp INTEGER NOT NULL ); - CREATE VIEW packets_v AS - SELECT o.id, t.raw_hex, - strftime('%Y-%m-%dT%H:%M:%fZ', o.timestamp, 'unixepoch') AS timestamp, - obs.id AS observer_id, obs.name AS observer_name, - o.direction, o.snr, o.rssi, o.score, t.hash, t.route_type, - t.payload_type, t.payload_version, o.path_json, t.decoded_json, - t.created_at - FROM observations o - JOIN transmissions t ON t.id = o.transmission_id - LEFT JOIN observers obs ON obs.rowid = o.observer_idx; ` if _, err := conn.Exec(schema); err != nil { t.Fatal(err) @@ -567,51 +559,6 @@ func TestGetNewTransmissionsSince(t *testing.T) { } } -func TestGetObservationsForHash(t *testing.T) { - db := setupTestDB(t) - defer db.Close() - seedTestData(t, db) - - obs, err := db.GetObservationsForHash("abc123def4567890") - if err != nil { - t.Fatal(err) - } - if len(obs) != 2 { - t.Errorf("expected 2 observations, got %d", len(obs)) - } -} - -func TestGetPacketByIDFound(t *testing.T) { - db := setupTestDB(t) - defer db.Close() - seedTestData(t, db) - - pkt, err := db.GetPacketByID(1) - if err != nil { - t.Fatal(err) - } - if pkt == nil { - t.Fatal("expected packet, got nil") - } - if pkt["hash"] != "abc123def4567890" { - t.Errorf("expected hash abc123def4567890, got %v", pkt["hash"]) - } -} - -func TestGetPacketByIDNotFound(t *testing.T) { - db := setupTestDB(t) - defer db.Close() - seedTestData(t, db) - - pkt, err := db.GetPacketByID(9999) - if err != nil { - t.Fatal(err) - } - if pkt != nil { - t.Error("expected nil for nonexistent packet ID") - } -} - func TestGetTransmissionByIDFound(t *testing.T) { db := setupTestDB(t) defer db.Close() @@ -654,34 +601,6 @@ func TestGetPacketByHashNotFound(t *testing.T) { } } -func TestGetRecentPacketsForNode(t *testing.T) { - db := setupTestDB(t) - defer db.Close() - seedTestData(t, db) - - packets, err := db.GetRecentPacketsForNode("aabbccdd11223344", "TestRepeater", 20) - if err != nil { - t.Fatal(err) - } - if len(packets) == 0 { - t.Error("expected packets for TestRepeater") - } -} - -func TestGetRecentPacketsForNodeDefaultLimit(t *testing.T) { - db := setupTestDB(t) - defer db.Close() - seedTestData(t, db) - - packets, err := db.GetRecentPacketsForNode("aabbccdd11223344", "TestRepeater", 0) - if err != nil { - t.Fatal(err) - } - if packets == nil { - t.Error("expected non-nil result") - } -} - func TestGetObserverIdsForRegion(t *testing.T) { db := setupTestDB(t) defer db.Close() @@ -731,46 +650,6 @@ func TestGetObserverIdsForRegion(t *testing.T) { }) } -func TestGetNodeHealth(t *testing.T) { - db := setupTestDB(t) - defer db.Close() - seedTestData(t, db) - - t.Run("found", func(t *testing.T) { - result, err := db.GetNodeHealth("aabbccdd11223344") - if err != nil { - t.Fatal(err) - } - if result == nil { - t.Fatal("expected result, got nil") - } - node, ok := result["node"].(map[string]interface{}) - if !ok { - t.Fatal("expected node object") - } - if node["name"] != "TestRepeater" { - t.Errorf("expected TestRepeater, got %v", node["name"]) - } - stats, ok := result["stats"].(map[string]interface{}) - if !ok { - t.Fatal("expected stats object") - } - if stats["totalPackets"] == nil { - t.Error("expected totalPackets in stats") - } - }) - - t.Run("not found", func(t *testing.T) { - result, err := db.GetNodeHealth("nonexistent") - if err != nil { - t.Fatal(err) - } - if result != nil { - t.Error("expected nil for nonexistent node") - } - }) -} - func TestGetChannelMessages(t *testing.T) { db := setupTestDB(t) defer db.Close() @@ -813,48 +692,6 @@ func TestGetChannelMessages(t *testing.T) { }) } -func TestGetTimestamps(t *testing.T) { - db := setupTestDB(t) - defer db.Close() - seedTestData(t, db) - - t.Run("with results", func(t *testing.T) { - ts, err := db.GetTimestamps("2020-01-01") - if err != nil { - t.Fatal(err) - } - if len(ts) == 0 { - t.Error("expected timestamps") - } - }) - - t.Run("no results", func(t *testing.T) { - ts, err := db.GetTimestamps("2099-01-01") - if err != nil { - t.Fatal(err) - } - if len(ts) != 0 { - t.Errorf("expected 0 timestamps, got %d", len(ts)) - } - }) -} - -func TestGetObservationCount(t *testing.T) { - db := setupTestDB(t) - defer db.Close() - seedTestData(t, db) - - count := db.GetObservationCount("abc123def4567890") - if count != 2 { - t.Errorf("expected 2, got %d", count) - } - - count = db.GetObservationCount("nonexistent") - if count != 0 { - t.Errorf("expected 0 for nonexistent, got %d", count) - } -} - func TestBuildPacketWhereFilters(t *testing.T) { db := setupTestDB(t) defer db.Close() @@ -1278,29 +1115,6 @@ func TestOpenDBInvalidPath(t *testing.T) { } } -func TestGetNodeHealthNoName(t *testing.T) { - db := setupTestDB(t) - defer db.Close() - - // Insert a node without a name - db.conn.Exec(`INSERT INTO observers (id, name, iata) VALUES ('obs1', 'Observer One', 'SJC')`) - db.conn.Exec(`INSERT INTO nodes (public_key, role, last_seen, first_seen, advert_count) - VALUES ('deadbeef12345678', 'repeater', '2026-01-15T10:00:00Z', '2026-01-01T00:00:00Z', 5)`) - db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json) - VALUES ('DDEE', 'deadbeefhash1234', '2026-01-15T10:05:00Z', 1, 4, - '{"pubKey":"deadbeef12345678","type":"ADVERT"}')`) - db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp) - VALUES (1, 1, 11.0, -91, '["dd"]', 1736935500)`) - - result, err := db.GetNodeHealth("deadbeef12345678") - if err != nil { - t.Fatal(err) - } - if result == nil { - t.Fatal("expected result, got nil") - } -} - func TestGetChannelMessagesObserverFallback(t *testing.T) { db := setupTestDB(t) defer db.Close() @@ -1381,20 +1195,6 @@ func TestQueryGroupedPacketsWithFilters(t *testing.T) { } } -func TestGetTracesEmpty(t *testing.T) { - db := setupTestDB(t) - defer db.Close() - seedTestData(t, db) - - traces, err := db.GetTraces("nonexistenthash1") - if err != nil { - t.Fatal(err) - } - if len(traces) != 0 { - t.Errorf("expected 0 traces, got %d", len(traces)) - } -} - func TestNullHelpers(t *testing.T) { // nullStr if nullStr(sql.NullString{Valid: false}) != nil { @@ -1468,6 +1268,53 @@ func TestGetChannelsStaleMessage(t *testing.T) { } } +func TestNodeTelemetryFields(t *testing.T) { + db := setupTestDB(t) + defer db.Close() + + // Insert node with telemetry data + db.conn.Exec(`INSERT INTO nodes (public_key, name, role, lat, lon, last_seen, first_seen, advert_count, battery_mv, temperature_c) + VALUES ('pk_telem1', 'SensorNode', 'sensor', 37.0, -122.0, '2026-01-01T00:00:00Z', '2026-01-01T00:00:00Z', 5, 3700, 28.5)`) + + // Test via GetNodeByPubkey + node, err := db.GetNodeByPubkey("pk_telem1") + if err != nil { + t.Fatal(err) + } + if node == nil { + t.Fatal("expected node, got nil") + } + if node["battery_mv"] != 3700 { + t.Errorf("battery_mv=%v, want 3700", node["battery_mv"]) + } + if node["temperature_c"] != 28.5 { + t.Errorf("temperature_c=%v, want 28.5", node["temperature_c"]) + } + + // Test via GetNodes + nodes, _, _, err := db.GetNodes(50, 0, "sensor", "", "", "", "", "") + if err != nil { + t.Fatal(err) + } + if len(nodes) != 1 { + t.Fatalf("expected 1 sensor node, got %d", len(nodes)) + } + if nodes[0]["battery_mv"] != 3700 { + t.Errorf("GetNodes battery_mv=%v, want 3700", nodes[0]["battery_mv"]) + } + + // Test node without telemetry — fields should be nil + db.conn.Exec(`INSERT INTO nodes (public_key, name, role, last_seen, first_seen, advert_count) + VALUES ('pk_notelem', 'PlainNode', 'repeater', '2026-01-01T00:00:00Z', '2026-01-01T00:00:00Z', 3)`) + node2, _ := db.GetNodeByPubkey("pk_notelem") + if node2["battery_mv"] != nil { + t.Errorf("expected nil battery_mv for node without telemetry, got %v", node2["battery_mv"]) + } + if node2["temperature_c"] != nil { + t.Errorf("expected nil temperature_c for node without telemetry, got %v", node2["temperature_c"]) + } +} + func TestMain(m *testing.M) { os.Exit(m.Run()) } diff --git a/cmd/server/go.mod b/cmd/server/go.mod index 55343edb..1ef2c8af 100644 --- a/cmd/server/go.mod +++ b/cmd/server/go.mod @@ -1,4 +1,4 @@ -module github.com/meshcore-analyzer/server +module github.com/corescope/server go 1.22 diff --git a/cmd/server/main.go b/cmd/server/main.go index e8d2d8b4..b2e859f3 100644 --- a/cmd/server/main.go +++ b/cmd/server/main.go @@ -116,7 +116,7 @@ func main() { var tableName string err = database.conn.QueryRow("SELECT name FROM sqlite_master WHERE type='table' AND name='transmissions'").Scan(&tableName) if err == sql.ErrNoRows { - log.Fatalf("[db] table 'transmissions' not found — is this a MeshCore Analyzer database?") + log.Fatalf("[db] table 'transmissions' not found — is this a CoreScope database?") } stats, err := database.GetStats() @@ -155,7 +155,7 @@ func main() { log.Printf("[static] directory %s not found — API-only mode", absPublic) router.PathPrefix("/").HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.Header().Set("Content-Type", "text/html") - w.Write([]byte(`
Frontend not found. API available at /api/
`)) + w.Write([]byte(`Frontend not found. API available at /api/
`)) }) } @@ -182,7 +182,7 @@ func main() { httpServer.Close() }() - log.Printf("[server] MeshCore Analyzer (Go) listening on http://localhost:%d", cfg.Port) + log.Printf("[server] CoreScope (Go) listening on http://localhost:%d", cfg.Port) if err := httpServer.ListenAndServe(); err != http.ErrServerClosed { log.Fatalf("[server] %v", err) } diff --git a/cmd/server/routes.go b/cmd/server/routes.go index b1c84c8c..14d1c9dd 100644 --- a/cmd/server/routes.go +++ b/cmd/server/routes.go @@ -240,7 +240,7 @@ func (s *Server) handleConfigTheme(w http.ResponseWriter, r *http.Request) { theme := LoadTheme(".") branding := mergeMap(map[string]interface{}{ - "siteName": "MeshCore Analyzer", + "siteName": "CoreScope", "tagline": "Real-time MeshCore LoRa mesh network analyzer", }, s.cfg.Branding, theme.Branding) @@ -626,12 +626,7 @@ func (s *Server) handlePacketTimestamps(w http.ResponseWriter, r *http.Request) writeJSON(w, s.store.GetTimestamps(since)) return } - ts, err := s.db.GetTimestamps(since) - if err != nil { - writeError(w, 500, err.Error()) - return - } - writeJSON(w, ts) + writeJSON(w, []string{}) } var hashPattern = regexp.MustCompile(`^[0-9a-f]{16}$`) @@ -645,10 +640,8 @@ var perfHexFallback = regexp.MustCompile(`[0-9a-f]{8,}`) func (s *Server) handlePacketDetail(w http.ResponseWriter, r *http.Request) { param := mux.Vars(r)["id"] var packet map[string]interface{} - var err error if s.store != nil { - // Use in-memory store for lookups if hashPattern.MatchString(strings.ToLower(param)) { packet = s.store.GetPacketByHash(param) } @@ -661,40 +654,22 @@ func (s *Server) handlePacketDetail(w http.ResponseWriter, r *http.Request) { } } } - } else { - // Fallback to DB - if hashPattern.MatchString(strings.ToLower(param)) { - packet, err = s.db.GetPacketByHash(param) - } - if packet == nil { - id, parseErr := strconv.Atoi(param) - if parseErr == nil { - packet, err = s.db.GetTransmissionByID(id) - if packet == nil { - packet, err = s.db.GetPacketByID(id) - } - } - } } - if err != nil || packet == nil { + if packet == nil { writeError(w, 404, "Not found") return } - // Build observation list hash, _ := packet["hash"].(string) var observations []map[string]interface{} if s.store != nil { observations = s.store.GetObservationsForHash(hash) - } else { - observations, _ = s.db.GetObservationsForHash(hash) } observationCount := len(observations) if observationCount == 0 { observationCount = 1 } - // Parse path from path_json var pathHops []interface{} if pj, ok := packet["path_json"]; ok && pj != nil { if pjStr, ok := pj.(string); ok && pjStr != "" { @@ -876,18 +851,16 @@ func (s *Server) handleNodeDetail(w http.ResponseWriter, r *http.Request) { func (s *Server) handleNodeHealth(w http.ResponseWriter, r *http.Request) { pubkey := mux.Vars(r)["pubkey"] - var result map[string]interface{} - var err error if s.store != nil { - result, err = s.store.GetNodeHealth(pubkey) - } else { - result, err = s.db.GetNodeHealth(pubkey) - } - if err != nil || result == nil { - writeError(w, 404, "Not found") + result, err := s.store.GetNodeHealth(pubkey) + if err != nil || result == nil { + writeError(w, 404, "Not found") + return + } + writeJSON(w, result) return } - writeJSON(w, result) + writeError(w, 404, "Not found") } func (s *Server) handleBulkHealth(w http.ResponseWriter, r *http.Request) { @@ -902,118 +875,7 @@ func (s *Server) handleBulkHealth(w http.ResponseWriter, r *http.Request) { return } - rows, err := s.db.conn.Query("SELECT public_key, name, role, lat, lon, last_seen FROM nodes ORDER BY last_seen DESC LIMIT ?", limit) - if err != nil { - writeError(w, 500, err.Error()) - return - } - defer rows.Close() - - type nodeDbInfo struct { - pk, name, role, lastSeen string - lat, lon interface{} - } - var nodes []nodeDbInfo - for rows.Next() { - var pk string - var name, role, lastSeen sql.NullString - var lat, lon sql.NullFloat64 - rows.Scan(&pk, &name, &role, &lat, &lon, &lastSeen) - nodes = append(nodes, nodeDbInfo{ - pk: pk, name: nullStrVal(name), role: nullStrVal(role), - lastSeen: nullStrVal(lastSeen), - lat: nullFloat(lat), lon: nullFloat(lon), - }) - } - - // Batch query: per-node transmission stats - todayStart := time.Now().UTC().Truncate(24 * time.Hour).Format(time.RFC3339) - results := make([]BulkHealthEntry, 0, len(nodes)) - for _, n := range nodes { - pk := "%" + n.pk + "%" - np := "%" + n.name + "%" - - var txCount, obsCount, packetsToday int - var avgSnr sql.NullFloat64 - var lastHeard sql.NullString - - whereClause := "t.decoded_json LIKE ?" - queryArgs := []interface{}{pk} - if n.name != "" { - whereClause = "(t.decoded_json LIKE ? OR t.decoded_json LIKE ?)" - queryArgs = []interface{}{pk, np} - } - - s.db.conn.QueryRow(fmt.Sprintf("SELECT COUNT(*) FROM transmissions t WHERE %s", whereClause), queryArgs...).Scan(&txCount) - - if txCount > 0 { - // Observation count - s.db.conn.QueryRow(fmt.Sprintf(`SELECT COALESCE(SUM( - (SELECT COUNT(*) FROM observations oi WHERE oi.transmission_id = t.id) - ), 0) FROM transmissions t WHERE %s`, whereClause), queryArgs...).Scan(&obsCount) - - // Packets today - todayArgs := append(queryArgs, todayStart) - s.db.conn.QueryRow(fmt.Sprintf("SELECT COUNT(*) FROM transmissions t WHERE %s AND t.first_seen > ?", whereClause), todayArgs...).Scan(&packetsToday) - - // Avg SNR from best observation per transmission - s.db.conn.QueryRow(fmt.Sprintf(`SELECT AVG(o.snr) FROM transmissions t - LEFT JOIN observations o ON o.id = ( - SELECT id FROM observations WHERE transmission_id = t.id AND snr IS NOT NULL LIMIT 1 - ) WHERE %s AND o.snr IS NOT NULL`, whereClause), queryArgs...).Scan(&avgSnr) - - // Last heard - s.db.conn.QueryRow(fmt.Sprintf("SELECT MAX(t.first_seen) FROM transmissions t WHERE %s", whereClause), queryArgs...).Scan(&lastHeard) - } - - lh := n.lastSeen - if lastHeard.Valid && lastHeard.String > lh { - lh = lastHeard.String - } - - // Per-observer breakdown - pvWhere := "pv.decoded_json LIKE ?" - if n.name != "" { - pvWhere = "(pv.decoded_json LIKE ? OR pv.decoded_json LIKE ?)" - } - obsSQL := fmt.Sprintf(`SELECT pv.observer_id, pv.observer_name, AVG(pv.snr) as avgSnr, AVG(pv.rssi) as avgRssi, COUNT(*) as packetCount - FROM packets_v pv WHERE %s AND pv.observer_id IS NOT NULL - GROUP BY pv.observer_id ORDER BY packetCount DESC`, pvWhere) - obsRows, _ := s.db.conn.Query(obsSQL, queryArgs...) - observers := make([]NodeObserverStatsResp, 0) - if obsRows != nil { - for obsRows.Next() { - var oID, oName sql.NullString - var oSnr, oRssi sql.NullFloat64 - var oPktCount int - obsRows.Scan(&oID, &oName, &oSnr, &oRssi, &oPktCount) - observers = append(observers, NodeObserverStatsResp{ - ObserverID: nullStr(oID), ObserverName: nullStr(oName), - AvgSnr: nullFloat(oSnr), AvgRssi: nullFloat(oRssi), - PacketCount: oPktCount, - }) - } - obsRows.Close() - } - - results = append(results, BulkHealthEntry{ - PublicKey: n.pk, - Name: nilIfEmpty(n.name), - Role: nilIfEmpty(n.role), - Lat: n.lat, - Lon: n.lon, - Stats: NodeStatsResp{ - TotalTransmissions: txCount, - TotalObservations: obsCount, - TotalPackets: txCount, - PacketsToday: packetsToday, - AvgSnr: nullFloat(avgSnr), - LastHeard: nilIfEmpty(lh), - }, - Observers: observers, - }) - } - writeJSON(w, results) + writeJSON(w, []BulkHealthEntry{}) } func (s *Server) handleNetworkStatus(w http.ResponseWriter, r *http.Request) { @@ -1033,49 +895,123 @@ func (s *Server) handleNodePaths(w http.ResponseWriter, r *http.Request) { writeError(w, 404, "Not found") return } - - pk := "%" + pubkey[:8] + "%" - name := "" - if n, ok := node["name"]; ok && n != nil { - name = fmt.Sprintf("%v", n) + if s.store == nil { + writeError(w, 503, "Packet store unavailable") + return } - whereClause := "path_json LIKE ?" - args := []interface{}{pk} - if name != "" { - whereClause = "(path_json LIKE ? OR path_json LIKE ?)" - args = append(args, "%"+name+"%") + prefix1 := strings.ToLower(pubkey) + if len(prefix1) > 2 { + prefix1 = prefix1[:2] } - - pathSQL := fmt.Sprintf("SELECT path_json, hash, MAX(timestamp) as lastSeen, COUNT(*) as cnt FROM packets_v WHERE path_json IS NOT NULL AND path_json != '[]' AND %s GROUP BY path_json ORDER BY cnt DESC LIMIT 50", whereClause) - rows, _ := s.db.conn.Query(pathSQL, args...) - - paths := make([]PathEntryResp, 0) - var totalPaths, totalTx int - if rows != nil { - defer rows.Close() - for rows.Next() { - var pj, hash string - var lastSeen sql.NullString - var cnt int - rows.Scan(&pj, &hash, &lastSeen, &cnt) - var hops []string - if json.Unmarshal([]byte(pj), &hops) != nil { - continue - } - hopEntries := make([]PathHopResp, 0, len(hops)) - for _, h := range hops { - hopEntries = append(hopEntries, PathHopResp{ - Prefix: h, Name: h, Pubkey: nil, Lat: nil, Lon: nil, - }) - } - paths = append(paths, PathEntryResp{ - Hops: hopEntries, Count: cnt, - LastSeen: nullStr(lastSeen), SampleHash: hash, - }) - totalPaths++ - totalTx += cnt + prefix2 := strings.ToLower(pubkey) + if len(prefix2) > 4 { + prefix2 = prefix2[:4] + } + s.store.mu.RLock() + _, pm := s.store.getCachedNodesAndPM() + type pathAgg struct { + Hops []PathHopResp + Count int + LastSeen string + SampleHash string + } + pathGroups := map[string]*pathAgg{} + totalTransmissions := 0 + hopCache := make(map[string]*nodeInfo) + resolveHop := func(hop string) *nodeInfo { + if cached, ok := hopCache[hop]; ok { + return cached } + r := pm.resolve(hop) + hopCache[hop] = r + return r + } + for _, tx := range s.store.packets { + hops := txGetParsedPath(tx) + if len(hops) == 0 { + continue + } + found := false + for _, hop := range hops { + hl := strings.ToLower(hop) + if hl == prefix1 || hl == prefix2 || strings.HasPrefix(hl, prefix2) { + found = true + break + } + } + if !found { + continue + } + + totalTransmissions++ + resolvedHops := make([]PathHopResp, len(hops)) + sigParts := make([]string, len(hops)) + for i, hop := range hops { + resolved := resolveHop(hop) + entry := PathHopResp{Prefix: hop, Name: hop} + if resolved != nil { + entry.Name = resolved.Name + entry.Pubkey = resolved.PublicKey + if resolved.HasGPS { + entry.Lat = resolved.Lat + entry.Lon = resolved.Lon + } + sigParts[i] = resolved.PublicKey + } else { + sigParts[i] = hop + } + resolvedHops[i] = entry + } + + sig := strings.Join(sigParts, "→") + agg := pathGroups[sig] + if agg == nil { + pathGroups[sig] = &pathAgg{ + Hops: resolvedHops, + Count: 1, + LastSeen: tx.FirstSeen, + SampleHash: tx.Hash, + } + continue + } + agg.Count++ + if tx.FirstSeen > agg.LastSeen { + agg.LastSeen = tx.FirstSeen + agg.SampleHash = tx.Hash + } + } + s.store.mu.RUnlock() + + paths := make([]PathEntryResp, 0, len(pathGroups)) + for _, agg := range pathGroups { + var lastSeen interface{} + if agg.LastSeen != "" { + lastSeen = agg.LastSeen + } + paths = append(paths, PathEntryResp{ + Hops: agg.Hops, + Count: agg.Count, + LastSeen: lastSeen, + SampleHash: agg.SampleHash, + }) + } + sort.Slice(paths, func(i, j int) bool { + if paths[i].Count == paths[j].Count { + li := "" + lj := "" + if paths[i].LastSeen != nil { + li = fmt.Sprintf("%v", paths[i].LastSeen) + } + if paths[j].LastSeen != nil { + lj = fmt.Sprintf("%v", paths[j].LastSeen) + } + return li > lj + } + return paths[i].Count > paths[j].Count + }) + if len(paths) > 50 { + paths = paths[:50] } writeJSON(w, NodePathsResponse{ @@ -1086,8 +1022,8 @@ func (s *Server) handleNodePaths(w http.ResponseWriter, r *http.Request) { "lon": node["lon"], }, Paths: paths, - TotalPaths: totalPaths, - TotalTransmissions: totalTx, + TotalPaths: len(pathGroups), + TotalTransmissions: totalTransmissions, }) } @@ -1101,7 +1037,6 @@ func (s *Server) handleNodeAnalytics(w http.ResponseWriter, r *http.Request) { days = 365 } - // Use in-memory store when available (fast path) if s.store != nil { result, err := s.store.GetNodeAnalytics(pubkey, days) if err != nil || result == nil { @@ -1112,471 +1047,43 @@ func (s *Server) handleNodeAnalytics(w http.ResponseWriter, r *http.Request) { return } - // Fallback: SQL path (no in-memory store) - node, err := s.db.GetNodeByPubkey(pubkey) - if err != nil || node == nil { - writeError(w, 404, "Not found") - return - } - - name := "" - if n, ok := node["name"]; ok && n != nil { - name = fmt.Sprintf("%v", n) - } - - fromISO := time.Now().Add(-time.Duration(days) * 24 * time.Hour).Format(time.RFC3339) - toISO := time.Now().Format(time.RFC3339) - - pk := "%" + pubkey + "%" - np := "%" + name + "%" - whereClause := "decoded_json LIKE ? OR decoded_json LIKE ?" - if name == "" { - whereClause = "decoded_json LIKE ?" - np = pk - } - timeWhere := fmt.Sprintf("(%s) AND timestamp > ?", whereClause) - - // Activity timeline - actSQL := fmt.Sprintf(`SELECT substr(timestamp, 1, 13) || ':00:00Z' as bucket, COUNT(*) as count - FROM packets_v WHERE %s GROUP BY bucket ORDER BY bucket`, timeWhere) - aRows, _ := s.db.conn.Query(actSQL, pk, np, fromISO) - activityTimeline := make([]TimeBucket, 0) - if aRows != nil { - defer aRows.Close() - for aRows.Next() { - var bucket string - var count int - aRows.Scan(&bucket, &count) - b := bucket - activityTimeline = append(activityTimeline, TimeBucket{Bucket: &b, Count: count}) - } - } - - // SNR trend - snrSQL := fmt.Sprintf(`SELECT timestamp, snr, rssi, observer_id, observer_name - FROM packets_v WHERE %s AND snr IS NOT NULL ORDER BY timestamp`, timeWhere) - sRows, _ := s.db.conn.Query(snrSQL, pk, np, fromISO) - snrTrend := make([]SnrTrendEntry, 0) - if sRows != nil { - defer sRows.Close() - for sRows.Next() { - var ts string - var snr, rssi sql.NullFloat64 - var obsID, obsName sql.NullString - sRows.Scan(&ts, &snr, &rssi, &obsID, &obsName) - snrTrend = append(snrTrend, SnrTrendEntry{ - Timestamp: ts, SNR: nullFloat(snr), RSSI: nullFloat(rssi), - ObserverID: nullStr(obsID), ObserverName: nullStr(obsName), - }) - } - } - - // Packet type breakdown - ptSQL := fmt.Sprintf("SELECT payload_type, COUNT(*) as count FROM packets_v WHERE %s GROUP BY payload_type", timeWhere) - ptRows, _ := s.db.conn.Query(ptSQL, pk, np, fromISO) - packetTypeBreakdown := make([]PayloadTypeCount, 0) - if ptRows != nil { - defer ptRows.Close() - for ptRows.Next() { - var pt, count int - ptRows.Scan(&pt, &count) - packetTypeBreakdown = append(packetTypeBreakdown, PayloadTypeCount{PayloadType: pt, Count: count}) - } - } - - // Observer coverage - ocSQL := fmt.Sprintf(`SELECT observer_id, observer_name, COUNT(*) as packetCount, - AVG(snr) as avgSnr, AVG(rssi) as avgRssi, MIN(timestamp) as firstSeen, MAX(timestamp) as lastSeen - FROM packets_v WHERE %s AND observer_id IS NOT NULL - GROUP BY observer_id ORDER BY packetCount DESC`, timeWhere) - ocRows, _ := s.db.conn.Query(ocSQL, pk, np, fromISO) - observerCoverage := make([]NodeObserverStatsResp, 0) - if ocRows != nil { - defer ocRows.Close() - for ocRows.Next() { - var obsID, obsName, first, last sql.NullString - var pktCount int - var avgSnr, avgRssi sql.NullFloat64 - ocRows.Scan(&obsID, &obsName, &pktCount, &avgSnr, &avgRssi, &first, &last) - observerCoverage = append(observerCoverage, NodeObserverStatsResp{ - ObserverID: nullStr(obsID), ObserverName: nullStr(obsName), - PacketCount: pktCount, AvgSnr: nullFloat(avgSnr), AvgRssi: nullFloat(avgRssi), - FirstSeen: nullStr(first), LastSeen: nullStr(last), - }) - } - } - - // Hop distribution from path_json - hdSQL := fmt.Sprintf(`SELECT path_json FROM packets_v WHERE %s AND path_json IS NOT NULL AND path_json != '[]'`, timeWhere) - hdRows, _ := s.db.conn.Query(hdSQL, pk, np, fromISO) - hopCounts := map[string]int{} - if hdRows != nil { - defer hdRows.Close() - for hdRows.Next() { - var pj string - hdRows.Scan(&pj) - var hops []interface{} - if json.Unmarshal([]byte(pj), &hops) == nil { - key := fmt.Sprintf("%d", len(hops)) - if len(hops) >= 4 { - key = "4+" - } - hopCounts[key]++ - } - } - } - // Also count zero-hop packets - zhSQL := fmt.Sprintf(`SELECT COUNT(*) FROM packets_v WHERE %s AND (path_json IS NULL OR path_json = '[]')`, timeWhere) - var zeroHops int - s.db.conn.QueryRow(zhSQL, pk, np, fromISO).Scan(&zeroHops) - if zeroHops > 0 { - hopCounts["0"] += zeroHops - } - hopDistribution := make([]HopDistEntry, 0) - for _, h := range []string{"0", "1", "2", "3", "4+"} { - if c, ok := hopCounts[h]; ok { - hopDistribution = append(hopDistribution, HopDistEntry{Hops: h, Count: c}) - } - } - - // Uptime heatmap - uhSQL := fmt.Sprintf(`SELECT - CAST(strftime('%%w', timestamp) AS INTEGER) as dayOfWeek, - CAST(strftime('%%H', timestamp) AS INTEGER) as hour, - COUNT(*) as count - FROM packets_v WHERE %s - GROUP BY dayOfWeek, hour ORDER BY count DESC`, timeWhere) - uhRows, _ := s.db.conn.Query(uhSQL, pk, np, fromISO) - uptimeHeatmap := make([]HeatmapCell, 0) - if uhRows != nil { - defer uhRows.Close() - for uhRows.Next() { - var dow, hr, cnt int - uhRows.Scan(&dow, &hr, &cnt) - uptimeHeatmap = append(uptimeHeatmap, HeatmapCell{DayOfWeek: dow, Hour: hr, Count: cnt}) - } - } - - // Computed stats - totalPackets := 0 - for _, entry := range activityTimeline { - totalPackets += entry.Count - } - - var snrMean, snrStdDev float64 - var snrCount int - snrStatsSQL := fmt.Sprintf(`SELECT COUNT(*), COALESCE(AVG(snr),0), COALESCE( - SQRT(AVG(snr*snr) - AVG(snr)*AVG(snr)), 0) - FROM packets_v WHERE %s AND snr IS NOT NULL`, timeWhere) - s.db.conn.QueryRow(snrStatsSQL, pk, np, fromISO).Scan(&snrCount, &snrMean, &snrStdDev) - _ = snrCount - - signalGrade := "D" - if snrMean >= 10 { - signalGrade = "A" - } else if snrMean >= 7 { - signalGrade = "A-" - } else if snrMean >= 4 { - signalGrade = "B+" - } else if snrMean >= 1 { - signalGrade = "B" - } else if snrMean >= -3 { - signalGrade = "C" - } - - relayCount := 0 - for _, h := range []string{"1", "2", "3", "4+"} { - relayCount += hopCounts[h] - } - var relayPct float64 - if totalPackets > 0 { - relayPct = round(float64(relayCount)*100.0/float64(totalPackets), 1) - } - - var avgPacketsPerDay float64 - if days > 0 { - avgPacketsPerDay = round(float64(totalPackets)/float64(days), 1) - } - - // Longest silence - var longestSilenceMs int - var longestSilenceStart interface{} - if len(activityTimeline) >= 2 { - for i := 1; i < len(activityTimeline); i++ { - var t1Str, t2Str string - if activityTimeline[i-1].Bucket != nil { - t1Str = *activityTimeline[i-1].Bucket - } - if activityTimeline[i].Bucket != nil { - t2Str = *activityTimeline[i].Bucket - } - t1, e1 := time.Parse(time.RFC3339, t1Str) - t2, e2 := time.Parse(time.RFC3339, t2Str) - if e1 == nil && e2 == nil { - gap := int(t2.Sub(t1).Milliseconds()) - if gap > longestSilenceMs { - longestSilenceMs = gap - longestSilenceStart = t1Str - } - } - } - } - - // Availability - totalHours := float64(days) * 24 - activeHours := float64(len(activityTimeline)) - availabilityPct := round(activeHours*100.0/totalHours, 1) - if availabilityPct > 100 { - availabilityPct = 100 - } - - writeJSON(w, NodeAnalyticsResponse{ - Node: node, - TimeRange: TimeRangeResp{From: fromISO, To: toISO, Days: days}, - ActivityTimeline: activityTimeline, - SnrTrend: snrTrend, - PacketTypeBreakdown: packetTypeBreakdown, - ObserverCoverage: observerCoverage, - HopDistribution: hopDistribution, - PeerInteractions: []PeerInteraction{}, - UptimeHeatmap: uptimeHeatmap, - ComputedStats: ComputedNodeStats{ - AvailabilityPct: availabilityPct, - LongestSilenceMs: longestSilenceMs, - LongestSilenceStart: longestSilenceStart, - SignalGrade: signalGrade, - SnrMean: round(snrMean, 1), - SnrStdDev: round(snrStdDev, 1), - RelayPct: relayPct, - TotalPackets: totalPackets, - UniqueObservers: len(observerCoverage), - UniquePeers: 0, - AvgPacketsPerDay: avgPacketsPerDay, - }, - }) + writeError(w, 404, "Not found") } // --- Analytics Handlers --- func (s *Server) handleAnalyticsRF(w http.ResponseWriter, r *http.Request) { + region := r.URL.Query().Get("region") if s.store != nil { - region := r.URL.Query().Get("region") writeJSON(w, s.store.GetAnalyticsRF(region)) return } - // Fallback: basic RF analytics from SQL - region := r.URL.Query().Get("region") - regionFilter := "" - var rArgs []interface{} - if region != "" { - regionFilter = "AND observer_id IN (SELECT id FROM observers WHERE iata = ?)" - rArgs = append(rArgs, region) - } - - // SNR/RSSI stats - rfSQL := fmt.Sprintf(`SELECT COUNT(*) as cnt, AVG(snr) as avgSnr, MIN(snr) as minSnr, MAX(snr) as maxSnr, - AVG(rssi) as avgRssi, MIN(rssi) as minRssi, MAX(rssi) as maxRssi - FROM packets_v WHERE snr IS NOT NULL %s`, regionFilter) - var cnt int - var avgSnr, minSnr, maxSnr, avgRssi, minRssi, maxRssi sql.NullFloat64 - s.db.conn.QueryRow(rfSQL, rArgs...).Scan(&cnt, &avgSnr, &minSnr, &maxSnr, &avgRssi, &minRssi, &maxRssi) - - // Payload type distribution - ptSQL := fmt.Sprintf(`SELECT payload_type, COUNT(DISTINCT hash) as count FROM packets_v WHERE 1=1 %s GROUP BY payload_type ORDER BY count DESC`, regionFilter) - ptRows, _ := s.db.conn.Query(ptSQL, rArgs...) - payloadTypes := make([]PayloadTypeEntry, 0) - ptNames := map[int]string{0: "REQ", 1: "RESPONSE", 2: "TXT_MSG", 3: "ACK", 4: "ADVERT", 5: "GRP_TXT", 7: "ANON_REQ", 8: "PATH", 9: "TRACE", 11: "CONTROL"} - if ptRows != nil { - defer ptRows.Close() - for ptRows.Next() { - var pt, count int - ptRows.Scan(&pt, &count) - name := ptNames[pt] - if name == "" { - name = fmt.Sprintf("UNK(%d)", pt) - } - payloadTypes = append(payloadTypes, PayloadTypeEntry{Type: pt, Name: name, Count: count}) - } - } - - // Total counts - var totalAll int - countSQL := fmt.Sprintf("SELECT COUNT(*) FROM packets_v WHERE 1=1 %s", regionFilter) - s.db.conn.QueryRow(countSQL, rArgs...).Scan(&totalAll) - var totalTx int - txSQL := fmt.Sprintf("SELECT COUNT(DISTINCT hash) FROM packets_v WHERE 1=1 %s", regionFilter) - s.db.conn.QueryRow(txSQL, rArgs...).Scan(&totalTx) - writeJSON(w, RFAnalyticsResponse{ - TotalPackets: cnt, - TotalAllPackets: totalAll, - TotalTransmissions: totalTx, - SNR: SignalStats{ - Min: nullFloatVal(minSnr), Max: nullFloatVal(maxSnr), - Avg: nullFloatVal(avgSnr), Median: 0, Stddev: 0, - }, - RSSI: SignalStats{ - Min: nullFloatVal(minRssi), Max: nullFloatVal(maxRssi), - Avg: nullFloatVal(avgRssi), Median: 0, Stddev: 0, - }, + SNR: SignalStats{}, + RSSI: SignalStats{}, SnrValues: Histogram{Bins: []HistogramBin{}, Min: 0, Max: 0}, RssiValues: Histogram{Bins: []HistogramBin{}, Min: 0, Max: 0}, PacketSizes: Histogram{Bins: []HistogramBin{}, Min: 0, Max: 0}, - MinPacketSize: 0, - MaxPacketSize: 0, - AvgPacketSize: 0, - PacketsPerHour: []HourlyCount{}, - PayloadTypes: payloadTypes, - SnrByType: []PayloadTypeSignal{}, - SignalOverTime: []SignalOverTimeEntry{}, - ScatterData: []ScatterPoint{}, - TimeSpanHours: 0, + PacketsPerHour: []HourlyCount{}, + PayloadTypes: []PayloadTypeEntry{}, + SnrByType: []PayloadTypeSignal{}, + SignalOverTime: []SignalOverTimeEntry{}, + ScatterData: []ScatterPoint{}, }) } func (s *Server) handleAnalyticsTopology(w http.ResponseWriter, r *http.Request) { + region := r.URL.Query().Get("region") if s.store != nil { - region := r.URL.Query().Get("region") writeJSON(w, s.store.GetAnalyticsTopology(region)) return } - // SQL fallback — compute basic topology from path_json - region := r.URL.Query().Get("region") - regionFilter := "" - var rArgs []interface{} - if region != "" { - regionFilter = "AND observer_id IN (SELECT id FROM observers WHERE iata = ?)" - rArgs = append(rArgs, region) - } - - pathSQL := fmt.Sprintf("SELECT path_json, snr FROM packets_v WHERE path_json IS NOT NULL AND path_json != '[]' %s", regionFilter) - pathRows, _ := s.db.conn.Query(pathSQL, rArgs...) - - hopCountMap := map[int]int{} - repeaterCounts := map[string]int{} - pairCounts := map[string]int{} - nodesSeen := map[string]bool{} - hopsVsSnrSum := map[int]float64{} - hopsVsSnrCnt := map[int]int{} - - if pathRows != nil { - defer pathRows.Close() - for pathRows.Next() { - var pj string - var snr sql.NullFloat64 - pathRows.Scan(&pj, &snr) - var hops []string - if json.Unmarshal([]byte(pj), &hops) != nil || len(hops) == 0 { - continue - } - hc := len(hops) - if hc > 25 { - hc = 25 - } - hopCountMap[hc]++ - for _, h := range hops { - nodesSeen[h] = true - repeaterCounts[h]++ - } - for i := 0; i+1 < len(hops); i++ { - pair := hops[i] + ":" + hops[i+1] - pairCounts[pair]++ - } - if snr.Valid { - hopsVsSnrSum[hc] += snr.Float64 - hopsVsSnrCnt[hc]++ - } - } - } - - var totalHopCount, maxHops int - for h, c := range hopCountMap { - totalHopCount += h * c - if h > maxHops { - maxHops = h - } - } - totalPaths := 0 - for _, c := range hopCountMap { - totalPaths += c - } - var avgHops float64 - if totalPaths > 0 { - avgHops = round(float64(totalHopCount)/float64(totalPaths), 1) - } - - hopDistribution := make([]TopologyHopDist, 0) - for h := 1; h <= maxHops; h++ { - if c, ok := hopCountMap[h]; ok { - hopDistribution = append(hopDistribution, TopologyHopDist{Hops: h, Count: c}) - } - } - - type kv struct { - k string - v int - } - var repSorted []kv - for k, v := range repeaterCounts { - repSorted = append(repSorted, kv{k, v}) - } - sort.Slice(repSorted, func(i, j int) bool { return repSorted[i].v > repSorted[j].v }) - topRepeaters := make([]TopRepeater, 0) - for i, rp := range repSorted { - if i >= 20 { - break - } - topRepeaters = append(topRepeaters, TopRepeater{Hop: rp.k, Count: rp.v, Name: nil, Pubkey: nil}) - } - - var pairSorted []kv - for k, v := range pairCounts { - pairSorted = append(pairSorted, kv{k, v}) - } - sort.Slice(pairSorted, func(i, j int) bool { return pairSorted[i].v > pairSorted[j].v }) - topPairs := make([]TopPair, 0) - for i, p := range pairSorted { - if i >= 20 { - break - } - parts := strings.SplitN(p.k, ":", 2) - topPairs = append(topPairs, TopPair{ - HopA: parts[0], HopB: parts[1], Count: p.v, - NameA: nil, NameB: nil, PubkeyA: nil, PubkeyB: nil, - }) - } - - hopsVsSnr := make([]HopsVsSnr, 0) - for h := 1; h <= maxHops; h++ { - if cnt, ok := hopsVsSnrCnt[h]; ok && cnt > 0 { - hopsVsSnr = append(hopsVsSnr, HopsVsSnr{ - Hops: h, Count: cnt, AvgSnr: round(hopsVsSnrSum[h]/float64(cnt), 1), - }) - } - } - - obsList := make([]ObserverRef, 0) - obsRows, _ := s.db.conn.Query("SELECT id, name FROM observers") - if obsRows != nil { - defer obsRows.Close() - for obsRows.Next() { - var oid string - var oname sql.NullString - obsRows.Scan(&oid, &oname) - obsList = append(obsList, ObserverRef{ID: oid, Name: nullStr(oname)}) - } - } - writeJSON(w, TopologyResponse{ - UniqueNodes: len(nodesSeen), - AvgHops: avgHops, - MedianHops: 0, - MaxHops: maxHops, - HopDistribution: hopDistribution, - TopRepeaters: topRepeaters, - TopPairs: topPairs, - HopsVsSnr: hopsVsSnr, - Observers: obsList, + HopDistribution: []TopologyHopDist{}, + TopRepeaters: []TopRepeater{}, + TopPairs: []TopPair{}, + HopsVsSnr: []HopsVsSnr{}, + Observers: []ObserverRef{}, PerObserverReach: map[string]*ObserverReach{}, MultiObsNodes: []MultiObsNode{}, BestPathList: []BestPathEntry{}, @@ -1604,64 +1111,18 @@ func (s *Server) handleAnalyticsChannels(w http.ResponseWriter, r *http.Request) } func (s *Server) handleAnalyticsDistance(w http.ResponseWriter, r *http.Request) { + region := r.URL.Query().Get("region") if s.store != nil { - region := r.URL.Query().Get("region") writeJSON(w, s.store.GetAnalyticsDistance(region)) return } - // SQL fallback - region := r.URL.Query().Get("region") - regionFilter := "" - var rArgs []interface{} - if region != "" { - regionFilter = "AND observer_id IN (SELECT id FROM observers WHERE iata = ?)" - rArgs = append(rArgs, region) - } - - nodeLocMap := s.db.GetNodeLocations() - _ = nodeLocMap - - pathSQL := fmt.Sprintf("SELECT path_json, hash, timestamp, snr FROM packets_v WHERE path_json IS NOT NULL AND path_json != '[]' %s ORDER BY timestamp DESC LIMIT 5000", regionFilter) - pathRows, _ := s.db.conn.Query(pathSQL, rArgs...) - - var totalHops, totalPaths int - var maxDist, distSum float64 - topHops := make([]DistanceHop, 0) - topPaths := make([]DistancePath, 0) - catStats := map[string]*CategoryDistStats{} - distOverTime := make([]DistOverTimeEntry, 0) - - if pathRows != nil { - defer pathRows.Close() - for pathRows.Next() { - var pj, hash string - var ts sql.NullString - var snr sql.NullFloat64 - pathRows.Scan(&pj, &hash, &ts, &snr) - var hops []string - if json.Unmarshal([]byte(pj), &hops) != nil || len(hops) == 0 { - continue - } - totalPaths++ - totalHops += len(hops) - } - } - - var avgDist float64 - if totalHops > 0 { - avgDist = round(distSum/float64(totalHops), 2) - } - writeJSON(w, DistanceAnalyticsResponse{ - Summary: DistanceSummary{ - TotalHops: totalHops, TotalPaths: totalPaths, - AvgDist: avgDist, MaxDist: maxDist, - }, - TopHops: topHops, - TopPaths: topPaths, - CatStats: catStats, + Summary: DistanceSummary{}, + TopHops: []DistanceHop{}, + TopPaths: []DistancePath{}, + CatStats: map[string]*CategoryDistStats{}, DistHistogram: nil, - DistOverTime: distOverTime, + DistOverTime: []DistOverTimeEntry{}, }) } @@ -1887,102 +1348,147 @@ func (s *Server) handleObserverDetail(w http.ResponseWriter, r *http.Request) { func (s *Server) handleObserverAnalytics(w http.ResponseWriter, r *http.Request) { id := mux.Vars(r)["id"] days := queryInt(r, "days", 7) - since := time.Now().Add(-time.Duration(days) * 24 * time.Hour).Format(time.RFC3339) + if days < 1 { + days = 1 + } + if days > 365 { + days = 365 + } + if s.store == nil { + writeError(w, 503, "Packet store unavailable") + return + } - // Timeline - bucketH := 4 + since := time.Now().Add(-time.Duration(days) * 24 * time.Hour) + s.store.mu.RLock() + obsList := s.store.byObserver[id] + filtered := make([]*StoreObs, 0, len(obsList)) + for _, obs := range obsList { + if obs.Timestamp == "" { + continue + } + t, err := time.Parse(time.RFC3339Nano, obs.Timestamp) + if err != nil { + t, err = time.Parse(time.RFC3339, obs.Timestamp) + } + if err != nil { + t, err = time.Parse("2006-01-02 15:04:05", obs.Timestamp) + } + if err != nil { + continue + } + if t.Equal(since) || t.After(since) { + filtered = append(filtered, obs) + } + } + sort.Slice(filtered, func(i, j int) bool { return filtered[i].Timestamp > filtered[j].Timestamp }) + + bucketDur := 24 * time.Hour if days <= 1 { - bucketH = 1 - } else if days > 7 { - bucketH = 24 + bucketDur = time.Hour + } else if days <= 7 { + bucketDur = 4 * time.Hour } - // Timeline — packet count per time bucket - bucketFmt := fmt.Sprintf("strftime('%%Y-%%m-%%dT', timestamp) || printf('%%02d', (CAST(strftime('%%H', timestamp) AS INTEGER) / %d) * %d) || ':00:00Z'", bucketH, bucketH) - tlSQL := fmt.Sprintf(`SELECT %s as label, COUNT(*) as count - FROM packets_v WHERE observer_id = ? AND timestamp > ? - GROUP BY label ORDER BY label`, bucketFmt) - tlRows, _ := s.db.conn.Query(tlSQL, id, since) - timeline := make([]TimeBucket, 0) - if tlRows != nil { - defer tlRows.Close() - for tlRows.Next() { - var label string - var count int - tlRows.Scan(&label, &count) - l := label - timeline = append(timeline, TimeBucket{Label: &l, Count: count}) + formatLabel := func(t time.Time) string { + if days <= 1 { + return t.UTC().Format("15:04") } + if days <= 7 { + return t.UTC().Format("Mon 15:04") + } + return t.UTC().Format("Jan 02") } - // Nodes timeline — unique nodes per time bucket - ntSQL := fmt.Sprintf(`SELECT %s as label, COUNT(DISTINCT hash) as count - FROM packets_v WHERE observer_id = ? AND timestamp > ? - GROUP BY label ORDER BY label`, bucketFmt) - ntRows, _ := s.db.conn.Query(ntSQL, id, since) - nodesTimeline := make([]TimeBucket, 0) - if ntRows != nil { - defer ntRows.Close() - for ntRows.Next() { - var label string - var count int - ntRows.Scan(&label, &count) - l := label - nodesTimeline = append(nodesTimeline, TimeBucket{Label: &l, Count: count}) - } - } - - // SNR distribution - snrSQL := `SELECT - CAST(snr / 2 AS INTEGER) * 2 as rangeStart, - COUNT(*) as count - FROM packets_v WHERE observer_id = ? AND timestamp > ? AND snr IS NOT NULL - GROUP BY rangeStart ORDER BY rangeStart` - snrRows, _ := s.db.conn.Query(snrSQL, id, since) - snrDistribution := make([]SnrDistributionEntry, 0) - if snrRows != nil { - defer snrRows.Close() - for snrRows.Next() { - var rangeStart, count int - snrRows.Scan(&rangeStart, &count) - snrDistribution = append(snrDistribution, SnrDistributionEntry{ - Range: fmt.Sprintf("%d to %d", rangeStart, rangeStart+2), - Count: count, - }) - } - } - - // Packet type breakdown - ptSQL := `SELECT payload_type, COUNT(*) as count FROM packets_v WHERE observer_id = ? AND timestamp > ? GROUP BY payload_type` - ptRows, _ := s.db.conn.Query(ptSQL, id, since) packetTypes := map[string]int{} - if ptRows != nil { - defer ptRows.Close() - for ptRows.Next() { - var pt, count int - ptRows.Scan(&pt, &count) - packetTypes[strconv.Itoa(pt)] = count - } - } + timelineCounts := map[int64]int{} + nodeBucketSets := map[int64]map[string]struct{}{} + snrBuckets := map[int]*SnrDistributionEntry{} + recentPackets := make([]map[string]interface{}, 0, 20) - // Recent packets - rpSQL := `SELECT id, raw_hex, timestamp, observer_id, observer_name, direction, snr, rssi, score, hash, route_type, payload_type, payload_version, path_json, decoded_json, created_at - FROM packets_v WHERE observer_id = ? AND timestamp > ? ORDER BY timestamp DESC LIMIT 20` - rpRows, _ := s.db.conn.Query(rpSQL, id, since) - recentPackets := make([]map[string]interface{}, 0) - if rpRows != nil { - defer rpRows.Close() - for rpRows.Next() { - p := scanPacketRow(rpRows) - if p != nil { - recentPackets = append(recentPackets, p) + for i, obs := range filtered { + ts, err := time.Parse(time.RFC3339Nano, obs.Timestamp) + if err != nil { + ts, err = time.Parse(time.RFC3339, obs.Timestamp) + } + if err != nil { + ts, err = time.Parse("2006-01-02 15:04:05", obs.Timestamp) + } + if err != nil { + continue + } + bucketStart := ts.UTC().Truncate(bucketDur).Unix() + timelineCounts[bucketStart]++ + if nodeBucketSets[bucketStart] == nil { + nodeBucketSets[bucketStart] = map[string]struct{}{} + } + + enriched := s.store.enrichObs(obs) + if pt, ok := enriched["payload_type"].(int); ok { + packetTypes[strconv.Itoa(pt)]++ + } + if decodedRaw, ok := enriched["decoded_json"].(string); ok && decodedRaw != "" { + var decoded map[string]interface{} + if json.Unmarshal([]byte(decodedRaw), &decoded) == nil { + for _, k := range []string{"pubKey", "srcHash", "destHash"} { + if v, ok := decoded[k].(string); ok && v != "" { + nodeBucketSets[bucketStart][v] = struct{}{} + } + } } } + for _, hop := range parsePathJSON(obs.PathJSON) { + if hop != "" { + nodeBucketSets[bucketStart][hop] = struct{}{} + } + } + if obs.SNR != nil { + bucket := int(*obs.SNR) / 2 * 2 + if *obs.SNR < 0 && int(*obs.SNR) != bucket { + bucket -= 2 + } + if snrBuckets[bucket] == nil { + snrBuckets[bucket] = &SnrDistributionEntry{Range: fmt.Sprintf("%d to %d", bucket, bucket+2)} + } + snrBuckets[bucket].Count++ + } + if i < 20 { + recentPackets = append(recentPackets, enriched) + } + } + s.store.mu.RUnlock() + + buildTimeline := func(counts map[int64]int) []TimeBucket { + keys := make([]int64, 0, len(counts)) + for k := range counts { + keys = append(keys, k) + } + sort.Slice(keys, func(i, j int) bool { return keys[i] < keys[j] }) + out := make([]TimeBucket, 0, len(keys)) + for _, k := range keys { + lbl := formatLabel(time.Unix(k, 0)) + out = append(out, TimeBucket{Label: &lbl, Count: counts[k]}) + } + return out + } + + nodeCounts := make(map[int64]int, len(nodeBucketSets)) + for k, nodes := range nodeBucketSets { + nodeCounts[k] = len(nodes) + } + snrKeys := make([]int, 0, len(snrBuckets)) + for k := range snrBuckets { + snrKeys = append(snrKeys, k) + } + sort.Ints(snrKeys) + snrDistribution := make([]SnrDistributionEntry, 0, len(snrKeys)) + for _, k := range snrKeys { + snrDistribution = append(snrDistribution, *snrBuckets[k]) } writeJSON(w, ObserverAnalyticsResponse{ - Timeline: timeline, + Timeline: buildTimeline(timelineCounts), PacketTypes: packetTypes, - NodesTimeline: nodesTimeline, + NodesTimeline: buildTimeline(nodeCounts), SnrDistribution: snrDistribution, RecentPackets: recentPackets, }) @@ -2112,35 +1618,6 @@ func (s *Server) handleAudioLabBuckets(w http.ResponseWriter, r *http.Request) { } buckets[typeName] = picked } - } else { - // Fallback: direct DB query when store is not loaded - ptSQL := `SELECT payload_type, id, raw_hex, hash, decoded_json, path_json, observer_id, timestamp - FROM ( - SELECT *, ROW_NUMBER() OVER (PARTITION BY payload_type ORDER BY length(raw_hex)) as rn - FROM packets_v WHERE raw_hex IS NOT NULL - ) sub WHERE rn <= 8` - rows, err := s.db.conn.Query(ptSQL) - if err != nil { - writeJSON(w, AudioLabBucketsResponse{Buckets: buckets}) - return - } - defer rows.Close() - - for rows.Next() { - var pt, id int - var rawHex, hash, decodedJSON, pathJSON, obsID, ts sql.NullString - rows.Scan(&pt, &id, &rawHex, &hash, &decodedJSON, &pathJSON, &obsID, &ts) - typeName := payloadTypeNames[pt] - if typeName == "" { - typeName = "UNKNOWN" - } - buckets[typeName] = append(buckets[typeName], AudioLabPacket{ - Hash: nullStr(hash), RawHex: nullStr(rawHex), - DecodedJSON: nullStr(decodedJSON), ObservationCount: 1, - PayloadType: pt, PathJSON: nullStr(pathJSON), - ObserverID: nullStr(obsID), Timestamp: nullStr(ts), - }) - } } writeJSON(w, AudioLabBucketsResponse{Buckets: buckets}) diff --git a/cmd/server/routes_test.go b/cmd/server/routes_test.go index 8daa3e0a..fdbe07cc 100644 --- a/cmd/server/routes_test.go +++ b/cmd/server/routes_test.go @@ -18,7 +18,9 @@ func setupTestServer(t *testing.T) (*Server, *mux.Router) { hub := NewHub() srv := NewServer(db, cfg, hub) store := NewPacketStore(db) - store.Load() + if err := store.Load(); err != nil { + t.Fatalf("store.Load failed: %v", err) + } srv.store = store router := mux.NewRouter() srv.RegisterRoutes(router) @@ -722,6 +724,9 @@ func TestNodePathsFound(t *testing.T) { if body["paths"] == nil { t.Error("expected paths in response") } + if got, ok := body["totalTransmissions"].(float64); !ok || got < 1 { + t.Errorf("expected totalTransmissions >= 1, got %v", body["totalTransmissions"]) + } } func TestNodePathsNotFound(t *testing.T) { @@ -832,6 +837,9 @@ func TestObserverAnalytics(t *testing.T) { if body["recentPackets"] == nil { t.Error("expected recentPackets") } + if recent, ok := body["recentPackets"].([]interface{}); !ok || len(recent) == 0 { + t.Errorf("expected non-empty recentPackets, got %v", body["recentPackets"]) + } }) t.Run("custom days", func(t *testing.T) { @@ -1251,6 +1259,11 @@ func TestNodeAnalyticsNoNameNode(t *testing.T) { cfg := &Config{Port: 3000} hub := NewHub() srv := NewServer(db, cfg, hub) + store := NewPacketStore(db) + if err := store.Load(); err != nil { + t.Fatalf("store.Load failed: %v", err) + } + srv.store = store router := mux.NewRouter() srv.RegisterRoutes(router) @@ -1282,6 +1295,11 @@ func TestNodeHealthForNoNameNode(t *testing.T) { cfg := &Config{Port: 3000} hub := NewHub() srv := NewServer(db, cfg, hub) + store := NewPacketStore(db) + if err := store.Load(); err != nil { + t.Fatalf("store.Load failed: %v", err) + } + srv.store = store router := mux.NewRouter() srv.RegisterRoutes(router) @@ -1521,8 +1539,6 @@ func TestHandlerErrorPaths(t *testing.T) { router := mux.NewRouter() srv.RegisterRoutes(router) - // Drop the view to force query errors - db.conn.Exec("DROP VIEW IF EXISTS packets_v") t.Run("stats error", func(t *testing.T) { db.conn.Exec("DROP TABLE IF EXISTS transmissions") @@ -1563,7 +1579,7 @@ func TestHandlerErrorTraces(t *testing.T) { router := mux.NewRouter() srv.RegisterRoutes(router) - db.conn.Exec("DROP VIEW IF EXISTS packets_v") + db.conn.Exec("DROP TABLE IF EXISTS observations") req := httptest.NewRequest("GET", "/api/traces/abc123def4567890", nil) w := httptest.NewRecorder() @@ -1697,13 +1713,12 @@ func TestHandlerErrorTimestamps(t *testing.T) { router := mux.NewRouter() srv.RegisterRoutes(router) - db.conn.Exec("DROP VIEW IF EXISTS packets_v") - + // Without a store, timestamps returns empty 200 req := httptest.NewRequest("GET", "/api/packets/timestamps?since=2020-01-01", nil) w := httptest.NewRecorder() router.ServeHTTP(w, req) - if w.Code != 500 { - t.Errorf("expected 500 for timestamps error, got %d", w.Code) + if w.Code != 200 { + t.Errorf("expected 200 for timestamps without store, got %d", w.Code) } } @@ -1740,8 +1755,8 @@ func TestHandlerErrorBulkHealth(t *testing.T) { req := httptest.NewRequest("GET", "/api/nodes/bulk-health", nil) w := httptest.NewRecorder() router.ServeHTTP(w, req) - if w.Code != 500 { - t.Errorf("expected 500, got %d", w.Code) + if w.Code != 200 { + t.Errorf("expected 200, got %d", w.Code) } } @@ -1876,7 +1891,9 @@ func TestGetNodeHashSizeInfoFlipFlop(t *testing.T) { db := setupTestDB(t) seedTestData(t, db) store := NewPacketStore(db) -store.Load() +if err := store.Load(); err != nil { + t.Fatalf("store.Load failed: %v", err) +} pk := "abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890" db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, role) VALUES (?, 'TestNode', 'repeater')", pk) @@ -1934,7 +1951,17 @@ for _, field := range arrayFields { if body[field] == nil { t.Errorf("field %q is null, expected []", field) } + } } +func TestObserverAnalyticsNoStore(t *testing.T) { + _, router := setupNoStoreServer(t) + req := httptest.NewRequest("GET", "/api/observers/obs1/analytics", nil) + w := httptest.NewRecorder() + router.ServeHTTP(w, req) + + if w.Code != 503 { + t.Fatalf("expected 503, got %d", w.Code) + } } func min(a, b int) int { if a < b { diff --git a/cmd/server/store.go b/cmd/server/store.go index 60444282..9d3b3dd4 100644 --- a/cmd/server/store.go +++ b/cmd/server/store.go @@ -62,7 +62,7 @@ type StoreObs struct { type PacketStore struct { mu sync.RWMutex db *DB - packets []*StoreTx // sorted by first_seen DESC + packets []*StoreTx // sorted by first_seen ASC (oldest first; newest at tail) byHash map[string]*StoreTx // hash → *StoreTx byTxID map[int]*StoreTx // transmission_id → *StoreTx byObsID map[int]*StoreObs // observation_id → *StoreObs @@ -176,7 +176,7 @@ func (s *PacketStore) Load() error { FROM transmissions t LEFT JOIN observations o ON o.transmission_id = t.id LEFT JOIN observers obs ON obs.rowid = o.observer_idx - ORDER BY t.first_seen DESC, o.timestamp DESC` + ORDER BY t.first_seen ASC, o.timestamp DESC` } else { loadSQL = `SELECT t.id, t.raw_hex, t.hash, t.first_seen, t.route_type, t.payload_type, t.payload_version, t.decoded_json, @@ -184,7 +184,7 @@ func (s *PacketStore) Load() error { o.snr, o.rssi, o.score, o.path_json, o.timestamp FROM transmissions t LEFT JOIN observations o ON o.transmission_id = t.id - ORDER BY t.first_seen DESC, o.timestamp DESC` + ORDER BY t.first_seen ASC, o.timestamp DESC` } rows, err := s.db.conn.Query(loadSQL) @@ -368,28 +368,32 @@ func (s *PacketStore) QueryPackets(q PacketQuery) *PacketResult { results := s.filterPackets(q) total := len(results) - if q.Order == "ASC" { - sorted := make([]*StoreTx, len(results)) - copy(sorted, results) - sort.Slice(sorted, func(i, j int) bool { - return sorted[i].FirstSeen < sorted[j].FirstSeen - }) - results = sorted - } - - // Paginate + // results is oldest-first (ASC). For DESC (default) read backwards from the tail; + // for ASC read forwards. Both are O(page_size) — no sort copy needed. start := q.Offset - if start >= len(results) { + if start >= total { return &PacketResult{Packets: []map[string]interface{}{}, Total: total} } - end := start + q.Limit - if end > len(results) { - end = len(results) + pageSize := q.Limit + if start+pageSize > total { + pageSize = total - start } - packets := make([]map[string]interface{}, 0, end-start) - for _, tx := range results[start:end] { - packets = append(packets, txToMap(tx)) + packets := make([]map[string]interface{}, 0, pageSize) + if q.Order == "ASC" { + for _, tx := range results[start : start+pageSize] { + packets = append(packets, txToMap(tx)) + } + } else { + // DESC: newest items are at the tail; page 0 = last pageSize items reversed + endIdx := total - start + startIdx := endIdx - pageSize + if startIdx < 0 { + startIdx = 0 + } + for i := endIdx - 1; i >= startIdx; i-- { + packets = append(packets, txToMap(results[i])) + } } return &PacketResult{Packets: packets, Total: total} } @@ -719,15 +723,16 @@ func (s *PacketStore) GetTimestamps(since string) []string { s.mu.RLock() defer s.mu.RUnlock() - // packets sorted newest first — scan from start until older than since + // packets sorted oldest-first — scan from tail until we reach items older than since var result []string - for _, tx := range s.packets { + for i := len(s.packets) - 1; i >= 0; i-- { + tx := s.packets[i] if tx.FirstSeen <= since { break } result = append(result, tx.FirstSeen) } - // Reverse to get ASC order + // result is currently newest-first; reverse to return ASC order for i, j := 0, len(result)-1; i < j; i, j = i+1, j-1 { result[i], result[j] = result[j], result[i] } @@ -777,23 +782,30 @@ func (s *PacketStore) QueryMultiNodePackets(pubkeys []string, limit, offset int, total := len(filtered) - if order == "ASC" { - sort.Slice(filtered, func(i, j int) bool { - return filtered[i].FirstSeen < filtered[j].FirstSeen - }) - } - + // filtered is oldest-first (built by iterating s.packets forward). + // Apply same DESC/ASC pagination logic as QueryPackets. if offset >= total { return &PacketResult{Packets: []map[string]interface{}{}, Total: total} } - end := offset + limit - if end > total { - end = total + pageSize := limit + if offset+pageSize > total { + pageSize = total - offset } - packets := make([]map[string]interface{}, 0, end-offset) - for _, tx := range filtered[offset:end] { - packets = append(packets, txToMap(tx)) + packets := make([]map[string]interface{}, 0, pageSize) + if order == "ASC" { + for _, tx := range filtered[offset : offset+pageSize] { + packets = append(packets, txToMap(tx)) + } + } else { + endIdx := total - offset + startIdx := endIdx - pageSize + if startIdx < 0 { + startIdx = 0 + } + for i := endIdx - 1; i >= startIdx; i-- { + packets = append(packets, txToMap(filtered[i])) + } } return &PacketResult{Packets: packets, Total: total} } @@ -926,15 +938,14 @@ func (s *PacketStore) IngestNewFromDB(sinceID, limit int) ([]map[string]interfac DecodedJSON: r.decodedJSON, } s.byHash[r.hash] = tx - // Prepend (newest first) - s.packets = append([]*StoreTx{tx}, s.packets...) + s.packets = append(s.packets, tx) // oldest-first; new items go to tail s.byTxID[r.txID] = tx s.indexByNode(tx) if tx.PayloadType != nil { pt := *tx.PayloadType - // Prepend to maintain newest-first order (matches Load ordering) + // Append to maintain oldest-first order (matches Load ordering) // so GetChannelMessages reverse iteration stays correct - s.byPayloadType[pt] = append([]*StoreTx{tx}, s.byPayloadType[pt]...) + s.byPayloadType[pt] = append(s.byPayloadType[pt], tx) } if _, exists := broadcastTxs[r.txID]; !exists { @@ -1079,8 +1090,6 @@ func (s *PacketStore) IngestNewFromDB(sinceID, limit int) ([]map[string]interfac s.cacheMu.Unlock() } - log.Printf("[poller] IngestNewFromDB: found %d new txs, maxID %d->%d", len(result), sinceID, newMaxID) - return result, newMaxID } @@ -1263,8 +1272,7 @@ func (s *PacketStore) IngestNewObservations(sinceObsID, limit int) int { s.subpathCache = make(map[string]*cachedResult) s.cacheMu.Unlock() - log.Printf("[poller] IngestNewObservations: updated %d existing txs, maxObsID %d->%d", - len(updatedTxs), sinceObsID, newMaxObsID) + // analytics caches cleared; no per-cycle log to avoid stdout overhead } return newMaxObsID @@ -1888,7 +1896,7 @@ func (s *PacketStore) GetChannelMessages(channelHash string, limit, offset int) msgMap := map[string]*msgEntry{} var msgOrder []string - // Iterate type-5 packets oldest-first (byPayloadType is in load order = newest first) + // Iterate type-5 packets oldest-first (byPayloadType is ASC = oldest first) type decodedMsg struct { Type string `json:"type"` Channel string `json:"channel"` @@ -1899,8 +1907,7 @@ func (s *PacketStore) GetChannelMessages(channelHash string, limit, offset int) } grpTxts := s.byPayloadType[5] - for i := len(grpTxts) - 1; i >= 0; i-- { - tx := grpTxts[i] + for _, tx := range grpTxts { if tx.DecodedJSON == "" { continue } @@ -4069,13 +4076,13 @@ func (s *PacketStore) GetNodeHealth(pubkey string) (map[string]interface{}, erro lhVal = lastHeard } - // Recent packets (up to 20, newest first — packets are already sorted DESC) + // Recent packets (up to 20, newest first — read from tail of oldest-first slice) recentLimit := 20 if len(packets) < recentLimit { recentLimit = len(packets) } recentPackets := make([]map[string]interface{}, 0, recentLimit) - for i := 0; i < recentLimit; i++ { + for i := len(packets) - 1; i >= len(packets)-recentLimit; i-- { p := txToMap(packets[i]) delete(p, "observations") recentPackets = append(recentPackets, p) diff --git a/cmd/server/testdata/golden/shapes.json b/cmd/server/testdata/golden/shapes.json index d04e74e9..86a5b5fd 100644 --- a/cmd/server/testdata/golden/shapes.json +++ b/cmd/server/testdata/golden/shapes.json @@ -996,6 +996,12 @@ "elementShape": { "type": "number" } + }, + "battery_mv": { + "type": "nullable_number" + }, + "temperature_c": { + "type": "nullable_number" } } }, @@ -1097,6 +1103,12 @@ }, "last_heard": { "type": "string" + }, + "battery_mv": { + "type": "nullable_number" + }, + "temperature_c": { + "type": "nullable_number" } } } diff --git a/config.example.json b/config.example.json index 6991efe8..c53a2714 100644 --- a/config.example.json +++ b/config.example.json @@ -10,7 +10,7 @@ "key": "/path/to/key.pem" }, "branding": { - "siteName": "MeshCore Analyzer", + "siteName": "CoreScope", "tagline": "Real-time MeshCore LoRa mesh network analyzer", "logoUrl": null, "faviconUrl": null @@ -32,7 +32,7 @@ "observer": "#8b5cf6" }, "home": { - "heroTitle": "MeshCore Analyzer", + "heroTitle": "CoreScope", "heroSubtitle": "Find your nodes to start monitoring them.", "steps": [ { "emoji": "📡", "title": "Connect", "description": "Link your node to the mesh" }, diff --git a/db.js b/db.js index f96c728c..c8352cd1 100644 --- a/db.js +++ b/db.js @@ -33,7 +33,9 @@ db.exec(` lon REAL, last_seen TEXT, first_seen TEXT, - advert_count INTEGER DEFAULT 0 + advert_count INTEGER DEFAULT 0, + battery_mv INTEGER, + temperature_c REAL ); CREATE TABLE IF NOT EXISTS observers ( @@ -60,7 +62,9 @@ db.exec(` lon REAL, last_seen TEXT, first_seen TEXT, - advert_count INTEGER DEFAULT 0 + advert_count INTEGER DEFAULT 0, + battery_mv INTEGER, + temperature_c REAL ); CREATE INDEX IF NOT EXISTS idx_nodes_last_seen ON nodes(last_seen); @@ -324,6 +328,22 @@ for (const col of ['model', 'firmware', 'client_version', 'radio', 'battery_mv', } } +// --- One-time migration: add telemetry columns to nodes and inactive_nodes --- +{ + const done = db.prepare(`SELECT 1 FROM _migrations WHERE name = 'node_telemetry_v1'`).get(); + if (!done) { + console.log('[migration] Adding telemetry columns to nodes/inactive_nodes...'); + const nodeCols = db.pragma('table_info(nodes)').map(c => c.name); + if (!nodeCols.includes('battery_mv')) db.exec(`ALTER TABLE nodes ADD COLUMN battery_mv INTEGER`); + if (!nodeCols.includes('temperature_c')) db.exec(`ALTER TABLE nodes ADD COLUMN temperature_c REAL`); + const inactiveCols = db.pragma('table_info(inactive_nodes)').map(c => c.name); + if (!inactiveCols.includes('battery_mv')) db.exec(`ALTER TABLE inactive_nodes ADD COLUMN battery_mv INTEGER`); + if (!inactiveCols.includes('temperature_c')) db.exec(`ALTER TABLE inactive_nodes ADD COLUMN temperature_c REAL`); + db.prepare(`INSERT INTO _migrations (name) VALUES ('node_telemetry_v1')`).run(); + console.log('[migration] node telemetry columns added'); + } +} + // --- Prepared statements --- const stmts = { upsertNode: db.prepare(` @@ -339,6 +359,12 @@ const stmts = { incrementAdvertCount: db.prepare(` UPDATE nodes SET advert_count = advert_count + 1 WHERE public_key = @public_key `), + updateNodeTelemetry: db.prepare(` + UPDATE nodes SET + battery_mv = COALESCE(@battery_mv, battery_mv), + temperature_c = COALESCE(@temperature_c, temperature_c) + WHERE public_key = @public_key + `), upsertObserver: db.prepare(` INSERT INTO observers (id, name, iata, last_seen, first_seen, packet_count, model, firmware, client_version, radio, battery_mv, uptime_secs, noise_floor) VALUES (@id, @name, @iata, @last_seen, @first_seen, 1, @model, @firmware, @client_version, @radio, @battery_mv, @uptime_secs, @noise_floor) @@ -511,6 +537,14 @@ function incrementAdvertCount(publicKey) { stmts.incrementAdvertCount.run({ public_key: publicKey }); } +function updateNodeTelemetry(data) { + stmts.updateNodeTelemetry.run({ + public_key: data.public_key, + battery_mv: data.battery_mv ?? null, + temperature_c: data.temperature_c ?? null, + }); +} + function upsertNode(data) { const now = new Date().toISOString(); stmts.upsertNode.run({ @@ -898,4 +932,4 @@ function moveStaleNodes(nodeDays) { return moved; } -module.exports = { db, schemaVersion, observerIdToRowid, resolveObserverIdx, insertTransmission, upsertNode, incrementAdvertCount, upsertObserver, updateObserverStatus, getPackets, getPacket, getTransmission, getNodes, getNode, getObservers, getStats, searchNodes, getNodeHealth, getNodeAnalytics, removePhantomNodes, moveStaleNodes }; +module.exports = { db, schemaVersion, observerIdToRowid, resolveObserverIdx, insertTransmission, upsertNode, incrementAdvertCount, updateNodeTelemetry, upsertObserver, updateObserverStatus, getPackets, getPacket, getTransmission, getNodes, getNode, getObservers, getStats, searchNodes, getNodeHealth, getNodeAnalytics, removePhantomNodes, moveStaleNodes }; diff --git a/decoder.js b/decoder.js index 2f06bfed..16270de4 100644 --- a/decoder.js +++ b/decoder.js @@ -135,10 +135,32 @@ function decodeAdvert(buf) { off += 8; } if (result.flags.hasName) { - let name = appdata.subarray(off).toString('utf8'); - // Strip non-printable characters (< 0x20 except tab/newline) and DEL + // Find null terminator to separate name from trailing telemetry bytes + let nameEnd = appdata.length; + for (let i = off; i < appdata.length; i++) { + if (appdata[i] === 0x00) { nameEnd = i; break; } + } + let name = appdata.subarray(off, nameEnd).toString('utf8'); name = name.replace(/[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]/g, ''); result.name = name; + off = nameEnd; + // Skip null terminator(s) + while (off < appdata.length && appdata[off] === 0x00) off++; + } + + // Telemetry bytes after name: battery_mv(2 LE) + temperature_c(2 LE, signed, /100) + // Only sensor nodes (advType=4) carry telemetry bytes. + if (result.flags.sensor && off + 4 <= appdata.length) { + const batteryMv = appdata.readUInt16LE(off); + const tempRaw = appdata.readInt16LE(off + 2); + const tempC = tempRaw / 100.0; + if (batteryMv > 0 && batteryMv <= 10000) { + result.battery_mv = batteryMv; + } + // Raw int16 / 100 → °C; accept -50°C to 100°C (raw: -5000 to 10000) + if (tempRaw >= -5000 && tempRaw <= 10000) { + result.temperature_c = tempC; + } } } diff --git a/docker-compose.yml b/docker-compose.yml index 8a474c58..27d5360c 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -3,8 +3,8 @@ services: prod: - image: meshcore-analyzer:latest - container_name: meshcore-prod + image: corescope:latest + container_name: corescope-prod restart: unless-stopped ports: - "${PROD_HTTP_PORT:-80}:${PROD_HTTP_PORT:-80}" @@ -24,8 +24,8 @@ services: retries: 3 staging: - image: meshcore-analyzer:latest - container_name: meshcore-staging + image: corescope:latest + container_name: corescope-staging restart: unless-stopped ports: - "${STAGING_HTTP_PORT:-81}:${STAGING_HTTP_PORT:-81}" @@ -52,8 +52,8 @@ services: args: APP_VERSION: ${APP_VERSION:-unknown} GIT_COMMIT: ${GIT_COMMIT:-unknown} - image: meshcore-go:latest - container_name: meshcore-staging-go + image: corescope-go:latest + container_name: corescope-staging-go restart: unless-stopped ports: - "${STAGING_GO_HTTP_PORT:-82}:80" diff --git a/docker/supervisord-go.conf b/docker/supervisord-go.conf index 9253beee..8c783615 100644 --- a/docker/supervisord-go.conf +++ b/docker/supervisord-go.conf @@ -14,8 +14,8 @@ stdout_logfile_maxbytes=0 stderr_logfile=/dev/stderr stderr_logfile_maxbytes=0 -[program:meshcore-ingestor] -command=/app/meshcore-ingestor -config /app/config.json +[program:corescope-ingestor] +command=/app/corescope-ingestor -config /app/config.json directory=/app autostart=true autorestart=true @@ -24,8 +24,8 @@ stdout_logfile_maxbytes=0 stderr_logfile=/dev/stderr stderr_logfile_maxbytes=0 -[program:meshcore-server] -command=/app/meshcore-server -config-dir /app -db /app/data/meshcore.db -public /app/public -port 3000 +[program:corescope-server] +command=/app/corescope-server -config-dir /app -db /app/data/meshcore.db -public /app/public -port 3000 directory=/app autostart=true autorestart=true diff --git a/docker/supervisord.conf b/docker/supervisord.conf index c1e29880..f55f34d4 100644 --- a/docker/supervisord.conf +++ b/docker/supervisord.conf @@ -14,7 +14,7 @@ stdout_logfile_maxbytes=0 stderr_logfile=/dev/stderr stderr_logfile_maxbytes=0 -[program:meshcore-analyzer] +[program:corescope] command=node /app/server.js directory=/app autostart=true diff --git a/docs/CUSTOMIZATION.md b/docs/CUSTOMIZATION.md index 640f2f90..6de40ffb 100644 --- a/docs/CUSTOMIZATION.md +++ b/docs/CUSTOMIZATION.md @@ -27,7 +27,7 @@ No restart needed. The server picks up changes to `theme.json` on every page loa **Bare metal / PM2 / systemd:** ```bash # Same directory as server.js and config.json -cp theme.json /path/to/meshcore-analyzer/ +cp theme.json /path/to/corescope/ ``` Check the server logs on startup — it tells you where it's looking: diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index a2e93d28..724f5e15 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -1,6 +1,6 @@ -# Deploying MeshCore Analyzer +# Deploying CoreScope -Get MeshCore Analyzer running with automatic HTTPS on your own server. +Get CoreScope running with automatic HTTPS on your own server. ## Table of Contents @@ -19,7 +19,7 @@ Get MeshCore Analyzer running with automatic HTTPS on your own server. ## What You'll End Up With -- MeshCore Analyzer running at `https://your-domain.com` +- CoreScope running at `https://your-domain.com` - Automatic HTTPS certificates (via Let's Encrypt + Caddy) - Built-in MQTT broker for receiving packets from observers - SQLite database for packet storage (auto-created) @@ -83,8 +83,8 @@ docker --version The easiest way — use the management script: ```bash -git clone https://github.com/Kpa-clawbot/meshcore-analyzer.git -cd meshcore-analyzer +git clone https://github.com/Kpa-clawbot/corescope.git +cd corescope ./manage.sh setup ``` @@ -111,8 +111,8 @@ flowchart LR ### 1. Download the code ```bash -git clone https://github.com/Kpa-clawbot/meshcore-analyzer.git -cd meshcore-analyzer +git clone https://github.com/Kpa-clawbot/corescope.git +cd corescope ``` ### 2. Create your config @@ -153,10 +153,10 @@ Save and close. Caddy handles certificates, renewals, and HTTP→HTTPS redirects ### 4. Build and run ```bash -docker build -t meshcore-analyzer . +docker build -t corescope . docker run -d \ - --name meshcore-analyzer \ + --name corescope \ --restart unless-stopped \ -p 80:80 \ -p 443:443 \ @@ -164,7 +164,7 @@ docker run -d \ -v $(pwd)/caddy-config/Caddyfile:/etc/caddy/Caddyfile:ro \ -v meshcore-data:/app/data \ -v caddy-data:/data/caddy \ - meshcore-analyzer + corescope ``` What each flag does: @@ -184,12 +184,12 @@ Open `https://your-domain.com`. You should see the analyzer home page. Check the logs: ```bash -docker logs meshcore-analyzer +docker logs corescope ``` Expected output: ``` -MeshCore Analyzer running on http://localhost:3000 +CoreScope running on http://localhost:3000 MQTT [local] connected to mqtt://localhost:1883 [pre-warm] 12 endpoints in XXXms ``` @@ -215,7 +215,7 @@ Add a remote broker to `mqttSources` in your `config.json`: } ``` -Restart: `docker restart meshcore-analyzer` +Restart: `docker restart corescope` ### Option B: Run your own observer @@ -271,12 +271,12 @@ If you already run a reverse proxy, skip Caddy entirely and proxy directly to th ```bash docker run -d \ - --name meshcore-analyzer \ + --name corescope \ --restart unless-stopped \ -p 3000:3000 \ -v $(pwd)/config.json:/app/config.json:ro \ -v meshcore-data:/app/data \ - meshcore-analyzer + corescope ``` Then configure your existing proxy to forward traffic to `localhost:3000`. @@ -287,12 +287,12 @@ For local testing or a LAN-only setup, use the default Caddyfile that ships in t ```bash docker run -d \ - --name meshcore-analyzer \ + --name corescope \ --restart unless-stopped \ -p 80:80 \ -v $(pwd)/config.json:/app/config.json:ro \ -v meshcore-data:/app/data \ - meshcore-analyzer + corescope ``` ## MQTT Security @@ -315,7 +315,7 @@ password_file /etc/mosquitto/passwd ``` After starting the container, create users: ```bash -docker exec -it meshcore-analyzer mosquitto_passwd -c /etc/mosquitto/passwd myuser +docker exec -it corescope mosquitto_passwd -c /etc/mosquitto/passwd myuser ``` **Option 3: Use TLS** — For production, configure Mosquitto with TLS certificates. See the [Mosquitto docs](https://mosquitto.org/man/mosquitto-conf-5.html). @@ -331,7 +331,7 @@ Packet data is stored in `meshcore.db` inside the data volume. **Using manage.sh (easiest):** ```bash -./manage.sh backup # Saves to ./backups/meshcore-TIMESTAMP.db +./manage.sh backup # Saves to ./backups/corescope-TIMESTAMP/ ./manage.sh backup ~/my-backup.db # Custom path ./manage.sh restore ./backups/some-file.db # Restore (backs up current DB first) ``` @@ -345,7 +345,7 @@ If you used `-v ./analyzer-data:/app/data` instead of a Docker volume, the datab ```bash crontab -e # Add: -0 3 * * * cd /path/to/meshcore-analyzer && ./manage.sh backup +0 3 * * * cd /path/to/corescope && ./manage.sh backup ``` ## Updating @@ -398,11 +398,11 @@ Center the map on your area in `config.json`: | Problem | Likely cause | Fix | |---------|-------------|-----| -| Site shows "connection refused" | Container not running | `docker ps` to check, `docker logs meshcore-analyzer` for errors | +| Site shows "connection refused" | Container not running | `docker ps` to check, `docker logs corescope` for errors | | HTTPS not working | Port 80 blocked | Open port 80 — Caddy needs it for ACME challenges | | "too many certificates" error | Let's Encrypt rate limit (5/domain/week) | Use a different subdomain, bring your own cert, or wait a week | | Certificate won't provision | DNS not pointed at server | `dig your-domain` must show your server IP before starting | -| No packets appearing | No observer connected | `docker exec meshcore-analyzer mosquitto_sub -t 'meshcore/#' -C 1 -W 10` — if silent, no data is coming in | +| No packets appearing | No observer connected | `docker exec corescope mosquitto_sub -t 'meshcore/#' -C 1 -W 10` — if silent, no data is coming in | | Container crashes on startup | Bad JSON in config | `python3 -c "import json; json.load(open('config.json'))"` to validate | | "address already in use" | Another web server on 80/443 | Stop it: `sudo systemctl stop nginx apache2` | | Slow on Raspberry Pi | First build is slow | Normal — subsequent builds use cache. Runtime performance is fine. | diff --git a/docs/HASH-PREFIX-DISAMBIGUATION.md b/docs/HASH-PREFIX-DISAMBIGUATION.md index 61bae815..e67eb669 100644 --- a/docs/HASH-PREFIX-DISAMBIGUATION.md +++ b/docs/HASH-PREFIX-DISAMBIGUATION.md @@ -1,4 +1,4 @@ -# Hash Prefix Disambiguation in MeshCore Analyzer +# Hash Prefix Disambiguation in CoreScope ## Section 1: Executive Summary diff --git a/docs/api-spec.md b/docs/api-spec.md index c92d70b7..d126fed8 100644 --- a/docs/api-spec.md +++ b/docs/api-spec.md @@ -1,4 +1,4 @@ -# MeshCore Analyzer — API Contract Specification +# CoreScope — API Contract Specification > **Authoritative contract.** Both the Node.js and Go backends MUST conform to this spec. > The frontend relies on these exact shapes. Breaking changes require a spec update first. @@ -1547,7 +1547,7 @@ Theme and branding configuration (merged from config.json + theme.json). ```jsonc { "branding": { - "siteName": string, // default: "MeshCore Analyzer" + "siteName": string, // default: "CoreScope" "tagline": string // default: "Real-time MeshCore LoRa mesh network analyzer" // ... additional branding keys from config/theme files }, diff --git a/docs/go-migration.md b/docs/go-migration.md index 688b8ca4..9a362e50 100644 --- a/docs/go-migration.md +++ b/docs/go-migration.md @@ -1,6 +1,6 @@ # Migrating from Node.js to Go Engine -Guide for existing MeshCore Analyzer users switching from the Node.js Docker image to the Go version. +Guide for existing CoreScope users switching from the Node.js Docker image to the Go version. > **Status (July 2025):** The Go engine is fully functional for production use. > Go images are **not yet published to Docker Hub** — you build locally from source. @@ -24,11 +24,11 @@ Guide for existing MeshCore Analyzer users switching from the Node.js Docker ima ## Prerequisites - **Docker** 20.10+ and **Docker Compose** v2 (verify: `docker compose version`) -- An existing MeshCore Analyzer deployment running the Node.js image +- An existing CoreScope deployment running the Node.js image - The repository cloned locally (needed to build the Go image): ```bash - git clone https://github.com/meshcore-dev/meshcore-analyzer.git - cd meshcore-analyzer + git clone https://github.com/Kpa-clawbot/meshcore-analyzer.git + cd corescope git pull # get latest ``` - Your `config.json` and `caddy-config/Caddyfile` in place (the same ones you use now) @@ -122,7 +122,7 @@ docker compose --profile staging-go build staging-go Or build directly: ```bash -docker build -f Dockerfile.go -t meshcore-go:latest \ +docker build -f Dockerfile.go -t corescope-go:latest \ --build-arg APP_VERSION=$(git describe --tags 2>/dev/null || echo unknown) \ --build-arg GIT_COMMIT=$(git rev-parse --short HEAD 2>/dev/null || echo unknown) \ . @@ -151,7 +151,7 @@ Once satisfied, update `docker-compose.yml` to use the Go image for prod: ```yaml services: prod: - image: meshcore-go:latest # was: meshcore-analyzer:latest + image: corescope-go:latest # was: corescope:latest build: context: . dockerfile: Dockerfile.go # add this @@ -174,9 +174,9 @@ docker compose up -d prod ./manage.sh stop # Build the Go image -docker build -f Dockerfile.go -t meshcore-analyzer:latest . +docker build -f Dockerfile.go -t corescope:latest . -# Start (manage.sh uses the meshcore-analyzer:latest image) +# Start (manage.sh uses the corescope:latest image) ./manage.sh start ``` @@ -248,7 +248,7 @@ These should match (or be close to) your pre-migration numbers. ```bash # Watch container logs for MQTT messages -docker logs -f meshcore-prod --tail 20 +docker logs -f corescope-prod --tail 20 # Or use manage.sh ./manage.sh mqtt-test @@ -279,13 +279,13 @@ If something goes wrong, switching back is straightforward: ```yaml services: prod: - image: meshcore-analyzer:latest # back to Node.js + image: corescope:latest # back to Node.js # Remove the build.dockerfile line if you added it ``` ```bash # Rebuild Node.js image if needed -docker build -t meshcore-analyzer:latest . +docker build -t corescope:latest . docker compose up -d --force-recreate prod ``` @@ -295,8 +295,8 @@ docker compose up -d --force-recreate prod ```bash ./manage.sh stop -# Rebuild Node.js image (overwrites the meshcore-analyzer:latest tag) -docker build -t meshcore-analyzer:latest . +# Rebuild Node.js image (overwrites the corescope:latest tag) +docker build -t corescope:latest . ./manage.sh start ``` @@ -310,9 +310,9 @@ docker build -t meshcore-analyzer:latest . Or manually: ```bash -docker stop meshcore-prod +docker stop corescope-prod cp backups/pre-go-migration/meshcore.db ~/meshcore-data/meshcore.db -docker start meshcore-prod +docker start corescope-prod ``` --- @@ -348,7 +348,7 @@ docker start meshcore-prod |------|---------|-----| | `engine` field in `/api/health` | Not present or `"node"` | Always `"go"` | | MQTT URL scheme | Uses `mqtt://` / `mqtts://` natively | Auto-converts to `tcp://` / `ssl://` (transparent) | -| Process model | Single Node.js process (server + ingestor) | Two binaries: `meshcore-ingestor` + `meshcore-server` (managed by supervisord) | +| Process model | Single Node.js process (server + ingestor) | Two binaries: `corescope-ingestor` + `corescope-server` (managed by supervisord) | | Memory management | Configurable via `packetStore.maxMemoryMB` | Loads all packets; no configurable limit | | Startup time | Faster (no compilation) | Slightly slower (loads all packets from DB into memory) | @@ -393,4 +393,4 @@ The following gaps have been identified. Check the GitHub issue tracker for curr 3. **Go ingestor missing `meshcore/self_info` handling** — The local node identity topic is not processed. Low impact but breaks parity. -4. **No Docker Hub publishing for Go images** — Users must build locally. CI/CD pipeline should publish `meshcore-go:latest` alongside the Node.js image. +4. **No Docker Hub publishing for Go images** — Users must build locally. CI/CD pipeline should publish `corescope-go:latest` alongside the Node.js image. diff --git a/manage.sh b/manage.sh index 09716c39..24fc84ff 100755 --- a/manage.sh +++ b/manage.sh @@ -1,13 +1,13 @@ #!/bin/bash -# MeshCore Analyzer — Setup & Management Helper +# CoreScope — Setup & Management Helper # Usage: ./manage.sh [command] # # Idempotent: safe to cancel and re-run at any point. # Each step checks what's already done and skips it. set -e -CONTAINER_NAME="meshcore-analyzer" -IMAGE_NAME="meshcore-analyzer" +CONTAINER_NAME="corescope" +IMAGE_NAME="corescope" DATA_VOLUME="meshcore-data" CADDY_VOLUME="caddy-data" STATE_FILE=".setup-state" @@ -201,7 +201,7 @@ TOTAL_STEPS=6 cmd_setup() { echo "" echo "═══════════════════════════════════════" - echo " MeshCore Analyzer Setup" + echo " CoreScope Setup" echo "═══════════════════════════════════════" echo "" @@ -501,7 +501,7 @@ prepare_staging_config() { if [ ! -f "$staging_config" ] || [ "$prod_config" -nt "$staging_config" ]; then info "Copying production config to staging..." cp "$prod_config" "$staging_config" - sed -i 's/"siteName":\s*"[^"]*"/"siteName": "MeshCore Analyzer — STAGING"/' "$staging_config" + sed -i 's/"siteName":\s*"[^"]*"/"siteName": "CoreScope — STAGING"/' "$staging_config" log "Staging config created at ${staging_config} with STAGING site name." else log "Staging config is up to date." @@ -541,13 +541,13 @@ cmd_start() { prepare_staging_db prepare_staging_config - info "Starting production container (meshcore-prod) on ports ${PROD_HTTP_PORT:-80}/${PROD_HTTPS_PORT:-443}..." - info "Starting staging container (meshcore-staging) on port ${STAGING_HTTP_PORT:-81}..." + info "Starting production container (corescope-prod) on ports ${PROD_HTTP_PORT:-80}/${PROD_HTTPS_PORT:-443}..." + info "Starting staging container (corescope-staging) on port ${STAGING_HTTP_PORT:-81}..." docker compose --profile staging up -d log "Production started on ports ${PROD_HTTP_PORT:-80}/${PROD_HTTPS_PORT:-443}/${PROD_MQTT_PORT:-1883}" log "Staging started on port ${STAGING_HTTP_PORT:-81} (MQTT: ${STAGING_MQTT_PORT:-1884})" else - info "Starting production container (meshcore-prod) on ports ${PROD_HTTP_PORT:-80}/${PROD_HTTPS_PORT:-443}..." + info "Starting production container (corescope-prod) on ports ${PROD_HTTP_PORT:-80}/${PROD_HTTPS_PORT:-443}..." docker compose up -d prod log "Production started. Staging NOT running (use --with-staging to start both)." fi @@ -586,12 +586,12 @@ cmd_stop() { if $COMPOSE_MODE; then case "$TARGET" in prod) - info "Stopping production container (meshcore-prod)..." + info "Stopping production container (corescope-prod)..." docker compose stop prod log "Production stopped." ;; staging) - info "Stopping staging container (meshcore-staging)..." + info "Stopping staging container (corescope-staging)..." docker compose stop staging log "Staging stopped." ;; @@ -617,12 +617,12 @@ cmd_restart() { local TARGET="${1:-prod}" case "$TARGET" in prod) - info "Restarting production container (meshcore-prod)..." + info "Restarting production container (corescope-prod)..." docker compose up -d --force-recreate prod log "Production restarted." ;; staging) - info "Restarting staging container (meshcore-staging)..." + info "Restarting staging container (corescope-staging)..." docker compose --profile staging up -d --force-recreate staging log "Staging restarted." ;; @@ -698,19 +698,19 @@ cmd_status() { if $COMPOSE_MODE; then echo "═══════════════════════════════════════" - echo " MeshCore Analyzer Status (Compose)" + echo " CoreScope Status (Compose)" echo "═══════════════════════════════════════" echo "" # Production - show_container_status "meshcore-prod" "Production" + show_container_status "corescope-prod" "Production" echo "" # Staging - if container_running "meshcore-staging"; then - show_container_status "meshcore-staging" "Staging" + if container_running "corescope-staging"; then + show_container_status "corescope-staging" "Staging" else - info "Staging (meshcore-staging): Not running (use --with-staging to start both)" + info "Staging (corescope-staging): Not running (use --with-staging to start both)" fi echo "" @@ -804,7 +804,7 @@ cmd_logs() { docker compose logs -f --tail="$LINES" prod ;; staging) - if container_running "meshcore-staging"; then + if container_running "corescope-staging"; then info "Tailing staging logs..." docker compose logs -f --tail="$LINES" staging else @@ -843,10 +843,10 @@ cmd_promote() { # Show what's currently running local staging_image staging_created prod_image prod_created - staging_image=$(docker inspect meshcore-staging --format '{{.Config.Image}}' 2>/dev/null || echo "not running") - staging_created=$(docker inspect meshcore-staging --format '{{.Created}}' 2>/dev/null || echo "N/A") - prod_image=$(docker inspect meshcore-prod --format '{{.Config.Image}}' 2>/dev/null || echo "not running") - prod_created=$(docker inspect meshcore-prod --format '{{.Created}}' 2>/dev/null || echo "N/A") + staging_image=$(docker inspect corescope-staging --format '{{.Config.Image}}' 2>/dev/null || echo "not running") + staging_created=$(docker inspect corescope-staging --format '{{.Created}}' 2>/dev/null || echo "N/A") + prod_image=$(docker inspect corescope-prod --format '{{.Config.Image}}' 2>/dev/null || echo "not running") + prod_created=$(docker inspect corescope-prod --format '{{.Created}}' 2>/dev/null || echo "N/A") echo " Staging: ${staging_image} (created ${staging_created})" echo " Prod: ${prod_image} (created ${prod_created})" @@ -863,8 +863,8 @@ cmd_promote() { mkdir -p "$BACKUP_DIR" if [ -f "$PROD_DATA/meshcore.db" ]; then cp "$PROD_DATA/meshcore.db" "$BACKUP_DIR/" - elif container_running "meshcore-prod"; then - docker cp meshcore-prod:/app/data/meshcore.db "$BACKUP_DIR/" + elif container_running "corescope-prod"; then + docker cp corescope-prod:/app/data/meshcore.db "$BACKUP_DIR/" else warn "Could not backup production database." fi @@ -878,7 +878,7 @@ cmd_promote() { info "Waiting for production health check..." local i health for i in $(seq 1 30); do - health=$(container_health "meshcore-prod") + health=$(container_health "corescope-prod") if [ "$health" = "healthy" ]; then log "Production healthy after ${i}s" break @@ -918,7 +918,7 @@ cmd_update() { cmd_backup() { TIMESTAMP=$(date +%Y%m%d-%H%M%S) - BACKUP_DIR="${1:-./backups/meshcore-${TIMESTAMP}}" + BACKUP_DIR="${1:-./backups/corescope-${TIMESTAMP}}" mkdir -p "$BACKUP_DIR" info "Backing up to ${BACKUP_DIR}/" @@ -972,7 +972,7 @@ cmd_restore() { if [ -d "./backups" ]; then echo "" echo " Available backups:" - ls -dt ./backups/meshcore-* 2>/dev/null | head -10 | while read d; do + ls -dt ./backups/meshcore-* ./backups/corescope-* 2>/dev/null | head -10 | while read d; do if [ -d "$d" ]; then echo " $d/ ($(ls "$d" | wc -l) files)" elif [ -f "$d" ]; then @@ -1019,7 +1019,7 @@ cmd_restore() { # Backup current state first info "Backing up current state..." - cmd_backup "./backups/meshcore-pre-restore-$(date +%Y%m%d-%H%M%S)" + cmd_backup "./backups/corescope-pre-restore-$(date +%Y%m%d-%H%M%S)" docker stop "$CONTAINER_NAME" 2>/dev/null || true @@ -1105,7 +1105,7 @@ cmd_reset() { cmd_help() { echo "" - echo "MeshCore Analyzer — Management Script" + echo "CoreScope — Management Script" echo "" echo "Usage: ./manage.shHow familiar are you with MeshCore?