meshcore-analyzer/docs/go-migration.md

Migrating from Node.js to Go Engine

Guide for existing CoreScope users switching from the Node.js Docker image to the Go version.

Status (July 2025): The Go engine is fully functional for production use. Go images are not yet published to Docker Hub — you build locally from source.

---

Table of Contents

1. Prerequisites
2. Backup
3. Config Changes
4. Switch to Go
5. DB Compatibility
6. Verification
7. Rollback to Node.js
8. Known Differences
9. FAQ

---

Prerequisites

• Docker 20.10+ and Docker Compose v2 (verify: docker compose version)
• An existing CoreScope deployment running the Node.js image
• The repository cloned locally (needed to build the Go image):

  git clone https://github.com/Kpa-clawbot/meshcore-analyzer.git
  cd corescope
  git pull   # get latest
  

• Your config.json and caddy-config/Caddyfile in place (the same ones you use now)

---

Backup

Always back up before switching engines. The Go engine applies the same v3 schema, but once Go writes to your DB, you want a restore point.

Using manage.sh

./manage.sh backup

This backs up:

• meshcore.db (SQLite database)
• config.json
• Caddyfile
• theme.json (if present)

Backups are saved to ./backups/meshcore-<timestamp>/.

Manual backup

mkdir -p backups/pre-go-migration
cp ~/meshcore-data/meshcore.db backups/pre-go-migration/
cp config.json backups/pre-go-migration/
cp caddy-config/Caddyfile backups/pre-go-migration/

Adjust paths if your data directory differs (check PROD_DATA_DIR in your .env or the default ~/meshcore-data).

---

Config Changes

The Go engine reads the same config.json as Node.js. No changes are required for a basic migration. However, there are a few things to be aware of:

MQTT broker URLs (automatic)

Node.js uses mqtt:// and mqtts:// scheme prefixes. The Go MQTT library (paho) uses tcp:// and ssl://. The Go ingestor normalizes this automatically — your existing mqtt://localhost:1883 config works as-is.

retention.nodeDays (compatible)

Both engines support retention.nodeDays (default: 7). Stale nodes are moved to the inactive_nodes table on the same schedule. No config change needed.

packetStore.maxMemoryMB (both engines read it)

Both engines take a memory budget for the in-memory packet store from packetStore.maxMemoryMB. In the Go server it does three things:

• Startup: the cold load stops once the budget is reached, so the server starts with the newest packets that fit rather than everything in SQLite.
• While running: the store evicts oldest-first when it passes the budget, down to 85% of it.
• Go heap: when neither GOMEMLIMIT nor runtime.maxMemoryMB is set, GOMEMLIMIT is derived from this value plus 50% headroom.

Unset (the default) means no limit: every packet within retention is loaded and none is evicted for memory. An old Node-era value left in a config file is therefore not inert — it caps the Go store too. Check it before upgrading, and read /api/perf (packetStore.trackedMB against maxMB) afterwards.

channelKeys / channel-rainbow.json (compatible)

Both engines load channel encryption keys:

• From channelKeys in config.json (inline map)
• From channel-rainbow.json next to config.json
• Go also supports CHANNEL_KEYS_PATH env var and channelKeysPath config field

No changes needed.

cacheTTL (compatible)

Both engines read cacheTTL from config. Go serves the same values via /api/config/cache.

Go-only config fields

Field	Description	Default
dbPath	SQLite path (also settable via DB_PATH env var)	data/meshcore.db
logLevel	Ingestor log verbosity	(unset)
channelKeysPath	Path to channel keys file	channel-rainbow.json next to config

These are optional and safe to add without breaking Node.js (Node ignores unknown fields).

---

Switch to Go

Option A: Docker Compose (recommended)

The docker-compose.yml already has a staging-go service for testing. To run Go in production:

Step 1: Build the Go image

docker compose --profile staging-go build staging-go

Or build directly:

docker build -f Dockerfile.go -t corescope-go:latest \
  --build-arg APP_VERSION=$(git describe --tags 2>/dev/null || echo unknown) \
  --build-arg GIT_COMMIT=$(git rev-parse --short HEAD 2>/dev/null || echo unknown) \
  .

Step 2: Test with staging-go first

Run the Go image on a separate port alongside your Node.js production:

# Copies your production DB to staging directory
mkdir -p ~/meshcore-staging-data
cp ~/meshcore-data/meshcore.db ~/meshcore-staging-data/
cp config.json ~/meshcore-staging-data/config.json

# Start the Go staging container (port 82 by default)
docker compose --profile staging-go up -d staging-go

Verify at http://your-server:82 — see Verification below.

Step 3: Switch production to Go

Once satisfied, update docker-compose.yml to use the Go image for prod:

services:
  prod:
    image: corescope-go:latest          # was: corescope:latest
    build:
      context: .
      dockerfile: Dockerfile.go        # add this
    # ... everything else stays the same

Then rebuild and restart:

docker compose build prod
docker compose up -d prod

Option B: manage.sh (legacy single-container)

⚠️ manage.sh does not currently support an --engine flag. You must manually switch the image.

# Stop the current container
./manage.sh stop

# Build the Go image
docker build -f Dockerfile.go -t corescope:latest .

# Start (manage.sh uses the corescope:latest image)
./manage.sh start

Note: This replaces the Node.js image tag. To switch back, you'll need to rebuild from Dockerfile (see Rollback).

---

DB Compatibility

Schema

Both engines use the same v3 schema:

Table	Purpose	Shared?
nodes	Mesh nodes from adverts	✅ Both read/write
observers	MQTT feed sources	✅ Both read/write
inactive_nodes	Nodes past retention window	✅ Both read/write
transmissions	Deduplicated packets	✅ Both read/write
observations	Per-observer sightings	✅ Both read/write
_migrations	One-time migration tracking	✅ Both read/write

Can Go read a Node.js DB?

Yes. The Go ingestor and server open existing v3 databases with no issues. If the database is pre-v3 (no observations table), Go creates it automatically using the same v3 schema.

Can Node.js read a Go-modified DB?

Yes. Go writes the same schema and data formats. You can switch back to Node.js and it will read the DB normally.

SQLite WAL mode

Both engines use WAL (Write-Ahead Logging) mode for concurrent access. The Go image runs two processes (ingestor + server) writing to the same DB file — same as Node.js running a single process.

Migration on first run

When Go opens a database for the first time:

1. Creates missing tables (transmissions, observations, nodes, observers, inactive_nodes) with CREATE TABLE IF NOT EXISTS
2. Runs the advert_count_unique_v1 migration if not already done (recalculates advert counts)
3. Does NOT modify existing data

---

Verification

After starting the Go engine, verify it's working:

1. Check the engine field

curl -s http://localhost/api/health | jq '.engine'
# Expected: "go"

curl -s http://localhost/api/stats | jq '.engine'
# Expected: "go"

The Node.js engine does not include an engine field (or returns "node"). The Go engine always returns "engine": "go".

2. Check packet counts

curl -s http://localhost/api/stats | jq '{totalPackets, totalNodes, totalObservers}'

These should match (or be close to) your pre-migration numbers.

3. Check MQTT ingestion

# Watch container logs for MQTT messages
docker logs -f corescope-prod --tail 20

# Or use manage.sh
./manage.sh mqtt-test

You should see MQTT [source] packet: log lines as new data arrives.

4. Check the UI

Open the web UI in your browser. Navigate through:

• Nodes — list should be populated
• Packets — table should show data
• Map — markers should appear
• Live — new packets should stream via WebSocket

5. Check WebSocket

Open browser DevTools → Network → WS tab. You should see a WebSocket connection to / with periodic packet broadcasts.

---

Rollback to Node.js

If something goes wrong, switching back is straightforward:

Docker Compose

services:
  prod:
    image: corescope:latest    # back to Node.js
    # Remove the build.dockerfile line if you added it

# Rebuild Node.js image if needed
docker build -t corescope:latest .

docker compose up -d --force-recreate prod

manage.sh (legacy)

./manage.sh stop

# Rebuild Node.js image (overwrites the corescope:latest tag)
docker build -t corescope:latest .

./manage.sh start

Restore from backup (if DB issues)

./manage.sh restore ./backups/pre-go-migration

Or manually:

docker stop corescope-prod
cp backups/pre-go-migration/meshcore.db ~/meshcore-data/meshcore.db
docker start corescope-prod

---

Known Differences

Fully supported in Go

Feature	Notes
Raw packet ingestion (Format 1)	Cisien/meshcoretomqtt format — full parity
Companion bridge channel messages (Format 2)	meshcore/message/channel/<n> — full parity
Companion bridge direct messages (Format 2b)	meshcore/message/direct/<id> — full parity
Channel key decryption	AES-CTR decryption of GRP_TXT payloads — implemented
WebSocket broadcast	Real-time packet streaming to browsers
In-memory packet store	Loads all packets from DB on startup, serves from RAM
All API endpoints	Full REST API parity (see /api/health, /api/stats, etc.)
Node retention / aging	Moves stale nodes to inactive_nodes per retention.nodeDays

Not yet supported in Go

Feature	Impact	Workaround
Companion bridge advertisements	meshcore/advertisement topic not handled by Go ingestor	Users relying on companion bridge adverts must stay on Node.js or wait for Go support
Companion bridge self_info	meshcore/self_info topic not handled	Same as above — minimal impact (only affects local node identity)
packetStore.maxMemoryMB config	Read by both engines, but Go's store accounts memory differently	Review the value before upgrading — see packetStore.maxMemoryMB
Docker Hub images	Go images not published yet	Build locally with docker build -f Dockerfile.go
manage.sh --engine flag	Can't toggle engines via manage.sh	Manual image swap required (see Switch to Go)

Behavioral differences

Area	Node.js	Go
engine field in /api/health	Not present or "node"	Always "go"
MQTT URL scheme	Uses mqtt:// / mqtts:// natively	Auto-converts to tcp:// / ssl:// (transparent)
Process model	Single Node.js process (server + ingestor)	Two binaries: corescope-ingestor + corescope-server (managed by supervisord)
Memory management	Configurable via packetStore.maxMemoryMB	Same setting: caps the cold load, evicts oldest-first above it, and sets GOMEMLIMIT when that is otherwise unset
Startup time	Faster (no compilation)	Slightly slower (loads all packets from DB into memory)

---

FAQ

Can I run Go alongside Node.js?

Yes, but not writing to the same DB simultaneously across containers. SQLite supports concurrent readers but cross-container writes via mounted volumes can cause locking issues.

The recommended approach is:

1. Run Go on staging (separate DB copy, separate port)
2. Verify it works
3. Stop Node.js, switch production to Go

Do I need to change my observer configs?

No. Observers publish to MQTT topics — they don't know or care which engine is consuming the data.

Will my theme.json and customizations carry over?

Yes. The Go server reads theme.json from the data directory (same as Node.js). All CSS variable-based theming works identically since the frontend is the same.

What about the in-memory packet store size?

The Go server loads all packets from the database on startup. For large databases (100K+ packets), this may use more memory than Node.js with a configured limit. Monitor memory usage after switching.

Is the frontend different?

No. Both engines serve the exact same public/ directory. The frontend JavaScript is identical.

---

Migration Gaps (Tracked Issues)

The following gaps have been identified. Check the GitHub issue tracker for current status:

1. manage.sh has no --engine flag — Users must manually swap Docker images to switch between Node.js and Go. An --engine go|node flag would simplify this.

2. Go ingestor missing meshcore/advertisement handling — Companion bridge advertisement messages are not processed by the Go ingestor. Users who receive node advertisements via companion bridge (not raw packets) will miss node upserts.

3. Go ingestor missing meshcore/self_info handling — The local node identity topic is not processed. Low impact but breaks parity.

4. No Docker Hub publishing for Go images — Users must build locally. CI/CD pipeline should publish corescope-go:latest alongside the Node.js image.