diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index de28629..3f06f91 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -393,6 +393,8 @@ jobs: run: scripts/test-vacuum-maintenance.sh - name: Test immutable release rollback and schema compatibility gates run: scripts/test-replace-container.sh + - name: Test split infrastructure resolver dry run + run: scripts/test-infra-compose.sh secrets: name: Secret scan diff --git a/HANDOFF.md b/HANDOFF.md index a261068..da34731 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -1,76 +1,38 @@ -# B1 owner baseline auto-rebaseline handoff +# Mission C5a handoff -## Status +Status: done +Branch: `fix/ukmesh-highs-a` +Push state: not pushed; coordinator relays this branch. -The refresh-script dependency is resolved. The reviewed script and operations -documentation from `4be9542` are now on this branch as cherry-pick `7f8f4b8`. -The script MD5 is `5479d7aa1fee57d4f81d35e4fdf69699`, matching the requested -deployed version. The parser's expected summary format matches the checked-in -script's validation output emitter, and a fixture-based parser test covers that -format. The fixture uses synthetic values; the full refresh validation pass has -not been run. +## Commits -## Branch and commits +- `865b5dd` — `fix(api): allowlist public node status fields` +- `794ee7d` — `fix(ops): resolve split infrastructure containers exactly` -- Branch: `fix/baseline-auto-rebaseline` (initial B1 base: `d44ef5e`). -- Prior B1 commits: `4325f98` (`feat(ops): auto-rebaseline owner ACL inventory`) - and `ead7ee9` (`docs(ops): handoff owner baseline autorebaseline`). -- Dependency commit: `7f8f4b8`, cherry-picked from reviewed source commit - `4be9542` (`ops: add reviewed owner-authorization baseline refresh script`). -- This B1b handoff, evidence, and parser fixture update is committed separately. -- Push state: not pushed; coordinator relays the branch. +## Changes -## Files changed +- #42: `/node-status/latest` and `/mqtt-nodes` return explicit public fields. Raw status JSON, hardware/firmware details, and unknown fields are omitted. Existing historical rows remain unchanged; no database sanitisation is needed because these public responses now use field-by-field projections. Owner-authenticated status handling is unchanged. +- #46: Added a shared resolver for exact `timescaledb`, `redis`, and `mosquitto` services in an external Compose project/file. Backup, release migration verification, provisioning, network relabeling, and chunk maintenance use it. Backup archives the Compose file it actually selected. The isolated restore drill continues to target its generated disposable containers on a private network. +- Updated the privacy and operations runbooks and added a non-destructive resolver dry-run to CI. -- `backend/src/owner/autoRebaseline.ts` — readiness decisions and refresh - validation summary parsing. -- `backend/src/tools/autoRebaseline.ts` — read-only check and timer wrapper. -- `deploy/systemd/user/owner-baseline-autorebaseline.service` and - `deploy/systemd/user/owner-baseline-autorebaseline.timer` — user service and - ten-minute timer units. -- `INSTALL.md` — VPS install, verification, and disable instructions for Ben. -- `scripts/refresh-owner-baseline.sh` — reviewed validation and optional apply - script; executable bit preserved. -- `docs/operations.md` — reviewed owner-baseline refresh procedure. -- `backend/src/owner/autoRebaseline.test.ts` — fixture-backed validation output - parser test. -- `backend/src/owner/fixtures/refresh-owner-baseline-output.txt` — sample - stdout matching the script's summary emitter; fixture values are synthetic. -- `evidence/owner-baseline-autorebaseline-check.txt` — latest read-only check - output. -- `HANDOFF.md` — this report. +## Tests and checks -## Tests and evidence +- `cd backend && TMPDIR="$OLDPWD/.tmp-c5a" node --import tsx --test src/api/routes/misc.test.ts src/api/routes/nodeStatus.test.ts` — passed, 3/3. +- `cd backend && npm run typecheck` — passed. +- `TMPDIR="$PWD/.tmp-c5a" npm --prefix scripts test --cache="$PWD/.tmp-c5a/npm-cache"` — passed (observer-key, resolver dry-run, and newuser tests). +- `TMPDIR="$PWD/.tmp-c5a" scripts/test-replace-container.sh` — passed. +- `bash -n` on modified shell scripts and `git diff --check` — passed. +- `scripts/test-vacuum-maintenance.sh` was not run because it starts a Docker test container, prohibited by the VPS constraints. CI retains this existing integration test. -- Focused test: `cd backend && node --import tsx --test src/owner/autoRebaseline.test.ts` — passed, 6 tests. -- Backend type check: `npm --prefix backend run typecheck` — passed. -- Backend build: `npm --prefix backend run build` — passed. -- Read-only check: `node backend/dist/tools/autoRebaseline.js --check` — passed - with `action=skip`, `reason=owner-baseline-current`, `mismatches=[]`, - `refreshScriptAvailable=true`, `scriptValidation=not-run-in-check-mode`, and - `applied=false`. +Backend and scripts dependencies were installed from their lockfiles with npm caches inside this worktree. -Exact `--check` output: +## Gated items for Ben -```json -{ - "mode": "check", - "action": "skip", - "reason": "owner-baseline-current", - "mismatches": [], - "refreshScriptAvailable": true, - "scriptValidation": "not-run-in-check-mode", - "applied": false -} -``` +- Run the real encrypted backup and isolated restore drill on an approved operations host before relying on the updated backup chain. Follow [the backup/restore runbook](docs/runbook-backup-restore.md). +- Deployment and any production migration remain for the coordinator/Ben. No live database writes or service/container operations were performed here. -No refresh script invocation, `--apply`, service/container action, timer -installation, or timer enablement was performed. +## Workspace note -## Ben-gated next steps +The temporary npm cache directory `.tmp-c5a/` remains untracked in this worktree. The cleanup command was rejected by the execution guard with: “rm -f style commands are not permitted. Use a safer approach”. -- Run the full no-apply refresh validation pass and review its result. This was - not run here because the brief gates it to Ben; the script's validation - requires starting a one-shot container and writes a candidate baseline. -- After review, install and enable the user timer using the instructions in - `INSTALL.md`. No unit was installed or enabled by this task. +OpenViking and Tokensave tools were not exposed in this session, so shared-memory and semantic-index actions were unavailable. diff --git a/backend/src/api/publicStatus.ts b/backend/src/api/publicStatus.ts new file mode 100644 index 0000000..99bcdd5 --- /dev/null +++ b/backend/src/api/publicStatus.ts @@ -0,0 +1,62 @@ +/** Approved fields for the unauthenticated node status response contracts. */ +export type PublicStatusCounters = { + battery_mv: number | null; + uptime_secs: number | null; + tx_air_secs: number | null; + rx_air_secs: number | null; + channel_utilization: number | null; + air_util_tx: number | null; +}; + +export type PublicLatestNodeStatusDTO = PublicStatusCounters & { + time: string; + node_id: string; + network: string | null; + name: string | null; + iata: string | null; +}; + +export type PublicMqttNodeStatusDTO = PublicStatusCounters & { + node_id: string; + name: string | null; + last_seen: string; + packets_24h: string; +}; + +type PublicStatusCounterSource = Partial; + +function publicStatusCounters(row: PublicStatusCounterSource): PublicStatusCounters { + return { + battery_mv: row.battery_mv ?? null, + uptime_secs: row.uptime_secs ?? null, + tx_air_secs: row.tx_air_secs ?? null, + rx_air_secs: row.rx_air_secs ?? null, + channel_utilization: row.channel_utilization ?? null, + air_util_tx: row.air_util_tx ?? null, + }; +} + +export function toPublicLatestNodeStatusDTO( + row: PublicLatestNodeStatusDTO, +): PublicLatestNodeStatusDTO { + return { + time: row.time, + node_id: row.node_id, + network: row.network, + name: row.name, + iata: row.iata, + ...publicStatusCounters(row), + }; +} + +export function toPublicMqttNodeStatusDTO( + row: PublicMqttNodeStatusDTO, +): PublicMqttNodeStatusDTO { + return { + node_id: row.node_id, + name: row.name, + last_seen: row.last_seen, + ...publicStatusCounters(row), + packets_24h: row.packets_24h, + }; +} diff --git a/backend/src/api/routes/misc.test.ts b/backend/src/api/routes/misc.test.ts index 5b167a1..c948e98 100644 --- a/backend/src/api/routes/misc.test.ts +++ b/backend/src/api/routes/misc.test.ts @@ -37,3 +37,64 @@ test('channel feed history route forwards its bounded channel and scope', async server.close(); } }); + +test('public MQTT node status omits owner-only and unknown telemetry fields', async () => { + const storedRow = { + node_id: 'b'.repeat(64), + name: 'Public MQTT node', + last_seen: '2026-09-23T12:00:00.000Z', + battery_mv: 3900, + uptime_secs: 4200, + channel_utilization: 8, + air_util_tx: 4, + rx_air_secs: 18, + tx_air_secs: 9, + packets_24h: '12', + hardware_model: 'private model', + firmware_version: 'private firmware', + stats: { + wifi_ssid: 'private wifi', + reset_reason: 'private reset detail', + fs_free_bytes: 1234, + mqtt: { broker_uri: 'mqtts://private', broker_username: 'private-user' }, + future_owner_field: 'private future value', + }, + future_status_field: 'private future value', + }; + const router = Router(); + registerMiscRoutes(router, { + query: async (text) => { + assert.doesNotMatch(text, /nss\.stats/); + return { rows: [storedRow as unknown as T] }; + }, + getRecentPackets: async () => [], + getRecentPacketEvents: async () => [], + getPacketDetail: async () => null, + getChannelMessageHistory: async () => [], + getPublicVisibilityGeneration: async () => 1, + packetDetailLimiter: passThroughLimiter as Parameters[1]['packetDetailLimiter'], + }); + const app = express(); + app.use(router); + const server = app.listen(0, '127.0.0.1'); + await once(server, 'listening'); + try { + const { port } = server.address() as AddressInfo; + const response = await fetch(`http://127.0.0.1:${port}/mqtt-nodes?network=ukmesh`); + assert.equal(response.status, 200); + assert.deepEqual(await response.json(), [{ + node_id: storedRow.node_id, + name: storedRow.name, + last_seen: storedRow.last_seen, + battery_mv: storedRow.battery_mv, + uptime_secs: storedRow.uptime_secs, + channel_utilization: storedRow.channel_utilization, + air_util_tx: storedRow.air_util_tx, + rx_air_secs: storedRow.rx_air_secs, + tx_air_secs: storedRow.tx_air_secs, + packets_24h: storedRow.packets_24h, + }]); + } finally { + server.close(); + } +}); diff --git a/backend/src/api/routes/misc.ts b/backend/src/api/routes/misc.ts index 1d6aa66..ddce527 100644 --- a/backend/src/api/routes/misc.ts +++ b/backend/src/api/routes/misc.ts @@ -16,6 +16,7 @@ import { decodePlannedNodeCursor, listPublicPlannedNodes, } from '../../repositories/plannedNodes.js'; +import { toPublicMqttNodeStatusDTO } from '../publicStatus.js'; type QueryFn = ( text: string, @@ -218,7 +219,6 @@ export function registerMiscRoutes(router: Router, deps: MiscRouteDeps): void { air_util_tx: number | null; rx_air_secs: number | null; tx_air_secs: number | null; - stats: Record | null; packets_24h: string; }>( @@ -232,7 +232,6 @@ export function registerMiscRoutes(router: Router, deps: MiscRouteDeps): void { nss.air_util_tx, nss.rx_air_secs, nss.tx_air_secs, - nss.stats, COALESCE(pc.packet_count, 0) AS packets_24h FROM node_identity_status_samples nss LEFT JOIN node_identity_nodes n ON n.node_id = nss.node_id @@ -250,7 +249,9 @@ export function registerMiscRoutes(router: Router, deps: MiscRouteDeps): void { ORDER BY nss.node_id, COALESCE(nss.uptime_secs, 0) DESC, nss.time DESC`, params, ); - res.json(result.rows.filter((r) => Number(r.packets_24h) > 0)); + res.json(result.rows + .filter((r) => Number(r.packets_24h) > 0) + .map(toPublicMqttNodeStatusDTO)); } catch (err) { console.error('[api] GET /mqtt-nodes', (err as Error).message); res.status(500).json({ error: 'Internal server error' }); diff --git a/backend/src/api/routes/nodeStatus.test.ts b/backend/src/api/routes/nodeStatus.test.ts new file mode 100644 index 0000000..f5c29d1 --- /dev/null +++ b/backend/src/api/routes/nodeStatus.test.ts @@ -0,0 +1,43 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; +import { toPublicLatestNodeStatusDTO } from '../publicStatus.js'; + +test('latest node status DTO omits owner-only and unknown telemetry fields', () => { + const storedRow = { + time: '2026-09-23T12:00:00.000Z', + node_id: 'a'.repeat(64), + network: 'ukmesh', + name: 'Public node', + iata: 'NCL', + battery_mv: 3910, + uptime_secs: 7200, + tx_air_secs: 14, + rx_air_secs: 28, + channel_utilization: 7, + air_util_tx: 3, + hardware_model: 'private model', + firmware_version: 'private firmware', + stats: { + wifi_ssid: 'private wifi', + reset_reason: 'private reset detail', + fs_free_bytes: 1234, + mqtt: { broker_uri: 'mqtts://private', broker_username: 'private-user' }, + future_owner_field: 'private future value', + }, + future_status_field: 'private future value', + }; + + assert.deepEqual(toPublicLatestNodeStatusDTO(storedRow), { + time: storedRow.time, + node_id: storedRow.node_id, + network: storedRow.network, + name: storedRow.name, + iata: storedRow.iata, + battery_mv: storedRow.battery_mv, + uptime_secs: storedRow.uptime_secs, + tx_air_secs: storedRow.tx_air_secs, + rx_air_secs: storedRow.rx_air_secs, + channel_utilization: storedRow.channel_utilization, + air_util_tx: storedRow.air_util_tx, + }); +}); diff --git a/backend/src/api/routes/nodeStatus.ts b/backend/src/api/routes/nodeStatus.ts index 095642c..df881dc 100644 --- a/backend/src/api/routes/nodeStatus.ts +++ b/backend/src/api/routes/nodeStatus.ts @@ -7,6 +7,10 @@ import { parseBoundedInteger, parseBoundedString, } from '../utils/input.js'; +import { + toPublicLatestNodeStatusDTO, + type PublicLatestNodeStatusDTO, +} from '../publicStatus.js'; const router = Router(); @@ -26,28 +30,11 @@ router.get('/node-status/latest', async (req, res) => { } const identityWhereClause = `WHERE ${identityConditions.join(' AND ')}`; - const result = await query<{ - time: string; - node_id: string; - network: string | null; - battery_mv: number | null; - uptime_secs: number | null; - tx_air_secs: number | null; - rx_air_secs: number | null; - channel_utilization: number | null; - air_util_tx: number | null; - stats: Record | null; - name: string | null; - iata: string | null; - hardware_model: string | null; - firmware_version: string | null; - }>( + const result = await query( `WITH identity_sources AS MATERIALIZED ( SELECT n.node_id AS canonical_id, n.name, n.iata, - n.hardware_model, - n.firmware_version, source.source_node_id FROM node_identity_nodes n CROSS JOIN LATERAL unnest(n.identity_source_ids) AS source(source_node_id) @@ -63,15 +50,12 @@ router.get('/node-status/latest', async (req, res) => { nss.rx_air_secs, nss.channel_utilization, nss.air_util_tx, - nss.stats, s.name, - s.iata, - s.hardware_model, - s.firmware_version + s.iata FROM identity_sources s JOIN LATERAL ( SELECT time, network, battery_mv, uptime_secs, tx_air_secs, - rx_air_secs, channel_utilization, air_util_tx, stats + rx_air_secs, channel_utilization, air_util_tx FROM node_status_samples WHERE node_id = s.source_node_id AND network = ANY($1::text[]) @@ -85,7 +69,7 @@ router.get('/node-status/latest', async (req, res) => { params, ); - res.json(result.rows); + res.json(result.rows.map(toPublicLatestNodeStatusDTO)); } catch (err) { console.error('[api] GET /node-status/latest', (err as Error).message); res.status(500).json({ error: 'Internal server error' }); diff --git a/docs/privacy.md b/docs/privacy.md index 14f3e30..ac70e40 100644 --- a/docs/privacy.md +++ b/docs/privacy.md @@ -64,7 +64,23 @@ The service stores radio traffic observed on the MeshCore network. deletion procedure in the ops playbook (rows by src/rx node id across packets, packet_paths, decryptions, path facts). -## 3. Cookies +## 3. Device status telemetry + +Status samples retain a JSON telemetry object for the authenticated owner +portal. The unauthenticated `/node-status/latest` and `/mqtt-nodes` APIs expose +only their explicit public fields: node identity, sample/availability data, +battery, uptime, airtime, channel utilization, and recent packet counts. They +do not return the telemetry object or forward unrecognized fields. Wi-Fi +identifiers, MQTT broker/account details, reset and configuration fields, +firmware metadata, and filesystem details are available only through +owner-authenticated endpoints. Unrecognized keys stay in storage until they +are deliberately added to a response contract. + +Existing status samples do not need a database rewrite: public responses are +projected through the same field allowlist regardless of when the sample was +stored. + +## 4. Cookies LocalStorage-only choices (cookie consent, theme). No third-party cookies. PECR does not apply to the mesh radio data (not a public electronic diff --git a/docs/runbook-backup-restore.md b/docs/runbook-backup-restore.md index a473b3a..df8cb1c 100644 --- a/docs/runbook-backup-restore.md +++ b/docs/runbook-backup-restore.md @@ -59,6 +59,12 @@ export MESHCORE_INFRA_PROJECT_NAME=meshcore-infra scripts/backup.sh ``` +`MESHCORE_INFRA_COMPOSE_FILE` can select a non-default external Compose file; +`MESHCORE_INFRA_PROJECT_DIR` can set its project directory explicitly. The +shared resolver checks that Compose project and requires one running container +for each exact `timescaledb`, `redis`, and `mosquitto` service. It does not +search container names across projects. + For the returned receipt and archive: ```bash diff --git a/docs/runbook-release-rollback.md b/docs/runbook-release-rollback.md index d411347..aeeba7c 100644 --- a/docs/runbook-release-rollback.md +++ b/docs/runbook-release-rollback.md @@ -45,6 +45,8 @@ export COSIGN_CERTIFICATE_OIDC_ISSUER='https://token.actions.githubusercontent.c export RESTORE_RECEIPT_PATH=/secure/receipts/latest.json export RESTORE_RECEIPT_VERIFY_KEY=/secure/receipts/verify.pem export RELEASE_RECEIPT_SIGNING_KEY=/secure/release/receipt-signing.pem +export MESHCORE_INFRA_DIR=/home/ben/ukmesh/meshcore-infra +export MESHCORE_INFRA_PROJECT_NAME=meshcore-infra scripts/replace-container.sh backend \ --image="$BACKEND_IMAGE" \ @@ -52,6 +54,13 @@ scripts/replace-container.sh backend \ --source-revision="$SOURCE_REVISION" ``` +If the infrastructure Compose file has another path, set +`MESHCORE_INFRA_COMPOSE_FILE` and, when needed, +`MESHCORE_INFRA_PROJECT_DIR`. The release script validates that external +project and resolves its single running `timescaledb` container before it +starts the migration job; the post-migration schema check uses that exact +container ID. + Repeat for each application service, passing its digest as `--image` and the same signed backend digest as `--backend-image`. The script verifies signatures and revision labels, runs migrations, starts the prior backend against the diff --git a/scripts/backup.sh b/scripts/backup.sh index 38ef029..cb0bec0 100755 --- a/scripts/backup.sh +++ b/scripts/backup.sh @@ -5,8 +5,18 @@ umask 077 script_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" project_dir="$(cd -- "${script_dir}/.." && pwd)" app_project_name="${COMPOSE_PROJECT_NAME:-meshcore-analytics}" -infra_project_dir="${MESHCORE_INFRA_DIR:-${project_dir}/../meshcore-infra}" +if [ -n "${MESHCORE_INFRA_COMPOSE_FILE:-}" ]; then + infra_compose_file="$MESHCORE_INFRA_COMPOSE_FILE" + infra_project_dir="${MESHCORE_INFRA_PROJECT_DIR:-${MESHCORE_INFRA_DIR:-$(dirname -- "$infra_compose_file")}}" +else + infra_project_dir="${MESHCORE_INFRA_PROJECT_DIR:-${MESHCORE_INFRA_DIR:-${project_dir}/../meshcore-infra}}" + infra_compose_file="$infra_project_dir/docker-compose.yml" +fi infra_project_name="${MESHCORE_INFRA_PROJECT_NAME:-meshcore-infra}" +export MESHCORE_INFRA_PROJECT_DIR="$infra_project_dir" +export MESHCORE_INFRA_COMPOSE_FILE="$infra_compose_file" +export MESHCORE_INFRA_PROJECT_NAME="$infra_project_name" +source "$script_dir/lib/infra-compose.sh" output_dir="${BACKUP_OUTPUT_DIR:?BACKUP_OUTPUT_DIR must name an encrypted backup target}" encryption_cert="${BACKUP_ENCRYPTION_CERT:?BACKUP_ENCRYPTION_CERT must name a public X.509 certificate}" signing_key="${BACKUP_RECEIPT_SIGNING_KEY:?BACKUP_RECEIPT_SIGNING_KEY must name the offline receipt signing key}" @@ -29,8 +39,8 @@ for path in "$encryption_cert" "$signing_key"; do exit 66 } done -test -f "$infra_project_dir/docker-compose.yml" || { - echo "meshcore-infra compose file is missing: $infra_project_dir/docker-compose.yml" >&2 +test -f "$infra_compose_file" || { + echo "infrastructure Compose file is missing: $infra_compose_file" >&2 exit 66 } openssl x509 -in "$encryption_cert" -noout >/dev/null @@ -66,30 +76,27 @@ if (( (8#$env_mode & 8#077) != 0 )); then echo "protected runtime configuration .env must not be group/world accessible" >&2 exit 66 fi +infra_env_file="$infra_project_dir/.env" +if [ ! -f "$infra_env_file" ] || [ -L "$infra_env_file" ]; then + echo "protected infrastructure configuration .env is missing or unsafe" >&2 + exit 66 +fi +infra_env_mode="$(stat -c '%a' "$infra_env_file")" +if (( (8#$infra_env_mode & 8#077) != 0 )); then + echo "protected infrastructure configuration .env must not be group/world accessible" >&2 + exit 66 +fi app_compose=( docker compose --project-directory "$project_dir" -f "$project_dir/docker-compose.yml" -f "$project_dir/docker-compose.live.yml" --project-name "$app_project_name" ) -infra_compose=( - docker compose --project-directory "$infra_project_dir" - -f "$infra_project_dir/docker-compose.yml" - --project-name "$infra_project_name" -) "${app_compose[@]}" config -q -"${infra_compose[@]}" config -q +meshcore_infra_compose config -q declare -A service_container_ids=() for service in timescaledb redis mosquitto; do - container_id="$("${infra_compose[@]}" ps -q "$service")" - test -n "$container_id" || { - echo "required service is not running: $service" >&2 - exit 69 - } - if [ "$(docker inspect "$container_id" --format '{{.State.Status}}')" != "running" ]; then - echo "required service is not running: $service" >&2 - exit 69 - fi + container_id="$(resolve_infra_container "$service")" service_container_ids["$service"]="$container_id" done timescaledb_container="${service_container_ids[timescaledb]}" @@ -190,8 +197,15 @@ tar -C "$project_dir" -czf "$payload_dir/configuration.tgz" \ logging \ backend/src/db/migrations backend/src/db/schema backend/src/db/owner-auth.sql \ scripts/backup.sh scripts/restore-drill.sh scripts/sync-latest.sh scripts/replace-container.sh \ + scripts/lib/infra-compose.sh scripts/unify-networks.sh scripts/relabel-packets-per-chunk.sh \ scripts/bootstrap-mosquitto.sh vacuum-compressed-chunks.sh -tar -C "$infra_project_dir" -czf "$payload_dir/infra-configuration.tgz" \ +infra_configuration_stage="$tmp_dir/infra-configuration" +mkdir -p "$infra_configuration_stage" +cp -- "$infra_project_dir/.env" "$infra_configuration_stage/.env" +cp -- "$infra_compose_file" "$infra_configuration_stage/docker-compose.yml" +cp -- "$infra_project_dir/README.md" "$infra_configuration_stage/README.md" +cp -- "$infra_project_dir/ROLLBACK.md" "$infra_configuration_stage/ROLLBACK.md" +tar -C "$infra_configuration_stage" -czf "$payload_dir/infra-configuration.tgz" \ .env docker-compose.yml README.md ROLLBACK.md completed_at="$(date -u +%Y-%m-%dT%H:%M:%S.000Z)" diff --git a/scripts/lib/infra-compose.sh b/scripts/lib/infra-compose.sh new file mode 100644 index 0000000..c76cf2d --- /dev/null +++ b/scripts/lib/infra-compose.sh @@ -0,0 +1,70 @@ +#!/usr/bin/env bash + +meshcore_infra_compose() { + local project_name="${MESHCORE_INFRA_PROJECT_NAME:-meshcore-infra}" + local project_dir="${MESHCORE_INFRA_PROJECT_DIR:-${MESHCORE_INFRA_DIR:-}}" + local compose_file="${MESHCORE_INFRA_COMPOSE_FILE:-}" + + if [ -z "$compose_file" ]; then + if [ -z "$project_dir" ]; then + echo 'MESHCORE_INFRA_PROJECT_DIR or MESHCORE_INFRA_COMPOSE_FILE is required' >&2 + return 64 + fi + compose_file="$project_dir/docker-compose.yml" + fi + if [ -z "$project_dir" ]; then + project_dir="$(dirname -- "$compose_file")" + fi + if [ ! -f "$compose_file" ]; then + echo "infrastructure Compose file is missing: $compose_file" >&2 + return 66 + fi + + docker compose \ + --project-directory "$project_dir" \ + --project-name "$project_name" \ + -f "$compose_file" \ + "$@" +} + +resolve_infra_container() { + local service="${1:-}" + case "$service" in + timescaledb|redis|mosquitto) ;; + *) + echo "unsupported infrastructure service: ${service:-}" >&2 + return 64 + ;; + esac + + local raw_ids + if ! raw_ids="$(meshcore_infra_compose ps -q "$service")"; then + echo "could not resolve infrastructure service: $service" >&2 + return 69 + fi + + local -a ids=() + local id + while IFS= read -r id; do + id="${id//$'\r'/}" + if [ -n "${id//[[:space:]]/}" ]; then + ids+=("$id") + fi + done <<<"$raw_ids" + if [ "${#ids[@]}" -ne 1 ]; then + echo "expected one infrastructure container for $service; found ${#ids[@]}" >&2 + return 69 + fi + + local state + if ! state="$(docker inspect --format '{{.State.Status}}' "${ids[0]}")"; then + echo "could not inspect infrastructure container for $service" >&2 + return 69 + fi + if [ "$state" != 'running' ]; then + echo "infrastructure container for $service is not running" >&2 + return 69 + fi + + printf '%s\n' "${ids[0]}" +} diff --git a/scripts/newuser.sh b/scripts/newuser.sh index 88aff81..7009172 100755 --- a/scripts/newuser.sh +++ b/scripts/newuser.sh @@ -3,11 +3,22 @@ set -Eeuo pipefail IFS=$'\n\t' +readonly SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" readonly REPO_DIR="${NEWUSER_REPO_DIR:-/home/ben/ukmesh/meshcore-analytics}" readonly ENV_FILE="${REPO_DIR}/.env" -readonly MOSQUITTO_CONTAINER="$(docker ps --format '{{.Names}}' | grep -E '^meshcore-infra-mosquitto-1$|mosquitto-1$' | head -1)" +if [ -n "${MESHCORE_INFRA_COMPOSE_FILE:-}" ]; then + infra_compose_file="$MESHCORE_INFRA_COMPOSE_FILE" + infra_project_dir="${MESHCORE_INFRA_PROJECT_DIR:-${MESHCORE_INFRA_DIR:-$(dirname -- "$infra_compose_file")}}" +else + infra_project_dir="${MESHCORE_INFRA_PROJECT_DIR:-${MESHCORE_INFRA_DIR:-$(dirname -- "$REPO_DIR")/meshcore-infra}}" + infra_compose_file="$infra_project_dir/docker-compose.yml" +fi +export MESHCORE_INFRA_PROJECT_DIR="$infra_project_dir" +export MESHCORE_INFRA_COMPOSE_FILE="$infra_compose_file" +source "$SCRIPT_DIR/lib/infra-compose.sh" +readonly MOSQUITTO_CONTAINER="$(resolve_infra_container mosquitto)" readonly BACKEND_CONTAINER='meshcore-analytics-backend-1' -readonly TIMESCALEDB_CONTAINER="$(docker ps --format '{{.Names}}' | grep -E '^meshcore-infra-timescaledb-1$|timescaledb-1$' | head -1)" +readonly TIMESCALEDB_CONTAINER="$(resolve_infra_container timescaledb)" readonly OWNER_DATABASE='meshcore_owner_auth' readonly POSTGRES_USER='meshcore' readonly BROKER_URL='wss://mqtt.ukmesh.com:443' diff --git a/scripts/package.json b/scripts/package.json index fbc4ca3..72cd210 100644 --- a/scripts/package.json +++ b/scripts/package.json @@ -6,7 +6,7 @@ "scripts": { "generate-key": "tsx generate-observer-key.ts", "register-observer": "tsx register-observer.ts", - "test": "bash test-generate-observer-key.sh && bash test-newuser.sh", + "test": "bash test-generate-observer-key.sh && bash test-infra-compose.sh && bash test-newuser.sh", "typecheck": "tsc --noEmit --target ES2022 --module NodeNext --moduleResolution NodeNext --strict --types node generate-observer-key.ts register-observer.ts" }, "dependencies": { diff --git a/scripts/relabel-packets-per-chunk.sh b/scripts/relabel-packets-per-chunk.sh index 3b13867..bcc9253 100755 --- a/scripts/relabel-packets-per-chunk.sh +++ b/scripts/relabel-packets-per-chunk.sh @@ -3,11 +3,25 @@ set -Eeuo pipefail cd "$(dirname "$0")/.." +script_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +project_dir="$(cd -- "$script_dir/.." && pwd)" +if [ -n "${MESHCORE_INFRA_COMPOSE_FILE:-}" ]; then + infra_compose_file="$MESHCORE_INFRA_COMPOSE_FILE" + infra_project_dir="${MESHCORE_INFRA_PROJECT_DIR:-${MESHCORE_INFRA_DIR:-$(dirname -- "$infra_compose_file")}}" +else + infra_project_dir="${MESHCORE_INFRA_PROJECT_DIR:-${MESHCORE_INFRA_DIR:-${project_dir}/../meshcore-infra}}" + infra_compose_file="$infra_project_dir/docker-compose.yml" +fi +export MESHCORE_INFRA_PROJECT_DIR="$infra_project_dir" +export MESHCORE_INFRA_COMPOSE_FILE="$infra_compose_file" +source "$script_dir/lib/infra-compose.sh" + log() { printf '[%s] %s\n' "$(date -Is)" "$*" } -PSQL=(docker compose exec -T timescaledb psql -U "${POSTGRES_USER:-meshcore}" -d "${POSTGRES_DB:-meshcore}" -v ON_ERROR_STOP=1 -X) +timescaledb_container="$(resolve_infra_container timescaledb)" +PSQL=(docker exec -i "$timescaledb_container" psql -U "${POSTGRES_USER:-meshcore}" -d "${POSTGRES_DB:-meshcore}" -v ON_ERROR_STOP=1 -X) # `test` remains an isolated network. This maintenance operation only folds # historical production labels into the unified UKMesh scope. LEGACY_SQL="'teesside','northeast'" diff --git a/scripts/replace-container.sh b/scripts/replace-container.sh index de6586a..7a7396c 100755 --- a/scripts/replace-container.sh +++ b/scripts/replace-container.sh @@ -49,6 +49,16 @@ done script_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" project_dir="$(cd -- "${script_dir}/.." && pwd)" project_name="${COMPOSE_PROJECT_NAME:-meshcore-analytics}" +if [ -n "${MESHCORE_INFRA_COMPOSE_FILE:-}" ]; then + infra_compose_file="$MESHCORE_INFRA_COMPOSE_FILE" + infra_project_dir="${MESHCORE_INFRA_PROJECT_DIR:-${MESHCORE_INFRA_DIR:-$(dirname -- "$infra_compose_file")}}" +else + infra_project_dir="${MESHCORE_INFRA_PROJECT_DIR:-${MESHCORE_INFRA_DIR:-${project_dir}/../meshcore-infra}}" + infra_compose_file="$infra_project_dir/docker-compose.yml" +fi +export MESHCORE_INFRA_PROJECT_DIR="$infra_project_dir" +export MESHCORE_INFRA_COMPOSE_FILE="$infra_compose_file" +source "$script_dir/lib/infra-compose.sh" release_dir="${RELEASE_STATUS_DIR:-/home/ben/meshcore-releases}" cosign_key="${COSIGN_PUBLIC_KEY:-}" cosign_identity_regexp="${COSIGN_CERTIFICATE_IDENTITY_REGEXP:-}" @@ -127,6 +137,8 @@ if ! docker compose --project-name "$project_name" config --services | grep -Fxq echo "unknown Compose service: $service" >&2 exit 65 fi +meshcore_infra_compose config -q +timescaledb_container="$(resolve_infra_container timescaledb)" if [ "$(git rev-parse HEAD)" != "$source_revision" ]; then echo "working checkout does not match the release source revision" >&2 exit 65 @@ -256,7 +268,7 @@ echo "Running the required migration job with the signed backend image..." BACKEND_IMAGE="$backend_image" \ docker compose --project-name "$project_name" run --rm db-migrate schema_version="$( - docker compose --project-name "$project_name" exec -T timescaledb \ + docker exec -i "$timescaledb_container" \ psql -U "${POSTGRES_USER:-meshcore}" -d "${POSTGRES_DB:-meshcore}" -Atc \ "SELECT COALESCE(MAX(((regexp_match(name, '^([0-9]+)_'))[1])::int), 0) FROM schema_migrations" )" diff --git a/scripts/restore-drill.sh b/scripts/restore-drill.sh index 838c8e5..474f337 100755 --- a/scripts/restore-drill.sh +++ b/scripts/restore-drill.sh @@ -53,6 +53,8 @@ test "$actual_archive_sha" = "$expected_archive_sha" || { exit 65 } +# This drill uses generated container names on a private network; all restore +# and smoke commands target those disposable containers directly. run_id="meshcore-restore-$$" network_name="${run_id}-network" postgres_name="${run_id}-postgres" diff --git a/scripts/test-infra-compose.sh b/scripts/test-infra-compose.sh new file mode 100755 index 0000000..c832ca9 --- /dev/null +++ b/scripts/test-infra-compose.sh @@ -0,0 +1,107 @@ +#!/usr/bin/env bash +set -euo pipefail + +script_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +test_root="$(mktemp -d)" +trap 'rm -rf -- "$test_root"' EXIT + +mkdir -p "$test_root/bin" "$test_root/external-infra" +cat >"$test_root/external-infra/compose.production.yml" <<'YAML' +services: + timescaledb: + image: timescale/timescaledb:fixture + redis: + image: redis:fixture + mosquitto: + image: eclipse-mosquitto:fixture +YAML +cat >"$test_root/bin/docker" <<'EOF' +#!/usr/bin/env bash +set -euo pipefail +printf '%s\n' "$*" >>"$MOCK_DOCKER_LOG" + +command_name="${1:-}" +shift || true +case "$command_name" in + compose) + while [ "$#" -gt 0 ]; do + case "$1" in + --project-directory|--project-name|-f) + shift 2 + ;; + config) + exit 0 + ;; + ps) + shift + test "${1:-}" = '-q' + service="${2:-}" + if [ "${MOCK_PS_OUTPUT+x}" = x ]; then + printf '%s\n' "$MOCK_PS_OUTPUT" + else + printf 'container-%s\n' "$service" + fi + exit 0 + ;; + *) + echo "unexpected mock Compose command: $*" >&2 + exit 90 + ;; + esac + done + ;; + inspect) + if [ "${MOCK_STATE:-running}" = 'running' ]; then + printf 'running\n' + else + printf '%s\n' "$MOCK_STATE" + fi + ;; + *) + echo "unexpected mock Docker command: $command_name $*" >&2 + exit 91 + ;; +esac +EOF +chmod 0755 "$test_root/bin/docker" + +export PATH="$test_root/bin:$PATH" +export MOCK_DOCKER_LOG="$test_root/docker.log" +export MESHCORE_INFRA_PROJECT_DIR="$test_root/external-infra" +export MESHCORE_INFRA_COMPOSE_FILE="$test_root/external-infra/compose.production.yml" +export MESHCORE_INFRA_PROJECT_NAME='meshcore-infra-test' +source "$script_dir/lib/infra-compose.sh" + +meshcore_infra_compose config -q +for service in timescaledb redis mosquitto; do + resolved="$(resolve_infra_container "$service")" + test "$resolved" = "container-$service" + grep -Fq -- \ + "compose --project-directory $MESHCORE_INFRA_PROJECT_DIR --project-name $MESHCORE_INFRA_PROJECT_NAME -f $MESHCORE_INFRA_COMPOSE_FILE ps -q $service" \ + "$MOCK_DOCKER_LOG" +done + +unset MESHCORE_INFRA_PROJECT_DIR MESHCORE_INFRA_DIR +test "$(resolve_infra_container timescaledb)" = 'container-timescaledb' +grep -Fq -- \ + "compose --project-directory $(dirname -- "$MESHCORE_INFRA_COMPOSE_FILE") --project-name $MESHCORE_INFRA_PROJECT_NAME -f $MESHCORE_INFRA_COMPOSE_FILE ps -q timescaledb" \ + "$MOCK_DOCKER_LOG" + +if MOCK_PS_OUTPUT=$'container-one\ncontainer-two' resolve_infra_container timescaledb >/dev/null 2>&1; then + echo 'ambiguous service resolution unexpectedly succeeded' >&2 + exit 1 +fi +if MOCK_PS_OUTPUT='' resolve_infra_container redis >/dev/null 2>&1; then + echo 'missing service resolution unexpectedly succeeded' >&2 + exit 1 +fi +if MOCK_STATE=exited resolve_infra_container mosquitto >/dev/null 2>&1; then + echo 'stopped service resolution unexpectedly succeeded' >&2 + exit 1 +fi +if resolve_infra_container backend >/dev/null 2>&1; then + echo 'unsupported service resolution unexpectedly succeeded' >&2 + exit 1 +fi + +printf 'split infrastructure resolver dry-run tests passed\n' diff --git a/scripts/test-newuser.sh b/scripts/test-newuser.sh index fac0530..8b5e51f 100755 --- a/scripts/test-newuser.sh +++ b/scripts/test-newuser.sh @@ -9,7 +9,8 @@ readonly NODE_C='C3C3C3C3C3C3C3C3C3C3C3C3C3C3C3C3C3C3C3C3C3C3C3C3C3C3C3C3C3C3C3C test_root="$(mktemp -d)" trap 'rm -rf -- "$test_root"' EXIT -mkdir -p "$test_root/bin" "$test_root/repo" +mkdir -p "$test_root/bin" "$test_root/repo" "$test_root/infra" +touch "$test_root/infra/docker-compose.yml" printf 'OWNER_MQTT_USERNAME_MAP=alice=%s,bob=%s\n' "$NODE_A" "$NODE_C" \ >"$test_root/repo/.env" @@ -20,9 +21,6 @@ set -Eeuo pipefail printf 'ARGS %s\n' "$*" >>"$FAKE_DOCKER_LOG" case ${1:-} in - ps) - printf '%s\n' meshcore-infra-mosquitto-1 meshcore-infra-timescaledb-1 - ;; inspect) if [[ $* == *State.Running* ]]; then printf 'true\n' @@ -31,6 +29,12 @@ case ${1:-} in fi ;; compose) + case "$*" in + *'ps -q mosquitto') printf 'infra-mosquitto-id\n' ;; + *'ps -q timescaledb') printf 'infra-timescaledb-id\n' ;; + *' up -d --no-deps backend') ;; + *) printf 'unexpected docker compose invocation: %s\n' "$*" >&2; exit 1 ;; + esac ;; exec) sql='' @@ -60,6 +64,7 @@ export FAKE_DOCKER_LOG="$test_root/docker.log" output="$test_root/output.log" PATH="$test_root/bin:$PATH" \ NEWUSER_REPO_DIR="$test_root/repo" \ +MESHCORE_INFRA_PROJECT_DIR="$test_root/infra" \ TMPDIR="$test_root" \ bash "$SCRIPT_DIR/newuser.sh" --link alice "$NODE_B" >"$output" 2>&1 @@ -70,6 +75,10 @@ grep -Fq 'Password: unchanged (existing credential)' "$output" grep -Fq 'INSERT INTO owner_account_nodes' "$FAKE_DOCKER_LOG" grep -Fq 'compose -f docker-compose.yml -f docker-compose.live.yml up -d --no-deps backend' \ "$FAKE_DOCKER_LOG" +grep -Fq "compose --project-directory $test_root/infra --project-name meshcore-infra -f $test_root/infra/docker-compose.yml ps -q mosquitto" \ + "$FAKE_DOCKER_LOG" +grep -Fq "compose --project-directory $test_root/infra --project-name meshcore-infra -f $test_root/infra/docker-compose.yml ps -q timescaledb" \ + "$FAKE_DOCKER_LOG" if grep -Fq 'mosquitto_passwd -b' "$FAKE_DOCKER_LOG"; then printf 'link flow unexpectedly changed the existing credential\n' >&2 exit 1 diff --git a/scripts/test-replace-container.sh b/scripts/test-replace-container.sh index ac2af8a..1973c5d 100755 --- a/scripts/test-replace-container.sh +++ b/scripts/test-replace-container.sh @@ -61,46 +61,64 @@ command_name="${1:-}" shift || true case "$command_name" in compose) - if [ "${1:-}" = "--project-name" ]; then - shift 2 - fi - case "${1:-}" in - config) - if [ "${2:-}" = "--services" ]; then - printf '%s\n' backend db-migrate timescaledb - elif [ "${2:-}" != "-q" ]; then - printf 'services: {}\n' - fi - ;; - ps) - printf 'current-backend\n' - ;; - run) - test "${*: -1}" = "db-migrate" - ;; - exec) - if [[ " $* " == *" timescaledb "* ]]; then - printf '30\n' - elif [[ "$*" == *"/readyz"* ]]; then - printf '{"status":"ready"}\n' - elif [[ "$*" == *"/metrics"* ]]; then - printf 'meshcore_process_start_time_seconds 1\n' - else - printf 'unexpected docker compose exec invocation: %s\n' "$*" >&2 - exit 91 - fi - ;; - up) - if [ "${MOCK_FAIL_DESIRED_DEPLOY:-false}" = "true" ] \ - && [ "${BACKEND_IMAGE:-}" = "$MOCK_DESIRED_IMAGE" ]; then - exit 1 - fi - ;; - *) - printf 'unexpected docker compose invocation: %s\n' "$*" >&2 - exit 92 - ;; - esac + project_name='' + while [ "$#" -gt 0 ]; do + case "$1" in + --project-name) + project_name="$2" + shift 2 + ;; + --project-directory|-f) + shift 2 + ;; + config) + if [ "${2:-}" = "--services" ]; then + printf '%s\n' backend db-migrate + elif [ "${2:-}" != "-q" ]; then + printf 'services: {}\n' + fi + exit 0 + ;; + ps) + shift + test "${1:-}" = '-q' + service="${2:-}" + if [ "$project_name" = 'meshcore-infra' ]; then + test "$service" = 'timescaledb' + printf 'infra-timescaledb\n' + else + printf 'current-backend\n' + fi + exit 0 + ;; + run) + test "${*: -1}" = "db-migrate" + exit 0 + ;; + exec) + if [[ "$*" == *"/readyz"* ]]; then + printf '{"status":"ready"}\n' + elif [[ "$*" == *"/metrics"* ]]; then + printf 'meshcore_process_start_time_seconds 1\n' + else + printf 'unexpected docker compose exec invocation: %s\n' "$*" >&2 + exit 91 + fi + exit 0 + ;; + up) + if [ "${MOCK_FAIL_DESIRED_DEPLOY:-false}" = "true" ] \ + && [ "${BACKEND_IMAGE:-}" = "$MOCK_DESIRED_IMAGE" ]; then + exit 1 + fi + exit 0 + ;; + *) + printf 'unexpected docker compose invocation: %s\n' "$*" >&2 + exit 92 + ;; + esac + done ;; pull) ;; @@ -124,6 +142,9 @@ case "$command_name" in *'.State.Running'*) printf 'true\n' ;; + *'.State.Status'*) + printf 'running\n' + ;; *'.State.Health'*) printf 'healthy\n' ;; @@ -137,6 +158,10 @@ case "$command_name" in printf 'compat-container-id\n' ;; exec) + if [[ "$*" == *"infra-timescaledb"*"psql"* ]]; then + printf '30\n' + exit 0 + fi if [ "${MOCK_COMPAT_READY:-true}" != "true" ]; then exit 1 fi @@ -178,12 +203,15 @@ run_case() { trap 'rm -rf -- "$test_root"' RETURN mkdir -p \ - "${test_root}/project/scripts" \ + "${test_root}/project/scripts/lib" \ + "${test_root}/infra" \ "${test_root}/fake-bin" \ "${test_root}/releases" cp "$replace_script" "${test_root}/project/scripts/replace-container.sh" + cp "${script_dir}/lib/infra-compose.sh" "${test_root}/project/scripts/lib/infra-compose.sh" chmod 0755 "${test_root}/project/scripts/replace-container.sh" printf 'services: {}\n' >"${test_root}/project/docker-compose.yml" + printf 'services: {}\n' >"${test_root}/infra/docker-compose.yml" local now now="$(date -u +%Y-%m-%dT%H:%M:%SZ)" @@ -228,6 +256,7 @@ run_case() { RESTORE_RECEIPT_PATH="${test_root}/restore-receipt.json" \ RESTORE_RECEIPT_VERIFY_KEY="${test_root}/receipt-verify.pem" \ RELEASE_STATUS_DIR="${test_root}/releases" \ + MESHCORE_INFRA_PROJECT_DIR="${test_root}/infra" \ COMPATIBILITY_TIMEOUT_SECONDS=1 \ "${test_root}/project/scripts/replace-container.sh" \ backend \ @@ -252,6 +281,12 @@ run_case() { test "$(jq -r '.status' "$release_status")" = "$expected_status" test "$(jq -r '.schema_version' "$release_status")" = "30" test "$(jq -r '.prior_image' "$release_status")" = "$prior_image" + local resolver_line migration_line + resolver_line="$(grep -nF 'ps -q timescaledb' "${test_root}/docker.log" | cut -d: -f1)" + migration_line="$(grep -nF 'compose --project-name meshcore-analytics run --rm db-migrate' "${test_root}/docker.log" | cut -d: -f1)" + test "$resolver_line" -lt "$migration_line" + grep -Fq 'exec -i infra-timescaledb psql -U meshcore -d meshcore -Atc' \ + "${test_root}/docker.log" local up_count up_count="$( diff --git a/scripts/unify-networks.sh b/scripts/unify-networks.sh index 5792a10..069061d 100755 --- a/scripts/unify-networks.sh +++ b/scripts/unify-networks.sh @@ -1,10 +1,24 @@ #!/usr/bin/env bash set -Eeuo pipefail -cd "$(dirname "$0")/.." +script_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +project_dir="$(cd -- "$script_dir/.." && pwd)" +cd "$project_dir" + +if [ -n "${MESHCORE_INFRA_COMPOSE_FILE:-}" ]; then + infra_compose_file="$MESHCORE_INFRA_COMPOSE_FILE" + infra_project_dir="${MESHCORE_INFRA_PROJECT_DIR:-${MESHCORE_INFRA_DIR:-$(dirname -- "$infra_compose_file")}}" +else + infra_project_dir="${MESHCORE_INFRA_PROJECT_DIR:-${MESHCORE_INFRA_DIR:-${project_dir}/../meshcore-infra}}" + infra_compose_file="$infra_project_dir/docker-compose.yml" +fi +export MESHCORE_INFRA_PROJECT_DIR="$infra_project_dir" +export MESHCORE_INFRA_COMPOSE_FILE="$infra_compose_file" +source "$script_dir/lib/infra-compose.sh" MODE="${1:-audit}" -PSQL=(docker compose exec -T timescaledb psql -U "${POSTGRES_USER:-meshcore}" -d "${POSTGRES_DB:-meshcore}" -v ON_ERROR_STOP=1 -X) +timescaledb_container="$(resolve_infra_container timescaledb)" +PSQL=(docker exec -i "$timescaledb_container" psql -U "${POSTGRES_USER:-meshcore}" -d "${POSTGRES_DB:-meshcore}" -v ON_ERROR_STOP=1 -X) LEGACY_SQL="'teesside','northeast'" RUN_ID="${NETWORK_UNIFICATION_RUN_ID:-network-unification-$(date -u +%Y%m%dT%H%M%SZ)}" diff --git a/vacuum-compressed-chunks.sh b/vacuum-compressed-chunks.sh index abec917..459d5b5 100755 --- a/vacuum-compressed-chunks.sh +++ b/vacuum-compressed-chunks.sh @@ -59,18 +59,26 @@ if [ -n "$chunk" ]; then fi cd "$repo_dir" -# Resolve the compose project that owns the timescaledb service. The test harness -# drives the script with COMPOSE_FILE/COMPOSE_PROJECT_NAME set (its own fixture); -# in the two-project production layout the database lives in the separate -# meshcore-infra project (mirrors scripts/backup.sh). -if [ -z "${COMPOSE_FILE:-}" ]; then - infra_dir="${MESHCORE_INFRA_DIR:-${repo_dir}/../meshcore-infra}" - compose_base=(docker compose --project-directory "$infra_dir" -f "$infra_dir/docker-compose.yml" --project-name "${MESHCORE_INFRA_PROJECT_NAME:-meshcore-infra}") +if [ -n "${MESHCORE_INFRA_COMPOSE_FILE:-}" ]; then + infra_compose_file="$MESHCORE_INFRA_COMPOSE_FILE" + infra_project_dir="${MESHCORE_INFRA_PROJECT_DIR:-${MESHCORE_INFRA_DIR:-$(dirname -- "$infra_compose_file")}}" + infra_project_name="${MESHCORE_INFRA_PROJECT_NAME:-meshcore-infra}" +elif [ -n "${COMPOSE_FILE:-}" ]; then + infra_compose_file="$COMPOSE_FILE" + infra_project_dir="$(dirname -- "$infra_compose_file")" + infra_project_name="${COMPOSE_PROJECT_NAME:-meshcore-infra}" else - compose_base=(docker compose) + infra_project_dir="${MESHCORE_INFRA_PROJECT_DIR:-${MESHCORE_INFRA_DIR:-${repo_dir}/../meshcore-infra}}" + infra_compose_file="$infra_project_dir/docker-compose.yml" + infra_project_name="${MESHCORE_INFRA_PROJECT_NAME:-meshcore-infra}" fi -"${compose_base[@]}" config -q -psql_command=("${compose_base[@]}" exec -T timescaledb psql -X -v ON_ERROR_STOP=1 -U "${POSTGRES_USER:-meshcore}" -d "$database") +export MESHCORE_INFRA_PROJECT_DIR="$infra_project_dir" +export MESHCORE_INFRA_COMPOSE_FILE="$infra_compose_file" +export MESHCORE_INFRA_PROJECT_NAME="$infra_project_name" +source "$repo_dir/scripts/lib/infra-compose.sh" +meshcore_infra_compose config -q +timescaledb_container="$(resolve_infra_container timescaledb)" +psql_command=(docker exec -i "$timescaledb_container" psql -X -v ON_ERROR_STOP=1 -U "${POSTGRES_USER:-meshcore}" -d "$database") database_exists="$("${psql_command[@]}" -Atc \ "SELECT 1 FROM pg_database WHERE datname = current_database() AND NOT datistemplate")" @@ -138,7 +146,7 @@ case "$action" in ;; esac available_bytes="$( - "${compose_base[@]}" exec -T timescaledb \ + docker exec "$timescaledb_container" \ df -PB1 /var/lib/postgresql/data | awk 'NR==2 {print $4}' )"