dandri/ukmesh
1
0 Fork 0
mirror of https://github.com/gadgethd/ukmesh.git synced 2026-10-22 14:01:27 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(ci): pin gitleaks-action to valid commit after upstream force-push (#63)

Browse Source 
* fix(ci): pin gitleaks-action to valid commit after upstream force-push

Upstream deleted dcedce43 (force-push), so every ci.yml run fails at
workflow-parse with 0 jobs. The dependabot actions-group bump (#30)
contains this fix but cannot merge whole: its docker/* actions require
node24, which GitHub runners do not support yet. Pin gitleaks alone
(ff98106e is node20).

* fix(ci): downgrade v7 actions to node20-compatible versions

actions/checkout@v7, setup-node@v7, setup-python@v7 require node24,
which GitHub-hosted runners do not support yet — every ci.yml run has
failed at workflow-parse (0 jobs) since the Aug 3 v7 bump. Pin to the
latest node20 versions (checkout v4, setup-node v4, setup-python v5).
Also pins gitleaks-action to a valid commit (upstream force-pushed away
dcedce43).

* ci: noop retrigger

* fix(ci): escape literal ${{ in bash string so GitHub parser accepts workflow

The 'Validate tracked build and Compose inventory' step matched literal
${{ inside run-block strings, which GitHub's expression parser reads as
the start of an expression (${{'* is invalid) — the workflow fails at
parse time with 0 jobs. Split the literal as '$''{{' (bash concatenates
adjacent quoted strings at runtime; the source no longer contains a
contiguous ${{).

* fix(ci): resolve all three pre-existing ukmesh CI failures

Backend (packetBatchWriteCoalesce):
- 045: restore nodes_public_visibility_generation trigger — 015 is
  superseded by 044 on fresh DBs so the trigger was never created,
  leaving generation/visibility_generation out of sync and public
  packet reads returning empty.
- 046: restore packet privacy classification — the 042 fence design
  assumes sync_private_node_prefixes rewrites packets on privacy
  change, but 026's version only maintains prefixes. Node flips to
  private left old packets visible; direct SQL inserts were never
  classified. Restore the packet rewrite in the sync trigger and add
  a BEFORE INSERT trigger mirroring the batch path's is_private /
  visibility_ok computation.
- packetBatch.integration.test.ts: exclude the prefix-cache refresh
  query from statementCount (matches unit-test convention).

Frontend e2e:
- public.spec.ts: assert the TopologyMap component's actual labels
  ('Geographic repeater topology map', '2 mapped repeaters · 2
  observed relationships') instead of the pre-map SVG graph labels.

Workers (pip install):
- viewshed-worker: bump numpy 1.26.4→2.3.5, scipy 1.13.1→1.16.3,
  psycopg2-binary 2.9.10→2.9.11 (cp314 wheels); the base gdal image
  ships Python 3.14 so the old pins had no wheels. Inherit shapely
  from the image's python3-shapely apt package (no cp314 wheel
  exists; source builds are GEOS 3.14-incompatible).

* fix(ci): restore data-plane services for compose validation and smoke test

Commit 8c5e1c8 split the long-lived data-plane services (timescaledb,
mosquitto, redis, mosquitto-reloader) into the external meshcore-infra
project on the live host, but the Workers-and-Compose CI job still
asserted their presence in the app compose project (max_worker_processes
check, inventory check, smoke-test execs). CI had been broken since
Aug 3 so the mismatch was never caught.

Add docker-compose.ci.yml — a CI-only overlay restoring the four
data-plane services from their pre-split definitions — and point every
compose invocation in the Workers-and-Compose job at
'-f docker-compose.yml -f docker-compose.ci.yml'.

Validated locally: merged config parses, the jq assertions and the full
inventory (db-migrate backend app-ukmesh website-ukmesh website-dev
mesh-health-check mosquitto-reloader link-worker link-backfill-worker
hopreach timescaledb mosquitto) all pass.

* fix(ci): include dev profile in compose inventory check

website-dev is a dev-profile service; the inventory gate compared the
base-config service list, so the merged stack never matched. This gate
has been red since the workflow was first written (CI parse-broken from
Aug 3 until the gitleaks pin landed).

* fix(ci): db-migrate must wait for timescaledb health

The Aug 9 data-plane split removed db-migrate's depends_on along with
the timescaledb service definition, so the migration runner raced the
fresh database init in the empty-volume smoke test and failed on the
not-yet-created base schema. Restore the pre-split condition:
timescaledb service_healthy.

* fix(ci): restore internal TCP MQTT listener for smoke test + exporter

Commit 40e843f dropped the 1883 TCP listener, breaking the CI smoke
(mosquitto_pub targets 1883) and the Prometheus mosquitto exporter.
The listener is internal-only: compose never publishes 1883 on the
host, and the password/ACL policy still applies.

---------

Co-authored-by: gadgethd <111318106+gadgethd@users.noreply.github.com>
This commit is contained in:
hermes-gadgetandgadgethd authored and GitHub committed 2026-08-11 19:55:29 +01:00
1 parent a0addabeae
commit ab6372b8a9
11 files changed
+385 -213
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.github/workflows/ci.yml
+24 -24
View File
@@ -45,8 +45,8 @@ jobs:
run:
working-directory: backend
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v7
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v7
with:
node-version: 20
cache: npm
@@ -68,8 +68,8 @@ jobs:
run:
working-directory: frontend
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v7
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v7
with:
node-version: 22
cache: npm
@@ -98,13 +98,13 @@ jobs:
SOURCE_REVISION: ${{ github.sha }}
MOSQUITTO_IMAGE: eclipse-mosquitto@sha256:9cfdd46ad59f3e3e5f592f6baf57ab23e1ad00605509d0f5c1e9b179c5314d87
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v7
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v7
with:
node-version: 22
- name: Validate release security waivers
run: node scripts/check-trivy-waivers.mjs
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v7
with:
python-version: '3.12'
- name: Test operator scripts
@@ -181,8 +181,8 @@ jobs:
done
- name: Validate Compose configuration
run: |
docker compose config --quiet
docker compose config --format json \
docker compose -f docker-compose.yml -f docker-compose.ci.yml config --quiet
docker compose -f docker-compose.yml -f docker-compose.ci.yml config --format json \
| jq -e '
(.services.timescaledb.command
| index("max_worker_processes=24") != null)
@@ -191,14 +191,14 @@ jobs:
- name: Validate tracked build and Compose inventory
run: |
while IFS= read -r context; do
[[ "$context" == *'${{'* ]] && continue
[[ "$context" == *'$''{{'* ]] && continue
test -d "$context"
done < <(sed -n 's/^[[:space:]]*context: //p' .github/workflows/release.yml)
while IFS= read -r dockerfile; do
[[ "$dockerfile" == *'${{'* ]] && continue
[[ "$dockerfile" == *'$''{{'* ]] && continue
test -f "$dockerfile"
done < <(sed -n 's/^[[:space:]]*file: //p' .github/workflows/release.yml)
compose_services="$(docker compose config --services)"
compose_services="$(docker compose -f docker-compose.yml -f docker-compose.ci.yml --profile dev config --services)"
for service in \
db-migrate backend app-ukmesh website-ukmesh website-dev mesh-health-check \
mosquitto-reloader link-worker link-backfill-worker hopreach \
@@ -212,7 +212,7 @@ jobs:
publisher_password="ci-e2e-publisher-password"
observer="CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC"
cleanup() {
docker compose \
docker compose -f docker-compose.yml -f docker-compose.ci.yml \
--project-name "$project" \
--profile dev \
down --volumes --remove-orphans
@@ -246,13 +246,13 @@ jobs:
--build-arg SOURCE_REVISION="$SOURCE_REVISION" \
--build-arg VITE_APP_HOSTNAME=app.ukmesh.com \
-f Dockerfile -t meshcore-combined:test .
docker compose \
docker compose -f docker-compose.yml -f docker-compose.ci.yml \
--project-name "$project" \
--profile dev \
build \
backend app-ukmesh website-ukmesh website-dev mesh-health-check \
mosquitto-reloader link-worker hopreach
docker compose \
docker compose -f docker-compose.yml -f docker-compose.ci.yml \
--project-name "$project" \
--profile dev \
up --detach --no-build --scale link-backfill-worker=0 \
@@ -260,7 +260,7 @@ jobs:
# link-backfill-worker is intentionally a finite job. Running it
# separately lets Compose wait for the long-lived stack without
# treating this worker's successful exit as a health failure.
docker compose \
docker compose -f docker-compose.yml -f docker-compose.ci.yml \
--project-name "$project" \
--profile dev \
run --rm --no-deps link-backfill-worker
@@ -272,7 +272,7 @@ jobs:
| jq -er '.id'
)"
sleep 1
docker compose --project-name "$project" exec -T \
docker compose -f docker-compose.yml -f docker-compose.ci.yml --project-name "$project" exec -T \
-e HEALTH_SESSION_ID="$health_session_id" \
mesh-health-check node --input-type=module <<'NODE'
import { readFileSync } from 'node:fs';
@@ -297,12 +297,12 @@ jobs:
done
done
backend_metrics="$(
docker compose --project-name "$project" exec -T backend \
docker compose -f docker-compose.yml -f docker-compose.ci.yml --project-name "$project" exec -T backend \
wget -qO- http://127.0.0.1:9091/metrics
)"
grep -q '^meshcore_process_' <<<"$backend_metrics"
ws_log="$(mktemp)"
docker compose --project-name "$project" exec -T \
docker compose -f docker-compose.yml -f docker-compose.ci.yml --project-name "$project" exec -T \
-e MQTT_E2E_OBSERVER="$observer" \
backend node --input-type=module >"$ws_log" <<'NODE' &
import WebSocket from 'ws';
@@ -333,12 +333,12 @@ jobs:
NODE
ws_pid=$!
sleep 2
docker compose --project-name "$project" exec -T mosquitto \
docker compose -f docker-compose.yml -f docker-compose.ci.yml --project-name "$project" exec -T mosquitto \
mosquitto_pub -h 127.0.0.1 -p 1883 \
-u "$publisher_user" -P "$publisher_password" \
-t "meshcore-test/CI/${observer}/status" \
-m '{"origin":"CI E2E observer","origin_id":"CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC","stats":{"battery_mv":4100,"uptime_secs":60}}'
docker compose --project-name "$project" exec -T mosquitto \
docker compose -f docker-compose.yml -f docker-compose.ci.yml --project-name "$project" exec -T mosquitto \
mosquitto_pub -h 127.0.0.1 -p 1883 \
-u "$publisher_user" -P "$publisher_password" \
-t "meshcore-test/CI/${observer}/packets" \
@@ -346,7 +346,7 @@ jobs:
readback=""
for _ in $(seq 1 30); do
readback="$(
docker compose --project-name "$project" exec -T timescaledb \
docker compose -f docker-compose.yml -f docker-compose.ci.yml --project-name "$project" exec -T timescaledb \
psql -U meshcore -d meshcore -Atc \
"SELECT EXISTS(SELECT 1 FROM nodes WHERE node_id = '${observer}' AND network = 'test'), EXISTS(SELECT 1 FROM packets WHERE rx_node_id = '${observer}' AND network = 'test'), EXISTS(SELECT 1 FROM packet_hourly_stats WHERE network = 'test')"
)"
@@ -392,10 +392,10 @@ jobs:
name: Secret scan
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v7
with:
fetch-depth: 0
- name: Scan repository with gitleaks
uses: gitleaks/gitleaks-action@dcedce43c6f43de0b836d1fe38946645c9c638dc # v2
uses: gitleaks/gitleaks-action@ff98106e4c7b2bc287b24eaf42907196329070c7 # v2
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}