Files
0af70cf2c1 fix(build): pin musl thread stacks at 8 MiB and check the artifacts (#2410)
## Summary

Pins musl thread stacks at 8 MiB on every link, and fixes a crash in
**native Alpine builds** (`make build`, `make test`, or `go test` on an
Alpine host).

**The released images are not affected.** An earlier version of this
description said public endpoints could crash the shipped image. That
was wrong, and review caught it: see "Who is affected".

## Root cause

- musl sizes every non-main thread's stack from the ELF `PT_GNU_STACK`
size, with a floor of 128 KiB and a cap of 8 MiB (glibc defaults to 8
MiB).
- cgo calls run on the calling thread's stack, and SQLite's
`multiSelect` recurses about 850 bytes per `UNION ALL` term.
- `lookupPrefixes` (up to 400 terms) and `sharedEightBytePrefixes` (up
to 450 terms) overflow a 128 KiB stack and SIGSEGV the whole process.

## Who is affected

| Link | `PT_GNU_STACK` without the flag | Thread stack | Wide-UNION
tests |
|---|---|---|---|
| Alpine gcc + GNU ld (`make build` / `make test` / `go test` on Alpine)
| `0` | 128 KiB | **SIGSEGV** |
| zig 0.16.0, x86_64-linux-musl (Dockerfile, `crossbuild`, release job)
| `0x1000000` (16 MiB) | 8 MiB | pass 3/3 |
| zig 0.16.0, aarch64-linux-musl | `0x1000000` (16 MiB) | 8 MiB | pass
3/3, run on an arm64 node |

So for the released images the flag pins a value that zig's default
already provides. It stops being a property of a linker default, and a
build check now fails if it is lost. For native Alpine links it is the
fix: with the flag those tests pass.

Control: a zig arm64 link with `-Wl,-z,stack-size=131072` gets 128 KiB
threads and the same tests SIGSEGV in cgo on the arm64 node.

## Fix

- Every musl link passes `-Wl,-z,stack-size=8388608`: the `Dockerfile`,
the `Makefile` (`build` and `test` on Linux, and `crossbuild`) and the
release job in `deploy.yml`.
- This reserves address space only. Memory is committed as pages are
touched.
- **The artifacts are checked.** The Dockerfile's builder stage and the
release job's "Verify release binaries" step read `PT_GNU_STACK` with
`readelf` and fail unless it is exactly 8 MiB. Verified: the builder
stage passes for `linux/amd64` and `linux/arm64`, and fails with
`--build-arg STACK_LDFLAG=` (`/corescope-server: PT_GNU_STACK is not 8
MiB`, memsz `0x1000000`).

## Testing

- `TestMuslThreadStackWideUnionQueries`
(`cmd/server/musl_thread_stack_linux_test.go`, Linux only) runs both
queries off the main thread and asserts exact results, so a failed chunk
cannot pass as "nothing matched". Under `golang:1.27-alpine` with gcc:
SIGSEGV without the flag, pass with it. On CI's glibc runners it passes
either way, which is why the artifact checks above exist.
- `tests/unit/test-issue-1956-release-routing.js`: the release verify
step must check both binaries, and a `0x000000` or `0x1000000` header
must fail the release.
- `tests/unit/test-dockerfile-buildplatform.js`: all three `go build`
lines carry `STACK_LDFLAG`, and the `readelf` check covers all three
binaries.
- `sh test-all.sh` passes. `go vet` passes for `cmd/server` on Linux
with gofmt clean.
- Not verified: a Windows `go vet`. The fix there is the
`_linux_test.go` file name; no Windows machine was available to run it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---

## Merge order

These four PRs were split out of #2396 and are **independent — they can
merge in any order.** Verified: every pair is conflict-free, and all
four merged together pass `go vet`, `go test ./...` (all 16 modules) and
`sh test-all.sh`. (That combined run predates the review fixes on #2409
and #2410.)

Suggested order, none of it urgent:

1. **#2408** — never lose rows during startup / a failed poll.
2. **#2407** — explicit `observers.idx` primary key.
3. **#2409** — content-hash migration moved into the ingestor.
4. **#2410** — pin musl thread stacks at 8 MiB. Released images are not
affected; it fixes native Alpine builds.

After these land, #2396 (Postgres) will be rebased onto master.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: efiten <erwin.fiten@gmail.com>
2026-11-18 18:35:41 +01:00

207 lines
9.4 KiB
Docker

# syntax=docker/dockerfile:1
# Build stage always runs natively on the builder's arch ($BUILDPLATFORM) and
# cross-compiles to $TARGETOS/$TARGETARCH. No QEMU for compilation.
#
# The SQLite driver is github.com/mattn/go-sqlite3, which is cgo, so the Go
# toolchain alone can no longer cross-compile this: it needs a C compiler that
# can target the other architecture. `zig cc` is that compiler. Targeting musl
# makes the result fully static (see -extldflags below), so the runtime stage
# has no libc dependency on the base image at all.
#
# BUILDPLATFORM is set automatically by BuildKit (the default builder since
# Docker 23, and what buildx uses) to the machine doing the build. Declare it
# without a default: a default here overrides BuildKit's value, so a native
# build on an arm64 host (a Raspberry Pi, Apple silicon) pulled the amd64 Go
# image and failed with "exec format error" at the first RUN.
ARG BUILDPLATFORM
FROM --platform=$BUILDPLATFORM golang:1.27-alpine AS builder
ARG APP_VERSION=unknown
ARG GIT_COMMIT=unknown
# musl sizes every non-main thread's stack from the ELF PT_GNU_STACK size
# (floor 128 KiB, cap 8 MiB). cgo calls into SQLite run on the calling thread's
# stack, and a deep query (multiSelect recursing ~850 B per UNION ALL term)
# overflows 128 KiB and SIGSEGVs the process. zig's linker already writes a
# 16 MiB PT_GNU_STACK, so these binaries got 8 MiB threads before this flag
# existed; GNU ld writes 0, which is where the crash was seen (a native Alpine
# gcc link). The flag pins the size so it no longer rests on a linker default,
# and the readelf check after the builds fails the image if it is ever lost.
# Reserved address space, not resident memory. Go rejects this flag in
# #cgo LDFLAGS, so it must go in -extldflags.
ARG STACK_LDFLAG="-Wl,-z,stack-size=8388608"
ARG BUILD_TIME=unknown
# Provided by buildx for multi-arch builds
ARG TARGETOS
ARG TARGETARCH
# Keep these in step with the Makefile: netgo/osusergo preserve the pure-Go
# resolver and user lookup the binaries had under CGO_ENABLED=0, and
# sqlite_omit_load_extension drops the dlopen path so -static links cleanly.
ENV GO_BUILD_TAGS=netgo,osusergo,sqlite_omit_load_extension \
ZIG_GLOBAL_CACHE_DIR=/tmp/zig-cache
# Pinned zig, checksum-verified. Arch comes from `uname -m` rather than
# TARGETARCH because this stage is pinned to BUILDPLATFORM — it has to work when
# someone builds on an arm64 machine too.
ARG ZIG_VERSION=0.16.0
# binutils: readelf, for the thread-stack check after the builds.
RUN apk add --no-cache curl xz binutils && \
case "$(uname -m)" in \
x86_64) ZA=x86_64; ZSHA=70e49664a74374b48b51e6f3fdfbf437f6395d42509050588bd49abe52ba3d00 ;; \
aarch64) ZA=aarch64; ZSHA=ea4b09bfb22ec6f6c6ceac57ab63efb6b46e17ab08d21f69f3a48b38e1534f17 ;; \
*) echo "unsupported builder arch $(uname -m)" >&2; exit 1 ;; \
esac && \
curl -sSLo /tmp/zig.tar.xz "https://ziglang.org/download/${ZIG_VERSION}/zig-${ZA}-linux-${ZIG_VERSION}.tar.xz" && \
echo "${ZSHA} /tmp/zig.tar.xz" | sha256sum -c - && \
mkdir -p /opt/zig && tar -xJf /tmp/zig.tar.xz -C /opt/zig --strip-components=1 && \
ln -s /opt/zig/zig /usr/local/bin/zig && rm /tmp/zig.tar.xz && \
zig version
# zigcc resolves TARGETARCH to a zig target triple once, so the three build
# steps below stay readable and cannot disagree with each other.
RUN printf '%s\n' '#!/bin/sh' \
'case "$TARGETARCH" in' \
' amd64) t=x86_64-linux-musl ;;' \
' arm64) t=aarch64-linux-musl ;;' \
' *) echo "unsupported TARGETARCH=$TARGETARCH" >&2; exit 1 ;;' \
'esac' \
'exec zig cc -target "$t" "$@"' > /usr/local/bin/zigcc && chmod +x /usr/local/bin/zigcc
ENV CC=zigcc CGO_ENABLED=1
# Build server
WORKDIR /build/server
COPY cmd/server/go.mod cmd/server/go.sum ./
COPY internal/geofilter/ ../../internal/geofilter/
COPY internal/sigvalidate/ ../../internal/sigvalidate/
COPY internal/packetpath/ ../../internal/packetpath/
COPY internal/dbconfig/ ../../internal/dbconfig/
COPY internal/dbschema/ ../../internal/dbschema/
COPY internal/prunequeue/ ../../internal/prunequeue/
COPY internal/perfio/ ../../internal/perfio/
COPY internal/mbcapqueue/ ../../internal/mbcapqueue/
COPY internal/lora/ ../../internal/lora/
COPY internal/users/ ../../internal/users/
COPY internal/channel/ ../../internal/channel/
COPY internal/mailer/ ../../internal/mailer/
RUN --mount=type=cache,target=/root/.cache/go-build \
--mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/tmp/zig-cache \
go mod download
COPY cmd/server/ ./
RUN --mount=type=cache,target=/root/.cache/go-build \
--mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/tmp/zig-cache \
GOOS=${TARGETOS} GOARCH=${TARGETARCH} \
go build -trimpath -tags ${GO_BUILD_TAGS} \
-ldflags "-s -w -extldflags '-static -Wl,-s ${STACK_LDFLAG}' -X main.Version=${APP_VERSION} -X main.Commit=${GIT_COMMIT} -X main.BuildTime=${BUILD_TIME}" \
-o /corescope-server .
# Build ingestor
WORKDIR /build/ingestor
COPY cmd/ingestor/go.mod cmd/ingestor/go.sum ./
COPY internal/channel/ ../../internal/channel/
COPY internal/geofilter/ ../../internal/geofilter/
COPY internal/sigvalidate/ ../../internal/sigvalidate/
COPY internal/packetpath/ ../../internal/packetpath/
COPY internal/dbconfig/ ../../internal/dbconfig/
COPY internal/dbschema/ ../../internal/dbschema/
COPY internal/prunequeue/ ../../internal/prunequeue/
COPY internal/perfio/ ../../internal/perfio/
COPY internal/mbcapqueue/ ../../internal/mbcapqueue/
RUN --mount=type=cache,target=/root/.cache/go-build \
--mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/tmp/zig-cache \
go mod download
COPY cmd/ingestor/ ./
RUN --mount=type=cache,target=/root/.cache/go-build \
--mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/tmp/zig-cache \
GOOS=${TARGETOS} GOARCH=${TARGETARCH} \
go build -trimpath -tags ${GO_BUILD_TAGS} \
-ldflags "-s -w -extldflags '-static -Wl,-s ${STACK_LDFLAG}'" \
-o /corescope-ingestor .
# Build decrypt CLI
WORKDIR /build/decrypt
COPY cmd/decrypt/go.mod cmd/decrypt/go.sum ./
COPY internal/channel/ ../../internal/channel/
RUN --mount=type=cache,target=/root/.cache/go-build \
--mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/tmp/zig-cache \
go mod download
COPY cmd/decrypt/ ./
RUN --mount=type=cache,target=/root/.cache/go-build \
--mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/tmp/zig-cache \
GOOS=${TARGETOS} GOARCH=${TARGETARCH} \
go build -trimpath -tags ${GO_BUILD_TAGS} \
-ldflags "-s -w -extldflags '-static -Wl,-s ${STACK_LDFLAG}' -X main.version=${APP_VERSION}" \
-o /corescope-decrypt .
# The thread-stack size must be in every binary's PT_GNU_STACK (see
# STACK_LDFLAG). The Go test for it runs on glibc in CI, where it passes with
# or without the flag, so the artifact is checked here.
RUN for f in /corescope-server /corescope-ingestor /corescope-decrypt; do \
readelf -lW "$f" | grep -Eq 'GNU_STACK .* 0x0*800000 +RW' || \
{ echo "$f: PT_GNU_STACK is not 8 MiB, musl thread stacks are not pinned" >&2; readelf -lW "$f" | grep GNU_STACK >&2; exit 1; }; \
done
# Runtime image
FROM alpine:3.24
RUN apk add --no-cache mosquitto mosquitto-clients supervisor caddy wget
# Account for the Go services when a deployment opts into running them
# unprivileged with RUN_AS_UID=1000, the usual choice. Creating it costs
# nothing when it goes unused, and saves the entrypoint creating one at start.
#
# Nothing in the image is given to it here. -H matters for that: without it
# busybox adduser creates the home directory itself, and /app would be
# corescope-owned mode 2755 — the setgid bit then giving gid 1000 to /app/data
# and to every file written into it, which Docker copies into each new named
# volume. The home field still points at /app; only the directory is left to
# the WORKDIR below, root-owned as in earlier releases.
RUN addgroup -S -g 1000 corescope && \
adduser -S -u 1000 -G corescope -H -h /app -s /sbin/nologin corescope
# Default: the services run as root, exactly as in earlier releases. The
# entrypoint overrides this when RUN_AS_UID asks for an unprivileged account.
ENV CORESCOPE_SERVICE_USER=root
WORKDIR /app
# Go binaries (statically linked; they do not use this image's libc)
COPY --from=builder /corescope-server /corescope-ingestor /corescope-decrypt /app/
# Frontend assets + config
COPY public/ ./public/
COPY config.example.json channel-rainbow.json ./
# Bake git commit SHA — manage.sh and CI write .git-commit before build
# Default to "unknown" if not provided
RUN echo "unknown" > .git-commit
# Supervisor + Mosquitto + Caddy config
COPY docker/supervisord-go.conf /etc/supervisor/conf.d/supervisord.conf
COPY docker/supervisord-go-no-mosquitto.conf /etc/supervisor/conf.d/supervisord-no-mosquitto.conf
COPY docker/supervisord-go-no-caddy.conf /etc/supervisor/conf.d/supervisord-no-caddy.conf
COPY docker/supervisord-go-no-mosquitto-no-caddy.conf /etc/supervisor/conf.d/supervisord-no-mosquitto-no-caddy.conf
COPY docker/mosquitto.conf /etc/mosquitto/mosquitto.conf
COPY docker/Caddyfile /etc/caddy/Caddyfile
# Data directory
RUN mkdir -p /app/data /var/lib/mosquitto /data/caddy && \
chown -R mosquitto:mosquitto /var/lib/mosquitto
# Entrypoint
COPY docker/entrypoint-go.sh /entrypoint.sh
# Sourced by the entrypoint, so deliberately not executable.
COPY docker/select-service-user.sh /select-service-user.sh
RUN chmod +x /entrypoint.sh
EXPOSE 80 443 1883
VOLUME ["/app/data", "/data/caddy"]
ENTRYPOINT ["/entrypoint.sh"]