Edge access logs, IP header hygiene, block lists, fail2ban jails

This commit is contained in:
MrAlders0n committed 2026-09-02 22:25:01 -04:00
1 parent 30c101d190
commit ee19469633
13 files changed
+378 -18

No files matched your search

@@ -0,0 +1,44 @@
# Beacon behind a host Apache (the analyzer.meshmapper.net shape). Drop these lines into the
# *:443 VirtualHost next to the existing ProxyPass rules. Needs mod_headers, mod_proxy,
# mod_proxy_http, mod_proxy_wstunnel, mod_authz_core (all already loaded on that box).
# dedicated log so fail2ban and the runbook can parse it (other_vhosts uses vhost_combined)
ErrorLog ${APACHE_LOG_DIR}/analyzer-ssl-error.log
CustomLog ${APACHE_LOG_DIR}/analyzer-ssl-access.log combined
# client-IP hygiene: the app trusts these headers, so only Apache may set them.
# mod_proxy re-adds a clean X-Forwarded-For after the unset.
RequestHeader unset X-Forwarded-For early
RequestHeader unset X-Real-IP early
RequestHeader unset True-Client-IP early
RequestHeader set X-Real-IP "%{REMOTE_ADDR}s"
# scraper User-Agent signatures live in /etc/apache2/beacon-blocklist-ua.conf; a match sets
# beacon_block, which the /api and /ws blocks below reject
Include /etc/apache2/beacon-blocklist-ua.conf
# manual block list for /api and /ws: edit /etc/apache2/beacon-blocklist.conf, then
# `apachectl configtest && systemctl reload apache2`
<Location "/api">
<RequireAll>
Require all granted
Require not env beacon_block
Include /etc/apache2/beacon-blocklist.conf
</RequireAll>
</Location>
<Location "/ws">
<RequireAll>
Require all granted
Require not env beacon_block
Include /etc/apache2/beacon-blocklist.conf
</RequireAll>
</Location>
# /etc/apache2/beacon-blocklist.conf — one `Require not ip <ip-or-cidr>` per line.
# Keep the TEST-NET placeholder so the file is never empty:
#
# Require not ip 192.0.2.0/24
#
# /etc/apache2/beacon-blocklist-ua.conf — one SetEnvIfNoCase per scraper User-Agent signature:
#
# SetEnvIfNoCase User-Agent "mesh\.hansimgamr\.net" beacon_block
+50 -9
View File
@@ -1,14 +1,55 @@
# Type 1 (All-in-One): API + Web on same server
{$DOMAIN} {
handle /api/* {
reverse_proxy tower-app:8080
}
# defines @blocked_ip and @blocked_ua; edit conf.d/blocklist.caddy then `caddy reload`
import conf.d/blocklist.caddy
handle /ws {
reverse_proxy tower-app:8080
}
# JSON access log for the whole site (API, SPA and assets together, so per-IP joins are one file)
log {
output file /var/log/caddy/access.json {
mode 0644
roll_size 100MiB
roll_keep 20
roll_keep_for 720h
}
format json
}
handle {
reverse_proxy tower-web:80
}
# opt-in: 403 API calls whose Referer names another site. Spoofable; enable only if the
# logs show a specific foreign host with sustained volume.
# @foreign_referer {
# header Referer *
# not header Referer https://{$DOMAIN}/*
# }
handle /api/* {
respond @blocked_ip "Forbidden" 403 {
close
}
respond @blocked_ua "Forbidden" 403 {
close
}
# respond @foreign_referer "Forbidden" 403
reverse_proxy app:8080 {
# the app trusts these headers for the client IP, so only Caddy may set them
header_up -True-Client-IP
header_up X-Real-IP {remote_host}
}
}
handle /ws {
respond @blocked_ip "Forbidden" 403 {
close
}
respond @blocked_ua "Forbidden" 403 {
close
}
reverse_proxy app:8080 {
header_up -True-Client-IP
header_up X-Real-IP {remote_host}
}
}
handle {
reverse_proxy web:80
}
}
+123
View File
@@ -0,0 +1,123 @@
# Beacon API abuse controls: edge log, block list, fail2ban
Staged posture: **log → throttle (beacon-server 429s) → firewall-ban repeat offenders.** Real
users include mobile users behind carrier CGNAT, so thresholds are generous and bans are short.
Two deployment shapes, two file sets:
| Shape | Edge | Files |
|---|---|---|
| Caddy in Docker (type 1, e.g. dev.meshcore.ca) | `docker-deployment-type1/data/Caddy/…` | `fail2ban/caddy/` |
| Host Apache in front of the containers (analyzer.meshmapper.net) | `app_config/apache/analyzer-vhost.snippet.conf` | `fail2ban/apache/` |
## 1. Edge: log + client-IP hygiene + manual block list
**Caddy.** `Caddyfile.proxy` now writes a JSON access log to `data/Caddy/logs/access.json`, strips
`True-Client-IP` and overwrites `X-Real-IP` on the way to the app (beacon-server trusts those headers
for the client IP), and 403s anything listed in `data/Caddy/conf.d/blocklist.caddy`. The compose
file mounts `conf.d` and `logs`; new mounts need one `docker compose up -d caddy`, after that:
```bash
docker compose exec caddy caddy validate --config /etc/caddy/Caddyfile
docker compose exec caddy caddy reload --config /etc/caddy/Caddyfile
```
**Apache.** Apply the snippet to the `*:443` vhost and create `/etc/apache2/beacon-blocklist.conf`
and `/etc/apache2/beacon-blocklist-ua.conf`, then `apachectl configtest && systemctl reload apache2`.
**Two ways to block.** By source IP (`remote_ip` / `Require not ip`) or by User-Agent signature
(`@blocked_ua` / `SetEnvIfNoCase`). A scraper that identifies itself, like the
`mesh.hansimgamr.net am-i-connected` client found on day 0, is better blocked by UA: it survives an
IP change and leaves the operator's own browser alone. Both apply to `/api/*` and `/ws` only.
Smoke test on either box (chi's log line must show the real client IP, not 1.2.3.4):
```bash
curl -s -o /dev/null -w '%{http_code}\n' -H 'True-Client-IP: 1.2.3.4' -H 'X-Real-IP: 1.2.3.4' https://$DOMAIN/api/v1/scopes
docker compose logs --tail 5 app
```
## 2. fail2ban
Two jails per box, same thresholds:
| Jail | Counts | Trips at | Why |
|---|---|---|---|
| `beacon-api-flood` | every `/api/` request | 900 in 60 s | a browser session peaks ~150 in its first 30 s (auto-chained pages), then ~2/min; 15 req/s for a full minute is no browser |
| `beacon-api-429` | `/api/` responses with status 429 | 300 in 10 min | only fires on clients that keep hammering through the server's throttle |
Both use `backend = polling` (the Debian default `systemd` cannot tail files) and a flat 10 min ban.
Install:
```bash
sudo cp filter.d/*.conf /etc/fail2ban/filter.d/
sudo cp jail.d/beacon-http.conf /etc/fail2ban/jail.d/
# Caddy box only: the ban chain must come up after dockerd
sudo install -D fail2ban.service.d/after-docker.conf /etc/systemd/system/fail2ban.service.d/after-docker.conf
sudo systemctl daemon-reload && sudo systemctl restart fail2ban
sudo fail2ban-regex <logpath> /etc/fail2ban/filter.d/beacon-api-flood.conf # date hits > 0, matches == /api/ lines
sudo fail2ban-client status beacon-api-flood
```
**Observe first.** The Caddy jail file ships with `banaction = dummy`; on the Apache box add the same
line per jail. Would-be bans land in `/var/run/fail2ban/fail2ban.dummy`. Run 3–7 days; every IP
there must be an obvious scraper, otherwise raise `maxretry`.
**Enforce.** Caddy box: traffic to Docker-published ports is DNAT'ed through the FORWARD path, which
the default nftables action (input hook) never sees, so switch to
`banaction = nftables[type=allports, chain=f2b-forward, chain_hook=forward]` (a second chain in the
existing `f2b-table`; `iptables-allports[chain=DOCKER-USER]` is equivalent). Apache box: remove the
`dummy` line, the inherited nftables input-hook action works because Apache listens on the host.
Then:
```bash
sudo fail2ban-client reload
sudo nft list table inet f2b-table # addr-set-beacon-api-flood present
sudo fail2ban-client set beacon-api-flood banip <phone-on-cellular> # site unreachable from it
sudo fail2ban-client set beacon-api-flood unbanip <phone-on-cellular>
```
Once beacon-server's per-IP limit is live, re-derive the flood `maxretry` as ~3× the per-minute
allowance. Consider `bantime.increment` only after a clean week.
## 3. Finding scrapers by hand
Caddy JSON (`LOG=/opt/docker/beacon/data/Caddy/logs/access.json`):
```bash
# top IPs, last hour
jq -r 'select(.ts > (now-3600)) | .request.remote_ip' "$LOG" | sort | uniq -c | sort -rn | head -20
# IPs with >200 /api/ hits and zero /ws upgrades, last 24h (the SPA always opens a socket)
jq -r 'select(.ts > (now-86400)) | [.request.remote_ip, (.request.uri|startswith("/api/")|tostring), (.request.uri=="/ws"|tostring)] | @tsv' "$LOG" \
| awk -F'\t' '{a[$1]+=($2=="true"); w[$1]+=($3=="true")} END {for (ip in a) if (a[ip]>200 && w[ip]==0) print a[ip], ip}' | sort -rn | head
# IPs asking limit= above 200 (the SPA never does)
jq -r '(.request.uri|capture("[?&]limit=(?<n>[0-9]+)")?) as $c | select($c != null and ($c.n|tonumber) > 200) | .request.remote_ip' "$LOG" | sort | uniq -c | sort -rn | head
# Referer / Origin hosts hitting /api/ (a reverse-proxying site usually forwards its visitors' Referer)
jq -r 'select(.request.uri|startswith("/api/")) | [(.request.headers.Referer[0] // "-"), (.request.headers.Origin[0] // "-")] | @tsv' "$LOG" \
| sed -E 's#(https?://[^/[:space:]]+)[^[:space:]]*#\1#g' | sort | uniq -c | sort -rn | head -30
# IPs with >50 /api/ hits and no SPA/asset fetch, last 24h (a native app looks like this too)
jq -r 'select(.ts > (now-86400)) | [.request.remote_ip, (.request.uri|startswith("/api/")|tostring), ((.request.uri|startswith("/api/")|not) and .request.uri!="/ws"|tostring)] | @tsv' "$LOG" \
| awk -F'\t' '{a[$1]+=($2=="true"); s[$1]+=($3=="true")} END {for (ip in a) if (a[ip]>50 && s[ip]==0) print a[ip], ip}' | sort -rn | head
# peak /api/ requests per IP per minute: calibrate maxretry / the server limit at ~3x the largest legitimate bucket
jq -r 'select(.request.uri|startswith("/api/")) | "\(.request.remote_ip) \((.ts/60)|floor)"' "$LOG" | sort | uniq -c | sort -rn | head -20
# spoof attempts (client-sent IP headers are logged as received)
jq -r 'select(.request.headers["True-Client-IP"] or .request.headers["X-Real-IP"]) | [.request.remote_ip, (.request.headers["True-Client-IP"][0]//"-")] | @tsv' "$LOG" | sort | uniq -c | sort -rn | head
```
Apache combined (`LOG=/var/log/apache2/analyzer-ssl-access.log`, run with sudo; `$7` is the request
path, `$11` the quoted Referer):
```bash
awk '{print $1}' "$LOG" | sort | uniq -c | sort -rn | head -20
awk '$7 ~ /^\/api\// {a[$1]++} $7=="/ws" {w[$1]++} END {for (ip in a) if (a[ip]>200 && !(ip in w)) print a[ip], ip}' "$LOG" | sort -rn | head
awk '$7 ~ /^\/api\/.*[?&]limit=/ {match($7,/[?&]limit=[0-9]+/); n=substr($7,RSTART+7,RLENGTH-7); if (n+0>200) print $1}' "$LOG" | sort | uniq -c | sort -rn | head
awk '$7 ~ /^\/api\// {print $11}' "$LOG" | sed -E 's#"(https?://[^/"]+).*#\1#' | sort | uniq -c | sort -rn | head -30
awk '$7 ~ /^\/api\// {a[$1]++} $7 !~ /^\/api\// && $7!="/ws" {s[$1]++} END {for (ip in a) if (a[ip]>50 && !(ip in s)) print a[ip], ip}' "$LOG" | sort -rn | head
awk '$7 ~ /^\/api\// {print $1, substr($4,2,17)}' "$LOG" | sort | uniq -c | sort -rn | head -20
```
A site that reverse-proxies the API shows up as one high-volume, API-only IP with no `/ws` and
usually a foreign Referer. Put it in the block list and reload.
Operations: `fail2ban-client status <jail>`, `fail2ban-client get <jail> banip --with-time`,
`fail2ban-client set <jail> unbanip <ip>`, `fail2ban-client reload <jail>` after editing a filter.
@@ -0,0 +1,5 @@
# fail2ban filter: beacon-api-429
# Requests to /api/ that beacon-server already throttled (status 429). Only persistence is punished.
[Definition]
failregex = ^<HOST> \S+ \S+ \[?\]? ?"[A-Z]+ /api/[^"]*" 429
ignoreregex =
@@ -0,0 +1,6 @@
# fail2ban filter: beacon-api-flood
# Apache `combined` log (Ubuntu variant). Every request to /api/ from an IP counts as one hit;
# the jail's maxretry/findtime decide what "too many" is. fail2ban has already cut the date out.
[Definition]
failregex = ^<HOST> \S+ \S+ \[?\]? ?"[A-Z]+ /api/
ignoreregex =
@@ -0,0 +1,31 @@
# Beacon API jails for a host-Apache deployment (analyzer.meshmapper.net shape).
# Same conventions as meshmapper-http.conf on that box: no [DEFAULT], backend = polling per jail,
# ignoreip = loopback, Tailscale, the Subscriber.
# banaction = dummy is observe mode (would-be bans land in /var/run/fail2ban/fail2ban.dummy).
# To enforce, delete those lines: the inherited default (nftables, input hook) works as-is because
# Apache listens on the host. bantime is a flat 10m to start: a banned CGNAT address hurts every
# user behind it.
[beacon-api-flood]
enabled = true
backend = polling
filter = beacon-api-flood
port = http,https
banaction = dummy
logpath = /var/log/apache2/analyzer-ssl-access.log
maxretry = 900
findtime = 60
bantime = 10m
ignoreip = 127.0.0.1/8 ::1 100.64.0.0/10 192.99.151.15
[beacon-api-429]
enabled = true
backend = polling
filter = beacon-api-429
port = http,https
banaction = dummy
logpath = /var/log/apache2/analyzer-ssl-access.log
maxretry = 300
findtime = 600
bantime = 10m
ignoreip = 127.0.0.1/8 ::1 100.64.0.0/10 192.99.151.15
@@ -0,0 +1,4 @@
# /etc/systemd/system/fail2ban.service.d/after-docker.conf
# The forward-hook / DOCKER-USER ban chain only makes sense once dockerd has set up its rules.
[Unit]
After=docker.service
@@ -0,0 +1,7 @@
# fail2ban filter: beacon-api-429
# Requests to /api/ that beacon-server already throttled (status 429). A client that keeps hammering
# through 429s is ignoring the app's own limit; this jail only enforces persistence.
[Definition]
datepattern = "ts":{EPOCH}
failregex = ^\{"level":"\w+",.*?"request":\{"remote_ip":"<HOST>","remote_port":"\d+",.*?"uri":"/api/.*?"status":429,
ignoreregex =
@@ -0,0 +1,7 @@
# fail2ban filter: beacon-api-flood
# Caddy JSON access log, one object per line. Every request to /api/ from an IP counts as one hit;
# the jail's maxretry/findtime decide what "too many" is.
[Definition]
datepattern = "ts":{EPOCH}
failregex = ^\{"level":"\w+",.*?"request":\{"remote_ip":"<HOST>","remote_port":"\d+",.*?"uri":"/api/
ignoreregex =
@@ -0,0 +1,36 @@
# Beacon API jails for a Caddy-in-Docker deployment (type 1). See app_config/fail2ban/README.md.
#
# DELIBERATE choices:
# - No [DEFAULT] section: it would merge into the global default and change the sshd jail.
# - backend = polling per jail: the Debian default is systemd, which cannot tail log FILES.
# - banaction: traffic to Docker-published 80/443 is DNAT'ed through the FORWARD path, so the
# default nftables action (input hook) never sees it. Start in observe mode (dummy), then switch
# to a forward-hook chain in the existing f2b-table:
# nftables[type=allports, chain=f2b-forward, chain_hook=forward]
# (equivalent: iptables-allports[chain=DOCKER-USER])
# - ignoreip: loopback, RFC1918 (Docker bridges), Tailscale. Add the operator's home IP if wanted.
# - bantime is a flat 10m: a banned CGNAT address hurts every user behind it.
[beacon-api-flood]
enabled = true
backend = polling
filter = beacon-api-flood
logpath = /opt/docker/beacon/data/Caddy/logs/access.json
banaction = dummy
ignoreip = 127.0.0.0/8 ::1 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 100.64.0.0/10
findtime = 60
maxretry = 900
bantime = 10m
maxmatches = 5
[beacon-api-429]
enabled = true
backend = polling
filter = beacon-api-429
logpath = /opt/docker/beacon/data/Caddy/logs/access.json
banaction = dummy
ignoreip = 127.0.0.0/8 ::1 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 100.64.0.0/10
findtime = 10m
maxretry = 300
bantime = 10m
maxmatches = 5
@@ -1,13 +1,54 @@
{$DOMAIN} {
handle /api/* {
reverse_proxy tower-app:8080
}
# defines @blocked_ip and @blocked_ua; edit conf.d/blocklist.caddy then `caddy reload`
import conf.d/blocklist.caddy
handle /ws {
reverse_proxy tower-app:8080
}
# JSON access log for the whole site (API, SPA and assets together, so per-IP joins are one file)
log {
output file /var/log/caddy/access.json {
mode 0644
roll_size 100MiB
roll_keep 20
roll_keep_for 720h
}
format json
}
handle {
reverse_proxy tower-web:80
}
# opt-in: 403 API calls whose Referer names another site. Spoofable; enable only if the
# logs show a specific foreign host with sustained volume.
# @foreign_referer {
# header Referer *
# not header Referer https://{$DOMAIN}/*
# }
handle /api/* {
respond @blocked_ip "Forbidden" 403 {
close
}
respond @blocked_ua "Forbidden" 403 {
close
}
# respond @foreign_referer "Forbidden" 403
reverse_proxy app:8080 {
# the app trusts these headers for the client IP, so only Caddy may set them
header_up -True-Client-IP
header_up X-Real-IP {remote_host}
}
}
handle /ws {
respond @blocked_ip "Forbidden" 403 {
close
}
respond @blocked_ua "Forbidden" 403 {
close
}
reverse_proxy app:8080 {
header_up -True-Client-IP
header_up X-Real-IP {remote_host}
}
}
handle {
reverse_proxy web:80
}
}
@@ -0,0 +1,13 @@
# Manual block list for /api/* and /ws.
# @blocked_ip: one IP or CIDR per remote_ip line. Keep at least one line: 192.0.2.0/24 is TEST-NET
# (never routable) and is a safe placeholder.
# @blocked_ua: one header_regexp line per scraper User-Agent signature (Go regexp; (?i) = case-insensitive).
# Apply changes with:
# docker compose exec caddy caddy validate --config /etc/caddy/Caddyfile
# docker compose exec caddy caddy reload --config /etc/caddy/Caddyfile
@blocked_ip {
remote_ip 192.0.2.0/24
}
@blocked_ua {
header_regexp User-Agent "(?i)mesh\.hansimgamr\.net"
}
@@ -53,6 +53,8 @@ services:
env_file: .env
volumes:
- ./data/Caddy/CaddyFile/Caddyfile.proxy:/etc/caddy/Caddyfile
- ./data/Caddy/conf.d:/etc/caddy/conf.d:ro
- ./data/Caddy/logs:/var/log/caddy
- ./data/Caddy/data:/data/caddy/
- ./data/Caddy/config:/config/caddy/
depends_on: