test(server): set the #1239 lock-hold threshold from measurement, 150µs to 5ms (#2039)

TestComputeAnalyticsDistanceLockHoldDuration failed on two consecutive master commits (5430bc79 at 222µs, 89377333 at 156µs), both passing on a re-run of the identical tree, neither touching cmd/server runtime code. The flat 150µs limit sat inside the healthy band.

Measured, not assumed:

  healthy    156µs, 222µs, 402µs   three commits, 402µs from run 35252186369
  regressed  201203µs              fork run 35252430017, RLock deliberately
                                   held across the whole compute

A factor of 500 apart, so the limit only had to stop sitting inside the healthy band. 5ms is 12x above the worst healthy reading and 40x below the measured regression. The four numbers and the run IDs are in the doc comment.

The first attempt (a1767c77) calibrated against a control where readers churned a second store the writer never locks, on the assumption that their CPU load was slowing the writer. CI measured that control at 0µs: the readers cost the writer nothing, the variance is lock handoff, and the control could not see what it was meant to subtract. 5e457961 replaces it. Both commits are kept in this branch's history, and issue #2038 is corrected where it argued against raising the limit.

Methodology untouched: same eight readers, same 200 writer cycles, same 20000 hops and 200 paths.

Merged by the interim maintainer without a second human reviewer. Not run locally: cmd/server needs cgo for the #1992 driver and this machine has no C toolchain, so CI (run 35253209412) is the check, and the mutation run above is what proves the assertion still fails on a real regression.

Fixes #2038

cmd/server/distance_lock_contention_test.go

@@ -17,10 +17,22 @@ import (
 // continuously, while the test goroutine measures how long it takes to
 // complete W bare mu.Lock()/mu.Unlock() cycles. Each writer cycle must
 // wait for ALL currently-holding RLocks to release. Pre-fix, every reader
-// holds RLock for the entire compute (~ms), so each writer cycle waits
-// behind an active reader → avg cycle hundreds of microseconds to
-// milliseconds. Post-fix, readers hold RLock only long enough to grab
-// slice headers (microseconds), so writer cycles complete unimpeded.
+// holds RLock for the entire compute, so each writer cycle waits behind an
+// active reader. Post-fix, readers hold RLock only long enough to grab
+// slice headers, so writer cycles complete unimpeded.
+//
+// The threshold comes from measurement, and the gap it has to straddle is
+// enormous (issue #2038). On CI runners:
+//
+//	healthy    156µs, 222µs, 402µs   (three readings, three commits)
+//	regressed  201203µs              (RLock deliberately held across the compute)
+//
+// The old limit was a flat 150µs, which sits *inside* the healthy band, so
+// it failed on two consecutive master commits that passed on re-run without
+// a byte changed. 5ms is 12x above the worst healthy reading and 40x below
+// the regression, which is the widest possible separation from both. The
+// scale of the gap is the point: a #1239 regression is not marginal, it
+// serializes a millisecond-scale compute behind every writer.
 func TestComputeAnalyticsDistanceLockHoldDuration(t *testing.T) {
 	if testing.Short() {
 		t.Skip("skipping concurrency timing test in -short mode")
@@ -118,12 +130,11 @@ func TestComputeAnalyticsDistanceLockHoldDuration(t *testing.T) {
 	t.Logf("avg writer Lock/Unlock cycle: %dµs over %d cycles (total %v) with %d concurrent readers, %d hops, %d paths",
 		avgMicros, WriterCycles, elapsed, Readers, N, len(paths))
 
-	// If readers hold the main RLock for their entire compute, every
-	// writer Lock cycle waits for an active reader to release: avg cycle
-	// >> 100µs at this data scale. After the refactor, readers hold the
-	// main RLock only long enough to snapshot slice headers (<1µs), so
-	// writer cycles complete in tens of microseconds.
-	const MaxAvgMicros = 150
+	// See the measurements in the doc comment: healthy runs land in the
+	// hundreds of microseconds, a regression two orders of magnitude above
+	// that. Anything in between is a genuine change in lock-hold behaviour
+	// and deserves to be looked at, not re-run.
+	const MaxAvgMicros = 5000
 	if avgMicros > MaxAvgMicros {
 		t.Fatalf("avg writer Lock/Unlock cycle %dµs exceeds %dµs threshold — computeAnalyticsDistance is holding the main RLock for too long and blocking writers (issue #1239)",
 			avgMicros, MaxAvgMicros)