#spindle torture

these are load, abuse and recovery scenarios for spindle's control plane: the job queue, jetstream ingest, hydrant catch-up, repo membership, retries, notifies, webhooks, logs, admission and quotas. they don't measure how much an engine can run. every scenario uses a fake engine that only sleeps and prints, so none of these numbers tell you anything about vms.

it only runs on linux, since rss and fds come from /proc. a gauge it can't read fails the scenario instead of quietly reading as zero.

#how it works

the scenarios are go tests in spindle/torture_*_test.go behind the torture build tag. each one boots a real spindle in process (New, Start, the real router and xrpc handlers, the jetstream client, the db and the job workers) and points it at fakes:

• fake knots (tortureKnot) serve records and git over http-backend, sign service auth with real keys as did:web on loopback, and double as every owner's pds. you can make them slow, take them down or break their git
• the fake jetstream and hydrant from fakes_test.go and hydrant_test.go
• one in-memory identity directory for every did, behind the same cache the real resolver uses, so a lookup it counts really was a cold fetch
• knotOutbox, which delivers notifies the way knot2 does: one in flight per spindle, oldest first, backoff from 1s doubling up to 60s with no jitter, and any 4xx gets dropped

spindle runs with the production defaults except for dev mode. loopback knots need it, and it also turns netguard off.

running in process is how a scenario can read the queue depth, swap in the fakes and restart spindle on the same db. the catch is that the fakes share the process too, so cpu and rss include them.

#running

it needs cgo, sqlite and the hydrant ffi like the normal spindle tests, so go through the repo's nix shell if your machine doesn't have those.

go test -tags 'torture libsqlite3' -c -o /tmp/torture.test ./spindle/
cd spindle
/tmp/torture.test -torture.list
/tmp/torture.test -test.run 'TestTorture/notify-outbox-flush' -test.v -test.timeout 20m \
  -torture.out /tmp/torture.jsonl

one scenario takes anywhere from a few seconds to about ten minutes, and all of them take about 40, so give a full run -test.timeout 90m.

on a shared box, cap it:

systemd-run --user --scope -p MemoryMax=4G -p CPUQuota=800% -- /tmp/torture.test ...

flags:

• -torture.list prints every scenario and what it does, without running any
• -test.run 'TestTorture/<name>' picks scenarios, plain TestTorture runs them all
• -torture.out <file> appends one json line per scenario
• -torture.scale 2 doubles every load param (repos, knots, events)
• -torture.set repos=200,step_ms=500 sets params directly. each result lists its params. only the loads and the knobs a check or this doc depends on are params, the rest are literals in the scenario
• -torture.log warn prints spindle's logs from that level up (the default is error). warnings and errors get counted per message in errors either way

#sizing the pools

pool-sizing measures, it doesn't pass or fail on speed. it runs one size at a time so it fits under MemoryMax. pool is the repo job pool and notify is the number of notify slots.

for pool in 16 64 256 1024; do
  /tmp/torture.test -test.run 'TestTorture/pool-sizing' -torture.set pool=$pool,notify=64 -torture.out /tmp/pool.jsonl
done

for each size you get what it costs idle (idle_rss_mb, idle_goroutines, idle_fds) and what it costs to catch up on about 5000 repos (60% on one big knot, 30% over four small ones and 10% on a slow one) with live events and pushes landing on top:

• rss_mb, goroutines, fds and knots_inflight, sampled, as _max
• catchup_s, from boot until the catch-up is done
• apply_all_s, until every changed repo got applied
• refreshes_per_s, notify_accepted_per_s, probe_write latency and errors

drain_budget (30m by default) is how long the catch-up gets. a run that runs out of it reports no time and fails "catch-up finished within the budget", so set -test.timeout past it. the other check is that the slow knot never holds more than its fair share of the pool while other knots have repos waiting. that's read off the pool's own bookkeeping, jobs running per knot against knots with jobs queued.

the defaults (256 repo job workers, 64 notify slots) come from this. behind a 5000 repo catch-up, the p99 for a live event was about 119s with 16 workers and 2.5s with 256, at about 685MB peak rss. 1024 workers only got it to 1.4s and cost about 990MB, 1775 fds and 2258 goroutines. 16 or 64 notify slots made no real difference.

#results

each json line has:

• params
• metrics: latency percentiles as _p50_ms up to _max_ms, and every sampled gauge as _start, _max, _end and _slope_per_min
• errors: http answers and log messages, counted by kind
• invariants: things spindle should always do

a broken invariant or a setup that fails (t.Fatal) fails the test and sets pass: false. everything should pass on the current code, so a red scenario means a regression or something new, never a known failure.

the sampled gauges are goroutines, heap_mb, rss_mb, fds, queued_jobs, pipelines, repo_retries, db_mb and engine_running, plus whatever a scenario adds.

some invariant names start with a short id like A1, N2, ING-1 or Q1. those ids aren't defined anywhere in the repo, so go by the name and its detail.

checks that are just pass or fail and don't need load live in ordinary tests next to the code they cover, so go test ./spindle/... catches them without the tag. the tag is only for what needs scale or time.