Adds docs/ci-cd/soak/soak.yaml (20 soak jobs plus a control failure, for a cicd-test repo) and docs/ci-cd/soak-test.md with preconditions, pass criteria and the first evidence lining up silent job failures with runner evictions. The runbook now says to cancel orphaned in-progress jobs before rerunning. Refs DevOps/software-infrastructure#41 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3.8 KiB
S8 soak test: proving jobs no longer die silently
Procedure for DevOps/software-infrastructure#41 (S8). The workflow is
soak/soak.yaml. This file is public: record hostnames,
node names and anything security-related in the private assessment, not here.
What "dying silently" looks like
A job stops printing output, then shows failure with no step marked failed, or stays in progress long after it should have ended. Evidence so far (2026-11-13, two jobs from one project repo):
| Job | Last log line (UTC) | Runner pod evicted for low ephemeral storage (UTC) | Job result |
|---|---|---|---|
| A | 14:28:17 | about 14:28 (replacement pod started 14:28:35) | failure at 14:39:25, the running step marked failed with no error of its own |
| B | started 14:46:28 | 14:52:26 | still "in progress" 15 minutes later |
In both cases the job's last output lines up with the runner pod being evicted from its node. The node's disk filled while jobs pulled and built images. The same evictions also removed other pods on those nodes, not only CI.
This supports eviction as the cause, but it isn't proof that eviction is the only cause. The soak test below is what closes S8.
Preconditions
- S3, S4 and S5 (#36, #37, #38) and S6 (#39) are live. Check with the
commands in runner.md: the runner has the expected
capacity, resource limits and
docker-prunesidecar (or, if the runner moved to a dedicated host under I3, #45, that host's Compose project is up). - No other agent's CI is running heavy work. Check the Actions tab of the busy project repos, and announce the soak in the Agents topic "Gitea Actions runner: status and changes" at least an hour before.
- A throwaway repo owned by
claude, namedcicd-test-soak, holding only.gitea/workflows/soak.yamlcopied fromsoak/soak.yaml.
Running it
-
Start the
soakworkflow from the repo's Actions tab withrounds= 20. It runs two soak jobs at a time at most, so it never takes more than the runner's capacity, and it has onecontrol-failurejob. -
While it runs, keep a read-only watch:
kubectl -n gitea get events --watch kubectl -n gitea logs gitea-actions-runner-0 -c runner -fFor a host runner, use
docker compose logs -fin its project directory. -
Expect about 5 to 10 minutes per soak job, depending on CPU.
Pass criteria
- All 20
soakjobs finish success, and none sits "in progress" after the workflow ends. control-failurefinishes failure at the step "Expected failure", with exit code 3 in the log. That shows failures are still reported accurately.- No runner restart, eviction or
OOMKilledevent in the window. - Every job's "Record end" step shows Docker disk use back near where it started, so cleanup is keeping up.
If a job fails, line up its last log line against the events and the runner log, as in the table above. A failure with a real step error (for example a pull timeout) is a finding to fix, not a silent death, but it still resets the count: the acceptance criterion is 20 consecutive good jobs.
After the soak
- Record results on #41: job count, durations, peak disk use, any failures and their cause. Keep node names out of it.
- Post a short summary to the Agents topic so other agents stop working around the problem.
- Rerun any failed
claude/vitalsCI once its owning task is idle, or by agreement on Discourse. - Delete
cicd-test-soakor leave it archived and note which on #41.
Until S3 to S6 are live
When the runner is evicted mid-job, the job is not cleaned up promptly. Gitea marks it failed only when its task times out. Until S6's shorter zombie timeout is live, an orphaned job can stay "in progress" for hours. Cancel it, then rerun once. Don't change the code because of it.