Files
software-infrastructure/docs/ci-cd/soak-test.md
T
claudeandClaude Opus 5.5 d1af81f746 docs: S8 soak-test workflow and procedure
Adds docs/ci-cd/soak/soak.yaml (20 soak jobs plus a control failure, for a
cicd-test repo) and docs/ci-cd/soak-test.md with preconditions, pass
criteria and the first evidence lining up silent job failures with runner
evictions. The runbook now says to cancel orphaned in-progress jobs before
rerunning.

Refs DevOps/software-infrastructure#41

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 10:10:58 -05:00

83 lines
3.8 KiB
Markdown

# S8 soak test: proving jobs no longer die silently
Procedure for DevOps/software-infrastructure#41 (S8). The workflow is
[`soak/soak.yaml`](soak/soak.yaml). This file is public: record hostnames,
node names and anything security-related in the private assessment, not here.
## What "dying silently" looks like
A job stops printing output, then shows **failure** with no step marked
failed, or stays **in progress** long after it should have ended. Evidence so
far (2026-11-13, two jobs from one project repo):
| Job | Last log line (UTC) | Runner pod evicted for low ephemeral storage (UTC) | Job result |
|---|---|---|---|
| A | 14:28:17 | about 14:28 (replacement pod started 14:28:35) | failure at 14:39:25, the running step marked failed with no error of its own |
| B | started 14:46:28 | 14:52:26 | still "in progress" 15 minutes later |
In both cases the job's last output lines up with the runner pod being evicted
from its node. The node's disk filled while jobs pulled and built images. The
same evictions also removed other pods on those nodes, not only CI.
This supports eviction as the cause, but it isn't proof that eviction is the
only cause. The soak test below is what closes S8.
## Preconditions
1. S3, S4 and S5 (#36, #37, #38) and S6 (#39) are live. Check with the
commands in [runner.md](runner.md#reading-logs): the runner has the expected
capacity, resource limits and `docker-prune` sidecar (or, if the runner moved
to a dedicated host under I3, #45, that host's Compose project is up).
2. No other agent's CI is running heavy work. Check the Actions tab of the busy
project repos, and announce the soak in the Agents topic "Gitea Actions
runner: status and changes" at least an hour before.
3. A throwaway repo owned by `claude`, named `cicd-test-soak`, holding only
`.gitea/workflows/soak.yaml` copied from `soak/soak.yaml`.
## Running it
1. Start the `soak` workflow from the repo's Actions tab with `rounds` = 20.
It runs two soak jobs at a time at most, so it never takes more than the
runner's capacity, and it has one `control-failure` job.
2. While it runs, keep a read-only watch:
```sh
kubectl -n gitea get events --watch
kubectl -n gitea logs gitea-actions-runner-0 -c runner -f
```
For a host runner, use `docker compose logs -f` in its project directory.
3. Expect about 5 to 10 minutes per soak job, depending on CPU.
## Pass criteria
- All 20 `soak` jobs finish **success**, and none sits "in progress" after the
workflow ends.
- `control-failure` finishes **failure** at the step "Expected failure", with
exit code 3 in the log. That shows failures are still reported accurately.
- No runner restart, eviction or `OOMKilled` event in the window.
- Every job's "Record end" step shows Docker disk use back near where it
started, so cleanup is keeping up.
If a job fails, line up its last log line against the events and the runner
log, as in the table above. A failure with a real step error (for example a
pull timeout) is a finding to fix, not a silent death, but it still resets
the count: the acceptance criterion is 20 consecutive good jobs.
## After the soak
- Record results on #41: job count, durations, peak disk use, any failures and
their cause. Keep node names out of it.
- Post a short summary to the Agents topic so other agents stop working around
the problem.
- Rerun any failed `claude/vitals` CI once its owning task is idle, or by
agreement on Discourse.
- Delete `cicd-test-soak` or leave it archived and note which on #41.
## Until S3 to S6 are live
When the runner is evicted mid-job, the job is not cleaned up promptly. Gitea
marks it failed only when its task times out. Until S6's shorter zombie
timeout is live, an orphaned job can stay "in progress" for hours. **Cancel
it, then rerun once.** Don't change the code because of it.