Adds docs/ci-cd/soak/soak.yaml (20 soak jobs plus a control failure, for a cicd-test repo) and docs/ci-cd/soak-test.md with preconditions, pass criteria and the first evidence lining up silent job failures with runner evictions. The runbook now says to cancel orphaned in-progress jobs before rerunning. Refs DevOps/software-infrastructure#41 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
83 lines
3.8 KiB
Markdown
83 lines
3.8 KiB
Markdown
# S8 soak test: proving jobs no longer die silently
|
|
|
|
Procedure for DevOps/software-infrastructure#41 (S8). The workflow is
|
|
[`soak/soak.yaml`](soak/soak.yaml). This file is public: record hostnames,
|
|
node names and anything security-related in the private assessment, not here.
|
|
|
|
## What "dying silently" looks like
|
|
|
|
A job stops printing output, then shows **failure** with no step marked
|
|
failed, or stays **in progress** long after it should have ended. Evidence so
|
|
far (2026-11-13, two jobs from one project repo):
|
|
|
|
| Job | Last log line (UTC) | Runner pod evicted for low ephemeral storage (UTC) | Job result |
|
|
|---|---|---|---|
|
|
| A | 14:28:17 | about 14:28 (replacement pod started 14:28:35) | failure at 14:39:25, the running step marked failed with no error of its own |
|
|
| B | started 14:46:28 | 14:52:26 | still "in progress" 15 minutes later |
|
|
|
|
In both cases the job's last output lines up with the runner pod being evicted
|
|
from its node. The node's disk filled while jobs pulled and built images. The
|
|
same evictions also removed other pods on those nodes, not only CI.
|
|
|
|
This supports eviction as the cause, but it isn't proof that eviction is the
|
|
only cause. The soak test below is what closes S8.
|
|
|
|
## Preconditions
|
|
|
|
1. S3, S4 and S5 (#36, #37, #38) and S6 (#39) are live. Check with the
|
|
commands in [runner.md](runner.md#reading-logs): the runner has the expected
|
|
capacity, resource limits and `docker-prune` sidecar (or, if the runner moved
|
|
to a dedicated host under I3, #45, that host's Compose project is up).
|
|
2. No other agent's CI is running heavy work. Check the Actions tab of the busy
|
|
project repos, and announce the soak in the Agents topic "Gitea Actions
|
|
runner: status and changes" at least an hour before.
|
|
3. A throwaway repo owned by `claude`, named `cicd-test-soak`, holding only
|
|
`.gitea/workflows/soak.yaml` copied from `soak/soak.yaml`.
|
|
|
|
## Running it
|
|
|
|
1. Start the `soak` workflow from the repo's Actions tab with `rounds` = 20.
|
|
It runs two soak jobs at a time at most, so it never takes more than the
|
|
runner's capacity, and it has one `control-failure` job.
|
|
2. While it runs, keep a read-only watch:
|
|
|
|
```sh
|
|
kubectl -n gitea get events --watch
|
|
kubectl -n gitea logs gitea-actions-runner-0 -c runner -f
|
|
```
|
|
|
|
For a host runner, use `docker compose logs -f` in its project directory.
|
|
3. Expect about 5 to 10 minutes per soak job, depending on CPU.
|
|
|
|
## Pass criteria
|
|
|
|
- All 20 `soak` jobs finish **success**, and none sits "in progress" after the
|
|
workflow ends.
|
|
- `control-failure` finishes **failure** at the step "Expected failure", with
|
|
exit code 3 in the log. That shows failures are still reported accurately.
|
|
- No runner restart, eviction or `OOMKilled` event in the window.
|
|
- Every job's "Record end" step shows Docker disk use back near where it
|
|
started, so cleanup is keeping up.
|
|
|
|
If a job fails, line up its last log line against the events and the runner
|
|
log, as in the table above. A failure with a real step error (for example a
|
|
pull timeout) is a finding to fix, not a silent death, but it still resets
|
|
the count: the acceptance criterion is 20 consecutive good jobs.
|
|
|
|
## After the soak
|
|
|
|
- Record results on #41: job count, durations, peak disk use, any failures and
|
|
their cause. Keep node names out of it.
|
|
- Post a short summary to the Agents topic so other agents stop working around
|
|
the problem.
|
|
- Rerun any failed `claude/vitals` CI once its owning task is idle, or by
|
|
agreement on Discourse.
|
|
- Delete `cicd-test-soak` or leave it archived and note which on #41.
|
|
|
|
## Until S3 to S6 are live
|
|
|
|
When the runner is evicted mid-job, the job is not cleaned up promptly. Gitea
|
|
marks it failed only when its task times out. Until S6's shorter zombie
|
|
timeout is live, an orphaned job can stay "in progress" for hours. **Cancel
|
|
it, then rerun once.** Don't change the code because of it.
|