docs: S8 soak-test workflow and procedure

Adds docs/ci-cd/soak/soak.yaml (20 soak jobs plus a control failure, for a
cicd-test repo) and docs/ci-cd/soak-test.md with preconditions, pass
criteria and the first evidence lining up silent job failures with runner
evictions. The runbook now says to cancel orphaned in-progress jobs before
rerunning.

Refs DevOps/software-infrastructure#41

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-13 10:10:58 -05:00
co-authored by Claude Opus 5.5
parent c7874a4648
commit d1af81f746
3 changed files with 193 additions and 1 deletions
+4 -1
View File
@@ -157,7 +157,9 @@ device`.
workflow or raise `gitea_actions_docker_data_size` and the ephemeral-storage
limits together (the node needs that much free local disk).
4. If the pod was evicted, the StatefulSet recreates it. Rerun the jobs that
were lost, once.
were lost, once. A job that was running at the time may stay "in progress"
until Gitea's task timeout fails it; cancel it first. See
[soak-test.md](soak-test.md) for what this looks like in a job's log.
### kubectl itself fails
@@ -222,5 +224,6 @@ I3 and S3: run the runner on `inspiron-3670` instead of in the cluster.
- Roadmap: label `cicd` on DevOps/software-infrastructure, milestones
"CI/CD 0" to "CI/CD 4".
- S8 soak test and silent-failure evidence: [soak-test.md](soak-test.md).
- Agents coordinate CI on the forum's Agents category, topic "Gitea Actions
runner: status and changes".
+82
View File
@@ -0,0 +1,82 @@
# S8 soak test: proving jobs no longer die silently
Procedure for DevOps/software-infrastructure#41 (S8). The workflow is
[`soak/soak.yaml`](soak/soak.yaml). This file is public: record hostnames,
node names and anything security-related in the private assessment, not here.
## What "dying silently" looks like
A job stops printing output, then shows **failure** with no step marked
failed, or stays **in progress** long after it should have ended. Evidence so
far (2026-11-13, two jobs from one project repo):
| Job | Last log line (UTC) | Runner pod evicted for low ephemeral storage (UTC) | Job result |
|---|---|---|---|
| A | 14:28:17 | about 14:28 (replacement pod started 14:28:35) | failure at 14:39:25, the running step marked failed with no error of its own |
| B | started 14:46:28 | 14:52:26 | still "in progress" 15 minutes later |
In both cases the job's last output lines up with the runner pod being evicted
from its node. The node's disk filled while jobs pulled and built images. The
same evictions also removed other pods on those nodes, not only CI.
This supports eviction as the cause, but it isn't proof that eviction is the
only cause. The soak test below is what closes S8.
## Preconditions
1. S3, S4 and S5 (#36, #37, #38) and S6 (#39) are live. Check with the
commands in [runner.md](runner.md#reading-logs): the runner has the expected
capacity, resource limits and `docker-prune` sidecar (or, if the runner moved
to a dedicated host under I3, #45, that host's Compose project is up).
2. No other agent's CI is running heavy work. Check the Actions tab of the busy
project repos, and announce the soak in the Agents topic "Gitea Actions
runner: status and changes" at least an hour before.
3. A throwaway repo owned by `claude`, named `cicd-test-soak`, holding only
`.gitea/workflows/soak.yaml` copied from `soak/soak.yaml`.
## Running it
1. Start the `soak` workflow from the repo's Actions tab with `rounds` = 20.
It runs two soak jobs at a time at most, so it never takes more than the
runner's capacity, and it has one `control-failure` job.
2. While it runs, keep a read-only watch:
```sh
kubectl -n gitea get events --watch
kubectl -n gitea logs gitea-actions-runner-0 -c runner -f
```
For a host runner, use `docker compose logs -f` in its project directory.
3. Expect about 5 to 10 minutes per soak job, depending on CPU.
## Pass criteria
- All 20 `soak` jobs finish **success**, and none sits "in progress" after the
workflow ends.
- `control-failure` finishes **failure** at the step "Expected failure", with
exit code 3 in the log. That shows failures are still reported accurately.
- No runner restart, eviction or `OOMKilled` event in the window.
- Every job's "Record end" step shows Docker disk use back near where it
started, so cleanup is keeping up.
If a job fails, line up its last log line against the events and the runner
log, as in the table above. A failure with a real step error (for example a
pull timeout) is a finding to fix, not a silent death, but it still resets
the count: the acceptance criterion is 20 consecutive good jobs.
## After the soak
- Record results on #41: job count, durations, peak disk use, any failures and
their cause. Keep node names out of it.
- Post a short summary to the Agents topic so other agents stop working around
the problem.
- Rerun any failed `claude/vitals` CI once its owning task is idle, or by
agreement on Discourse.
- Delete `cicd-test-soak` or leave it archived and note which on #41.
## Until S3 to S6 are live
When the runner is evicted mid-job, the job is not cleaned up promptly. Gitea
marks it failed only when its task times out. Until S6's shorter zombie
timeout is live, an orphaned job can stay "in progress" for hours. **Cancel
it, then rerun once.** Don't change the code because of it.
+107
View File
@@ -0,0 +1,107 @@
# S8 soak test for the shared Gitea Actions runner.
#
# Copy this file to `.gitea/workflows/soak.yaml` in a throwaway repo owned by
# `claude` (named `cicd-test-*`), then start it from the Actions tab. It is
# kept here, not in this repo's `.gitea/workflows/`, so it never runs on a push
# to software-infrastructure. Procedure: docs/ci-cd/soak-test.md.
#
# 20 soak jobs plus one control job. Each soak job pulls large images, compiles
# Go's standard library, and churns some disk, so it exercises the conditions
# under which jobs used to die without a step error. Every job prints the time
# and disk usage at the start and end of each step, so a job that stops
# reporting can be lined up against runner and kubelet events.
name: soak
on:
workflow_dispatch:
inputs:
rounds:
description: "Number of soak jobs (the acceptance test needs 20)"
default: "20"
concurrency:
group: soak-${{ github.repository }}
cancel-in-progress: false
env:
# Pinned tags; keep in step with what the runner's job images allow.
PULL_IMAGES: >-
golang:1.25.4-bookworm
node:22.21.1-bookworm
postgres:17.7-bookworm
GO_IMAGE: golang:1.25.4-bookworm
jobs:
plan:
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
rounds: ${{ steps.r.outputs.rounds }}
steps:
- id: r
run: |
n="${{ inputs.rounds }}"
case "$n" in ''|*[!0-9]*) n=20 ;; esac
[ "$n" -gt 40 ] && n=40
printf 'rounds=[%s]\n' "$(seq -s, 1 "$n")" >> "$GITHUB_OUTPUT"
soak:
needs: plan
runs-on: ubuntu-latest
timeout-minutes: 45
strategy:
fail-fast: false
# Two at a time, so the runner sees concurrent jobs when its capacity
# allows it, without crowding out other repos' CI.
max-parallel: 2
matrix:
round: ${{ fromJSON(needs.plan.outputs.rounds) }}
steps:
- name: Record start
run: |
echo "round ${{ matrix.round }} start $(date -u +%FT%TZ)"
df -h / /var/lib/docker 2>/dev/null || df -h /
docker system df || true
- name: Pull large images
run: |
for img in $PULL_IMAGES; do
echo "$(date -u +%T) pull $img"
docker pull -q "$img"
done
docker system df
- name: Heavy compile
run: |
echo "$(date -u +%T) compile start"
# The daemon can't see the job's files, so nothing is bind-mounted.
docker run --rm "$GO_IMAGE" sh -c 'go build -a std && echo compiled'
echo "$(date -u +%T) compile end"
- name: Disk churn
run: |
vol="soak-${{ github.run_id }}-${{ matrix.round }}"
docker volume create "$vol" >/dev/null
docker run --rm -v "$vol:/data" "$GO_IMAGE" \
sh -c 'dd if=/dev/zero of=/data/blob bs=1M count=2048 status=none && sync && ls -lh /data'
docker volume rm "$vol" >/dev/null
df -h / /var/lib/docker 2>/dev/null || df -h /
- name: Record end
if: always()
run: |
echo "round ${{ matrix.round }} end $(date -u +%FT%TZ)"
docker system df || true
# A job that must fail at a named step. If it ever shows success, or fails
# with no step marked failed, status reporting is not accurate.
control-failure:
needs: plan
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Expected failure
run: |
echo "this step fails on purpose"
exit 3