From d1af81f7462d069247f5577e7a8a966c5a231b94 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 13 Nov 2026 10:10:58 -0500 Subject: [PATCH] docs: S8 soak-test workflow and procedure Adds docs/ci-cd/soak/soak.yaml (20 soak jobs plus a control failure, for a cicd-test repo) and docs/ci-cd/soak-test.md with preconditions, pass criteria and the first evidence lining up silent job failures with runner evictions. The runbook now says to cancel orphaned in-progress jobs before rerunning. Refs DevOps/software-infrastructure#41 Co-Authored-By: Claude Opus 5.5 --- docs/ci-cd/runner.md | 5 +- docs/ci-cd/soak-test.md | 82 +++++++++++++++++++++++++++++ docs/ci-cd/soak/soak.yaml | 107 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 193 insertions(+), 1 deletion(-) create mode 100644 docs/ci-cd/soak-test.md create mode 100644 docs/ci-cd/soak/soak.yaml diff --git a/docs/ci-cd/runner.md b/docs/ci-cd/runner.md index 7073c12..3dbd4a8 100644 --- a/docs/ci-cd/runner.md +++ b/docs/ci-cd/runner.md @@ -157,7 +157,9 @@ device`. workflow or raise `gitea_actions_docker_data_size` and the ephemeral-storage limits together (the node needs that much free local disk). 4. If the pod was evicted, the StatefulSet recreates it. Rerun the jobs that - were lost, once. + were lost, once. A job that was running at the time may stay "in progress" + until Gitea's task timeout fails it; cancel it first. See + [soak-test.md](soak-test.md) for what this looks like in a job's log. ### kubectl itself fails @@ -222,5 +224,6 @@ I3 and S3: run the runner on `inspiron-3670` instead of in the cluster. - Roadmap: label `cicd` on DevOps/software-infrastructure, milestones "CI/CD 0" to "CI/CD 4". +- S8 soak test and silent-failure evidence: [soak-test.md](soak-test.md). - Agents coordinate CI on the forum's Agents category, topic "Gitea Actions runner: status and changes". diff --git a/docs/ci-cd/soak-test.md b/docs/ci-cd/soak-test.md new file mode 100644 index 0000000..bbfb269 --- /dev/null +++ b/docs/ci-cd/soak-test.md @@ -0,0 +1,82 @@ +# S8 soak test: proving jobs no longer die silently + +Procedure for DevOps/software-infrastructure#41 (S8). The workflow is +[`soak/soak.yaml`](soak/soak.yaml). This file is public: record hostnames, +node names and anything security-related in the private assessment, not here. + +## What "dying silently" looks like + +A job stops printing output, then shows **failure** with no step marked +failed, or stays **in progress** long after it should have ended. Evidence so +far (2026-11-13, two jobs from one project repo): + +| Job | Last log line (UTC) | Runner pod evicted for low ephemeral storage (UTC) | Job result | +|---|---|---|---| +| A | 14:28:17 | about 14:28 (replacement pod started 14:28:35) | failure at 14:39:25, the running step marked failed with no error of its own | +| B | started 14:46:28 | 14:52:26 | still "in progress" 15 minutes later | + +In both cases the job's last output lines up with the runner pod being evicted +from its node. The node's disk filled while jobs pulled and built images. The +same evictions also removed other pods on those nodes, not only CI. + +This supports eviction as the cause, but it isn't proof that eviction is the +only cause. The soak test below is what closes S8. + +## Preconditions + +1. S3, S4 and S5 (#36, #37, #38) and S6 (#39) are live. Check with the + commands in [runner.md](runner.md#reading-logs): the runner has the expected + capacity, resource limits and `docker-prune` sidecar (or, if the runner moved + to a dedicated host under I3, #45, that host's Compose project is up). +2. No other agent's CI is running heavy work. Check the Actions tab of the busy + project repos, and announce the soak in the Agents topic "Gitea Actions + runner: status and changes" at least an hour before. +3. A throwaway repo owned by `claude`, named `cicd-test-soak`, holding only + `.gitea/workflows/soak.yaml` copied from `soak/soak.yaml`. + +## Running it + +1. Start the `soak` workflow from the repo's Actions tab with `rounds` = 20. + It runs two soak jobs at a time at most, so it never takes more than the + runner's capacity, and it has one `control-failure` job. +2. While it runs, keep a read-only watch: + + ```sh + kubectl -n gitea get events --watch + kubectl -n gitea logs gitea-actions-runner-0 -c runner -f + ``` + + For a host runner, use `docker compose logs -f` in its project directory. +3. Expect about 5 to 10 minutes per soak job, depending on CPU. + +## Pass criteria + +- All 20 `soak` jobs finish **success**, and none sits "in progress" after the + workflow ends. +- `control-failure` finishes **failure** at the step "Expected failure", with + exit code 3 in the log. That shows failures are still reported accurately. +- No runner restart, eviction or `OOMKilled` event in the window. +- Every job's "Record end" step shows Docker disk use back near where it + started, so cleanup is keeping up. + +If a job fails, line up its last log line against the events and the runner +log, as in the table above. A failure with a real step error (for example a +pull timeout) is a finding to fix, not a silent death, but it still resets +the count: the acceptance criterion is 20 consecutive good jobs. + +## After the soak + +- Record results on #41: job count, durations, peak disk use, any failures and + their cause. Keep node names out of it. +- Post a short summary to the Agents topic so other agents stop working around + the problem. +- Rerun any failed `claude/vitals` CI once its owning task is idle, or by + agreement on Discourse. +- Delete `cicd-test-soak` or leave it archived and note which on #41. + +## Until S3 to S6 are live + +When the runner is evicted mid-job, the job is not cleaned up promptly. Gitea +marks it failed only when its task times out. Until S6's shorter zombie +timeout is live, an orphaned job can stay "in progress" for hours. **Cancel +it, then rerun once.** Don't change the code because of it. diff --git a/docs/ci-cd/soak/soak.yaml b/docs/ci-cd/soak/soak.yaml new file mode 100644 index 0000000..efbb109 --- /dev/null +++ b/docs/ci-cd/soak/soak.yaml @@ -0,0 +1,107 @@ +# S8 soak test for the shared Gitea Actions runner. +# +# Copy this file to `.gitea/workflows/soak.yaml` in a throwaway repo owned by +# `claude` (named `cicd-test-*`), then start it from the Actions tab. It is +# kept here, not in this repo's `.gitea/workflows/`, so it never runs on a push +# to software-infrastructure. Procedure: docs/ci-cd/soak-test.md. +# +# 20 soak jobs plus one control job. Each soak job pulls large images, compiles +# Go's standard library, and churns some disk, so it exercises the conditions +# under which jobs used to die without a step error. Every job prints the time +# and disk usage at the start and end of each step, so a job that stops +# reporting can be lined up against runner and kubelet events. + +name: soak + +on: + workflow_dispatch: + inputs: + rounds: + description: "Number of soak jobs (the acceptance test needs 20)" + default: "20" + +concurrency: + group: soak-${{ github.repository }} + cancel-in-progress: false + +env: + # Pinned tags; keep in step with what the runner's job images allow. + PULL_IMAGES: >- + golang:1.25.4-bookworm + node:22.21.1-bookworm + postgres:17.7-bookworm + GO_IMAGE: golang:1.25.4-bookworm + +jobs: + plan: + runs-on: ubuntu-latest + timeout-minutes: 5 + outputs: + rounds: ${{ steps.r.outputs.rounds }} + steps: + - id: r + run: | + n="${{ inputs.rounds }}" + case "$n" in ''|*[!0-9]*) n=20 ;; esac + [ "$n" -gt 40 ] && n=40 + printf 'rounds=[%s]\n' "$(seq -s, 1 "$n")" >> "$GITHUB_OUTPUT" + + soak: + needs: plan + runs-on: ubuntu-latest + timeout-minutes: 45 + strategy: + fail-fast: false + # Two at a time, so the runner sees concurrent jobs when its capacity + # allows it, without crowding out other repos' CI. + max-parallel: 2 + matrix: + round: ${{ fromJSON(needs.plan.outputs.rounds) }} + steps: + - name: Record start + run: | + echo "round ${{ matrix.round }} start $(date -u +%FT%TZ)" + df -h / /var/lib/docker 2>/dev/null || df -h / + docker system df || true + + - name: Pull large images + run: | + for img in $PULL_IMAGES; do + echo "$(date -u +%T) pull $img" + docker pull -q "$img" + done + docker system df + + - name: Heavy compile + run: | + echo "$(date -u +%T) compile start" + # The daemon can't see the job's files, so nothing is bind-mounted. + docker run --rm "$GO_IMAGE" sh -c 'go build -a std && echo compiled' + echo "$(date -u +%T) compile end" + + - name: Disk churn + run: | + vol="soak-${{ github.run_id }}-${{ matrix.round }}" + docker volume create "$vol" >/dev/null + docker run --rm -v "$vol:/data" "$GO_IMAGE" \ + sh -c 'dd if=/dev/zero of=/data/blob bs=1M count=2048 status=none && sync && ls -lh /data' + docker volume rm "$vol" >/dev/null + df -h / /var/lib/docker 2>/dev/null || df -h / + + - name: Record end + if: always() + run: | + echo "round ${{ matrix.round }} end $(date -u +%FT%TZ)" + docker system df || true + + # A job that must fail at a named step. If it ever shows success, or fails + # with no step marked failed, status reporting is not accurate. + control-failure: + needs: plan + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - name: Expected failure + run: | + echo "this step fails on purpose" + exit 3