docs: S8 soak-test workflow and procedure
Adds docs/ci-cd/soak/soak.yaml (20 soak jobs plus a control failure, for a cicd-test repo) and docs/ci-cd/soak-test.md with preconditions, pass criteria and the first evidence lining up silent job failures with runner evictions. The runbook now says to cancel orphaned in-progress jobs before rerunning. Refs DevOps/software-infrastructure#41 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -157,7 +157,9 @@ device`.
|
||||
workflow or raise `gitea_actions_docker_data_size` and the ephemeral-storage
|
||||
limits together (the node needs that much free local disk).
|
||||
4. If the pod was evicted, the StatefulSet recreates it. Rerun the jobs that
|
||||
were lost, once.
|
||||
were lost, once. A job that was running at the time may stay "in progress"
|
||||
until Gitea's task timeout fails it; cancel it first. See
|
||||
[soak-test.md](soak-test.md) for what this looks like in a job's log.
|
||||
|
||||
### kubectl itself fails
|
||||
|
||||
@@ -222,5 +224,6 @@ I3 and S3: run the runner on `inspiron-3670` instead of in the cluster.
|
||||
|
||||
- Roadmap: label `cicd` on DevOps/software-infrastructure, milestones
|
||||
"CI/CD 0" to "CI/CD 4".
|
||||
- S8 soak test and silent-failure evidence: [soak-test.md](soak-test.md).
|
||||
- Agents coordinate CI on the forum's Agents category, topic "Gitea Actions
|
||||
runner: status and changes".
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
# S8 soak test: proving jobs no longer die silently
|
||||
|
||||
Procedure for DevOps/software-infrastructure#41 (S8). The workflow is
|
||||
[`soak/soak.yaml`](soak/soak.yaml). This file is public: record hostnames,
|
||||
node names and anything security-related in the private assessment, not here.
|
||||
|
||||
## What "dying silently" looks like
|
||||
|
||||
A job stops printing output, then shows **failure** with no step marked
|
||||
failed, or stays **in progress** long after it should have ended. Evidence so
|
||||
far (2026-11-13, two jobs from one project repo):
|
||||
|
||||
| Job | Last log line (UTC) | Runner pod evicted for low ephemeral storage (UTC) | Job result |
|
||||
|---|---|---|---|
|
||||
| A | 14:28:17 | about 14:28 (replacement pod started 14:28:35) | failure at 14:39:25, the running step marked failed with no error of its own |
|
||||
| B | started 14:46:28 | 14:52:26 | still "in progress" 15 minutes later |
|
||||
|
||||
In both cases the job's last output lines up with the runner pod being evicted
|
||||
from its node. The node's disk filled while jobs pulled and built images. The
|
||||
same evictions also removed other pods on those nodes, not only CI.
|
||||
|
||||
This supports eviction as the cause, but it isn't proof that eviction is the
|
||||
only cause. The soak test below is what closes S8.
|
||||
|
||||
## Preconditions
|
||||
|
||||
1. S3, S4 and S5 (#36, #37, #38) and S6 (#39) are live. Check with the
|
||||
commands in [runner.md](runner.md#reading-logs): the runner has the expected
|
||||
capacity, resource limits and `docker-prune` sidecar (or, if the runner moved
|
||||
to a dedicated host under I3, #45, that host's Compose project is up).
|
||||
2. No other agent's CI is running heavy work. Check the Actions tab of the busy
|
||||
project repos, and announce the soak in the Agents topic "Gitea Actions
|
||||
runner: status and changes" at least an hour before.
|
||||
3. A throwaway repo owned by `claude`, named `cicd-test-soak`, holding only
|
||||
`.gitea/workflows/soak.yaml` copied from `soak/soak.yaml`.
|
||||
|
||||
## Running it
|
||||
|
||||
1. Start the `soak` workflow from the repo's Actions tab with `rounds` = 20.
|
||||
It runs two soak jobs at a time at most, so it never takes more than the
|
||||
runner's capacity, and it has one `control-failure` job.
|
||||
2. While it runs, keep a read-only watch:
|
||||
|
||||
```sh
|
||||
kubectl -n gitea get events --watch
|
||||
kubectl -n gitea logs gitea-actions-runner-0 -c runner -f
|
||||
```
|
||||
|
||||
For a host runner, use `docker compose logs -f` in its project directory.
|
||||
3. Expect about 5 to 10 minutes per soak job, depending on CPU.
|
||||
|
||||
## Pass criteria
|
||||
|
||||
- All 20 `soak` jobs finish **success**, and none sits "in progress" after the
|
||||
workflow ends.
|
||||
- `control-failure` finishes **failure** at the step "Expected failure", with
|
||||
exit code 3 in the log. That shows failures are still reported accurately.
|
||||
- No runner restart, eviction or `OOMKilled` event in the window.
|
||||
- Every job's "Record end" step shows Docker disk use back near where it
|
||||
started, so cleanup is keeping up.
|
||||
|
||||
If a job fails, line up its last log line against the events and the runner
|
||||
log, as in the table above. A failure with a real step error (for example a
|
||||
pull timeout) is a finding to fix, not a silent death, but it still resets
|
||||
the count: the acceptance criterion is 20 consecutive good jobs.
|
||||
|
||||
## After the soak
|
||||
|
||||
- Record results on #41: job count, durations, peak disk use, any failures and
|
||||
their cause. Keep node names out of it.
|
||||
- Post a short summary to the Agents topic so other agents stop working around
|
||||
the problem.
|
||||
- Rerun any failed `claude/vitals` CI once its owning task is idle, or by
|
||||
agreement on Discourse.
|
||||
- Delete `cicd-test-soak` or leave it archived and note which on #41.
|
||||
|
||||
## Until S3 to S6 are live
|
||||
|
||||
When the runner is evicted mid-job, the job is not cleaned up promptly. Gitea
|
||||
marks it failed only when its task times out. Until S6's shorter zombie
|
||||
timeout is live, an orphaned job can stay "in progress" for hours. **Cancel
|
||||
it, then rerun once.** Don't change the code because of it.
|
||||
@@ -0,0 +1,107 @@
|
||||
# S8 soak test for the shared Gitea Actions runner.
|
||||
#
|
||||
# Copy this file to `.gitea/workflows/soak.yaml` in a throwaway repo owned by
|
||||
# `claude` (named `cicd-test-*`), then start it from the Actions tab. It is
|
||||
# kept here, not in this repo's `.gitea/workflows/`, so it never runs on a push
|
||||
# to software-infrastructure. Procedure: docs/ci-cd/soak-test.md.
|
||||
#
|
||||
# 20 soak jobs plus one control job. Each soak job pulls large images, compiles
|
||||
# Go's standard library, and churns some disk, so it exercises the conditions
|
||||
# under which jobs used to die without a step error. Every job prints the time
|
||||
# and disk usage at the start and end of each step, so a job that stops
|
||||
# reporting can be lined up against runner and kubelet events.
|
||||
|
||||
name: soak
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
rounds:
|
||||
description: "Number of soak jobs (the acceptance test needs 20)"
|
||||
default: "20"
|
||||
|
||||
concurrency:
|
||||
group: soak-${{ github.repository }}
|
||||
cancel-in-progress: false
|
||||
|
||||
env:
|
||||
# Pinned tags; keep in step with what the runner's job images allow.
|
||||
PULL_IMAGES: >-
|
||||
golang:1.25.4-bookworm
|
||||
node:22.21.1-bookworm
|
||||
postgres:17.7-bookworm
|
||||
GO_IMAGE: golang:1.25.4-bookworm
|
||||
|
||||
jobs:
|
||||
plan:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
outputs:
|
||||
rounds: ${{ steps.r.outputs.rounds }}
|
||||
steps:
|
||||
- id: r
|
||||
run: |
|
||||
n="${{ inputs.rounds }}"
|
||||
case "$n" in ''|*[!0-9]*) n=20 ;; esac
|
||||
[ "$n" -gt 40 ] && n=40
|
||||
printf 'rounds=[%s]\n' "$(seq -s, 1 "$n")" >> "$GITHUB_OUTPUT"
|
||||
|
||||
soak:
|
||||
needs: plan
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 45
|
||||
strategy:
|
||||
fail-fast: false
|
||||
# Two at a time, so the runner sees concurrent jobs when its capacity
|
||||
# allows it, without crowding out other repos' CI.
|
||||
max-parallel: 2
|
||||
matrix:
|
||||
round: ${{ fromJSON(needs.plan.outputs.rounds) }}
|
||||
steps:
|
||||
- name: Record start
|
||||
run: |
|
||||
echo "round ${{ matrix.round }} start $(date -u +%FT%TZ)"
|
||||
df -h / /var/lib/docker 2>/dev/null || df -h /
|
||||
docker system df || true
|
||||
|
||||
- name: Pull large images
|
||||
run: |
|
||||
for img in $PULL_IMAGES; do
|
||||
echo "$(date -u +%T) pull $img"
|
||||
docker pull -q "$img"
|
||||
done
|
||||
docker system df
|
||||
|
||||
- name: Heavy compile
|
||||
run: |
|
||||
echo "$(date -u +%T) compile start"
|
||||
# The daemon can't see the job's files, so nothing is bind-mounted.
|
||||
docker run --rm "$GO_IMAGE" sh -c 'go build -a std && echo compiled'
|
||||
echo "$(date -u +%T) compile end"
|
||||
|
||||
- name: Disk churn
|
||||
run: |
|
||||
vol="soak-${{ github.run_id }}-${{ matrix.round }}"
|
||||
docker volume create "$vol" >/dev/null
|
||||
docker run --rm -v "$vol:/data" "$GO_IMAGE" \
|
||||
sh -c 'dd if=/dev/zero of=/data/blob bs=1M count=2048 status=none && sync && ls -lh /data'
|
||||
docker volume rm "$vol" >/dev/null
|
||||
df -h / /var/lib/docker 2>/dev/null || df -h /
|
||||
|
||||
- name: Record end
|
||||
if: always()
|
||||
run: |
|
||||
echo "round ${{ matrix.round }} end $(date -u +%FT%TZ)"
|
||||
docker system df || true
|
||||
|
||||
# A job that must fail at a named step. If it ever shows success, or fails
|
||||
# with no step marked failed, status reporting is not accurate.
|
||||
control-failure:
|
||||
needs: plan
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
steps:
|
||||
- name: Expected failure
|
||||
run: |
|
||||
echo "this step fails on purpose"
|
||||
exit 3
|
||||
Reference in New Issue
Block a user