docs: S8 soak-test workflow and procedure
Adds docs/ci-cd/soak/soak.yaml (20 soak jobs plus a control failure, for a cicd-test repo) and docs/ci-cd/soak-test.md with preconditions, pass criteria and the first evidence lining up silent job failures with runner evictions. The runbook now says to cancel orphaned in-progress jobs before rerunning. Refs DevOps/software-infrastructure#41 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -157,7 +157,9 @@ device`.
|
|||||||
workflow or raise `gitea_actions_docker_data_size` and the ephemeral-storage
|
workflow or raise `gitea_actions_docker_data_size` and the ephemeral-storage
|
||||||
limits together (the node needs that much free local disk).
|
limits together (the node needs that much free local disk).
|
||||||
4. If the pod was evicted, the StatefulSet recreates it. Rerun the jobs that
|
4. If the pod was evicted, the StatefulSet recreates it. Rerun the jobs that
|
||||||
were lost, once.
|
were lost, once. A job that was running at the time may stay "in progress"
|
||||||
|
until Gitea's task timeout fails it; cancel it first. See
|
||||||
|
[soak-test.md](soak-test.md) for what this looks like in a job's log.
|
||||||
|
|
||||||
### kubectl itself fails
|
### kubectl itself fails
|
||||||
|
|
||||||
@@ -222,5 +224,6 @@ I3 and S3: run the runner on `inspiron-3670` instead of in the cluster.
|
|||||||
|
|
||||||
- Roadmap: label `cicd` on DevOps/software-infrastructure, milestones
|
- Roadmap: label `cicd` on DevOps/software-infrastructure, milestones
|
||||||
"CI/CD 0" to "CI/CD 4".
|
"CI/CD 0" to "CI/CD 4".
|
||||||
|
- S8 soak test and silent-failure evidence: [soak-test.md](soak-test.md).
|
||||||
- Agents coordinate CI on the forum's Agents category, topic "Gitea Actions
|
- Agents coordinate CI on the forum's Agents category, topic "Gitea Actions
|
||||||
runner: status and changes".
|
runner: status and changes".
|
||||||
|
|||||||
@@ -0,0 +1,82 @@
|
|||||||
|
# S8 soak test: proving jobs no longer die silently
|
||||||
|
|
||||||
|
Procedure for DevOps/software-infrastructure#41 (S8). The workflow is
|
||||||
|
[`soak/soak.yaml`](soak/soak.yaml). This file is public: record hostnames,
|
||||||
|
node names and anything security-related in the private assessment, not here.
|
||||||
|
|
||||||
|
## What "dying silently" looks like
|
||||||
|
|
||||||
|
A job stops printing output, then shows **failure** with no step marked
|
||||||
|
failed, or stays **in progress** long after it should have ended. Evidence so
|
||||||
|
far (2026-11-13, two jobs from one project repo):
|
||||||
|
|
||||||
|
| Job | Last log line (UTC) | Runner pod evicted for low ephemeral storage (UTC) | Job result |
|
||||||
|
|---|---|---|---|
|
||||||
|
| A | 14:28:17 | about 14:28 (replacement pod started 14:28:35) | failure at 14:39:25, the running step marked failed with no error of its own |
|
||||||
|
| B | started 14:46:28 | 14:52:26 | still "in progress" 15 minutes later |
|
||||||
|
|
||||||
|
In both cases the job's last output lines up with the runner pod being evicted
|
||||||
|
from its node. The node's disk filled while jobs pulled and built images. The
|
||||||
|
same evictions also removed other pods on those nodes, not only CI.
|
||||||
|
|
||||||
|
This supports eviction as the cause, but it isn't proof that eviction is the
|
||||||
|
only cause. The soak test below is what closes S8.
|
||||||
|
|
||||||
|
## Preconditions
|
||||||
|
|
||||||
|
1. S3, S4 and S5 (#36, #37, #38) and S6 (#39) are live. Check with the
|
||||||
|
commands in [runner.md](runner.md#reading-logs): the runner has the expected
|
||||||
|
capacity, resource limits and `docker-prune` sidecar (or, if the runner moved
|
||||||
|
to a dedicated host under I3, #45, that host's Compose project is up).
|
||||||
|
2. No other agent's CI is running heavy work. Check the Actions tab of the busy
|
||||||
|
project repos, and announce the soak in the Agents topic "Gitea Actions
|
||||||
|
runner: status and changes" at least an hour before.
|
||||||
|
3. A throwaway repo owned by `claude`, named `cicd-test-soak`, holding only
|
||||||
|
`.gitea/workflows/soak.yaml` copied from `soak/soak.yaml`.
|
||||||
|
|
||||||
|
## Running it
|
||||||
|
|
||||||
|
1. Start the `soak` workflow from the repo's Actions tab with `rounds` = 20.
|
||||||
|
It runs two soak jobs at a time at most, so it never takes more than the
|
||||||
|
runner's capacity, and it has one `control-failure` job.
|
||||||
|
2. While it runs, keep a read-only watch:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
kubectl -n gitea get events --watch
|
||||||
|
kubectl -n gitea logs gitea-actions-runner-0 -c runner -f
|
||||||
|
```
|
||||||
|
|
||||||
|
For a host runner, use `docker compose logs -f` in its project directory.
|
||||||
|
3. Expect about 5 to 10 minutes per soak job, depending on CPU.
|
||||||
|
|
||||||
|
## Pass criteria
|
||||||
|
|
||||||
|
- All 20 `soak` jobs finish **success**, and none sits "in progress" after the
|
||||||
|
workflow ends.
|
||||||
|
- `control-failure` finishes **failure** at the step "Expected failure", with
|
||||||
|
exit code 3 in the log. That shows failures are still reported accurately.
|
||||||
|
- No runner restart, eviction or `OOMKilled` event in the window.
|
||||||
|
- Every job's "Record end" step shows Docker disk use back near where it
|
||||||
|
started, so cleanup is keeping up.
|
||||||
|
|
||||||
|
If a job fails, line up its last log line against the events and the runner
|
||||||
|
log, as in the table above. A failure with a real step error (for example a
|
||||||
|
pull timeout) is a finding to fix, not a silent death, but it still resets
|
||||||
|
the count: the acceptance criterion is 20 consecutive good jobs.
|
||||||
|
|
||||||
|
## After the soak
|
||||||
|
|
||||||
|
- Record results on #41: job count, durations, peak disk use, any failures and
|
||||||
|
their cause. Keep node names out of it.
|
||||||
|
- Post a short summary to the Agents topic so other agents stop working around
|
||||||
|
the problem.
|
||||||
|
- Rerun any failed `claude/vitals` CI once its owning task is idle, or by
|
||||||
|
agreement on Discourse.
|
||||||
|
- Delete `cicd-test-soak` or leave it archived and note which on #41.
|
||||||
|
|
||||||
|
## Until S3 to S6 are live
|
||||||
|
|
||||||
|
When the runner is evicted mid-job, the job is not cleaned up promptly. Gitea
|
||||||
|
marks it failed only when its task times out. Until S6's shorter zombie
|
||||||
|
timeout is live, an orphaned job can stay "in progress" for hours. **Cancel
|
||||||
|
it, then rerun once.** Don't change the code because of it.
|
||||||
@@ -0,0 +1,107 @@
|
|||||||
|
# S8 soak test for the shared Gitea Actions runner.
|
||||||
|
#
|
||||||
|
# Copy this file to `.gitea/workflows/soak.yaml` in a throwaway repo owned by
|
||||||
|
# `claude` (named `cicd-test-*`), then start it from the Actions tab. It is
|
||||||
|
# kept here, not in this repo's `.gitea/workflows/`, so it never runs on a push
|
||||||
|
# to software-infrastructure. Procedure: docs/ci-cd/soak-test.md.
|
||||||
|
#
|
||||||
|
# 20 soak jobs plus one control job. Each soak job pulls large images, compiles
|
||||||
|
# Go's standard library, and churns some disk, so it exercises the conditions
|
||||||
|
# under which jobs used to die without a step error. Every job prints the time
|
||||||
|
# and disk usage at the start and end of each step, so a job that stops
|
||||||
|
# reporting can be lined up against runner and kubelet events.
|
||||||
|
|
||||||
|
name: soak
|
||||||
|
|
||||||
|
on:
|
||||||
|
workflow_dispatch:
|
||||||
|
inputs:
|
||||||
|
rounds:
|
||||||
|
description: "Number of soak jobs (the acceptance test needs 20)"
|
||||||
|
default: "20"
|
||||||
|
|
||||||
|
concurrency:
|
||||||
|
group: soak-${{ github.repository }}
|
||||||
|
cancel-in-progress: false
|
||||||
|
|
||||||
|
env:
|
||||||
|
# Pinned tags; keep in step with what the runner's job images allow.
|
||||||
|
PULL_IMAGES: >-
|
||||||
|
golang:1.25.4-bookworm
|
||||||
|
node:22.21.1-bookworm
|
||||||
|
postgres:17.7-bookworm
|
||||||
|
GO_IMAGE: golang:1.25.4-bookworm
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
plan:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
timeout-minutes: 5
|
||||||
|
outputs:
|
||||||
|
rounds: ${{ steps.r.outputs.rounds }}
|
||||||
|
steps:
|
||||||
|
- id: r
|
||||||
|
run: |
|
||||||
|
n="${{ inputs.rounds }}"
|
||||||
|
case "$n" in ''|*[!0-9]*) n=20 ;; esac
|
||||||
|
[ "$n" -gt 40 ] && n=40
|
||||||
|
printf 'rounds=[%s]\n' "$(seq -s, 1 "$n")" >> "$GITHUB_OUTPUT"
|
||||||
|
|
||||||
|
soak:
|
||||||
|
needs: plan
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
timeout-minutes: 45
|
||||||
|
strategy:
|
||||||
|
fail-fast: false
|
||||||
|
# Two at a time, so the runner sees concurrent jobs when its capacity
|
||||||
|
# allows it, without crowding out other repos' CI.
|
||||||
|
max-parallel: 2
|
||||||
|
matrix:
|
||||||
|
round: ${{ fromJSON(needs.plan.outputs.rounds) }}
|
||||||
|
steps:
|
||||||
|
- name: Record start
|
||||||
|
run: |
|
||||||
|
echo "round ${{ matrix.round }} start $(date -u +%FT%TZ)"
|
||||||
|
df -h / /var/lib/docker 2>/dev/null || df -h /
|
||||||
|
docker system df || true
|
||||||
|
|
||||||
|
- name: Pull large images
|
||||||
|
run: |
|
||||||
|
for img in $PULL_IMAGES; do
|
||||||
|
echo "$(date -u +%T) pull $img"
|
||||||
|
docker pull -q "$img"
|
||||||
|
done
|
||||||
|
docker system df
|
||||||
|
|
||||||
|
- name: Heavy compile
|
||||||
|
run: |
|
||||||
|
echo "$(date -u +%T) compile start"
|
||||||
|
# The daemon can't see the job's files, so nothing is bind-mounted.
|
||||||
|
docker run --rm "$GO_IMAGE" sh -c 'go build -a std && echo compiled'
|
||||||
|
echo "$(date -u +%T) compile end"
|
||||||
|
|
||||||
|
- name: Disk churn
|
||||||
|
run: |
|
||||||
|
vol="soak-${{ github.run_id }}-${{ matrix.round }}"
|
||||||
|
docker volume create "$vol" >/dev/null
|
||||||
|
docker run --rm -v "$vol:/data" "$GO_IMAGE" \
|
||||||
|
sh -c 'dd if=/dev/zero of=/data/blob bs=1M count=2048 status=none && sync && ls -lh /data'
|
||||||
|
docker volume rm "$vol" >/dev/null
|
||||||
|
df -h / /var/lib/docker 2>/dev/null || df -h /
|
||||||
|
|
||||||
|
- name: Record end
|
||||||
|
if: always()
|
||||||
|
run: |
|
||||||
|
echo "round ${{ matrix.round }} end $(date -u +%FT%TZ)"
|
||||||
|
docker system df || true
|
||||||
|
|
||||||
|
# A job that must fail at a named step. If it ever shows success, or fails
|
||||||
|
# with no step marked failed, status reporting is not accurate.
|
||||||
|
control-failure:
|
||||||
|
needs: plan
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
timeout-minutes: 5
|
||||||
|
steps:
|
||||||
|
- name: Expected failure
|
||||||
|
run: |
|
||||||
|
echo "this step fails on purpose"
|
||||||
|
exit 3
|
||||||
Reference in New Issue
Block a user