S4: Set runner resources, capacity, and job timeouts #37

Open
opened 2026-11-09 17:05:57 +00:00 by claude · 1 comment
Member

The runner has no CPU or memory requests or limits and runs two jobs at once, on nodes that mostly have 1 CPU. It uses default job and shutdown timeouts.

Work

  • Add CPU and memory requests and limits for the runner and DinD containers.
  • Set capacity to what the chosen node can sustain (likely 1 until a build host exists).
  • Set runner.timeout, shutdown_timeout, fetch_timeout, and a matching terminationGracePeriodSeconds so jobs can finish or report during a restart.
  • Set container.force_pull: false and pin the three runner image labels instead of latest where practical.

Acceptance

  • Values live in role defaults and the rendered config is shown in a comment.
  • A forced pod restart produces a clear job failure rather than a silent one.

Depends on: S2. Label ready-to-deploy when committed.

The runner has no CPU or memory requests or limits and runs two jobs at once, on nodes that mostly have 1 CPU. It uses default job and shutdown timeouts. **Work** - Add CPU and memory requests and limits for the runner and DinD containers. - Set `capacity` to what the chosen node can sustain (likely 1 until a build host exists). - Set `runner.timeout`, `shutdown_timeout`, `fetch_timeout`, and a matching `terminationGracePeriodSeconds` so jobs can finish or report during a restart. - Set `container.force_pull: false` and pin the three runner image labels instead of `latest` where practical. **Acceptance** - Values live in role defaults and the rendered config is shown in a comment. - A forced pod restart produces a clear job failure rather than a silent one. Depends on: S2. Label `ready-to-deploy` when committed.
claude added this to the CI/CD 0: Stabilize the runner milestone 2026-11-09 17:05:57 +00:00
claude added the
priority
high
cicd
labels 2026-11-09 17:05:57 +00:00
claude added this to the CI/CD Roadmap project 2026-11-09 17:07:33 +00:00
claude moved this to To Do in CI/CD Roadmap on 2026-11-09 17:07:59 +00:00
Author
Member

Committed, ready to deploy together with S3 (#36). DevOps/ansible-role-gitea@5fb2d9c, bumped in 91b81e5.

What changed (all role defaults, with comments)

  • Resources. Runner container: request 50m CPU / 64Mi, limit 500m / 256Mi. DinD: request 500m CPU / 2Gi, limit 2 CPU / 8Gi, plus S3's ephemeral-storage values. Sized for one job at a time on a 2-CPU, 16 GB worker, which is S3's recommended node. A job that goes over the DinD memory limit is OOM-killed inside the pod and fails with a step error, rather than the node evicting the whole runner.
  • capacity: 1 (was 2).
  • Timeouts. timeout: 2h, kept below Gitea's ENDLESS_TASK_TIMEOUT from S6 (#39) so the runner, which can say why, fails a long job first. fetch_timeout: 10s.
  • Shutdown. Chart 0.1.2 doesn't expose terminationGracePeriodSeconds, so the pod keeps Kubernetes' 30 s default. I set shutdown_timeout: 15s and report_close_timeout: 10s so that on SIGTERM the runner cancels running jobs and reports them to Gitea inside those 30 s. Before this, shutdown_timeout was 0. DinD runs as a native sidecar, so it stops after the runner and stays up while jobs are cancelled.
  • Images. force_pull: false. The three labels are pinned to runner-images:ubuntu-24.04-v26.11.03 and ubuntu-22.04-v26.11.03. On 2026-11-10 those have the same digests as the floating ubuntu-24.04 and ubuntu-22.04 tags, so nothing changes for jobs except that the image stops moving.
  • Option names were checked against gitea-runner generate-config from the docker.gitea.com/runner:2.0.1 image that's deployed.

Rendered runner config (Ansible render of the role values, then helm template on chart 0.1.2):

container:
  require_docker: true
  docker_timeout: 300s
  force_pull: false
runner:
  capacity: 1
  timeout: 2h
  shutdown_timeout: 15s
  report_close_timeout: 10s
  fetch_timeout: 10s
  labels:
    - "ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-24.04-v26.11.03"
    - "ubuntu-24.04:docker://docker.gitea.com/runner-images:ubuntu-24.04-v26.11.03"
    - "ubuntu-22.04:docker://docker.gitea.com/runner-images:ubuntu-22.04-v26.11.03"

The StatefulSet renders the requests and limits above for runner and dind.

Tested: the values above went through Ansible (stub secrets, Helm module replaced by a stub that echoes its values) and then through helm template. gitea.yaml --syntax-check passes, and --list-tasks --tags actions is unchanged: only the runner tasks. Nothing was applied.

Deploy. This comes along with S3, because S3's node-label check is already in the role:

kubectl label node <chosen-node> eom.dev/ci-runner=true
ansible-playbook -i inventories/eom.dev.yml gitea.yaml --tags actions

Add your usual vault option.

  • Effect: the runner pod is recreated on the labeled node with the new limits and config. Gitea itself isn't touched.
  • Risk: the restart kills any running CI jobs, so check the Actions pages first. If the chosen node can't fit a 500m CPU request, the pod stays Pending. In that case lower gitea_actions_dind_cpu_request or pick another node. With capacity 1, jobs queue instead of running in parallel.
  • Rollback: git revert 91b81e5 in the superproject, then rerun the same command.

Not yet verified: the second acceptance criterion, that a forced pod restart produces a clear job failure, can only be checked after deploy. That's S8 (#41).

— eom-cicd-roadmap

**Committed, ready to deploy together with S3 (#36).** DevOps/ansible-role-gitea@5fb2d9c, bumped in 91b81e5. **What changed (all role defaults, with comments)** - **Resources.** Runner container: request 50m CPU / 64Mi, limit 500m / 256Mi. DinD: request 500m CPU / 2Gi, limit 2 CPU / 8Gi, plus S3's ephemeral-storage values. Sized for one job at a time on a 2-CPU, 16 GB worker, which is S3's recommended node. A job that goes over the DinD memory limit is OOM-killed inside the pod and fails with a step error, rather than the node evicting the whole runner. - **`capacity: 1`** (was 2). - **Timeouts.** `timeout: 2h`, kept below Gitea's `ENDLESS_TASK_TIMEOUT` from S6 (#39) so the runner, which can say why, fails a long job first. `fetch_timeout: 10s`. - **Shutdown.** Chart 0.1.2 doesn't expose `terminationGracePeriodSeconds`, so the pod keeps Kubernetes' 30 s default. I set `shutdown_timeout: 15s` and `report_close_timeout: 10s` so that on SIGTERM the runner cancels running jobs and reports them to Gitea inside those 30 s. Before this, `shutdown_timeout` was 0. DinD runs as a native sidecar, so it stops after the runner and stays up while jobs are cancelled. - **Images.** `force_pull: false`. The three labels are pinned to `runner-images:ubuntu-24.04-v26.11.03` and `ubuntu-22.04-v26.11.03`. On 2026-11-10 those have the same digests as the floating `ubuntu-24.04` and `ubuntu-22.04` tags, so nothing changes for jobs except that the image stops moving. - Option names were checked against `gitea-runner generate-config` from the `docker.gitea.com/runner:2.0.1` image that's deployed. **Rendered runner config** (Ansible render of the role values, then `helm template` on chart 0.1.2): ```yaml container: require_docker: true docker_timeout: 300s force_pull: false runner: capacity: 1 timeout: 2h shutdown_timeout: 15s report_close_timeout: 10s fetch_timeout: 10s labels: - "ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-24.04-v26.11.03" - "ubuntu-24.04:docker://docker.gitea.com/runner-images:ubuntu-24.04-v26.11.03" - "ubuntu-22.04:docker://docker.gitea.com/runner-images:ubuntu-22.04-v26.11.03" ``` The StatefulSet renders the requests and limits above for `runner` and `dind`. **Tested:** the values above went through Ansible (stub secrets, Helm module replaced by a stub that echoes its values) and then through `helm template`. `gitea.yaml --syntax-check` passes, and `--list-tasks --tags actions` is unchanged: only the runner tasks. Nothing was applied. **Deploy.** This comes along with S3, because S3's node-label check is already in the role: ``` kubectl label node <chosen-node> eom.dev/ci-runner=true ansible-playbook -i inventories/eom.dev.yml gitea.yaml --tags actions ``` Add your usual vault option. - **Effect:** the runner pod is recreated on the labeled node with the new limits and config. Gitea itself isn't touched. - **Risk:** the restart **kills any running CI jobs**, so check the Actions pages first. If the chosen node can't fit a 500m CPU request, the pod stays Pending. In that case lower `gitea_actions_dind_cpu_request` or pick another node. With capacity 1, jobs queue instead of running in parallel. - **Rollback:** `git revert 91b81e5` in the superproject, then rerun the same command. **Not yet verified:** the second acceptance criterion, that a forced pod restart produces a clear job failure, can only be checked after deploy. That's S8 (#41). — eom-cicd-roadmap
claude added the ready-to-deploy label 2026-11-10 16:02:35 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DevOps/software-infrastructure#37