G4: Run www and membership tests in CI #52

Open
opened 2026-11-09 17:06:02 +00:00 by claude · 1 comment
Member

DevOps/www holds the membership and billing app, and has tests, but no workflow runs them. This is the most valuable first test gate.

Work: add a workflow in DevOps/www for the test suites in tests/ and membership/, using test settings and test keys only.

Acceptance

  • Tests run on push and PR with no live Stripe or production credentials.
  • A deliberately failing test fails the workflow.
  • Existing failures are fixed or filed.

Depends on: S8.

DevOps/www holds the membership and billing app, and has tests, but no workflow runs them. This is the most valuable first test gate. **Work**: add a workflow in DevOps/www for the test suites in `tests/` and `membership/`, using test settings and test keys only. **Acceptance** - Tests run on push and PR with no live Stripe or production credentials. - A deliberately failing test fails the workflow. - Existing failures are fixed or filed. Depends on: S8.
claude added this to the CI/CD 2: Quality gates milestone 2026-11-09 17:06:02 +00:00
claude added the
priority
critical
cicd
labels 2026-11-09 17:06:02 +00:00
claude added this to the CI/CD Roadmap project 2026-11-09 17:07:33 +00:00
Author
Member

Prepared ahead of S8: workflow committed in DevOps/www, tested locally, not yet run on the runner. Commits DevOps/www@7fe6047, @74fd1d8, @fd22bc6.

What's committed

  • ci/run-tests.sh runs each suite in throwaway containers built from the checkout: site (node --test tests/*.test.mjs), membership (Django on in-memory SQLite) and postgres (the same Django suite on a disposable PostgreSQL 17 with tmpfs storage), or all.
    • Test containers run with --network none. The PostgreSQL run uses an --internal Docker network shared only with the database. So no test can reach Stripe, SMTP, LDAP, Dex or production, whatever the code does. No secrets or keys are passed in. The DB password is random per run, and test_settings already forces STRIPE_LIVE_MODE = False.
    • The checkout goes to Docker as a build context (minus .git), not a bind mount, because the runner's Docker daemon can't see job files.
    • Containers, the network and the images are removed on exit, including on failure.
  • ci/site-test.Dockerfile (node 24.22.2-slim) and ci/membership-test.Dockerfile (same python:3.12-slim base as membership/Dockerfile, with the whole repo copied in, because some tests read htdocs/ and membership/tiers.eom.json).
  • .gitea/workflows/tests.yaml: jobs site and membership (SQLite step, then PostgreSQL step). Manual-only until S8 (#41), with the push/pull_request trigger in its header. No secrets, contents: read, concurrency per ref, timeouts 15 and 30 min, actions/checkout pinned by SHA. The push started no runs; I checked.
  • README "Verification" points to sh ci/run-tests.sh all.

Existing failures, fixed

  • Three test expectations were stale after the homepage and members page switched from Matrix to Harmony (c5d7dc7): two node tests and one Django test still expected Matrix, the old chat link and icon, and the old Gold kicker. Updated to the intended content (7fe6047).
  • test_published_tier_copy_matches_role_defaults read a repo-root file before its own skip check, so it errored in the app-only image that membership/compose.test.yaml builds. Reordered (fd22bc6). That fixture now passes too (217 tests, 6 skipped).

Tested on the workstation, in containers

Case Result
site 75/75 pass
membership (SQLite) 217 run, OK, 5 skipped
postgres 217 run, OK, 2 skipped (the 3 PostgreSQL row-lock tests run here)
Planted failing node test site exits 1
Planted failing Django test membership and postgres exit 1
After every run no leftover containers, networks or images
Network inside a --network none container outbound HTTPS fails at name resolution

actionlint 1.7.12, shellcheck, hadolint 2.15.1 and yamllint pass (yamllint warns only about line length on the pinned uses: lines).

Not tested: a run on the shared runner.

Remaining before close

  1. S8 passes. Then switch the trigger to push/pull_request, run it once, and show a planted failure failing the workflow on the runner (in a cicd-test-* copy, not in www).

Follow-ups, not blocking

  • Skips that stay skipped in CI: the tier-copy check needs ansible-role-www checked out next to www, and the listmonk round trip needs a disposable listmonk. The first could be added later by cloning the public role in the job.
  • membership/%SystemDrive%/ProgramData/... holds three Windows cache files committed by accident. They're harmless. I left them alone; deleting them is Eric's call.

— eom-cicd-roadmap

**Prepared ahead of S8: workflow committed in DevOps/www, tested locally, not yet run on the runner.** Commits DevOps/www@7fe6047, @74fd1d8, @fd22bc6. **What's committed** - `ci/run-tests.sh` runs each suite in throwaway containers built from the checkout: `site` (`node --test tests/*.test.mjs`), `membership` (Django on in-memory SQLite) and `postgres` (the same Django suite on a disposable PostgreSQL 17 with tmpfs storage), or `all`. - Test containers run with `--network none`. The PostgreSQL run uses an `--internal` Docker network shared only with the database. So no test can reach Stripe, SMTP, LDAP, Dex or production, whatever the code does. No secrets or keys are passed in. The DB password is random per run, and `test_settings` already forces `STRIPE_LIVE_MODE = False`. - The checkout goes to Docker as a build context (minus `.git`), not a bind mount, because the runner's Docker daemon can't see job files. - Containers, the network and the images are removed on exit, including on failure. - `ci/site-test.Dockerfile` (node 24.22.2-slim) and `ci/membership-test.Dockerfile` (same `python:3.12-slim` base as `membership/Dockerfile`, with the whole repo copied in, because some tests read `htdocs/` and `membership/tiers.eom.json`). - `.gitea/workflows/tests.yaml`: jobs `site` and `membership` (SQLite step, then PostgreSQL step). **Manual-only until S8 (#41)**, with the `push`/`pull_request` trigger in its header. No secrets, `contents: read`, `concurrency` per ref, timeouts 15 and 30 min, `actions/checkout` pinned by SHA. The push started no runs; I checked. - README "Verification" points to `sh ci/run-tests.sh all`. **Existing failures, fixed** - Three test expectations were stale after the homepage and members page switched from Matrix to Harmony (c5d7dc7): two node tests and one Django test still expected Matrix, the old chat link and icon, and the old Gold kicker. Updated to the intended content (7fe6047). - `test_published_tier_copy_matches_role_defaults` read a repo-root file before its own skip check, so it errored in the app-only image that `membership/compose.test.yaml` builds. Reordered (fd22bc6). That fixture now passes too (217 tests, 6 skipped). **Tested on the workstation, in containers** | Case | Result | |---|---| | `site` | 75/75 pass | | `membership` (SQLite) | 217 run, OK, 5 skipped | | `postgres` | 217 run, OK, 2 skipped (the 3 PostgreSQL row-lock tests run here) | | Planted failing node test | `site` exits 1 | | Planted failing Django test | `membership` and `postgres` exit 1 | | After every run | no leftover containers, networks or images | | Network inside a `--network none` container | outbound HTTPS fails at name resolution | `actionlint` 1.7.12, `shellcheck`, `hadolint` 2.15.1 and `yamllint` pass (yamllint warns only about line length on the pinned `uses:` lines). **Not tested:** a run on the shared runner. **Remaining before close** 1. S8 passes. Then switch the trigger to `push`/`pull_request`, run it once, and show a planted failure failing the workflow on the runner (in a `cicd-test-*` copy, not in www). **Follow-ups, not blocking** - Skips that stay skipped in CI: the tier-copy check needs `ansible-role-www` checked out next to www, and the listmonk round trip needs a disposable listmonk. The first could be added later by cloning the public role in the job. - `membership/%SystemDrive%/ProgramData/...` holds three Windows cache files committed by accident. They're harmless. I left them alone; deleting them is Eric's call. — eom-cicd-roadmap
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DevOps/software-infrastructure#52