271 lines
16 KiB
Markdown
Executable File
271 lines
16 KiB
Markdown
Executable File
# Luanti WASM Kubernetes role
|
||
|
||
Deploy the **browser client**, not a Luanti game server. This follows the neighboring
|
||
Owncast role's controller-local, `kubernetes.core.k8s`, role defaults, Jinja manifest,
|
||
and deployment-playbook conventions. There is no Helm release, PVC, database,
|
||
game-server Deployment, or mutable server configuration.
|
||
|
||
## Architecture and prerequisites
|
||
|
||
- A stateless Deployment puts the nginx **web runtime target** and **lwproxy
|
||
sidecar in the same pod**. nginx serves TCP 8080 and forwards `/proxy` WebSocket
|
||
requests to `http://127.0.0.1:7777`. Browser game traffic becomes UDP to the
|
||
explicitly allowed external game destination.
|
||
- One **ClusterIP-only Service**, with **only TCP 8080**. Port 7777 is never
|
||
exposed by a Service, hostPort, NodePort, LoadBalancer, or Ingress.
|
||
- The proxy listens on `0.0.0.0:7777` so kubelet TCP startup/readiness/liveness
|
||
probes can reach the pod IP. Binding only to localhost would make those probes
|
||
fail. The mandatory NetworkPolicy permits ingress **only to TCP 8080**;
|
||
normal kubelet/node health-probe traffic is exempt from Kubernetes network
|
||
isolation. Confirm your CNI's node-probe behavior in staging.
|
||
- Kubernetes with `apps/v1`, `networking.k8s.io/v1` and a **NetworkPolicy-enforcing
|
||
CNI** is required. The role cannot detect a CNI that silently ignores policy;
|
||
there is deliberately **no policy-disable or unsupported-CNI bypass**.
|
||
- Use a Linux Ansible controller (including WSL), Python 3.10+ and ansible-core
|
||
2.16–2.19. Ansible 2.19 needs Python 3.11+ on the controller. Install
|
||
[requirements.txt](requirements.txt) in your controller environment and the
|
||
collection in [requirements.yml](requirements.yml). The version is pinned to
|
||
match the neighboring role; the small CIDR filter uses only Python's standard
|
||
library, not `ansible.utils` or `netaddr`.
|
||
- The controller must have a working kubeconfig or `K8S_AUTH_*` settings and
|
||
permissions to manage namespaces, ConfigMaps, Deployments, Services, Ingresses,
|
||
and NetworkPolicies. No kubeconfig, credentials, RBAC binding, or service-account
|
||
token is placed in the workload. Kubernetes API certificate verification stays
|
||
enabled; use your normal trusted controller configuration.
|
||
- Optional ingress needs a WebSocket-capable ingress controller, DNS, and a
|
||
**browser-trusted TLS certificate**. cert-manager is optional. With no issuer,
|
||
create the named TLS Secret yourself in the workload namespace. A private
|
||
`ca-issuer` requires distributing that CA's trust to every browser.
|
||
|
||
## Build and supply real images
|
||
|
||
There are **no default image names**: the application CI does not publish these
|
||
runtime images. Build, scan, and push both images to a registry you control. The
|
||
following commands are conceptual, run **from your Luanti application checkout**;
|
||
set `REGISTRY`, `RELEASE`, and `GAME_HOST` to your actual values first:
|
||
|
||
```sh
|
||
docker build --target web \
|
||
--build-arg LUANTI_SERVER_ADDRESS="$GAME_HOST" \
|
||
--build-arg LUANTI_SERVER_PORT=30000 \
|
||
-t "$REGISTRY/luanti-web:$RELEASE" .
|
||
docker build -f docker/proxy.Dockerfile -t "$REGISTRY/lwproxy:$RELEASE" .
|
||
docker push "$REGISTRY/luanti-web:$RELEASE"
|
||
docker push "$REGISTRY/lwproxy:$RELEASE"
|
||
```
|
||
|
||
Use the application's `web` target, **not** its `build` or `artifacts` target.
|
||
The confirmed image contracts are:
|
||
|
||
| Image | Runtime contract |
|
||
| --- | --- |
|
||
| web | nginx entrypoint; `/usr/share/nginx/html`; `/etc/nginx/nginx.conf`; numeric UID/GID 101; TCP 8080 |
|
||
| proxy | `lwproxy --config /etc/lwproxy.toml`; numeric UID/GID 65532; TCP 7777 |
|
||
|
||
ConfigMap mounts replace exactly those two config paths. Preserve these contracts
|
||
in custom images. Root filesystems are read-only, nginx writes only to a bounded
|
||
memory-backed `/tmp`, and the proxy requires no writable volume.
|
||
|
||
The game server **hostname and port are embedded at image build time** through
|
||
`LUANTI_SERVER_ADDRESS` / `LUANTI_SERVER_PORT`. The role does not edit or mount
|
||
`server.conf`, and changing role variables does **not** change the embedded
|
||
destination. Rebuild the web image to change it. Keep the DNS A/AAAA addresses,
|
||
`luanti_wasm_game_cidrs`, and `luanti_wasm_game_port` aligned with the destination
|
||
and address families your proxy actually uses. The IPv4 proxy listener can still
|
||
serve a dual-stack pod via nginx loopback; egress IPv6 requires cluster support.
|
||
|
||
Both image variables require a non-`latest` explicit tag or a complete
|
||
`@sha256:` digest (64 lowercase hex characters). Mutable non-latest tags are
|
||
accepted, but **digests are recommended** for reproducibility and supply-chain
|
||
verification. Use admission signature/provenance policies and scan your builds.
|
||
Reusing a mutable tag alone does not trigger a rollout. Image pull credentials
|
||
must be pre-created Kubernetes Secrets, referenced by name, never embedded here.
|
||
|
||
## Deploy
|
||
|
||
From the role checkout, install dependencies and make the parent directory a
|
||
role search path:
|
||
|
||
```sh
|
||
python -m pip install -r requirements.txt
|
||
ansible-galaxy collection install -r requirements.yml
|
||
export ANSIBLE_ROLES_PATH="$(dirname "$PWD")${ANSIBLE_ROLES_PATH:+:$ANSIBLE_ROLES_PATH}"
|
||
ansible-playbook -i localhost, playbooks/deploy-eom.yml -e @deployment-vars.yml
|
||
```
|
||
|
||
Create your own deployment variables file. This example contains intentionally
|
||
nonexistent registry names and documentation-only IPs; **replace every fixture
|
||
with your pushed images and resolved game destination**:
|
||
|
||
```yaml
|
||
eom_luanti_wasm_web_image: registry.example.invalid/your-team/luanti-web:your-release
|
||
eom_luanti_wasm_proxy_image: registry.example.invalid/your-team/lwproxy:your-release
|
||
eom_luanti_wasm_game_cidrs:
|
||
- 192.0.2.15/32
|
||
- 2001:db8::15/128
|
||
# Optional; only approved content-service addresses, never 0.0.0.0/0 or ::/0.
|
||
eom_luanti_wasm_https_cidrs: []
|
||
```
|
||
|
||
[playbooks/deploy-eom.yml](playbooks/deploy-eom.yml) requires the three
|
||
`eom_luanti_wasm_*` image/game inputs before any Kubernetes operation. It enables
|
||
TLS ingress at **luanti.eom.dev**, ingress class **nginx**, using the
|
||
**ca-issuer ClusterIssuer** and the **luanti-wasm-tls** Secret. It assumes the
|
||
image was built for your intended server (the app currently defaults to
|
||
luanti.eom.dev:30000); it does not create that server. If sharing this hostname,
|
||
ensure its web ingress and external game UDP routing resolve as intended.
|
||
|
||
For another site, invoke the role directly:
|
||
|
||
```yaml
|
||
- name: Deploy browser game client
|
||
hosts: localhost
|
||
connection: local
|
||
gather_facts: false
|
||
vars:
|
||
luanti_wasm_web_image: "{{ my_pushed_web_image }}"
|
||
luanti_wasm_proxy_image: "{{ my_pushed_proxy_image }}"
|
||
luanti_wasm_game_cidrs: "{{ my_game_server_cidrs }}"
|
||
luanti_wasm_ingress_enabled: true
|
||
luanti_wasm_hostname: play.example.org
|
||
luanti_wasm_ingress_class: nginx
|
||
luanti_wasm_tls_secret_name: browser-game-tls
|
||
# Empty issuer: supply the browser-trusted TLS Secret externally.
|
||
luanti_wasm_cluster_issuer: ""
|
||
luanti_wasm_ingress_peers:
|
||
- namespaceSelector:
|
||
matchLabels:
|
||
kubernetes.io/metadata.name: ingress-nginx
|
||
podSelector:
|
||
matchLabels:
|
||
app.kubernetes.io/component: controller
|
||
roles:
|
||
- ansible-role-luanti-wasm
|
||
```
|
||
|
||
Inspect your controller labels before using the example ingress selectors.
|
||
`namespaceSelector` and `podSelector` in **one peer** are ANDed; separate peer
|
||
entries are ORed. A podSelector alone selects pods in the workload namespace.
|
||
Ingress peers may also use validated `ipBlock` CIDRs with contained `except`
|
||
networks. An empty peer list omits `from`, allowing any source to TCP 8080 only.
|
||
|
||
## Configuration reference
|
||
|
||
All settings are in [defaults/main.yml](defaults/main.yml).
|
||
|
||
| Variable(s), prefixed `luanti_wasm_` | Default / purpose |
|
||
| --- | --- |
|
||
| `namespace`, `name` | `luanti-wasm`; DNS labels, at most 53 characters to reserve suffix space; name starts with a letter |
|
||
| `web_image`, `proxy_image` | Empty, **required**; distinct runtime image inputs |
|
||
| `image_pull_policy`, `image_pull_secrets` | `IfNotPresent`, `[]`; secrets are `{name: existing-secret}` objects |
|
||
| `replicas` | Integer `1`; may scale to a positive integer, but see WebSocket caveats below |
|
||
| `web_resources`, `proxy_resources` | Each requests `100m` CPU / `64Mi`; limits `500m` / `256Mi`; require both requests and limits |
|
||
| `tmp_size_limit` | `64Mi`; positive whole `Ki`, `Mi`, or `Gi` memory-backed nginx scratch space, charged to pod/container memory usage |
|
||
| `node_selector`, `tolerations` | `{}`, `[]` |
|
||
| `wait`, `wait_timeout` | `true`, `300` seconds; wait for Deployment readiness, configurable for GitOps/slow starts |
|
||
| `ingress_enabled`, `hostname`, `ingress_class` | `false`, empty, empty; disabled ingress is removed on subsequent runs |
|
||
| `tls_enabled`, `tls_secret_name` | `true`, name plus `-tls`; **TLS must be true whenever ingress is enabled** |
|
||
| `cluster_issuer` | Empty; optional cert-manager ClusterIssuer name |
|
||
| `ingress_annotations` | nginx read/send timeout `3600` seconds and SSL redirect enabled; replace/extend the dictionary for your controller |
|
||
| `websocket_timeout_seconds` | Integer `3600`; nginx side of the long-lived WebSocket; also adjust ingress timeouts to match |
|
||
| `ingress_peers` | `[]`, any source on **8080 only**; restrict to your ingress controller in production |
|
||
| `game_cidrs`, `game_port` | Empty, **required nonempty**, and UDP `30000`; canonical IPv4/IPv6 CIDRs only |
|
||
| `https_cidrs` | `[]`, no external TCP 443 by default |
|
||
| `http_allowed_hosts` | Exact `content.luanti.org`, `servers.luanti.org`; JSON-escaped array written as valid TOML |
|
||
|
||
Resource quantities are ultimately validated by the Kubernetes API; tune limits
|
||
for actual traffic and account for nginx tmpfs. Requests must not exceed limits.
|
||
Role assertions validate identities, image references, scalars, collection/peer
|
||
shapes, ingress hostname/TLS, exact proxy hostnames, and canonical CIDRs. Scoped
|
||
IPv6, host bits, malformed addresses, and `/0` wildcard networks are rejected.
|
||
Use narrow `/32` and `/128` destination rules where possible. The role does not
|
||
make DNS lookups to silently broaden or update an allowlist.
|
||
|
||
## Security and operational boundaries
|
||
|
||
- NetworkPolicy is applied **before the Deployment**, selecting its pod labels.
|
||
Ingress permits TCP 8080 only; egress permits **only** DNS TCP/UDP 53 to
|
||
`kube-system` pods labeled `k8s-app: kube-dns`, game UDP to the explicit CIDRs
|
||
and port, and optional TCP 443 to explicit HTTPS CIDRs. There are no empty
|
||
egress peers or wildcard destinations. This assumes CoreDNS/kube-dns with those
|
||
labels. NodeLocal DNSCache, custom DNS namespaces, NAT, host-network ingress,
|
||
and CNI-specific service-IP handling require operator review and testing;
|
||
do not work around mismatches by opening arbitrary egress.
|
||
- Kubernetes NetworkPolicies are **additive**: another policy can widen access.
|
||
Audit all policies selecting these pods, node firewall rules, and namespace
|
||
access. NetworkPolicy is pod-wide, not per-container; nginx and lwproxy share
|
||
the same narrow egress permissions. It does not protect against privileged
|
||
node access or enforce loopback isolation.
|
||
- Kubernetes standard NetworkPolicy **does not support FQDN egress**. The proxy
|
||
host allowlist is application-layer filtering, not protection from **DNS
|
||
rebinding**. Keep CIDRs narrow; allowlisted hosts resolving outside them will
|
||
fail closed. Never add pod, service, node, metadata, or internal control-plane
|
||
networks to external content allowlists. Content/CDN addresses change; maintain
|
||
approved CIDRs yourself, or use a separately reviewed CNI-specific FQDN/egress
|
||
gateway policy outside this role. Do not use `0.0.0.0/0` or `::/0` as a shortcut.
|
||
- Without HTTPS CIDRs, content/server-list HTTP proxy requests are intentionally
|
||
blocked even though those hostnames appear in TOML. DNS and game traffic still
|
||
work. Redirects, CDN aliases, extra ports, or new hostnames require explicit
|
||
review; no automatic unrestricted egress is added. Browser-direct external
|
||
fetches are not governed by pod NetworkPolicy and must satisfy COEP/CORS too.
|
||
- Web and proxy run as numeric non-root users **101** and **65532**, drop **ALL**
|
||
capabilities, disallow privilege escalation, use read-only roots, and inherit
|
||
`seccompProfile: RuntimeDefault`. The pod never mounts an API token. There are
|
||
no embedded credentials or certificates: reference existing pull/TLS Secrets
|
||
and authenticate only on the controller to avoid leaking secrets into source.
|
||
- TLS terminates at ingress; keep the in-cluster HTTP hop within a trusted,
|
||
isolated cluster network. Game UDP is the native Luanti protocol, not added
|
||
TLS. Configure cluster ingress TLS policy/ciphers and certificate renewal in
|
||
your ingress infrastructure. SharedArrayBuffer/threaded WASM needs a **secure
|
||
context** and the preserved COOP `same-origin`, COEP `require-corp`, and
|
||
`X-Content-Type-Options: nosniff` headers on pages/workers. Do not strip them at
|
||
ingress. If ingress is disabled, provide trusted HTTPS elsewhere (or use
|
||
localhost port-forwarding for local testing), not remote plain HTTP.
|
||
- Non-release paths use `no-cache`; only paths matching the app's immutable
|
||
**12-lowercase-hex release directory** regex get one-year immutable caching.
|
||
A SHA256 checksum of the full rendered ConfigMap changes the pod template
|
||
whenever either runtime config changes, because `subPath` mounts do not
|
||
update in-place. Upstream runtime/image contract changes need revalidation.
|
||
- Existing WebSocket sessions belong to one proxy pod. Scaling is supported for
|
||
new sessions, not session migration; rolling updates/restarts disconnect active
|
||
games. A startup/readiness TCP probe proves the listener is reachable, not
|
||
successful game UDP connectivity. Deployment wait does not prove ingress TLS,
|
||
NetworkPolicy enforcement, content access, or an actual browser game session.
|
||
|
||
## Offline tests and staging validation
|
||
|
||
With dependencies installed, run from the role checkout:
|
||
|
||
```sh
|
||
sh tests/run.sh
|
||
```
|
||
|
||
[tests/test.yml](tests/test.yml) includes only `tasks_from: validate.yml` (with
|
||
`public: true`) and `tasks_from: render.yml`, never the deployment tasks. It uses
|
||
documentation-network and nonexistent-registry fixtures, renders and parses real
|
||
YAML, and checks:
|
||
|
||
- same-pod layout, ClusterIP 8080-only routing, actual image/config mount contracts;
|
||
- headers, immutable-cache regex, loopback WebSocket upstream and proxy bind;
|
||
- security contexts, probes, tmpfs limits, resources, and absence of PVCs;
|
||
- default-deny port isolation and exact DNS/game/optional HTTPS destinations;
|
||
- deterministic config checksum and config-triggered rollout changes;
|
||
- actual EOM playbook variables, TLS ingress, optional cert-manager and annotations;
|
||
- rejection of empty/unversioned/latest images, malformed/wildcard CIDRs, bad
|
||
names/hostnames, peer shapes, missing TLS, invalid resources/types and timeouts.
|
||
|
||
[tests/run.sh](tests/run.sh) also syntax-checks the deployment playbook and runs
|
||
its missing-input preflight, verifying that it fails before namespace creation.
|
||
It runs standard-library unit tests in [tests/test_network.py](tests/test_network.py)
|
||
for canonical IPv4/IPv6 parsing and strict ingress CIDR exception containment.
|
||
Negative test assertion failures are **expected and rescued**; the final recap
|
||
must still show `failed=0` and the runner's success message. No cluster, image
|
||
pull, container build, or WASM compiler is required. Run in a network-disabled
|
||
sandbox for extra assurance.
|
||
|
||
Before production, separately verify admission/server-side schema acceptance,
|
||
actual image startup with read-only roots, CNI drop behavior on 7777 and unwanted
|
||
egress, allowed UDP game connectivity, trusted TLS/HTTPS redirects, and browser
|
||
`crossOriginIsolated` plus WebSocket game join. These runtime properties cannot
|
||
be established by offline manifest rendering alone.
|