diff --git a/README.md b/README.md index b6c8069..65ce309 100644 --- a/README.md +++ b/README.md @@ -81,7 +81,7 @@ An empty list deliberately starts an **isolated validator**. There is no automat discovery, bootnode list, or automatic reciprocal configuration: configure the other operators' nodes too. All nodes must run the same network/genesis/consensus. For mainnet, application startup additionally requires at least two distinct -public HTTPS peer origins, an arrived image-pinned genesis time, the exact chain-ID +peer origins (public HTTPS by default), an arrived image-pinned genesis time, the exact chain-ID approval, and explicit unaudited-risk acceptance. These switches are not launch approval from this role; the project currently records a NO-GO decision. The role does not discover or verify peer independence. @@ -102,7 +102,8 @@ HTTPS peer origins on another port, add that port to rules. Browser Wikimedia image fetches are client-side; validators no longer fetch random Wikipedia articles. -For an explicitly private testnet only, set `wikideck_allow_private_peers: true` +For an explicitly private testnet or operator-controlled mainnet candidate cluster, +set `wikideck_allow_private_peers: true` and provide `wikideck_network_policy_peer_egress` rules with destination selectors and numeric TCP ports. This opt-in permits **both private addresses and HTTP** in the application; it does not automatically open network policy. For example: @@ -128,7 +129,12 @@ wikideck_network_policy_peer_egress: The receiving node must separately allow the sender through its ingress policy. Service DNAT handling depends on the CNI; the example selects the destination pod and its target port 3001 rather than opening all destinations on Service port 80. -Do not enable private-peer exceptions for public peer configuration. +Do not enable private-peer exceptions for public peer configuration. Updated +mainnet candidate images advertise `privatePeersSupported: true` in their manifest; +older images reject the private opt-in. Browser HTTPS, chain approval, risk +acknowledgment and arrived genesis remain mandatory. Private HTTP RPC is not +authenticated or encrypted; keep it within a trusted cluster boundary or add +independently configured mesh transport protection. Deploy multiple validators as **separate role instances with different names and PVCs** (and separate hostnames if using ingress), not by scaling one Deployment. @@ -136,6 +142,45 @@ Sharing a machine/cluster/storage failure domain does not provide independent operator security. Empty local chains can synchronize from configured peers; never seed a second running node by sharing its writable PVC. +### Three-node latest candidate playbook + +[playbooks/wikideck.yaml](playbooks/wikideck.yaml) configures the public `wikideck` +node plus `wikideck-peer-a` and `wikideck-peer-b` within the same namespace. Each +uses its own Service and 2 TiB PVC (6 TiB requested in total), reciprocal internal +DNS peer URLs, scoped validator-pod ingress/egress, and public TCP 443 egress for +Wikipedia article and Wikimedia statistics requests used by mining templates. +Standard NetworkPolicy cannot select FQDNs; private address ranges remain excluded +from that public HTTPS rule. Only the public node gets +the `wikideck.eom.dev` TLS ingress. All three are created before readiness waits. +This deploys validators, not miners, and does not establish independent operators. + +The playbook uses this role checkout directly, not a separately installed older +Galaxy role. It defaults to `ericomeehan/wikideck:latest` with pull policy `Always` +and `wikideck_candidate_risk_ack: operator-accepts-unaudited-risk`. This explicit +candidate workflow is not the digest-pinned public-launch example. Before creating +ledger resources, a short-lived hardened Kubernetes Job runs the image's +`--network-manifest` command. The playbook generates `wikideck_candidate_chain_id` +from its output and rejects older images without private-peer support, non-mainnet +images, or a future image-baked genesis. The Job is removed even on failure; it +mounts no ledger, wallet secrets, or service-account token. + +Rebuild and publish `latest` from the updated application before running this +playbook. The default Dockerfile genesis is November 1, 2026, 00:00 UTC +(`1793491200`), so a normal rebuild can start now without custom genesis arguments. +Older images retain their old genesis; publishing updated source is required. +Never change the chain identity of an existing ledger without a deliberate backup +and migration plan. Do not move `latest` while inspection/deployment is in progress: +all nodes must pull the same image. For reproducible public releases use the +digest-pinned [examples/playbook.yml](examples/playbook.yml) instead. A mutable tag +with `Always` is pulled when pods start. The playbook records the inspected +container image ID in a pod-template annotation, so rerunning after publishing a +new `latest` triggers a rollout; unchanged image IDs do not force a restart. +Preserve existing data and keep the same genesis for updates to an operating chain. + +The controller needs Job creation/deletion and Pod/log reads in addition to the +role's permissions. Manifest inspection needs a live cluster, so this playbook +rejects check mode; the role's offline render tests require no cluster credentials. + ## Variables All defaults are in [defaults/main.yml](defaults/main.yml). diff --git a/defaults/main.yml b/defaults/main.yml index b55a151..899cbbe 100644 --- a/defaults/main.yml +++ b/defaults/main.yml @@ -19,7 +19,7 @@ wikideck_image_pull_secrets: [] # Static peer origins, at most 16. Empty means an isolated validator, not a network. wikideck_peers: [] -# Test/private networks only; permits private addresses AND HTTP in the app. +# Operator-controlled test/private candidate clusters; permits private addresses AND HTTP. wikideck_allow_private_peers: false # Explicit mainnet candidate startup gates. Never set these before launch approval. wikideck_mainnet_launch: "" @@ -51,6 +51,8 @@ wikideck_resources: wikideck_node_selector: {} wikideck_tolerations: [] wikideck_affinity: {} +# Pod-template annotations can trigger a rollout when a mutable image tag changes. +wikideck_pod_annotations: {} # Trust only literal proxy source IPs/CIDRs whose edge overwrites forwarded headers. wikideck_trusted_proxies: [] # Optional private operations listener; application binds it to container loopback. diff --git a/playbooks/wikideck.yaml b/playbooks/wikideck.yaml index 0ab3ab9..a037e1d 100644 --- a/playbooks/wikideck.yaml +++ b/playbooks/wikideck.yaml @@ -1,45 +1,268 @@ -- name: Deploy Wikideck +- name: Deploy a three-node private Wikideck mainnet candidate hosts: alpha-control-plane become: true vars_files: - ~/Desktop/secrets.yaml + vars: + # This is an operator-controlled candidate, not an independent public network. + wikideck_release_image: ericomeehan/wikideck:latest + wikideck_candidate_risk_ack: operator-accepts-unaudited-risk + wikideck_cluster_nodes: + - wikideck + - wikideck-peer-a + - wikideck-peer-b + wikideck_namespace: wikideck + wikideck_image: "{{ wikideck_release_image }}" + wikideck_image_pull_policy: Always + # Roll out latest only when the inspected image changes, not on every run. + wikideck_pod_annotations: + wikideck.eom.dev/image-id: "{{ wikideck_manifest_image_id }}" + wikideck_allow_private_peers: true + wikideck_mainnet_launch: "{{ wikideck_candidate_chain_id }}" + wikideck_mainnet_risk_ack: "{{ wikideck_candidate_risk_ack }}" + # Internal validators share the browser origin, but only the first has ingress. + wikideck_public_origin: https://wikideck.eom.dev + wikideck_ingress_host: wikideck.eom.dev + wikideck_ingress_class_name: nginx + wikideck_ingress_annotations: + cert-manager.io/cluster-issuer: ca-issuer + nginx.ingress.kubernetes.io/ssl-redirect: "true" + wikideck_storage_size: 2Ti + wikideck_storage_class: nfs-client + # Populate only after confirming the real proxy source IPs/CIDRs. + wikideck_trusted_proxies: "{{ wikideck_ingress_proxy_sources | default([]) }}" + # Start every node before waiting; the first must not wait for absent peers. + wikideck_wait: false + wikideck_wait_timeout: 900 + wikideck_network_policy_ingress_from: + - podSelector: + matchLabels: + app.kubernetes.io/name: wikideck + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: ingress-nginx + # Mining templates fetch Wikipedia articles and Wikimedia statistics over HTTPS. + # Standard NetworkPolicy cannot select FQDNs; permit public TCP 443, not private ranges. + wikideck_network_policy_https_to: + - ipBlock: + cidr: 0.0.0.0/0 + except: + - 10.0.0.0/8 + - 100.64.0.0/10 + - 127.0.0.0/8 + - 169.254.0.0/16 + - 172.16.0.0/12 + - 192.168.0.0/16 + - ipBlock: + cidr: "::/0" + except: + - "::1/128" + - "fc00::/7" + - "fe80::/10" + wikideck_network_policy_peer_egress: + - to: + - podSelector: + matchLabels: + app.kubernetes.io/name: wikideck + ports: + - protocol: TCP + port: 3001 pre_tasks: - - name: Require an explicitly approved, immutable mainnet release + - name: Validate candidate cluster inputs ansible.builtin.assert: that: - - wikideck_mainnet_deployment_approved | default(false) | bool - - wikideck_release_image is defined and wikideck_release_image is string and wikideck_release_image is match('^.+@sha256:[a-f0-9]{64}$') - - wikideck_mainnet_peers is defined and wikideck_mainnet_peers is sequence and wikideck_mainnet_peers is not string and wikideck_mainnet_peers | length >= 2 - - wikideck_mainnet_peers | unique | list | length == wikideck_mainnet_peers | length - - wikideck_candidate_chain_id is defined and wikideck_candidate_chain_id is match('^[a-f0-9]{64}$') + - not ansible_check_mode + - wikideck_release_image is string and wikideck_release_image is match('^[^\s@]+:latest$') + - wikideck_cluster_nodes is sequence and wikideck_cluster_nodes is not string and wikideck_cluster_nodes is not mapping + - wikideck_cluster_nodes | length == 3 + - wikideck_cluster_nodes | unique | list | length == 3 + - wikideck_cluster_nodes | select('match', '^[a-z]([a-z0-9-]*[a-z0-9])?$') | list | length == 3 + - wikideck_cluster_nodes | map('length') | max <= 50 - wikideck_candidate_risk_ack | default('') == 'operator-accepts-unaudited-risk' - - wikideck_release_genesis_time is defined and (wikideck_release_genesis_time | string) is match('^[1-9][0-9]{8,10}$') - - wikideck_release_genesis_time | int >= 1700000000 and wikideck_release_genesis_time | int <= 4102444800 - - wikideck_release_genesis_time | int <= ansible_date_time.epoch | int fail_msg: >- - Mainnet is currently NO-GO. Do not deploy until the operator has approved - a digest-pinned image with a verified frozen manifest, arrived genesis, - at least two independent public HTTPS peers, and explicit risk acceptance. - roles: - - role: ericomeehan.wikideck + This candidate playbook requires a latest-tag image, three unique node + names, and operator-accepts-unaudited-risk. Check mode cannot inspect the + image manifest; use the offline render tests instead. + run_once: true + + - name: Initialize manifest registry Secret references + ansible.builtin.set_fact: + wikideck_manifest_pull_secrets: [] + run_once: true + + - name: Convert registry Secret names to Kubernetes references + ansible.builtin.set_fact: + wikideck_manifest_pull_secrets: "{{ wikideck_manifest_pull_secrets + [{'name': item}] }}" + loop: "{{ wikideck_image_pull_secrets | default([]) }}" + run_once: true + + - name: Create namespace for read-only image inspection + kubernetes.core.k8s: + kubeconfig: "{{ wikideck_kubeconfig | default(omit, true) }}" + context: "{{ wikideck_kube_context | default(omit, true) }}" + state: present + apply: true + definition: + apiVersion: v1 + kind: Namespace + metadata: + name: "{{ wikideck_namespace }}" + run_once: true + + - name: Inspect latest without opening a ledger or starting a validator + run_once: true + block: + - name: Remove a stale manifest inspection Job + kubernetes.core.k8s: + kubeconfig: "{{ wikideck_kubeconfig | default(omit, true) }}" + context: "{{ wikideck_kube_context | default(omit, true) }}" + state: absent + api_version: batch/v1 + kind: Job + namespace: "{{ wikideck_namespace }}" + name: wikideck-candidate-manifest + wait: true + + - name: Run the image manifest command + kubernetes.core.k8s: + kubeconfig: "{{ wikideck_kubeconfig | default(omit, true) }}" + context: "{{ wikideck_kube_context | default(omit, true) }}" + state: present + wait: true + wait_timeout: 300 + wait_condition: + type: Complete + status: "True" + definition: + apiVersion: batch/v1 + kind: Job + metadata: + name: wikideck-candidate-manifest + namespace: "{{ wikideck_namespace }}" + spec: + backoffLimit: 0 + activeDeadlineSeconds: 240 + ttlSecondsAfterFinished: 600 + template: + spec: + restartPolicy: Never + automountServiceAccountToken: false + enableServiceLinks: false + imagePullSecrets: "{{ wikideck_manifest_pull_secrets }}" + securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + seccompProfile: + type: RuntimeDefault + containers: + - name: manifest + image: "{{ wikideck_release_image }}" + imagePullPolicy: Always + command: + - node + - --experimental-strip-types + - server/index.ts + - --network-manifest + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: [ALL] + resources: + requests: {cpu: 100m, memory: 128Mi} + limits: {cpu: "1", memory: 512Mi} + + - name: Locate the completed manifest pod + kubernetes.core.k8s_info: + kubeconfig: "{{ wikideck_kubeconfig | default(omit, true) }}" + context: "{{ wikideck_kube_context | default(omit, true) }}" + kind: Pod + namespace: "{{ wikideck_namespace }}" + label_selectors: + - job-name=wikideck-candidate-manifest + register: wikideck_manifest_pods + + - name: Require exactly one completed manifest pod + ansible.builtin.assert: + that: + - wikideck_manifest_pods.resources | length == 1 + - wikideck_manifest_pods.resources[0].status.phase == 'Succeeded' + + - name: Read the public image manifest + kubernetes.core.k8s_log: + kubeconfig: "{{ wikideck_kubeconfig | default(omit, true) }}" + context: "{{ wikideck_kube_context | default(omit, true) }}" + namespace: "{{ wikideck_namespace }}" + name: "{{ wikideck_manifest_pods.resources[0].metadata.name }}" + container: manifest + register: wikideck_manifest_log + + - name: Parse the image identity + ansible.builtin.set_fact: + wikideck_candidate_manifest: "{{ wikideck_manifest_log.log | from_json }}" + + - name: Require a compatible image and an arrived image-baked genesis + ansible.builtin.assert: + that: + - wikideck_candidate_manifest.profile == 'mainnet' + - wikideck_candidate_manifest.network == 'wikideck-randomx-mainnet-1' + - wikideck_candidate_manifest.privatePeersSupported | default(false) == true + - wikideck_candidate_manifest.chainId is string and wikideck_candidate_manifest.chainId is match('^[a-f0-9]{64}$') + - wikideck_candidate_manifest.genesisTime is integer + - wikideck_candidate_manifest.genesisTime >= 1700000000 and wikideck_candidate_manifest.genesisTime <= 4102444800 + - wikideck_candidate_manifest.genesisTime <= ansible_date_time.epoch | int + fail_msg: >- + Rebuild and publish latest from code supporting private candidate + peers, with a deliberately chosen, arrived mainnet genesis frozen + into the image. Runtime genesis overrides are not supported. + + - name: Generate the candidate chain approval from this image + ansible.builtin.set_fact: + wikideck_candidate_chain_id: "{{ wikideck_candidate_manifest.chainId }}" + wikideck_release_genesis_time: "{{ wikideck_candidate_manifest.genesisTime }}" + wikideck_manifest_image_id: "{{ wikideck_manifest_pods.resources[0].status.containerStatuses[0].imageID }}" + always: + - name: Remove the manifest inspection Job and its pod + kubernetes.core.k8s: + kubeconfig: "{{ wikideck_kubeconfig | default(omit, true) }}" + context: "{{ wikideck_kube_context | default(omit, true) }}" + state: absent + api_version: batch/v1 + kind: Job + namespace: "{{ wikideck_namespace }}" + name: wikideck-candidate-manifest + wait: true + tasks: + - name: Deploy each validator with its own service and persistent ledger + ansible.builtin.include_role: + # Use this updated checkout rather than an older installed Galaxy role. + name: "{{ playbook_dir }}/.." # noqa: role-name[path] vars: - wikideck_image: "{{ wikideck_release_image }}" - wikideck_image_pull_policy: IfNotPresent - wikideck_peers: "{{ wikideck_mainnet_peers }}" - wikideck_mainnet_launch: "{{ wikideck_candidate_chain_id }}" - wikideck_mainnet_risk_ack: "{{ wikideck_candidate_risk_ack }}" - wikideck_public_origin: https://wikideck.eom.dev - wikideck_ingress_enabled: true - wikideck_ingress_host: wikideck.eom.dev - wikideck_ingress_class_name: nginx - wikideck_ingress_annotations: - cert-manager.io/cluster-issuer: ca-issuer - # Match the actual ingress controller namespace; add its podSelector if known. - wikideck_network_policy_ingress_from: - - namespaceSelector: - matchLabels: - kubernetes.io/metadata.name: ingress-nginx - wikideck_storage_size: 2Ti - wikideck_storage_class: nfs-client - # Populate only after confirming the real proxy source IPs/CIDRs. - wikideck_trusted_proxies: "{{ wikideck_ingress_proxy_sources | default([]) }}" + wikideck_name: "{{ wikideck_cluster_node }}" + wikideck_existing_claim: "" + wikideck_ingress_enabled: "{{ wikideck_cluster_node == wikideck_cluster_nodes[0] }}" + wikideck_peers: >- + {{ wikideck_cluster_nodes | reject('equalto', wikideck_cluster_node) + | map('regex_replace', '^', 'http://') + | map('regex_replace', '$', '.' ~ wikideck_namespace ~ '.svc.cluster.local:80') | list }} + loop: "{{ wikideck_cluster_nodes }}" + loop_control: + loop_var: wikideck_cluster_node + run_once: true + + - name: Wait for all three validators after creating the full peer topology + kubernetes.core.k8s_info: + kubeconfig: "{{ wikideck_kubeconfig | default(omit, true) }}" + context: "{{ wikideck_kube_context | default(omit, true) }}" + api_version: apps/v1 + kind: Deployment + namespace: "{{ wikideck_namespace }}" + name: "{{ item }}" + wait: true + wait_timeout: "{{ wikideck_wait_timeout | int }}" + wait_condition: + type: Available + status: "True" + loop: "{{ wikideck_cluster_nodes }}" + run_once: true diff --git a/tasks/validate.yml b/tasks/validate.yml index 1f9f4d8..e66549a 100644 --- a/tasks/validate.yml +++ b/tasks/validate.yml @@ -19,6 +19,7 @@ - wikideck_resources.limits.cpu is defined and wikideck_resources.limits.memory is defined - wikideck_node_selector is mapping - wikideck_affinity is mapping + - wikideck_pod_annotations is mapping - wikideck_ingress_annotations is mapping - wikideck_storage_size is string and wikideck_storage_size | length > 0 - wikideck_storage_access_modes | length > 0 @@ -95,7 +96,7 @@ - wikideck_allow_private_peers or item is match('^https://') fail_msg: >- Peers must be HTTPS origins without credentials, paths, commas, queries or - fragments. Private/HTTP test peers require wikideck_allow_private_peers=true. + fragments. Private/HTTP cluster peers require wikideck_allow_private_peers=true. Wikideck additionally validates ports and public DNS/IP addresses at runtime. loop: "{{ wikideck_peers }}" diff --git a/templates/deployment.yml.j2 b/templates/deployment.yml.j2 index d0744cc..d22f15c 100644 --- a/templates/deployment.yml.j2 +++ b/templates/deployment.yml.j2 @@ -20,6 +20,7 @@ spec: app.kubernetes.io/instance: {{ wikideck_name | to_json }} template: metadata: + annotations: {{ wikideck_pod_annotations | to_json }} labels: app.kubernetes.io/name: wikideck app.kubernetes.io/instance: {{ wikideck_name | to_json }} diff --git a/tests/render.yml b/tests/render.yml index 0b40c4a..b20a4a0 100644 --- a/tests/render.yml +++ b/tests/render.yml @@ -13,6 +13,8 @@ wikideck_image: "{{ render_case.image | default('registry.example.com/wikideck:1.0.0') }}" wikideck_name: "{{ render_case.name | default('wikideck') }}" wikideck_namespace: "{{ render_case.namespace | default('wikideck') }}" + wikideck_image_pull_policy: "{{ render_case.pull_policy | default('IfNotPresent') }}" + wikideck_pod_annotations: "{{ render_case.pod_annotations | default({}) }}" wikideck_peers: "{{ render_case.peers | default([]) }}" wikideck_allow_private_peers: "{{ render_case.allow_private | default(false) }}" wikideck_mainnet_launch: "{{ render_case.mainnet_launch | default('') }}" @@ -84,6 +86,52 @@ - label: exact mainnet launch configuration mainnet_launch: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa risk_ack: operator-accepts-unaudited-risk + - label: latest private mainnet candidate public node + image: ericomeehan/wikideck:latest + pull_policy: Always + pod_annotations: + wikideck.eom.dev/image-id: docker-pullable://ericomeehan/wikideck@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa + peers: [http://wikideck-peer-a.wikideck.svc.cluster.local:80, http://wikideck-peer-b.wikideck.svc.cluster.local:80] + allow_private: true + ingress: true + mainnet_launch: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa + risk_ack: operator-accepts-unaudited-risk + # Public HTTPS is needed for Wikipedia mining templates and statistics. + https_to: + - ipBlock: + cidr: 0.0.0.0/0 + except: [10.0.0.0/8, 100.64.0.0/10, 127.0.0.0/8, 169.254.0.0/16, 172.16.0.0/12, 192.168.0.0/16] + - ipBlock: + cidr: "::/0" + except: ["::1/128", "fc00::/7", "fe80::/10"] + ingress_from: + - podSelector: + matchLabels: + app.kubernetes.io/name: wikideck + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: ingress-nginx + peer_egress: + - to: + - podSelector: + matchLabels: + app.kubernetes.io/name: wikideck + ports: [{protocol: TCP, port: 3001}] + - label: latest private mainnet candidate internal peer + image: ericomeehan/wikideck:latest + pull_policy: Always + name: wikideck-peer-a + peers: [http://wikideck.wikideck.svc.cluster.local:80, http://wikideck-peer-b.wikideck.svc.cluster.local:80] + allow_private: true + mainnet_launch: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa + risk_ack: operator-accepts-unaudited-risk + https_to: [] + peer_egress: + - to: + - podSelector: + matchLabels: + app.kubernetes.io/name: wikideck + ports: [{protocol: TCP, port: 3001}] loop_control: loop_var: render_case label: "{{ render_case.label }}" diff --git a/tests/render_case.yml b/tests/render_case.yml index 6995d6e..143a761 100644 --- a/tests/render_case.yml +++ b/tests/render_case.yml @@ -18,6 +18,7 @@ that: - deployment.spec.replicas == 1 - deployment.spec.strategy.type == 'Recreate' + - deployment.spec.template.metadata.annotations == wikideck_pod_annotations - deployment.spec.selector.matchLabels == deployment.spec.template.metadata.labels - wikideck_test_manifests.service.spec.selector == deployment.spec.selector.matchLabels - wikideck_test_manifests.networkpolicy.spec.podSelector.matchLabels == deployment.spec.selector.matchLabels @@ -33,6 +34,7 @@ - not container.securityContext.allowPrivilegeEscalation - container.securityContext.capabilities.drop == ['ALL'] - container.image == wikideck_image + - container.imagePullPolicy == wikideck_image_pull_policy - container.resources == wikideck_resources - container.ports[0].containerPort == 3001 - container.startupProbe.httpGet.path == '/healthz'