wikideck/docs/operations-recovery.md

Deployment, private monitoring and recovery — stage 4

Production remains NO-GO. This stage provides tools and a local recovery rehearsal, not an installed public edge, configured alerts or independent operational sign-off. No mainnet approval, node launch, real payment, production backup/restore or existing data modification was performed. Tools are available on Node 24 and included in the runtime image where noted.

Explicit reverse-proxy trust

WIKIDECK_TRUSTED_PROXIES is empty/untrusted by default. It accepts up to 16 comma-separated literal IPs or CIDRs (IPv4 prefix 8–32; IPv6 32–128). It rejects booleans, hop counts, names and universal /0 ranges. Set only the actual source address/subnet of the trusted edge connection. A large permitted subnet can still be unsafe: do not include unrelated tenants/containers. The candidate Compose file forwards the setting but leaves it empty.

The nginx example is not installed or deployment-validated. It illustrates TLS, request/body/connection/time limits and overwriting X-Forwarded-For with the actual remote client address rather than appending attacker input. Replace the reserved invalid hostname and certificate paths; review TLS/HSTS/ciphers, renewals, compression, static caching and per-IP/global admission for your edge. Validate nginx configuration on the intended version with real certificates before use. No actual TLS/certificate test was performed here.

Topology matters: nginx on the host forwarding to a container's published loopback port may appear inside the container as a Docker gateway, not 127.0.0.1. Determine the source and use a reviewed dedicated network; do not blindly copy a loopback trust list. Firewall direct backend access. Anyone who can connect from a trusted source can forge forwarded IPs unless the edge overwrites headers and isolates the backend. Application trust settings do not configure the firewall. Existing global RPC budgets remain independent of forwarded client IPs.

Private diagnostics

Set WIKIDECK_OPS_PORT to an unused integer port (1–65535) to enable a separate HTTP listener bound to 127.0.0.1 only. Unset/empty disables it. Do not reuse the main RPC port. The default/candidate Compose definitions do not enable or publish this port. For container monitoring, execute the checker inside that container/network namespace; publishing a container port does not make a loopback listener safely accessible.

Only GET /status is supported. Responses are no-store JSON capped at 64 KiB; connection count/timeouts are bounded. No password/token is required because access is confined to the local-process trust boundary. Therefore all users/processes with loopback access can read it; it is not suitable for an untrusted multi-user host without additional isolation. Never reverse proxy, tunnel broadly, add it to public routing, or publish it to the Internet. The listener has no administrative/mutation operations.

Reported aggregates:

• Node uptime and Node-process RSS; RSS excludes the RandomX subprocess, so monitor container/cgroup/host totals externally.
• Event-loop mean/max since listener startup (not rolling percentiles).
• Canonical height, DAG block count, pending transactions, ledger mutation queue, process-local reorg count, tip timestamp/age. Tip age is miner-timestamp-based, not proof of local receipt freshness or synchronization.
• Native subprocess failure/active/queue signals. available means no recorded failure/exit; it is not a startup known-vector test or proof of successful current hashing.
• Configured/connected peer counts, API/P2P active/rejected counts. Peer count is connectivity, not independently administered hashpower or consensus security. Counters reset on restart.
• Market mutation queue, last successful reconciliation timestamp and counts grouped by order state. No addresses, order IDs, balances or signed transactions are included. Diagnostics still reveal operational activity and must remain private.
• Disk bytes available and ledger/market DB/WAL sizes, sampled asynchronously every 30 seconds. Missing sample/failure is explicit; absence of disk is not healthy. First startup may precede the first successful sample. Sampling does not impose a quota or prevent exhaustion.

ops:check -- --port <private-port> reads only literal local loopback with a five-second deadline and 64 KiB streaming cap, emitting JSON plus a nonzero exit for warning conditions. No remote target or credentials are accepted. Built-in starting-point warning thresholds: observation clock/freshness >60s; no peers; empty/stale chain (>1h by tip timestamp); native unavailable; queue pressure (native/market >=48 or ledger >=192); mean event-loop delay >250ms; stale market reconciliation >60s; review/refund-required orders; disk sample >90s old/unavailable or <512 MiB free. A fresh empty isolated testnet correctly warns. These thresholds are not production SLOs and must be reviewed against real workload/chain timing.

Run the checker with your restricted local scheduler/monitoring agent and route warning codes to the actual on-call system. This repository does not create an alert account, install a scheduler or verify delivery. Externally monitor cumulative counter changes (reorgs/rejections), process/container totals, disk growth rate, peer/tip freshness, native failures and checker availability itself. A failed/unreachable operations listener must alert, not silently pass. Keep /healthz as liveness only; HTTP 200 is not launch/readiness evidence.

Stopped-node pair backup

The node:backup script supports create, verify, restore. It deliberately does not offer online pair backup or overwrite/force/reset. Choose WIKIDECK_NETWORK explicitly to match the data; it validates the derived chain/network identity without enabling launch. The runtime includes the script and server helpers; it does not require dev dependencies.

Create

1. Stop the node cleanly; verify all processes/writers using that data are stopped. Do not run Ledger/Market library writers directly while maintenance is active. Do not rely solely on a lock file for an arbitrary unmanaged process.
2. Invoke node:backup with action create, explicit --source <node-data-directory>, new --destination <backup-directory> and --consent. Source is the parent of the selected testnet/mainnet folder, not the individual SQLite file. Destination must be outside that data directory and must not exist. Its parent must already exist.
3. The tool acquires the same exclusive network node lock as normal startup. Any lock, even stale, refuses maintenance. The tool never deletes a preexisting lock. Inspect stale ownership/process status manually before separately authorized removal.
4. Validate SQLite quick checks and network/market metadata, then use SQLite VACUUM INTO for both ledger and market to produce standalone snapshot files (including committed source WAL contents, not a naive live-file copy). A symlinked network directory or database file is refused. Pair consistency relies on all writers staying stopped for the entire operation.
5. Database copies are checked, hashed with SHA-256 and flushed. A strict versioned manifest records names/sizes/hashes/network/profile/chain and creation time. The manifest completion marker is written/flushed last. POSIX directory entries are flushed; Windows/remote filesystem and storage-controller power-loss behavior requires real environment validation. Partial output is deliberately retained for investigation, not automatically trusted or deleted.

Only the two databases and manifest are included: not browser wallets, backup passwords, deployment configuration, secrets, certificates, peer settings, native binaries or logs. Keep each category independently backed up with versioned recovery instructions. Backups contain sensitive marketplace participant/order information and are not encrypted by this tool. Store in restricted, encrypted, access-audited off-host storage with tested retention; do not put them in Git/build contexts. Unix modes are restrictive where supported; Windows ACLs and network-share permissions must be enforced by the operator.

Verify

Action verify needs --source <backup-directory> only. It reads a bounded strict manifest, requires exactly one ledger and one market filename, checks regular files/sizes/hashes and refuses unexpected SQLite WAL/SHM/journal sidecars, then checks SQLite and metadata. It does not launch or initialize a new node. Verification opens SQLite read-only, but OS access timestamps and filesystem-specific read behavior are not a forensic immutability guarantee.

Hashes detect corruption, not authenticity: an attacker able to replace both manifest and files can recompute hashes. Sign/protect the backup package externally, keep trusted manifests off-host and restrict directory ownership. No wallet-control or signature proof is inferred from a checksum. This tool assumes trusted local filesystem administration; it does not defend against a privileged concurrent file-replacement attacker or hostile mounted storage.

Restore to a new isolated directory

1. Action restore requires explicit --source <verified-backup-directory>, a nonexistent --destination <new-data-directory>, and --consent. Existing destinations are refused even when empty. There is no in-place reset or deletion.
2. Verify the package before creating output, copy only fixed database names exclusively, recheck hashes/SQLite metadata and flush. No stale lock, user secrets or deployment approval variables are restored. A partial failed restore is retained; do not try to start it or merge it into a working node.
3. The tool reports proofsRevalidated: false: database/hash verification is not chain verification. Reopen/replay with the trusted reference RandomX release in an isolated reviewed rehearsal before declaring it usable. Preserve market instance identity, orders, fees, payout markers and nonce receipts together. Compare the expected tip/work/ownership/order state against the trusted recovery point.
4. Reconcile later chain events, payment state and any external processors before any future resumption of valuable activity. Restoring stale databases does not roll back on-chain transfers or real payouts. Live payments remain disabled and have no reviewed disaster recovery process in this release.

An operator must choose recovery point/retention/RPO/RTO, practice off-host recovery and approve any production restore explicitly. Never remove a lock or replace a running volume just to make a recovery test pass.

Evidence and remaining hold points

Automated tests cover default proxy distrust/spoofed-header rejection, explicit loopback proxy recognition, invalid trust/port settings, private listener and threshold checks, paired backup/restore identity preservation, refusal to overwrite/ignore locks/back up within data, and rejection of tampering, wrong network, incomplete packages and unexpected sidecars.

The native HTTP integration now cleanly stops its disposable nodes, backs up their real three-block RandomX chain and market journal, restores to a fresh disposable directory, opens/revalidates copied proofs with the actual reference engine, and checks canonical ownership, market instance, order buyer and nonce receipt history. It then restarts the original isolated fixture. This is a local automated drill, not an off-host operational recovery or crash/power-loss certification.

Final Node 24 TypeScript/Vite and production runtime builds passed. The suite passed 98 tests, 0 failed, 0 skipped. A separate packaged-runtime smoke drill ran with no external network, read-only root, dropped capabilities and a disposable 64 MiB tmpfs: monitoring correctly returned nonzero for isolation, empty chain, not-yet-reconciled market and low disk; clean stopped-node backup, verification and new-directory restore each exited zero. That empty-chain CLI drill reported proof revalidation as false, as designed; nonempty real-proof replay is covered by the integration drill above. No host volume or production wallet was used.

SEC-07/SEC-09 are only partially addressed: actual edge/source-IP/TLS checks, a production monitoring agent and verified alert delivery, off-host backup storage, long-history restore timing, filesystem/power-loss testing, independent peer/miner topology and named operators remain unverified. See operator launch register and security review. No startup or payment approval is granted by these tools.