wikideck/docs/operations-recovery.md

Deployment, private monitoring and recovery — stage 4

Production remains NO-GO. This stage provides tools and a local recovery rehearsal, not an installed public edge, configured alerts or independent operational sign-off. No mainnet approval, node launch, real payment, production backup/restore or existing data modification was performed. Tools are available on Node 24 and included in the runtime image where noted.

Explicit reverse-proxy trust

WIKIDECK_TRUSTED_PROXIES is empty/untrusted by default. It accepts up to 16 comma-separated literal IPs or CIDRs (IPv4 prefix 8–32; IPv6 32–128). It rejects booleans, hop counts, names and universal /0 ranges. Set only the actual source address/subnet of the trusted edge connection. A large permitted subnet can still be unsafe: do not include unrelated tenants/containers. The candidate Compose file forwards the setting but leaves it empty.

The nginx example is not installed or deployment-validated. It illustrates TLS, request/body/connection/time limits and overwriting X-Forwarded-For with the actual remote client address rather than appending attacker input. Replace the reserved invalid hostname and certificate paths; review TLS/HSTS/ciphers, renewals, compression, static caching and per-IP/global admission for your edge. Validate nginx configuration on the intended version with real certificates before use. No actual TLS/certificate test was performed here.

Topology matters: nginx on the host forwarding to a container's published loopback port may appear inside the container as a Docker gateway, not 127.0.0.1. Determine the source and use a reviewed dedicated network; do not blindly copy a loopback trust list. Firewall direct backend access. Anyone who can connect from a trusted source can forge forwarded IPs unless the edge overwrites headers and isolates the backend. Application trust settings do not configure the firewall. Existing global RPC budgets remain independent of forwarded client IPs.

Private diagnostics

Set WIKIDECK_OPS_PORT to an unused integer port (1–65535) to enable a separate HTTP listener bound to 127.0.0.1 only. Unset/empty disables it. Do not reuse the main RPC port. The default/candidate Compose definitions do not enable or publish this port. For container monitoring, execute the checker inside that container/network namespace; publishing a container port does not make a loopback listener safely accessible.

Only GET /status is supported. Responses are no-store JSON capped at 64 KiB; connection count/timeouts are bounded. No password/token is required because access is confined to the local-process trust boundary. Therefore all users/processes with loopback access can read it; it is not suitable for an untrusted multi-user host without additional isolation. Never reverse proxy, tunnel broadly, add it to public routing, or publish it to the Internet. The listener has no administrative/mutation operations.

Reported aggregates:

• Node uptime and Node-process RSS; RSS excludes the RandomX subprocess, so monitor container/cgroup/host totals externally.
• Event-loop mean/max since listener startup (not rolling percentiles).
• Canonical height, DAG block count, pending transactions, ledger mutation queue, process-local reorg count, tip timestamp/age. Tip age is miner-timestamp-based, not proof of local receipt freshness or synchronization.
• Native subprocess failure/active/queue signals. available means no recorded failure/exit; it is not a startup known-vector test or proof of successful current hashing.
• Configured/connected peer counts, API/P2P active/rejected counts. Peer count is connectivity, not independently administered hashpower or consensus security. Counters reset on restart.
• Market mutation queue, last successful reconciliation timestamp and counts grouped by order state. No addresses, order IDs, balances or signed transactions are included. Diagnostics still reveal operational activity and must remain private.
• Disk bytes available and ledger/market DB/WAL sizes, sampled asynchronously every 30 seconds. Missing sample/failure is explicit; absence of disk is not healthy. First startup may precede the first successful sample. Sampling does not impose a quota or prevent exhaustion.

ops:check -- --port <private-port> reads only literal local loopback with a five-second deadline and 64 KiB streaming cap, emitting JSON plus a nonzero exit for warning conditions. No remote target or credentials are accepted. Built-in starting-point warning thresholds: observation clock/freshness >60s; no peers; empty/stale chain (>1h by tip timestamp); native unavailable; queue pressure (native/market >=48 or ledger >=192); mean event-loop delay >250ms; stale market reconciliation >60s; review/refund-required orders; disk sample >90s old/unavailable or <512 MiB free. A fresh empty isolated testnet correctly warns. These thresholds are not production SLOs and must be reviewed against real workload/chain timing.

Run the checker with your restricted local scheduler/monitoring agent and route warning codes to the actual on-call system. This repository does not create an alert account, install a scheduler or verify delivery. Externally monitor cumulative counter changes (reorgs/rejections), process/container totals, disk growth rate, peer/tip freshness, native failures and checker availability itself. A failed/unreachable operations listener must alert, not silently pass. Keep /healthz as liveness only; HTTP 200 is not launch/readiness evidence.

Stopped-node pair backup

The node:backup script supports create, verify, restore. It deliberately does not offer online pair backup or overwrite/force/reset. Choose WIKIDECK_NETWORK explicitly to match the data; it validates the derived chain/network identity without enabling launch. The runtime includes the script and server helpers; it does not require dev dependencies.

Create

1. Stop the node cleanly; verify all processes/writers using that data are stopped. Do not run Ledger/Market library writers directly while maintenance is active. Do not rely solely on a lock file for an arbitrary unmanaged process.
2. Invoke node:backup with action create, explicit --source <node-data-directory>, new --destination <backup-directory> and --consent. Source is the parent of the selected testnet/mainnet folder, not the individual SQLite file. Destination must be outside that data directory and must not exist. Its parent must already exist.
3. The tool acquires the same exclusive network node lock as normal startup. Any lock, even stale, refuses maintenance. The tool never deletes a preexisting lock. Inspect stale ownership/process status manually before separately authorized removal.
4. Validate SQLite quick checks and network/market metadata, then use SQLite VACUUM INTO for both ledger and market to produce standalone snapshot files (including committed source WAL contents, not a naive live-file copy). A symlinked network directory or database file is refused. Pair consistency relies on all writers staying stopped for the entire operation.
5. Database copies are checked, hashed with SHA-256 and flushed. A strict versioned manifest records names/sizes/hashes/network/profile/chain and creation time. The manifest completion marker is written/flushed last. POSIX directory entries are flushed; Windows/remote filesystem and storage-controller power-loss behavior requires real environment validation. Partial output is deliberately retained for investigation, not automatically trusted or deleted.

Only the two databases and manifest are included: not browser wallets, backup passwords, deployment configuration, secrets, certificates, peer settings, native binaries or logs. Keep each category independently backed up with versioned recovery instructions. Backups contain sensitive marketplace participant/order information and are not encrypted by this tool. Store in restricted, encrypted, access-audited off-host storage with tested retention; do not put them in Git/build contexts. Unix modes are restrictive where supported; Windows ACLs and network-share permissions must be enforced by the operator.

Verify

Action verify needs --source <backup-directory> only. It reads a bounded strict manifest, requires exactly one ledger and one market filename, checks regular files/sizes/hashes and refuses unexpected SQLite WAL/SHM/journal sidecars, then checks SQLite and metadata. It does not launch or initialize a new node. Verification opens SQLite read-only, but OS access timestamps and filesystem-specific read behavior are not a forensic immutability guarantee.

Hashes detect corruption, not authenticity: an attacker able to replace both manifest and files can recompute hashes. Sign/protect the backup package externally, keep trusted manifests off-host and restrict directory ownership. No wallet-control or signature proof is inferred from a checksum. This tool assumes trusted local filesystem administration; it does not defend against a privileged concurrent file-replacement attacker or hostile mounted storage.

Restore to a new isolated directory

1. Action restore requires explicit --source <verified-backup-directory>, a nonexistent --destination <new-data-directory>, and --consent. Existing destinations are refused even when empty. There is no in-place reset or deletion.
2. Verify the package before creating output, copy only fixed database names exclusively, recheck hashes/SQLite metadata and flush. No stale lock, user secrets or deployment approval variables are restored. A partial failed restore is retained; do not try to start it or merge it into a working node.
3. The tool reports proofsRevalidated: false: database/hash verification is not chain verification. Reopen/replay with the trusted reference RandomX release in an isolated reviewed rehearsal before declaring it usable. Preserve market instance identity, orders, fees, payout markers and nonce receipts together. Compare the expected tip/work/ownership/order state against the trusted recovery point.
4. Reconcile later chain events, payment state and any external processors before any future resumption of valuable activity. Restoring stale databases does not roll back on-chain transfers or real payouts. Live payments remain disabled and have no reviewed disaster recovery process in this release.

An operator must choose recovery point/retention/RPO/RTO, practice off-host recovery and approve any production restore explicitly. Never remove a lock or replace a running volume just to make a recovery test pass.

Coordinated PostgreSQL + node custody backup rehearsal

The prior file-only node:backup still refuses PostgreSQL-bound node directories. A separate explicit paired backup helper and utility now create a small-sandbox rehearsal package containing the SQLite ledger/market pair and a custom-format PostgreSQL database dump. Creation holds the exact node-directory lock, then a dedicated PostgreSQL session advisory writer lock, checks the local custody-postgres.json binding against the database custodian identity, verifies any retained SQLite cutover source is frozen and its digest matches PostgreSQL's import_source, audits PostgreSQL liabilities under the lock, snapshots the active node files and runs pg_dump. A frozen SQLite source is deliberately omitted from the active backup pair; PostgreSQL is authoritative after cutover. The pair manifest is written last and binds network/chain/profile/custodian ID, the node manifest digest, PostgreSQL database name and dump size/hash. No restore/reset operation is included.

The PostgreSQL dump uses a separate read-only custody_backup login, not the app runtime or schema owner. The local Compose provisioner now creates its credential volume, the one-shot bootstrap grants only database connect, schema usage, table SELECT and sequence SELECT, and the database HBA permits only TLS/SCRAM for this role. Existing local v1 secret volumes get an additive backup-role secret upgrade without rotating database credentials. This changes the next explicit persistent-stack provision/migrate cycle; no running stack was restarted here. The helper image has pg_dump and is a one-shot maintenance container; those tools and backup credentials are not added to the long-running app image. The disposable integration test checks that the role can dump but cannot update balances, delete entries or advance journal sequences. A cold test restore also replays the paired node SQLite database through the native RandomX verifier and checks tip/NFT ownership; the test-only fake-PoW fixtures are not used for that proof-revalidation assertion.

Local sandbox invocation

This procedure is not yet a production backup runbook. Use only after reviewing the exact Compose project/volume names and destination permissions:

1. Stop only the app service and verify no other node/application process writes the same data directory. Leave the PostgreSQL service running. Do not stop/restart or remove any user deployment as part of development tests.
2. With the matching stopped-node directory, run the one-shot provision and migration services under an approved maintenance window so older v1 local secret volumes receive the additive read-only backup credential and database role. Verify existing passwords, CA and custodian ID were preserved; do not regenerate volumes.
3. Set WIKIDECK_BACKUP_DIR to an existing, restricted host directory on encrypted storage and WIKIDECK_BACKUP_NAME to a new leaf name. Run docker compose -p wikideck-custody-local -f compose.custody-backup.yaml run --rm backup. The helper requires an existing destination parent and refuses overwrite. The node lock blocks startup during the entire paired snapshot. A PostgreSQL session lock fences all compliant Wikideck custody writers; privileged/manual DB writers must also be operationally fenced.
4. Run the helper with verify --source /backups/<name> (overriding the service command) to check the strict pair manifest, exact node/root directory contents, hashes, recorded server/client versions, and that a compatible pg_restore --list parses the custom archive and includes required custody accounting tables. This is structural/content-presence validation, not a full restore or authenticity proof. Separately rehearse restoration into a new isolated PostgreSQL database; never point pg_restore at the authoritative database. Use matching or newer PostgreSQL client tools; this helper refuses a restore client older than the archived dump utility.
5. Store an encrypted off-host copy with separately protected authenticity metadata. This package is not encrypted by the utility, and colocating it with the source host does not satisfy disaster recovery.

Creation holds locks and runs a full financial audit; a large journal may exceed configured query/audit budgets and fail closed, so this needs measured long-history tests before production use. A database superuser/schema owner can bypass application advisory-lock discipline or rewrite records; external writer fencing remains mandatory. Hashes and archive parsing detect accidental damage, not a malicious coherent replacement; the manifest is unsigned. The pg_dump subprocess uses only a whitelisted connection environment, so inherited PGHOSTADDR or TLS overrides cannot redirect it away from the audited/writer-fenced endpoint. The PostgreSQL dump is not atomic with arbitrary unmanaged ledger or schema-owner activity. There is no WAL/PITR, remote scheduler, retention, restore orchestration, RPO/RTO claim, encryption, signature or production failover process in this increment. The local Compose backup service mounts the external host destination read-write into a container with a stopped-node source lock; review Docker volume access and the helper image before using it around any valuable data.

Evidence and remaining hold points

Automated tests cover default proxy distrust/spoofed-header rejection, explicit loopback proxy recognition, invalid trust/port settings, private listener and threshold checks, paired backup/restore identity preservation, refusal to overwrite/ignore locks/back up within data, and rejection of tampering, wrong network, incomplete packages and unexpected sidecars.

The native HTTP integration now cleanly stops its disposable nodes, backs up their real three-block RandomX chain and market journal, restores to a fresh disposable directory, opens/revalidates copied proofs with the actual reference engine, and checks canonical ownership, market instance, order buyer and nonce receipt history. It then restarts the original isolated fixture. This is a local automated drill, not an off-host operational recovery or crash/power-loss certification.

Coordinated custody recovery evidence (2026-10-20): the latest build passed 172 offline regressions and 31 isolated PostgreSQL integration tests. The paired-backup test mines a valid native-RandomX source block, writes fictitious PostgreSQL custody state, freezes/binds the simulated SQLite cutover source, creates and verifies the pair, restores PostgreSQL into a scratch database, and restores/replays the node database into a new directory with the reference RandomX engine. It verifies backup-login restrictions, mismatched identity/cutover detection, stale node-lock refusal, failed pg_dump cleanup/writer-lock release, output overwrite refusal, incomplete package handling, exact directory contents, rejection of a valid unrelated empty-database archive and a corrupt dump even when its unsigned manifest hash is recomputed. A hostile inherited PGHOSTADDR/TLS environment also fails to redirect the dump child. Tests use tmpfs/internal trust-auth databases and synthetic local files; this does not demonstrate protected persistent-volume provisioning, host-backed destination ACLs, off-host encryption or production operational recovery. The persistent sandbox was not restarted or migrated for this work.

Final Node 24 TypeScript/Vite and production runtime builds passed. The suite passed 98 tests, 0 failed, 0 skipped. A separate packaged-runtime smoke drill ran with no external network, read-only root, dropped capabilities and a disposable 64 MiB tmpfs: monitoring correctly returned nonzero for isolation, empty chain, not-yet-reconciled market and low disk; clean stopped-node backup, verification and new-directory restore each exited zero. That empty-chain CLI drill reported proof revalidation as false, as designed; nonempty real-proof replay is covered by the integration drill above. No host volume or production wallet was used.

SEC-07/SEC-09 are only partially addressed: actual edge/source-IP/TLS checks, a production monitoring agent and verified alert delivery, off-host backup storage, long-history restore timing, filesystem/power-loss testing, independent peer/miner topology and named operators remain unverified. See operator launch register and security review. No startup or payment approval is granted by these tools.