WikiDeck/wikideck
3
0
Fork 1
Code Issues 8 Pull Requests Actions Packages Projects 2 Releases Wiki Activity

Remove markets

Browse Source
This commit is contained in:
eric
2026-10-28 18:33:52 -04:00
parent eea2deb1fb
commit b3a1179703
141 changed files with 554 additions and 17365 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/operations-recovery.md
+22 -70
View File
@@ -1,90 +1,42 @@
# Deployment, private monitoring and recovery — stage 4
# Deployment, private monitoring, and recovery
**Production remains NO-GO.** This stage provides tools and a local recovery rehearsal, not an installed public edge, configured alerts or independent operational sign-off. No mainnet approval, node launch, real payment, production backup/restore or existing data modification was performed. Tools are available on Node 24 and included in the runtime image where noted.
**Production remains NO-GO.** These procedures are operator preparation, not an installed public edge, configured alerts, independent review, or deployment approval. Do not remove persistent node volumes as part of source cleanup.
## Explicit reverse-proxy trust
## Reverse-proxy trust
`WIKIDECK_TRUSTED_PROXIES` is empty/untrusted by default. It accepts up to 16 comma-separated **literal IPs or CIDRs** (IPv4 prefix 8–32; IPv6 32–128). It rejects booleans, hop counts, names and universal /0 ranges. Set only the actual source address/subnet of the trusted edge connection. A large permitted subnet can still be unsafe: do not include unrelated tenants/containers. The candidate Compose file forwards the setting but leaves it empty.
`WIKIDECK_TRUSTED_PROXIES` is empty/untrusted by default. It accepts up to 16 comma-separated literal IPs or bounded CIDRs. Configure only the actual source address of a trusted edge connection. The edge must overwrite forwarded headers, and firewall rules must prevent direct backend access. A trusted-proxy setting does not configure the firewall.
The [nginx example](../deploy/nginx.conf.example) is **not installed or deployment-validated**. It illustrates TLS, request/body/connection/time limits and overwriting `X-Forwarded-For` with the actual remote client address rather than appending attacker input. Replace the reserved invalid hostname and certificate paths; review TLS/HSTS/ciphers, renewals, compression, static caching and per-IP/global admission for your edge. Validate nginx configuration on the intended version with real certificates before use. No actual TLS/certificate test was performed here.
**Topology matters:** nginx on the host forwarding to a container's published loopback port may appear inside the container as a Docker gateway, not `127.0.0.1`. Determine the source and use a reviewed dedicated network; do not blindly copy a loopback trust list. Firewall direct backend access. Anyone who can connect from a trusted source can forge forwarded IPs unless the edge overwrites headers and isolates the backend. Application trust settings do not configure the firewall. Existing global RPC budgets remain independent of forwarded client IPs.
The [nginx example](../deploy/nginx.conf.example) is not installed or deployment-validated. Replace its reserved hostname and certificate paths, and validate TLS, renewal, forwarded-IP handling, body/connection/time limits, and edge admission controls on the intended host. Container proxy source addresses may differ from host loopback; inspect the actual topology instead of copying a trust list blindly.
## Private diagnostics
Set `WIKIDECK_OPS_PORT` to an unused integer port (1–65535) to enable a **separate HTTP listener bound to `127.0.0.1` only**. Unset/empty disables it. Do not reuse the main RPC port. The default/candidate Compose definitions do not enable or publish this port. For container monitoring, execute the checker inside that container/network namespace; publishing a container port does not make a loopback listener safely accessible.
Set `WIKIDECK_OPS_PORT` to an unused port (1–65535) to enable a separate HTTP listener bound to `127.0.0.1`; leave it unset to disable it. Never publish or reverse-proxy this port. Only `GET /status` is supported. It returns bounded, no-store aggregate JSON; any local process with loopback access can read it.
Only `GET /status` is supported. Responses are no-store JSON capped at 64 KiB; connection count/timeouts are bounded. No password/token is required because access is confined to the local-process trust boundary. Therefore **all users/processes with loopback access can read it**; it is not suitable for an untrusted multi-user host without additional isolation. Never reverse proxy, tunnel broadly, add it to public routing, or publish it to the Internet. The listener has no administrative/mutation operations.
Reported diagnostics include node uptime/RSS, event-loop delay, canonical height and tip age, DAG and pending counts, native-engine status, configured/connected peers, RPC budgets, and available disk plus ledger database/WAL sizes. Native subprocess memory is not included in Node RSS. Peer count is not a security score. Health is liveness, not sync readiness or consensus assurance.
Reported aggregates:
`ops:check -- --port <private-port>` reads only literal loopback with bounded time and response size. It emits warnings for stale/empty chain, peer isolation, native-engine failure, queue pressure, event-loop delay, and unavailable/low disk. Thresholds are starting points, not production SLOs. Route alerts to an external system and test delivery; this repository does not install a scheduler or alert destination.
- Node uptime and Node-process RSS; **RSS excludes the RandomX subprocess**, so monitor container/cgroup/host totals externally.
- Event-loop mean/max since listener startup (not rolling percentiles).
- Canonical height, DAG block count, pending transactions, ledger mutation queue, process-local reorg count, tip timestamp/age. Tip age is miner-timestamp-based, not proof of local receipt freshness or synchronization.
- Native subprocess failure/active/queue signals. `available` means no recorded failure/exit; it is **not** a startup known-vector test or proof of successful current hashing.
- Configured/connected peer counts, API/P2P active/rejected counts. Peer count is connectivity, not independently administered hashpower or consensus security. Counters reset on restart.
- Market mutation queue, last successful reconciliation timestamp and counts grouped by order state. No addresses, order IDs, balances or signed transactions are included. Diagnostics still reveal operational activity and must remain private.
- Disk bytes available and ledger/market DB/WAL sizes, sampled asynchronously every 30 seconds. Missing sample/failure is explicit; absence of `disk` is not healthy. First startup may precede the first successful sample. Sampling does not impose a quota or prevent exhaustion.
## Stopped-node ledger backup
`ops:check -- --port <private-port>` reads only literal local loopback with a five-second deadline and 64 KiB streaming cap, emitting JSON plus a nonzero exit for warning conditions. No remote target or credentials are accepted. Built-in **starting-point** warning thresholds: observation clock/freshness >60s; no peers; empty/stale chain (>1h by tip timestamp); native unavailable; queue pressure (native/market >=48 or ledger >=192); mean event-loop delay >250ms; stale market reconciliation >60s; `review`/`refund-required` orders; disk sample >90s old/unavailable or <512 MiB free. A fresh empty isolated testnet correctly warns. These thresholds are not production SLOs and must be reviewed against real workload/chain timing.
`node:backup` supports `create`, `verify`, and `restore`. New version-2 packages contain only the selected network's ledger database. Stop the node cleanly first; the tool takes the same exclusive network lock as startup, uses SQLite `VACUUM INTO`, checks SQLite integrity and chain metadata, hashes the snapshot, and writes the manifest last. It refuses to overwrite a destination, restore in place, or run when a node lock exists. Restore targets must not exist.
Run the checker with your restricted local scheduler/monitoring agent and route warning codes to the **actual on-call system**. This repository does not create an alert account, install a scheduler or verify delivery. Externally monitor cumulative counter changes (reorgs/rejections), process/container totals, disk growth rate, peer/tip freshness, native failures and checker availability itself. A failed/unreachable operations listener must alert, not silently pass. Keep `/healthz` as liveness only; HTTP 200 is not launch/readiness evidence.
Example workflow, using the `node:backup` package command with explicit `--source`, `--destination`, and `--consent` as required by the CLI:
## Stopped-node pair backup
1. Stop the node and confirm all writers to its data directory are stopped.
2. Create the backup in a new directory outside the node data directory. Keep wallet backups separately; this tool does not include browser wallets, secrets, configuration, binaries, or logs.
3. Run `verify` against the backup before copying it off-host.
4. Restore only to a new isolated data directory. The tool reports `proofsRevalidated: false`; reopen with the trusted RandomX implementation and verify the chain before resuming service.
The `node:backup` script supports `create`, `verify`, `restore`. It deliberately does **not** offer online pair backup or overwrite/force/reset. Choose `WIKIDECK_NETWORK` explicitly to match the data; it validates the derived chain/network identity without enabling launch. The runtime includes the script and server helpers; it does not require dev dependencies.
Backups are not encrypted or authenticated by this tool. Store them in restricted, encrypted off-host storage, protect the manifest separately, and test cold restoration. Hashes detect accidental corruption, not a coherent malicious replacement. Filesystem, power-loss, Windows ACL, and network-share durability need validation on the actual storage platform.
### Create
## Retiring legacy financial data safely
1. Stop the node cleanly; verify all processes/writers using that data are stopped. Do not run `Ledger`/`Market` library writers directly while maintenance is active. Do not rely solely on a lock file for an arbitrary unmanaged process.
2. Invoke `node:backup` with action `create`, explicit `--source <node-data-directory>`, **new** `--destination <backup-directory>` and `--consent`. Source is the parent of the selected testnet/mainnet folder, not the individual SQLite file. Destination must be outside that data directory and must not exist. Its parent must already exist.
3. The tool acquires **the same exclusive network node lock** as normal startup. Any lock, even stale, refuses maintenance. The tool never deletes a preexisting lock. Inspect stale ownership/process status manually before separately authorized removal.
4. Validate SQLite quick checks and network/market metadata, then use SQLite `VACUUM INTO` for **both ledger and market** to produce standalone snapshot files (including committed source WAL contents, not a naive live-file copy). A symlinked network directory or database file is refused. Pair consistency relies on all writers staying stopped for the entire operation.
5. Database copies are checked, hashed with SHA-256 and flushed. A strict versioned manifest records names/sizes/hashes/network/profile/chain and creation time. The manifest completion marker is written/flushed **last**. POSIX directory entries are flushed; Windows/remote filesystem and storage-controller power-loss behavior requires real environment validation. Partial output is deliberately retained for investigation, not automatically trusted or deleted.
Removing market/custody source code does **not** delete or resolve data left by an older deployment. The card-only node does not open or reconcile those files. If the selected network directory contains `market.sqlite`, `custody-sandbox.sqlite`, or a `custody-postgres.json` binding, new ledger-only backup creation refuses to proceed rather than silently omit those records. Preserve the existing volume and any PostgreSQL database; arrange an operator-reviewed archive before retiring it.
Only the two databases and manifest are included: **not** browser wallets, backup passwords, deployment configuration, secrets, certificates, peer settings, native binaries or logs. Keep each category independently backed up with versioned recovery instructions. Backups contain sensitive marketplace participant/order information and are **not encrypted by this tool**. Store in restricted, encrypted, access-audited off-host storage with tested retention; do not put them in Git/build contexts. Unix modes are restrictive where supported; Windows ACLs and network-share permissions must be enforced by the operator.
Historical version-1 paired backups containing `ledger.sqlite` and `market.sqlite` remain verifiable and restorable as opaque files for preservation. Restoration does not reactivate market services or imply that financial obligations have been resolved. External PostgreSQL data is not contained in a SQLite package. Do not delete old volumes, journals, database credentials, or backups as part of this source cleanup.
### Verify
An operator must choose retention, recovery point/time objectives, off-host storage, and restore ownership. Never remove a lock or replace a running volume just to make recovery succeed.
Action `verify` needs `--source <backup-directory>` only. It reads a bounded strict manifest, requires exactly one ledger and one market filename, checks regular files/sizes/hashes and refuses unexpected SQLite WAL/SHM/journal sidecars, then checks SQLite and metadata. It does not launch or initialize a new node. Verification opens SQLite read-only, but OS access timestamps and filesystem-specific read behavior are not a forensic immutability guarantee.
## Remaining operational evidence
**Hashes detect corruption, not authenticity:** an attacker able to replace both manifest and files can recompute hashes. Sign/protect the backup package externally, keep trusted manifests off-host and restrict directory ownership. No wallet-control or signature proof is inferred from a checksum. This tool assumes trusted local filesystem administration; it does not defend against a privileged concurrent file-replacement attacker or hostile mounted storage.
### Restore to a new isolated directory
1. Action `restore` requires explicit `--source <verified-backup-directory>`, a **nonexistent** `--destination <new-data-directory>`, and `--consent`. Existing destinations are refused even when empty. There is no in-place reset or deletion.
2. Verify the package before creating output, copy only fixed database names exclusively, recheck hashes/SQLite metadata and flush. No stale lock, user secrets or deployment approval variables are restored. A partial failed restore is retained; do not try to start it or merge it into a working node.
3. The tool reports `proofsRevalidated: false`: database/hash verification is not chain verification. Reopen/replay with the trusted reference RandomX release in an **isolated reviewed rehearsal** before declaring it usable. Preserve market instance identity, orders, fees, payout markers and nonce receipts together. Compare the expected tip/work/ownership/order state against the trusted recovery point.
4. Reconcile later chain events, payment state and any external processors before any future resumption of valuable activity. Restoring stale databases does not roll back on-chain transfers or real payouts. Live payments remain disabled and have no reviewed disaster recovery process in this release.
An operator must choose recovery point/retention/RPO/RTO, practice off-host recovery and approve any production restore explicitly. Never remove a lock or replace a running volume just to make a recovery test pass.
## Coordinated PostgreSQL + node custody backup rehearsal
The prior file-only `node:backup` still refuses PostgreSQL-bound node directories. A separate explicit [paired backup helper](../compose.custody-backup.yaml) and [utility](../server/coordinated-backup.ts) now create a small-sandbox rehearsal package containing the SQLite ledger/market pair and a custom-format PostgreSQL database dump. Creation holds the exact node-directory lock, then a dedicated PostgreSQL session advisory writer lock, checks the local `custody-postgres.json` binding against the database custodian identity, verifies any retained SQLite cutover source is frozen and its digest matches PostgreSQL's `import_source`, audits PostgreSQL liabilities under the lock, snapshots the active node files and runs `pg_dump`. A frozen SQLite source is deliberately omitted from the active backup pair; PostgreSQL is authoritative after cutover. The pair manifest is written last and binds network/chain/profile/custodian ID, the node manifest digest, PostgreSQL database name and dump size/hash. No restore/reset operation is included.
The PostgreSQL dump uses a **separate read-only `custody_backup` login**, not the app runtime or schema owner. The local Compose provisioner now creates its credential volume, the one-shot bootstrap grants only database connect, schema usage, table SELECT and sequence SELECT, and the database HBA permits only TLS/SCRAM for this role. Existing local v1 secret volumes get an additive backup-role secret upgrade without rotating database credentials. This changes the next explicit persistent-stack provision/migrate cycle; no running stack was restarted here. The helper image has `pg_dump` and is a one-shot maintenance container; those tools and backup credentials are not added to the long-running app image. The disposable integration test checks that the role can dump but cannot update balances, delete entries or advance journal sequences. A cold test restore also replays the paired node SQLite database through the native RandomX verifier and checks tip/NFT ownership; the test-only fake-PoW fixtures are not used for that proof-revalidation assertion.
### Local sandbox invocation
This procedure is **not yet a production backup runbook**. Use only after reviewing the exact Compose project/volume names and destination permissions:
1. Stop only the app service and verify no other node/application process writes the same data directory. Leave the PostgreSQL service running. Do not stop/restart or remove any user deployment as part of development tests.
2. With the matching stopped-node directory, run the one-shot provision and migration services under an approved maintenance window so older v1 local secret volumes receive the additive read-only backup credential and database role. Verify existing passwords, CA and custodian ID were preserved; do not regenerate volumes.
3. Set `WIKIDECK_BACKUP_DIR` to an **existing, restricted** host directory on encrypted storage and `WIKIDECK_BACKUP_NAME` to a new leaf name. Run `docker compose -p wikideck-custody-local -f compose.custody-backup.yaml run --rm backup`. The helper requires an existing destination parent and refuses overwrite. The node lock blocks startup during the entire paired snapshot. A PostgreSQL session lock fences all compliant Wikideck custody writers; privileged/manual DB writers must also be operationally fenced.
4. Run the helper with `verify --source /backups/<name>` (overriding the service command) to check the strict pair manifest, exact node/root directory contents, hashes, recorded server/client versions, and that a compatible `pg_restore --list` parses the custom archive and includes required custody accounting tables. This is structural/content-presence validation, not a full restore or authenticity proof. Separately rehearse restoration into a new isolated PostgreSQL database; never point `pg_restore` at the authoritative database. Use matching or newer PostgreSQL client tools; this helper refuses a restore client older than the archived dump utility.
5. Store an encrypted off-host copy with separately protected authenticity metadata. This package is **not encrypted** by the utility, and colocating it with the source host does not satisfy disaster recovery.
Creation holds locks and runs a full financial audit; a large journal may exceed configured query/audit budgets and fail closed, so this needs measured long-history tests before production use. A database superuser/schema owner can bypass application advisory-lock discipline or rewrite records; external writer fencing remains mandatory. Hashes and archive parsing detect accidental damage, not a malicious coherent replacement; the manifest is unsigned. The `pg_dump` subprocess uses only a whitelisted connection environment, so inherited `PGHOSTADDR` or TLS overrides cannot redirect it away from the audited/writer-fenced endpoint. The PostgreSQL dump is not atomic with arbitrary unmanaged ledger or schema-owner activity. There is no WAL/PITR, remote scheduler, retention, restore orchestration, RPO/RTO claim, encryption, signature or production failover process in this increment. The local Compose backup service mounts the external host destination read-write into a container with a stopped-node source lock; review Docker volume access and the helper image before using it around any valuable data.
## Evidence and remaining hold points
Automated tests cover default proxy distrust/spoofed-header rejection, explicit loopback proxy recognition, invalid trust/port settings, private listener and threshold checks, paired backup/restore identity preservation, refusal to overwrite/ignore locks/back up within data, and rejection of tampering, wrong network, incomplete packages and unexpected sidecars.
The native HTTP integration now cleanly stops its disposable nodes, backs up their **real three-block RandomX chain and market journal**, restores to a fresh disposable directory, opens/revalidates copied proofs with the actual reference engine, and checks canonical ownership, market instance, order buyer and nonce receipt history. It then restarts the original isolated fixture. This is a local automated drill, not an off-host operational recovery or crash/power-loss certification.
**Coordinated custody recovery evidence (2026-10-20):** the latest build passed 172 offline regressions and 31 isolated PostgreSQL integration tests. The paired-backup test mines a valid native-RandomX source block, writes fictitious PostgreSQL custody state, freezes/binds the simulated SQLite cutover source, creates and verifies the pair, restores PostgreSQL into a scratch database, and restores/replays the node database into a new directory with the reference RandomX engine. It verifies backup-login restrictions, mismatched identity/cutover detection, stale node-lock refusal, failed `pg_dump` cleanup/writer-lock release, output overwrite refusal, incomplete package handling, exact directory contents, rejection of a valid unrelated empty-database archive and a corrupt dump even when its unsigned manifest hash is recomputed. A hostile inherited `PGHOSTADDR`/TLS environment also fails to redirect the dump child. Tests use tmpfs/internal trust-auth databases and synthetic local files; this does not demonstrate protected persistent-volume provisioning, host-backed destination ACLs, off-host encryption or production operational recovery. The persistent sandbox was not restarted or migrated for this work.
Final Node 24 TypeScript/Vite and production runtime builds passed. The suite passed **98 tests, 0 failed, 0 skipped**. A separate packaged-runtime smoke drill ran with no external network, read-only root, dropped capabilities and a disposable 64 MiB tmpfs: monitoring correctly returned nonzero for isolation, empty chain, not-yet-reconciled market and low disk; clean stopped-node backup, verification and new-directory restore each exited zero. That empty-chain CLI drill reported proof revalidation as false, as designed; nonempty real-proof replay is covered by the integration drill above. No host volume or production wallet was used.
SEC-07/SEC-09 are only partially addressed: actual edge/source-IP/TLS checks, a production monitoring agent and verified alert delivery, off-host backup storage, long-history restore timing, filesystem/power-loss testing, independent peer/miner topology and named operators remain unverified. See [operator launch register](operator-launch.md) and [security review](security-review.md). No startup or payment approval is granted by these tools.
Verify the intended HTTPS/proxy/firewall topology, peer and tip freshness, alert delivery, disk growth, backup encryption/retention, and restore/replay time on the actual host. Assign an incident responder and document stop/rollback criteria. See the [card-only production plan](card-only-production-plan.md), [operator launch register](operator-launch.md), and [security review](security-review.md). No startup or payment approval is granted by these tools.