Hardened systemd deployment contract for read-only Monero observer

2026-10-22 10:08:15 -04:00
parent 158981e965
commit e835238d05
12 changed files with 170 additions and 7 deletions
@@ -0,0 +1,50 @@
+# Monero observer service (single Linux host)
+
+This is a supervised **observation-only** process for a dedicated Linux host. It is not a public API, deposit-address service, credit worker, withdrawal signer, or production-launch approval. It requires an independently deployed PostgreSQL instance and a private TLS 1.3 mTLS Wallet-RPC gateway. Do not point it at public Wallet RPC or expose either dependency to the Internet.
+
+## Deployment prerequisites
+
+- Linux x86-64 with systemd, Node.js 24, a dedicated `wikideck-observer` user/group, and a reviewed immutable application release installed read-only at `/opt/wikideck/current`.
+- A production PostgreSQL database reachable only on a private network, with verified TLS, encryption at rest, restricted administrative access, encrypted backups and a tested restore. Do not reuse the local sandbox database or runtime credential.
+- A mainnet Monero full node and watch-only Wallet RPC behind the already-tested mTLS method-allowlisting gateway. Network policy/firewall must permit the observer host to reach only the intended PostgreSQL and gateway addresses/ports. The gateway must not expose Wallet RPC directly.
+- Protected wallet observer/mTLS JSON and credential files owned by `wikideck-observer`, mode `0600`. Keep the mTLS private key and PostgreSQL URL out of shell history, process arguments, environment values, source control and logs. Store host disks and backups encrypted; define and test certificate rotation before use.
+- The chosen wallet is independently verified as the intended mainnet watch-only wallet; its spending key is not present on this host. A config endpoint match is not proof that the remote service is the intended wallet.
+
+The example JSON files in this directory are templates only. Copy them to `/etc/wikideck/monero-observer/`, replace the gateway hostname, and install the matching client certificate/key and gateway CA there. The endpoint in both configs must be identical and the certificate chain must be checked for validity dates, key strength and SHA-2 signatures. The application's certificate loader requires owner-only POSIX file permissions.
+
+## One-time database preparation
+
+Use the dedicated schema-owner PostgreSQL credential file and CA file in a restricted operator session. The operation below is explicit, transactional, digest-checked and idempotent; it does **not** create the observer login, add a runtime grant, or enable deposits. Do not run it against a database with unrelated or valuable data before reviewing its schema and backup.
+
+```sh
+WIKIDECK_NETWORK=mainnet \
+WIKIDECK_CUSTODY_PG_URL_FILE=/secure/operator/schema-owner-url \
+WIKIDECK_CUSTODY_PG_CA_FILE=/secure/operator/postgres-ca.crt \
+WIKIDECK_MAINNET_SCHEMA_MIGRATION=explicit-schema-migration-only \
+node --experimental-strip-types scripts/custody-db.ts migrate-mainnet-schema
+```
+
+Have the database administrator create a separate `LOGIN` role with no administrative attributes and no role memberships, and place its TLS PostgreSQL URL in `postgres-url`. Then issue only the observer grants using the schema-owner credential:
+
+```sh
+WIKIDECK_NETWORK=mainnet \
+WIKIDECK_CUSTODY_PG_URL_FILE=/secure/operator/schema-owner-url \
+WIKIDECK_CUSTODY_PG_CA_FILE=/secure/operator/postgres-ca.crt \
+node --experimental-strip-types scripts/custody-db.ts grant-monero-observer-runtime wikideck_observer
+```
+
+The role name in the command must exactly match the separately provisioned login. The worker role cannot read journals or balances, allocate addresses, post credits/releases, or submit transfers. Restrict the schema-owner credential to this one-time operation and remove it from the host afterward. `custody-db` continues to refuse all other mainnet financial actions.
+
+## Install and operate
+
+Install the service unit from this directory at `/etc/systemd/system/wikideck-monero-observer.service`. Create `/etc/wikideck/monero-observer/` with owner `wikideck-observer` and mode `0700`. Install the two runtime JSON files and all referenced certificates/keys plus `postgres-url` and `postgres-ca.crt`, each owned by that user and mode `0600`. The PostgreSQL URL must use the observer role and must not contain TLS downgrade parameters. Ensure the `current` application release is root-owned and not writable by the service account.
+
+After reviewing the unit against the host's systemd version, reload systemd and start the unit. Confirm the process is running and that logs contain only bounded scan aggregates; logs intentionally omit wallet addresses, transaction IDs, amounts and credentials. Configure an alert outside this host for service exits, sustained scan lag, database/gateway unavailability, WAL/disk pressure and unexpected worker restarts. The worker itself does not deliver alerts.
+
+The service uses a PostgreSQL session advisory lock so a second instance cannot scan concurrently. SIGTERM/SIGINT stops its polling loop and releases the lock. A failed RPC/database batch is retried; a failed scan does not advance the durable cursor. Logs are available through the system journal under identifier `wikideck-monero-observer`.
+
+## Release boundary
+
+This deployment only persists read-only chain observations and scan progress. It does **not** issue customer deposit addresses, expose an address to a user, calculate customer credit, release pending liabilities, enable marketplace checkout, or move XMR. There is no public endpoint or UI connected to these rows. A scan is not proof of solvency or authorization to accept deposits.
+
+The repository still lacks a production Monero daemon/Wallet-RPC/gateway deployment, certificate and secret lifecycle automation, nonempty owned-output validation against the deployed topology, external monitoring/restore evidence, a reviewed customer deposit/withdrawal policy, and live settlement integration. The mainnet candidate is independently marked NO-GO and its current candidate genesis time is 2027-01-01; deploying this worker does not clear any launch gate. The service file is a starting deployment contract, not a certification or instruction to accept funds.
@@ -0,0 +1,6 @@
+{
+  "url": "https://monero-rpc.internal.example/json_rpc",
+  "clientCertificateFile": "/etc/wikideck/monero-observer/client.crt",
+  "clientPrivateKeyFile": "/etc/wikideck/monero-observer/client.key",
+  "serverCaFile": "/etc/wikideck/monero-observer/gateway-ca.crt"
+}
@@ -0,0 +1,4 @@
+{
+  "network": "mainnet",
+  "url": "https://monero-rpc.internal.example/json_rpc"
+}
@@ -0,0 +1,50 @@
+[Unit]
+Description=Wikideck read-only Monero mainnet observation worker
+Wants=network-online.target
+After=network-online.target
+StartLimitIntervalSec=300
+StartLimitBurst=5
+
+[Service]
+Type=simple
+User=wikideck-observer
+Group=wikideck-observer
+WorkingDirectory=/opt/wikideck/current
+Environment=WIKIDECK_MONERO_OBSERVER_WORKER=1
+Environment=WIKIDECK_MONERO_OBSERVER=1
+Environment=WIKIDECK_MONERO_WALLET_RPC_CONFIG_FILE=/etc/wikideck/monero-observer/wallet-rpc.json
+Environment=WIKIDECK_MONERO_MTLS_CONFIG_FILE=/etc/wikideck/monero-observer/mtls.json
+Environment=WIKIDECK_CUSTODY_PG_URL_FILE=/etc/wikideck/monero-observer/postgres-url
+Environment=WIKIDECK_CUSTODY_PG_CA_FILE=/etc/wikideck/monero-observer/postgres-ca.crt
+Environment=WIKIDECK_MONERO_OBSERVER_POLL_MS=30000
+ExecStart=/usr/bin/node --experimental-strip-types /opt/wikideck/current/scripts/monero-observer-worker.ts
+Restart=on-failure
+RestartSec=10s
+TimeoutStopSec=40s
+KillSignal=SIGTERM
+UMask=0077
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=wikideck-monero-observer
+
+NoNewPrivileges=true
+CapabilityBoundingSet=
+AmbientCapabilities=
+PrivateTmp=true
+PrivateDevices=true
+ProtectSystem=strict
+ProtectHome=true
+ProtectKernelTunables=true
+ProtectKernelModules=true
+ProtectKernelLogs=true
+ProtectControlGroups=true
+ProtectClock=true
+ProtectHostname=true
+RestrictSUIDSGID=true
+RestrictRealtime=true
+RestrictNamespaces=true
+LockPersonality=true
+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
+
+[Install]
+WantedBy=multi-user.target
@@ -34,6 +34,8 @@ Status: active engineering plan; **not launch or payment authorization**. Update

 **Separate Monero observation worker (2026-10-21):** added a standalone opt-in process, durable bounded catch-up (100,000-block pages, four pages per batch, 1,000-block overlap), PostgreSQL session-lock leadership, restart resumption, and redacted aggregate progress logs. It persists observations only: it has no journal, balance, credit, release, address-allocation, or transfer capability. It is not started by the API and is not included in the sandbox stack as a running service. A separate role grant is available; on mainnet the `custody-db` CLI now permits only this exact observer-role grant and continues to reject migrations, audits, coordinator grants, and all financial actions. The operator must separately create a non-admin, no-membership database login, provide its protected PostgreSQL URL and the protected mTLS client/observer configuration, and run the worker under a supervised POSIX service. This is not yet a complete deployable Monero topology: no mainnet Wallet RPC/gateway/network-policy service, secret provisioning, monitoring destination, key lifecycle, nonempty owned-output test, or deposit-credit automation is provided. Validation: production test-image build succeeded; offline suite **195/195** and disposable PostgreSQL suite **37/37** passed. Worker evidence is synthetic catch-up/restart/leader fencing and journal exclusion only; it does not authorize mainnet deposits or payments.

+**Single-host worker deployment increment (2026-10-22):** added a hardened systemd unit and protected configuration templates for the stated apartment-host model. Added a distinct mainnet-only, exact-acknowledgement PostgreSQL schema migration action; it runs the existing transactional, digest-checked migrations and grants no runtime permissions. PostgreSQL URL/CA files now require POSIX owner-only permissions. The worker remains non-root, private-network-only by host firewall contract, and separate from the API. This advances deployment preparation, not live service: no external host/DB/gateway was contacted, no credentials were provisioned, and no service was started. The private Monero daemon/Wallet-RPC/gateway, external monitoring and recovery, production database provisioning and independent review remain unprovided. Validation: image build passed, offline suite **196/196**, PostgreSQL suite **37/37**. See [the systemd deployment contract](../deploy/systemd/README.md).
+
 ## 1A. MVP release boundary — proposed safe first production-hosted release

 The apartment/single-operator constraint changes the rollout and availability claims, not the safety evidence required to hold customer funds or valuable assets. Recommended MVP: **a public, clearly labeled non-monetary beta on the existing testnet**, with real XMR completely disabled, no custodial balances/deposits/withdrawals, and marketplace checkout/delivery disabled for valuable assets. Keep mining, browsing and wallet use only if they are explicitly described as experimental and users accept that data/ownership may be lost. This would be a production-hosted beta, **not mainnet production**, a custody MVP, or an independently secured chain.
@@ -9,7 +9,8 @@ import { PostgresCustody } from '../server/market/postgres-custody.ts';
 import { IS_MAINNET } from '../shared/network.ts';
 import { resolveCustodyIncident } from '../server/market/custody-incident.ts';
 import { loadMoneroDepositReviewApproval, loadMoneroDepositReviewPolicy } from '../server/market/monero-deposit-review-config.ts';
-import { assertCustodyDbActionAllowedOnNetwork, parseCustodyDbArguments } from '../server/market/custody-db-args.ts';
+import { assertCustodyDbActionAllowedOnNetwork, assertMainnetSchemaMigrationAcknowledged,
+  parseCustodyDbArguments } from '../server/market/custody-db-args.ts';
 import { prepareDepositCreditReview, postReviewedDepositPendingCredit, prepareDepositCreditReleaseReview,
  releaseReviewedDepositCredit, reconcileDepositCreditReorgHolds, prepareDepositReorgResolution,
  resolveReleasedDepositReorgHold, type DepositCreditApproval, type DepositCreditReleaseApproval,
@@ -19,10 +20,11 @@ import { prepareDepositCreditReview, postReviewedDepositPendingCredit, prepareDe
 const commandArgs = process.argv.slice(2);
 const depositActions = ['deposit-prepare-credit', 'deposit-stage-credit', 'deposit-prepare-release', 'deposit-release',
  'deposit-reconcile-holds', 'deposit-prepare-resolve-hold', 'deposit-resolve-hold'];
-const custodyActions = ['migrate', 'grant-runtime', 'grant-monero-observer-runtime', 'grant-monero-credit-runtime',
+const custodyActions = ['migrate', 'migrate-mainnet-schema', 'grant-runtime', 'grant-monero-observer-runtime', 'grant-monero-credit-runtime',
  'import-sqlite', 'audit', 'resolve-integrity'];
 const { action, argument, extraArgument } = parseCustodyDbArguments(commandArgs);
 assertCustodyDbActionAllowedOnNetwork(action, IS_MAINNET);
+assertMainnetSchemaMigrationAcknowledged(action, process.env.WIKIDECK_MAINNET_SCHEMA_MIGRATION);
 if (!custodyActions.includes(action) && !depositActions.includes(action)) throw new Error('Unsupported custody operation');
 try {
  const config = await postgresConfig();
@@ -32,7 +34,12 @@ try {
  } else {
    const pool = new Pool(config);
    try {
-      if (action === 'migrate') { await migrateCustody(pool); console.log('Custody PostgreSQL schema verified/migrated. No funds activated.'); }
+      if (action === 'migrate' || action === 'migrate-mainnet-schema') {
+        await migrateCustody(pool);
+        console.log(action === 'migrate-mainnet-schema'
+          ? 'Mainnet custody schema verified/migrated; no runtime grants, deposits, or payments enabled.'
+          : 'Custody PostgreSQL schema verified/migrated. No funds activated.');
+      }
      if (action === 'grant-runtime') { await grantCustodyRuntime(pool, argument ?? ''); console.log('Runtime grants applied; no credentials printed.'); }
      if (action === 'grant-monero-observer-runtime') {
        await grantMoneroObserverRuntime(pool, argument ?? '');
@@ -1,5 +1,6 @@
 export const CUSTODY_DB_ACTION_ARGUMENT_COUNTS = {
  migrate: 0,
+  'migrate-mainnet-schema': 0,
  'grant-runtime': 1,
  'grant-monero-observer-runtime': 1,
  'grant-monero-credit-runtime': 1,
@@ -19,11 +20,20 @@ export type CustodyDbAction = keyof typeof CUSTODY_DB_ACTION_ARGUMENT_COUNTS;
 export type CustodyDbArguments = { action: CustodyDbAction; argument?: string; extraArgument?: string };

 export function assertCustodyDbActionAllowedOnNetwork(action: CustodyDbAction, isMainnet: boolean): void {
-  if (isMainnet && action !== 'grant-monero-observer-runtime') {
+  if (action === 'migrate-mainnet-schema' && !isMainnet) {
+    throw new Error('Mainnet schema migration action requires the mainnet profile');
+  }
+  if (isMainnet && action !== 'grant-monero-observer-runtime' && action !== 'migrate-mainnet-schema') {
    throw new Error('Only the separately scoped Monero observation-role grant is allowed on mainnet');
  }
 }

+export function assertMainnetSchemaMigrationAcknowledged(action: CustodyDbAction, acknowledgement: string | undefined): void {
+  if (action === 'migrate-mainnet-schema' && acknowledgement !== 'explicit-schema-migration-only') {
+    throw new Error('Mainnet schema migration requires explicit schema-migration acknowledgement');
+  }
+}
+
 export function parseCustodyDbArguments(args: readonly string[]): CustodyDbArguments {
  const [action, ...values] = args;
  if (!action || !Object.hasOwn(CUSTODY_DB_ACTION_ARGUMENT_COUNTS, action)) {
@@ -10,6 +10,7 @@ async function secretFile(path: string | undefined): Promise<string> {
  if (!path) throw new Error('Custody PostgreSQL secret file is required');
  const info = await lstat(path);
  if (!info.isFile() || info.isSymbolicLink() || info.size > 16384 || info.size === 0) throw new Error('Invalid PostgreSQL secret file');
+  if (process.platform !== 'win32' && (info.mode & 0o077) !== 0) throw new Error('PostgreSQL secret file permissions are too broad');
  return (await readFile(path, 'utf8')).trim();
 }
 /** No URL logging, no TLS options in a connection string overriding verified transport. */
@@ -1,6 +1,7 @@
 import test from 'node:test';
 import assert from 'node:assert/strict';
-import { assertCustodyDbActionAllowedOnNetwork, parseCustodyDbArguments } from '../server/market/custody-db-args.ts';
+import { assertCustodyDbActionAllowedOnNetwork, assertMainnetSchemaMigrationAcknowledged,
+  parseCustodyDbArguments } from '../server/market/custody-db-args.ts';

 test('custody database CLI parses two-argument deposit operations without shifting arguments', () => {
  assert.deepEqual(parseCustodyDbArguments(['deposit-prepare-credit', 'a'.repeat(64), `${'b'.repeat(64)}:${'c'.repeat(64)}`]), {
@@ -16,6 +17,12 @@ test('custody database CLI parses two-argument deposit operations without shifti

 test('mainnet custody CLI permits only the observation-role grant', () => {
  assert.doesNotThrow(() => assertCustodyDbActionAllowedOnNetwork('grant-monero-observer-runtime', true));
+  assert.doesNotThrow(() => assertCustodyDbActionAllowedOnNetwork('migrate-mainnet-schema', true));
+  assert.throws(() => assertCustodyDbActionAllowedOnNetwork('migrate-mainnet-schema', false), /requires the mainnet profile/);
+  assert.throws(() => assertMainnetSchemaMigrationAcknowledged('migrate-mainnet-schema', undefined), /acknowledgement/);
+  assert.throws(() => assertMainnetSchemaMigrationAcknowledged('migrate-mainnet-schema', 'yes'), /acknowledgement/);
+  assert.doesNotThrow(() => assertMainnetSchemaMigrationAcknowledged('migrate-mainnet-schema', 'explicit-schema-migration-only'));
+  assert.doesNotThrow(() => assertMainnetSchemaMigrationAcknowledged('grant-monero-observer-runtime', undefined));
  for (const action of ['migrate', 'grant-runtime', 'grant-monero-credit-runtime', 'audit', 'deposit-release'] as const) {
    assert.throws(() => assertCustodyDbActionAllowedOnNetwork(action, true), /only the separately scoped/i);
    assert.doesNotThrow(() => assertCustodyDbActionAllowedOnNetwork(action, false));
@@ -28,3 +28,24 @@ test('database HBA requires TLS+SCRAM over TCP and provisioning refuses partial
  assert.match(setup, /subjectAltName=DNS:database/);
  assert.match(setup, /-checkend/);
 });
+
+test('standalone Monero observer service is non-root, filesystem-hardened and has no public listener', async () => {
+  const unit = await read('deploy/systemd/wikideck-monero-observer.service');
+  const guide = await read('deploy/systemd/README.md');
+  const observer = JSON.parse(await read('deploy/systemd/wallet-rpc.example.json')) as { network: string; url: string };
+  const mtls = JSON.parse(await read('deploy/systemd/mtls.example.json')) as { url: string; clientPrivateKeyFile: string };
+  assert.match(unit, /User=wikideck-observer/);
+  assert.match(unit, /ProtectSystem=strict/);
+  assert.match(unit, /ProtectHome=true/);
+  assert.match(unit, /NoNewPrivileges=true/);
+  assert.match(unit, /CapabilityBoundingSet=/);
+  assert.match(unit, /RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX/);
+  assert.match(unit, /WIKIDECK_MONERO_OBSERVER_WORKER=1/);
+  assert.doesNotMatch(unit, /^\s*(?:ListenStream|ListenDatagram|SocketUser)=/m);
+  assert.equal(observer.network, 'mainnet');
+  assert.equal(observer.url, mtls.url);
+  assert.equal(mtls.clientPrivateKeyFile, '/etc/wikideck/monero-observer/client.key');
+  assert.match(guide, /migrate-mainnet-schema/);
+  assert.match(guide, /does \*\*not\*\* issue customer deposit addresses/);
+  assert.doesNotMatch(`${unit}\n${guide}`, /(?:password|api[_-]?key)\s*[:=]\s*[^\s<]+/i);
+});
@@ -1,6 +1,6 @@
 import test from 'node:test';
 import assert from 'node:assert/strict';
-import { mkdtemp, writeFile, rm } from 'node:fs/promises';
+import { mkdtemp, writeFile, chmod, rm } from 'node:fs/promises';
 import { join } from 'node:path';
 import { tmpdir } from 'node:os';
 import { postgresConfig } from '../server/market/postgres-config.ts';
@@ -18,6 +18,11 @@ test('PostgreSQL verified TLS remains the default and cannot be downgraded throu
  for (const mode of ['', 'off', 'disable', 'false', 'mesh', 'TLS']) {
    await assert.rejects(postgresConfig({ ...env, WIKIDECK_CUSTODY_PG_TRANSPORT: mode }), /configuration/);
  }
+  if (process.platform !== 'win32') {
+    await chmod(file, 0o644);
+    await assert.rejects(postgresConfig(env), /configuration/);
+    await chmod(file, 0o600);
+  }
  await writeFile(file, 'postgresql://fixture:fixture-only@database.cluster.example/custody?sslmode=disable');
  await assert.rejects(postgresConfig(env), /configuration/);
 });
@@ -772,7 +772,7 @@ test('configuration requires secret files, verified TLS, literal backend, and lo
  await assert.rejects(openCustodyStorage(dir, options, { WIKIDECK_CUSTODY_STORAGE: 'postgres',
    WIKIDECK_CUSTODY_CANONICAL_REHEARSAL: 'yes', WIKIDECK_PG_DISPOSABLE_TEST: '1' }), /exactly 0 or 1/);
  await assert.rejects(postgresConfig({}), /configuration/);
-  await writeFile(file, 'postgresql://user:secret@database.example/custody');
+  await writeFile(file, 'postgresql://user:secret@database.example/custody', { mode: 0o600 });
  const secure = await postgresConfig({ WIKIDECK_CUSTODY_PG_URL_FILE: file }); assert.equal((secure.ssl as { rejectUnauthorized: boolean }).rejectUnauthorized, true);
  await assert.rejects(postgresConfig({ WIKIDECK_CUSTODY_PG_URL_FILE: file, WIKIDECK_CUSTODY_PG_LOCAL_TEST: '1' }), /configuration/);
  await writeFile(file, 'postgresql://user:secret@127.0.0.1/custody?sslmode=disable');