Monitoring should distinguish process health, consensus readiness, application progress, and committed transaction evidence.
Monitoring layers
| Layer | Evidence |
|---|---|
| Process | Daemon alive, logs, file descriptors, disk, and bounded shutdown. |
| Consensus | Role, node ID, peer connectivity, worker/primary progress, and fatal-error channel. |
| Application | Canonical application head, app hash, module metrics, and execution errors. |
| RPC | /status, complete block_header, transaction observation, body/batch rejection, and WebSocket health. |
| Resources | Per-validator CPU, RSS, heap, GC, goroutines, disk, network, and scrape cadence. |
Canonical cluster check
For a validator set, bind each RPC endpoint to its declared metrics endpoint by canonical node ID, not row position. Capture one common complete height from each endpoint, then compare block time, block hash, and app hash. Aggregate timestamps must not hide a missing validator sample.
Transaction evidence
For live performance or financial workflows, serialize the exact signed envelope, SHA-256 identity, admission result, submission time, committed height, execution code, gas wanted/used, and query cadence. A result snapshot is complete only after workers have stopped and the cleanup state is valid.
Alert on
- canonical header divergence;
- application head ahead of retained header for longer than the bounded reconciliation window;
- stopped or under-connected consensus;
- failed or stale metrics scrapes;
- resource identity mismatch between RPC and metrics;
- accepted transactions that remain unobserved;
- repeated RPC body/batch rejection or EVM filter-limit errors;
- shutdown cleanup timeouts.