Health and Incidents

​

Pipeline Overview

​

Leader Election

​

NodeHealthMachine

See also: Domain Model — Node State Machine for the canonical transition table.

​

Health Event Outbox

1. Atomic claim: UPDATE SET status='processing' WHERE id IN (SELECT ... FOR UPDATE SKIP LOCKED)
2. Dispatch to composite handlers:
   • CompositeTransitionHandler -> incident service, migration handler, state log handler
   • CompositeDimensionHandler -> incident service
3. Mark processed or failed
4. Cleanup: Remove old processed entries, recover stale processing entries from crashed workers

​

Two Evaluation Systems

​

Heartbeat Evaluator

1. ListSnapshots() — single 3-way JOIN: nodes + agent_registrations + node_health_state
2. EvaluateHeartbeat() per node — pure decision function, no I/O
3. ApplyHeartbeatDecisions() — batch state transitions with optimistic locking
4. BatchUpdateMutations() — persist failure counters to node_health_state
5. Trigger migrations for eligible nodes (deterministic workflow IDs)

• Transition — state change (if any)
• HealthMutation — bookkeeping (consecutive_failures, down_since, last_heartbeat_check)
• Migration — whether to trigger migration

1. No heartbeat for HeartbeatTimeout (60s) -> increment consecutive_failures
2. consecutive_failures >= ConsecutiveFailuresForDown (3) -> transition to DOWN
3. DOWN for MigrationGracePeriod (60s) -> trigger migration (if cooldown allows)
4. Heartbeat received while DOWN -> recover to HEALTHY/SYNCING, reset counter

​

Policy Evaluator

1. Query Victoria Metrics for active nodes (circuit breaker protected)
2. Evaluate compiled CEL programs per chain profile
3. First-match-wins verdict ordering: critical > degraded > ok > unknown
4. Write verdict to health machine (emits HealthDimensionEvent on change)

See also: Workflows — Metrics Flow for the full observation pipeline.

​

Incident Service

​

Event Categorization

​

Incident Lifecycle

1. Create/Upsert: Dedup by (node_id, category) partial unique index. New occurrence -> create. Existing open incident -> increment occurrence_count, update last_seen_at, merge metadata (JSONB concatenation).
2. Escalation: Warning incidents escalate to critical after:
   • app_degraded: occurrence_count >= threshold OR duration > escalation_duration
   • sync_stalled: duration > escalation_duration
3. Flapping detection: If occurrence_count >= threshold within time window, incident marked is_flapping (persisted to dedicated column, survives restarts).
4. Auto-resolution: When node recovers (DOWN -> HEALTHY/SYNCING, app_health -> OK, sync_status -> syncing/synced):
   • Increment resolution_debounce counter (per-incident, scoped to node_id + category)
   • Normal incidents resolve after debounce threshold
   • Flapping incidents require doubled threshold
   • Resolution resets debounce counter
5. User resolution: resolveUserID() fetches owner from subscription (nullable on failure — incidents still created without user_id).

​

Recovery Signals

​

Notification Pipeline

​

Dispatcher

1. Enqueue: Fire-and-forget to buffered channel (overflow drops with metric)
2. Batch collection: Wait for batch window (50ms default) or buffer full
3. Correlation: Group notifications by ChainProfileID. If count >= CorrelationMinNodes, send single correlated summary instead of individual notifications
4. Mass failure guard: If batch >= MassFailureThreshold, send single summary
5. Rate limiting: Apply sliding window limits before sending
6. Send: Fan out to all registered channels (Slack, Telegram, Email, Webhook)
7. Retry: Failed sends -> notification outbox with exponential backoff

​

Rate Limiter

​

Notification Channels

type Notifier interface {
    Send(ctx context.Context, notification Notification) error
    Name() string
}

See also: Extending — Adding a Notification Channel for how to add new channels.

​

Notification Outbox

​

Metrics

​

Rolling Uptime

​

Architecture

HealthTransitionEvent → CompositeTransitionHandler → StateLogHandler → node_state_log (append-only)
                                                                              │
                                                          UptimeWorker (every 5min) → node_uptime_hourly
                                                                                              │
                                                                                  API → SUM(buckets) → % uptime

​

StateLogHandler

1. Closes the previous state entry (exited_at = event.Timestamp)
2. Inserts a new state entry (exited_at = NULL, representing the current state)

​

UptimeWorker

1. Find the last complete bucket (or node.created_at if none exist)
2. Compute each hour from that point to the current hour
3. For each hour, classify state intervals as uptime, downtime, or unknown
4. Upsert bucket via INSERT ... ON CONFLICT DO UPDATE (idempotent)
5. Mark past hours as is_complete = true; current hour remains is_complete = false (recomputed each tick)
6. Purge buckets older than 90 days

​

UptimeService

• Summary: SUM(uptime_seconds) / SUM(total_seconds) over the requested window
• History: Group buckets by granularity (hourly or daily) for chart data

​

API Endpoints

• window: 24h, 7d, 30d (default: 30d)
• granularity: hourly, daily (default: hourly for 24h, daily otherwise)

​

Data Tables

See also: Domain Model — Database Schema for table definitions and indexes.

​

Cleanup Services

​

Cleanup Matrix

​

SubscriptionCleaner

​

BackupCleaner

• Called by SubscriptionCleaner on node termination
• Periodic CleanupOrphanedBackups() for already-terminated nodes
• Returns nil if S3 not configured; non-blocking on failures

​

TerraformCleaner

• Builds set of active host IDs (non-terminated + recently-terminated nodes)
• Only deletes state for hosts NOT in active set
• Age threshold prevents deletion of recently-stopped hosts

​

Auth CleanupJob

​

MaintenanceCleanup

​

Workflow Compensation Patterns

​

Resource Lifecycle

​

Post-Upgrade Health Validation

​

Related Documents

• Overview — Health evaluator service description
• Domain Model — Node state machine, incident model
• Workflows — Temporal workflow compensation patterns
• Extending — Adding health event handlers, notification channels