Node Upgrade Rollout System

Last verified: 2026-02-13

​

Overview

• Chain-agnostic: Same 8 action primitives serve all chains. No chain-specific code in the upgrade engine.
• Runtime-agnostic: RuntimeAdapter interface abstracts systemd/Docker. Adding a runtime requires only a new adapter.
• Idempotent: Three-layer idempotency (completed store, checkpoints, system state checks) ensures safe re-entry after crashes.
• Compensating: Per-action Compensate() in reverse order restores previous state on failure.

​

Architecture

​

Manifest Loader

• Auto mode: Provide upgrade_id + chain_profile_id + node_type. The control plane loads the manifest from {ConfigDir()}/upgrades/{chain_profile_id}/{upgrade_id}.yaml and auto-populates binary_name, target_version, target_url, target_checksum, migration_type, recovery_plan, config_changes, and manifest_content_hash.
• Manual mode: Omit upgrade_id. Operator provides all fields explicitly (existing behavior).

# {ConfigDir}/upgrades/{chain_profile_id}/{upgrade_id}.yaml
upgrade_id: v1.5.0-hotfix
chain_profile_id: celestia-mocha
node_type: full
binary_name: celestia-node
target_version: v0.21.0
target_url: https://releases.celestia.org/v0.21.0/celestia-node-linux-amd64
target_checksum: sha256:abc123...
state_compatibility:
  migration_type: compatible
  notes: "State format unchanged"
config_changes:
  - path: /etc/celestia/config.toml
    template: config.toml.tmpl
    overrides:
      max_peers: "100"

​

Agent Three-Layer Architecture

​

Workflow Hierarchy

​

Agent Upgrade Handler

​

UpgradeAction Interface

type UpgradeAction interface {
    Name() string
    Execute(ctx context.Context, rt RuntimeAdapter, params ActionParams) error
    IsAlreadyDone(ctx context.Context, rt RuntimeAdapter, params ActionParams) (bool, error)
    Compensate(ctx context.Context, rt RuntimeAdapter, params ActionParams) error
    Timeout() time.Duration
}

​

RuntimeAdapter Interface

type RuntimeAdapter interface {
    runtime.Runtime  // Start, Stop, Restart, Status, ServiceName

    AcquireArtifact(ctx context.Context, url, checksum string) (ArtifactHandle, error)
    VerifyArtifact(ctx context.Context, handle ArtifactHandle, checksum string) error
    IsArtifactInstalled(ctx context.Context, name, checksum string) (bool, error)
    BackupState(ctx context.Context, backupDir string) error
    InstallArtifact(ctx context.Context, handle ArtifactHandle, target string) error
    WriteConfig(ctx context.Context, path string, content []byte) error
    IsConfigWritten(ctx context.Context, path string, expectedHash string) (bool, error)
    ReloadDaemon(ctx context.Context) error
    RestoreState(ctx context.Context, backupDir string) error
    IsArtifactAcquired(ctx context.Context, url, checksum string) (bool, error)
    IsBackedUp(ctx context.Context, backupDir string) (bool, error)
}

​

Action Primitives

​

Executor

1. Check IsAlreadyDone() via adapter (skip if true)
2. Execute with per-action timeout budget
3. On failure: call Compensate() on all completed actions in reverse order
4. Checkpoint completed actions for crash recovery

​

Phase Reporting

​

Three-Layer Idempotency

​

Runtime Adapters

​

Systemd Adapter

​

Docker Adapter

See also: Extending for how to add a new runtime adapter.

​

Data Model

​

Tables

​

Status Enums

​

NodeConfigState

See also: Domain Model for canonical node and subscription models.

​

Temporal Workflows

​

RolloutGroupWorkflow

Workflow ID: rollout-group-{group_id}
Search Attributes: GroupID, ChainProfileID

​

RolloutWorkflow

Workflow ID: rollout-{rollout_id}
Search Attributes: RolloutID, ChainProfileID, Strategy, GroupID

1. Activity: ResolveRolloutTargets — filter eligible nodes, assign batch indexes
2. Per batch: launch child UpgradeNodeWorkflow per node, wait for batch completion
3. Activity: UpdateRolloutProgress — update counters
4. Evaluate failure threshold (unless force_continue=true)
5. Canary: after batch 0, extended observation (2x health_wait), validate canary health
6. continue-as-new with remaining batches (event history resets per batch)
7. Final batch: Activity FinalizeRollout (set terminal status, forced DB-Temporal sync)

​

UpgradeNodeWorkflow

Workflow ID: upgrade-{rollout_id}-{node_id}
Search Attributes: NodeID, RolloutID, CorrelationID

​

Activity Profiles

See also: Workflows for existing Temporal patterns and compensation conventions.

​

Rollout Strategies

​

Rolling

• Upgrade batch_size nodes at a time (sequential batches, parallel within batch)
• After each batch: check failure threshold
• If failed / attempted > threshold -> auto-pause
• Operator decides: resume, retry, cancel, rollback

​

Canary

• Batch 0: canary_size random nodes
• Extended observation: 2x health_wait_duration after canary batch
• If canary healthy -> proceed with remaining nodes in rolling batches
• If canary fails -> auto-pause
• Breaking migrations (migration_type: breaking): canary batch is blocking — rollout cannot proceed until operator sends approve_canary signal

​

All-at-Once

• All nodes in single batch (no sequential batching)
• For hard forks with block height deadlines
• force_continue=true: complete all nodes regardless of failures
• SkipValidation=true: skip health wait for deadline pressure

​

Compensation and Rollback

​

Per-Action Compensation

1. Configs restored before daemon-reload
2. Daemon-reload before start
3. Service started last

​

Backup Directory

.upgrade-backup/
  {upgrade_id}/
    manifest.json          # Atomic marker (written last)
    binaries/
      celestia-node        # Previous binary
    configs/
      config.toml          # Previous config files
    container_config.json  # Docker: previous container inspect

​

Rollback Guarantee Tiers

1. Binary + config restored (Tier 1 always holds)
2. Node starts -> old binary encounters incompatible state -> crash loop
3. Health validation detects crash loop
4. Critical incident created with recovery_plan from manifest
5. Node set to degraded
6. Operator-initiated reprovisioning is the recovery path

​

Full Rollout Rollback

1. Pause rollout (stop new batches)
2. For each succeeded node: launch UpgradeNodeWorkflow with swapped versions (target = source)
3. Wait for all rollback workflows
4. Mark rollout rolled_back

​

Group Rollback (rollback_all Policy)

• Reverse-order rollback of ALL completed components
• Each component rolled back via RolloutWorkflow with swapped versions
• Group marked rolled_back

​

Health Validation

​

WaitForHealthValidation

1. At least one heartbeat received (agent alive)
2. application_health != critical
3. sync_status != stalled
4. Node state not down or failed
5. No crashloop (fewer than N restarts within health_wait_duration)

​

Maintenance State Exemption

• Heartbeat evaluator skips maintenance nodes (existing behavior)
• Policy evaluator skips maintenance nodes
• No false alarms during upgrade window

​

MaintenanceCleanup Loop

See also: Health and Incidents for the health evaluation pipeline.

​

Validation Rules

​

Creation-Time Rules

​

State Compatibility Validation

​

Config Rendering Pipeline

1. Load base template from hoodcloud-chain-configs/
2. Build TemplateVars from chain profile + node config (same variables as CONFIGURE)
3. Apply variable_overrides from upgrade manifest
4. Render via Go text/template -> full content
5. Include in UpgradePayload.config_files

​

API Surface

​

Rollout Endpoints

​

Rollout Group Endpoints

​

Integration Points

​

Incident Categories

See also: Health and Incidents — Incident Service for existing incident categories.

​

Notification Events

​

NATS Event Subjects

• Rollout-level: events.rollout.{rolloutID}.started, .progress, .completed
• Per-node: events.node.{nodeID}.upgrade_started, .upgrade_completed, .upgrade_phase_changed

​

Prometheus Metrics

​

Operational Guide

​

Creating a Rollout

1. POST /api/v1/rollouts with upgrade_id, chain_profile_id, node_type, source_version, strategy, and batch size
2. Control plane loads manifest, auto-populates binary details, state compatibility, and config changes
3. POST /api/v1/rollouts/{id}/start to begin execution

1. POST /api/v1/rollouts with chain_profile_id, node_type, binary_name, source_version, target_version, target_binary_url, target_binary_checksum, strategy, and batch size
2. POST /api/v1/rollouts/{id}/start to begin execution

1. POST /api/v1/rollout-groups with desired_versions, component_order (entries may include upgrade_id for auto-population), and failure_policy
2. POST /api/v1/rollout-groups/{id}/start

​

Monitoring Progress

• GET /api/v1/rollouts/{id} — overall progress (counters)
• GET /api/v1/rollouts/{id}/nodes — per-node status with upgrade_phase
• GET /api/v1/rollout-groups/{id}/status — per-component progress
• Temporal UI: workflow status, heartbeat details (action_index, action_name, progress_pct)

​

Pausing and Resuming

• POST /api/v1/rollouts/{id}/pause — pauses after current batch completes
• POST /api/v1/rollouts/{id}/resume — continues from paused state
• Auto-pause triggers: failure threshold exceeded, canary failure

​

Handling Failures

• Individual node failures: agent runs per-action compensation automatically
• Retry a failed node: POST /api/v1/rollouts/{id}/nodes/{nodeId}/retry
• FAILED_ROLLBACK nodes: manual intervention required (critical incident created)

​

Full Rollback

1. POST /api/v1/rollouts/{id}/rollback
2. System pauses the rollout, then launches reverse upgrades for all succeeded nodes
3. Uses the same UpgradeNodeWorkflow with swapped versions

​

Canary Approval for Breaking Migrations

1. Canary batch executes and completes
2. Rollout blocks — waiting for operator approval
3. Operator validates canary nodes manually
4. Operator sends approve_canary signal to resume
5. Remaining batches proceed

​

Chain-Agnostic Verification

1. Create chain config YAML with runtime: { type: systemd } and binary definition
2. Provision nodes
3. Create rollout with new version -> agent injects SystemdAdapter
4. Executor runs same 8 actions. No code changes.

1. Create chain config with runtime: { type: docker } and image digest
2. Provision nodes
3. Create rollout -> agent injects DockerAdapter
4. Executor runs same 8 actions. No code changes.

• New imperative runtime (e.g., LXC, Podman): implement RuntimeAdapter. Zero changes to actions, executor, workflows, or API.

​

Related Documents

• Workflows — Temporal workflow patterns, compensation conventions
• Health and Incidents — Health evaluation pipeline, incident management
• Domain Model — Node state machine, subscription lifecycle
• Extending — Adding runtime adapters, notification channels