> ## Documentation Index > Fetch the complete documentation index at: https://docs.hoodcloud.io/llms.txt > Use this file to discover all available pages before exploring further. # Domain Model > Core domain objects, state machines, business rules, and database schema > **Last verified:** 2026-02-13 | Commit scope: `bc0fb41` ## Domain Objects ### Node **Location:** `internal/models/node.go` | Field | Type | Description | | ----------------------------- | ----------------- | ------------------------------------------------------------- | | `id` | UUID | Unique identifier | | `subscription_id` | UUID | Owning subscription | | `chain_profile_id` | string | Chain identifier (e.g., `celestia-mocha`) | | `node_type` | string | Node type (`full`, `archive`) | | `state` | NodeState | Infrastructure lifecycle state (owned by heartbeat evaluator) | | `sync_status` | SyncStatus | Chain sync status (`syncing`, `synced`) | | `application_health` | ApplicationHealth | Application health (owned by policy evaluator) | | `provider` | string | Cloud provider (internal) | | `instance_type` | string | VM instance type (internal) | | `host_id` | string | Provider host identifier (internal) | | `host_ip` | string | Host IP address (internal) | | `region` | string | Cloud region (internal) | | `last_heartbeat` | timestamp | Last agent heartbeat | | `last_migration_triggered_at` | timestamp | Migration cooldown tracking | **Field Ownership:** All state mutations flow through `NodeHealthMachine` (`internal/health/machine.go`), which enforces transitions, uses optimistic locking, and emits events via `health_event_outbox`. | Field | Writer | Machine Method | Purpose | | -------------------- | ------------------- | --------------------------- | --------------------------------- | | `state` | Heartbeat Evaluator | `ApplyHeartbeatDecisions()` | Infrastructure liveness | | `application_health` | Policy Evaluator | `UpdateApplicationHealth()` | Application health (CEL policies) | | `sync_status` | NATS sync events | `UpdateSyncStatus()` | Chain sync progress | **State values:** * `state`: `provisioning`, `syncing`, `healthy`, `degraded`, `down`, `maintenance`, `terminating`, `terminated`, `failed` * `application_health`: `unknown`, `ok`, `degraded`, `critical` **Display state:** `DisplayState()` computes a combined state for API responses — infrastructure states take precedence, then application\_health, then sync\_status. **Invariants:** * Node belongs to exactly one subscription * Infrastructure fields (`provider`, `host_id`, `host_ip`) are internal-only * Terminal states (`terminated`, `failed`) have no outgoing transitions * Terminal transitions auto-reset `sync_status` and `application_health` to `unknown` * Migration cooldown prevents rapid re-triggering (default: 1h) ### Node State Machine ```mermaid theme={null} stateDiagram-v2 [*] --> provisioning provisioning --> syncing provisioning --> terminating provisioning --> failed syncing --> healthy syncing --> degraded syncing --> down syncing --> terminating healthy --> degraded healthy --> down healthy --> maintenance healthy --> terminating degraded --> healthy degraded --> down degraded --> maintenance degraded --> terminating down --> syncing down --> healthy down --> maintenance down --> terminating maintenance --> syncing maintenance --> healthy maintenance --> terminating terminating --> terminated terminating --> failed terminated --> [*] failed --> [*] ``` **Transition table** (defined in `models.validTransitions`): | From | Valid targets | | ---------------------- | ------------------------------------------ | | `provisioning` | syncing, terminating, failed | | `syncing` | healthy, degraded, down, terminating | | `healthy` | degraded, down, maintenance, terminating | | `degraded` | healthy, down, maintenance, terminating | | `down` | syncing, healthy, maintenance, terminating | | `maintenance` | syncing, healthy, terminating | | `terminating` | terminated, failed | | `terminated`, `failed` | — (terminal) | **API:** `CanTransitionTo(target)`, `IsTerminal()`, `ValidSourceStates(target)`. Force updates via `ForceUpdateState()` bypass validation for compensation. > **See also:** [Health and Incidents](./health-and-incidents) for how transitions are triggered. ### Subscription **Location:** `internal/models/subscription.go` | Field | Type | Source | Description | | ------------------------- | -------------------- | --------------- | -------------------------------------- | | `id` | UUID | System | Unique identifier | | `user_id` | string | Auth context | Owner | | `chain_profile_id` | string | User input | Chain profile | | `node_type` | string | User input | Node type | | `provider` | string | System-resolved | Cloud provider (from duration mapping) | | `region` | string | System-resolved | Deployment region | | `instance_type` | string | System-resolved | Compute instance type | | `duration` | string | User input | `1w`, `2w`, `1m`, `3m`, `6m` | | `status` | SubscriptionStatus | System | Lifecycle state | | `expires_at` | timestamp | System | Subscription period end | | `grace_period_expires_at` | timestamp (nullable) | System | Renewal deadline after expiration | | `payment_id` | string (nullable) | System | Payment service reference | **User-provided:** `chain_profile_id`, `node_type`, `duration`. **System-resolved:** `provider`, `region`, `instance_type` (from chain profile + duration mapping). API responses hide system-resolved fields. ### Subscription State Machine ```mermaid theme={null} stateDiagram-v2 [*] --> pending_payment : POST /subscriptions pending_payment --> DELETED : 30m TTL expired pending_payment --> active : payment.completed (NATS) active --> expiring : expires_at < now (cleanup scheduler) expiring --> terminated : grace_period_expires_at < now (cleanup scheduler) note right of pending_payment : No nodes provisioned note right of expiring : 24h grace period for renewal note right of terminated : Nodes terminated, backups deleted ``` | Status | Meaning | | ----------------- | ------------------------------------------------------------ | | `pending_payment` | Created but unpaid. No nodes provisioned. PaymentID is null. | | `active` | Paid and running. Nodes provisioned and serving. | | `expiring` | Past `expires_at`, in 24h grace period. Nodes still running. | | `terminated` | Grace period ended. All nodes terminated, backups deleted. | **Invariants:** * Infrastructure fields resolved at creation time (not user-provided) * `pending_payment` subscriptions deleted after TTL (default 30m) * Expiration triggers 24h grace period before termination * Abandoned `pending_payment` subscriptions have `payment_id = null` > **See also:** [Workflows — Subscription Lifecycle](./workflows#subscription-lifecycle) for execution flow. ### Incident **Location:** `internal/models/incident.go` | Field | Type | Description | | --------------------- | -------------------- | ------------------------------------------------- | | `id` | UUID | Unique identifier | | `node_id` | UUID | Associated node | | `subscription_id` | UUID | Associated subscription | | `user_id` | UUID (nullable) | Resolved owner (notification routing) | | `chain_profile_id` | string | Chain identifier | | `category` | IncidentCategory | Incident type | | `severity` | IncidentSeverity | Current severity | | `status` | IncidentStatus | Lifecycle status | | `title` | string | Human-readable title | | `description` | string | Detailed description | | `metadata` | map | Trigger-specific context | | `occurrence_count` | int | Dedup counter | | `first_seen_at` | timestamp | First occurrence | | `last_seen_at` | timestamp | Most recent occurrence | | `acknowledged_at` | timestamp (nullable) | User acknowledgment | | `resolved_at` | timestamp (nullable) | Resolution time | | `is_flapping` | bool | Flapping flag (persisted, survives restarts) | | `resolution_debounce` | int | Per-incident debounce counter for auto-resolution | **Categories:** | Category | Trigger | Initial Severity | | ------------------ | ----------------------------------------- | ---------------- | | `node_down` | State transition to `down` | critical | | `app_critical` | `application_health` -> `critical` | critical | | `app_degraded` | `application_health` -> `degraded` | warning | | `sync_stalled` | `sync_status` -> `stalled` (CEL pipeline) | warning | | `provision_failed` | State -> `failed` from `provisioning` | critical | | `migration_failed` | State -> `failed` from `maintenance` | critical | **Incident Status Lifecycle:** ```mermaid theme={null} stateDiagram-v2 [*] --> open : incident created open --> acknowledged : user acknowledges open --> resolved : manual resolve open --> auto_resolved : healthy transition acknowledged --> resolved : manual resolve acknowledged --> auto_resolved : healthy transition ``` **Dedup:** Partial unique index on `(node_id, category) WHERE status NOT IN ('resolved', 'auto_resolved')`. Upsert increments `occurrence_count`, updates `last_seen_at`. **No foreign keys:** Incidents are historical records that survive node/subscription deletion. > **See also:** [Health and Incidents](./health-and-incidents#incident-service) for the full incident pipeline. ### User **Location:** `internal/models/user.go` | Field | Type | Description | | ------------------- | ----------------------- | ---------------------------------------------------- | | `id` | UUID | Unique identifier | | `wallet_address` | \*string (nullable) | Blockchain wallet address (checksummed for Ethereum) | | `email` | \*string (nullable) | Email address | | `wallet_public_key` | \[]byte | Public key bytes from signature | | `wallet_type` | \*WalletType (nullable) | `ethereum`, `solana` | | `external_auth_id` | \*string (nullable) | Clerk `user_xxxx` ID | **Identity combinations:** | Scenario | `wallet_address` | `wallet_type` | `external_auth_id` | | ---------------------------- | ---------------- | ------------- | ------------------ | | Clerk user (no wallet) | nil | nil | set | | Clerk user + Ethereum wallet | set | `ethereum` | set | | Clerk user + Solana wallet | set | `solana` | set | **Invariants:** Wallet address stored checksummed (EIP-55), lookups case-insensitive (lowercase index). Email optional and unique when provided. `external_auth_id` unique when not null. Created via Clerk webhook. ### API Key **Location:** `internal/models/apikey.go`, `internal/models/apikey_scope.go` | Field | Type | Description | | -------------- | -------------------- | ------------------------------------- | | `id` | UUID | Unique identifier | | `user_id` | UUID | Owning user | | `name` | string | Human-readable name (max 100 chars) | | `key_hash` | string | SHA-256 hash (plaintext never stored) | | `scopes` | string\[] | Granted permissions | | `expires_at` | timestamp (nullable) | Expiration | | `revoked_at` | timestamp (nullable) | Revocation | | `last_used_at` | timestamp (nullable) | Last validation | | `deleted_at` | timestamp (nullable) | Soft-delete | **Scopes:** `nodes:read`, `nodes:write`, `subscriptions:read`, `subscriptions:write`, `chains:read`, `keys:export`, `payments:read`, `payments:write`, `api-keys:manage`, `*` (wildcard) **Business rules:** At least one scope required. Max 50 active keys per user. Min 1h expiration. Soft-deleted for audit trail. Rotation is atomic (new key + old revoked in one transaction). Plaintext returned only on creation/rotation. ### Node Keys **Location:** `migrations/001_initial_schema.sql:88` **Encryption hierarchy:** ``` Master Key (Vault) --encrypts--> DEK (32-byte AES, stored encrypted) --encrypts--> Chain Key (mnemonic, stored encrypted) ``` **User-provided secrets flow:** ``` Client fetches X25519 public key (GET /api/v1/crypto/public-key) -> Client encrypts with NaCl sealed box (tweetnacl) -> Sealed ciphertext stored in provisioning_input_secrets -> PrepareUserProvidedKeys activity: decrypt with server X25519 private key (Vault) -> Re-encrypt with node DEK (AES-256-GCM) -> stored in node_keys ``` **Invariants:** Master key never leaves Vault. DEK in plaintext only in agent memory during operation. All keys deleted on subscription expiration. ### Auth Nonce **Location:** `internal/models/auth.go` — Single-use challenge for wallet verification. 64-char random, 5-min TTL. ### Refresh Token **Location:** `internal/models/auth.go` — SHA-256 hashed, 7-day TTL. Atomic rotation on refresh (old revoked, new created in one transaction). Multiple active tokens per user (different devices). ### Rollout Group **Location:** `internal/models/rollout.go` Coordinates multi-binary upgrades (e.g., ethereum-holesky: geth + lighthouse). Executes component rollouts in declared order. | Field | Type | Description | | ---------------------- | ------------------ | --------------------------------------------------------------------------------------------------- | | `id` | UUID | Unique identifier | | `chain_profile_id` | string | Target chain | | `status` | RolloutGroupStatus | Lifecycle state | | `failure_policy` | GroupFailurePolicy | `partial_ok`, `rollback_all`, `manual` (REQUIRED, no default) | | `component_order` | JSONB | Ordered array of `{binary_name, version, url, checksum, upgrade_id (optional for auto-population)}` | | `desired_versions` | JSONB | Target state: `{binary_name: version}` | | `strategy` | string | `rolling`, `canary`, `all_at_once` | | `batch_size` | int | Nodes per batch | | `canary_size` | int | Canary batch size | | `failure_threshold` | float | Auto-pause threshold (0.0–1.0) | | `health_wait_duration` | interval | Health gate duration between components | | `created_by` | string | Operator identifier | **Group status values:** `pending`, `running`, `paused`, `completed`, `partial`, `failed`, `cancelled`, `rolled_back` **Concurrency:** Partial unique index on `(chain_profile_id) WHERE status IN ('pending', 'running', 'paused')` — one active group per chain. Standalone single-component rollouts do not require a group. ### Rollout **Location:** `internal/models/rollout.go` Single-component upgrade rollout. Tracks progress, strategy, and per-node status. | Field | Type | Description | | ------------------------ | --------------- | ------------------------------------------ | | `id` | UUID | Unique identifier | | `group_id` | UUID (nullable) | Parent rollout group (null for standalone) | | `component_index` | int | Ordering within group | | `chain_profile_id` | string | Target chain | | `binary_name` | string | Binary to upgrade | | `source_version` | string | Current version | | `target_version` | string | Target version | | `target_binary_url` | string | Fully resolved artifact URL | | `target_binary_checksum` | string | SHA256 checksum (REQUIRED) | | `strategy` | RolloutStrategy | `rolling`, `canary`, `all_at_once` | | `status` | RolloutStatus | Lifecycle state | | `batch_size` | int | Nodes per batch | | `failure_threshold` | float | Auto-pause threshold | | `health_wait_duration` | interval | Post-batch health gate | | `total_nodes` | int | Denormalized progress counter | | `succeeded_nodes` | int | Denormalized progress counter | | `failed_nodes` | int | Denormalized progress counter | | `manifest_content_hash` | string | SHA256 of manifest at creation time | **Rollout status values:** `pending`, `scheduled`, `running`, `paused`, `completed`, `failed`, `cancelled`, `rolled_back` **Concurrency:** Partial unique index on `(chain_profile_id) WHERE status IN ('pending', 'scheduled', 'running', 'paused')`. #### Rollout Status State Machine ```mermaid theme={null} stateDiagram-v2 [*] --> pending : POST /rollouts pending --> scheduled : schedule (future) pending --> running : POST /rollouts/{id}/start scheduled --> running : scheduled time reached running --> paused : signal or auto-pause running --> completed : all nodes succeeded running --> failed : unrecoverable failure paused --> running : signal resume paused --> cancelled : signal cancel paused --> rolled_back : signal rollback running --> cancelled : signal cancel running --> rolled_back : rollback triggered ``` ### Rollout Node **Location:** `internal/models/rollout.go` Per-node upgrade tracking within a rollout. | Field | Type | Description | | --------------------- | ----------------- | --------------------------------------- | | `id` | UUID | Unique identifier | | `rollout_id` | UUID | Parent rollout | | `node_id` | UUID | Target node | | `batch_index` | int | Assigned batch | | `status` | NodeUpgradeStatus | Lifecycle state | | `upgrade_phase` | UpgradePhase | Current execution phase (display state) | | `previous_version` | string | Version before upgrade | | `previous_binary_url` | string | URL before upgrade (for rollback) | | `previous_checksum` | string | Checksum before upgrade | | `error_message` | string | Failure details | | `attempt_count` | int | Retry counter | **Node upgrade status values:** `pending`, `in_progress`, `validating`, `succeeded`, `failed`, `rolled_back`, `skipped` #### UpgradePhase Enum Display states derived from executor progress through the action sequence: | Phase | Meaning | | ----------------- | ------------------------------------------------ | | `preparing` | Backup + artifact acquisition + verification | | `stopped` | Service stopped | | `mutating` | Artifact/config replacement in progress | | `starting` | Service start issued | | `verifying` | Health validation in progress | | `completed` | Health gate passed | | `rollbacking` | Compensation in progress | | `rolled_back` | All compensations succeeded | | `failed_rollback` | Compensation failed — manual intervention needed | ### NodeConfigState **Location:** `internal/models/node_config_state.go` Tracks binary version per node. Pre-existing `node_config_state` table now has Go model and repository. | Field | Type | Description | | ---------------- | ----------------- | ---------------------------------- | | `node_id` | UUID | Node reference | | `binary_version` | string (nullable) | Currently installed binary version | | `config_version` | string (nullable) | Currently applied config version | Written by `UpdateNodeBinaryVersion` activity after successful upgrade, and by provisioning workflow after initial CONFIGURE. > **See also:** [Workflows — Upgrade Rollout Workflows](./workflows#upgrade-rollout-workflows) for the full workflow hierarchy. ## Business Rules ### Key Management 1. Keys exist in plaintext only temporarily for node operation 2. Encrypted recovery backups only within subscription TTL 3. No human-accessible key material (no SSH, keys only via encrypted gRPC) 4. Recovery without backup is impossible by design 5. User-provided secrets use client-side NaCl sealed box encryption ### Migration 1. **Gradual detection:** Consecutive failures tracked per node (default: 3 required for DOWN) 2. **Cooldown:** 1h between migration attempts (persisted in DB) 3. **Auto-recovery:** Heartbeat during DOWN grace period -> back to HEALTHY, counter reset 4. **Steps:** Verify backup -> provision new infra -> restore keys -> start node -> destroy old 5. **Non-blocking:** No user approval required | Parameter | Default | Purpose | | -------------------------- | ------- | ------------------------------------------ | | HeartbeatTimeout | 60s | Time without heartbeat to count as failure | | ConsecutiveFailuresForDown | 3 | Failures before DOWN state | | MigrationGracePeriod | 60s | Wait after DOWN before migration | | MigrationCooldown | 1h | Min time between migrations | ### Subscription Lifecycle 1. **Two-phase creation:** `POST /subscriptions` creates with `pending_payment`. Idempotent — returns existing if same user/chain/nodeType/duration. 2. **Pending cleanup:** Abandoned `pending_payment` deleted after TTL (default 30m) 3. **Expiration:** `active` -> `expiring` (24h grace period). Nodes continue running. 4. **Termination:** Grace period expired -> terminate all nodes, delete backups -> `terminated` ## Database Schema Summary ### Main App Database | Table | Purpose | | ---------------------------- | ---------------------------------------------------------------------- | | `users` | User identity (wallet, email, external\_auth\_id) | | `refresh_tokens` | JWT refresh tokens (hashed) | | `api_keys` | API authentication (scoped, soft-delete) | | `subscriptions` | Subscription lifecycle (status, duration, payment\_id, grace\_period) | | `nodes` | Node lifecycle (state, sync\_status, application\_health) | | `node_health_state` | Per-node health tracking (consecutive\_failures, down\_since, version) | | `health_event_outbox` | Health events pending dispatch | | `incidents` | Health incidents (no foreign keys, survives deletion) | | `notification_outbox` | Failed notifications queued for retry | | `agent_registrations` | Agent connectivity (last\_seen\_at, running\_command\_ids) | | `node_keys` | Encrypted key material (encrypted\_dek, encrypted\_chain\_key) | | `provisioning_inputs` | Provisioning input values (text/proof, subscription-scoped) | | `provisioning_input_secrets` | User-provided secrets (sealed box ciphertext) | | `node_tokens` | Internal authentication | | `node_config_state` | Per-node version tracking (binary\_version, config\_version) | | `rollout_groups` | Multi-component rollout coordination | | `rollouts` | Single-component upgrade rollouts | | `rollout_nodes` | Per-node upgrade tracking within a rollout | ### Payment Service Database Separate PostgreSQL instance. See [Payment Service](./payment-service#database-schema) for schema. | Data | Payment Service | Main App | | --------------- | --------------------- | --------------- | | User identity | Mirror (customer\_id) | Primary (users) | | Payment records | Primary | Reference only | | Subscriptions | Reference only | Primary | ### Key Indexes | Index | Table | Purpose | | ------------------------------------------------------------------- | --------------------- | ---------------------------- | | `wallet_address` (LOWER, unique) | `users` | Case-insensitive lookup | | `email` (unique where not null) | `users` | Email uniqueness | | `external_auth_id` (partial unique) | `users` | External auth mapping | | `token_hash` (unique) | `refresh_tokens` | Token validation | | `state` | `nodes` | Health evaluation queries | | `subscription_id` | `nodes` | Subscription cleanup | | `last_seen_at` | `agent_registrations` | Heartbeat timeout | | `expires_at` | `subscriptions` | Expiration cleanup | | `(node_id, category) WHERE status NOT IN (resolved, auto_resolved)` | `incidents` | Dedup upsert | | `payment_id WHERE status = 'pending_payment'` | `subscriptions` | Payment lookup | | `created_at WHERE status = 'pending_payment'` | `subscriptions` | TTL cleanup | | `(chain_profile_id) WHERE status IN active` | `rollout_groups` | One active group per chain | | `(chain_profile_id) WHERE status IN active` | `rollouts` | One active rollout per chain | | `(rollout_id, node_id) UNIQUE` | `rollout_nodes` | Dedup per rollout-node pair | | `(rollout_id, status)` | `rollout_nodes` | Batch status queries | ## State Storage Locations | State | Storage | Access Pattern | | -------------------- | ------------------------------------------------------------ | ----------------------------------------------------- | | Infrastructure state | PostgreSQL (`nodes.state`) | Written via NodeHealthMachine by heartbeat evaluator | | Application health | PostgreSQL (`nodes.application_health`) | Written via NodeHealthMachine by policy evaluator | | Health tracking | PostgreSQL (`node_health_state`) | Per-node: consecutive\_failures, down\_since, version | | Health events | PostgreSQL (`health_event_outbox`) | Emitted by health machine, consumed by outbox worker | | Incidents | PostgreSQL (`incidents`) | No foreign keys (survives deletion) | | Notification retry | PostgreSQL (`notification_outbox`) | Exponential backoff retry | | Encrypted keys | PostgreSQL (`node_keys`) | Written once, read for injection/migration | | Key backups | S3 | Created at provisioning, deleted at termination | | Terraform state | Files (`/app/terraform-state/{node_id}/`) | Per-node, managed by Terraform | | Pending commands | Redis queue | Pushed by orchestrator, consumed by agent | | Command progress | Redis (`progress:{commandID}`) | Written by gateway, read by orchestrator | | In-flight workflows | Temporal | Managed by Temporal server | | Metrics | Victoria Metrics TSDB | 30-day retention, queried by policy evaluator | | Rollout state | PostgreSQL (`rollouts`, `rollout_nodes`) | Written by workflow activities | | Version tracking | PostgreSQL (`node_config_state`) | Updated after successful upgrade/provision | | Upgrade backup | Node filesystem (`{data_dir}/.upgrade-backup/{upgrade_id}/`) | Per-upgrade, retained 24h | | Metrics transport | NATS JetStream | 1-hour retention | ## Related Documents * [Overview](./overview) — System overview and service descriptions * [Workflows](./workflows) — State transition triggers, execution flows * [Health and Incidents](./health-and-incidents) — Health evaluation pipeline * [Extending](./extending) — Adding wallet types