> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hoodcloud.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Architecture Overview

> System overview, technology stack, and service descriptions for HoodCloud v1

## What is HoodCloud?

HoodCloud is a fully managed blockchain node infrastructure service implemented in Go. It provisions, monitors, and maintains non-validator blockchain nodes (full nodes, archive nodes, indexers) without requiring users to manage infrastructure, software upgrades, or monitoring.

**Primary Responsibilities:**

* Provision blockchain nodes on cloud infrastructure (Hetzner, OVH, extensible via module registry)
* Generate and securely manage cryptographic key material
* Monitor node health and automatically trigger migrations on failure
* Apply configuration changes and software updates via declarative recipes
* Apply targeted binary and config upgrades via rollout orchestration
* Terminate nodes and clean up resources when subscriptions expire

**What This Is NOT:**

* Not a validator service
* Not a public RPC endpoint provider
* Not a self-service platform (admin-controlled provisioning in v1)
* Not highly available by default (single-host nodes in v1)

## Application Type

HoodCloud is a **distributed control plane system** consisting of six services:

| Service          | Transport         | Port           | Purpose                                                                     |
| ---------------- | ----------------- | -------------- | --------------------------------------------------------------------------- |
| API Server       | HTTP              | 8080           | Business logic — nodes, subscriptions, chains, payments, key export         |
| Auth Server      | HTTP              | 8081           | Identity, authentication, wallet registration, API keys                     |
| Agent Gateway    | gRPC              | 9090           | Ops-agent communication (heartbeat, commands, events)                       |
| Orchestrator     | Temporal worker   | —              | Durable workflow execution (provision, migrate, terminate, upgrade rollout) |
| Health Evaluator | Background daemon | 9090 (metrics) | Health evaluation, incidents, notifications, cleanup (leader + standby)     |
| Migration Runner | One-shot CLI      | —              | Database schema migrations (`cmd/migrate`), runs as pre-deploy step         |
| Ops Agent        | gRPC client       | —              | On-host agent (one per node VM)                                             |

A separate **Payment Service** handles payment processing as an isolated microservice. See [Payment Service Architecture](./payment-service).

## Technology Stack

| Component               | Technology                                                                                               |
| ----------------------- | -------------------------------------------------------------------------------------------------------- |
| Language                | Go 1.25                                                                                                  |
| Workflow Engine         | Temporal (durable workflows)                                                                             |
| Database                | PostgreSQL (node state, subscriptions, keys, users; per-service connection pools and statement timeouts) |
| Payment Database        | PostgreSQL (separate instance, payment records)                                                          |
| Cache/Queue             | Redis (command queue, progress storage, idempotency)                                                     |
| Event Streaming         | NATS JetStream (agent events, metrics transport, payment events; 3-node cluster, R=3 in production)      |
| RPC Protocol            | gRPC with Protocol Buffers                                                                               |
| Service-to-Service Auth | mTLS (TLS 1.3, mutual certificate verification)                                                          |
| Infrastructure          | Terraform (declarative provisioning via module registry)                                                 |
| Encryption              | AES-256-GCM (node key material), NaCl sealed box (user-provided secrets)                                 |
| Authentication          | Clerk (external auth provider), JWT (RS256)                                                              |
| Node Hosting            | Hetzner, OVH (VPS, Public Cloud, Dedicated) — extensible                                                 |
| Cloud Services          | AWS (S3), Secrets (HashiCorp Vault), Vault AWS Credential Provider                                       |
| Metrics Storage         | Victoria Metrics TSDB                                                                                    |
| Policy Engine           | CEL (Common Expression Language)                                                                         |
| Distributed Tracing     | OpenTelemetry SDK -> OTel Collector -> Grafana Tempo                                                     |
| Status Page             | Gatus (health monitoring dashboard)                                                                      |

## High-Level System Architecture

```mermaid theme={null}
graph TD
    subgraph ControlPlane["Control Plane"]
        API["API Server<br/>(HTTP:8080)"]
        Auth["Auth Server<br/>(HTTP:8081)"]
        Gateway["Agent Gateway<br/>(gRPC:9090)"]
        Temporal["Orchestrator<br/>(Temporal)"]
        Health["Health Evaluator<br/>(Health Machine,<br/>Incidents,<br/>Notifications)"]
        Consumer["NATS<br/>Consumer"]

        API -.JWT validation.-> Auth
        Consumer -."payment.completed events".-> Temporal

        DB[(PostgreSQL<br/>Node State, Keys,<br/>Subscriptions, Users, Auth)]
        Redis[(Redis<br/>Command Queue,<br/>Progress Storage,<br/>Idempotency)]
        NATS[(NATS JetStream<br/>Agent Events,<br/>Metrics,<br/>Payment Events)]

        API --> DB
        Gateway --> DB
        Temporal --> DB
        Health --> DB
        Consumer --> DB

        API --> Redis
        Gateway --> Redis
        Temporal --> Redis

        Gateway --> NATS
        Health --> NATS
        Consumer --> NATS
    end

    subgraph PaymentService["Payment Service (Isolated)"]
        PayGRPC["gRPC Server<br/>:50051"]
        PayNATS["NATS Publisher"]
        PayDB[(PostgreSQL<br/>Payment Records)]

        PayGRPC --> PayDB
        PayNATS --> PayDB
    end

    subgraph NodeHosts["Node Hosts"]
        OpsAgent["Ops Agent (gRPC Client)<br/>- Heartbeat every 15s<br/>- Sync status tracking<br/>- Metric collection<br/>- Command execution"]
        NodeRuntime["Node Runtime (systemd)<br/>- celestia-bridge, geth, etc."]
    end

    Gateway -."gRPC (mTLS)".-> OpsAgent
    PayGRPC -."gRPC (mTLS)".-> ControlPlane
    PayNATS -.payment events.-> NATS
    OpsAgent --> NodeRuntime
```

## Provisioning Workflow Sequence

```mermaid theme={null}
sequenceDiagram
    participant API as API Server
    participant Temporal
    participant Orchestrator
    participant Infra as Infrastructure
    participant Agent as Ops Agent

    API->>Temporal: CreateNode()
    Temporal->>Orchestrator: Assign Task
    Orchestrator->>Orchestrator: GenerateKeys
    Orchestrator->>Infra: RunTerraform
    Infra->>Infra: Provision VM
    Infra-->>Orchestrator: VM Created
    Infra->>Agent: Boot VM
    Orchestrator->>Orchestrator: WaitForAgent
    Agent-->>Orchestrator: Register()
    Orchestrator->>Agent: SendConfigure
    Agent-->>Orchestrator: Apply Config
    Orchestrator->>Agent: InjectKeys
    Agent-->>Orchestrator: Write Keys
    Orchestrator->>Agent: StartNode
    Agent-->>Orchestrator: systemctl start
    Orchestrator-->>Temporal: Complete
    Temporal-->>API: Complete
```

## Security Boundaries

Keys exist in plaintext only temporarily for node operation. No permanent key retention — keys and backups are deleted on subscription expiration. No SSH access to nodes; all management via ops-agent. User-provided secrets are client-side encrypted (NaCl sealed box).

> **See also:** [CLAUDE.md — Security Boundaries](../../CLAUDE.md) for the canonical security rules.

## Service Descriptions

### API Server (`cmd/api-server/`)

**Purpose:** Business logic — nodes, subscriptions, chains, payments, key export.

* **Entry point:** `cmd/api-server/main.go` (thin, delegates to `internal/app/bootstrap/api_server.go`)
* **Port:** 8080 (HTTP)
* **Dependencies:** PostgreSQL, Redis, Temporal, Vault, NATS (optional for payment consumer)
* **Auth:** `DualAuthMiddleware` — JWT (primary) + API key (programmatic). See `internal/api/middleware.go`
* **Key packages:** `internal/api/` (handlers, routing, middleware), `internal/service/` (business logic)

**Startup:** Load config -> Init telemetry -> Init secrets (Vault) -> Connect PostgreSQL -> Init repos -> Connect Temporal -> Init chain config -> Init services -> Connect Redis -> Start HTTP server -> Wait for signal -> Graceful shutdown.

> **Note:** Database migrations are handled by the dedicated `cmd/migrate` binary as a pre-deploy step, not at service startup.

> **See also:** [CLAUDE.md — Server Separation](../../CLAUDE.md) for endpoint lists.

### Auth Server (`cmd/auth-server/`)

**Purpose:** Identity, authentication, wallet registration, API key management.

* **Entry point:** `cmd/auth-server/main.go` (thin, delegates to `internal/app/bootstrap/auth_server.go`)
* **Port:** 8081 (HTTP), 9094 (metrics)
* **Dependencies:** PostgreSQL, Vault (JWT keys). Minimal — no Temporal, no Redis
* **Key packages:** `internal/auth/` (service, handlers, JWT), `internal/authprovider/` (Clerk adapter)

**Key features:**

* Clerk webhook endpoint (`POST /webhooks/clerk`) for user lifecycle sync
* JWT session management (RS256, 15m access / 7d refresh, atomic rotation)
* Chain-agnostic wallet registration via `SignatureVerifierRegistry`
* API key CRUD and rotation
* IP-based rate limiting (20 req/min, Redis-backed for global enforcement across instances)

> **See also:** [Clerk Setup](../operations/clerk-setup) for operational configuration.

### Agent Gateway (`cmd/agent-gateway/`)

**Purpose:** gRPC endpoint for ops-agent communication.

* **Entry point:** `cmd/agent-gateway/main.go`
* **Port:** 9090 (gRPC), 9091 (metrics HTTP)
* **Dependencies:** PostgreSQL, Redis, NATS
* **Key packages:** `internal/grpc/` (server), `internal/commandqueue/` (Redis queue + progress)

**Responsibilities:**

* Agent registration and heartbeat processing
* Command queue delivery (Redis -> agent via heartbeat response)
* DEK retrieval for key decryption
* Progress tracking for long-running commands (Redis `progress:{commandID}`)
* Event forwarding to NATS

### Orchestrator (`cmd/orchestrator/`)

**Purpose:** Temporal workflow worker — executes provision, migrate, and terminate workflows.

* **Entry point:** `cmd/orchestrator/main.go`
* **Dependencies:** PostgreSQL, Redis, Temporal, Vault, Terraform, S3
* **Key packages:** `internal/workflows/` (workflow definitions), `internal/activities/` (activity implementations), `internal/terraform/` (infrastructure provisioning)

**Registered workflows:** `ProvisionNodeWorkflow`, `MigrateNodeWorkflow`, `TerminateNodeWorkflow`, `RolloutGroupWorkflow`, `RolloutWorkflow`, `UpgradeNodeWorkflow`

**Startup:** Load config -> Init telemetry -> Init secrets -> Connect PostgreSQL -> Init repos + crypto -> Init chain config + Terraform -> Connect Redis -> Connect Temporal -> Create worker -> Register workflows + activities -> Start worker -> Wait for signal.

### Health Evaluator (`cmd/health-evaluator/`)

**Purpose:** Background daemon for health evaluation, incident management, notifications, and cleanup.

* **Entry point:** `cmd/health-evaluator/main.go`
* **Port:** 9090 (metrics HTTP)
* **Dependencies:** PostgreSQL, Temporal (for migration triggers), NATS (event subscription), S3 (backup cleanup), Victoria Metrics (metrics queries)
* **Key packages:** `internal/health/` (machine, evaluator, outbox, cleanup), `internal/incident/` (service, notifier), `internal/observation/` (policy evaluation, metrics ingestion), `internal/uptime/` (state log handler, uptime worker)

**Leader election:** Uses PostgreSQL advisory lock-based leader election. One active leader, N-1 hot standby instances. 9 of 11 goroutine loops are leader-gated; 2 run on all instances (outbox worker, metrics ingester). Leader election uses a dedicated `pgx.Conn` (not pooled) for advisory lock persistence. On leader failure, a standby acquires the lock within one evaluation interval (15-30s).

**Subsystems run concurrently:**

| Subsystem            | Leader-gated? | Description                                                                                          |
| -------------------- | ------------- | ---------------------------------------------------------------------------------------------------- |
| Heartbeat evaluator  | YES           | 30s cycle, batch evaluation via 3-way JOIN snapshot                                                  |
| Policy evaluator     | YES           | 60s cycle, CEL policy evaluation against Victoria Metrics                                            |
| Outbox worker        | NO            | Polls `health_event_outbox`, dispatches to handlers (`FOR UPDATE SKIP LOCKED` — multi-instance safe) |
| Incident pipeline    | YES           | Incident service + notification dispatcher (Slack, Telegram, Email, Webhook)                         |
| Uptime worker        | YES           | 5min cycle, materializes hourly uptime buckets from state transition log                             |
| Metrics ingester     | NO            | NATS -> Victoria Metrics (idempotent writes)                                                         |
| Subscription cleanup | YES           | Expiration, grace period, pending payment TTL                                                        |
| Backup cleanup       | YES           | S3 orphaned backup removal                                                                           |
| Terraform cleanup    | YES           | Orphaned state directory removal                                                                     |
| Maintenance cleanup  | YES           | Stuck maintenance node recovery                                                                      |

> **See also:** [Health and Incidents](./health-and-incidents) for the full pipeline architecture.

### Ops Agent (`cmd/ops-agent/`)

**Purpose:** Lightweight on-host agent for node lifecycle management.

* **Entry point:** `cmd/ops-agent/main.go`
* **Runs on:** Each node VM (installed via cloud-init)
* **Communication:** gRPC client -> Agent Gateway, NATS publisher for metrics
* **Key packages:** `internal/opsagent/` (agent core, commands, recipes, config, state tracking, observation, upgrade)

**Responsibilities:**

* Lifecycle control (start/stop/restart node process via systemd)
* Upgrade execution via three-layer architecture (actions, runtime adapters, executor)
* Configuration application via declarative recipes (`hoodcloud-chain-configs/recipes/`)
* System and chain metric collection via observation runner
* Sync status tracking (events forwarded via gRPC -> NATS)
* Key injection (encrypted key material decrypted with DEK in memory)
* Progress monitoring for long-running operations (snapshot downloads)
* Self-update mechanism

**Startup:** Load config -> Create metrics collector -> Create agent -> Init state tracker -> Init observation runner (if `observation.yaml` exists) -> Start gRPC server -> Register with control plane -> Fetch DEK -> Start state tracking + observation + heartbeat loops -> Wait for signal -> Stop node -> Clear DEK -> Shutdown.

## Key Packages

| Package                                     | Purpose                                                              |
| ------------------------------------------- | -------------------------------------------------------------------- |
| `internal/contracts/`                       | Canonical interfaces (repositories, services, notifiers)             |
| `internal/models/`                          | Domain model types (Node, Subscription, User, Incident, etc.)        |
| `internal/database/`                        | PostgreSQL repositories (auth/, ops/, user/ domains)                 |
| `internal/service/`                         | Business logic services (NodeService, SubscriptionService, etc.)     |
| `internal/crypto/`                          | AES-256-GCM encryption, NaCl sealed box                              |
| `internal/vault/`                           | HashiCorp Vault client (AppRole, Transit, PKI, circuit breaker)      |
| `internal/chains/`                          | Chain profile loading (local filesystem or S3 with version polling)  |
| `internal/upgrade/manifest/`                | Upgrade manifest reader (YAML manifests from chain config directory) |
| `internal/provision/input/`                 | Provisioning input framework (schema, validation, storage)           |
| `internal/wallet/`                          | Chain-agnostic wallet verification (Ethereum, Solana)                |
| `internal/consumers/`                       | NATS consumers (payment events, idempotency)                         |
| `internal/observation/`                     | Metrics collection, transport, CEL policy evaluation                 |
| `internal/uptime/`                          | Rolling uptime calculation (state log handler, uptime worker)        |
| `internal/terraform/`                       | Terraform execution and self-describing module registry              |
| `internal/health/leader.go`                 | PostgreSQL advisory lock leader election (generic, reusable)         |
| `internal/app/bootstrap/leader_election.go` | Leader election bootstrap pattern                                    |
| `internal/correlation/`                     | Correlation ID propagation                                           |
| `internal/telemetry/`                       | OpenTelemetry distributed tracing                                    |

## Related Documents

**Architecture:**

* [Domain Model](./domain-model) — Domain objects, state machines, business rules, DB schema
* [Workflows](./workflows) — Temporal workflows, provisioning inputs, NATS consumers
* [NATS JWT Operator Mode](./nats-jwt) — JWT authentication, account structure, credential flows
* [Health and Incidents](./health-and-incidents) — Health evaluation, incidents, notifications, cleanup
* [Payment Service](./payment-service) — Isolated payment microservice
* [Extending](./extending) — Extension points, adding chains/providers/channels
* [Developer Guide](./developer-guide) — Reading guide, patterns, debugging

**Operations:**

* [Environment Variables](../operations/environment-variables) — Complete env var reference
* [Vault](../operations/vault) — Vault setup, secret structure, operations
* [Deployment and Operations](../operations/deployment-and-operations) — Local dev, day-to-day ops