Developer Guide

​

How to Read This Codebase

​

Recommended Reading Order

1. Domain models — internal/models/node.go, internal/models/subscription.go. Understand state machines and business rules.
   • See: Domain Model
2. Request flow — cmd/api-server/main.go -> internal/app/bootstrap/api_server.go -> internal/api/router.go -> internal/api/middleware.go -> handlers -> services -> repositories.
   • See: Workflows — HTTP Request Flow
3. Workflows — internal/workflows/provision.go, internal/activities/. Understand the Temporal orchestration pattern.
   • See: Workflows
4. Upgrade rollout — internal/workflows/rollout.go, internal/opsagent/upgrade/. Understand three-level workflow hierarchy and agent-side three-layer architecture.
   • See: Workflows — Upgrade Rollout Workflows
5. Agent communication — proto/agent.proto, internal/grpc/server.go, cmd/ops-agent/main.go. Understand bidirectional gRPC communication.
6. Health monitoring — internal/health/leader.go -> internal/health/machine.go -> internal/health/heartbeat.go -> internal/health/outbox.go -> internal/incident/service.go -> internal/incident/notifier/. Uptime: internal/uptime/state_log_handler.go -> internal/uptime/worker.go.
   • See: Health and Incidents
7. Chain configuration — hoodcloud-chain-configs/chains/. Understand the declarative, chain-agnostic approach.
   • See: Extending — Adding a New Chain

​

Key Files

​

Payment Service Key Files

​

Common Patterns

​

Repository Pattern

​

Service Layer

​

Temporal Workflows

​

Declarative Configuration

​

Event-Driven Architecture

Evaluator -> NodeHealthMachine -> health_event_outbox -> OutboxWorker -> handlers

​

Graceful Degradation

​

Error Handling

​

Error Types and Handling

​

Workflow Error Propagation

Activity fails -> Temporal retry policy (3 attempts, exponential backoff)
  -> All retries fail -> workflow receives error
  -> compensate() called
  -> Compensating activities (delete keys, destroy infra, update state to "failed")
  -> Workflow returns error result
  -> Metrics incremented, logs written

​

HTTP Error Responses

{
  "error": "validation failed",
  "details": "chain_profile_id is required"
}

​

Temporal Retry Policies

​

Tracing Code Paths

​

”How does a node get marked as DOWN?”

1. Health evaluator leader fires every 30s (leader-gated via advisory lock) -> ListSnapshots() (3-way JOIN, includes state_version)
2. EvaluateHeartbeat() per node — pure function, no I/O. If heartbeat timed out: increment ConsecutiveFailures. If >= ConsecutiveFailuresForDown (3): return transition to DOWN.
3. NodeHealthMachine.ApplyHeartbeatDecisions() — batch update with optimistic locking on both node_health_state.version and nodes.state_version. Inserts HealthTransitionEvent into health_event_outbox.
4. OutboxWorker (NOT leader-gated, uses FOR UPDATE SKIP LOCKED) dispatches to CompositeTransitionHandler -> incident service creates node_down incident -> migration handler evaluates grace period and cooldown, triggers migration with deterministic workflow ID (migrate-node-{nodeID}-{healthStateVersion}).

​

”How does a node get upgraded?”

1. Operator creates rollout via POST /api/v1/rollouts — either with upgrade_id (auto mode: manifest auto-populates binary details, state compatibility, config changes) or with all fields explicit (manual mode)
2. Operator starts rollout via POST /api/v1/rollouts/{id}/start
3. RolloutWorkflow launched in Temporal → ResolveRolloutTargets assigns nodes to batches
4. Per batch: launches UpgradeNodeWorkflow children in parallel
5. UpgradeNodeWorkflow: PreCheckNode → UpdateNodeState(maintenance) → SendUpgradeCommand
6. Agent receives COMMAND_TYPE_UPGRADE (value 9) in heartbeat → executor runs 8 actions via RuntimeAdapter
7. WaitForCommandCompletion → UpdateNodeBinaryVersion → WaitForHealthValidation → UpdateNodeState(syncing)
8. On failure: agent runs per-action compensation (reverse order), workflow marks node failed/rolled_back

​

”How does uptime get computed?”

1. State transition emitted by NodeHealthMachine -> health_event_outbox
2. OutboxWorker dispatches to CompositeTransitionHandler -> StateLogHandler.OnTransition() closes previous node_state_log entry, inserts new entry
3. UptimeWorker (5min cycle) finds last complete bucket per node, computes hourly buckets from node_state_log entries, upserts into node_uptime_hourly
4. API request: UptimeService queries node_uptime_hourly with SUM(uptime_seconds) / SUM(total_seconds) over requested window

​

”How does an agent send a heartbeat?”

1. Agent heartbeat loop (15s) -> sendHeartbeat()
2. gRPC ControlPlaneService.Heartbeat() with node_id
3. Control plane updates agent_registrations.last_seen_at, returns pending commands from Redis queue
4. Agent executes commands, reports results via ReportEvent()

​

Debugging Tips

​

Database Migrations

go run cmd/migrate/main.go          # Apply pending migrations

​

Leader Election

# Check which instance is the leader
curl -s http://localhost:9090/metrics | grep health_evaluator_is_leader

# Verify advisory lock is held
psql -c "SELECT * FROM pg_locks WHERE locktype = 'advisory';"

​

Logging

LOG_LEVEL=debug ./bin/api-server

​

Database Guardrails

​

Database Queries

-- Node and health state
SELECT id, chain_profile_id, state, last_heartbeat FROM nodes;
SELECT node_id, consecutive_failures, down_since, current_state FROM node_health_state;

-- Active incidents
SELECT id, node_id, category, severity, status, occurrence_count
FROM incidents WHERE status NOT IN ('resolved', 'auto_resolved');

-- Pending health events
SELECT id, event_type, status FROM health_event_outbox WHERE status = 'pending';

-- Agent connectivity
SELECT node_id, agent_version, last_seen_at FROM agent_registrations;

-- Uptime state log (recent transitions)
SELECT node_id, state, entered_at, exited_at, trigger
FROM node_state_log WHERE node_id = '<uuid>' ORDER BY entered_at DESC LIMIT 20;

-- Uptime hourly buckets
SELECT bucket_hour, uptime_seconds, downtime_seconds, unknown_seconds, is_complete
FROM node_uptime_hourly WHERE node_id = '<uuid>' ORDER BY bucket_hour DESC LIMIT 24;

​

Distributed Traces (Tempo)

1. Open Grafana -> Explore -> Tempo
2. Search by service name or trace ID
3. TraceQL queries: {resource.service.name="api-server"}, {status=error}

​

Temporal Workflows

1. Open Temporal UI: http://localhost:8233
2. Search for workflow ID: provision-{uuid}
3. View execution history (activities, retries, errors)

​

Redis Command Queue

redis-cli KEYS hoodcloud:commands:*
redis-cli LRANGE hoodcloud:commands:{node_id} 0 -1

​

NATS Streams

nats stream info HOODCLOUD_METRICS
nats stream info PAYMENTS
nats consumer info PAYMENTS main-app-payment

​

Victoria Metrics

curl "http://localhost:8428/api/v1/query?query=up{node_id='...'}"

​

Terraform State

# Local backend
cd /app/terraform-state/{host_id} && terraform show

# S3 backend (TERRAFORM_STATE_BACKEND=s3)
aws s3 ls s3://hoodcloud-terraform-state/nodes/ --recursive

​

Status Page (Gatus)

​

Running Tests

​

Unit Tests

go test ./...                          # All
go test ./internal/service/...         # Specific package
go test -v ./internal/workflows/...    # Verbose

​

Integration Tests

go test -tags=integration ./tests/integration/...

​

E2E Tests

docker compose -f tests/e2e/docker-compose.e2e.yml up -d
go test -tags=e2e ./tests/e2e/...
docker compose -f tests/e2e/docker-compose.e2e.yml down -v

​

Payment Service Tests

cd payment-service
make test

​

Related Documents

• Overview — System overview, tech stack, service descriptions
• Domain Model — Domain objects, state machines, business rules
• Workflows — HTTP flow, Temporal workflows, provisioning inputs
• Health and Incidents — Health evaluation, incidents, notifications
• Payment Service — Payment architecture
• Extending — Extension points, adding new capabilities
• Environment Variables — Configuration reference
• Deployment and Operations — Local dev, operations