Vault Integration - HoodCloud

HoodCloud uses HashiCorp Vault for secrets management.

Overview

Vault provides:

KV v2 Engine: Application secrets (DB passwords, API keys, JWT RSA keys)
Transit Engine: DEK encryption without exposing master key
AppRole Auth: Machine-to-machine authentication
Audit Logging: Complete access audit trail

Architecture

Local Development

docker compose -f docker-compose.dev.yml up handles Vault automatically. No manual setup needed. The dev compose includes:

vault — Dev mode (in-memory, no TLS, token: dev-token, http://localhost:8200)
vault-init — Runs scripts/vault-dev-init.sh to seed engines, policies, AppRole, keys

Shared volume pattern: the sidecar writes AppRole credentials to /vault/config/. Services mount this volume read-only and use an entrypoint wrapper:

entrypoint: ["/bin/sh", "-c", "export VAULT_ROLE_ID=$(cat /vault/config/role-id) && exec ./service-name"]

E2E tests (tests/e2e/docker-compose.e2e.yml) use the same script on port 8201.

Production Setup

Prerequisites

Docker and Docker Compose v2+
OpenSSL (TLS cert generation)
jq (migration script)

1. Generate TLS Certificates

VAULT_SERVER_IP=<vault-server-ip> ./scripts/generate-vault-certs.sh

Creates in infrastructure/docker/config/vault/tls/:

vault-ca.crt — CA certificate (distribute to clients)
vault-ca.key — CA private key
vault.crt — Vault server certificate (includes IP in SAN)
vault.key — Vault server private key

2. Prepare Data Volumes

cd infrastructure/docker
docker network create hoodcloud 2>/dev/null || true
docker run --rm \
    -v hoodcloud-vault-data:/vault/data \
    -v hoodcloud-vault-logs:/vault/logs \
    hashicorp/vault:1.15 chown -R vault:vault /vault/data /vault/logs

3. Deploy Vault

# Default: bind to 127.0.0.1:8200 (local only)
docker compose -f docker-compose.vault.yml --profile vault-prod up -d vault

# Remote access (for separate control plane server):
VAULT_BIND_ADDR=0.0.0.0 docker compose -f docker-compose.vault.yml --profile vault-prod up -d vault

Persist VAULT_BIND_ADDR=0.0.0.0 in .env for remote access.

Note: The command must be server (no -config= flag). The Docker entrypoint adds -config=/vault/config automatically. Explicit -config causes “address already in use”.

4. Configure Firewall

ufw allow ssh
ufw allow from <control-plane-ip> to any port 8200 comment "Control plane -> Vault"
ufw allow from <admin-ip> to any port 8200 comment "Admin -> Vault"
ufw enable

5. Initialize Vault

docker exec -it hoodcloud-vault vault operator init -key-shares=5 -key-threshold=3

Save the output: 5 unseal keys + root token.

# Unseal (3 of 5 keys)
docker exec -it hoodcloud-vault vault operator unseal <key-1>
docker exec -it hoodcloud-vault vault operator unseal <key-2>
docker exec -it hoodcloud-vault vault operator unseal <key-3>

6. Run Initialization Script

export VAULT_ADDR=https://<vault-server-ip>:8200
export VAULT_CACERT=infrastructure/docker/config/vault/tls/vault-ca.crt
export VAULT_TOKEN=<root-token>

./scripts/vault-init.sh

Creates:

KV v2 at secret/, Transit engine with hoodcloud-master key
AppRole roles: hoodcloud-control-plane, hoodcloud-auth-server, hoodcloud-payment-service, hoodcloud-admin
Policies for each role
Audit logging at /vault/logs/audit.log
Admin AppRole credentials (printed at the end — save these)

7. Revoke Root Token

vault token revoke <root-token>

The admin AppRole replaces the root token for day-to-day management. To regenerate a root token (emergency):

vault operator generate-root -init
vault operator generate-root  # provide unseal keys

8. Generate JWT RS256 Keys

Generate the RSA keypair for JWT signing:

openssl genrsa -out jwt_private.pem 2048
openssl rsa -in jwt_private.pem -pubout -out jwt_public.pem

9. Generate X25519 Sealed Box Keypair

Generate the NaCl sealed box keypair for user-provided secret encryption. These are mandatory — api-server and orchestrator fail fast at startup without them.

// Use crypto.GenerateSealedBoxKeyPair() from internal/crypto/sealedbox.go
pub, priv, err := box.GenerateKey(rand.Reader)
fmt.Println("sealed_box_public_key=" + base64.StdEncoding.EncodeToString(pub[:]))
fmt.Println("sealed_box_private_key=" + base64.StdEncoding.EncodeToString(priv[:]))

Note: These must be a mathematically related X25519 keypair, not random bytes. For local dev, vault-dev-init.sh seeds them automatically.

10. Populate Secrets

vault kv put secret/hoodcloud/app-credentials \
    db_password="GENERATE_STRONG_PASSWORD" \
    redis_password="GENERATE_STRONG_PASSWORD" \
    jwt_private_key=@jwt_private.pem \
    jwt_public_key=@jwt_public.pem \
    nats_operator_signing_seed="<operator signing seed from nats-jwt-setup>" \
    nats_agent_account_signing_seed="<AGENT account signing seed from nats-jwt-setup>" \
    nats_ctrl_account_signing_seed="<CTRL account signing seed from nats-jwt-setup>" \
    nats_agent_account_pub="<AGENT account public key from nats-jwt-setup>" \
    nats_ctrl_account_pub="<CTRL account public key from nats-jwt-setup>" \
    grpc_config_signing_key="base64-encoded-ecdsa-key" \
    clerk_secret_key="sk_live_..." \
    clerk_webhook_signing_secret="whsec_..." \
    sealed_box_public_key="<base64-X25519-public-key>" \
    sealed_box_private_key="<base64-X25519-private-key>" \
    incident_slack_webhook_url="https://hooks.slack.com/services/T.../B.../..." \
    incident_telegram_bot_token="123456:ABC-..." \
    incident_telegram_chat_id="-100..." \
    incident_email_api_url="https://api.sendgrid.com/v3/mail/send" \
    incident_email_api_key="SG.xxxx" \
    incident_email_from="[email protected]" \
    incident_email_to="[email protected]" \
    incident_webhook_url="https://hooks.example.com/incidents" \
    incident_webhook_secret="whsec_..." \
    terraform_env_vars='{"HCLOUD_TOKEN":"your-token","OVH_ENDPOINT":"ovh-eu"}' \
    provider_env_vars='{"ovh_subsidiary":"EU"}'

vault kv put secret/hoodcloud/master-key key="$(openssl rand -base64 32)"

# Cloudflare API token for NATS TLS certificate renewal (DNS-01 challenge)
# Token must have Zone:DNS:Edit + Zone:Zone:Read permissions
vault kv put secret/infra/certbot/cloudflare api_token=<cloudflare-api-token>

Important: terraform_env_vars and provider_env_vars must be JSON objects, not JSON strings. The application deserializes them as map[string]string.

Incident notification fields are optional. If none are configured, incidents are tracked in the database but no external notifications are sent. Or use the interactive migration script:

./scripts/vault-migrate-secrets.sh --manual

11. Deploy AppRole Credentials

# Get role_id (static, reuse across deployments)
ROLE_ID=$(vault read -field=role_id auth/approle/role/hoodcloud-control-plane/role-id)

# Generate and deploy secret_id to control plane
vault write -field=secret_id -f auth/approle/role/hoodcloud-control-plane/secret-id | \
    ssh root@<control-plane-ip> 'mkdir -p /opt/hoodcloud/secrets && cat > /opt/hoodcloud/secrets/vault-secret-id && chmod 644 /opt/hoodcloud/secrets/vault-secret-id'

Note: Use chmod 644 — containers run as non-root. The volume is mounted :ro.

12. Configure Control Plane Services

Copy the Vault CA certificate:

scp infrastructure/docker/config/vault/tls/vault-ca.crt root@<control-plane-ip>:/opt/hoodcloud/secrets/vault-ca.crt

Add to .env:

VAULT_ADDR=https://<vault-server-ip>:8200
VAULT_ROLE_ID=<role-id>
VAULT_SECRET_ID_PATH=/secrets/vault-secret-id
VAULT_CACHE_TTL=5m
VAULT_MASTER_KEY_PATH=hoodcloud/master-key
VAULT_APP_CREDENTIALS_PATH=hoodcloud/app-credentials
VAULT_TLS_CA_FILE=/secrets/vault-ca.crt

All five services (api-server, auth-server, agent-gateway, orchestrator, health-evaluator) need:

Vault env vars in docker-compose.yml
Volume mount: /opt/hoodcloud/secrets:/secrets:ro

Recreate services:

docker compose up -d api-server auth-server orchestrator health-evaluator agent-gateway

Important: Use up -d, not restart. restart does not reload .env changes.

13. Save Credentials and Clean Up

Store securely (password manager):

Credential	Storage
5 unseal keys	Distribute to separate trusted people
Admin AppRole role_id + secret_id	Password manager
Vault CA certificate (`vault-ca.crt`)	Safe to keep in repo (gitignored keys are not committed)

Delete from Vault server:

ssh root@<vault-server-ip> 'rm -f /root/vault-init-keys.json'

Secret Structure

secret/hoodcloud/
├── app-credentials          # Application secrets
│   ├── db_password
│   ├── redis_password
│   ├── jwt_private_key      # RSA private key PEM (auth-server only)
│   ├── jwt_public_key       # RSA public key PEM (all services)
│   ├── nats_operator_signing_seed
│   ├── nats_agent_account_signing_seed
│   ├── nats_ctrl_account_signing_seed
│   ├── nats_agent_account_pub
│   ├── nats_ctrl_account_pub
│   ├── grpc_config_signing_key
│   ├── clerk_secret_key
│   ├── clerk_webhook_signing_secret
│   ├── sealed_box_public_key   # base64 X25519 (32 bytes, mandatory)
│   ├── sealed_box_private_key  # base64 X25519 (32 bytes, mandatory)
│   ├── incident_slack_webhook_url      # optional
│   ├── incident_telegram_bot_token     # optional
│   ├── incident_telegram_chat_id       # optional
│   ├── incident_email_api_url          # optional
│   ├── incident_email_api_key          # optional
│   ├── incident_email_from             # optional
│   ├── incident_email_to               # optional
│   ├── incident_webhook_url            # optional
│   ├── incident_webhook_secret         # optional
│   ├── payment_client_cert  # PEM (optional, mTLS)
│   ├── payment_client_key   # PEM (optional, mTLS)
│   ├── payment_ca_cert      # PEM (optional, mTLS)
│   ├── terraform_env_vars   # JSON object: {"HCLOUD_TOKEN": "..."}
│   └── provider_env_vars    # JSON object: {"ovh_subsidiary": "EU"}
├── master-key               # Master encryption key
│   └── key
└── providers/               # Cloud provider credentials
    ├── hetzner/
    │   └── HCLOUD_TOKEN
    └── ovh/
        ├── OVH_APPLICATION_KEY
        ├── OVH_APPLICATION_SECRET
        └── OVH_CONSUMER_KEY

secret/infra/
└── certbot/
    └── cloudflare               # Cloudflare API token for DNS-01 challenge
        └── api_token            # Zone:DNS:Edit + Zone:Zone:Read scoped

secret/payment-service/
└── credentials              # Payment service secrets
    ├── db_password
    ├── redis_password
    ├── stripe_secret_key
    ├── stripe_webhook_secret
    └── nats_ctrl_signing_seed   # CTRL account signing seed (for NATS JWT auth)

transit/keys/
└── hoodcloud-master         # Transit encryption key (DEK wrapping)

Note: The payment service’s nats_ctrl_account_pub (CTRL account public key) is not stored in Vault. It comes from the NATS_CTRL_ACCOUNT_PUB environment variable or YAML config. Only the signing seed is a secret and must be in Vault.

Configuration

All Vault-related environment variables (control plane and payment service) are documented in Environment Variables.

Note: The payment service uses VAULT_ADDRESS (not VAULT_ADDR). See Environment Variables - Payment Service Vault for details.

Features

Circuit Breaker

State	Behavior
Closed	Normal operation, requests go to Vault
Open	Vault unreachable, return cached values
Half-Open	Test request to check recovery

Failure threshold: 3 consecutive failures
Recovery timeout: 30s (exponential backoff up to 5 min)

Secret Caching

Default TTL: 5 minutes (VAULT_CACHE_TTL)
Expired cache used as fallback when circuit is open
Cache cleared on service restart

Token Renewal

Renewal attempted at 75% of TTL
On failure, re-authenticates with AppRole credentials

Credential Expiry Monitoring

hoodcloud_credential_expiry_warning{credential_type="vault_token"} == 1

Warnings fire when < 25% of the original TTL remains. Configure alerts to schedule service restarts before expiry.

Operations

Unseal After Restart

Vault seals on every restart. Unseal with 3 of 5 keys:

docker exec -it hoodcloud-vault vault operator unseal <key-1>
docker exec -it hoodcloud-vault vault operator unseal <key-2>
docker exec -it hoodcloud-vault vault operator unseal <key-3>

While sealed, services use cached secrets (circuit breaker). If cache expires before unseal, services fail.

Authenticate as Admin

After root token is revoked:

export VAULT_ADDR=https://<vault-server-ip>:8200
export VAULT_CACERT=infrastructure/docker/config/vault/tls/vault-ca.crt

VAULT_TOKEN=$(vault write -field=token auth/approle/login \
    role_id=<admin-role-id> \
    secret_id=<admin-secret-id>)
export VAULT_TOKEN

vault kv list secret/hoodcloud/

Rotate Secrets

vault kv put secret/hoodcloud/app-credentials \
    db_password="new-password" \
    # ... all other fields (kv put replaces the entire secret)

# Restart services to pick up new values
docker compose restart api-server auth-server orchestrator health-evaluator agent-gateway

Rotate AppRole Secret ID

vault write -field=secret_id -f auth/approle/role/hoodcloud-control-plane/secret-id \
    > /opt/hoodcloud/secrets/vault-secret-id
chmod 644 /opt/hoodcloud/secrets/vault-secret-id

docker compose restart api-server auth-server orchestrator health-evaluator agent-gateway

View Audit Logs

docker exec hoodcloud-vault tail -f /vault/logs/audit.log | jq 'select(.request.path | startswith("secret/"))'

# Failed auth attempts
docker exec hoodcloud-vault grep -i "permission denied" /vault/logs/audit.log | jq .

Verify Secret Storage

# Verify terraform_env_vars is stored as JSON object (not string)
vault kv get -format=json secret/hoodcloud/app-credentials | \
    jq '.data.data.terraform_env_vars | type'
# Should output: "object"

Troubleshooting

Service Cannot Authenticate

Error: authenticate with Vault: permission denied

Check Vault status: vault status (sealed?)
Verify role exists: vault read auth/approle/role/hoodcloud-control-plane

Regenerate secret_id:

vault write -field=secret_id -f auth/approle/role/hoodcloud-control-plane/secret-id \
    > /opt/hoodcloud/secrets/vault-secret-id
chmod 644 /opt/hoodcloud/secrets/vault-secret-id

Secret Not Found

vault kv get secret/hoodcloud/app-credentials  # Verify it exists
echo $VAULT_APP_CREDENTIALS_PATH                # Check path matches

Circuit Breaker Open

Warn: Vault circuit breaker open, using cached value

curl -sk ${VAULT_ADDR}/v1/sys/health | jq .
vault status  # If sealed, unseal

TLS Connection Errors

Error: x509: certificate signed by unknown authority

Set VAULT_TLS_CA_FILE=/secrets/vault-ca.crt in .env, or for CLI:

export VAULT_CACERT=infrastructure/docker/config/vault/tls/vault-ca.crt

Security Model

AppRole Credential Security (Phase 0)

secret_id_num_uses=0 — Reusable for re-authentication after Vault restarts
secret_id_ttl=0 — No expiry
secret_id_bound_cidrs — Usable only from bound server IP
token_bound_cidrs — Tokens valid only from bound server IP

Compromised SecretID is useless from any IP other than the bound server.

Phase 1 Upgrade: SecretID Rotation

Add TTL to SecretIDs (secret_id_ttl=7d) and implement rotation via CI/Ansible. See scripts/vault-init.sh for AppRole configuration. Options:

Vault Agent sidecar for automated token management
Response wrapping (secret_id_num_uses=1) for strongest security

​Overview

​Architecture

​Local Development

​Production Setup

​Prerequisites

​1. Generate TLS Certificates

​2. Prepare Data Volumes

​3. Deploy Vault

​4. Configure Firewall

​5. Initialize Vault

​6. Run Initialization Script

​7. Revoke Root Token

​8. Generate JWT RS256 Keys

​9. Generate X25519 Sealed Box Keypair

​10. Populate Secrets

​11. Deploy AppRole Credentials

​12. Configure Control Plane Services

​13. Save Credentials and Clean Up

​Secret Structure

​Configuration

​Features

​Circuit Breaker

​Secret Caching

​Token Renewal

​Credential Expiry Monitoring

​Operations

​Unseal After Restart

​Authenticate as Admin

​Rotate Secrets

​Rotate AppRole Secret ID

​View Audit Logs

​Verify Secret Storage

​Troubleshooting

​Service Cannot Authenticate

​Secret Not Found

​Circuit Breaker Open

​TLS Connection Errors

​Security Model

​AppRole Credential Security (Phase 0)

​Phase 1 Upgrade: SecretID Rotation