ADR 0004: Hermes Labs — AI Agent Experimentation Server

Source: g2labs-in/linode-infra@f96c94b · docs/adr/0004-hermes-labs-ai-agent-server.md · updated 2026-09-13

Date: 2026-06-09 Status: accepted

Context

We need a dedicated server to host Hermes Agent (Nous Research) for experimentation with AI agent workflows. The goal is to test contact/CRM management and restaurant bot use cases for our Billuu restaurant POS SaaS before committing to production infrastructure. Currently we have no AI agent infrastructure and no way to experiment with conversational AI workflows.

The server must be:

Isolated from production services (dokploy-prod, billuu-db, billuu-monitoring)
On the same Tailscale network for private inter-VPS communication
Low-cost for experimentation (~$12/mo)
Capable of running Hermes Agent with Discord/Telegram adapters

Decision

Provision a GCP e2-small VM (hermes-labs) in asia-south1-a, joined to the existing Tailscale network. Install Hermes Agent via the official install script with a local terminal backend (no Docker, saves RAM). Use agent-native CRM with SQLite for structured contact storage instead of deploying a heavier CRM solution.

Alternatives considered

Option	Pros	Cons	Rejected because
Run Hermes on dokploy-prod	No extra cost	Competes with app for RAM, no isolation	Unacceptable for production
Run Hermes on billuu-monitoring	Already GCP	Monitoring stack needs dedicated resources	Wrong purpose
Deploy Twenty CRM (Node.js + Postgres)	Full-featured CRM	Heavy (Docker, PostgreSQL), overkill for experimentation	Too complex for initial experiments
Flat file storage	Simple	No query capability, no structure	Not practical for CRM
Hermes on dedicated GCP e2-small	Isolated, low-cost, on Tailscale	$12/mo extra	Chosen

Consequences

Positive: Dedicated space for AI experiments, no impact on production services, Tailscale connectivity to all VPSs, SQLite for fast iteration, easy to scale up if needed
Negative: $12/mo added cost (covered by GCP credits for ~2 years), manual backup management for SQLite, no automated monitoring yet
Neutral: Hermes Agent uses external model providers (OpenCode Go / OpenRouter), no local inference

Remote Desktop Architecture

The Hermes Desktop app on the local machine connects to the remote gateway on hermes-labs. This required solving several integration issues:

Architecture:

Gateway runs on port 8642 (API server, OpenAI-compatible, handles tool execution)
Dashboard runs on port 8080 (web UI, /api/status, /api/ws proxy)
Desktop app connects to dashboard (8080), not API server (8642)
Dashboard proxies WebSocket to gateway for real-time chat

Auth model:

Dashboard uses HERMES_DASHBOARD_SESSION_TOKEN env var (<see ~/.hermes/.env on hermes-labs>)
Desktop sends X-Hermes-Session-Token header with the same token
API server uses separate API_SERVER_KEY for external clients
--insecure flag needed because dashboard refuses 0.0.0.0 bind without OAuth providers

Why not port 8642? Desktop needs /api/status (dashboard-only endpoint) for liveness checks. API server only serves OpenAI-compatible REST/WebSocket, not the lifecycle endpoints the desktop requires.

Why --insecure? Dashboard enforces OAuth gate on non-loopback binds when no auth providers are registered. --insecure skips this. Security is handled by Tailscale (private network only) + session token.

Token storage gotcha: Desktop’s connection.json stores tokens encrypted via Electron’s safeStorage. You can’t hand-edit plaintext tokens there. Use HERMES_DESKTOP_REMOTE_URL + HERMES_DESKTOP_REMOTE_TOKEN env vars instead (bypasses encryption).

Scaling Path

If experiments need more RAM: Upgrade to e2-medium (4GB, ~$24/mo)
If experiments succeed: Production bots get dedicated servers with proper monitoring
If experiments fail: Delete the GCP instance, stop $12/mo cost

ADR 001: Self-hosted PostgreSQL (production DB infrastructure)
ADR 002: Dedicated DB VPS (fault isolation pattern)
ADR 0003: PgBouncer connection pooling (production DB optimization)