Hermes lab runbook

Source: g2labs-in/linode-infra@17e7500 · docs/runbooks/hermes-labs.md · updated 2026-09-16

Last verified: 2026-09-16 on hermes-labs (install, provider, persona, gateway reboot proof). Bot tokens still pending.

Dyson is the lab agent on the Azure VM hermes-labs. This runbook covers what runs there, how to rebuild it, how its state is backed up, and how to recover it.

Where it runs

Fact	Value
Host	Azure VM `hermes-labs`, resource group `billuu-prod`, centralindia
Size	Standard_B2ats_v2, 2 vCPU, 1 GB RAM
Disk	64 GB P6
Public IP	`20.198.3.153`
User	`codejack`, key-only SSH
Tag	`env=lab`
Tailscale	`hermes-labs-1` at `100.101.91.58`
Cost	B2ats_v2 and one P6 disk fit the free tier for 12 months. The public IP is billed, about $3.65/mo

ssh codejack@20.198.3.153            # public IP, always works
ssh codejack@100.101.91.58           # tailnet, from a machine that is on it

What runs there

Piece	Detail
Hermes Agent	v0.21.3, installed at `/home/codejack/.hermes`, CLI at `~/.local/bin/hermes`
Default model	`model.default` in `~/.hermes/config.yaml`, set to `opencode-go/deepseek-v4.1-flash`
Provider	OpenCode Go. Key in `~/.hermes/.env` as `OPENCODE_GO_API_KEY`
Terminal backend	`local`, no Docker. 1 GB of RAM has no room for it
Gateway	systemd `hermes-gateway.service`, runs as `codejack`, enabled at boot
Persona	`azure/hermes-SOUL.md` is the source of `~/.hermes/SOUL.md`
Bots	Telegram `@Dyson_v1_bot`, wired and polling. Discord pending
Desktop	Hermes Desktop on the laptop, over an app-managed SSH connection. The app spawns and forwards its own isolated backend; no service on the VM

The agent is the team CRM for Billuu. Its one job is the contact database, SQLite at ~/.hermes/data/contacts.db.

Bot tokens and their allowlists go in ~/.hermes/.env: TELEGRAM_BOT_TOKEN with TELEGRAM_ALLOWED_USERS (set), and DISCORD_BOT_TOKEN with DISCORD_ALLOWED_USERS (pending). The smoke check reports their presence.

Rebuild from scratch

Every command runs from this repo. About 30 minutes, most of it waiting on the VM.

1. Provision the VM

azure/hermes-cloud-init.yaml holds no secrets, so it goes in as custom-data unchanged. It adds a 2 GB swapfile, because 1 GB of RAM is half of what the agent wants, then enables UFW (22 and tailscale0), fail2ban, and the Tailscale package.

az group create -n billuu-prod -l centralindia

az vm create -g billuu-prod -n hermes-labs -l centralindia \
  --image Ubuntu2404 --size Standard_B2ats_v2 \
  --admin-username codejack --ssh-key-values ~/.ssh/id_ed25519.pub \
  --os-disk-size-gb 64 --storage-sku Premium_LRS \
  --custom-data azure/hermes-cloud-init.yaml --tags env=lab

SKU restrictions flags on this subscription are unreliable. A size that runs can be flagged NotAvailableForSubscription, this one included. Confirm with az vm create --validate before believing the flag.

Verify the bootstrap:

ssh codejack@20.198.3.153 'cloud-init status --long; free -h | head -2; sudo ufw status | head -5'

2. Join Tailscale

The Tailscale auth key is single use and short lived, so the join is interactive:

ssh codejack@20.198.3.153 'sudo tailscale up'

Approve the node in the admin console. The tailnet names it hermes-labs-1, because an earlier node still holds the bare name. The address should be 100.101.91.58.

3. Install Hermes Agent

ssh codejack@20.198.3.153
curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash
hermes --version

The agent state lands in /home/codejack/.hermes. The version at the time of writing is v0.21.3.

4. Wire the provider

The OpenCode Go key is reused from the workstation, at ~/.local/share/opencode/auth.json. Put its value in ~/.hermes/.env as OPENCODE_GO_API_KEY, then set model.default in ~/.hermes/config.yaml to opencode-go/deepseek-v4.1-flash. Values never enter this repo.

Check the model without the TUI:

hermes -z "Reply with exactly the word READY"

5. Install the gateway service

The install needs root, and sudo drops HOME, which the installer needs to find the user’s state. One command covers both quirks:

sudo HOME=$HOME /home/codejack/.local/bin/hermes gateway install --system \
  --run-as-user codejack --start-now --start-on-login

Plain hermes gateway install --system fails without root. Under sudo the binary needs its full path, because root’s PATH has no ~/.local/bin.

Check the unit:

systemctl status hermes-gateway
systemctl is-enabled hermes-gateway

6. Deploy the persona

scp azure/hermes-SOUL.md codejack@20.198.3.153:.hermes/SOUL.md

azure/hermes-SOUL.md is the canonical copy. Change it there, never on the VM alone.

7. Run the smoke check

scp azure/hermes-smoke.sh codejack@20.198.3.153:~
ssh codejack@20.198.3.153
export PATH=$PATH:$HOME/.local/bin && bash hermes-smoke.sh

One PASS, FAIL, or SKIP line per check: version, one-shot model call, gateway unit state, bot token presence. The bot token check skips until the tokens are wired. The script exits non-zero on any FAIL.

8. Prove the reboot

ssh codejack@20.198.3.153 'sudo reboot'
# wait about a minute, then
ssh codejack@20.198.3.153 'systemctl is-active hermes-gateway'

Run the smoke check again once the host answers. The last reboot proof, 2026-09-16, came back in 70 seconds with every check green.

9. Connect the laptop desktop

On the laptop, with Hermes Desktop installed: Settings → Gateways → Add connection → SSH, name Dyson, host hermes-labs (the SSH alias). Save, then Test until it reads Reachable. Make it Primary so the app opens on Dyson. The app spawns and manages the remote backend over the tunnel; nothing else needs to run on the VM.

State and backup

The whole agent lives in ~/.hermes. A dedicated private repo, Aakash-chouksey/hermes-lab, holds the nightly copy. This repo keeps only the tooling and this runbook.

Piece	Where
Canonical script	`azure/hermes-backup.sh` in this repo
Script on the VM	`/home/codejack/hermes-backup.sh`
Clone	`/home/codejack/hermes-backup/repo`
Auth	SSH deploy key at `/home/codejack/.ssh/hermes-backup`, set per clone with `core.sshCommand`. No personal token
Age identity	`/home/codejack/hermes-backup/age.key`, also kept by the owner in a password manager
Age recipient	committed in the backup repo at `tooling/age-recipient.txt`
Ping URL	`/home/codejack/hermes-backup/backup.env` on the VM only
Schedule	`hermes-backup.timer`, daily 03:30 UTC, `Persistent=true`, `RandomizedDelaySec=300`
Unit sources	`hermes-backup.service` and `hermes-backup.timer`, committed in `azure/`

The script mirrors itself into the backup repo at tooling/hermes-backup.sh on every run, so the backup carries its own tooling.

Layout in the backup repo:

Path	Contents
`state/config.yaml`, `state/SOUL.md`, `state/auth.json`, `state/channel_directory.json`, `state/install_id`, `state/.hermes_history`	single files, same names as in `~/.hermes`
`state/memories/`, `state/sessions/`, `state/skills/`, `state/cron/`, `state/hooks/`, `state/platforms/`, `state/pairing/`, `state/state/`, `state/terminal-sessions/`, `state/pending_messages/`, `state/sandboxes/`, `state/data/`	directories, same names. `data/` holds the CRM contacts database
`state/kanban.db`, `state/state.db`, `state/shared-state.db`	consistent copies, taken through python3’s sqlite3 backup API instead of a raw file copy
`secrets/.env.age`	`~/.hermes/.env`, encrypted with age
`tooling/hermes-backup.sh`	the backup script itself
`tooling/age-recipient.txt`	the age public recipient

Backed up is the agent state, nothing else. The owner’s personal notes are not in scope.

Excluded, because it is reproducible or large:

hermes-agent/     code, 1.2 GB
node/             208 MB
bin/              85 MB of browser binaries
audio_cache/, image_cache/, provider model caches, models_dev_cache.json
logs/, runtime/, lock and pid files
database -wal and -shm files

One healthchecks.io check watches the job. It is named Dyson backup, period 24 hours. The script pings it on success and pings the /fail endpoint on failure, so a failed run and a silent one both alert.

Recovery on a fresh host

Install Hermes, step 3 above.
Clone Aakash-chouksey/hermes-lab to /home/codejack/hermes-backup/repo.
Copy the contents of state/ into ~/.hermes/.
Decrypt the env file and lock it down:

cd /home/codejack/hermes-backup
age -d -i age.key repo/secrets/.env.age > ~/.hermes/.env
chmod 600 ~/.hermes/.env

Reinstall the gateway service, step 5 above. The unit is not in the backup.
Run the smoke check, step 7 above.
Confirm the dashboard on port 8080 over Tailscale, then check the bot round trip.

Three things are not in the backup and need re-establishing:

Item	Where it comes from
Deploy key	The private half lived only on the VM. Generate a new pair and add the public half to `Aakash-chouksey/hermes-lab` as a deploy key
Age identity	The owner’s password manager, or `/home/codejack/hermes-backup/age.key` on the VM if it survives. Without it `secrets/.env.age` cannot be read and every key in it must be reissued
Repo access	The owner’s GitHub account, `Aakash-chouksey`

Reach it from the laptop

The laptop runs Hermes Desktop (part of the Omarchy Hermes install). Its connection to Dyson is an SSH entry in Desktop’s registry, pointing at this host’s SSH alias. The app does the rest: it opens a control master to hermes-labs, finds the Hermes install, spawns an isolated hermes serve on the VM, forwards a local port to it, and adopts a session token over the tunnel. No credentials are typed, and no service runs on the VM while the app is closed. Reconnects reuse the same spawned server.

Registry entry (in ~/.config/Hermes/connections.json on the laptop):

{
  "id": "dyson",
  "kind": "ssh",
  "label": "Dyson",
  "host": "hermes-labs"
}

The same connection can be created from the app: Settings → Gateways → Add connection → SSH, name Dyson, host hermes-labs.

Do not put a LocalForward line in the laptop’s hermes-labs SSH config block. Desktop manages its own forward; a fixed forward on the same port fails the connection with bind [::1]:9119: Address already in use. That cost one debug cycle on 2026-09-16.

Optional browser path instead of the app: run hermes serve --host 0.0.0.0 --port 9119 as a service with HERMES_DASHBOARD_BASIC_AUTH_* credentials set in ~/.hermes/.env, then browse http://100.101.91.58:9119 over Tailscale from any tailnet device. It costs about 140 MB of RAM on a 1 GB box, so it is off by default.

Quirks and gotchas

gateway install     needs root, and needs HOME. sudo HOME=$HOME <full path>/hermes gateway install
                    --system --run-as-user codejack --start-now --start-on-login
interactive chat    a TUI, so it needs a real pty. Scripted checks use the one-shot path hermes -z
tailnet name        the node is hermes-labs-1, not hermes-labs. An earlier node holds the bare name
RAM                1 GB is tight. The 2 GB swapfile from cloud-init is not optional
cost                the VM and the disk are free for 12 months. The public IP is billed, about $3.65/mo
ports              no always-on server. Desktop spawns an isolated backend per connect and forwards an ephemeral local port
forward            never add LocalForward for hermes-labs to the laptop SSH config. Desktop owns the forward
dead addresses     GCP-era 100.88.24.82 and 34.180.12.153 are obsolete. Never use them
SKU flags          restrictions flags lie on this subscription. Validate with az vm create --validate
frozen disk        the GCP-era ~/.hermes, bot tokens included, exists only on the frozen GCP disk.
                    Rescue or recreate it before any purge
naming             the agent instance is Dyson. The repo file azure/hermes-SOUL.md is its persona source
deploys            change this repo, then deploy. Never edit the VM copy alone

azure/hermes-cloud-init.yaml                  VM bootstrap, no secrets
azure/hermes-smoke.sh                         the rerunnable verification
azure/hermes-SOUL.md                          the persona, deployed to ~/.hermes/SOUL.md
azure/hermes-backup.sh                        the backup script, mirrored into the backup repo
.scratch/hermes-labs-setup/issue.md           the PRD and its status updates
.scratch/hermes-labs-install-agent/issue.md   the install record
docs/adr/0004-hermes-labs-ai-agent-server.md  the desktop and dashboard auth model
CONTEXT.md                                    the lab host entry