Environments
Source:
g2labs-in/billuu-docs@9f1dcd7·content/platform/environments.md· updated 2026-09-16
Billuu runs on two Azure VMs, one managed Postgres, and a set of free-tier services around them. The infrastructure repo at /home/codejack/code/linode is the source of truth for this page. Anything marked retired is kept only as history.
The estate was rebuilt on Azure and Cloudflare on 2026-09-13 and the observability layer closed on 2026-09-15. Older Linode and GCP material is still present in the repo, so read every row carefully.
Request flow
visitor | vCloudflare edge (DNS, TLS, WAF, rate limit, Access) | +-- billuu.com, www, billing, order -> Netlify (DNS only, no proxy) | +-- testing.billuu.com (proxied) -> Traefik on billuu-app -> backend :8000 | +-- dokploy.billuu.com (proxied) -> Cloudflare Access gate -> Traefik -> Dokploy :3000
billuu-app -> Grafana Cloud (metrics and logs, outbound push by Alloy)billuu-app -> healthchecks.io (dead-man pings, outbound)UptimeRobot -> public URLs, from outside the estateServers
| Host | Provider | Size | Disk | Region | Public IPv4 | Tailscale | Role |
|---|---|---|---|---|---|---|---|
billuu-app | Azure, resource group billuu-prod | Standard_B2ls_v2, 2 vCPU, 4 GB | 64 GB StandardSSD | centralindia | 20.235.64.99, static | 100.86.125.42 | production |
hermes-labs | Azure, resource group billuu-prod | Standard_B2ats_v2, 2 vCPU, 1 GB | 64 GB P6 | centralindia | 20.198.3.153 | node hermes-labs-1, 100.101.91.58 | lab, tag env=lab |
Both hosts are Ubuntu 24.04, key-only SSH, root login disabled, and running fail2ban. UFW allows 22 and everything on tailscale0 on both hosts, and billuu-app also allows 80 and 443. azure/cloud-init.yaml builds billuu-app and azure/hermes-cloud-init.yaml builds hermes-labs.
Every host joins the Tailscale network choukseyaakash62@. hermes-labs is joined as hermes-labs-1. The line in AGENTS.md that says the join is still pending is stale.
billuu-app costs about $44 per month, covered by the Azure credit until around 2026-10-13. hermes-labs fits the free tier for 12 months, with its public IP billed at about $3.65 per month after the credit. Two budgets, billuu-50 and billuu-150, email on spend thresholds. az consumption cannot read amounts on this offer, so the Azure portal is the only spend view.
Services on billuu-app
| Service | Form | Notes |
|---|---|---|
| Dokploy | container dokploy/dokploy v0.30.6 | admin panel at https://dokploy.billuu.com, behind Cloudflare Access, also reachable on port 3000 over Tailscale |
| Traefik | Dokploy-managed container | reverse proxy on ports 80 and 443, Let’s Encrypt certificates |
| Backend | Docker Swarm service billuu-backend-poxd1a | Node and Express on port 8000, built from the stabilize-phase-1-2 branch |
| Redis | Swarm service billuu-redis-engwdt, Redis 8 | internal only, no published port, volume billuu-redis-engwdt-data |
| Dokploy Postgres | container dokploy-postgres | Dokploy’s own database, not the application database |
| Alloy | systemd alloy.service | ships host metrics, container logs, and the journal to Grafana Cloud |
| Firewall unit | systemd billuu-firewall.service | re-applies the port 443 filter at boot, after docker.service |
| Host jobs | /etc/cron.d/billuu-* | backups, edge IP refresh, heartbeat, firewall guard |
Dokploy owns /etc/dokploy/traefik/dynamic/. Those files are regenerated, so never hand-edit them.
Services on hermes-labs
| Piece | Detail |
|---|---|
| Hermes Agent | v0.21.3 at /home/codejack/.hermes, CLI at ~/.local/bin/hermes |
| Model | OpenCode Go, default opencode-go/deepseek-v4.1-flash, key in ~/.hermes/.env |
| Gateway | systemd hermes-gateway.service, runs as codejack, enabled at boot |
| Persona | azure/hermes-SOUL.md is the source of ~/.hermes/SOUL.md |
| Bots | Telegram @Dyson_v1_bot, wired in polling mode. Discord token pending |
| State backup | hermes-backup.timer daily at 03:30 UTC, mirrors state to the private repo Aakash-chouksey/hermes-lab |
| Desktop access | Hermes Desktop on the laptop, over an app-managed SSH connection to 20.198.3.153 |
There is no always-on web service on the VM. The desktop app spawns its own backend per connection.
Databases
| Database | Where | Used for |
|---|---|---|
Supabase kqrcogxpfdsxzyclgosm | Mumbai, pooler host aws-0-ap-south-1.pooler.supabase.com | production, 7 tenant schemas |
Supabase cingknteccavcehxdynw | Mumbai | team test database, safe to break |
Supabase katcvvfauvheglmlxiem | Sydney, aws-1-ap-southeast-2 | paused by the owner on purpose, retired from service, do not delete yet |
Dokploy Postgres dokploy-postgres | on billuu-app | Dokploy’s control plane, holds every app config and environment variable |
The application reaches Postgres through the Supabase pooler. The session pooler on port 5432 is used by pg_dump and long jobs, because pg_dump cannot work through a transaction pooler. The application uses the transaction pooler on port 6543. The running backend was observed pointing at aws-0-ap-south-1.pooler.supabase.com:5432.
Supabase’s free tier has no point-in-time recovery and a database size cap. Both are recorded as live risks, not solved problems.
DNS
DNS lives on Cloudflare, zone billuu.com, since 2026-09-13. Nameservers ada.ns.cloudflare.com and tate.ns.cloudflare.com are set at Namecheap with Custom DNS. The zone holds 13 records.
| Type | Name | Content | Proxy |
|---|---|---|---|
| A | testing, dokploy, main | 20.235.64.99 | testing and dokploy proxied, main not |
| CNAME | billuu.com, www | billu-brewwlabs.netlify.app | DNS only |
| CNAME | billing | billubybrewwlabs.netlify.app | DNS only |
| CNAME | order | billuu-customer-app.netlify.app | DNS only |
| MX | billuu.com | eforward1 to eforward5.registrar-servers.com, priorities 10/10/10/15/20 | n/a |
| TXT | billuu.com | v=spf1 include:spf.efwd.registrar-servers.com ~all | n/a |
main.billuu.com is unused and kept for parity.
The four Netlify hostnames stay DNS only. Netlify issues their certificates, so proxying them breaks TLS. Never proxy them.
One Cloudflare rate limit rule covers testing.billuu.com on /api/auth at 40 requests per 10 seconds per IP, blocked for 10 seconds with a 429. The Free plan fixes the period and the mitigation timeout at 10 seconds and requires cf.colo.id in the characteristics.
TLS termination
| Layer | Terminates | Certificate |
|---|---|---|
| Cloudflare edge | testing.billuu.com, dokploy.billuu.com | Cloudflare-managed, SSL mode Full (strict) |
Traefik on billuu-app | the same two hostnames at the origin | Let’s Encrypt, HTTP challenge |
| Netlify | billuu.com, www, billing, order | Netlify-managed |
Port 80 stays open to the world on purpose. Let’s Encrypt validates the HTTP challenge from its own addresses, which are not Cloudflare edge addresses.
Renewal of the panel certificate is a known problem, because Cloudflare Access answers the challenge URL. See deploy-and-rollback.md for the fix options, and .scratch/panel-certificate-renewal/issue.md for the deadline.
Access to hosts and admin surfaces
| Path | How |
|---|---|
SSH to billuu-app | ssh codejack@20.235.64.99. Tailscale SSH to 100.86.125.42 refuses port 22 |
SSH to hermes-labs | ssh codejack@20.198.3.153, or ssh codejack@100.101.91.58 from the tailnet |
| Dokploy panel | https://dokploy.billuu.com, Cloudflare Access login, then the Dokploy password |
| Dokploy API and CLI | API key in the password manager, DOKPLOY_URL in ~/.config/billuu/infra.env |
Port 443 accepts Cloudflare edge IPs and anything on tailscale0, enforced in both DOCKER-USER and UFW by /usr/local/bin/billuu-firewall.sh. /etc/billuu/cloudflare-ips.txt holds the 21 edge ranges and /usr/local/bin/billuu-cf-refresh.sh refreshes them daily at 03:15 UTC. The DOCKER-USER hook matches inbound traffic only, -i eth0. Without that match, container egress on 443 is dropped as well and no container can reach an HTTPS service.
Monitoring and alerting
Each layer sits outside the thing it watches.
| Layer | Watches | Where |
|---|---|---|
Grafana Cloud free stack wittyspring1257 | host metrics, container and journal logs, three alert rules | https://wittyspring1257.grafana.net, folder Billuu, dashboard billuu-app overview |
| Alloy | ships from billuu-app, node metrics every 60 s, docker logs, systemd journal | pushes to prometheus-prod-43-prod-ap-south-1.grafana.net and logs-prod-028.grafana.net |
| healthchecks.io | five dead-man checks | email plus Telegram |
| UptimeRobot | four external HTTP monitors every 5 minutes | testing.billuu.com, billuu.com, order.billuu.com, billing.billuu.com |
| Host heartbeat | billuu-heartbeat.sh every 5 minutes, pings HC_HOST_URL | healthchecks.io |
| Firewall guard | billuu-firewall.sh --guard every 5 minutes, re-applies after drift | /var/log/billuu-firewall.log |
Grafana Cloud rules in group billuu-app are memory above 85 percent for 5 minutes, disk above 80 percent for 10 minutes, and a metrics-pipeline-silent rule built on absent(node_load1{instance="billuu-app"}) for 5 minutes. Contact points billuu-email and billuu-telegram receive every alert through the root policy, with a 30 second group wait and a 4 hour repeat.
The single chat for machine alerts is the Telegram group billuu-alerts. Its bots are @Billuu_monitor_bot for Grafana and @HealthchecksBot for healthchecks.io.
Backups
Every backup job runs on billuu-app, reads /etc/billuu/backup.env as root with mode 600, and copies to the Cloudflare R2 bucket billuu-pg-backup.
| Schedule (UTC) | Job | Script | Output |
|---|---|---|---|
| 19:30 daily | production database dump | billuu-db-backup.sh | r2:billuu-pg-backup/db/ |
| 19:45 daily | Dokploy database dump | billuu-dokploy-backup.sh | r2:billuu-pg-backup/dokploy/ |
| 19:50 daily | host config and Redis volume | billuu-volume-backup.sh | r2:billuu-pg-backup/volumes/ |
| 20:30 Sunday | restore verification of both dumps | billuu-db-verify.sh | log line with counts, alert on failure |
| 03:15 daily | Cloudflare edge IP refresh | billuu-cf-refresh.sh | host firewall |
| every 5 minutes | host heartbeat | billuu-heartbeat.sh | healthchecks.io |
Retention is 30 days offsite and 3 days locally under /var/backups/billuu/, /var/backups/dokploy/, and /var/backups/volumes/. Each job pings its own healthchecks.io check and pings /fail on error, so failure and silence both alert.
Recovery point is the last 01:00 IST dump, so at most one day of data loss. Recovery time is about 20 to 30 minutes. A restore is always manual and follows docs/runbooks/database-restore.md.
billuu-volume-backup.sh tars /etc/dokploy and the Redis volume. The Dokploy Postgres data directory is not tarred, because a live data directory makes a corrupt snapshot. It is captured as a logical dump instead.
Retired infrastructure
Do not resurrect any of these. They are kept in the repo as history only.
| Retired | Provider | State |
|---|---|---|
dokploy-prod, billuu-db | Linode | offline since July and August 2026, instances pending deletion in the Linode console, local API token revoked so the CLI is console-only |
billuu-app, billuu-monitoring, hermes-labs | GCP project billuu-brewwlabs | terminated 2026-09-12 when the billing account closed, compute commands refuse to run, project pending rescue or purge |
| Prometheus, Loki, Grafana, promtail, exporters | GCP billuu-monitoring | retired with the self-hosted monitoring VM, files kept under monitoring/ for reference, do not deploy them |
| Render Key Value instance, Oregon | Render | unused since Redis moved in as a Dokploy resource, cancellation pending |
Supabase project katcvvfauvheglmlxiem | Supabase, Sydney | paused by the owner, keep paused until the data copy is confirmed |
Addresses that belonged to retired hosts and must not be used:
| Address | Belonged to |
|---|---|
34.93.144.23 | the GCP static IP that testing and dokploy pointed at |
100.88.24.82, 34.180.12.153 | the GCP hermes-labs VM, replaced by 100.101.91.58 and 20.198.3.153 |
dokploy.3dpnow.com | the Linode-era Dokploy panel domain, DNS removed |
application id xN9BDsHL2daVdVhr5cNXY | the backend application on the decommissioned GCP Dokploy instance |
Known gaps
| Gap | Where it is tracked |
|---|---|
Certificate expiry is not monitored for dokploy.billuu.com or testing.billuu.com | .scratch/alerting-baseline/issue.md |
| The 5xx and timeout rule waits on Traefik access logs | .scratch/traefik-access-logs/issue.md |
Tailscale SSH to billuu-app refuses port 22 over the tailnet | CONTEXT.md, known issues |
| Azure patch windows and maintenance configuration are not set yet | .scratch/azure-patch-window/issue.md |
| No staging environment exists, so there is one deployed backend and one shared test database | .scratch/staging-environment-split/issue.md |
| Redis is reproducible only from the Dokploy UI, not from a file in the repo | .scratch/infra-docs-refresh/issue.md |
Sources
Infrastructure repo /home/codejack/code/linode:
CONTEXT.mdREADME.mdAGENTS.mddocs/roadmap.mddocs/runbooks/build-from-scratch.mddocs/runbooks/database-restore.mddocs/runbooks/hermes-labs.mdhost/README.mdhost/backup.env.examplehost/scripts/billuu-db-backup.shhost/scripts/billuu-volume-backup.shhost/systemd/billuu-firewall.serviceazure/cloud-init.yamlazure/hermes-cloud-init.yamlazure/infra.env.examplemonitoring/README.mdmonitoring/alloy/config.alloy.scratch/alerting-baseline/issue.md.scratch/azure-patch-window/issue.md.scratch/decommission-render-redis/issue.md.scratch/dokploy-cli-reauth/issue.md.scratch/dokploy-ui-domain-conflict/issue.md.scratch/dokploy-volume-backups/issue.md.scratch/gcp-project-decommission/issue.md.scratch/infra-docs-refresh/issue.md.scratch/linode-decommission/issue.md.scratch/panel-certificate-renewal/issue.md.scratch/postgres-region-migration/issue.md.scratch/production-restore/issue.md.scratch/stabilize-branch-deploy-blockers/issue.md.scratch/staging-environment-split/issue.md.scratch/traefik-access-logs/issue.mddocs/superpowers/plans/2026-09-13-billuu-cost-and-resilience.md