Deploy and rollback

Source: g2labs-in/billuu-docs@a1403f9 · content/platform/deploy-and-rollback.md · updated 2026-09-26

How a change reaches production, how to prove it landed, and what the infrastructure repo records about undoing one. The repo at /home/codejack/code/linode is the source of truth.

Deploys here are manual. Nothing in the repo records an automatic trigger for the backend or the public sites.

What deploys what

Component	Platform	Source	Trigger
Backend API	Dokploy on `billuu-app`	`g2labs-in/brewlabs-billu-backend`, branch `main`	manual, from the panel or the Dokploy CLI. `autoDeploy` is off
Super admin panel	Cloudflare Worker `billuu-admin`	`g2labs-in/brewlabs-billu-admin`, branch `main`	`npm run deploy:cf`, or a push to `main` through `.github/workflows/deploy.yml`
Public sites	Netlify	not recorded in the infra repo	not recorded in the infra repo
Host scripts, cron files, units	`billuu-app` filesystem	`host/` in the infra repo	manual copy with `scp` and `install`
Lab agent and its bootstrap	`hermes-labs`	`azure/hermes-cloud-init.yaml`, `azure/hermes-SOUL.md`	manual, per `docs/runbooks/hermes-labs.md`

Backend deploy path

The backend runs as the Docker Swarm service billuu-backend-poxd1a on port 8000. Dokploy builds the image on the host from the Git branch, then Traefik routes testing.billuu.com to it.

1. Read the running environment set first

Dokploy replaces the whole environment set on every save. Read the live values before touching anything, and keep a copy of the working set.

ssh codejack@20.235.64.99 'CID=$(docker ps --filter name=billuu-backend-poxd1a -q | head -1); \
  docker inspect "$CID" --format "{{range .Config.Env}}{{println .}}{{end}}"'

2. Deploy

Through the panel, open https://dokploy.billuu.com, pick billuu_backend, press Deploy. The panel requires a Cloudflare Access login first.

Through the CLI:

source ~/.config/billuu/infra.env
dokploy application deploy --applicationId 2iDk-vtF1XibpA2a8SLcZ --json

The application id is 2iDk-vtF1XibpA2a8SLcZ. Older ids in the repo belong to the decommissioned GCP Dokploy instance.

3. Watch the build

Build logs land in /etc/dokploy/logs/billuu-backend-poxd1a/ on the host.

The build type is nixpacks. The Dockerfile on this branch still copies the pre-src/ directory layout, so a Docker build fails on paths that no longer exist. nixpacks is the workaround, and replacing it needs the fix in .scratch/stabilize-branch-deploy-blockers/issue.md.

Keep the Dokploy publish directory empty. When it is set, Dokploy runs a second static build that can overwrite the application image with an nginx one.

4. Startup guards that decide the deploy

The application refuses to start quietly. A missing value shows up as exit code 1 rather than a warning.

Guard	Effect
`REDIS_ENABLED=true`	without it, `createRedisConnection()` returns a mock client to BullMQ and the process exits 1 during startup
`DATA_FIRST_LENIENT=true`	set on the running host to get past the migration-discipline guard. The code comment marks it “NOT recommended”, and `.scratch/stabilize-branch-deploy-blockers/issue.md` tracks removing it
`DATABASE_URL`, `CONTROL_PLANE_DATABASE_URL`	point at the Supabase Mumbai pooler. A wrong or paused project fails startup

Dokploy CLI notes

Symptom	Cause and workaround
`one` or `validate` endpoints return 400	the CLI runs 0.29.4 against server 0.30.6, so several single-item endpoints are confirmed broken. Use `all` endpoints or the panel
`save-environment` wipes unrelated variables	it replaces the whole set. Send the complete set, read first
`save-environment` rejects the call	add `--buildArgs "" --buildSecrets "" --createEnvFile`
`docker stack deploy` ignores a `.env` file	stack deploy does not read `.env` files, unlike `docker compose`

Super admin panel deploy path

The panel is a static bundle served by a Cloudflare Worker with an assets binding and no script. It is live at superadmin.billuu.com.

npm run build
npm run deploy:cf        # wrangler deploy, needs CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID

The token needs Workers Scripts edit and DNS edit for the billuu.com zone. The one in ~/.config/billuu/infra.env has both, and does not carry Access permissions, so the Cloudflare Access application is managed in the dashboard rather than through the API.

.github/workflows/deploy.yml builds and deploys on a push to main. Its CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID repository secrets were set on 2026-09-26, and its trigger was corrected at the same time: it had named the panel’s old branch, themeing-and-Updated-code-2026-07-01, which was correct while main held an unrelated history, and which no longer exists now that the history was moved aside. No run has executed yet, so the job is configured but unproven, and npm run deploy:cf remains the way to deploy by hand.

The hostname is not a free choice. refreshTokens reads the refresh token from req.cookies?.refreshToken and nowhere else, and the login response sets it HttpOnly; Secure; SameSite=Strict. SameSite is a site rule, not an origin rule, so a panel on a *.workers.dev name would never receive that cookie and every session would die after 24 hours with no way back. superadmin.billuu.com is same-site with the API on testing.billuu.com, which is why it is the host.

Rollback is a redeploy of the previous Worker version, listed under Deployments in the Cloudflare dashboard. Worker versions are immutable and retained, so this is exact rather than a rebuild.

Verifying one is not a curl. The panel sits behind a Cloudflare Access application, so an automated request gets a 302 to the Access login and a 200 from /health tells you nothing about the panel:

curl -s -o /dev/null -w '%{http_code}\n' https://superadmin.billuu.com/    # 302 to the Access login is correct

A 302 says the gate is up, not that the panel works. The real check is a person clicking through, or a local panel pointed at the deployed API, which the backend’s CORS allowlist already permits on http://localhost:5174.

Frontend deploy path

The public sites are Netlify-hosted and served over four hostnames.

Hostname	Netlify site	Purpose
`billuu.com`, `www`	`billu-brewwlabs.netlify.app`	marketing site
`billing.billuu.com`	`billubybrewwlabs.netlify.app`	billing portal
`order.billuu.com`	`billuu-customer-app.netlify.app`	customer ordering app, QR codes point here

The DNS records for all four stay DNS only. Netlify issues their certificates, so proxying them breaks TLS. Never proxy them.

The infra repo records no build command, branch, or trigger for these sites, and no rollback procedure. A frontend release is therefore not something this repo can currently describe or undo. The three public hostnames are monitored from outside by UptimeRobot at a five minute interval, which is the only recorded signal that a frontend deploy landed.

The landing page billuu_landingPage is not deployed. Cloudflare Pages is its intended home, per CONTEXT.md and docs/superpowers/plans/2026-09-13-azure-migration-and-cloudflare-split.md.

Deploying host files

Scripts, cron entries, units, and logrotate rules live in host/ in the infra repo and are copied to the host by hand. Edit in the repo first, never on the host alone.

scp host/scripts/billuu-firewall.sh codejack@100.86.125.42:/tmp/
ssh codejack@100.86.125.42 'sudo install -m 755 -o root -g root /tmp/billuu-firewall.sh /usr/local/bin/'

Here	On the host	Mode
`host/scripts/*.sh`	`/usr/local/bin/`	755, root owned
`host/cron/billuu-*`	`/etc/cron.d/`	644, root owned
`host/systemd/billuu-firewall.service`	`/etc/systemd/system/`, then `systemctl daemon-reload && systemctl enable --now billuu-firewall`	as installed
`host/logrotate/billuu-*`	`/etc/logrotate.d/`	as installed

Note that host/README.md uses the tailnet address in its copy commands while AGENTS.md and CONTEXT.md say Tailscale SSH to billuu-app refuses port 22. Use the public IP 20.235.64.99 and treat the tailnet command as pending.

Verify a deploy

Check	Command	Expected
Health endpoint	`curl -s https://testing.billuu.com/health`	`{"success":true,"data":{"db":"ok","redis":"ok","environment":"production", ...}}`
Services on the host	`ssh codejack@20.235.64.99 'docker service ls'`	`billuu-backend-poxd1a` and `billuu-redis-engwdt` at 1/1
Application logs	`ssh codejack@20.235.64.99 'sudo docker service logs --tail 200 billuu-backend-poxd1a'`	no restart loop, no guard warning
Build log	`/etc/dokploy/logs/billuu-backend-poxd1a/` on the host	the new build finished and produced the image
Metrics and logs	dashboard `billuu-app overview` in the `Billuu` folder, `https://wittyspring1257.grafana.net`	host metrics continue, no alert firing
Logs in Grafana	Explore, datasource `grafanacloud-wittyspring1257-logs`, `{job="docker"}` or `{job="systemd-journal"}`	the new lines are present
From outside the estate	UptimeRobot monitors	`testing.billuu.com` still green

A green /health is not the end of a deploy that changes behavior. .scratch/production-smoke-test/issue.md holds the full checklist, and it needs a real outlet. It covers web login, POS login, menu load, order create and edit, receipt print, offline mode, the customer QR flow, reports, and panel access.

Rollback

The infrastructure repo records no rollback runbook for a deploy, and no rollback drill has been performed. That gap is written down in .scratch/infra-docs-refresh/issue.md and listed as unmet acceptance criteria in .scratch/staging-environment-split/issue.md.

What the repo does document, and what each lever covers:

Situation	Lever	Where it is recorded
Bad code on the branch	The deploy is a Git build, so the deployed image follows the branch content. Revert the change on `stabilize-phase-1-2` and deploy again	`CONTEXT.md` describes the Git build. No procedure or deployment-history rollback is written down
Bad environment change	Send the previous full env set with `save-environment` and redeploy. This is why the running set is read and kept first	`README.md`
Broken Dokploy state	Restore the Dokploy database dump. It holds the deploy settings and the environment variables	`docs/runbooks/database-restore.md`
Bad database change	Restore into a new Supabase project, then repoint `DATABASE_URL` and `CONTROL_PLANE_DATABASE_URL` in Dokploy and redeploy	`docs/runbooks/database-restore.md`
Host config regression	`/etc/dokploy` is tarred nightly to `r2:billuu-pg-backup/volumes/`. The repo copy under `host/` is the source of truth for the scripts themselves	`host/scripts/billuu-volume-backup.sh`, `host/README.md`
Host-level cutover	Rollback is a DNS revert. The migration plan keeps the old record for a day	`docs/superpowers/plans/2026-09-13-billuu-cost-and-resilience.md`
Frontend	No rollback procedure is recorded	`docs.manifest.yaml` lists no frontend runbook, and `.scratch/infra-docs-refresh/issue.md` records the missing rollback runbook

Restores are always manual and never run over a live database. The verified procedure is docs/runbooks/database-restore.md.

Known issues

Certificate renewal for the panel is broken by Cloudflare Access

dokploy.billuu.com sits behind a Cloudflare Access application. Let’s Encrypt validates its HTTP challenge at /.well-known/acme-challenge/..., Access answers with a login page instead, and the renewal fails. SSL mode is Full (strict), so the panel breaks when the certificate expires.

Date	Event
2026-11-12	the renewal attempt lands, about 30 days before expiry
2026-11-01	the deadline to finish this work
2026-12-12	the panel certificate expires

Two fixes are recorded in .scratch/panel-certificate-renewal/issue.md:

Install a Cloudflare Origin Certificate on the host as the Traefik default certificate. It is valid for 15 years and removes renewal entirely. Files go to /etc/dokploy/traefik/dynamic/certs/ with a TLS store beside them, written as the container sees those paths.
Add an Access application policy for dokploy.billuu.com with action bypass and the path selector /.well-known/acme-challenge/*. Smaller change, but the renewal stays a thing to watch every 60 days.

The same trap applies to any hostname that gets an Access application later, including testing.billuu.com. Certificate expiry is not monitored yet, which is recorded in .scratch/alerting-baseline/issue.md.

Recorded verification for either option:

echo | openssl s_client -connect 104.21.72.183:443 -servername dokploy.billuu.com 2>/dev/null \
  | openssl x509 -noout -issuer -dates

Stale workflows and config that no longer reflect reality

Artifact	What it says	Why it is stale
`brewlabs-billu-backend/.github/workflows/deploy.yml`	on a push to `main` or `master`, build an image, push it to GHCR, then SSH to `$PROD_HOST` and run `docker compose pull` and `docker compose up -d` in `/opt/brewwlabs-pos`	production is deployed by Dokploy from the `stabilize-phase-1-2` branch, and no file in the infra repo records that host, path, or registry. The workflow is present on `stabilize-phase-1-2` and absent from `main`
`/home/codejack/code/opencode/billuu/.github/workflows/deploy.yml`	build and push backend and frontend images to Docker Hub on a push to `main`	it has no deploy step, and the workspace root is not a Git repository, so nothing triggers it
`brewlabs-billu-frontend/vercel.json`	rewrite `/api/` to `https://pos-backend-6afs.onrender.com/api/`	the host appears nowhere in the live estate. The Render resource the infra repo records is a Key Value instance pending cancellation
`brewlabs-billu-frontend/.env.example`	`VITE_API_BASE_URL=https://billuu.3dpnow.com/api`, with a commented line naming another Render host	`3dpnow.com` is the Linode-era domain the infra repo records as dead

Treat all four as history, not as a deploy path. The deployed backend path is the Git branch plus Dokploy.

Deploy blockers recorded on the branch

Blocker	Detail
Stale Dockerfile paths	the branch moved code into `src/` but the `Dockerfile` still copies top-level `config`, `models`, and `routes`. `docker build` fails with `"/app/config": not found`. `nixpacks` is the live workaround
Migration discipline guard	the Data-First guard reports model changes without migrations and `app.js` exits 1 in production. The live host sets `DATA_FIRST_LENIENT=true`, which the code comment marks “NOT recommended”

Both are tracked in .scratch/stabilize-branch-deploy-blockers/issue.md.

Changes that can take HTTPS away from every container

Any edit to the port 443 filter must keep the DOCKER-USER hook matching inbound traffic only, with -i eth0. Without that match, the rule also catches container egress on 443 and the containers lose HTTPS to Razorpay, Google, and GitHub. This broke one deploy and is the first thing to check when outbound calls stop working.

Runbooks

The infra repo holds three runbooks. The docs site projects them under platform/runbooks/ per billuu-docs/docs.manifest.yaml.

Runbook	Covers
`docs/runbooks/build-from-scratch.md`	an empty cloud account to a verified production, including the trap list
`docs/runbooks/database-restore.md`	the backup set, the dead-man hooks, and the restore procedure
`docs/runbooks/hermes-labs.md`	the lab host, rebuild, state backup, and recovery

Two runbooks named in .scratch/infra-docs-refresh/issue.md do not exist yet, roll back a deploy and restart Redis.

Sources

Infrastructure repo /home/codejack/code/linode:

CONTEXT.md
README.md
AGENTS.md
docs/roadmap.md
docs/runbooks/build-from-scratch.md
docs/runbooks/database-restore.md
host/README.md
host/backup.env.example
host/scripts/billuu-db-backup.sh
host/scripts/billuu-volume-backup.sh
host/systemd/billuu-firewall.service
host/cron/billuu-db-verify
azure/infra.env.example
.scratch/alerting-baseline/issue.md
.scratch/infra-docs-refresh/issue.md
.scratch/panel-certificate-renewal/issue.md
.scratch/production-restore/issue.md
.scratch/production-smoke-test/issue.md
.scratch/stabilize-branch-deploy-blockers/issue.md
.scratch/staging-environment-split/issue.md
.scratch/traefik-access-logs/issue.md
docs/superpowers/plans/2026-09-13-azure-migration-and-cloudflare-split.md
docs/superpowers/plans/2026-09-13-billuu-cost-and-resilience.md

Application repos:

brewlabs-billu-backend/.github/workflows/deploy.yml
brewlabs-billu-frontend/vercel.json
brewlabs-billu-frontend/.env.example
billuu/.github/workflows/deploy.yml

Docs site:

billuu-docs/docs.manifest.yaml