Skip to content
Billuu

Build the Billuu infrastructure from scratch

Source: g2labs-in/linode-infra@968c507 · docs/runbooks/build-from-scratch.md · updated 2026-09-14

This guide takes you from an empty cloud account to a running production, on Azure or any other provider. It repeats nothing about the why of the design. For that, read CONTEXT.md, the plans in docs/superpowers/plans/, and the ADRs.

Working time: about two hours. Nothing here is destructive to an existing environment. Numbers and IDs in this file are examples from the Azure build; substitute your own.

0. What you need before you start

ItemWhy
A domain you can change nameservers onthe whole stack hangs off it
A Cloudflare accountDNS, TLS, proxy, WAF, rate limit, Access
A Supabase accountmanaged Postgres with a pooler
Access to the GitHub org that holds the backend repoDokploy deploys from Git
An object storage account with an S3 APIoffsite backups. Cloudflare R2 is what we use
A healthchecks.io accounttells you when a backup stops running
A Tailscale accountprivate admin access with no public ports
A password managerevery secret, none of them in this repo

Write down, in the password manager, the names of the artifacts you will create: VM IP, Tailscale auth key, Dokploy API key, Cloudflare API token, R2 keys, database URLs, app secret set. Values only, never in git.

1. Decisions to make first

DecisionGuidance
Regionput the app near the database and near the users. Ours: database Mumbai, host Pune (centralindia)
VM size2 vCPU and 4 GB is the floor. The backend plus Docker plus Traefik fits, with headroom for builds
Disk64 GB. Dokploy builds images on the host, so leave room
Databasemanaged, never self-hosted, unless you enjoy on-call. Supabase or any Postgres with a pooler
Backupsobject storage in a different provider from the app host
DNS and edgeCloudflare, from day one. You will want the proxy, the WAF, and Access later

2. Cost estimate

Prices move. Check before you commit, and get approval before creating anything billable.

ItemAzure referenceTypical alternatives
VM, 2 vCPU / 4 GBStandard_B2ls_v2, about $35/moHetzner CX22 about €4, DigitalOcean 2 GB about $12, Linode 4 GB about $24
Disk, 64 GB SSDabout $5/moincluded on most providers
Static IPv4about $3.65/mofree on most providers, small charge on some
Managed PostgresSupabase free tier, or $25 for point in time recoveryNeon, RDS, Cloud SQL
DNS, proxy, WAFCloudflare Free, $0Cloudflare Free
Object storageR2 free tier, about 10 GBBackblaze B2, S3
Alertinghealthchecks.io free tierBetter Stack, UptimeRobot

3. Phase 1: create the host

Azure, with the resource group, firewall, and a static IP:

Terminal window
az group create -n billuu-prod -l centralindia


az network nsg create -g billuu-prod -n billuu-nsg -l centralindia
az network nsg rule create -g billuu-prod --nsg-name billuu-nsg -n allow-ssh \
  --priority 100 --protocol Tcp --destination-port-ranges 22 --access Allow
az network nsg rule create -g billuu-prod --nsg-name billuu-nsg -n allow-http \
  --priority 110 --protocol Tcp --destination-port-ranges 80 --access Allow
az network nsg rule create -g billuu-prod --nsg-name billuu-nsg -n allow-https \
  --priority 120 --protocol Tcp --destination-port-ranges 443 --access Allow


az network public-ip create -g billuu-prod -n billuu-app-ip -l centralindia \
  --sku Standard --allocation-method Static


envsubst '${TS_AUTHKEY}' < azure/cloud-init.yaml > /tmp/billuu-ci.yaml


az vm create -g billuu-prod -n billuu-app -l centralindia \
  --image Ubuntu2404 --size Standard_B2ls_v2 \
  --admin-username codejack --ssh-key-values ~/.ssh/id_ed25519.pub \
  --public-ip-address billuu-app-ip --nsg billuu-nsg \
  --os-disk-size-gb 64 --storage-sku StandardSSD_LRS \
  --custom-data /tmp/billuu-ci.yaml
rm -f /tmp/billuu-ci.yaml

azure/cloud-init.yaml installs Docker, Tailscale, UFW, and fail2ban, and joins the tailnet. It expects TS_AUTHKEY from your password manager.

Other providers:

ProviderCreateFirewallStatic IP
Hetznerhcloud server createcloud firewallassign a primary IP
DigitalOceandoctl compute droplet createcloud firewallreserved IP
Linodelinode-cli linodes createcloud firewallreserved IP
GCPgcloud compute instances createVPC firewall rulesreserved address

The rest of this guide assumes Ubuntu 24.04 and works the same everywhere.

Verify:

Terminal window
ssh codejack@<vm-ip> 'cloud-init status --long; docker --version; tailscale status | head -3'

4. Phase 2: hardening check

Terminal window
ssh codejack@<tailscale-ip> 'sudo ufw status | head -8; sudo fail2ban-client status sshd'
ssh codejack@<tailscale-ip> 'sudo sshd -T | grep -E "permitrootlogin|passwordauthentication"'

Expected: 22, 80, 443 open, everything on tailscale0 allowed, fail2ban active, root login no, password auth no.

Delete the Tailscale auth key now. It is single use and short lived by design.

5. Phase 3: Dokploy

Terminal window
ssh codejack@<tailscale-ip> 'curl -sSL https://dokploy.com/install.sh | sudo sh'

Open http://<tailscale-ip>:3000, create the first account, then Settings, API/CLI, Create API Key. Store it as DOKPLOY_API_KEY, with DOKPLOY_URL alongside.

For command line use, install the CLI where you work, not on the host:

Terminal window
dokploy auth -u "$DOKPLOY_URL" -t "$DOKPLOY_API_KEY"
dokploy project all --json

Create the project, the environment, and the application. Connect the GitHub org through the browser flow, pick the repository and the branch you deploy from.

6. Phase 4: the database

  1. Create a Supabase project in the region your host can reach quickly.
  2. Note two connection strings from the dashboard, both on the pooler host:
session pooler, port 5432       used by pg_dump and long jobs
transaction pooler, port 6543   used by the application at runtime
  1. Store both in the password manager. The application uses the transaction pooler, the backup script uses the session pooler, because pg_dump cannot work through a transaction pooler.
  2. Apply the schema. Either restore a dump (see docs/runbooks/database-restore.md) or let the application create its control plane tables on first boot.

Traps, all of which we hit:

  • The direct database hostname db.<ref>.supabase.co may not resolve. Use the pooler.
  • Connections require PGSSLMODE=require.
  • Supabase free tier has no point in time recovery. Nightly logical dumps are your only safety net.
  • A paused free project answers tenant/user not found on the pooler. That means paused, not lost.

7. Phase 5: Redis

Add Redis as a Dokploy resource on the same network as the application. Copy the generated password and build the URL:

REDIS_URL=redis://default:<password>@<redis-service>:6379
REDIS_ENABLED=true

REDIS_ENABLED=true is not optional. With it unset the application hands a mock client to the queue workers and the process exits with code 1 during startup.

8. Phase 6: the application

Set the environment before the first deploy. Dokploy replaces the whole set on every save, so keep a copy of what works.

Required groups:

DATABASE_URL, CONTROL_PLANE_DATABASE_URL     transaction pooler
JWT_SECRET, REFRESH_TOKEN_SECRET, ACCESS_TOKEN_SECRET, ENCRYPTION_KEY
REDIS_URL, REDIS_ENABLED=true
NODE_ENV=production, PORT=8000
FRONTEND_URL, RENDER_EXTERNAL_URL, GOOGLE_CALLBACK_URL
RAZORPAY_*, CLOUDINARY_*, WHATSAPP_*, EMAIL_*, GOOGLE_CLIENT_*
DB_SYNC=false

Add the domain, port 8000, HTTPS with Let’s Encrypt. Then deploy.

Traps:

  • Choose a build type. nixpacks builds this repository without extra work. If you use the Dockerfile, confirm it matches the current directory layout, or the build fails on missing paths.
  • If you set a publish directory, Dokploy also runs a static build and can overwrite the image with nginx. Leave it empty.
  • The application refuses to start in production when the schema guard finds models without matching migrations. DATA_FIRST_LENIENT=true starts it with a warning. Fix the migrations when you can.
  • save-environment requires --buildArgs "" --buildSecrets "" --createEnvFile with the CLI.

Verify before touching DNS:

Terminal window
ssh codejack@<vm-ip> 'CID=$(docker ps --filter name=<service> -q | head -1); \
  docker exec "$CID" node -e "require(\"http\").get(\"http://127.0.0.1:8000/health\",r=>r.pipe(process.stdout))"'

Expect {"success":true,...,"db":"ok","redis":"ok"}.

9. Phase 7: DNS and Cloudflare

  1. Add the Zone in Cloudflare, free plan. Cloudflare scans existing records.
  2. Compare the scan against the true list. Missing records cause outages, not warnings. Ours were:
A      testing, dokploy, main     -> the host's static IP
CNAME  @ and www                  -> the Netlify site
CNAME  billing                    -> the billing portal
CNAME  order                      -> the customer app
MX     apex, five records         -> the mail provider
TXT    apex, SPF                  -> the mail provider
  1. Keep the Netlify records DNS only (grey cloud). Netlify manages their own certificates.
  2. Change nameservers at the registrar to the pair Cloudflare assigns.
  3. Wait for the zone to become Active. Verify before and after:
Terminal window
curl -H 'accept: application/dns-json' \
  "https://cloudflare-dns.com/dns-query?name=testing.example.com&type=A"
  1. Set SSL/TLS mode to Full (strict). The origin already has a real certificate.
  2. Flip the API and panel records to Proxied, one at a time, testing after each.

Trap: once a hostname is proxied, any Cloudflare Access application on it answers the ACME HTTP challenge, so Let’s Encrypt renewal fails about 60 days later. Either install a Cloudflare Origin Certificate on the origin, or add an Access bypass policy for /.well-known/acme-challenge/*.

10. Phase 8: backups and alerting

Four scripts and five cron entries. Copy them from host/ in this repo, do not retype them.

billuu-db-backup.sh         nightly production dump   -> r2:.../db/
billuu-dokploy-backup.sh    nightly Dokploy dump      -> r2:.../dokploy/
billuu-volume-backup.sh     host config and Redis     -> r2:.../volumes/
billuu-db-verify.sh         weekly restore proof
billuu-cf-refresh.sh        refresh the Cloudflare edge IP list

Steps:

  1. Create a bucket, for example billuu-pg-backup.
  2. Create an R2 API token scoped to that bucket, permission Object Read & Write. Store both keys.
  3. Write /etc/billuu/backup.env, mode 600, root owned. It holds the database URL, the R2 keys, and the alert URLs.
  4. Create two healthchecks.io checks: one daily with a two hour grace, one weekly with a six hour grace. Put the ping URLs in HC_URL and HC_VERIFY_URL.
  5. Install the scripts in /usr/local/bin, the cron files in /etc/cron.d, and logrotate entries.
  6. Run each script once and read its log.

Trap: a bucket-scoped R2 token cannot HEAD or create buckets. Every rclone call needs --s3-no-check-bucket, or the upload fails with AccessDenied on CreateBucket.

Trap: the postgres image starts the server twice on first boot. A readiness check must confirm two successful queries with a gap, or a restore hits the temporary server.

11. Phase 9: protections

Panel gate. Cloudflare Zero Trust, one time setup: pick a team name, pick the free plan. Then add an Access application for the panel hostname, with an allow policy listing emails, and One-time PIN as the login method.

Origin firewall. Once the API and panel are proxied, restrict port 443 to Cloudflare edge IPs plus Tailscale. Docker publishes ports through docker-proxy, so the rule must exist in the DOCKER-USER chain as well as UFW. billuu-firewall.sh in this repo does both, and billuu-cf-refresh.sh keeps the IP list current. Keep port 80 open: Let’s Encrypt validates from its own addresses.

Rate limiting. One Cloudflare rule on /api/auth/*, 40 requests per 10 seconds per IP, block for 10 seconds. On the free plan the period and the mitigation timeout are both fixed at 10 seconds, and cf.colo.id is required in the characteristics.

Client IPs. After proxying, Traefik must trust the Cloudflare ranges, or logs and audit entries show edge IPs instead of visitors.

12. Phase 10: verification checklist

Run all of these. Each has an expected result.

[ ] ssh over Tailscale works, public SSH still works on 22
[ ] https://<api host>/health returns 200 with db ok and redis ok
[ ] a real login works from the frontend
[ ] an order can be created end to end
[ ] https://<panel host> shows the Access gate, then the Dokploy login
[ ] direct connection to the VM IP on 443 is refused or times out
[ ] all five cron entries exist and one backup ran with a green healthchecks check
[ ] the weekly verify script prints table and row counts in its log
[ ] a scratch restore of the newest dump succeeds
[ ] SSL mode is Full (strict) and the certificate chain validates

13. Rebuilding an existing environment instead of building fresh

If the old environment is gone but the backups survive, this path is faster:

  1. Create the host and install Dokploy, as above.
  2. Restore the Dokploy dump into Dokploy’s own Postgres, or paste the environment set from the password manager.
  3. Restore the production dump into the database (new Supabase project or scratch instance).
  4. Point the application at it and redeploy.
  5. Repoint DNS.

The Dokploy dump carries the deploy settings and the environment variables. That is why we back it up.

14. Traps, in one list

REDIS_ENABLED unset                   app exits 1 during startup, workers get a mock client
DATA_FIRST_LENIENT unset              production startup blocked by the migration guard
publishDirectory set to "."           Dokploy runs a static nginx build and overwrites the image
stale Dockerfile paths                build fails on COPY of directories that no longer exist
save-environment                      replaces the whole set, not a merge
pg_dump on the transaction pooler     fails or hangs. Use the session pooler, port 5432
rclone without --s3-no-check-bucket   AccessDenied on CreateBucket
postgres image first boot             temporary server, readiness checks must be patient
Access on a proxied host              Let's Encrypt renewal fails. Use an Origin Certificate
proxying Netlify records              breaks their certificates. Keep them DNS only
NSG or UFW closed on 80               certificate issuance fails
Cloudflare edge IPs change            refresh the list daily, or the origin starts rejecting traffic
DOCKER-USER hook without -i eth0      the rule also matches container egress, so every container
                                      loses HTTPS to the internet. Match inbound only

15. Where to look next

CONTEXT.md                            what is true today: hosts, DNS, firewall, backups
docs/runbooks/database-restore.md     the restore procedure, verified
docs/adr/                             why the decisions were made
docs/superpowers/plans/               the cost and migration plans
.scratch/                             every open issue, one folder each

Host files live in host/ in the repo: the scripts, the cron entries, the systemd unit, the logrotate rules, and an example config with empty values. Deploy them with scp then sudo install -m 755, never by editing on the host alone.