Clone My Server

A field guide to one bare-metal box that runs everything: dev environments with automatic public subdomains, git-push production deploys, a monitoring stack that pages an AI agent, and a terminal workspace where a fleet of coding agents builds software in parallel. Everything below is reproducible; snippets are sanitized copies of what actually runs.

0Philosophy

Three ideas hold this whole setup together:

• One box, no PaaS. A single beefy bare-metal server is cheaper and simpler than a constellation of managed services, and it gives agents a stable, fully-inspectable world to operate in. Dev and prod live side by side in two directories.
• Convention over configuration. Drop a folder in ~/workspace/ and it has a public HTTPS dev URL within a minute. Push to a bare repo and it deploys. No dashboards, no YAML ceremonies.
• Agents are first-class operators. Every terminal session runs inside a workspace manager that tracks agent state. Alerts get piped to an agent, deploys are run by an agent, and features are built by a supervised hierarchy of agents.

1The machine

A Scaleway bare-metal server (their Elastic Metal line — any dedicated-server vendor works) running stock Ubuntu 24.04 LTS. This one has an AMD EPYC 12-core / 24-thread CPU, 128 GB RAM and ~1 TB of NVMe in software RAID. That's massively overprovisioned for web serving — the headroom is for agents: a dozen concurrent coding agents with their builds, test runs and dev servers eat CPU and RAM in bursts.

Directory layout — the backbone convention

~/workspace/<project>/   # dev checkouts — every folder here gets a *.dev.<domain> URL
~/deploy/<project>/      # prod checkouts — deployed by git hooks, run by systemd
~/git/<project>.git      # bare repos — the push target that triggers deploys
~/tools/<tool>/          # small operational tools (bridges, monitors)

One regular user (member of sudo and docker) owns everything. System-level services (Caddy, BIND, monitoring containers) are the only root-adjacent things.

2Caddy: reverse proxy & automatic TLS

Caddy terminates TLS for every hostname and reverse-proxies to local ports. It's the only thing listening on 80/443. Why Caddy: automatic certificates (including on-demand issuance for wildcard subdomains, no DNS plugin needed) and a config file simple enough to be generated by a shell script — which is exactly what happens (§3).

The config has three layers:

1. Global options — ACME email and an on_demand_tls ask-endpoint (a local Caddy vhost that just answers 200, so any subdomain that reaches the box gets a cert).
2. import /etc/caddy/sites.d/*.caddyfile — hand-written vhosts for anything special (static sites, apex domains, custom headers). One file per site.
3. Two generated wildcard blocks — *.dev.<domain> and *.<domain>, rewritten every minute by devproxy-sync. Never edited by hand.

The reverse-proxy block pattern

@myapp host myapp.dev.cosmictaco.dev
handle @myapp {
    reverse_proxy localhost:5100 {
        flush_interval -1          # stream immediately — REQUIRED for SSE / AI token streams
        transport http {
            dial_timeout 2s        # dead backend fails fast instead of hanging 30s
        }
    }
}

Static site pattern (sites.d)

# /etc/caddy/sites.d/landing.caddyfile
landing.cosmictaco.dev {
    root * /var/www/landing
    file_server
    encode gzip
}

Unmatched subdomains fall through to a static fallback page (root * /var/www/fallback + try_files /index.html), so a typo'd or stopped project shows a friendly page instead of a connection error.

3devproxy-sync — folders become URLsthe killer feature

The single highest-leverage script on the box. A cron job scans the two project directories every minute and regenerates the wildcard Caddy blocks:

• ~/workspace/myapp/ → https://myapp.dev.cosmictaco.dev (default port 3000)
• ~/deploy/myapp/ → https://myapp.cosmictaco.dev (default port 3001)

Create a folder, and one minute later it has a public HTTPS URL. This matters more than it sounds: agents can spin up a project and hand you a shareable link with zero proxy configuration, from a phone you can check any dev server, and webhooks/OAuth callbacks work against real HTTPS URLs in dev.

Per-project port config: .devproxy.json

The script

Full sanitized copy — installed at /usr/local/bin/devproxy-sync, run by cron every minute (* * * * * /usr/local/bin/devproxy-sync). It writes to a temp file and only touches Caddy when the config actually changed, so the cron is free when nothing moved. Run it manually with sudo devproxy-sync to skip the wait.

#!/bin/bash
# devproxy-sync: generate Caddyfile for dev + prod subdomains
#
# Dev:  ~/workspace/<project> → <project>.dev.cosmictaco.dev (port 3000 default)
# Prod: ~/deploy/<project>   → <project>.cosmictaco.dev (via .devproxy.json or 3001)
#
# .devproxy.json format:
#   { "port": 3939 }                    → <project>:3939
#   { "web": 3939, "api": 4000 }        → <project>-web:3939, <project>-api:4000
#   No file                             → dev:3000, prod:3001

WORKSPACE="/home/youruser/workspace"
DEPLOYDIR="/home/youruser/deploy"
CADDYFILE="/etc/caddy/Caddyfile"
FALLBACK="/var/www/fallback"
DEV_DOMAIN="dev.cosmictaco.dev"
PROD_DOMAIN="cosmictaco.dev"
TMPFILE=$(mktemp)

# ── Helper: scan a directory and collect entries ──
scan_dir() {
    local basedir="$1"
    local default_port="$2"
    local entries=""

    for dir in "$basedir"/*/; do
        [ -d "$dir" ] || continue
        project=$(basename "$dir")
        config="$dir/.devproxy.json"

        if [ -f "$config" ]; then
            simple_port=$(grep -oP '"port"\s*:\s*\K\d+' "$config" 2>/dev/null)
            if [ -n "$simple_port" ]; then
                entries="$entries $project:$simple_port"
            else
                while IFS="=" read -r app port; do
                    app=$(echo "$app" | tr -d " \"")
                    port=$(echo "$port" | tr -d " \",}")
                    [ -z "$app" ] || [ -z "$port" ] && continue
                    entries="$entries ${project}-${app}:$port"
                done < <(grep -oP "\"[^\"]+\"\s*:\s*\d+" "$config" | sed "s/\"\([^\"]*\)\"\s*:\s*\(.*\)/\1=\2/")
            fi
        else
            entries="$entries $project:$default_port"
        fi
    done

    echo "$entries"
}

# ── Helper: write a wildcard block ──
write_block() {
    local domain="$1"
    local entries="$2"

    cat >> "$TMPFILE" << EOF

*.${domain} {
    tls {
        on_demand
    }

EOF

    if [ -n "$entries" ]; then
        for entry in $entries; do
            subdomain="${entry%%:*}"
            port="${entry##*:}"
            matcher="${subdomain//[-.]/_}_$(echo "$domain" | tr '.' '_')"
            cat >> "$TMPFILE" << EOF
    @${matcher} host ${subdomain}.${domain}
    handle @${matcher} {
        reverse_proxy localhost:${port} {
            flush_interval -1
            transport http {
                dial_timeout 2s
            }
        }
    }

EOF
        done
    fi

    cat >> "$TMPFILE" << EOF
    handle {
        root * ${FALLBACK}
        try_files /index.html
        file_server
    }

    handle_errors {
        root * ${FALLBACK}
        rewrite * /index.html
        file_server
    }
}
EOF
}

# ── Global config ──
cat > "$TMPFILE" << HEADER
{
    email you@example.com
    on_demand_tls {
        ask http://localhost:5555
    }
}

http://localhost:5555 {
    respond 200
}

import /etc/caddy/sites.d/*.caddyfile

HEADER

# ── Scan directories ──
DEV_ENTRIES=$(scan_dir "$WORKSPACE" 3000)
PROD_ENTRIES=$(scan_dir "$DEPLOYDIR" 3001)

# ── Write blocks ──
write_block "$DEV_DOMAIN" "$DEV_ENTRIES"
write_block "$PROD_DOMAIN" "$PROD_ENTRIES"

# ── Reload if changed ──
if ! diff -q "$TMPFILE" "$CADDYFILE" > /dev/null 2>&1; then
    sudo cp "$TMPFILE" "$CADDYFILE"
    if systemctl is-active --quiet caddy; then
        sudo systemctl reload caddy 2>/dev/null
    else
        sudo systemctl start caddy 2>/dev/null
    fi
fi

rm -f "$TMPFILE"

4DNS, VPN & firewall

DNS

Two wildcard A records are all the routing needs: *.dev.<domain> → server IP and *.<domain> → server IP. Any DNS host works (this setup has used a self-hosted BIND as well as managed providers — wildcards + short TTLs are the only requirement). If you self-host BIND, the box runs named as just another systemd service; keep the zone files in git.

Tailscale — the admin plane

Tailscale (WireGuard mesh VPN) connects the server, laptop and phone. Everything administrative rides the VPN: SSH, database clients, internal dashboards. The public internet only ever sees ports 80/443.

Firewall posture (UFW)

• 80/443 open to the world — Caddy only.
• SSH allowed only on the Tailscale interface. The public port 22 is redirected to an endlessh tarpit that feeds scanner bots an endless SSH banner at glacial speed. Real SSH is unreachable without the VPN.
• A high UDP range open for mosh roaming sessions.
• fail2ban as a second layer, and unattended-upgrades for security patches.

5Deploys: git push → live

Deployment is a bare git repo plus a hook. No CI service, no registry, no containers for app code.

# one-time setup per project
git init --bare ~/git/myapp.git
# then in your checkout:
git remote add deploy ~/git/myapp.git     # or user@host:git/myapp.git from a laptop
git push deploy main                      # ← this is the whole deploy interface

The post-receive hook

~/git/myapp.git/hooks/post-receive checks out the pushed commit into ~/deploy/myapp, builds, copies the env file, restarts the service, and logs everything to ~/deploy/myapp.deploy.log:

#!/bin/bash
set -euo pipefail

APP="myapp"
DEPLOY_DIR="/home/youruser/deploy/$APP"
SERVICE="$APP.service"
LOG="/home/youruser/deploy/$APP.deploy.log"

echo "=== Deploy $APP started at $(date) ===" | tee "$LOG"

# Only deploy on main branch
while read oldrev newrev refname; do
    branch=$(git rev-parse --symbolic --abbrev-ref "$refname" 2>/dev/null || echo "$refname")
    if [ "$branch" != "main" ] && [ "$refname" != "refs/heads/main" ]; then
        echo "Push to $branch — skipping deploy" | tee -a "$LOG"
        exit 0
    fi
done

mkdir -p "$DEPLOY_DIR"
GIT_WORK_TREE="$DEPLOY_DIR" git checkout -f main 2>&1 | tee -a "$LOG"
cd "$DEPLOY_DIR"

bun install --frozen-lockfile 2>&1 | tee -a "$LOG"
bun run build 2>&1 | tee -a "$LOG"

# env files live OUTSIDE the repo and are copied in at deploy time
cp "/home/youruser/secrets/$APP.env.local" "$DEPLOY_DIR/.env.local"

sudo systemctl restart "$SERVICE" 2>&1 | tee -a "$LOG"
echo "=== Deploy $APP finished at $(date) ===" | tee -a "$LOG"

systemd: one simple unit per app…

# /etc/systemd/system/myapp.service
[Unit]
Description=My App
After=network.target

[Service]
Type=simple
User=youruser
WorkingDirectory=/home/youruser/deploy/myapp
ExecStart=/home/youruser/.bun/bin/bun run start
Restart=always
RestartSec=5
KillMode=control-group
TimeoutStopSec=10
Environment=NODE_ENV=production
Environment=PORT=3001

[Install]
WantedBy=multi-user.target

…and a template unit for a monorepo fleet

The big app on this box is a Bun + Turborepo Next.js monorepo whose prod runs as a fleet of ~15 processes (web, API gateways, a dozen queue workers). One systemd template unit covers them all — %i is the package name:

# /etc/systemd/system/fleet-prod@.service
[Unit]
Description=Prod fleet — %i
After=network-online.target
Wants=network-online.target
PartOf=fleet-prod.target

[Service]
Type=simple
User=youruser
WorkingDirectory=/home/youruser/deploy/monorepo
EnvironmentFile=/home/youruser/deploy/monorepo/.env.local
Environment=PATH=/home/youruser/.bun/bin:/usr/local/bin:/usr/bin:/bin
ExecStart=/home/youruser/.bun/bin/bun run --filter=%i start
Restart=always
RestartSec=5
KillMode=control-group
TimeoutStopSec=20

[Install]
WantedBy=fleet-prod.target

sudo systemctl enable --now fleet-prod@api fleet-prod@worker-mail fleet-prod@worker-jobs
systemctl list-units 'fleet-prod@*'        # the whole fleet at a glance

The deploy-bot pane

Deploys are never fired from a background shell. A dedicated agent pane (an AI agent session named "deploy bot" in the workspace manager, §8) runs them: it pushes, tails the deploy log, curls the health endpoints, and reports. If a deploy goes sideways there's a visible, scrollable terminal with the whole story — and an agent with context already loaded to fix it. Alerts are piped into the same pane (§7).

Next evolution (in progress): blue/green — deploy to a second port, health-check, then flip the proxy target, for zero-downtime fleet restarts.

6Data services

Databases run as Docker containers with remapped host ports, one shared instance per engine for all dev work:

docker run -d --name app-postgres --restart unless-stopped \
  -p 127.0.0.1:5440:5432 \
  -e POSTGRES_PASSWORD=<YOUR_PASSWORD> \
  pgvector/pgvector:pg16

docker run -d --name app-redis --restart unless-stopped \
  -p 127.0.0.1:6390:6379 \
  redis:7-alpine

• Non-default ports (5440/6390), bound to 127.0.0.1. Nothing on the public internet, no collision with any distro-packaged postgres, and connection strings are unambiguous about which instance they mean.
• pgvector image so every project gets vector search for free — table stakes for AI apps.
• One instance, database-per-environment discipline: myapp_dev, myapp_test, myapp_prod as separate databases (separate Redis DB indexes likewise). Postgres is far cheaper per-database than per-instance; 128 GB of RAM does the rest. The one hard rule: prod and dev never share a database, and the monitoring stack (§7) touches neither.
• An occasional project that needs a different major version gets its own container on yet another remapped port.

7Monitoring & alerts — that page an agent

A small Prometheus-compatible stack runs as Docker containers on the monitoring port decade, every component bound to 127.0.0.1:

Watching the watchers

• External probes: Better Stack monitors the public prod URLs from outside — if the whole box or its network dies, the inside stack can't tell you.
• Dead-man's switch: a Watchdog alert fires forever by design and is routed to a Better Stack heartbeat URL. The external service alerts when the heartbeat stops — covering "monitoring itself is down", the failure mode all self-hosted monitoring shares.

The alert → agent bridge

The interesting part. A small script polls Alertmanager and injects newly-firing alerts into the deploy-bot agent pane together with a runbook prompt. The AI agent — which already has the deploy context, log locations and health-check commands in its head — investigates autonomously: snapshot forensics first, then remediation (restart the wedged unit, verify health), then an incident report. Two design rules learned the hard way:

• The bridge is a dumb pipe. It never restarts anything itself; all judgment (forensics before restart, deploy windows, confirmation) stays with the agent that has context.
• Busy-guard: never inject text while the pane's agent is mid-turn — queued injections corrupt a running turn. Alerts persist in Alertmanager, so waiting for the next poll cycle loses nothing.

8herdr — the terminal workspace managerthe control plane

herdr is a terminal workspace manager built for running AI agents: workspaces → tabs → panes, like tmux, but every pane knows whether the agent inside it is idle, working, blocked or done, and the whole thing is drivable over a socket CLI. The user connects with herdr --remote from a Mac (and mobile); every agent session on the server lives in a herdr pane. That one property — agent status as a queryable API — is what makes multi-agent orchestration (§10) possible.

The CLI surface

herdr workspace list
herdr workspace create --cwd ~/workspace/myapp --label "myapp" --no-focus
herdr workspace focus <id>   |   herdr workspace close <id>

herdr pane list [--workspace <id>]      # every pane + agent_status + cwd
herdr pane read <pane_id> [--lines N]   # read a pane's screen (spy on another agent)
herdr pane run <pane_id> '<command>'    # run a command in a pane
herdr pane send-text <pane_id> '<text>' # type text (single argv, quotes-safe)
herdr pane send-keys <pane_id> Enter    # send raw keys

herdr wait output <pane_id> --match "<text>" --timeout 30000
herdr wait agent-status <pane_id> --status idle|working|blocked|done --timeout 600000

Load-bearing gotchas (all learned live)

• send-text never submits in an agent TUI — the trailing Enter is swallowed by the input box. Always follow with a separate herdr pane send-keys <pane> Enter, then poll until agent_status flips to working; if it's still idle after ~10s, resend Enter.
• Never pass a prompt as a CLI argument (claude "do the thing…") — inner quotes will eventually mangle it. Launch the agent bare, then deliver the prompt with send-text "$(cat prompt.md)".
• Detect herdr by probing the socket (herdr pane list exits 0), not via env vars — they aren't exported into every pane's shell.

9AI agent setup

Three coding agents are installed side by side, and all three report status to herdr:

Context discovery — write it once, every agent reads it

• User-global: ~/.claude/CLAUDE.md holds machine-wide facts (this whole document's content, in condensed form: devproxy, deploy system, herdr recipes, policies). Both Claude Code and omp auto-load it at session start.
• Per-repo: AGENTS.md at the repo root (with CLAUDE.md symlinked to it) holds repo-specific rules. Agents discover it by walking up from the cwd.
• Rule of thumb: machine-wide facts go in the global file, repo rules in the repo file — and nothing an agent could derive from the code itself goes in either.

Memory

Agents keep a persistent memory directory (~/.claude/…/memory/): one file per fact with a one-line description in frontmatter, plus a MEMORY.md index that gets loaded each session. Corrections the user makes once ("never default subagents to the cheap model") become memory files and stop being repeated mistakes.

Skills

Reusable procedures live as skills — ~/.claude/skills/<name>/ (user-global) and <repo>/.claude/skills/<name>/ (per-repo) — small markdown playbooks the agent loads on demand: "set up devproxy for a project", "spawn a worktree agent", "run the agent factory" (§10). The pattern that works: skills encode proven multi-step recipes with their gotchas, not general knowledge the model already has.

10The agent factorythe crown jewel

The setup above compounds into this: a pattern for building a batch of features in parallel with a three-tier hierarchy of agents, proven over multiple live runs.

dispatcher (1 session — your pane)
│   plans waves, spawns orchestrators, merges PRs, GC's worktrees
├── orchestrator A (claude pane, own git worktree + branch + herdr workspace)
│   ├── worker: omp pane        — backend legwork
│   └── worker: claude pane     — frontend
├── orchestrator B ( … )
└── orchestrator C ( … )

Tier responsibilities

• Dispatcher (the session you talk to): splits the feature batch into waves, creates a git worktree + branch + herdr workspace per feature, spawns one orchestrator pane each, watches them, squash-merges finished PRs, and deletes worktrees and workspaces afterwards. The spawner owns the cleanup, always.
• Orchestrators (one per feature, isolated worktree): plan → spawn workers → review the diff to convergence → open the PR → wait for CI green. They are deliberately denied merge rights — a permission classifier blocks pushing past branch protection. They end their run by printing MERGE-READY: #<PR> and going idle; the dispatcher does the actual merge. One agent, one branch, one PR.
• Workers: implement under the orchestrator's instructions. Workers never commit — the orchestrator owns git. That single rule eliminates a whole class of half-committed-state disasters.

The rules that make it work

• Wave-plan by serialization points. Dependency-free features run in the same wave; dependent ones wait. Crucially, identify shared mutable state: only ONE orchestrator per wave may touch the database schema (two agents running codegen on the same migration journal = guaranteed conflict). Global lint/typecheck runs once at the end, not per agent.
• Isolation via git worktrees. Each orchestrator gets .claude/worktrees/<slug> — same repo object store, independent working copy. Worktree hygiene is a hard rule: whoever creates a worktree removes it when the branch is merged (each one carries its own node_modules; twenty stale worktrees once cost 15 GB).
• Naming for observability. Every pane gets a structured title (NF <scope> <id> for orchestrators, NFW … for workers) so a phone screen full of panes stays legible.
• Sustained-idle watching. A watcher loop polls herdr pane list and only treats an orchestrator as finished after ~3 consecutive idle polls — and not if the pane still shows a running background shell (an agent waiting on a worker looks idle for one poll). Then it verifies the claim: is the PR actually open? is CI actually green? Never trust a status line alone.
• Review to convergence. Orchestrators run adversarial review passes on their workers' diffs and iterate until a pass finds nothing — before the PR is ever opened.
• Prompt delivery discipline — §8's gotchas, mechanically applied: write the prompt to a file, send-text "$(cat prompt.md)", separate send-keys Enter, poll for working.

What this buys: a batch of five features lands in an afternoon with one human reviewing PRs and making merge calls, and every intermediate step is visible as a scrollable terminal pane rather than an opaque job log.

11Quality of life

• Bun everywhere. One runtime + package manager for every JS project (bun install, bun run, Bun.serve). No npm/pnpm/yarn/node-version matrix, and it's fast enough that agents stop timing out on installs. (This very page is served by a 20-line Bun.serve script.)
• Subscription/limits monitoring. Agent subscriptions have usage windows; small monitors track remaining quota so a long factory run doesn't silently starve mid-wave — scheduling heavy waves is a resource-planning problem like any other.
• Chat → agent bridge. Same philosophy as the alert bridge (§7): a small poller watches a chat channel (self-hosted chat server) and pipes messages into a designated agent pane, with the same busy-guard. Net effect: you can message your server from your phone and an agent with full local context answers, deploys, or investigates.
• Server-rendered docs like this page: any doc worth sharing gets a folder in ~/workspace, a static Bun.serve, a .devproxy.json — and it's live on a subdomain a minute later. Eat the dogfood.