Scaling Beyond Shared Staging: Building On-Demand Ephemeral Test Environments with Docker Compose

Picture this: it's Friday afternoon in your favorite co-working space in Barcelona. You've finally finished the feature you've been chasing all week. You push a branch, open a PR, and try to deploy it to staging — only to find your teammate is already there, seeding test data for a demo. Twenty minutes later, your build is queued, the database is in some weird half-migrated state, and your coffee is cold. Sound familiar?

Welcome to the most universal pain in modern software delivery: the shared staging environment. It worked when your team had three engineers and one product. It does not scale when you're shipping ten merges a day across services in Madrid, Valencia, and Malaga. Today we're leveling up. We're going to replace shared staging with on-demand ephemeral test environments — full stacks that spin up per pull request, live exactly as long as the PR does, and disappear cleanly when the work is merged.

By the end of this post you'll have a production-ready Docker Compose setup, a GitHub Actions workflow that creates per-PR environments, a teardown script that won't leak resources, and the confidence to retire your shared staging server forever.

Why Does Shared Staging Always Break When You Need It Most?

Shared staging breaks because one mutable database, one config, and one queue mean any teammate's bug can mask or amplify yours.

That's the answer in one sentence. The longer answer is that shared staging is a coordination problem dressed up as a technical one. The 2024 State of DevOps Reportfrom DORA found that elite-performing teams deploy to production multiple times per day and recover from incidents in under an hour. You cannot hit those numbers if your test environment is a single shared resource that has to be reset by hand every time someone runs a destructive migration. Meanwhile, the Stack Overflow 2024 Developer Survey reported that more than half of professional developers (about 57%) work with Docker daily — meaning the tool you need to fix this problem is almost certainly already in your toolbox.

The shared-staging anti-pattern fails in three predictable ways:

• State drift. Engineer A runs a destructive migration. Engineer B's test assumes the old schema. Both PRs "pass" locally and break each other in staging.
• Contention. Two PRs race to deploy. The losing PR's artifacts overwrite the winning PR's artifacts. Now nobody knows which build is actually live.
• Trust erosion. "Works in staging" stops meaning anything. Reviewers either click approve without testing or refuse to merge until they've booked a 30-minute slot to use staging — both terrible outcomes.

From "Staging Is Down Again" to On-Demand Environments

The leveling-up move is conceptually simple: stop treating "staging" as a place. Start treating it as a function of a pull request. When a PR opens, your CI builds a fresh stack — app, database, cache, queue, the works — gives it a unique URL, posts the link as a PR comment, and tears the whole thing down on close. Every reviewer gets a private playground. Every PR gets a clean schema. No one waits in line.

This is the same pattern that platforms like Vercel, Netlify, and Render sell as "Preview Deployments." Those are great for static sites, but they break down the moment your stack needs a real database, a Redis queue, or a background worker. The good news: you don't need a managed platform to get the same workflow. Docker Compose plus a thin GitHub Actions wrapper will give you 90% of the benefit at 10% of the cost.

Your First Ephemeral Environment: A Production-Ready docker-compose.yml

Let's start with the foundation. The trick to making one Compose file serve every PR on a single host is to parametrize everything that has to be unique per environment: ports, volume names, project name, container names. Docker Compose has a built-in mechanism for this called the COMPOSE_PROJECT_NAME variable.

# docker-compose.ephemeral.yml
# Spin up with:
#   COMPOSE_PROJECT_NAME=pr-1234 APP_PORT=18234 \
#   docker compose -f docker-compose.ephemeral.yml up -d --build

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile
    image: myapp:${GIT_SHA:-latest}
    environment:
      DATABASE_URL: postgres://app:app@db:5432/app
      REDIS_URL: redis://cache:6379/0
      APP_ENV: ephemeral
      PR_NUMBER: ${PR_NUMBER:-local}
    ports:
      - "${APP_PORT:-3000}:3000"
    depends_on:
      db:
        condition: service_healthy
      cache:
        condition: service_started
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 5s
      timeout: 3s
      retries: 20

  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: app
      POSTGRES_PASSWORD: app
      POSTGRES_DB: app
    volumes:
      - dbdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U app -d app"]
      interval: 3s
      timeout: 2s
      retries: 30

  cache:
    image: redis:7-alpine
    command: ["redis-server", "--save", "", "--appendonly", "no"]

  migrate:
    image: myapp:${GIT_SHA:-latest}
    command: ["npm", "run", "migrate:deploy"]
    environment:
      DATABASE_URL: postgres://app:app@db:5432/app
    depends_on:
      db:
        condition: service_healthy
    restart: "no"

volumes:
  dbdata:

A few things to notice. First, the app port is parameterized so multiple PR environments can coexist on a single host without colliding. Second, the migrate service is a one-shot job that runs schema migrations on startup — it depends on the database being healthy and exits cleanly when done. Third, healthchecks gate the dependency graph: the app won't start until the database is genuinely accepting connections, not just "the process exists." That single change eliminates the most common class of CI flakiness in Compose-based setups.

Wiring It Into CI: GitHub Actions Spinning Up Per-PR Environments

Now the orchestration layer. We want a GitHub Actions workflow that listens to PR events, provisions an environment on a Docker host, and posts the URL as a comment so reviewers can click straight into the running stack. We'll assume you have a single VM (in DigitalOcean, Hetzner, or Fly.io) reachable over SSH that runs the actual containers — the cheapest viable ephemeral platform.

# .github/workflows/ephemeral-env.yml
name: Ephemeral PR Environment

on:
  pull_request:
    types: [opened, synchronize, reopened, closed]

concurrency:
  group: ephemeral-pr-${{ github.event.pull_request.number }}
  cancel-in-progress: true

jobs:
  deploy:
    if: github.event.action != 'closed'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Compute environment metadata
        id: meta
        run: |
          PR=${{ github.event.pull_request.number }}
          PORT=$((10000 + PR % 50000))
          echo "pr=$PR"            >> "$GITHUB_OUTPUT"
          echo "port=$PORT"        >> "$GITHUB_OUTPUT"
          echo "project=pr-$PR"    >> "$GITHUB_OUTPUT"
          echo "url=https://pr-$PR.preview.example.com" >> "$GITHUB_OUTPUT"

      - name: Set up SSH
        run: |
          mkdir -p ~/.ssh && chmod 700 ~/.ssh
          echo "${{ secrets.PREVIEW_HOST_KEY }}" > ~/.ssh/id_ed25519
          chmod 600 ~/.ssh/id_ed25519
          ssh-keyscan ${{ secrets.PREVIEW_HOST }} >> ~/.ssh/known_hosts

      - name: Sync code to preview host
        run: |
          rsync -az --delete \
            --exclude='.git' --exclude='node_modules' \
            ./ deploy@${{ secrets.PREVIEW_HOST }}:/srv/previews/${{ steps.meta.outputs.project }}/

      - name: Deploy stack
        run: |
          ssh deploy@${{ secrets.PREVIEW_HOST }} bash -s <<EOF
            set -euo pipefail
            cd /srv/previews/${{ steps.meta.outputs.project }}
            export COMPOSE_PROJECT_NAME=${{ steps.meta.outputs.project }}
            export APP_PORT=${{ steps.meta.outputs.port }}
            export PR_NUMBER=${{ steps.meta.outputs.pr }}
            export GIT_SHA=${{ github.event.pull_request.head.sha }}
            docker compose -f docker-compose.ephemeral.yml pull --quiet || true
            docker compose -f docker-compose.ephemeral.yml up -d --build --wait
          EOF

      - name: Post preview URL
        uses: actions/github-script@v7
        with:
          script: |
            const url = "${{ steps.meta.outputs.url }}";
            const body = `Ephemeral environment ready: ${url}\n\nFresh DB, fresh cache, isolated state. Tear-down happens automatically on PR close.`;
            await github.rest.issues.createComment({
              issue_number: ${{ steps.meta.outputs.pr }},
              owner: context.repo.owner,
              repo: context.repo.repo,
              body,
            });

Two important details. The concurrency block cancels any in-flight deployment for the same PR — if you push twice in 30 seconds, only the latest commit's environment exists. Without this, your Docker host ends up running multiple zombie builds for the same PR and burning CPU. The deterministic port allocation (10000 + PR % 50000) guarantees the same PR always gets the same port, which makes URLs stable across pushes — important for reviewers bookmarking a preview.

The Cleanup Problem: Tearing Environments Down Without Leaks

Here's the boring truth nobody talks about in the "preview environments" hype: spinning up is the easy half. Cleanup is where teams quietly leak disk, ports, and database volumes for months until someone notices the host is at 100% disk on a Saturday morning. The cleanup layer needs two mechanisms — a webhook-driven teardown for the happy path, and a janitor cron for the inevitable cases where the webhook fails.

#!/usr/bin/env bash
# /srv/previews/cleanup.sh
# Run on the preview host, e.g. every 30 minutes via systemd timer.
# - Tears down environments for closed PRs (driven by GitHub list).
# - Forces shutdown of any environment older than MAX_AGE_HOURS.

set -euo pipefail

PREVIEW_DIR="/srv/previews"
MAX_AGE_HOURS="${MAX_AGE_HOURS:-24}"
GH_REPO="${GH_REPO:-acme/myapp}"

# Get list of open PR numbers from GitHub.
open_prs=$(gh pr list --repo "$GH_REPO" --state open --json number \
  --jq '.[].number' | sort -n | tr '\n' ' ')

shopt -s nullglob
for dir in "$PREVIEW_DIR"/pr-*/; do
  project=$(basename "$dir")
  pr_num="${project#pr-}"

  # Reason 1: PR is closed → tear down.
  if ! grep -qw "$pr_num" <<< "$open_prs"; then
    echo "Tearing down $project (PR closed)"
    (cd "$dir" && \
      COMPOSE_PROJECT_NAME="$project" \
      docker compose -f docker-compose.ephemeral.yml down -v --remove-orphans)
    rm -rf "$dir"
    continue
  fi

  # Reason 2: environment older than MAX_AGE_HOURS → tear down.
  age_hours=$(( ( $(date +%s) - $(stat -c %Y "$dir") ) / 3600 ))
  if (( age_hours > MAX_AGE_HOURS )); then
    echo "Tearing down $project (idle for $age_hours h)"
    (cd "$dir" && \
      COMPOSE_PROJECT_NAME="$project" \
      docker compose -f docker-compose.ephemeral.yml down -v --remove-orphans)
    rm -rf "$dir"
  fi
done

# Final sweep: prune dangling images & volumes weekly.
if [[ "$(date +%u)" == "7" ]]; then
  docker image prune -af --filter "until=168h"
  docker volume prune -f
fi

The -v flag on docker compose down is non-negotiable — without it, named volumes (like the Postgres data directory) stick around forever. --remove-orphanssweeps containers from removed services, which matters when you rename or delete services between PRs. The double-loop check (closed PR or too old) means you survive both the normal close-PR happy path and the silent failures where the GitHub webhook never fires.

Shared Staging vs Ephemeral: A Side-by-Side Comparison

If you're still on the fence about whether the engineering effort pays for itself, this comparison should help:

How Do You Handle Database State in Ephemeral Environments?

Seed every environment from a tiny fixture or sanitized snapshot. Migrations run on container start, giving each PR a known schema.

That short answer is the whole strategy in one breath. The longer version: keep a tiny seeds/dev.sql file (a few hundred rows of representative data) checked into the repo. Run it as the last step of your migrate service. If your team needs richer data, store a sanitized weekly snapshot in object storage and have the migrate service download it and replay. Critically: never let an ephemeral environment touch production data directly. The whole point of isolation is undermined the moment a leaked test job writes to the prod replica.

Troubleshooting and Debugging

When ephemeral environments misbehave, the symptoms are almost always one of these five categories. Here's how to diagnose each in under five minutes.

• Environment never becomes healthy. Run docker compose -p pr-1234 ps and look for a service in unhealthystate. Then docker compose -p pr-1234 logs --tail=200 app. Nine times out of ten the app booted before the database accepted connections — fix the healthcheck on db.
• Port already in use. Two PRs hashed to the same port, or a previous teardown failed. Check with ss -tlnp | grep :PORT, then run a targeted docker compose -p pr-OLD down -v.
• Migrations hang. Almost always a missing depends_on with condition: service_healthy. Without it, migrate races the database and silently retries forever.
• Disk filling up. docker system df will tell you whether images, volumes, or logs are the culprit. Add a weekly docker image prune -af to your janitor cron.
• Slow build times. Cache aggressively. Mount the BuildKit cache via --cache-from pointing at your registry, and order your Dockerfile so dependency installs come before source copies. A 4-minute build can drop to 40 seconds with just those two changes.

Edge Cases and Gotchas

These are the things that will bite you between week 2 and week 8 of running this in production. File them away now and save yourself the late-night debugging sessions.

• Cookie domain collisions. If all your previews share a wildcard domain like *.preview.example.com, a session cookie scoped to .preview.example.comleaks across PRs. Always scope cookies to the exact subdomain.
• Background jobs. Workers in one preview can pick up queue messages from another if they share a Redis. Always namespace queue keys with the project name.
• Reverse proxy timeouts. If you front your previews with Nginx or Caddy, set generous timeouts during cold-start (60s+). Compose pulls and migrations on first request can push past the default 30 seconds.
• Webhook lifecycle. Stripe, Slack, and similar services don't know about your previews. Use a signed query parameter (?pr=1234) on a single test webhook URL and route inside your app rather than registering a webhook per PR.
• Image bloat. Every PR that builds and tags an image creates a new registry artifact. Without lifecycle policies, registries balloon. Set a 14-day retention on PR-tagged images.
• DNS propagation. If you use a wildcard A record, you're fine. If you create a DNS record per PR, expect 30–120 seconds of resolution lag. Reviewers will think the environment is broken.

Where to Go From Here

You now have everything you need to retire shared staging: a Compose file, a CI workflow, a cleanup script, and a list of gotchas. Start small. Pick one service. Get it running on a single preview host. Watch how reviewers behave when they can click a link instead of waiting for a slot. The first time you see two reviewers happily working two different PRs at the same time on a Friday afternoon, you'll know you've leveled up.

From here, the next moves are platform-shaped: add automatic seed data refreshes, wire in Cypress or Playwright runs against each preview, expose your previews to designers and PMs for async review. Each of these is a small additional layer on the foundation you just built. None of them require Kubernetes, a managed PaaS, or a six-figure platform budget. They just require the willingness to treat your test environment as a function of your code, not a place your code visits.

Ship the first one this week. Your future self — and every reviewer in Barcelona, Madrid, Valencia, and Malaga who'll never have to wait their turn at staging again — will thank you.