Last updated on

The preview database that was still production


Part 4 of 8 in the series Running a production database, carefully. Full series ↓

I thought I'd already shipped Neon preview branching. I'd merged the PR that wired it up, flipped the RUN_PREVIEW_MIGRATIONS flag, and written a fail-closed guard so a preview could never migrate the wrong database. On paper the feature was done.

It wasn't. Not one preview deploy had ever actually branched. The feature was "launched" the way a rocket is launched while it's still bolted to the pad. Everything in place, nothing off the ground. This is the story of finding that out, proving the fix, and the two small things that quietly refused to work along the way.

The feature that never ran

The tell was in the build logs. My last preview deploys predated the merge that added the diagnostic, so they couldn't have exercised the new path. But the louder signal was in the environment variables. In Vercel, DATABASE_URL was a single static value, set 23 days earlier, identical on both Production and Preview. Every POSTGRES_* and PG* var matched it.

That is exactly the fingerprint of the base Neon storage integration: it injects one production connection string into every environment. Branch-per-preview is supposed to override DATABASE_URL at deploy time with a throwaway branch. A static prod value sitting on Preview doesn't prove branching is off, but it doesn't prove it's on either. And with no preview build to inspect, I had a feature I couldn't observe. (Getting a connection string's name wrong has bitten this project in the other direction too: calling a pulled-down prod string DATABASE_URI would have let a local production build read prod. That's pulling production data into local dev safely.

So the whole job came down to one question: is the integration actually creating a branch, or is every preview quietly talking to production? That second possibility wasn't hypothetical. It had already bitten this project twice, as Trap #1 in when two migrations collide and as the most expensive lesson in adding drafts was the easy part — back when the only defense was migrating production first and being careful.

Trap #1: the toggle that isn't where you look

The setting that controls this, Create Database Branch For Deployment → Preview, was off. Finding where to turn it on took longer than turning it on.

The Neon Console has an Integrations page for the Vercel connection. It offers exactly one button: "Manage Neon subscription," which links back to Vercel. Vercel's own integration page has a "Connect a Project" dialog with the checkbox I wanted, but my project was already connected, so it sat greyed out and unclickable. Neon points at Vercel, Vercel's dialog won't let you edit an existing connection, and you go in circles convinced you have to tear the whole thing down and reconnect.

You don't. The escape hatch is a row menu that's easy to miss: Vercel → Storage → your database → Projects → the ⋮ menu → Update Project Connection. That opens the same configuration in an editable state, checkbox and all. Tick Preview, leave Production unchecked (production still migrates by hand, on purpose), leave the variable prefix blank so DATABASE_URL keeps its name, and save. No reconnect, no downtime.

For the managed integration this is the one true home of the switch. It is not in the Neon Console, and it is not in Vercel's environment-variables screen. It's behind a three-dot menu one level deeper than anywhere you'd think to look.

The guard that made testing boring

Before opening anything against production, I went back and read the guard I'd written earlier, because I was about to lean on it hard. On a preview build, vercel-build.sh does this:

if [[ -z "${DATABASE_URL:-}" ]]; then
  echo "preview migrate ABORTED — no DATABASE_URL (Neon branch) present." >&2
  exit 1
fi
preview_host="${DATABASE_URL#*@}"; preview_host="${preview_host%%/*}"
if [[ "$preview_host" == "$prod_host" ]]; then
  echo "preview migrate ABORTED — resolved DB host is PRODUCTION." >&2
  exit 1
fi
pnpm payload migrate < /dev/null

It runs payload migrate only when a DATABASE_URL exists and its host is not production. If branching were still off, the preview would fall back to the production host and the build would fail loudly instead of migrating prod from a pull request. That single comparison turns a scary test into a boring one: the worst case is a red build, never a corrupted database.

Which meant I could just open a throwaway PR and read the result.

Proving it

I wrote a deliberately disposable migration — create a _preview_branch_probe table, drop it in the down migration, touching nothing the app knows about — and opened a PR whose only purpose was to make a preview run payload migrate.

The build log answered every question at once:

vercel-build[diag]: DATABASE_URI host -> ep-soft-boat-...neon.tech (production)
vercel-build[diag]: DATABASE_URL host -> (a different host)
vercel-build: preview deploy — applying pending migrations to the branch DB...
INFO: Migrating: 20260809_231416_preview_branch_probe
INFO: Migrated: 20260809_231416_preview_branch_probe (50ms)

Green build. The guard let the migration run, which it only does when the host differs from production. And in the Neon console a new branch had appeared: preview/verify-preview-branching, a child of main, using zero extra storage because it's copy-on-write.

The detail I liked most was what didn't happen. Only the probe migration was pending on that branch. An older migration already sat applied, which is only true if the branch was cloned from a production that already had it. The isolation wasn't a mock. It was a real, current copy of prod that my PR could scribble on freely.

Here's the whole decision, since it fits in one small picture:

Trap #2: cleanup that piles up

One win uncovered the next problem. The managed integration creates a branch named preview/<git-branch> per pull request, but it only deletes that branch when Vercel garbage-collects the underlying deployment — and Vercel keeps preview deployments for six months by default. Left alone, every PR leaves a database branch lying around for half a year.

The fix is a small GitHub Action that deletes the branch the moment the PR closes:

on:
  pull_request:
    types: [closed]
jobs:
  delete-branch:
    runs-on: ubuntu-latest
    steps:
      - uses: neondatabase/delete-branch-action@v3
        with:
          project_id: ${{ vars.NEON_PROJECT_ID }}
          branch: preview/${{ github.head_ref }}
          api_key: ${{ secrets.NEON_API_KEY }}

Because the branch name is derived deterministically from the git branch, preview/${{ github.head_ref }} always names the right one. closed covers both merges and abandoned PRs, so nothing leaks. And if Vercel's own retention fires later, the branch is already gone and the delete no-ops.

Simple. It also didn't work the first time.

Trap #3: the cleanup that lied

I merged the workflow, closed the throwaway PR, and watched. Nothing got deleted. Two separate reasons, and both are worth knowing.

The first was in the run log:

env:
  NEON_API_KEY:
...
ERROR: Cannot run interactive auth in CI

The API key was empty. I'd added NEON_PROJECT_ID as a repository variable but never added NEON_API_KEY as a repository secret. GitHub has several places to put secrets (Dependabot, Codespaces, Environments) that an Actions workflow can't see. The value has to land specifically as a repository Actions secret. The GitHub API made it unambiguous: total_count: 0. Once the secret was actually there, a re-run found the branch and deleted it.

The second reason was subtler, and it's the one I'll remember. Closing the pull request didn't trigger the workflow at all — there was no failed run to look at, because no run had started. For a pull_request event, GitHub uses the workflow file as it exists on the PR's own ref. My throwaway branch had been created before the cleanup workflow existed, so as far as that PR was concerned, the workflow wasn't there. The fix was almost silly: reopen the PR and close it again. Reopening recomputes the PR against the current main, which now contains the workflow, and this time closing it fired the job and deleted the branch cleanly.

Every future PR branches off a main that already has the workflow, so this only bites branches that predate it. Good to know it's a one-time hazard and not a standing one.

What I'd tell past-me

  • "The flag is on" is not "the feature ran." If you can't point at a log line showing the new path executing, assume it never has.
  • A static DATABASE_URL on Preview is a smell, not a verdict. Branch-per-preview overrides it at deploy time; the only proof is a build log or a branch in the console.
  • The managed integration's real switch lives behind ⋮ → Update Project Connection. Not in Neon, not in the env-vars screen. Save yourself the reconnect panic.
  • Write the fail-closed guard first. It's the difference between testing against production nervously and testing against it not caring.
  • A workflow only runs for PRs whose branch already contains it. Merging it to main doesn't retroactively arm the PRs that predate it. Reopen-and-close, or clean those up by hand once.
  • GitHub has more than one secrets drawer. If a job says it has no key, check that the secret is a repository Actions secret and not sitting in the wrong one.

The feature works now, and it cleans up after itself. But the part I actually enjoyed was the shape of the whole thing: a preview that couldn't touch production even when I'd misconfigured it, so that verifying the happy path never risked the real one. That's what the guardrail really bought me: I could go looking for problems in production's plumbing knowing I couldn't cause one.

That same fail-closed habit runs through the rest of this database's plumbing. The daily backup endpoint returns 401 rather than run without its secret, and it refuses to write anything at all if it can't encrypt it. That's the backups for a database I can't run pg_dump against.

Comments

No comments yet. Be the first to comment.

Leave a comment