Deploying vs. Operating: What Broke After the Site Went Live

Last post ended with the site actually live — Vercel, Neon, Blob storage, real content migrated over, DNS pointed at the right place. That felt like the finish line while I was writing it. It wasn't. Deploying something once and keeping it correct while people (in this case, just me) actually use it turned out to be a different skill, with its own failure modes that a successful deploy doesn't warn you about. Three separate incidents in the days after launch made that distinction concrete rather than theoretical.

Locked out of my own CMS

The first one showed up when I tried to publish a real post: a permissions error, with no further explanation, only in production. Locally, creating a post worked without issue — same code, same collection config, same access rules. That "same code, different result" gap is usually the fastest lead you can get, because it rules out the code itself and points straight at environment or configuration.

The actual cause was a single missing environment variable: SERVER_URL, which Payload uses to verify that write requests are actually coming from the app's own origin. My config had a fallback — process.env.SERVER_URL || 'http://localhost:3000' — that was meant to make local dev work without extra setup. In production, with SERVER_URL never set in Vercel's environment variables, that fallback silently made the live app think its own origin was localhost. Every write request from the real domain then looked like a cross-origin request from somewhere Payload didn't recognize, and got rejected.

Nothing about this was a bug in the traditional sense. The code did exactly what it was written to do. The problem was a value that needed to exist in one specific place and didn't, and the failure mode it produced — a vague "no permissions" message — gave no hint that an environment variable was the actual cause. Fixed by setting SERVER_URL in Vercel's Production environment and redeploying. Worth saying plainly: this is the kind of bug that operating a live service produces and local development simply can't, because local development never has to ask "which of several possible environments is this request actually coming from."

The list that wouldn't update

Once publishing worked, the next thing I noticed was stranger: a new post would save successfully, show up correctly in the admin panel, and then simply not appear on the public blog page or the homepage's recent-posts list. Not a slow update, not a caching delay measured in seconds — it genuinely wouldn't show up, even after a hard refresh.

I'd already run into a version of this problem before launch: generateStaticParams() queries the database at build time to know which individual post pages to pre-render, which is why the database had to be migrated before a successful build was even possible (that was the "order I planned was backwards" story from the last post). What I hadn't accounted for was that the listing pages — the blog index and the homepage's "latest posts" section — have the exact same constraint, and nothing about my earlier fix touched them. Both are plain Next.js Server Components with no dynamic APIs and no revalidate export, which means Next prerenders them once at build time and Vercel serves that static output indefinitely. A post existing in the database was never the same thing as a post existing in the pre-rendered HTML, and I'd only fixed that gap for individual post pages, not for the pages that list them.

There were three honest ways to handle this, and they're a real tradeoff, not just an implementation detail. Redeploy after every edit is free but defeats the purpose of having a CMS. Time-based revalidation (export const revalidate = 300) is simple but leaves a window where you'd publish something and have no way to know if it actually went live yet. I went with the third option: an afterChange hook on the Posts collection that calls Next's revalidatePath() directly, so saving a post immediately invalidates the cached blog index, homepage, and that post's own page. Payload's hooks run inside the same Next.js app now (this is the collapsed-into-one-app architecture from earlier posts), so this is a same-process function call, not a webhook round trip. The only real wrinkle was that revalidatePath() throws when it's called outside of an actual Next.js request — which happens every time my seed script runs a batch of database writes from a standalone process — so the hook needed a try/catch around it. A revalidation failure should never be able to take down the actual save.

The afternoon a storage bucket didn't exist

The third one started small and got serious fast. Two icon images in the site footer — GitHub and LinkedIn — were broken in production. Straightforward enough: I traced it to those two files having been uploaded before the Vercel Blob storage plugin was ever configured, back when local disk was the only option, which meant their database records still pointed at a local-disk file path that doesn't exist on Vercel's ephemeral filesystem. Payload writes a file's serving URL onto its database record at upload time and doesn't recompute it later, so turning on Blob storage afterward didn't retroactively fix documents that already existed. Normal enough problem, and I had what looked like an obvious fix: re-upload the two files now that Blob storage was active.

That's where it stopped being straightforward. Replacing the file on the existing record failed with a server error. Creating a brand-new record failed the same way. Uploading a completely different file type failed identically. That pattern — every operation failing the same way regardless of what I changed about the request — meant the problem wasn't in the two files or their configuration at all. The actual error, once I pulled it from Vercel's function logs instead of the vague "something went wrong" the admin UI showed: This store does not exist. The Vercel Blob store itself — the entire bucket every uploaded file had been living in — was gone.

For a few minutes, that was a genuinely bad moment, not a minor inconvenience. Vercel doesn't version or restore deleted Blob stores, so if real content had been uploaded directly to production and existed nowhere else, it was permanently gone with no recovery path. I checked scope before doing anything else: the Media collection had exactly two documents in it, both the already-broken icon files I already had local copies of. No other content had been lost — but I want to be honest that I didn't know that going in, and the right first move when you hit "this data store doesn't exist anymore" is always to establish the actual blast radius before you do anything else, not to assume it's contained.

The fix, once scope was confirmed: create a new Blob store, update the environment variable pointing at it, redeploy so the running app actually picks up the new value, and then re-upload both icons through Payload's own update path rather than a raw file upload — specifically so Payload would regenerate and persist a correct URL against the new store, on the same document IDs, so nothing referencing them needed to change.

The actual distinction

None of these three were architecture problems. The code that broke in each case was correct, or close to it — a missing environment variable, a caching boundary I'd only partially accounted for, a piece of third-party infrastructure that disappeared out from under a running app with no warning. Getting a site deployed once is a project with a defined end state: you either have it live or you don't. Operating it is not a project, it's an ongoing condition, and the failure modes that show up there are different in kind, not just in frequency — they're about what happens when real requests hit a real environment over real time, not about whether the initial architecture was sound.

That's the actual reason I think this is worth separating from the last post rather than folding it in as more "smaller friction." A deploy retrospective and an operations retrospective are different documents, because they're testing different things. I'd rather be direct about that than pretend the deploy post already covered it.

Still owed: the post about the actual content model — the collections, how the rich text editor is wired into the frontend, the parts of the admin setup that are easier to show than describe. That one's next, genuinely, and this time there's no deploy detour left to interrupt it.