Last updated on
The Cache That Didn't Know the Database Changed
Filed under Operations
The problem
A couple of maintenance scripts on this site fix published content directly: things like repairing a broken series membership without going through the admin UI. They write straight to the database through the CMS's server-side API, skipping the deployed app entirely.
The pages themselves rely on on-demand cache invalidation. When a post or series changes, a hook fires that tells the framework to regenerate the affected page, but that call only works inside a live request: a page load, a server action, a webhook. A standalone script has no such context. The hook still fires, but the call has nothing to act on. It doesn't throw or log anything, it just quietly does nothing.
That gap showed up during a real fix. I ran one of these scripts against production to repair a series whose post list had drifted out of sync. The database write succeeded immediately, confirmed by a follow-up query, but the live page kept showing the old, broken state. It only caught up once I opened the same document in the admin UI and hit save, a real request where the invalidation call actually worked. The affected pages had no time-based fallback either, so without that manual save, the page would have stayed wrong indefinitely.
The solution
The scripts still write directly to the database; that's still the right tool for a batch content fix. What changed is that they now trigger the invalidation from the one place it actually works: the live server.
After a successful write, each script calls a new authenticated endpoint on the deployed site whose only job is running that same invalidation, this time from inside a real request. A few choices were deliberate:
- The credential travels as a request header, not a URL parameter, so it can't end up in server logs. It gets its own dedicated secret rather than reusing an existing one, since a script-triggered call and a routine scheduled job are different levels of access.
- The endpoint only accepts a fixed allowlist of paths, so a leaked credential can't be used to force-refresh arbitrary pages, and it caps how many paths one request can refresh, so it can't be used to trigger excessive work in a single call.
- If the call fails or the credential is missing, the script only logs a warning. It never fails the underlying fix, since the database write already succeeded by that point. A timeout keeps a hung connection from blocking the script indefinitely.
As a backstop, the affected pages also refresh themselves automatically every hour, regardless of whether the on-demand call ran. A missed or failed revalidation now self-heals within the hour instead of staying wrong forever.
What we learned
- An incident that quietly resolves itself is still worth a writeup. The symptom went away once I happened to re-save through the admin UI. The actual defect, a structural gap in how these scripts interact with caching, was still sitting there for the next person to trip over.
- A try/catch can hide a structural gap, not just a flaky error. The invalidation call wasn't unreliable, it was being called from a context where it fundamentally can't do anything. Swallowing the exception made both cases look identical from the outside.
- A credential's boundary should match who's using it, not what's convenient to reuse. Reusing an existing secret would have worked mechanically. Giving the script-triggered path its own credential cost a few extra lines of configuration in exchange for keeping two different kinds of access separate.
- Hardening a feature right after the happy path works is worth the extra pass. The path cap and the timeout weren't in the original plan. Both came from asking, right after the first version worked, what the worst misuse of it could look like.
Comments
No comments yet. Be the first to comment.