A copy-on-write system that copied everything

The problem

pgoverlay gives you a Postgres database per pull request. You seed one dataset once, and every branch after that is supposed to be an OverlayFS copy-on-write view of it: one read-only base shared by every branch, a writable layer holding only what that branch changes, thrown away without ceremony. The entire value proposition is the copying that does not happen.

So the number that mattered was not the 61.9 seconds. It was the writable layer. After creating a branch of a 5.00 GiB database (no queries run, no rows written, nothing but a boot), the branch's writable layer held 5.05 GiB. The 1 GiB dataset behaved the same way: 1.00 GiB of database, 1.05 GiB of writable layer. Whatever else was happening, the system had made a full physical copy of the data on a code path whose whole purpose is not to.

The explanation I expected, and why it was wrong

pgoverlay seeds with pg_basebackup, so a branch's first boot is a crash recovery: Postgres replays WAL to bring the data directory to a consistent state. That is a real, known cost, and it was the obvious suspect: a slow, I/O-heavy recovery pass that touches everything.

The Postgres log refused to support it. The recovery it was blamed on reported:

redo done ... elapsed: 0.00 s

Zero. The replay was not slow; the replay was nothing. Whatever was burning a minute and materialising five gigabytes happened before the work I had assumed was responsible, and I had been reading the wrong side of the log.

The diagnosis

Before replaying any WAL, Postgres runs a pass called SyncDataDirectory. Its job is to guarantee that nothing left over from before the crash is sitting dirty in the page cache unflushed. Under the default recovery_init_sync_method=fsync, it does that the literal way: it walks the entire data directory and opens every single file read-write in order to fsync it.

On an ordinary filesystem that is cheap (an open, an fsync, a close, mostly against already-clean files). On OverlayFS it is not, because of one rule: opening a file that lives in the read-only lower layer with O_RDWR forces a copy-up. Not of the pages you touch. Of the whole file, immediately, on open, before you have written a byte.

Those two behaviours compose into the exact opposite of the product. The sync pass opens every file in the dataset read-write; OverlayFS faithfully copies each one into the branch's writable layer; and by the time Postgres is ready to accept its first connection it has duplicated the entire database in the name of preparing for a WAL replay that had nothing to replay.

Proving it rather than believing it

That story is plausible, which is the problem with it. Two measurements were cheap enough that there was no excuse for shipping the explanation without them.

• The size is not approximately the database; it is the database. 5.05 GiB of writable layer against 5.00 GiB of data, and 1.05 against 1.00. A partial or incidental copy does not land on the dataset size twice at two different scales. A full copy-up of every file does.
• A single-variable control run. Same image, same dataset, same command, one flag changed: recovery_init_sync_method=syncfs. The branch came up with 16 KiB in its writable layer. If the mechanism were anything other than the sync method's file-open behaviour, changing only the sync method could not have moved the number by five orders of magnitude.

The fix

One flag, in the container entrypoint that starts a branch's Postgres:

exec docker-entrypoint.sh postgres -c recovery_init_sync_method=syncfs

syncfs() flushes the entire filesystem containing the data directory in one call, without opening individual files. The durability argument is that this is a superset of what the per-file pass covers. The property the pass exists to establish still holds, so crash-recovery semantics are unchanged. What it costs is specificity: syncfs can surface I/O errors belonging to unrelated files on the same filesystem. In this design a branch's filesystem is its own disposable overlay mount holding nothing else, so that class of noise has nothing to be noisy about.

It is not free. syncfs as a recovery init method is Linux-only and Postgres 14 or newer, which puts a hard floor under the project. Postgres 13 is therefore rejected when you register a source, loudly, rather than silently accepting it and quietly performing badly.

What changed, and how it was measured

Roughly 33× faster to create, and the storage a branch costs at rest drops by more than two orders of magnitude. That storage number is the one that decides whether you can afford a database per pull request.

What is still wrong with this

Two other things I would rather say than have someone find. The safety argument for syncfs is an argument: it reasons from what the two sync methods guarantee, and it has not been demonstrated with fault injection. And nothing in CI would catch this regression coming back. The fix is protected by a benchmark script that a human has to remember to run, which is exactly the kind of protection that fails silently a year later.

The repository states outright how much of its commit history was agent-assisted, and points at the two documents (the benchmark record and the deep dives) that required diagnosis rather than generation. The benchmark file still contains the pre-fix numbers that contradicted the project's own thesis, including the one above that looks better before the fix than after it.