fsync is the only thing between you and data loss, and it is slower than you think

An order service returns 201. The write is logged, the metric is incremented, the customer sees a confirmation page. Forty seconds later the host loses power. When it comes back, the order does not exist.

Nothing failed. No exception was thrown, no error was logged, no alert fired. The application called write(), the call returned success, and the application was entitled to believe it. The data was sitting in memory owned by the kernel and it evaporated with everything else in memory.

This is the part of durability that gets skipped, because the syscall that lies to you returns zero.

Four places your data can be, and only one of them is safe

There is a ladder between your variable and the physical medium, and each rung has different survival properties.

flowchart TB
    APP["Application buffer<br/>(language runtime)"] -->|"flush()"| PAGE["Kernel page cache<br/>(volatile RAM)"]
    PAGE -->|"fsync()"| DEV["Device write cache<br/>(volatile, on the drive)"]
    DEV -->|"FLUSH / FUA"| MEDIA["Persistent media<br/>(NAND or platter)"]

    style APP stroke:#ef4444,stroke-width:2px,color:#fff
    style PAGE stroke:#ef4444,stroke-width:2px,color:#fff
    style DEV stroke:#f59e0b,stroke-width:2px,color:#fff
    style MEDIA stroke:#4ade80,stroke-width:3px,color:#fff

In Python the ladder is three lines, and the middle one fools people constantly.

import os

with open("orders.log", "ab") as f:
    f.write(record)           # Python's buffer. Nothing has left the process.
    f.flush()                 # Buffer to kernel page cache. Still volatile.
    os.fsync(f.fileno())      # Page cache to device. Now it is durable.

f.flush() is not a durability operation. It pushes bytes out of the language runtime’s buffer into the kernel, which means the data now survives your process being killed. It does not survive the machine losing power, because the kernel page cache is RAM.

I have seen flush() called in a finally block and treated as the durability story more than once. It is a reasonable mistake. The method is named flush, the data does leave your process, and every test involving a process restart passes.

That first column is exactly why this bug reaches production. Every failure a team can easily simulate lands in the rows where write() is sufficient. You cannot test the bottom two rows by killing a process, and almost nobody has a way to cut power to a machine in CI.

Also worth knowing: os.fdatasync() exists and skips flushing metadata when the file size has not changed. For an append-only log where the size does change every write, it buys you less than you would hope. For in-place updates to a preallocated file, which is how most database WALs are written, it is a real saving.

The cost is that fsync is a barrier

The thing that makes fsync expensive is not the bytes. A WAL record is a few hundred bytes and moving it is nothing. The expense is that fsync does not return until the device confirms, which means the calling thread stops.

Rough orders of magnitude, and I would encourage measuring rather than trusting any table including this one:

• local NVMe, tens of microseconds to a few hundred
• SATA SSD, a few hundred microseconds
• spinning disk, 5ms to 10ms, dominated by rotational latency
• network attached block storage such as EBS, often around 0.5ms to 2ms, because the durability boundary is now a network round trip

That last line is the one that changed the calculus for everyone and got almost no attention when it did. On a cloud instance with network attached storage, fsync is a network operation. Your durability guarantee has the latency characteristics of a remote call, and it sits in the commit path of every transaction.

The arithmetic that follows is brutal and completely mechanical. A system that calls fsync once per commit, serially, has a commit ceiling of one divided by the fsync latency.

fsync latency 1ms   ->  ~1,000 commits/sec, per serial stream
fsync latency 5ms   ->  ~200 commits/sec
fsync latency 0.1ms ->  ~10,000 commits/sec

CPU count does not move this number. Query optimisation does not move it. Adding application instances does not move it, because they all queue behind the same device. You can have a database that is 3 percent busy and cannot accept another commit, which is the same confusing shape as a connection pool saturating while the database sits idle: the resource that is exhausted is not the one the dashboard is watching.

Group commit is how every real database escapes the ceiling

If the barrier costs the same whether it protects one transaction or two hundred, then batching is not an optimisation, it is the only way the numbers work.

sequenceDiagram
    participant T1 as Txn A
    participant T2 as Txn B
    participant T3 as Txn C
    participant WAL as WAL buffer
    participant DISK as Device

    T1->>WAL: append commit record
    T2->>WAL: append commit record
    T3->>WAL: append commit record
    WAL->>DISK: single fsync covering A, B, C
    DISK-->>WAL: durable
    WAL-->>T1: commit acknowledged
    WAL-->>T2: commit acknowledged
    WAL-->>T3: commit acknowledged
    Note over WAL,DISK: One barrier, three commits.<br/>Throughput scales, latency does not improve.

Postgres exposes this through commit_delay and commit_siblings, which tell a committing backend to pause briefly if other transactions are already in flight, so their commit records ride along in the same flush. MySQL has binlog group commit with its own delay setting. Both default to conservative values, and both reward measurement under real concurrency rather than a single threaded benchmark.

The trade is worth stating plainly, because it inverts the usual intuition. Group commit makes individual commits slightly slower on purpose. Under concurrency it multiplies throughput. A benchmark with one client will show group commit doing nothing or hurting, which is precisely the benchmark most people run.

The two knobs that get discussed as one

This is the part where I get twitchy, because the two most common durability settings do fundamentally different things and get compared in the same sentence constantly.

In Postgres, synchronous_commit = off means a commit returns before its WAL record has been flushed. A crash can lose the last few hundred milliseconds of committed transactions, bounded by roughly three times wal_writer_delay. The database recovers cleanly. It is internally consistent. You lost recent work and nothing else.

In Postgres, fsync = off disables the flushes that guarantee write ordering during recovery. A crash here can leave a database that cannot be recovered at all, because recovery depends on knowing that certain writes landed before others. You are not trading durability for speed. You are trading the ability to recover.

The MySQL equivalent has three states rather than two, and the middle one is the interesting one.

Setting 2 is the one worth understanding. The data reaches the kernel page cache at commit, so mysqld crashing loses nothing. Only a host level failure costs you the last second. For a lot of workloads that is a defensible trade, and it is a very different bet from setting 0, where a plain process crash is enough to lose transactions.

Every one of these is a legitimate choice. What is not legitimate is picking one without being able to say which failure it stops protecting you against.

Replication is a different failure domain, not a faster fsync

Here is where teams get into trouble in distributed systems, and it took me a while to see the shape of it.

fsync protects data against one machine dying. Replication protects data against one machine being lost. Those sound like the same sentence and they are not, because they cover different failure domains and neither one subsumes the other.

Kafka is the clearest example. Setting acks=all with min.insync.replicas=2 means the record reached the page cache of two or more brokers before the producer was acknowledged. No broker necessarily called fsync. Kafka made this choice deliberately: replication across brokers is cheaper than a disk barrier and covers the common failure, which is one broker crashing.

The gap is correlated failure. Three brokers in the same rack losing power at once lose whatever sat in three page caches, and acks=all was satisfied for all of it. Rack aware replica placement matters for exactly this reason, and it is a placement decision rather than a durability setting. If you genuinely need per-record disk durability, flush.messages=1 exists and it will cost you most of your throughput.

Postgres offers the mirror image with synchronous_standby_names, where a commit waits for a standby to acknowledge the WAL rather than waiting for local disk. You have swapped disk latency for network latency, and gained failure independence you cannot get from any local flush. Whether that is a win depends entirely on whether your network round trip beats your device, which on network attached storage it sometimes does.

The point I keep coming back to is that this is the same trade appearing at a different layer. Exactly-once delivery collapses into at-least-once plus idempotency once you look at the physics, and durability collapses into a choice of which failure you are buying insurance against. Neither has a setting that means safe.

Testing the thing you have never tested

Graceful shutdown proves nothing. SIGKILL proves your process can die without losing data that already reached the kernel, which is a real property and not the one you are worried about.

What tests fsync is removing power, and the practical stand-ins are:

• a virtual machine destroyed at the hypervisor level mid-write, which discards guest page cache the way a power cut would
• a fault injection layer such as dm-flakey under the filesystem, which can drop writes that were never flushed
• strace -e trace=fsync,fdatasync on the running process, which answers a narrower but immediately useful question: is the thing you believe is syncing actually calling the syscall at all

That last one is worth doing today on anything you consider a system of record. Watching an application you assumed was durable never issue a single fsync is a memorable ten seconds.

Also check that your storage honours flush, because the drive sits at the bottom of the ladder and it is the one rung you cannot inspect from software. Enterprise drives with power loss protection have a capacitor that lets them acknowledge a flush honestly while the data is still in their cache. Consumer drives historically did not, and some of them reported success on flush commands they had not completed. Cloud block storage generally handles this correctly, which is one of the few places where the managed service genuinely removes a class of problem rather than relocating it.

The uncomfortable summary is that durability is not a property your code has. It is a property of a chain that runs through your language runtime, the kernel, a filesystem, a block layer, possibly a network, and a physical device, and every link in it has a cache and an opinion about when to tell you it is done. fsync is where you get to interrupt that chain and demand an honest answer, and the reason it is slow is that an honest answer takes time.