Every replicated database has replication lag. Most engineers don't fully understand what happens when reads hit a stale replica, until production teaches them.
Replication lag is the gap between when a write is committed on the primary and when that write is visible on a replica. In a perfect world, this gap is zero. In the real world, it ranges from a few milliseconds to several seconds, and in bad situations (replica catching up after a restart, primary overloaded, large transactions) it can stretch to minutes. Most systems silently serve stale reads from replicas and never tell users. Here's what's actually happening inside the database engine.
In asynchronous replication (the default for MySQL, PostgreSQL, MongoDB, Cassandra, and most others), the primary accepts a write and acknowledges it to the client immediately. It then sends the write to replicas as a background task. The replica applies the write to its own storage. Until that application completes, anyone reading from the replica will see the old value. The primary doesn't wait for the replica before acking the write. That would make every write as slow as the slowest replica, which is unacceptable. This is the fundamental source of replication lag.
Imagine a user updates their profile photo. The write goes to the primary. The UI immediately redirects to the profile page, which reads from a replica, and the user sees their old photo. They refresh, still old. They are furious. This is the 'read-your-writes' consistency problem: you wrote something, but a subsequent read doesn't reflect it. The fix is routing: after any write, send that user's subsequent reads to the primary for a short window (or use a logical clock/timestamp to ensure the replica is caught up). Instagram uses this pattern extensively; after you post a photo, your feed reads are temporarily routed to the primary so you always see your own post immediately.
A subtler problem: a user reads from replica A and sees a message. They refresh. The request hits replica B, which is further behind, and the message is gone. From the user's perspective, data vanished. This is a violation of monotonic read consistency. The fix is session affinity: pin each user's reads to the same replica for the duration of their session. Consistent hashing on user_id to replica is the standard approach. The downside is uneven replica load when users with high read rates happen to hash to the same replica.
The hardest version of the problem: user A writes a comment. User A then writes a reply to that comment. User B reads the reply but not the original comment, because the reply landed on a replica that has caught up past the reply's timestamp but the original comment landed on a replica that is behind on a different shard. Causal consistency requires that if you see a write's effect, you also see all the writes that causally preceded it. Most systems don't guarantee this. Systems that do (MongoDB multi-document transactions, Google Spanner, CockroachDB) use vector clocks or hybrid logical clocks to track causal dependencies across nodes.
You can eliminate replication lag entirely with synchronous replication: the primary doesn't ack the write until at least one (semi-sync) or all (fully-sync) replicas have applied it. This gives you zero lag and strong read-after-write consistency. The cost: write latency is now bounded by the slowest replica in the replication group. In a multi-region setup, that means a write in Virginia that must wait for a replica in Singapore adds hundreds of milliseconds of latency to every write. Google Spanner accepts this cost in exchange for global strong consistency. Most applications don't need Spanner-level consistency and pay through the nose if they try to get it from a conventional RDBMS.
Most applications need read-your-writes consistency for the user who performed the write, and eventual consistency for everyone else. That's achievable cheaply: route the writing user's subsequent reads to the primary for 5-10 seconds after a write. For social feeds and timelines, eventual consistency is fine. If you see a post from 5 seconds ago instead of 3 seconds ago, nobody notices. Only reach for synchronous replication when you have a genuine consistency requirement that can't be met by application-level routing, and when you do, measure the latency impact before committing.