ct: Fix seen/apply window divergence
In the ctp_stm::fence_epoch do not demote write lock to read lock. Old behavior: if the epoch advances the seen window the write lock is acquired first. This means that all in-flight requests are committed. The problem is that after demotion the produce request that advances the seen window may still be scheduled concurrently with a request that carries previous epoch. There is a very slight room for them to be reordered and it was never observed but it is still possible. The fix is to avoid demoting the lock from write to read. The fence_epoch now returns a fence that carries a write lock if the request advances the "seen" window. This may potentially create a performance regression. To avoid this the second part of this commit (frontend.cc) releases units early. The units are released when the placeholder is enqueued into Raft. At this point the reordering of placeholders becomes impossible and it is safe to release the fence. This allows us to use full write lock in the fence if needed. The units are only being held up until the placeholder is queued which is a low latency in-memory operation that doesn't require any I/O. The final bit of this PR addresses the failed replication problem. Currently, if the placeholder replication fails the seen and apply windows diverge. When the fence is acquired the seen window is advanced before the placeholder is replicated and applied to the in-memory state of the STM. After that the background fiber of the ctp_stm applies the placehodler to the in-memory state and advances the applied window. When the replication of the placholder fails we can't roll-back the changes to the seen window because the placeholder could still be replicated and applied. The replication failure doesn't invalidate all queued placeholders after the failed one. So, because of that we may or may not end up in the incorrect state. To avoid this problem the frontend.cc triggers leadership step down after a failed replication. This only happens if the placeholder that moved seen window failed to replicate. Signed-off-by: Evgeny Lazin <4lazin@gmail.com>
E
Evgeny Lazin committed
1969fe95b42c27d1193a3d68341dbe9ac333b4aa
Parent: 918b445