Principal engineer · distributed systems · architecture case studies
A control plane spanning three regions with automatic promotion of the write leader. The interesting constraint was that promotion had to be safe under a network partition where each side could still see a majority of its own region.
Raft with a modified pre-vote phase to stop a partitioned node from disrupting a healthy term on rejoin. Deployed across 40 nodes.
Sharded by tenant with a work-stealing balancer. Median enqueue-to-start 8ms at peak; the tail is dominated by cold executors rather than the scheduler.