Dev Tools · 1h ago
Redis Clock Drift Causes Duplicate Job Execution
A nightly billing-reconciliation job ran twice on two workers despite using a Redis lock with a 30-second TTL. Worker A's local clock was 9 seconds behind Redis's clock due to a silent NTP failure, causing the lock to expire mid-job. Worker B then acquired the lock and started the same job, leading to duplicate processing.
Meridian48 take
The fix—auto-renewing locks instead of relying on fixed TTLs—is well-known but often skipped; this incident shows how subtle clock drift can break distributed locks in production.
Read the full reporting
Two Workers Both Held the 'Only One' Lock. The Clock Was Never Synced. →
DEV Community
redis-locksclock-drift