1. The Danger of In-Memory Cron in Clustered Environments
Why node-cron without distributed synchronization triggers catastrophic duplicate jobs
When deploying a Node.js web application to ECS, Kubernetes, or Auto-Scaling EC2 groups, multiple replica pods run in parallel. If an in-memory scheduler like `node-cron` triggers a job (e.g. sending daily digest emails or scraping job listings), every single container will execute the job simultaneously, causing duplicate billing, email spam, and database race conditions.
To safely run scheduled tasks in distributed environments, you must either use an external event dispatcher (AWS EventBridge / Cloud Tasks) or synchronize execution across instances using a distributed lock pattern.
2. Implementing Distributed Locking with Redis (Redlock)
Acquiring atomic execution tokens across clustered worker nodes
Using Redis atomic commands (`SET key value NX PX timeout`), only the first container to request the lock will receive authorization to run the scheduled task. Other instances immediately skip execution.
import cron from "node-cron";
import { Redis } from "ioredis";
import { runEngineeringJobScraper } from "../scrapers/jobScraper";
const redis = new Redis(process.env.REDIS_URL || "redis://localhost:6379");
export function initDistributedCron() {
// Schedule to run every 2 hours
cron.schedule("0 */2 * * *", async () => {
const lockKey = "locks:cron:engineering-scraper";
const lockTtlMs = 1000 * 60 * 30; // 30 minutes lease
const workerInstanceId = `worker-${process.env.HOSTNAME || Math.random().toString(36).substring(7)}`;
// SET key value NX PX: Atomic acquisition
const acquired = await redis.set(lockKey, workerInstanceId, "PX", lockTtlMs, "NX");
if (!acquired) {
console.log(`[${workerInstanceId}] Scraper job is already running on another node. Skipping.`);
return;
}
console.log(`[${workerInstanceId}] Lock acquired. Executing scheduled scraper...`);
const startTime = Date.now();
try {
const results = await runEngineeringJobScraper();
console.log(`[${workerInstanceId}] Successfully scraped ${results.totalJobs} jobs in ${Date.now() - startTime}ms`);
} catch (error) {
console.error(`[${workerInstanceId}] Scraper error:`, error);
// Optional: alert PagerDuty / Sentry
} finally {
// Release lock safely with Lua script to verify lock ownership
const releaseScript = `
if redis.call("get", KEYS[1]) == ARGV[1] then
return redis.call("del", KEYS[1])
else
return 0
end
`;
await redis.eval(releaseScript, 1, lockKey, workerInstanceId);
console.log(`[${workerInstanceId}] Lock released.`);
}
});
}3. Scaling from Simple Cron to Resilient BullMQ Job Queues
Handling exponential backoffs, concurrency limits, and worker pools
When tasks require processing thousands of sub-items (e.g. resizing images, generating clinical PDFs, dispatching FCM notifications), running them in a single monolithic cron loop risks timeouts.
Decomposing the batch into distinct jobs managed by BullMQ allows workers to process jobs concurrently with automatic retry logic and dead-letter queue isolation.
import { Queue, Worker, QueueEvents } from "bullmq";
import { redisConnection } from "../lib/redis";
import { sendPushNotification } from "../services/fcm";
export const notificationQueue = new Queue("push-notifications", {
connection: redisConnection,
defaultJobOptions: {
attempts: 4,
backoff: {
type: "exponential",
delay: 2000,
},
removeOnComplete: 1000,
removeOnFail: 5000,
},
});
// Dedicated background worker process
export const notificationWorker = new Worker(
"push-notifications",
async (job) => {
const { userId, title, body, metadata } = job.data;
await sendPushNotification(userId, { title, body, metadata });
},
{
connection: redisConnection,
concurrency: 25, // Process 25 notifications concurrently
}
);
