Cloudflare 14 ☁️ Cloudflare Queues & Workflows — Offloading background tasks with Queues/DLQs and orchestrating durable multi-step Workflows
In the traditional serverless model, execution is bound by strict timeouts and the request-response cycle. If a Worker needs to process a massive image, send a batch of emails, or coordinate a multi-day onboarding sequence, the 30-second execution limit becomes a brick wall. Cloudflare Queues and Cloudflare Workflows are designed to break these constraints by shifting execution from synchronous to asynchronous. Queues provide a high-throughput messaging backbone that decouples the “Producer” (the code that creates a task) from the “Consumer” (the code that executes it), ensuring that traffic spikes don’t overwhelm your downstream systems.
Cloudflare Workflows takes this a step further by providing a durable execution engine. Unlike a standard Worker script that starts from the top every time, a Workflow is composed of discrete “steps.” If a step fails or the environment restarts, the Workflow engine remembers exactly which steps were completed and resumes execution from the point of failure, maintaining state throughout the process. This enables complex, long-running business logic—like order fulfillment, content moderation pipelines, or automated subscription management—to run reliably over minutes, hours, or even days.
This chapter covers the architecture of asynchronous edge computing. We will explore how to configure Queues with Dead Letter Queues (DLQs) for robust error handling, how to use batching to optimize throughput, and how to define durable Workflows that survive restarts and handle complex retries. The sections that follow walk through the Producer/Consumer model of Queues, the step-based orchestration of Workflows, and the integration patterns that connect these services to the broader Cloudflare ecosystem.
Key point: Cloudflare Queues decouple services for high-volume asynchronous task processing, while Cloudflare Workflows provide durable, stateful orchestration for multi-step processes that must survive failures and handle long durations.
Why exists
The Request-Response Timeout Problem. Standard Workers have execution limits (usually 30 seconds of CPU time). Tasks that take longer—like generating a large PDF or processing a video—cannot be completed within a single HTTP request. Queues and Workflows allow these tasks to be moved into the background where they are not bound by the initial request’s timeout.
The Scalability and Spike Suppression Problem. If a website suddenly receives 100,000 sign-ups, and each sign-up triggers a heavy database write and an email send, the downstream database might crash. A Queue acts as a buffer, accepting the messages at high speed and allowing a Consumer to process them at a steady, manageable rate.
The State Recovery Problem. In a multi-step process, if the third step fails, a standard script loses all progress. You have to write complex logic to figure out where you left off. Workflows exist to solve this natively; the engine automatically persists the result of every successful step, ensuring that retries don’t repeat expensive or non-idempotent operations.
The Dead Letter Queue (DLQ) requirement. Some messages are “poisonous”—they cause errors every time they are processed. Without a Queue system, these errors might go unnoticed or loop infinitely. A DLQ provides a safe “holding area” for failed messages, allowing developers to inspect and debug them without stopping the rest of the pipeline.
The Trade-off of Distributed Retries. Implementing retries with exponential backoff manually is error-prone. Queues provide built-in retry logic that automatically re-enqueues failed messages with a delay, ensuring that temporary outages in external APIs don’t lead to permanent data loss.
The Long-Running Orchestration Problem. Some tasks require waiting for days (e.g., “Wait 3 days, then send a follow-up email”). Keeping a Worker alive for 3 days is impossible. Workflows allow for “sleep” steps that pause execution and resume it precisely when needed, consuming zero resources while waiting.
a. Cloudflare Queues: Producers, Consumers, and DLQs
Cloudflare Queues follow a simple messaging pattern. A Producer (any Worker with a Queue binding) sends a JavaScript object or string to the queue. The Consumer (a Worker specifically configured to listen to that queue) receives these messages. The magic happens in the middle: the Queue service handles the persistence, the delivery guarantees (at-least-once), and the retry logic.
// THE PRODUCER (Standard Worker)
export default {
async fetch(request, env) {
const payload = await request.json();
// Send a message to the queue
await env.MY_QUEUE.send({
userId: payload.id,
action: "generate_report",
timestamp: Date.now(),
});
return new Response("Task enqueued!");
},
};
The Consumer is defined by a queue handler. Instead of a fetch event, it responds to a queue event which contains a batch of messages. Batching is a critical performance feature; rather than firing one Worker per message, Cloudflare can bundle 100 messages into a single Worker execution, significantly reducing cost and overhead.
// THE CONSUMER (Queue Worker)
export default {
async queue(batch, env) {
for (const message of batch.messages) {
try {
await processTask(message.body);
message.ack(); // Explicitly acknowledge success
} catch (e) {
// If we don't ack, it will be retried based on queue settings
console.error("Failed to process", e);
}
}
},
};
Dead Letter Queues (DLQs) are secondary queues. If a message fails to be acknowledged after a certain number of retries (e.g., 3 attempts), it is automatically moved to the DLQ. This prevents a single bad message from blocking your consumer indefinitely. You can later point a different Worker at the DLQ to analyze the failures.
b. Cloudflare Workflows: Durable Step Orchestration
Cloudflare Workflows move beyond simple messaging into the realm of Durable Execution. A Workflow is defined as a class extending Workflow. Inside the run method, you use step.do() to wrap discrete blocks of logic. Each step.do() is a checkpoint. If the Workflow is interrupted, Cloudflare resumes execution at the next uncompleted step.
import { WorkflowEntrypoint, WorkflowStep } from 'cloudflare:workers';
export class MyOnboardingWorkflow extends WorkflowEntrypoint {
async run(event: any, step: WorkflowStep) {
// STEP 1: Create User in DB
const userId = await step.do('create user', async () => {
const res = await api.createUser(event.email);
return res.id;
});
// STEP 2: Pause for 24 hours
await step.sleep('wait for welcome', '24 hours');
// STEP 3: Send Welcome Email
await step.do('send email', async () => {
await emailProvider.send(userId, "Welcome!");
});
}
}
The step.do block serves two purposes: it isolates errors and it persists results. If “send email” fails, the “create user” step does not run again when the workflow retries. This is vital for operations that have side effects (like charging a credit card) where you must ensure they only happen once.
Workflows also handle State Persistence. The return value of any step.do is automatically saved to Cloudflare’s internal storage and passed into the next step. This eliminates the need to manually pass IDs and status flags through a database between different stages of a process.
c. Error Handling and Scaling Patterns
Both Queues and Workflows offer sophisticated error handling. In Queues, the retry configuration in wrangler.toml defines how many times a message should be attempted and the delay between them. In Workflows, you can define retry policies per step or for the entire workflow.
# wrangler.toml for a Queue Consumer
[[queues.consumers]]
queue = "my-tasks"
max_batch_size = 10
max_batch_timeout = 30 # seconds
max_retries = 3
dead_letter_queue = "my-tasks-errors"
A common pattern is the Fan-out. A single Producer sends a message to a Queue. That Queue’s Consumer then sends messages to multiple other Queues, effectively parallelizing a massive workload. For example, a “Newsletter” queue might have a consumer that breaks a list of 1 million emails into 1,000 batches of 1,000, sending each batch to a “Send Email” queue.
Concurrency control is another vital pattern. You can limit how many Consumer Workers are running simultaneously for a Queue. This is useful for protecting legacy databases that cannot handle high concurrency. If the queue fills up faster than the consumers can empty it, Cloudflare stores the messages until capacity becomes available.
For Workflows, the Instance Pattern is key. Every time you start a Workflow, you give it an ID. If you try to start a Workflow with the same ID, Cloudflare can either return the existing one or fail, preventing duplicate processing of the same business event (like a duplicate order ID).
Complete Example Session
// ============================================
// PART 1: QUEUE PRODUCER (Order Received)
// ============================================
export default {
async fetch(request, env) {
const order = await request.json();
await env.ORDERS_QUEUE.send(order);
return new Response("Order processing started", { status: 202 });
}
}
// ============================================
// PART 2: QUEUE CONSUMER (Triggers Workflow)
// ============================================
export default {
async queue(batch, env) {
for (const msg of batch.messages) {
// Launch a durable workflow for each order
await env.ORDER_WORKFLOW.create({
id: `order-${msg.body.id}`,
params: msg.body
});
msg.ack();
}
}
}
// ============================================
// PART 3: THE WORKFLOW CLASS
// ============================================
import { WorkflowEntrypoint } from 'cloudflare:workers';
export class OrderWorkflow extends WorkflowEntrypoint {
async run(event: any, step: any) {
// PART 4: STEP 1 - Inventory Check
const inventory = await step.do('check inventory', async () => {
return await db.check(event.productId);
});
if (!inventory.available) return;
// PART 5: STEP 2 - Payment (External API)
const payment = await step.do('process payment', {
retries: { limit: 5, delay: '10 seconds' }
}, async () => {
return await stripe.charge(event.amount, event.token);
});
// PART 6: STEP 3 - Shipping (Delay)
await step.sleep('wait for warehouse', '1 hour');
// PART 7: STEP 4 - Label Generation
const label = await step.do('generate label', async () => {
return await fedex.createLabel(event.address);
});
// PART 8: STEP 5 - Notification
await step.do('notify user', async () => {
await email.send(event.email, `Shipped! Tracking: ${label.id}`);
});
}
}
// ============================================
// PART 9: WRANGLER CONFIGURATION
// ============================================
// [[queues.producers]]
// binding = "ORDERS_QUEUE"
// queue = "orders"
// [[workflows]]
// name = "order-workflow"
// binding = "ORDER_WORKFLOW"
// class_name = "OrderWorkflow"
// ============================================
// PART 10: MONITORING
// ============================================
// Use 'wrangler workflows list' to see active instances
The example above demonstrated a complete asynchronous pipeline: an HTTP Worker enqueues a message, a Queue Worker consumes the batch to ensure reliability, and a Durable Workflow orchestrates the complex, multi-step business logic of order fulfillment.
Quick Reference
Queue Consumer Settings
| Setting | Description | Recommended |
|---|---|---|
max_batch_size | Messages per consumer run | 10–100 |
max_batch_timeout | Wait time to fill a batch | 10s–30s |
max_retries | Attempts before DLQ | 3–5 |
dead_letter_queue | Target for failed messages | Required for Prod |
Workflow Step Types
| Step Method | Purpose | Resumes On Failure |
|---|---|---|
step.do(id, fn) | Execute logic block | Yes (Skips if done) |
step.sleep(id, time) | Pause execution | Yes (Wait completes) |
workflow.create(id) | Start instance | N/A (Entry point) |
workflow.get(id) | Check status | N/A |
Reliability Comparison
| Feature | Queues | Workflows |
|---|---|---|
| Primary Use | High-volume tasks | Stateful orchestration |
| Duration | Minutes (via retry) | Days (via sleep) |
| State | Message only | Full JSON result per step |
| Error Handling | DLQs / Retries | Per-step retries / Resumption |
Best Practices
✅ Do This:
// Always wrap side-effects in step.do
await step.do('charge card', async () => { ... }); // ✅
// Use explicit message acknowledgement in Queues
message.ack(); // ✅
// Define a Dead Letter Queue for every production queue
dead_letter_queue = "my-queue-errors" // ✅
// Use batching to reduce costs in high-volume queues
max_batch_size = 50 // ✅
// Use idempotent keys for Workflow instances
env.WORKFLOW.create({ id: `user-${userId}` }); // ✅
❌ Don’t Do This:
// Don't perform side-effects outside of a step in Workflows
await chargeCard(); // ❌ (Will repeat if workflow restarts)
// Don't use Queues for logic requiring multi-step state
// (Use Workflows instead) // ❌
// Don't set infinite retries without a DLQ
// (Can cause a "poison pill" infinite loop) // ❌
// Don't pass massive binary blobs in Queue messages
// (Store in R2 and pass the key instead) // ❌
Common Pitfalls
| Pitfall | Why It Happens | Fix |
|---|---|---|
| Poison Messages | A specific message causes an unhandled crash | Use a DLQ and max_retries. |
| Duplicate Operations | Workflow step not wrapped in step.do | Ensure every external call is an isolated step. |
| Batch Timeout | Consumer runs before batch is full | Adjust max_batch_timeout to wait longer. |
| Workflow ID Conflict | Attempting to start the same ID twice | Catch the error or use unique IDs (e.g., UUIDs). |
| State Bloat | Passing massive objects between steps | Keep step return values small; use R2 for large data. |
| Visibility Timeout | Message processing takes longer than expected | Increase the visibility_timeout in queue settings. |
Real-World Examples
1. Image Processing Pipeline
A Producer uploads to R2 and enqueues a key; a Consumer generates thumbnails in the background.
2. Drip Marketing Campaigns
A Workflow that sends an email, sleeps for 3 days, checks if the user clicked, and sends another.
3. Log Aggregator
Collecting thousands of small log entries in a Queue and flushing them to D1 in batches of 100.
4. Subscription Renewal
A Workflow that attempts to charge a card, retries with backoff on failure, and cancels the account after 3 failed days.
5. Data Migration
Breaking a 1-million-row migration into chunks, enqueuing each chunk into a high-concurrency queue.
6. Webhook Proxy
Accepting high-speed webhooks from Stripe or GitHub and processing them slowly to avoid crashing a backend.
7. PDF Generation
A long-running step that calls a headless browser and stores the result in R2.
8. Order Fulfillment
A Workflow coordinating inventory, payment, warehouse API, and shipping notifications.
9. AI Content Moderation
Enqueuing user comments for an AI model to scan; if toxic, the Consumer deletes them from the DB.
10. IoT Heartbeat Monitor
A Workflow that sleeps and checks if a device has checked in; if not, it sends an alert.
Visual
Cloudflare Queue Pipeline
┌──────────────┐ ┌──────────────┐ ┌────────────────┐
│ PRODUCER │ │ CLOUDFLARE │ │ CONSUMER │
│ (Worker/API) │ │ QUEUE │ │ (Queue Worker) │
└──────┬───────┘ └──────┬───────┘ └────────┬───────┘
│ │ │
│ 1. .send(msg) │ │
├────────────────────►│ │
│ │ 2. Wait for Batch │
│ │ │
│ │ 3. Dispatch Batch │
│ │ (suppress spikes) │
│ ├──────────────────────►│
│ │ │
│ │ 4. Ack / Retry │
│ │◄──────────────────────┤
│ │ │
│ │ 5. Move to DLQ │
│ │ (if failed 3x) │
│ └──────────┬────────────┘
│ │
▼ ▼
[ Success ] [ Dead Letter Queue ]
Durable Workflow Resumption
┌────────────────────────────────────────────────────────┐
│ WORKFLOW: User Signup │
│ │
│ STEP 1: Create DB Record [ COMPLETED - Saved ✅ ] │
│ │
│ STEP 2: API Call (Stripe) [ FAILED - RETRYING ❌ ] │
│ │
│ [ SYSTEM RESTART / CRASH ] │
│ │
│ 1. Workflow resumes... │
│ 2. Checks step state... │
│ 3. SKIPS Step 1 (already in DB) │
│ 4. RUNS Step 2 Again │
│ │
│ STEP 3: Send Email [ WAITING... ] │
└────────────────────────────────────────────────────────┘
Batching vs Latency
┌────────────────────────────────────────────────────────┐
│ BATCHING EFFICIENCY │
│ │
│ Without Batching: │
│ [M] [M] [M] [M] ──► 4 Worker Executions ($$$) │
│ │
│ With Batching (size=4): │
│ [M, M, M, M] ──► 1 Worker Execution ($) │
│ │
│ Result: Lower cost, higher throughput, higher latency │
└────────────────────────────────────────────────────────┘
Summary
| Item | Value |
|---|---|
| Queue Delivery | At-least-once |
| Max Delay | 12 hours (Queue) / Days (Workflow) |
| Max Step Retries | Configurable per-step |
| Batch Limit | 100 messages |
| State Persistence | Automatic in step.do |
| Primary Tool | Wrangler CLI |
| Visibility Timeout | Duration a message is hidden while processing |
Key takeaways:
- Queues are for decoupling and scale. Use them to absorb traffic spikes and handle high-volume background tasks that don’t require complex multi-step coordination.
- Workflows are for reliability and state. Use them when you have a multi-step process where it’s unacceptable to lose progress or repeat sensitive operations like payments.
- Dead Letter Queues are your safety net. Always define a DLQ to capture “poison pills” that would otherwise cause infinite retry loops in your primary consumers.
- Batching is a cost-saver. By processing multiple messages in a single Worker execution, you reduce the per-request overhead and lower your Cloudflare bill.
- Durable execution is a paradigm shift. Moving from “run everything at once” to “run in isolated, persisted steps” makes your code significantly more resilient to network failures and system restarts.
Remember: Use Queues to manage load; use Workflows to manage complexity. By combining the two, you can build production-grade asynchronous systems that are both highly scalable and perfectly reliable.
Stop using slow, ad-bloated tool sites! 🤮
🔎 Search “KandZ Tools” on Google to use many professional utilities for free.
KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)
⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free
🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!