| | |

Netlify 16 🌐 Async Workloads — Building durable, event-driven queues, multi-step workflows, and automatic retries

⏱️ Estimated read: 13 min | |

Serverless functions have revolutionized how we deploy backend logic, but they have always been haunted by the “execution ghost”—the strict timeout. In a standard serverless environment, a function must complete its task within seconds (usually 10 to 30), or it is summarily killed by the infrastructure. This creates a massive hurdle for developers building complex systems like video processors, multi-day email drip campaigns, or order fulfillment pipelines that require waiting for third-party webhooks. Netlify Async Workloads, specifically Netlify Workflows, solve this by introducing durable execution.

Durable execution means that your code’s state—its variables, progress, and history—is persisted automatically. If a function fails halfway through a ten-step process, the system doesn’t just crash; it retries the specific failed step with exponential backoff. If your code needs to “sleep” for three days until a customer confirms an email, the execution pauses, frees up compute resources, and wakes up exactly where it left off seventy-two hours later. This paradigm shift moves us away from “fire and forget” functions toward “set and ensure” business logic.

This chapter explores the architectural foundations of Netlify Workflows and Background Functions. We will examine how to define multi-step processes that are resilient to transient network failures, how to manage state across long-running tasks, and how to trigger these workloads using events. By the end of this guide, you will understand how to build robust, asynchronous systems that handle the complexity of the real world without the overhead of managing dedicated queue servers or state databases.

The sections follow a progression from basic asynchronous execution to advanced durable orchestration. We will cover the core SDK patterns, the retry mechanisms that ensure reliability, and the integration strategies for connecting these workflows to your frontend and external APIs.

Key point: Netlify Workflows provide durable execution, allowing code to run for minutes, hours, or weeks by persisting state between “steps,” automatically retrying failures, and pausing execution during idle wait times to eliminate timeout constraints.


Why exists

Why exists

The execution timeout problem. Standard serverless functions are built for request-response cycles. They are optimized for speed, which means they are intentionally capped. When a task requires more than 10-30 seconds—such as generating a massive PDF or syncing a large dataset—a standard function simply dies. Workflows exist to break this “time-box,” allowing logic to span across multiple function invocations over any duration.

The partial failure problem. In a traditional script that calls three different APIs, if the third API fails, the entire script fails. You are then left with a “zombie” state: the first two actions happened, but the third didn’t. Workflows use “steps” to checkpoint progress. If step three fails, the system retries only step three, knowing that steps one and two are already safely committed.

The “sleep” cost problem. Running a server or a container just to wait for a timer or a webhook is a waste of money and resources. Workflows allow you to programmatically “sleep” or “wait.” During these periods, no compute is active, and you aren’t billed for idle time. The workflow is simply a record in a database until the wake-up condition is met.

The state management problem. Orchestrating multi-step tasks usually requires a database to track status (e.g., is_step_1_done: true). This adds significant “plumbing” code to every project. Workflows treat the code itself as the state. The variables you define at the start of a workflow are available at the end, even if the end happens five days later on a different physical server.

The visibility problem. Debugging asynchronous tasks is notoriously difficult because they happen “out of sight.” Netlify Workflows provide a built-in execution history. Developers can see exactly which step failed, what the input was, and why the retry logic was triggered, providing a level of observability that custom-built queue systems rarely offer without significant effort.

The infrastructure overhead problem. Building a durable queue system typically involves setting up Redis, RabbitMQ, or AWS SQS/Step Functions. This requires managing credentials, scaling the queue workers, and handling dead-letter queues. Netlify abstracts this entire layer into a single SDK, allowing developers to focus on the business logic rather than the message broker.


a. Defining Durable Workflows with the SDK

The core of Netlify Workflows is the workflow function provided by the @netlify/workflows library. Unlike standard functions, a workflow is a generator-like structure (though implemented via modern async/await patterns) that wraps your logic in “steps.” Every action that interacts with the outside world—like a database call or an API request—should be wrapped in a step.run call.

import { workflow } from "@netlify/workflows";

export const orderWorkflow = workflow("order-process", async ({ step }) => {
  // Step 1: Validate the order
  const validatedOrder = await step.run("validate-order", async () => {
    // Logic to check inventory
    return { id: "123", status: "valid" };
  });

  // Step 2: Charge the customer
  await step.run("charge-customer", async () => {
    // Logic to call Stripe
    console.log(`Charging for order ${validatedOrder.id}`);
  });
});

In this snippet, step.run acts as a checkpoint. When the workflow executes, Netlify records the output of “validate-order.” If the “charge-customer” step fails later, Netlify won’t re-run the validation; it will simply look up the cached result of the first step and proceed. This idempotency is what makes the workflow durable. It ensures that side effects (like charging a credit card) aren’t accidentally repeated during retries.

The workflow is triggered via a standard HTTP POST request to the Netlify endpoint. This allows your frontend or a webhook from a service like Shopify to kick off the process. Because the workflow returns a 202 Accepted immediately, your frontend doesn’t have to wait for the long-running process to finish, providing a snappy user experience.


b. Handling Long Waits and External Events

One of the most powerful features of durable workloads is the ability to pause execution. In traditional programming, setTimeout or sleep in a serverless function would just keep the bill running until a timeout occurs. In Netlify Workflows, step.sleep literally pauses the execution and shuts down the compute instance.

import { workflow } from "@netlify/workflows";

export const userOnboarding = workflow("user-onboarding", async ({ step }) => {
  await step.run("send-welcome-email", async () => {
    // Email logic
  });

  // Pause for 3 days
  await step.sleep("wait-for-trial-end", "3d");

  await step.run("send-upgrade-offer", async () => {
    // Check if user is still on free tier, then send offer
  });
});

Beyond simple timers, workflows can wait for external signals using step.waitForEvent. This is useful for “Human-in-the-loop” patterns, where a process needs an approval from an admin. The workflow stops at the waitForEvent line and won’t proceed until an external API call signals that the event has occurred.

The “wait” capability transforms how we think about business logic. Instead of writing a “cron job” that checks the database every hour for users who signed up three days ago, you simply write a workflow that describes a single user’s journey. This “per-entity” workflow model is much easier to reason about and scale than massive batch processing scripts.


c. Resilience and Automatic Retries

Failure is inevitable in distributed systems. A third-party API might be down for maintenance, or a database connection might flicker. Netlify Workflows handle this through a robust retry policy. By default, if a step.run fails, Netlify will retry it using exponential backoff.

You can customize this behavior for specific steps. For example, a mission-critical payment step might need a more aggressive or conservative retry strategy than a simple logging step.

await step.run("flaky-api-call", 
  {
    retries: 5,
    backoff: {
      initialInterval: "10s",
      multiplier: 2,
      maxInterval: "1h"
    }
  },
  async () => {
    // This will be retried up to 5 times on failure
    return await fetch("https://flaky-service.com/data");
  }
);

When all retries are exhausted, the workflow enters a “Failed” state. Developers can then inspect the failure in the Netlify UI, fix the underlying issue (like an expired API key), and manually trigger a “Resume” from the failed step. This manual intervention capability is a lifesaver for complex operations that cannot be fully automated or require human judgment after a technical failure.

The persistence of state also means that local variables are preserved. If you calculate a value in Step 1, it is available in Step 10, even if Step 10 happens a week later. Netlify achieves this by re-playing the workflow history. It doesn’t actually save the “memory heap” of the JS engine; it simply re-runs the function and replaces the step.run calls with their previously recorded results. This is why it is critical that code outside of step.run remains deterministic.


Complete Example Session

// ============================================
// PART 1: THE WORKFLOW DEFINITION
// ============================================
import { workflow } from "@netlify/workflows";

export const deliveryWorkflow = workflow("delivery-pipeline", async ({ step }) => {

// ============================================
// PART 2: INITIAL DATA PROCESSING
// ============================================
  const order = await step.run("fetch-order", async () => {
    return { id: "ORD-99", email: "customer@example.com", items: ["Laptop"] };
  });

// ============================================
// PART 3: EXTERNAL API INTEGRATION
// ============================================
  await step.run("charge-stripe", async () => {
    // Imagine a Stripe API call here
    console.log(`Charged for ${order.id}`);
  });

// ============================================
// PART 4: CONDITIONAL LOGIC BASED ON STEPS
// ============================================
  await step.run("notify-warehouse", async () => {
    // Logic to alert shipping team
    return { trackingId: "TRK-12345" };
  });

// ============================================
// PART 5: THE LONG SLEEP
// ============================================
  // Wait 24 hours for the package to be picked up
  await step.sleep("wait-for-pickup", "24h");

// ============================================
// PART 6: WAITING FOR AN EXTERNAL SIGNAL
// ============================================
  // Pause until the shipping provider sends a "delivered" webhook
  const deliveryStatus = await step.waitForEvent("shipment-delivered", {
    timeout: "7d" // If not delivered in 7 days, trigger timeout logic
  });

// ============================================
// PART 7: ERROR HANDLING WITHIN WORKFLOWS
// ============================================
  if (!deliveryStatus) {
    await step.run("alert-support", async () => {
      // Logic to notify support that delivery is late
    });
    return; // Exit workflow
  }

// ============================================
// PART 8: FINAL CUSTOMER COMMUNICATION
// ============================================
  await step.run("send-survey", async () => {
    console.log(`Sending satisfaction survey to ${order.email}`);
  });

// ============================================
// PART 9: CLEANUP OR STATE FINALIZATION
// ============================================
  await step.run("archive-order", async () => {
    // Final database update
    return { finalized: true };
  });
});

// ============================================
// PART 10: TRIGGERING THE WORKFLOW
// ============================================
// From a standard Netlify Function or Frontend:
// await fetch('/.netlify/functions/delivery-pipeline', { method: 'POST' });

The example session demonstrates a complete 10-part lifecycle of an asynchronous delivery pipeline, including charging, sleeping, waiting for external events, and error handling.


Quick Reference

Workflow Core Methods

MethodPurposeTimeout Behavior
step.run()Execute an atomic taskRetries on failure
step.sleep()Pause for a durationCompute stops, no cost
step.waitForEvent()Pause for external APIResumes via webhook/signal
step.invoke()Call another workflowParent waits for child

Duration & Limits

UnitFormatExample
Secondss30s
Minutesm15m
Hoursh2h
Daysd7d

Workflow States

StateMeaning
ActiveCurrently executing a step
SleepingPaused on a timer
WaitingPaused for an external event
ErroredRetrying after a failure
FailedMax retries hit, requires manual intervention

Best Practices

✅ Do This:

// Keep step names unique and descriptive
await step.run("calculate-tax-v2", async () => { ... });      // ✅

// Wrap every network call in a step
const data = await step.run("api-fetch", async () => { ... }); // ✅

// Use deterministic logic outside of steps
const constantValue = 42;                                     // ✅

// Handle wait timeouts explicitly
const event = await step.waitForEvent("evt", { timeout: "1h" }); // ✅

❌ Don’t Do This:

// Use Math.random() outside of a step (non-deterministic)
const rand = Math.random();                                   // ❌

// Use new Date() to check time (replay will get different date)
const now = new Date();                                       // ❌

// Run long loops outside of a step.run
for(let i=0; i<10000; i++) { heavyLogic(); }                 // ❌

// Perform side effects directly in the workflow body
await fetch("https://api.com/log");                           // ❌

Common Pitfalls

PitfallWhy It HappensFix
Side Effect ReplayCode outside step.run runs every retryMove all side effects into step.run
Non-DeterminismMath.random() or Date yields different results on replayPass these values into the workflow as input
Large Step PayloadsNetlify has limits on data passed between stepsStore large data in Netlify Blobs, pass the key
Infinite RetriesNot setting a retry limit on a doomed API callConfigure retries: X in the step options
Closure ConfusionAttempting to access local variables that weren’t returned by stepsEnsure steps return necessary state to the workflow scope
Timeout on First StepThe very first setup logic takes too longMove heavy setup into its own named step.run

Real-World Examples

1. Daily Summary Email

await step.sleep("wait-until-morning", "24h");
await step.run("send-email", async () => { /* email logic */ });

2. Payment Verification

const success = await step.run("verify-payment", async () => {
  return await stripe.confirm();
});

3. Image Optimization Pipeline

await step.run("resize", async () => { /* resize logic */ });
await step.run("upload-to-s3", async () => { /* upload */ });

4. Slack Approval Workflow

await step.run("post-to-slack", async () => { /* post msg */ });
await step.waitForEvent("slack-approved");

5. Abandoned Cart Recovery

await step.sleep("wait-1-hour", "1h");
await step.run("check-cart", async () => { /* logic */ });

6. Multi-Step User Import

for (const user of users) {
  await step.run(`import-${user.id}`, async () => { /* db */ });
}

7. Webhook Transformation

const data = await step.run("parse-webhook", async () => { return payload; });

8. License Key Expiration

await step.sleep("wait-one-year", "365d");
await step.run("disable-key", async () => { /* db */ });

9. Database Backup Rotation

await step.run("snapshot", async () => { /* backup */ });
await step.sleep("wait-for-rotation", "30d");

10. API Polling Substitute

// Instead of polling, wait for the specific event
await step.waitForEvent("processing-complete");

Visual

Workflow Lifecycle Execution

┌─────────────────────────────────────────────────────────┐
│  WORKFLOW EXECUTION ENGINE                              │
│                                                         │
│  1. Run Step A ──► Success ──► Save Result in History   │
│  2. Run Step B ──► FAIL ──┐                             │
│                           ▼                             │
│  3. Wait 10s (Backoff) ───┘                             │
│  4. Re-run Step B ──► Success ──► Save Result           │
│  5. SLEEP (7 Days) ──► SHUT DOWN COMPUTE                │
│                                                         │
│  [7 Days Later]                                         │
│                                                         │
│  6. WAKE UP ──► Lookup Step A & B History               │
│  7. Run Step C ──► Finalize                             │
└─────────────────────────────────────────────────────────┘

Durable State vs. Standard Memory

┌─────────────────────────────────────────────────────────┐
│  STANDARD FUNCTION            NETLIFY WORKFLOW          │
│  ┌──────────────┐             ┌──────────────┐          │
│  │ RAM MEMORY   │             │ STEP HISTORY │          │
│  │ (Volatile)   │             │ (Persistent) │          │
│  └──────┬───────┘             └──────┬───────┘          │
│         │                            │                  │
│   CRASH = LOSS                 CRASH = RESUME           │
│   TIMEOUT = KILL               TIMEOUT = N/A            │
│                                                         │
└─────────────────────────────────────────────────────────┘

Event-Driven Resumption

┌─────────────────────────────────────────────────────────┐
│  Workflow Paused: waitForEvent("order_shipped")         │
│                                                         │
│  [ External World ]                                     │
│  Shipping Provider API ──┐                              │
│                          │ (HTTP POST Webhook)          │
│                          ▼                              │
│  Netlify Event Handler ──┴──► Notify Workflow Engine    │
│                                        │                │
│  Workflow Resumes ◄────────────────────┘                │
│                                                         │
└─────────────────────────────────────────────────────────┘

Summary

ItemValue
Core Library@netlify/workflows
Primary Unitstep.run()
Time Managementstep.sleep()
Event Listeningstep.waitForEvent()
State PersistenceAutomatic via Replay
Retry StrategyExponential Backoff (Default)
Max DurationPractically Unlimited (Days/Weeks)
ConcurrencyManaged by Netlify Infrastructure
BillingCharged per execution/step, not idle time

Key takeaways:

  • Durable execution eliminates the timeout limitations of serverless. Workflows allow code to run for as long as necessary by splitting logic into discrete, checkpointed steps.
  • step.run is the fundamental building block for reliability. Every side effect should be wrapped in a step to ensure it is idempotent and can be retried without duplicating actions.
  • Sleeping and waiting are cost-free operations. Unlike standard serverless functions, Workflows do not charge for the time spent waiting for a timer or an external event.
  • Determinism is mandatory for the workflow body. Because the engine “replays” code to restore state, using non-deterministic functions (like Math.random()) outside of a step.run will cause inconsistencies.
  • Automatic retries provide high availability. Developers don’t need to write custom logic for network blips; the infrastructure handles the backoff and retry cycles automatically.
  • Manual intervention is possible. If a workflow fails after all retries, it can be inspected and manually resumed from the Netlify dashboard once the issue is resolved.

Remember: Netlify Workflows are designed for the “Long Tail” of logic. If a task takes 500ms, use a standard function. If a task takes 2 minutes, requires 3 retries, or needs to wait for a user to click a link next Tuesday, a Workflow is the superior architectural choice. It shifts the burden of state and reliability from your code to the Netlify platform.

Was this helpful?


Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!