n8n API Pagination & Rate Limits: Reliable Integrations Without Timeouts

Large API syncs fail for two reasons: you pull too much too fast (rate limits) or you don't paginate correctly (missed or duplicate records). These failures are silent and cumulative. You don't notice 200 missing records until someone asks why the dashboard numbers don't match the source.

This guide shows how to implement cursor and page pagination, respect provider limits, and add retries with backoff in n8n so your data syncs finish reliably every time.

Tip: Looking for webhook hardening? Read our companion guide: n8n Webhook Best Practices.

What You'll Build

A resilient n8n workflow that:

• Fetches all pages using page or cursor strategies without gaps
• Handles 429/5xx with retries and exponential backoff
• Batches processing using Split In Batches for memory safety
• Resumes safely using checkpoints to avoid re-fetching on restart
• Respects provider quotas by reading Retry-After headers

Understanding Pagination Models

Before building, you need to identify which pagination model the target API uses. Most APIs fall into one of two categories, and choosing the wrong pattern leads to missed or duplicate records.

1) Page/Offset Pagination

The API accepts page + limit or offset + limit parameters. You increment the page number (or offset) until you get fewer results than the limit.

• Pros: Simple to implement, easy to calculate total pages
• Cons: Unstable when data changes during iteration. If a record is inserted or deleted between page requests, you get duplicates or gaps.

// Example: GET /items?page={{$json.page}}&limit=100
const page = $json.page ?? 1
return [{ page, limit: 100 }]

n8n loop pattern:

1. Initialize page = 1 in a Set node
2. HTTP Request node fetches the page
3. Code node extracts items and checks count
4. IF node: items.length < limit means you're on the last page
5. Otherwise increment page and loop back to the HTTP Request

2) Cursor/Token Pagination (Recommended)

The API returns a cursor, next_token, or next URL in each response. You pass it back in the next request. When the cursor is null or absent, you've reached the end.

• Pros: Stable across data changes, no gaps or duplicates
• Cons: Cannot jump to arbitrary pages; must iterate sequentially

// Extract next cursor from API response
const next = $json.response?.meta?.next_cursor || null
return [{ cursor: next, hasMore: next !== null }]

n8n loop pattern:

1. Start with an empty cursor in a Set node
2. HTTP Request includes cursor parameter (omitted on first call)
3. Code node extracts items and next_cursor
4. IF node: continue while hasMore is true
5. Feed next_cursor back as cursor for the next iteration

When the API supports both, always choose cursor pagination. It handles concurrent writes without data integrity issues. The choice between REST, GraphQL, and cursor-friendly patterns is covered in our 2025 API design patterns guide.

Rate Limits & Backoff

Every production API enforces rate limits. When you exceed them, you get a 429 Too Many Requests response. Transient 5xx errors require the same treatment: wait and retry.

Reading Rate Limit Headers

Most APIs tell you their limits in response headers. Check for these before you hit the ceiling:

// Parse rate limit headers from HTTP response
const remaining = parseInt($headers["x-ratelimit-remaining"] || "100")
const resetAt = parseInt($headers["x-ratelimit-reset"] || "0")

if (remaining < 5) {
  const waitMs = Math.max(0, (resetAt * 1000) - Date.now()) + 500
  return [{ shouldThrottle: true, waitMs }]
}
return [{ shouldThrottle: false }]

Exponential Backoff with Jitter

When you do hit a 429 or 5xx, back off exponentially with random jitter to avoid thundering herd problems:

function backoff(attempt) {
  const base = 500  // ms
  const max = 16000
  const jitter = Math.floor(Math.random() * 250)
  return Math.min(max, base * 2 ** attempt) + jitter
}

let attempt = $json.attempt ?? 0
const status = $json.statusCode

if (status === 429 || (status >= 500 && status < 600)) {
  // Respect Retry-After header when present
  const retryAfter = $headers?.["retry-after"]
  const waitMs = retryAfter
    ? parseInt(retryAfter) * 1000
    : backoff(attempt)
  
  return [{ retry: true, waitMs, attempt: attempt + 1 }]
}
return [{ retry: false }]

Always prefer the Retry-After header over your own backoff calculation when the API provides it. Some providers (Shopify, GitHub) use this header to communicate exactly how long to wait.

Use a Wait node with the calculated waitMs, then loop back to the HTTP node. Cap attempts at 5-6 and route failures to a dead-letter queue (DLQ) for manual review.

Batching & Memory Safety

Pulling 50,000 records into memory at once crashes n8n or slows it to a crawl. Use Split In Batches to process records in manageable chunks.

Recommended batch sizes:

• Light transformations (rename fields, filter): 500
• API calls per record (enrichment, lookups): 50-100
• Database writes: 200-500

// Normalize each item before batch processing
return $json.items.map((item) => ({
  id: item.id,
  email: item.email,
  updatedAt: item.updated_at,
  source: "api_sync"
}))

Concurrency Control

When each batch triggers downstream API calls, limit concurrency to avoid overwhelming the target system:

// Process batch items sequentially when target has strict rate limits
const results = []
for (const item of $json.batch) {
  results.push({
    ...item,
    processedAt: new Date().toISOString()
  })
}
return results

For APIs with generous rate limits, you can process items in parallel. For strict APIs (like Salesforce at 100 requests per 15 seconds), sequential processing within each batch keeps you safe.

Checkpointing: Resume Where You Left Off

Production syncs get interrupted. Server restarts, deployment rollouts, and network blips all cause workflow failures. Without checkpoints, you re-fetch everything from the beginning.

Persist progress so restarts don't repeat work:

• Page model: store last successful page number
• Cursor model: store last cursor value
• Timestamp model: store highest updatedAt seen

// Save checkpoint after each successful batch
const checkpoint = {
  lastCursor: $json.cursor,
  lastUpdatedAt: $json.maxUpdatedAt,
  recordsProcessed: $json.totalProcessed,
  savedAt: new Date().toISOString()
}
// Write to your KV store, Supabase, or a simple JSON file
return [checkpoint]

Rules for safe checkpointing:

1. Update the checkpoint only after the batch is fully processed and committed
2. On restart, read the checkpoint first and resume from the stored position
3. Use updatedAt >= checkpoint.lastUpdatedAt (inclusive) to handle records that were mid-write during the last save
4. Keep a history of the last 5 checkpoints so you can roll back if data corruption is detected

Reference Architecture

A complete n8n pagination workflow follows this structure:

1. Set node: Initialize page/cursor/since from checkpoint (or defaults)
2. HTTP Request: Fetch one page of data
3. Code node: Extract items, next cursor, and rate limit headers
4. IF node: Rate limited? Route to Wait node, then back to HTTP Request
5. IF node: Retry needed (429/5xx)? Route to backoff Wait, increment attempt
6. Split In Batches: Process items in chunks
7. Code/HTTP node: Upsert to database, CRM, or destination API
8. Code node: Update checkpoint with current position
9. IF node: Has more pages? Loop back to step 2

Best Practices

1. Prefer cursor pagination when the API supports it. Page/offset breaks under concurrent writes.
2. Request minimal fields using fields or select parameters. Less data means faster responses and lower memory usage.
3. Respect Retry-After headers over your own backoff calculation. The API knows its capacity better than you do.
4. Use idempotent upserts with a unique key to handle duplicates from retries or overlapping pagination windows.
5. Log rate-limit metrics per run: how many 429s, average backoff duration, total records processed. This data helps you tune batch sizes and schedules.
6. Run long syncs in time windows (last 24 hours, since last checkpoint) rather than full table scans. Backfill historical data separately as a one-time job.
7. Set execution timeouts in n8n settings. A sync that runs indefinitely blocks the worker queue. Set a timeout, checkpoint progress, and pick up where you left off on the next scheduled run.

Troubleshooting

Problem	Cause	Fix
Duplicate records	Retries without idempotency keys	Use upserts with unique id from the source
Missing records (gaps)	Page/offset with concurrent writes	Switch to cursor pagination or lock time windows
429 storms	Too aggressive concurrency or batch size	Reduce concurrency, increase delay, check provider quotas
Memory pressure	Building giant arrays before processing	Lower batch size, stream through Split In Batches
Workflow hangs	Infinite retry loop without max attempts	Cap retries at 5-6 and route to DLQ
Stale data	Checkpoints not updating after failures	Only update checkpoint after successful commit

Deployment Considerations

• Scheduling: Run large syncs during off-peak hours for both your infrastructure and the target API. Most APIs have higher rate limits during off-peak windows.
• Timeouts: Increase execution timeout limits carefully in n8n settings. Prefer chunked runs with checkpoints over single long-running executions.
• Monitoring: Store execution summaries (record count, last cursor, error rate, duration) and alert when sync completion rate drops below 95%.
• Cost: Each HTTP request and n8n execution has a cost. Batch aggressively, request minimal fields, and cache responses when the API supports ETags or If-Modified-Since headers.

Real-World Examples

• CRM backfill: Syncing 2M contacts using cursor pagination with batch size 300, checkpointed hourly, running on a nightly schedule. Total sync time: 4 hours with zero duplicates.
• Shopify orders: Respecting Retry-After headers, pausing on 429, resuming with cursor. Handles Black Friday traffic spikes (10x normal order volume) without missing orders.
• GitHub issues: Using ETag caching to skip unchanged pages entirely. Reduces API calls by 80% on repos with low activity, staying well within GitHub's 5,000 requests/hour limit.
• Stripe events: Paginating through webhook event logs using starting_after cursor, processing each event idempotently by event.id, and persisting the last processed event for crash recovery.

Next Steps

1. Identify the pagination model your target API uses (check the docs for cursor, next_token, offset, or page parameters)
2. Add backoff logic and Retry-After handling to your HTTP nodes before you hit production traffic
3. Introduce checkpointing using a database or KV store so interrupted syncs resume cleanly
4. Monitor metrics (records/minute, 429 rate, backoff duration) and tune batch size and schedule based on real data

---

Related Reading

• All n8n articles
• AI Automation & Productivity category