Paginate Express API Results with Cursor and Offset Strategies
Cursor-based pagination is often superior to offset-based pagination for large datasets because it avoids performance degradation as you paginate deeper into the results.
Here’s an example of an Express API endpoint that supports both offset and cursor-based pagination, using a hypothetical items array.
const express = require('express');
const app = express();
const port = 3000;
// Sample data
const items = Array.from({ length: 1000 }, (_, i) => ({
id: i + 1,
name: `Item ${i + 1}`,
createdAt: new Date(Date.now() - (1000 - i) * 1000) // Items sorted by creation date
}));
app.get('/items', (req, res) => {
const { limit = 10, page, cursor, sortBy = 'createdAt' } = req.query;
let paginatedItems = [...items];
// Sorting
paginatedItems.sort((a, b) => {
if (a[sortBy] < b[sortBy]) return -1;
if (a[sortBy] > b[sortBy]) return 1;
return 0;
});
let results = {};
if (cursor) {
// Cursor-based pagination
const cursorIndex = paginatedItems.findIndex(item => item.id.toString() === cursor);
if (cursorIndex === -1) {
return res.status(400).json({ error: 'Invalid cursor' });
}
const startIndex = cursorIndex + 1;
const endIndex = startIndex + parseInt(limit);
results.data = paginatedItems.slice(startIndex, endIndex);
results.nextCursor = results.data.length > 0 ? results.data[results.data.length - 1].id : null;
results.prevCursor = paginatedItems[cursorIndex] ? paginatedItems[cursorIndex].id : null; // The item *before* the cursor
} else if (page) {
// Offset-based pagination
const pageNum = parseInt(page);
const offset = (pageNum - 1) * parseInt(limit);
const startIndex = offset;
const endIndex = startIndex + parseInt(limit);
results.data = paginatedItems.slice(startIndex, endIndex);
results.currentPage = pageNum;
results.totalPages = Math.ceil(items.length / parseInt(limit));
results.nextPage = pageNum < results.totalPages ? pageNum + 1 : null;
results.prevPage = pageNum > 1 ? pageNum - 1 : null;
} else {
// Default to first page offset pagination if no cursor or page provided
const offset = 0;
const startIndex = offset;
const endIndex = startIndex + parseInt(limit);
results.data = paginatedItems.slice(startIndex, endIndex);
results.currentPage = 1;
results.totalPages = Math.ceil(items.length / parseInt(limit));
results.nextPage = 1 < results.totalPages ? 2 : null;
results.prevPage = null;
}
res.json(results);
});
app.listen(port, () => {
console.log(`API listening at http://localhost:${port}`);
});
This setup allows you to fetch data like this:
- Offset:
GET /items?limit=5&page=3 - Cursor:
GET /items?limit=5&cursor=50(to get items after item with ID 50) - Cursor with sort:
GET /items?limit=5&cursor=50&sortBy=createdAt
The core problem this solves is efficiently retrieving large lists of data in manageable chunks. Offset pagination (page and limit) is simple: skip (page - 1) * limit records and take limit records. The issue arises because database systems often need to scan and discard records up to the offset, which becomes very slow for large offsets. Cursor pagination, on the other hand, uses a bookmark (the "cursor") to mark the last item seen. The next request asks for items after that cursor. This means the database only needs to find items greater than the cursor’s value, which is much faster, especially when sorting by an indexed column like id or createdAt.
Internally, the system first sorts the entire dataset based on the sortBy query parameter. For offset pagination, it calculates the startIndex and endIndex based on the page and limit and uses Array.prototype.slice to extract the portion. It then calculates totalPages, nextPage, and prevPage for navigation. For cursor pagination, it finds the index of the item matching the cursor ID. The startIndex is set to the item after the cursor, and endIndex is calculated. The nextCursor is the ID of the last item in the current page, and prevCursor is the ID of the item before the cursor (which essentially represents the "end" of the previous page). A crucial detail for cursor pagination is that the cursor value itself must be unique and sortable, typically an ID or a timestamp combined with a unique identifier.
The most surprising aspect of cursor pagination is how it fundamentally changes the concept of "page." With offset pagination, page 3 always contains the same items regardless of when you request it (assuming no new items are added before the current page). With cursor pagination, the "next page" is determined by the last item you received. If new items are added to the dataset after the cursor you’re using, your "next page" might contain fewer items than your limit, or it might even skip over items that were added after your cursor but before the end of your current page. This makes cursor pagination excellent for real-time feeds where you want to ensure you don’t miss anything, but it means you can’t reliably jump to a specific "page number."
The next thing you’ll likely want to tackle is implementing more sophisticated sorting, including multi-column sorting and reverse order.