Fixing the 10,000 Result Limit: Deep Pagination in Elasticsearch

If you have ever clicked through multiple pages of search results, you have experienced pagination. While most users rarely click past the first page or two, background workers, data exporters, synchronization jobs, and power users often need to traverse tens of thousands or even millions of documents.

If you attempt to do this in Elasticsearch using traditional offset-based pagination, you will quickly encounter this error:

Result window is too large, from + size must be less than or equal to: [10000]

This 10,000-document limit is not an arbitrary restriction. It is a safeguard built into Elasticsearch to protect your cluster from excessive memory consumption and expensive query execution.

In this guide, we’ll explore why this limit exists, the performance costs of deep pagination, how to implement Elasticsearch’s recommended solution using search_after and Point in Time (PIT), and how purpose-built search engines simplify pagination for user-facing search applications.

Why the 10,000 Limit Exists

To understand the limit, we first need to understand how traditional pagination works in Elasticsearch using the from and size parameters.

Suppose you want page 901, with 10 results per page:

GET /products/_search
{
  "from": 9000,
  "size": 10,
  "query": {
    "match": {
      "description": "shoes"
    }
  }
}

Because Elasticsearch is a distributed search engine, your index is typically divided across multiple shards. When this query executes:

1. The coordinating node receives the request.
2. Every shard finds its top 9,010 matching documents (from + size).
3. Each shard returns those candidates to the coordinating node.
4. The coordinating node merges all candidate results, sorts them, discards the first 9,000 documents, and finally returns the remaining 10.

The problem is that as from grows, every shard must collect and return increasingly larger candidate sets, while the coordinating node must merge and sort more results in memory. Most of this work is ultimately discarded because only the final page is returned to the client.

Rather than allowing queries to consume unbounded memory and CPU, Elasticsearch enforces the index.max_result_window setting, which defaults to 10,000.

Why Not Just Increase index.max_result_window?

While you can increase index.max_result_window, doing so simply shifts the cost to your cluster rather than solving the underlying scalability problem.

PUT /products/_settings
{
  "index.max_result_window": 1000000
}

Every larger result window requires more documents to be collected on every shard and more work for the coordinating node. Raising the limit may appear to fix the problem during development, but it often leads to slower queries, increased memory pressure, and reduced cluster stability in production.

For this reason, the Elasticsearch documentation recommends using search_after instead of large offsets.

Note: Older versions of Elasticsearch commonly used the Scroll API for deep traversal. Today, Elasticsearch recommends using Point in Time (PIT) together with search_after for most deep-pagination use cases.

The Recommended Solution: search_after with Point in Time (PIT)

Instead of skipping documents using an offset, Elasticsearch recommends the search_after parameter.

Rather than asking Elasticsearch to ignore the first 9,000 documents, search_after tells Elasticsearch where the previous page ended by supplying the sort values of the last document. This allows Elasticsearch to continue the search efficiently without repeatedly collecting and discarding all earlier results.

To ensure the result set remains consistent while the index continues to receive writes, search_after should be used together with a Point in Time (PIT). A PIT creates a lightweight snapshot of the index so documents do not move between pages because of refreshes, updates, or deletions.

Step 1: Open a Point in Time (PIT)

First, create a PIT.

POST /products/_pit?keep_alive=1m

The response returns a PIT ID:

{
  "id": "46ToAwMDaW5kZXg5BXV1SURzAAAAAWJlaGF2aW9yAAA="
}

The keep_alive parameter specifies how long Elasticsearch should keep the snapshot available.

Step 2: Execute the Initial Search

When using a PIT, there are two important rules:

1. Do not specify the index in the request URL. Always use /_search.
2. Include the PIT ID inside the request body.

You must also specify a deterministic sort order. Elasticsearch recommends using _shard_doc as the tie-breaker.

POST /_search
{
  "size": 10,
  "query": {
    "match": {
      "description": "shoes"
    }
  },
  "pit": {
    "id": "46ToAwMDaW5kZXg5BXV1SURzAAAAAWJlaGF2aW9yAAA=",
    "keep_alive": "1m"
  },
  "sort": [
    { "price": "asc" },
    { "_shard_doc": "desc" }
  ]
}

The last document in the response contains its sort values:

{
  "hits": {
    "hits": [
      {
        "_source": {
          "name": "Running Shoes",
          "price": 49.99
        },
        "sort": [49.99, 12]
      }
    ]
  }
}

Step 3: Retrieve Subsequent Pages

To retrieve the next page, copy the sort values from the last document of the previous page into the search_after parameter.

POST /_search
{
  "size": 10,
  "query": {
    "match": {
      "description": "shoes"
    }
  },
  "pit": {
    "id": "46ToAwMDaW5kZXg5BXV1SURzAAAAAWJlaGF2aW9yAAA=",
    "keep_alive": "1m"
  },
  "sort": [
    { "price": "asc" },
    { "_shard_doc": "desc" }
  ],
  "search_after": [49.99, 12]
}

Repeat this process until all documents have been retrieved.

Step 4: Close the PIT

Once pagination is complete, close the PIT to release cluster resources.

DELETE /_pit
{
  "id": "46ToAwMDaW5kZXg5BXV1SURzAAAAAWJlaGF2aW9yAAA="
}

The Trade-Offs of search_after

search_after is significantly more efficient than deep offset pagination, but it also introduces additional implementation complexity.

• Client-side state: Your application must keep track of the PIT ID and the sort values from the last document.
• Sequential navigation: Users cannot jump directly to page 50 or page 100. Each page depends on the previous one.
• PIT lifecycle management: Open PITs consume cluster resources until they are closed or expire.
• More application logic: Pagination becomes a stateful workflow rather than a simple page parameter.

For data exports, synchronization pipelines, or background jobs, these trade-offs are usually worthwhile because they provide predictable performance even across very large datasets.

An Alternative Approach: Purpose-Built Search Engines

The search_after + PIT approach is an excellent solution for sequentially traversing large result sets. However, for many user-facing applications such as e-commerce websites, documentation portals, or business directories, it introduces complexity that developers often don’t need.

Purpose-built search engines like Typesense take a different approach by exposing familiar page-based pagination without requiring applications to manage PIT IDs or pagination state.

Pagination in Typesense

Typesense supports familiar page-based pagination using the page and per_page parameters.

POST /collections/products/documents/search
{
  "q": "shoes",
  "query_by": "description",
  "page": 101,
  "per_page": 10
}

If your application already uses SQL-style pagination, for example, through GraphQL, Typesense also supports offset and limit.

POST /collections/products/documents/search
{
  "q": "shoes",
  "query_by": "description",
  "offset": 1000,
  "limit": 10
}

Unlike Elasticsearch, applications don’t need to manage cursor state, PIT IDs, or sort keys between requests.

While extremely large offsets still require additional work internally, as they do in most search engines, in practice, most user-facing search experiences operate well within reasonable pagination depths, making this simpler API a natural fit.

Protecting Against Excessive Pagination

Even if your search engine supports deep pagination efficiently, allowing clients to page indefinitely can expose your application to large-scale scraping or unnecessary resource consumption.

Typesense provides a built-in limit_hits parameter that limits how many documents a client can access through pagination.

To prevent clients from bypassing this restriction, you can embed limit_hits directly into a Scoped API Key generated on your backend.

const scopedApiKey = client.keys().generateScopedApiKey(
  searchOnlyApiKey,
  {
    limit_hits: 1000
  }
);

Any searches performed with this scoped key will only be able to paginate through the first 1,000 documents, without requiring additional application logic.

Elasticsearch vs. Typesense for Deep Pagination

Conclusion

Elasticsearch’s search_after + Point in Time (PIT) approach is designed to handle deep pagination efficiently while maintaining a consistent view of a changing index. It’s an excellent fit for large-scale data exports, synchronization pipelines, and other workloads where scalability and consistency are critical.

For user-facing search experiences, however, managing PIT lifecycles and cursor state can add unnecessary complexity. Typesense offers a simpler, page-based pagination API that feels more natural for interactive search applications.

Ultimately, both approaches are optimized for different use cases. The right choice depends on the type of search experience you’re building and the trade-offs you’re willing to make.