Size Elasticsearch Shards to Avoid Oversharding and Hotspots (2026)

The true cost of oversharding Elasticsearch isn’t just more disk space, it’s the exponential increase in network traffic and CPU contention that grinds your cluster to a halt.

Let’s see this in action. Imagine a search query hitting a cluster with 10,000 shards. Each shard needs to be contacted, its local results aggregated, and then sent back to the coordinating node. That’s 10,000 individual network requests and 10,000 potential CPU contexts for the search thread pool. Now, imagine that same query on a cluster with 100 shards. The difference is stark.

Here’s a simplified view of how Elasticsearch handles a search request:

Coordinating Node: Receives the search request.
Shard Routing: Determines which shards (primary and replica) hold the data relevant to the query.
Request Distribution: Sends the search request to all relevant primary and replica shards across the cluster.
Shard Execution: Each shard executes the search query locally, fetching matching documents.
Result Aggregation: Shards send their partial results back to the coordinating node.
Final Result: The coordinating node merges and sorts these partial results to form the final response.

The problem arises when the number of shards balloons. Each shard requires memory for its file handles, its shard state, and its thread pools. A cluster with too many shards, even if the total data size is manageable, can exhaust system resources. This leads to:

Increased Indexing Latency: More shards mean more overhead for each document ingested. The cluster spends more time managing shard metadata and distributing documents.
Slower Search Performance: As seen above, coordinating nodes have to manage an overwhelming number of shard requests, leading to significant latency.
Hotspots: If shards for a particular index are not evenly distributed across nodes, or if a few nodes are responsible for a disproportionately large number of shards, those nodes become overloaded. This is a "hotspot," where a few nodes do all the heavy lifting, impacting cluster stability and performance.

The core principle for avoiding oversharding and hotspots is to balance the number of shards with the amount of data they hold, and to ensure even distribution.

Key Levers for Control:

index.number_of_shards: This is the primary setting you control when creating an index. It defines the total number of primary shards for that index. Once set, it cannot be changed without reindexing.
index.number_of_replicas: This determines how many copies of each primary shard are maintained for fault tolerance and search scalability. While important for availability, it also doubles (or triples, etc.) the number of shards the cluster needs to manage.
Node Capacity: The CPU, RAM, and disk I/O of your individual nodes directly influence how many shards they can healthily host.
Data Volume per Shard: A common guideline is to aim for shards that are between 10GB and 50GB in size. Smaller shards incur more overhead; larger shards can make recovery and rebalancing slower.

Diagnosing Oversharding:

Cluster Health: Run GET _cluster/health?pretty. Look for status (should be green or yellow). If it’s red, you have unassigned shards, often a sign of problems.
Shard Count per Node: Use GET _cat/shards?v=true&h=index,shard,prirep,state,unassigned.reason,node,bytes and filter or sort by node to see how many shards and how much data each node is hosting. Look for nodes with significantly more shards or data than others.
Index Shard Count: GET _cat/indices?v=true&h=index,health,status,uuid,pri,rep,docs.count,store.size will show you the total number of primary and replica shards per index. If an index has thousands of shards, it’s a red flag.
Node Statistics: GET _nodes/stats/indices/shard_stats,thread_pool,fs,jvm?pretty provides detailed stats per node, including file counts, thread pool usage (search, index, merge), and disk usage. High thread pool rejection counts or excessive file descriptors are indicators of stress.

Fixing Oversharding & Hotspots:

Reindex to a New Index with Fewer Shards: This is the most common and effective solution.
1. Create a New Index: Define a new index with an appropriate number of shards. For example, if you have 1TB of data and want shards around 30GB, you’d aim for roughly 1000GB / 30GB = ~33 primary shards.
```
PUT my_new_index
{
  "settings": {
    "index": {
      "number_of_shards": 32,
      "number_of_replicas": 1
    }
  }
}
```
2. Run the Reindex API: Copy data from the old, oversharded index to the new one.
```
POST _reindex
{
  "source": {
    "index": "my_old_index"
  },
  "dest": {
    "index": "my_new_index"
  }
}
```
3. Verify and Switch: Once reindexing is complete and verified, update your application to point to my_new_index and then delete my_old_index.
Shrink API (for time-based indices): If your oversharding is due to many small indices (e.g., daily indices with very little data), you can use the _shrink API to merge them into a single, larger index with fewer shards. This is particularly useful for log data.
1. Create a Shrink Configuration: Define a new index with fewer shards.
```
PUT /my_shrunk_index
{
  "settings": {
    "index.number_of_shards": 5,
    "index.number_of_replicas": 1
  }
}
```
2. Execute Shrink:
```
POST /my_old_index/_shrink/my_shrunk_index
{
  "settings": {
    "index.number_of_shards": 5
  }
}
```
Why this works: The Shrink API effectively creates a new index from segments of an existing index, allowing you to consolidate many small indices into fewer, larger ones.
Force Merge API (to reduce segment count, not shard count): While not directly reducing shard count, force merging can reduce the number of segments within shards. Too many small segments can negatively impact search performance and increase memory overhead.
```
POST /my_index/_forcemerge?max_num_segments=1
```
Why this works: This consolidates segments within shards, reducing the overhead of searching across many small files. It’s a background process and can be resource-intensive, so run it during off-peak hours.
Adjusting index.routing.allocation.total_shards_per_node: If you have a hotspot, this setting can help by limiting the maximum number of shards allowed on a single node. This forces Elasticsearch to distribute new shards more evenly.
```
PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.total_shards_per_node": 1000
  }
}
```
Why this works: This is a cluster-wide setting that acts as a safety valve, preventing any single node from becoming overloaded with too many shards, thus mitigating hotspots.
Increase Node Resources: If your nodes are consistently hitting resource limits (CPU, RAM, disk I/O) even with a reasonable shard count, you may need to scale up your hardware or add more nodes to the cluster.

The next problem you’ll likely encounter is understanding how segment merging impacts query performance.