Couchbase nodes can go out of memory (OOM) and get killed by the OS, and the most common culprit is misconfigured memory quotas.
Cause 1: couchstore Memory Quota Too High
Diagnosis:
Check your node’s memory quotas via the Couchbase UI (Settings -> Memory) or the couchbase-cli command:
couchbase-cli memcached-tool <node_ip>:<port> --settings -o
Look for the couchstore quota. A common misconfiguration is setting it too high, leaving insufficient memory for other critical services.
Fix:
Reduce the couchstore quota. A good starting point is to allocate 25-30% of available RAM to couchstore. For a 32GB node, this might be around 8192 MB (8GB).
couchbase-cli memcached-tool <node_ip>:<port> --settings -i couchstore=<new_quota_in_MB>
Why it works: The couchstore process is responsible for reading and writing data to disk. If this quota is too large, it can consume a disproportionate amount of RAM, starving other essential Couchbase services like the KV engine.
Cause 2: index Memory Quota Too High
Diagnosis:
Similar to couchstore, check the index memory quota using the same memcached-tool command. High index memory can be a silent killer.
Fix:
Adjust the index quota. The ideal size depends heavily on your indexing strategy and data volume. Start by allocating 10-20% of RAM. For a 32GB node, 3072 MB (3GB) might be a reasonable initial value.
couchbase-cli memcached-tool <node_ip>:<port> --settings -i index=<new_quota_in_MB>
Why it works: The indexer service uses RAM to build and maintain indexes. Over-allocating here can lead to excessive memory consumption, especially during index creation or rebalancing.
Cause 3: Insufficient kv Memory Quota
Diagnosis:
The KV (Key-Value) cache is where Couchbase stores frequently accessed data in RAM for fast retrieval. Check its quota. If it’s too low, performance will suffer, and it might indirectly contribute to OOM by forcing more disk I/O and couchstore activity.
Fix:
Increase the kv quota. This is often the largest allocation, typically 50-70% of available RAM. For a 32GB node, 18432 MB (18GB) is a common starting point.
couchbase-cli memcached-tool <node_ip>:<port> --settings -i kv=<new_quota_in_MB>
Why it works: A larger KV cache means more data can be served directly from RAM, reducing the need to read from disk. This not only improves read performance but also lessens the load on couchstore.
Cause 4: n1ql Memory Quota Too High
Diagnosis: The N1QL query service also consumes memory for query execution, compilation, and caching. Check its quota.
Fix:
Reduce the n1ql quota if it’s set excessively high. For most workloads, 512MB to 2048MB is usually sufficient.
couchbase-cli memcached-tool <node_ip>:<port> --settings -i n1ql=<new_quota_in_MB>
Why it works: While N1QL needs memory to operate, an overly generous quota can be a wasteful use of RAM that could be better allocated to the KV cache or couchstore.
Cause 5: Swap Enabled on Nodes
Diagnosis:
Couchbase nodes should never have swap enabled. Check swapon --show on the node’s operating system. If any swap partitions or files are listed, swap is active.
Fix: Disable swap immediately.
sudo swapoff -a
# To make it permanent, edit /etc/fstab and comment out the swap line
Why it works: When the OS needs more memory than available, it starts using swap space on disk. This is orders of magnitude slower than RAM and causes extreme performance degradation. More critically, Couchbase’s memory management can be severely disrupted, leading to instability and OOM kills.
Cause 6: Other Services Consuming RAM
Diagnosis:
Use htop or top on the node’s OS to see what other processes are consuming memory. Sometimes, external agents or misconfigured applications on the same host can hog resources.
Fix: Identify and stop or reconfigure the offending processes. Ensure Couchbase is the primary application running on its dedicated nodes. Why it works: Couchbase needs to be able to rely on the allocated RAM. If other processes are stealing memory, Couchbase’s own memory management will fail, leading to OOM conditions.
Cause 7: Insufficient Total System RAM
Diagnosis: Even with perfect quota configuration, if the total RAM on the node is insufficient for the configured quotas plus the OS and other essential services, OOMs will occur. Sum up all your configured Couchbase memory quotas and add at least 2-4GB for the OS and other processes. If this sum exceeds your total physical RAM, you have a problem.
Fix: Add more RAM to the nodes or reduce the total configured Couchbase memory across all services. Why it works: There’s a hard limit to how much memory a machine has. If the total demand exceeds supply, the OS will eventually kill processes to reclaim memory.
After adjusting these quotas, you might encounter issues with memcached connections if you’ve set the kv quota too high or too low relative to the number of vBuckets.