Articles

Fix Apache Flink Backpressure: Diagnose and Unblock Slow Operators (2026)

April 16, 2026 5 min read

Your Flink job is choking because a downstream operator can’t keep up with the data rate from an upstream one, causing a buildup of unacknowledged records and eventually halting progress.

Causes and Fixes

1. Network Bottleneck

  • Diagnosis: Monitor network traffic between TaskManagers. Use netstat -s on the affected nodes to check for TCP retransmissions or errors. Look for high latency in network tools like ping or traceroute between the nodes running the upstream and downstream tasks.
  • Fix:
    • Increase Network Bandwidth: If possible, upgrade your network infrastructure.
    • Optimize Data Serialization: Use more efficient serialization formats like Kryo or Avro instead of Java serialization. For example, in your Flink configuration, set pipeline.serialization-config.kryo-registrator-class: com.example.MyKryoRegistrator. This reduces the amount of data sent over the wire.
    • Adjust TCP Buffer Sizes: On Linux, increase net.core.rmem_max and net.core.wmem_max in /etc/sysctl.conf to 1310720 (1.25MB) and sudo sysctl -p to apply. This allows TCP to buffer more data, smoothing out temporary network fluctuations.
  • Why it works: Less data to send or more capacity to handle it directly addresses the physical limits of data transfer.

2. Insufficient Task Slots / Parallelism Mismatch

  • Diagnosis: Examine the Flink Web UI. Navigate to the "Job Graph" view and select the backpressured operator. Check the "Input" and "Output" metrics for this operator. If the input rate is consistently higher than the output rate, and the input buffer is filling up (indicated by a high "Input Queue Length"), this is a strong sign. Also, verify the parallelism of the upstream and downstream operators. If the downstream operator has fewer parallel instances than the upstream one, it’s a direct bottleneck.
  • Fix:
    • Increase Downstream Parallelism: In your Flink job submission, set setParallelism(N) for the downstream operator, where N is equal to or greater than the upstream operator’s parallelism. For example, dataStream.keyBy(0).map(...).setParallelism(4);
    • Increase Total Task Slots: Ensure your Flink cluster has enough available TaskManager slots to accommodate the increased parallelism.
  • Why it works: More parallel instances of the downstream operator can process incoming data concurrently, matching the rate of the upstream producer.

3. Slow Operator Logic (CPU Bound)

  • Diagnosis: Access the Flink Web UI for the backpressured operator. Look at the "CPU Usage" metric for the TaskManager instances running this operator. If CPU usage is consistently at or near 100%, the operator’s computation is too slow. You can also check the "Back Pressure" tab in the UI for details on which input/output channels are blocked.
  • Fix:
    • Optimize Code: Profile the operator’s code to identify performance bottlenecks. This might involve improving algorithms, reducing object creation, or using more efficient data structures.
    • Increase Parallelism (if applicable): If the operator’s logic can be parallelized (e.g., stateless Map or FlatMap operations), increase its parallelism as described in "Insufficient Task Slots."
    • Offload Computation: If the computation is heavy, consider offloading it to external services or using Flink’s state management more efficiently.
  • Why it works: Faster computation means the operator can process data quicker, reducing the backlog. More parallelism allows the work to be distributed.

4. State Backend Performance Issues

  • Diagnosis: If your operator uses state (e.g., KeyedState in ProcessFunction or RichAggregateFunction), its performance can be limited by the state backend. Check the Flink Web UI for metrics related to state access latency (e.g., state.get or state.put duration). If these are high, or if you’re using a disk-based state backend like RocksDBStateBackend and observing high disk I/O wait times on the TaskManager nodes, this is the culprit.
  • Fix:
    • Tune State Backend: For RocksDBStateBackend, ensure you have sufficient RAM for block cache ( RocksDBConfigurableOptions.setBlockCacheSize(128 * 1024 * 1024) in your job configuration) and that your disks are fast (SSD is recommended).
    • Use FsStateBackend with sufficient memory: If your state size is manageable and you have ample RAM, FsStateBackend can be faster for frequent small state accesses.
    • Optimize State Access Patterns: Reduce the frequency of state reads/writes, use batching where possible, and avoid iterating over large state sets.
  • Why it works: Faster access to and persistence of state allows operators to process records without being blocked by I/O or state management overhead.

5. Serialization/Deserialization Overhead

  • Diagnosis: Even with efficient serialization formats, the act of serializing and deserializing every record between operators can become a bottleneck, especially if records are small and numerous. Monitor CPU usage on TaskManagers for serialization-related threads.
  • Fix:
    • Use Flink's Internal Ser/De: If possible, avoid custom serializers and rely on Flink’s optimized internal serializers (Kryo, Avro) which are often faster.
    • Avoid Pojo Serialization: If using Kryo, explicitly register POJOs (env.getConf().get(KryoRegistrator.class, KryoRegistrator.class)). Unregistered POJOs can lead to slower, generic serialization.
    • Reduce Data Transferred: If you’re serializing large objects, consider only sending the necessary fields downstream.
  • Why it works: Minimizing the CPU time spent converting data to and from bytes reduces the overall processing time for each record.

6. Backpressure Due to Event Time Processing / Watermarks

  • Diagnosis: If your job uses event time and you’re seeing backpressure, it might be because watermarks are not advancing quickly enough, or they are advancing too quickly and causing downstream operators (especially those with complex windowing or ProcessFunction logic) to fall behind. Check the "Watermarks" section in the Flink Web UI for the current watermark and its progression.
  • Fix:
    • Tune Watermark Strategy: Adjust your AssignerWithPeriodicWatermarks or AssignerWithPunctuatedWatermarks to generate watermarks more aggressively or less aggressively based on your data’s characteristics and downstream needs. For example, new AscendingTimestampExtractor<MyEvent>() { @Override public long extractAscendingTimestamp(MyEvent element) { return element.getTimestamp(); } } with a periodic watermark interval of 1000L (1 second) might be too slow if your data has gaps.
    • Increase Parallelism: If windowing or processing logic is complex, increasing parallelism can help process events and finalize windows faster once watermarks arrive.
    • Handle Late Data: Ensure your job is configured to handle late data appropriately (e.g., using allowedLateness in windowing) so that a sudden influx of late events doesn’t cause downstream recomputation and backpressure.
  • Why it works: A steady and predictable watermark progression allows downstream operators to trigger computations and clear their buffers efficiently.

The next error you’ll likely encounter, if you haven’t already, is a OutOfMemoryError on the TaskManager running the backpressured operator, as its input buffers continue to fill indefinitely.

Want structured learning?

Take the full Flink course →