Articles

Fix Flink YARN Container Failures During Job Execution (2026)

April 16, 2026 5 min read

A Flink YARN container is failing because the YARN ResourceManager is killing it due to excessive memory usage, specifically when the Flink TaskManager tries to allocate more heap memory than it’s been assigned.

Here are the common culprits and how to fix them:

1. TaskManager Heap Size Too Low:

  • Diagnosis: Check YARN container logs for java.lang.OutOfMemoryError: Java heap space or similar memory allocation errors. You can also check the YARN UI for the specific container’s memory usage. If it’s consistently hitting the YARN-allocated limit, this is likely the issue.
  • Fix: Increase the taskmanager.memory.heap.size configuration value in your Flink flink-conf.yaml. For example, if it’s set to 1024m and you’re seeing failures, try 2048m.
    # In flink-conf.yaml
taskmanager.memory.heap.size: 2048m
  • Why it works: Flink’s TaskManagers run as JVM processes. This setting directly controls the maximum heap size available to the Java Virtual Machine. Increasing it allows the TaskManager to hold more data in memory, preventing OutOfMemoryError during processing.

2. YARN Container Memory Allocation Too Low:

  • Diagnosis: Examine the YARN container logs for messages indicating the container was killed by YARN. Look for messages like Exceeded memory limits or Killed by the ApplicationMaster. You can also check the YARN UI for the specific container’s allocated memory versus actual usage.
  • Fix: Increase the memory allocated to Flink TaskManagers via YARN’s container settings. This is typically done when submitting the Flink job using the Flink YARN client or by setting flink.yarn.container.memory in flink-conf.yaml if you’re using a YARN session.
    # Example using flink-yarn-session.sh
./bin/flink-yarn-session.sh -m 1 -t 2048m -c 2

# Or in flink-conf.yaml (if submitting via YARN client with these defaults)
# flink.yarn.container.memory: 2048
    (Note: The exact parameter might vary slightly depending on Flink version and submission method. For flink-yarn-session.sh, -t is typically for TaskManager memory.)
  • Why it works: YARN enforces resource limits. If the TaskManager tries to use more memory than YARN has allocated to its container, YARN will terminate the container to protect the cluster. This setting ensures YARN grants enough memory to the container for Flink to operate.

3. Excessive Network Buffers:

  • Diagnosis: Flink’s network stack also consumes memory. If you have very high throughput or large state that needs to be shuffled, the default network buffer allocation might be insufficient, leading to container OOMs even if the heap is technically okay. Look for errors related to buffer allocation failures or OutOfDirectMemoryError.
  • Fix: Increase taskmanager.memory.network.fraction or taskmanager.memory.network.min and taskmanager.memory.network.max. A common starting point if network buffers are suspected is to increase the fraction.
    # In flink-conf.yaml
taskmanager.memory.network.fraction: 0.2  # e.g., 20% of total Flink managed memory
taskmanager.memory.network.min: 128m
taskmanager.memory.network.max: 512m
  • Why it works: Flink uses managed memory for its network buffers, which are crucial for data transfer between TaskManagers. Increasing the allocated fraction or absolute min/max allows more memory for these buffers, preventing bottlenecks and potential OOMs during high network I/O.

4. Checkpointing Overhead:

  • Diagnosis: If failures occur consistently during or shortly after a checkpoint, it might be due to checkpointing activity. Large state or inefficient checkpointing mechanisms can temporarily spike memory usage.
  • Fix:
    • Increase TaskManager Heap: As mentioned in point 1, a larger heap can absorb these spikes.
    • Optimize Checkpointing:
      • Incremental Checkpointing: If using RocksDB state backend, ensure incremental checkpointing is enabled.
      • State Size: Analyze your state size and consider state TTL or state partitioning if it’s excessively large.
      • Checkpoint Timeout: Increase execution.checkpointing.timeout if checkpoints are taking too long and timing out, which can lead to retries and accumulated memory pressure.
    # In flink-conf.yaml
execution.checkpointing.timeout: 10m # e.g., 10 minutes
  • Why it works: Larger heaps provide more headroom for the temporary memory allocations that occur during state snapshotting. Optimizing checkpointing reduces the peak memory demand during these operations.

5. JVM Metaspace Exhaustion:

  • Diagnosis: While less common for container failures directly tied to heap, if your application loads many classes (e.g., heavy UDF usage, dynamic class loading), the JVM’s Metaspace can fill up. This typically results in java.lang.OutOfMemoryError: Metaspace.
  • Fix: Increase the taskmanager.memory.jvm-metaspace.size in flink-conf.yaml.
    # In flink-conf.yaml
taskmanager.memory.jvm-metaspace.size: 256m
  • Why it works: Metaspace stores class metadata. Increasing its size allows the JVM to load more classes without running out of memory.

6. High Watermark or Event Time Processing Spikes:

  • Diagnosis: If your job uses event time and processes large windows, a sudden influx of late events or a rapid advancement of watermarks can cause significant internal state changes and memory pressure. This is hard to diagnose directly from logs but often correlates with specific data patterns.
  • Fix:
    • Increase TaskManager Heap/Network Buffers: As above, more memory helps absorb these spikes.
    • Tweak Watermark Strategy: If possible, adjust your watermark generation to be less aggressive or use a BoundedOutOfOrdernessTimestampExtractor with a reasonable maxOutOfOrderness value.
    • parallelism.default: Ensure your parallelism is sufficient to process the incoming data rate.
  • Why it works: More memory provides buffer for event time processing queues and internal data structures. Controlling watermark advancement limits the suddenness of state updates.

The next error you’ll likely encounter after fixing memory issues is a java.lang.OutOfMemoryError: Direct buffer memory if you haven’t also addressed network buffer sizing, or a job failure due to a different, unrelated configuration issue.

Want structured learning?

Take the full Flink course →