Fix Flink Watermarks Not Advancing in Streaming Jobs (2026)

The Flink job’s StreamTask failed to emit watermarks because the source operator stopped producing events, leaving no new timestamps to process.

This usually happens when the source operator itself is stuck, waiting for upstream data or encountering an internal error that halts its run() method. Since Flink relies on these emitted watermarks to track event time progress and trigger window computations, a stalled source effectively freezes the entire job’s time progression.

Here are the most common reasons Flink watermarks stop advancing and how to fix them:

Source Operator Deadlock or Hang

Diagnosis: Check the logs of your Flink TaskManager. Look for repeated log messages from the source operator indicating it’s waiting indefinitely for data, or for any exceptions occurring within the source’s run() or collect() methods. You can also use the Flink UI to inspect the source operator’s subtask, looking for long-running or stuck tasks.

Cause: The source operator might be blocked waiting for an external system (e.g., Kafka partition, database cursor) that is itself stalled or has encountered an error. This can also happen if the source is implementing custom logic that leads to a deadlock.

Fix:

1. Restart the source operator: If the Flink UI shows the source subtask as running but not making progress, you can try to restart it. In the Flink UI, navigate to the job, then to the specific operator, and find the "Restart" option for the relevant subtask. This will re-initialize the source.
2. Investigate the upstream system: If restarting doesn’t help, the problem lies outside Flink. For Kafka sources, check Kafka broker health, partition leadership, and consumer group offsets. For database sources, check the database connection, query performance, and any locks.
3. Add a timeout to source fetching: If your source allows it, configure a reasonable timeout for fetching data. For example, if using KafkaSource, ensure your Kafka client configurations (e.g., request.timeout.ms) are appropriate. If you’re writing a custom source, implement a mechanism to break out of blocking calls after a certain period.

Why it works: Restarting forces the source to re-establish its connection to the upstream system and re-initialize its state. Investigating the upstream system addresses the root cause of the data starvation. Adding timeouts prevents indefinite blocking.

Insufficient Parallelism or Bottlenecked Source

Diagnosis: Examine the Flink UI’s "Task Managers" and "Job Overview" sections. Look at the throughput (records per second) of your source operator compared to downstream operators. If the source is producing data but downstream operators are significantly slower, it can cause backpressure. However, if the source operator itself has very low throughput or is showing high CPU/memory usage and not keeping up with the expected data rate, it’s a bottleneck.

Cause: The source operator might not have enough parallel instances to handle the incoming data rate, or the single instance of the source operator is a performance bottleneck due to its own processing limitations.

Fix:

1. Increase source parallelism: If your source operator is configured with a parallelism of 1, and you have multiple partitions or data streams available, increase its parallelism to match the number of available input partitions or to a higher value that can handle the load. For example, if you’re reading from 8 Kafka partitions, set the source parallelism to at least 8. You’ll need to restart the job with a new job ID or use a savepoint.
2. Optimize source code: If the source is custom-written or uses complex logic, profile its performance. Identify and optimize any inefficient code, I/O operations, or CPU-intensive tasks within the source operator.

Why it works: Higher parallelism distributes the load across more task slots, allowing more data to be read concurrently. Optimizing the source code improves its intrinsic processing speed.

Serialization/Deserialization Errors in Source

Diagnosis: Scan Flink TaskManager logs for StackOverflowError, OutOfMemoryError, or specific deserialization exceptions (e.g., EOFException, InvalidClassException, JsonParseException) originating from the source operator.

Cause: The data being read by the source operator is malformed, corrupted, or in an unexpected format, causing serialization or deserialization failures. This can happen if the data schema changes upstream without Flink being updated, or if there are transient data corruption issues.

Fix:

1. Implement error handling in the source: For sources that can encounter malformed records, add robust error handling. This might involve skipping bad records, logging them for later analysis, or sending them to a dead-letter queue. For example, with KafkaSource, you can configure a DeserializationSchema with a failOnMissingField flag set to false or implement custom error handling within your deserialize method.
2. Cleanse or repair upstream data: If possible, identify and fix the source of the corrupted data in the upstream system.
3. Use a more resilient deserializer: If using a schema-based format like Avro or Protobuf, ensure your schema registry and Flink’s deserializer are compatible and handle schema evolution correctly.

Why it works: Graceful handling of bad records prevents the source operator from crashing or getting stuck. Fixing upstream data issues resolves the root cause of malformed input.

Source Operator Waiting for External System State

Diagnosis: Check the Flink UI for the source operator’s subtask. If it’s in a "RUNNING" state but the input queue is consistently empty and no records are being processed, it suggests it’s waiting for something external. Examine the TaskManager logs for messages indicating waiting for network responses, database locks, or specific external conditions.

Cause: The source operator might be waiting for a specific condition to be met in an external system, such as the completion of a batch job, a specific flag being set, or a file to appear. If that condition never materializes, the source will remain idle.

Fix:

1. Verify external system status: Manually check the status of the external system the source is dependent on. Ensure that the expected data or condition is indeed available.
2. Implement a timeout or retry mechanism: If the external system is prone to delays or occasional unavailability, add a timeout to the source’s polling or waiting logic. After a timeout, the source can either retry, report an error, or proceed with whatever data is available.
3. Adjust polling intervals: If the source is polling an external system, consider if the polling interval is too long, leading to perceived stagnation, or too short, causing excessive load.

Why it works: Verifying the external system confirms if the source is correctly waiting or if the external system is the problem. Timeouts and retries make the source more robust to transient external issues.

Flink Cluster Resource Starvation (Less Common for Source-Only Hang)

Diagnosis: While less likely to only affect watermark advancement without other symptoms, if the TaskManager running the source operator is severely starved of CPU or memory, it could lead to its run() loop becoming unresponsive, thus not emitting watermarks. Check TaskManager metrics for high CPU utilization, low available memory, or excessive garbage collection.

Cause: The Flink cluster nodes are overloaded, or the specific TaskManager running the source operator is undersized or experiencing issues that prevent its threads from executing properly.

Fix:

1. Scale Flink cluster: Increase the number of TaskManagers or the resources (CPU/memory) allocated to existing TaskManagers.
2. Adjust Flink JVM settings: Tune Flink’s JVM heap size (-Xmx, -Xms) and garbage collection settings for the TaskManagers.
3. Resource management: Ensure your cluster’s resource manager (e.g., YARN, Kubernetes) is correctly allocating resources to Flink TaskManagers.

Why it works: Providing sufficient resources ensures that the Flink processes, including the source operator’s threads, can execute without being starved, allowing them to process data and emit watermarks.

Once these issues are resolved, you should see your source operator processing data again, and consequently, Flink watermarks will start advancing. The next error you might encounter is related to downstream operator backpressure if they were also impacted by the watermark stall and are now struggling to catch up.