Process Events for Real-Time Analytics Pipelines (2026)

The core problem with real-time analytics is that the data is never truly "real-time" without a system that can process events as they happen, not in batches.

Let’s look at a simplified Kafka-to-Spark streaming pipeline. Imagine we’re tracking user clicks on a website and want to see the most popular pages in the last 5 minutes.

// Spark Streaming application (Scala)

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._
import org.apache.spark.sql.streaming.Trigger

object RealTimeAnalytics {
  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder()
      .appName("RealTimeAnalytics")
      .master("local[*]") // In production, this would be a cluster manager
      .getOrCreate()

    // Define the schema for incoming click events
    val schema = new org.apache.spark.sql.types.StructType()
      .add("timestamp", "timestamp")
      .add("userId", "string")
      .add("pageUrl", "string")

    // Read from Kafka
    val kafkaStreamDF = spark.readStream
      .format("kafka")
      .option("kafka.bootstrap.servers", "localhost:9092") // Your Kafka brokers
      .option("subscribe", "click_events") // The Kafka topic
      .load()

    // Parse the JSON value from Kafka
    val clicksDF = kafkaStreamDF.selectExpr("CAST(value AS STRING)")
      .select(from_json(col("value"), schema).as("data"))
      .select("data.*")

    // Filter for clicks in the last 5 minutes
    val fiveMinutesAgo = current_timestamp() - expr("interval 5 minutes")
    val recentClicksDF = clicksDF.filter(col("timestamp") >= fiveMinutesAgo)

    // Count page views per URL
    val pageCountsDF = recentClicksDF
      .withWatermark("timestamp", "10 minutes") // Allow for late data up to 10 minutes
      .groupBy(window(col("timestamp"), "5 minutes", "1 minute"), col("pageUrl"))
      .agg(count("*").as("views"))

    // Write results to the console (for demonstration)
    val query = pageCountsDF.writeStream
      .outputMode("update") // Or "append" if you only want new counts
      .format("console")
      .option("truncate", "false")
      .trigger(Trigger.ProcessingTime("10 seconds")) // Process new data every 10 seconds
      .start()

    query.awaitTermination()
  }
}

In this example, Spark is reading directly from Kafka. Each incoming message (a user click) is treated as an individual event. The window function with 5 minutes and 1 minute slide means we’re calculating counts for 5-minute intervals that advance every 1 minute. The ProcessingTime("10 seconds") trigger tells Spark to check for new data and perform computations at least every 10 seconds. This is the essence of event processing: breaking down continuous data streams into actionable, time-bound insights.

The system solves the problem of "stale data" by processing events individually or in micro-batches as they arrive, rather than waiting for large batches to accumulate. This allows for near-instantaneous analysis. The core internal mechanism is Spark Streaming’s micro-batching or continuous processing model, where data is read from a source (like Kafka), transformed through a series of operations, and then outputted to a sink. The Trigger defines how often these micro-batches are processed, and Watermarking handles out-of-order or late-arriving data by defining a threshold beyond which events are dropped.

The key levers you control are:

• Source Configuration: How you connect to your streaming source (e.g., Kafka bootstrap servers, topic, consumer group).
• Schema Definition: How you parse your incoming event data.
• Windowing and Aggregation: Defining the time windows (tumbling, sliding, session) and the aggregation functions (count, sum, avg) to derive insights.
• Watermarking: Setting the maxLate duration to manage how much late data you tolerate. This is crucial for accuracy when event ordering isn’t guaranteed.
• Triggers: Defining the frequency of processing (ProcessingTime, Once, Continuous). This directly impacts latency and resource utilization.

The one thing most people don’t realize is how aggressively Spark might drop data if your watermark is too small relative to the actual lateness of your data, even if your trigger interval is very short. If your Kafka producer is slow, or network latency is high, events arriving 5 minutes late might be processed by a window that has already been dropped by Spark if your withWatermark("timestamp", "4 minutes") is set too low. The watermark isn’t just about when Spark finishes processing a window; it’s about if Spark will even consider data for a window that has already passed its logical end time.

The next concept you’ll grapple with is handling stateful operations across complex event sequences, not just simple counts.