Evolve Event Schemas Without Breaking Existing Consumers (2026)

The most surprising truth about evolving event schemas is that you can usually break the "never break consumers" rule – you just have to break it smartly.

Let’s say you’re using Kafka and have an event schema for UserCreated.

{
  "type": "object",
  "properties": {
    "userId": { "type": "string" },
    "email": { "type": "string" },
    "timestamp": { "type": "integer" }
  },
  "required": ["userId", "email", "timestamp"]
}

A new requirement comes in: we need to track the signupSource (e.g., "web", "mobile_ios", "referral"). If you just add "signupSource": { "type": "string" } to your schema and deploy, any consumer that strictly validates against the new schema will start failing because it’s expecting signupSource and older messages won’t have it.

Here’s how we make it work. The key is "backward compatibility" and "forward compatibility," often managed by a schema registry.

Backward Compatibility: New code (producer) can write data that old code (consumer) can understand. Forward Compatibility: Old code (producer) can write data that new code (consumer) can understand.

For event schemas, especially in streaming systems like Kafka, the common approach is to enforce backward compatibility at the schema registry level. This means a new schema must be backward compatible with the previous one.

The Schema Registry (e.g., Confluent Schema Registry)

This is the central authority for your schemas. Producers register their schema versions, and consumers fetch the schema they expect to parse a message.

Scenario: Adding an Optional Field

Let’s evolve our UserCreated event. We want to add signupSource.

1. Define the New Schema:

{
  "type": "object",
  "properties": {
    "userId": { "type": "string" },
    "email": { "type": "string" },
    "timestamp": { "type": "integer" },
    "signupSource": { "type": "string" } // New field
  },
  "required": ["userId", "email", "timestamp"] // signupSource is NOT required
}

2. Register the New Schema:

You’d use your schema registry’s API or UI to register this new version. The registry will check if it’s compatible with the previous version (schema ID 1, in our example). Adding an optional field is generally backward compatible.

3. Update Producers:

Your producers that generate UserCreated events need to be updated to include signupSource if available. If it’s not available for a particular event, they should omit it.

Example Producer Logic (Conceptual):

def publish_user_created(user_data):
    schema_id = get_latest_schema_id("user_created") # Fetches schema ID from registry
    schema = get_schema_by_id(schema_id) # Fetches schema content

    event_payload = {
        "userId": user_data["id"],
        "email": user_data["email"],
        "timestamp": int(time.time()),
    }
    if "source" in user_data:
        event_payload["signupSource"] = user_data["source"]

    serialized_event = serialize_with_schema(event_payload, schema)
    kafka_producer.send("user_events", serialized_event)

4. Consumers Adapt:

• Old Consumers: These consumers were written against the schema without signupSource. They will happily continue to process messages. When they encounter a message with signupSource, their deserializer will simply ignore the unknown field, as is standard behavior for most JSON/Avro parsers when a field isn’t explicitly handled.
• New Consumers: These consumers are updated to expect signupSource. They will correctly parse messages with and without the field.

The Crucial Point: Optional Fields

The magic here is making the new field optional. In JSON Schema, this is done by not including the new field in the required array. If you were using Avro, you’d provide a default value for the new field, which also makes it optional and ensures backward compatibility.

Scenario: Renaming a Field

Let’s say we want to rename email to contactEmail. This is trickier because simply renaming a field breaks backward compatibility.

Original Schema (v1):

{
  "type": "object",
  "properties": {
    "userId": { "type": "string" },
    "email": { "type": "string" },
    "timestamp": { "type": "integer" }
  },
  "required": ["userId", "email", "timestamp"]
}

Attempt 1 (Breaks Backward Compatibility):

{
  "type": "object",
  "properties": {
    "userId": { "type": "string" },
    "contactEmail": { "type": "string" }, // Renamed
    "timestamp": { "type": "integer" }
  },
  "required": ["userId", "contactEmail", "timestamp"]
}

If you register this as v2 and producers start sending contactEmail, old consumers expecting email will fail.

The Solution: Add New, Deprecate Old

You don’t rename in place. You add the new field and keep the old one for a transition period.

New Schema (v2):

{
  "type": "object",
  "properties": {
    "userId": { "type": "string" },
    "email": { "type": "string" }, // Keep for backward compatibility
    "contactEmail": { "type": "string" }, // New field
    "timestamp": { "type": "integer" }
  },
  "required": ["userId", "email", "timestamp"] // Still required for old consumers
}

1. Register v2: This schema is backward compatible with v1.
2. Update Producers: Producers now send both email and contactEmail. For example, if the new email is test@example.com, they’d send:

   {
     "userId": "abc-123",
     "email": "test@example.com", // For old consumers
     "contactEmail": "test@example.com", // For new consumers
     "timestamp": 1678886400
   }
   

3. Update Consumers: New consumers are updated to read contactEmail. They can still read email for compatibility with older messages.
4. Deprecation: Over time, as you confirm all consumers are updated, you can:
   • Stop producing the email field.
   • Eventually, create a v3 schema that removes email and makes contactEmail required (or just removes email if it wasn’t required).

What if you need to make a field required that wasn’t before?

This is the hardest case for breaking compatibility. The standard advice is to not do this if possible. If you absolutely must:

1. Add the field as optional first (as shown in the signupSource example).
2. Update producers to send the field with a default value or null if it’s not available.
3. Update consumers to handle the optional field.
4. Migrate data: If possible, run a separate process to backfill the new required field for existing data.
5. Change schema to required: Finally, register a new schema version where the field is marked as required. This will only work for new messages produced after this change and consumed by consumers that understand the new schema. Old consumers will continue to work on old messages.

The core principle is that schema evolution is a coordinated dance between producers and consumers, guided by a schema registry that enforces rules. The most common and safest evolution is adding optional fields. More complex changes require a phased approach with temporary redundancy.

The next challenge you’ll face is how to handle complex data types within your schemas, like nested objects or arrays, and maintain compatibility there.