Save 25% or More on Your Kafka Costs | Take the Confluent Cost Savings Challenge

Unknown Magic Byte! How to Address Magic Byte Errors in Apache Kafka

Watch demo: Kafka streaming in 10 minutes

Get started with Confluent, for free

Written By

If you work with Kafka Streams, Apache Kafka® clients, and Schema Registry, you’ve likely come across this error: 

Caused by: org.apache.kafka.common.errors.SerializationException: Unknown magic byte!

This error might be frustrating and mysterious at first, but hopefully, after you review this blog post, you’ll feel more confident to handle it. 

 What is a magic byte?

Magic byte is the first few bits of data in a file, meant to help identify the file content. They are also known as “magic numbers.” So, what causes the “unknown magic byte” error in Kafka Streams? 

In general, there’s an essential problem within the publish/subscribe data architecture pattern: How do you make sure that the data formats of the publishers (in Kafka’s case, producers) match the data formats of the subscribers (in Kafka’s case, consumers)? 

Kafka solves this problem with a Schema Registry. The Schema Registry lives outside of the servers that host the producers and consumers, and topics. Messages are validated by the schema before being sent. Here’s an illustration from Confluent documentation that shows how it works:

Schemas are available in three formats: Avro, Protobuf, and JSON. The Schema Registry reads the file signature to determine the format. If it doesn’t match, the dreaded “unknown magic byte” error is thrown. 

How to address “unknown magic byte”

Generally, the “unknown magic byte” error means that you must reconcile serialization methods and check that your schemas use the same format on the production and consumption ends. You might have to do this in the client, in ksqlDB, or in Kafka Streams, depending on your project. 

I ran into the error myself recently in a Shakespeare app that I created. I was pulling data from the Folger Shakespeare API, and trying to join two topics created with the API results in ksqlDB. I had never implemented a Schema Registry in a client before, so I didn’t realize that it was necessary when you use ksqlDB. Nothing showed up in my streams after I created them! 

Luckily, ksqlDB has something called a “processing log,” which is a stream of metadata about your instance. Querying it resulted in an error with an “unknown magic byte” string. 

Turns out, when you work with ksqlDB in Confluent Cloud, and you create a stream, you need to include the value format so the value can be deserialized:


Since I created the stream using the “AVRO” format, I needed to set up an Avro schema in my client using confluent-schema-registry. You can view the full code that I added on my GitHub file page, but here’s the gist:

const registry = new SchemaRegistry({
 host: "",
 auth: {
   username: `${process.env.SCHEMA_USERNAME}`,
   password: `${process.env.SCHEMA_PASSWORD}`,

const schema = `
...schema definition here

const { id } = await registry.register({
 type: SchemaType.AVRO,

Magic byte tutorial

If you’re still curious about magic bytes, the following tutorial helps to solidify the concept of a magic byte. 


1. `git clone && cd magic-byte-illustration`

2. Now view in your text editor. It will look like the following:

3. PK is the file signature, or magic byte, for the zip file format. You can verify it by running `file`.

4. Let's change the file signature bytes so that `file` reads this file as a PDF.

Erase “PK” on line 1 and replace it with “%PDF”.

5. Run `file` to confirm. Note that while the extension is still ".zip", the file signature is for a PDF, so it's identified as a PDF. Pretty cool huh?


This post uses Node.js for the solution; but if you’re looking to resolve it in other languages, you can take a look at the documentation for the Java client, or ask about it in the Confluent Community Slack for other languages: 

Watch demo: Kafka streaming in 10 minutes

Get started with Confluent, for free

Did you like this blog post? Share it now