← Back

−85%bandwidth

When compression made things bigger

~85% less Kafka data transfer

judgment · migration

Zipy's browser SDK ran on sixty to seventy customers' websites. It used rrweb to record what visitors did: clicks, scrolls, typing, and full pictures of the page. Every event became JSON, which is plain text, and went to Apache Kafka, the system our services used to pass data to each other.

Three things went wrong at the same time. Customers' own pages stuttered, because building and sending large text messages is slow work to do inside someone else's website. Our servers spent too much memory and CPU on every message. And the Kafka bill grew with every new customer, with no end to it.

Why Protobuf is smaller

I changed the format from JSON to Protocol Buffers.

The saving is not compression. It is that the field names disappear. JSON repeats every field name in every message, so "timestamp" and "user_id" get written thousands of times in one session. Protobuf replaces each name with a number that takes a single byte.

The part I got wrong

rrweb events come in very different sizes. A full page snapshot is huge. A click is tiny.

My first plan was to compress everything, and the numbers proved me wrong. Compression adds a small header to each message. On a click, that header cost more than the compression saved, so the small events came out bigger than before.

So I changed the rule. Protobuf for everything, compression only above a size threshold.

Nobody had to upgrade

Changing the format took a few days. The hard part was that I could not make customers upgrade.

We shipped the SDK two ways. Through npm, where customers pick a version and can sit on it for years. And through a CDN, where everyone gets the new version immediately.

So I ran both formats at once. Old SDKs kept sending JSON and our services kept reading it. New SDKs sent Protobuf. Every message carried a marker naming its format, so the reader always knew what it was looking at. There was no switch-over day, and turning it off again was a config change rather than a release.

Measuring it was a separate problem. I could not compare production before and after, because the mix of customers shifts every day and I would have been comparing two different populations. So I took real recorded events, ran them through both versions in a test environment, and compared the same input twice.

That gave about 85% less data through Kafka on our real traffic mix. Page snapshots came out roughly 85% smaller and clicks about 55% smaller. The stutter on customer pages stopped and the servers calmed down.

Binary is unreadable

That is the cost, and it is a real one. You need the schema file to make sense of a message. So the schema has to stay in step across services, and I gave up being able to just look at a payload while debugging. We built a decode tool for developers, which helps, but it is not the same thing.

If I did this again I would set up a schema registry from the start. Ours worked on convention, which was fine while one team owned both ends of the pipe. The day a second team starts publishing to that topic, it stops being fine.