Context
Zipy's browser SDK ran on sixty to seventy customers' production websites. It used rrweb to record what users did — full DOM snapshots plus a continuous stream of clicks, scrolls, mutations and input changes. Every event was serialized as raw JSON and shipped to Kafka.
The problem
Three symptoms showed up at once, which is what made it a real problem rather than a nice-to-have:
- Jank on customers' own pages. Serializing and sending large JSON payloads caused rendering issues on busy sites — someone else's product, felt by their users.
- Producer load. Large string payloads cost disproportionate memory and CPU on every publish.
- Cost scaling linearly with growth. Every new customer made Kafka bandwidth, and the bill, bigger.
What I did
I moved serialization from JSON to Protocol Buffers across the pipeline. The saving isn't really compression — it's that the field names disappear. JSON writes every key in full in every message; Protobuf writes a one-byte tag and encodes small integers as varints.
Where my first plan was wrong
rrweb events vary wildly in size. A full snapshot is enormous; a click is tiny. My first approach applied compression uniformly on top of Protobuf, and the data disproved it: on small events the compression header cost more than it saved, so clicks got bigger.
I switched to a per-event-type strategy — Protobuf everywhere, compression only above a size threshold.
The decision that mattered
The encoding took days. The hard constraint was that I couldn't force customers to upgrade. The SDK shipped through npm, where customers pin versions indefinitely, and a CDN, where they auto-update immediately.
So I ran both formats in parallel. Old SDKs kept producing JSON and old consumers kept reading it; new SDKs produced Protobuf. A version marker in the message envelope told consumers which format they were reading. No customer had to upgrade, there was no coordinated cutover, and rollback was a config flip.
To measure honestly, I didn't diff production numbers — the customer mix shifts as CDN users update and npm users don't. I replayed real DOM event samples through an isolated producer–consumer pipeline in QA for a clean before/after on identical inputs.
What it cost
The binary is unreadable without the schema. I took on schema coordination between producer and consumers and gave up wire readability when debugging production. We kept a decode path in dev tooling, but losing the ability to just read a payload is a real cost.
Outcome
- ~85% less Kafka data transfer, weighted by the real traffic mix
- Snapshots ~85% smaller, clicks ~55% smaller
- Client-side rendering issues resolved; producer load down
- Lower inter-AZ and egress transfer, and less broker storage under retention
What I learned
Migration design matters more than migration execution. And measure a plan before trusting it — uniform compression looked right on paper. If I did it again, I'd bring in a schema registry earlier; convention-based schemas are the first thing to break once a second team produces to the same topic.