🔥 The DB Grill 🔥

Where database blog posts get flame-broiled to perfection

Reduce data transfer and storage (DTS) costs in Elastic Cloud
Originally from elastic.co/blog/feed
October 21, 2021 • Roasted by Jamie "Vendetta" Mitchell Read Original Article

Ah, another dispatch from the front. It’s just so heartwarming to see the old team finally getting around to these… enhancements. I read this with a real sense of pride.

It’s fantastic that they’re tackling Data Transfer and Storage costs. I vividly recall conversations where the monthly cloud bill for a single large customer looked more like the GDP of a small island nation. To see that now being addressed as a feature is just… chef’s kiss. For years, the unofficial motto was "if the customer is complaining about the bill, they're using it correctly." It’s wonderful to see that evolving.

And data relocation via snapshots! Truly groundbreaking. I remember the old recovery process, which was a bit more… artisanal. It mostly involved a series of frantic Slack messages, a shell script that one of the original engineers wrote on a dare back in 2016, and a whole lot of hoping the customer wouldn't check their uptime monitor for the next 72 hours. To have this formalized into something that doesn't require a blood sacrifice is a huge step forward for the SRE team's collective sanity.

...compression on indexing data...

Now this one, this is my favorite. The idea of adding compression to the indexing pipeline was on a whiteboard somewhere since the beginning, I'm sure of it. It was usually filed under "Ambitious Q4 Goals" right next to "Achieve Sentience" and "Fix Timestamps."

Seeing it live is a real testament to engineering focus. I’m certain they managed to implement this with absolutely no impact on indexing latency or query performance. They definitely didn't have to, say, rewrite the entire storage engine twice or quietly increase the recommended instance size to compensate. No, I'm sure it was a clean, simple project.

It all ladders up to the promise of lower or more predictable Elastic Cloud bills. Predictability is a great north star. It’s a refreshing change from the previous billing model, which I believe was based on a Fibonacci sequence tied to the number of support tickets filed that month. Customers will be so relieved to know their bill will now be predictably high.

Honestly, this is inspiring. It’s great to see the company tackling these foundational issues and presenting them as dazzling new innovations. Keep up the great work, everyone. Can't wait to see what you invent next. Maybe ACID compliance? One can dream.