Druid Summit Lakehouse Panel: A Deep Dive into Data Lakehouses and Apache Druid
At the inaugural in-person Druid Summit this past October, industry leaders gathered to explore the future of data, streaming analytics, and more. In these panels industry experts answered questions about streaming analytics, operations, optimization, real-time user interfaces, and data lakehouses; all in the context of Apache Druid.
I had the honor of moderating the Lakehouse panel, which is near and dear to my heart, after spending the better part of a year working with the original creators of Apache Iceberg at Tabular. This panel featured Atul Mohan, Gian Merlino, Will Xu, and Abishek Balaji Radhakrishnan. This panel was a perfect cap to two earlier sessions. Abishek and Venki Korukanti presented on ingestion of Delta Lake tables into Druid, and Atul presented on Apache Iceberg integration with Druid.
If you missed the session, the full recording is available on the Druid Summit site, but here are some key highlights worth noting:
Metadata and Unified Schemas
Atul kicked things off by discussing the use of a metadata platform like DataHub. This enables a unified schema catalog that spans Druid and data lake tables, making it easier for organizations to manage and query their data effectively.
Query Engine Enhancements
Gian and Will shared insights into the latest query engine enhancements for Druid. Druid is rapidly evolving to become an exceptional tool for querying not only Druid-native data but also data within lakehouse environments. It’s exciting to see how Druid is bridging the gap between traditional databases and modern data lakes.
The Philosophical and Technical Challenges of Deletes
The panel took an intriguing turn when the topic of deletes in Druid and lakehouse tables was brought up. This sparked a fascinating technical and philosophical discussion. Will Xu shared an intriguing technique used by Twitch to handle deletes in a way that ensures efficiency—though I won’t spoil the details here, you’ll definitely want to watch the video to get the full details.
Querying Lakehouse Tables Directly with Druid
A particularly exciting bit of news from the panel was the discussion around querying lakehouse tables directly with Druid, bypassing the need to pull data into Druid segment files. While this feature almost made it into Druid 31, there are still a few bugs to iron out. But don’t worry—this looks like it will be part of an upcoming release.
Roadmap Insights and Future Enhancements
The panel also offered a sneak peek into future enhancements to the Druid storage layer, along with other roadmap updates While the details were still under wraps, these insights gave the audience a sense of the exciting direction Druid is heading in. The knowledge shared by these experts was invaluable, sparking plenty of inspiration and discussion.
The Blurring of Lines Between Streaming and Batch
One question raised during the panel really got me thinking: Could Druid be the next great lakehouse? Traditionally, data lakes, warehouses, and lakehouses were designed for batch processing, while Druid has always excelled in the streaming space. But recent advancements in the data space are blurring the lines between batch and streaming workloads.
- Apache Iceberg, a table format designed for cloud object stores, has significantly reduced the latency of querying data.
- Stream-processing giant Confluent has adopted Iceberg and also acquired WarpStream, a Kafka engine that’s directly built on Amazon’s S3.
- Users with batch-driven use cases are increasingly demanding faster query times, and Druid is rising to the occasion with powerful enhancements such as Dart.
It’s an exciting time to be working in the data space, and Druid’s evolving role in this ecosystem is only going to get more crucial.
A Must-Watch for Data Enthusiasts
I highly encourage you to watch the full recording of the Lakehouse Panel and all the other excellent sessions from from Druid Summit 2024. Whether you’re deep in the world of data lakes, looking to optimize your streaming analytics, or simply curious about the latest advancements in Druid, there’s something for everyone.
I’m already looking forward to the next Druid Summit—hope to see you there!
Related articles
More from Imply on the same topic.
Search your Splunk DDSS archive without restoring it
Security teams often keep logs long after they have aged out of their SIEM platform’s searchable window. The data is there for investigations or audits, but accessing the data for insights can be complicated.
Claude makes dashboards now, Lumi is ready for them
This week Anthropic launched Claude Dashboards in beta. You connect a data source, ask a question in plain language, and Claude writes the queries and builds the dashboard.
Lumi Lounge Recap: 3 Takeaways from Splunk .conf26
The Imply Lumi Lounge sat directly across the street from Splunk .conf26 and reached venue capacity on both days. Digital signs around the Colorado Convention Center pointed attendees our way with a simple message: “Federated search, done right. Full SPL. Less spend.”
Agents don’t query like your detections do
A security team of four has run a major SIEM since February. Nobody on the team writes SPL. Nobody opens the console or looks at the dashboards. Every question goes through an AI agent, which writes the query, runs it, reads the result, and answers in plain language.
Why You Shouldn’t Have to Delete Your VPC Flow Logs
When a security incident happens, investigators almost always start with the same questions: Which systems communicated? Where did the traffic originate? What changed before the incident? Was data exfiltrated? AWS VPC Flow Logs are often the fastest way to answer those questions.
Splunk Smartstore vs Lumi Loglake: Two Very Different Ways to Search Logs in Object Storage
One copies data back before it can be searched. The other queries it where it lives. Lumi Loglake lets Splunk teams query logs directly in object storage, including AWS S3, Delta Lake, Apache Iceberg, using standard SPL.
Supercharging Schema-On-Read: Logs in Object Storage Don’t Need a Data Catalog
Machine data architectures are rapidly changing. As telemetry volumes continue to grow and as costs rise, organizations are increasingly moving logs and other machine data into object stores such as AWS S3.
Imply Lumi Loglake vs Splunk Federated Search for S3
Teams are increasingly moving log data into AWS S3 to reduce costs and extend retention. Both Lumi Loglake and Splunk Federated Search to S3 help you query data in AWS S3 to lower costs, however the two technologies have several notable differences.
A First Look at Lumi Loglake: Query Logs Where They Live
TL;DR: Imply Lumi Loglake is a lakehouse (separated compute/storage) architecture for unstructured logs that reduces costs from 40% up to orders of magnitude on your hardware/AWS/Azure bill used to run your SIEM/observability products.
Imply Lumi Major Release Preview: Continuing the Journey Towards Decoupled Observability/SIEM
We are getting ready to introduce the next major expansion of Imply Lumi and the observability warehouse. When we introduced the industry’s first observability warehouse, the goal was clear: decouple the observability stack by separating storage, compute, and access.
Query Lumi from Grafana: Now in Private Preview
Observability has a silo problem. Today, we’re removing another one. Imply Lumi’s Grafana Loki integration is now in Private Preview.
