New features in Delta Lake 3.0 simplify data format selection and integration tasks for enhanced efficiency
Databricks has unveiled Delta Lake 3.0, the newest enhancement to the Linux Foundation open source project Delta Lake. Delta Lake 3.0 introduces the Universal Format (UniForm), allowing Delta-stored data to be read as if it were in Apache Hudi or Apache Iceberg formats. This eliminates compatibility concerns and simplifies data format selection, as UniForm provides automatic support for Iceberg and Hudi within Delta Lake. The new version aims to streamline data integration tasks and enable the creation of high-performing, open lakehouses.
“Databricks created the lakehouse architecture, which is built on Delta Lake. We’re committed to making Delta Lake the open format that gives customers the most choice and flexibility, greatest control of their own data, and all the benefits of an open ecosystem,” said Ali Ghodsi, Co-Founder and CEO at Databricks. “Customers shouldn’t be limited by their choice of format. With this latest version of Delta Lake, we’re making it possible for users to easily work with whatever file formats they want, including Iceberg and Hudi, while still accessing Delta Lake’s industry-leading speed and scalability.”
The release of Delta Lake 3.0 addresses the challenge of data silos that many organizations face when adopting the data lakehouse architecture. UniForm facilitates interoperability by enabling tools to read from Iceberg, Delta, and Hudi. This advancement offers more streamlined access to diverse corporate data, including transactional, streaming, structured, and unstructured data across different formats.
Key features of Delta Lake 3.0 include:
- Delta Universal Format (UniForm): This feature allows data in Delta to be read as if it were in Iceberg or Hudi formats. UniForm generates necessary metadata for Hudi or Iceberg, eliminating the need for manual conversions between formats. Delta becomes the universal format, promoting compatibility and scalability across ecosystems.
- Delta Kernel: To combat connector fragmentation, the Delta Kernel ensures that connectors are built against a core Delta library, simplifying updates and maintenance. Developers can keep their connectors up-to-date without the burden of reworking them.
- Delta Liquid Clustering: This innovation tackles performance issues by introducing flexible data layout techniques that provide cost-efficient data clustering as data volumes grow, improving read and write performance.
“Delta Lake 3.0, including Universal Format and Kernel, underlines the open source community’s dedication to enhancing data reliability and delivering advanced analytics. This release is a step forward in creating a community-driven ecosystem of data integrity, seamless collaboration, and real-time analytics tools,” said Mike Dolan, SVP of Projects, The Linux Foundation.
Delta Lake’s enhancements aim to enable organizations to analyze data from various sources, build AI models, and foster secure data exchange. The Delta Sharing feature, supporting the open standard for secure data exchange, plays a pivotal role in this.
Delta Lake 3.0 is now available in preview as part of the Linux Foundation’s Delta Lake project.
About Utility Analytics Institute (UAI)
UAI Enables Utility Transformation Through Analytics
UAI is a utility-led membership organization that provides support to the industry to advance the analytics profession and utility organizations of all types, sizes, and analytics maturity levels, as well as analytics professionals throughout every phase of their career.
Transforming into a data decision-based company is one of the most difficult transitions a utility will have to make to thrive in the new energy economy. It’s more than just managing massive amounts of data, implementing the right tools and technology, and people and process management. It’s ensuring you have proper change management processes in place to address cultural challenges, as well as data management and governance plans, and best practice and compliant security strategies in place. It’s implementing the best organizational structure for your utility, and hiring and retaining talented staff, plus so much more! UAI brings together leading utilities who are serious about tackling these challenges and together we concentrate on utility analytics.
What’s UAI Membership all about? UAI serves multiple audiences providing different membership packages for each audience type. Learn more about how UAI unifies our community, serves each audience to help you meet your goals and address challenges, and how each audience collaborates to better serve the utility industry.
Contact Kevin Praet, Membership Relations, at kpraet@utilityanalytics.com to learn about the benefits of becoming a member of Utility Analytics Institute (UAI).
