The News: Databricks has disclosed its roadmap for developments that will be released in Q3. Read the roadmap information on the Databricks community site.
Databricks Discloses Roadmap for Q3 with Data Platform Capabilities
Analyst Take: Databricks continues to deliver on valuable capabilities for its data platform as evidenced by the disclosures for Q3 delivery and directional statements of what will follow. The significance of these capabilities goes beyond the new functions and represents investments by Databricks to continue evolving its data platform as usage in AI/ML environments expand. There were a plethora of new capabilities included in the Q3 roadmap disclosure – the significant ones include:
- Lakehouse Federation – Adds the ability to connect to databases and data warehouses without the need to copy data into the Databricks Lakehouse. This capability reduces duplication where the data access from the external sources will still meet performance needs. Also added as part of Lakehouse Federation is the addition of new sources with connection to Oracle databases and the ability to mount a Hive Metastore.
- Lakehouse IQ – An AI-powered function for customers to understand the use of the Lakehouse, and data including a search function is enabled with Lakehouse IQ.
- Lakehouse Monitoring – A new capability to monitor the data pipeline for the purpose of assuring data quality was added.
- Security and Compliance – Several improvements for improvements were added that are specific to the public cloud where Databricks is executing:
- ACLs, IAM improvements with SSO support, network security control, and support for Azure Storage Firewall.
- Encryption of data in the Lakehouse with support for customer-provided keys.
- Support for Azure Confidential Computing.
- Compliance certifications – HIPAA, PCI, FedRAMP
- Databricks Marketplace additions
- AI models that may be downloaded.
- Connections for sharing data (Delta Sharing) outside the Databricks environment.
- Clean room support.
- Cloudflare R2 integration – share live data with zero egress fees.
- MosaicML with gated availability.
- MLflow additions for management of large language models (LLMs).
- Databricks SQL for use of the Lakehouse as a Data Warehouse.
- Serverless execution with Azure and AWS.
- Delta Universal Format (UniForm) adds the ability to access data as if it were in Iceberg or Hudi format.
There were a number of future developments disclosed that will be in public preview in Q3, but general availability was not included in the information. The Futurum Group will include more information when the GA date is disclosed.
Key Takeaways
One of the big takeaways from the disclosure is the continued evolution of AI/ML development with additional capabilities delivered by Databricks in creating the environment. The features/functions represent advances that go beyond new capabilities but make it simpler or easier than before, when many environments were “hand-built” by specialists.
Databricks continues to be a leader in Data Platform, according to analysis by The Futurum Group. As the Data Platform market coalesces with acquisitions and organic development, Databricks has made strategic moves including the acquisition of Mosaic and integration of Unity Catalog that demonstrates the company is a leader in evolving the environment.
Disclosure: The Futurum Group is a research and advisory firm that engages or has engaged in research, analysis, and advisory services with many technology companies, including those mentioned in this article. The author does not hold any equity positions with any company mentioned in this article.
Analysis and opinions expressed herein are specific to the analyst individually and data and other information that might have been provided for validation, not those of The Futurum Group as a whole.
Other insights from The Futurum Group:
Databricks’ MosaicML Acquisition, LakehouseIQ Launch, Data + AI Summit Show Gen AI Savvy
The Cost of The Next Big Thing – Artificial Intelligence
Author Information
Randy has written numerous industry articles and papers as an educator and presenter, and he is the author of two books: Planning a Storage Strategy and Information Archiving – Economics and Compliance. The latter is the first book of its kind to explore information archiving in depth. Randy regularly teaches classes on Information Management technologies in the U.S. and Europe.
