2025
Fishing for AI-Powered Insights: Lakehouse Technologies
ML algorithms, and data lakes
Josef Sieber
Presenter
ML algorithms, and data lakes
Josef Sieber
Presenter
In the past decade, companies have been focusing on machine learning powered insights to drive their businesses forward. Recently, there has been a focus on agentic AI with LLMs trained on proprietary company data. To train these models, companies have turned to data lake technologies to address the exponential growth of data and the need for more flexible, scalable data management solutions. Many modern ML algorithms and architectures (neural networks, transformers, backpropagation, LSTMs) have been around for decades, but require a massive amount of data to train.
The volume, variety, and velocity of data required for modern ML have outpaced traditional data storage and processing systems. Data lakes offer a compelling solution by providing a centralized repository capable of storing vast amounts of raw, unstructured, and semi-structured data in native formats, well-suited for machine learning and artificial intelligence tasks.
In this session, we will discuss data lake technologies. We will cover the history of relational databases, data warehouses, ML algorithms, and data lakes. We will also dive into technical details of table formats like ACID guarantees of Delta Lake and Apache Iceberg, the underlying file formats like Apache Parquet, and how they come together to create the lakehouse for ML and AI.
Aniruth Narayanan
Aniruth Narayanan is a Product Manager on the Storage team at Databricks, the Data and AI company. Aniruth focuses on interoperable, open data infrastructure with Delta Lake and Apache Iceberg. Aniruth is an award winning public speaker, a former computer science instructor, and an avid basketball fan. Previously, Aniruth worked at Tesla (Software Release), Microsoft (Data Centers), Retool (Database/API Connectors), and Ernst & Young (ML Cybersecurity). You can read more about Aniruth at aniruthn.com.
GuidePoint Security · 3030 N Rocky Point Drive · Suite 600 · Tampa, FL