Data Lake

A data lake is a centralized storage repository that holds raw data in its native format — structured (database exports), semi-structured (JSON, CSV logs), and unstructured (images, PDFs, audio, raw text) — at massive scale and low cost, without requiring the data to be structured into a predefined schema before it's stored. This is the key distinction from a data warehouse: a warehouse enforces "schema-on-write" (data must be cleaned and structured before it goes in), while a data lake practices "schema-on-read" (data goes in as-is; structure and meaning are applied later, at query or processing time, by whatever tool reads it). Why it matters for AI/SaaS builders: data lakes have become especially relevant in the AI era because so much of the raw material for AI features — documents, chat transcripts, uploaded files, model outputs, eval logs — doesn't fit neatly into relational tables, and forcing a rigid schema onto it before storage would mean deciding upfront exactly how it'll be used, which is often unknown when the data first lands. Storing it raw in a data lake first (commonly just object storage like S3 with a defined folder/partition convention) preserves optionality: a new AI feature six months later can reprocess the same raw historical data in a way nobody anticipated when it was first collected. How it works: data lakes are typically built directly on object storage (S3, Google Cloud Storage, Azure Blob Storage) using open file formats (Parquet, Avro, JSON) organized into a partitioning scheme (e.g., `s3://lake/events/year=2026/month=07/day=02/`) that lets query engines scan only relevant partitions instead of the whole dataset. Query engines like Apache Spark, Presto/Trino, or AWS Athena can run SQL directly against files sitting in the lake without a separate load step, blurring the line between a "data lake" and a "data warehouse" — a hybrid pattern often called a "lakehouse" (popularized by Databricks) that adds warehouse-like transactional guarantees and schema enforcement on top of lake-style cheap, flexible raw storage. Worked example: an AI SaaS logs every raw LLM request/response pair (prompt, model, tokens, latency, output) as JSON files into an S3-based data lake, partitioned by date, long before deciding exactly what analysis they'll want to run on it. Months later, when the team wants to build a prompt-regression eval system comparing model behavior across versions, they run Athena queries directly against the historical raw JSON in the lake — no ETL pipeline had to be pre-built for a use case nobody had defined yet at collection time.