Data Lakes, Data Warehouses, and Lakehouse Architecture

The evolution of enterprise data architecture can be understood, in part, as an evolution in how organizations store, organize, process, and use information.

For many years, the enterprise data warehouse was the central destination for analytical data. It provided a structured environment in which organizations could integrate information from operational systems and make it available for reporting, business intelligence, and decision support. The warehouse brought discipline to enterprise analytics by emphasizing consistent definitions, controlled structures, governed data, and predictable performance.

As the volume, variety, and velocity of enterprise data increased, however, the traditional warehouse model began to encounter new challenges. Organizations increasingly needed to work with log data, machine-generated information, documents, images, events, application data, and other forms of information that did not fit naturally into highly structured relational models.

The data lake emerged as an architectural response to these challenges. It emphasized scalable storage, flexible data formats, and the ability to retain large amounts of raw and processed information. Data lakes became particularly important in cloud architectures because inexpensive and scalable object storage could provide a foundation for large enterprise data estates.

But the data lake introduced challenges of its own. Without strong governance, metadata, quality controls, and data management practices, a lake could become difficult to understand and trust. Organizations sometimes found themselves with large collections of data that were technically available but difficult to discover, interpret, govern, or use.

The lakehouse emerged as an attempt to combine important characteristics of data lakes and data warehouses. It seeks to provide flexible, scalable storage while also supporting stronger management, transactional capabilities, analytical performance, governance, and data engineering patterns. For the Enterprise Data Architect, the important question is not whether a lake, warehouse, or lakehouse is inherently the correct architecture. The important question is:

What combination of storage, processing, management, and access patterns best supports the enterprise’s information and intelligence requirements?

That question requires understanding the architectural strengths and limitations of each approach.

7.1 The Evolution of Enterprise Analytical Architecture

The enterprise analytical environment has evolved through several broad stages.

In the early stages of enterprise computing, organizations often performed analysis directly against operational systems. This approach worked when data volumes were relatively small and analytical requirements were limited. As organizations began demanding more sophisticated reporting, directly querying operational systems created problems. Analytical workloads could compete with transaction processing, and information from different applications was often difficult to combine.

The enterprise data warehouse addressed this problem by creating a separate environment for analytical processing. Data from operational systems was extracted, transformed, integrated, and loaded into a structured analytical repository. The warehouse provided a controlled environment for enterprise reporting. This architecture established an important principle:

Analytical workloads should be architected differently from transactional workloads.

That principle remains relevant today. However, enterprise data became increasingly diverse. Organizations began collecting information from websites, mobile applications, sensors, social platforms, application logs, APIs, external datasets, and machine-generated systems. Data was no longer exclusively structured into rows and columns. At the same time, organizations increasingly wanted to retain information before knowing exactly how it would be used. The data lake addressed this requirement. Instead of requiring every piece of information to be modeled and transformed before storage, organizations could retain data in more flexible forms and apply processing later.

This approach introduced the concept commonly described as schema-on-read. A warehouse traditionally emphasized defining structure before or during loading. A data lake could defer some structural decisions until data was consumed. The lake therefore expanded the range of information that could be retained and processed. But flexibility came with a price. Without strong architecture, a lake could become a collection of files, tables, pipelines, and datasets whose meaning was unclear. This created demand for approaches that combined the flexibility of a lake with more disciplined data management. The lakehouse concept emerged in this context.

7.2 Understanding the Data Warehouse

A data warehouse is an analytical data environment designed to support structured, governed, and performant analysis. Its traditional role is to integrate information from multiple operational systems into a consistent analytical model. For example, an enterprise may have separate systems for:

• Customers
• Accounts
• Transactions
• Products
• Employees
• Financial reporting

Each operational system may have its own data model and terminology. The warehouse provides an environment in which these representations can be integrated for analytical purposes. A financial institution, for example, may have one system that identifies a customer using a customer number, another using an account identifier, and another using a relationship identifier. The warehouse can establish analytical relationships among these representations. This is one reason data warehousing is not simply a storage technology. It is an information integration architecture.

The Role of Data Modeling

Traditional warehouses generally depend heavily on data modeling. Dimensional models, star schemas, normalized structures, and other analytical modeling techniques provide mechanisms for organizing information around business concepts. For example, an analytical model may contain Customer, Account, Product, Transaction, and Date structures. These structures make it possible to ask questions such as:

• How much revenue was generated by customer segment?
• Which products have the highest transaction volume?
• How has account activity changed over time?
• Which regions are experiencing growth?

The model creates consistency between the data and the questions the enterprise wants to ask. This is particularly important for regulated industries and executive reporting, where the definition of a metric can have significant consequences.

Strengths of the Warehouse

The traditional warehouse remains valuable because it provides strong support for structured analytical workloads. It generally emphasizes:

• Governed data
• Predictable analytical structures
• Performance
• SQL-based access
• Controlled transformation
• Enterprise reporting
• Consistent business definitions

The warehouse is therefore particularly well suited to information that has a relatively clear analytical purpose.

Limitations of the Traditional Warehouse

The same characteristics that make the warehouse effective can also create limitations. Highly structured models can make it harder to work with:

• Large volumes of raw data
• Semi-structured information
• Documents
• Images
• Logs
• Rapidly changing data structures
• Machine-generated data

Traditional warehouse architectures can also create significant transformation requirements. If data must be extensively modeled before analysts and data scientists can use it, the time between data generation and data availability can become longer. This is one of the reasons organizations began adopting data lake architectures.

7.3 Understanding the Data Lake

A data lake is a large-scale data repository designed to store information in a variety of formats, often using scalable object storage. The central architectural idea is flexibility. A data lake can retain:

• Structured data
• Semi-structured data
• Unstructured data
• Logs
• Events
• Documents
• Files
• Machine-generated information

The data does not necessarily need to be transformed into a highly structured analytical model before it is stored. This creates an important architectural capability. An organization can preserve information first and determine some of its future analytical uses later. This can be particularly useful for data science and AI workloads. For example, an organization may collect millions of application events. At the time of ingestion, it may not know every analytical question that will eventually be asked of those events. Retaining the raw information provides future flexibility.

Schema-on-Read

One of the concepts associated with data lakes is schema-on-read. The idea is that structure can be applied when the information is consumed rather than requiring all structure to be established at ingestion. This does not mean that data lakes should have no schemas. In well-designed data lakes, schemas still exist. The difference is that the architecture can support multiple representations and defer some modeling decisions.

Raw Data and Curated Data

A mature data lake commonly contains multiple stages of information. For example:

• Raw: information as received from source systems.
• Standardized: information transformed into consistent technical formats.
• Curated: information cleaned, enriched, and prepared for specific analytical or operational uses.
• Published: information made available through governed data products or analytical interfaces.

These stages should not be interpreted as mandatory physical zones for every implementation. They are architectural concepts that help establish data lifecycle and responsibility.

The Data Swamp Problem

The flexibility of a data lake can become a weakness when architecture and governance are insufficient. If an organization stores large amounts of data without clear ownership, metadata, quality controls, or lifecycle policies, users may struggle to determine:

• What a dataset represents
• Who owns it
• Whether it is trustworthy
• When it was created
• How it was transformed
• Whether sensitive information is present
• Whether another dataset already contains the same information

A data lake that is technically full but semantically unclear is not a successful enterprise data architecture. This is often described using the term data swamp. The problem is not the lake itself. The problem is the absence of sufficient architecture around it.

7.4 From Data Lake to Lakehouse

The lakehouse approach emerged from the recognition that organizations wanted the flexibility of data lakes while also requiring many of the capabilities associated with analytical warehouses. The lakehouse is therefore best understood as an architectural approach rather than simply a product category. At a conceptual level, a lakehouse seeks to combine:

• Scalable storage
• Flexible data formats
• Analytical processing
• Stronger data management
• Transactional capabilities
• Governance
• Performance
• Multiple workload support

The lakehouse can allow organizations to maintain a relatively open and scalable storage foundation while providing more structured management and analytical capabilities above it. This is particularly attractive in cloud environments. Cloud object storage can provide a durable foundation, while different processing engines can operate over the stored data.

Why This Matters

The architectural significance of the lakehouse is not simply that it combines two technologies. Its larger significance is that it can reduce the need to maintain completely separate data estates for different analytical workloads. For example, the same underlying data may support:

• Business intelligence
• Data engineering
• Machine learning
• Exploratory analytics
• Operational analytics
• AI retrieval

This does not mean every workload should use the same physical representation. It means the architecture can provide a more coherent data foundation across workloads.

7.5 Data Lakehouse Architecture

A conceptual lakehouse architecture can be viewed as a set of interacting layers. At the bottom is scalable storage. Above that are mechanisms for organizing and managing datasets. Processing engines provide transformation and analytical capabilities. Governance and metadata services provide enterprise control. Consumption layers provide access to users and applications.

Storage Layer

The storage layer provides the durable foundation. Object storage is commonly used because of its scalability and flexibility. However, storage alone does not provide the semantics required for enterprise analytics.

Table and Data Management Layer

The next layer provides structures that allow files and datasets to behave more like managed analytical tables. Capabilities can include:

• Schema management
• Transactional behavior
• Versioning
• Partitioning
• Metadata
• Data lifecycle management

These capabilities help address some of the weaknesses associated with unmanaged data lakes.

Processing Layer

The processing layer provides engines for transforming and querying data. Different engines may be optimized for different workloads. For example, one workload may emphasize interactive SQL while another may require distributed processing. The architecture should allow processing to evolve without forcing unnecessary redesign of the underlying information foundation.

Consumption Layer

The same underlying data may support multiple consumers. These may include:

• Analysts
• Business intelligence tools
• Data scientists
• Applications
• Machine learning pipelines
• Generative AI systems

This is where the lakehouse becomes particularly relevant to the broader enterprise architecture.

7.6 Batch, Streaming, and Lakehouse Architectures

The traditional data warehouse was heavily associated with batch processing. Data would be extracted from source systems on a scheduled basis, transformed, and loaded into analytical structures. This remains useful. Many business processes do not require second-by-second information. Financial reporting, regulatory reporting, historical analysis, and many planning workloads can operate effectively with scheduled processing. However, modern enterprises increasingly require streaming and near-real-time data. Examples include:

• Fraud detection
• Payment monitoring
• Customer interaction analysis
• Operational monitoring
• IoT
• Cybersecurity
• Application telemetry

The modern data architecture therefore needs to support multiple processing patterns.

Batch Processing

Batch processing works with accumulated data. Its strengths include:

• Predictable processing
• Efficient large-scale transformation
• Simpler operational models
• Strong suitability for historical processing

Streaming Processing

Streaming processes data continuously or in small increments. Its strengths include:

• Low latency
• Rapid event processing
• Real-time decision support
• Continuous monitoring

Micro-Batch

Some architectures use micro-batching, where data is processed in small frequent intervals rather than individually. This can provide a balance between continuous processing and traditional batch operations. The important architectural principle is:

Processing mode should be selected according to business latency requirements.

Not every dataset needs real-time processing. Real-time architecture introduces additional complexity and cost, so it should be justified by business value.

7.7 Data Architecture for Analytics and AI

The rise of AI changes the role of the analytical data platform. Traditional analytics often focuses on structured questions. For example:

What were our sales last quarter?

AI applications can require much broader forms of information. An AI system may need:

• Structured customer information
• Transaction history
• Product information
• Documents
• Policies
• Procedures
• Knowledge articles
• Application events
• Metadata
• Business definitions

This creates a convergence between data architecture and AI architecture. A lakehouse or modern analytical platform may become one component of a broader enterprise intelligence architecture.

Structured Data for AI

Structured data remains important for AI. Machine learning models often require well-prepared datasets. Analytical models can provide:

• Features
• Historical patterns
• Behavioral information
• Customer attributes
• Financial indicators

Unstructured Data for AI

Generative AI introduces significant demand for unstructured information. Documents and text may need to be:

• Collected
• Classified
• Parsed
• Chunked
• Enriched
• Indexed
• Embedded
• Retrieved

This means that enterprise data architecture increasingly needs to account for information beyond conventional tables.

Metadata Becomes More Important

As data is consumed by AI systems, metadata becomes critical. An AI application needs to know not only that information exists but also:

• What it means
• Whether it is current
• Whether it is authoritative
• Who owns it
• Whether it contains sensitive information
• Under what conditions it can be used

This reinforces a principle introduced earlier in the book:

Data without context is not sufficient for enterprise intelligence.

7.8 Governance Across Lakes, Warehouses, and Lakehouses

The physical architecture of the data platform does not replace governance. Governance must operate across the entire data lifecycle. A modern architecture should be able to answer:

• What is this data?
• Who owns it?
• Where did it come from?
• How was it transformed?
• Who can access it?
• How long should it be retained?
• How trustworthy is it?
• Can it be used for AI?

These questions require metadata, lineage, quality, security, privacy, and ownership.

Data Classification

Information should be classified according to its sensitivity and business significance. For example, an enterprise may distinguish:

• Public information
• Internal information
• Confidential information
• Restricted or regulated information

The precise categories will vary by organization. What matters is that classification can be connected to enforceable policies.

Data Lineage

Lineage allows the organization to understand how information moves through the architecture. For example:

Operational Database → Ingestion → Lakehouse Table → Transformation → Data Product → Dashboard

or:

Document Repository → Processing → Chunking → Embedding → Retrieval Index → AI Application

Lineage becomes increasingly important as architectures become more complex.

Data Quality

A lakehouse does not automatically make data trustworthy. Quality must be measured and managed. Relevant dimensions can include:

• Completeness
• Accuracy
• Consistency
• Timeliness
• Uniqueness
• Validity

The quality expectations should be linked to business use. A dataset used for regulatory reporting may require different controls from a dataset used for exploratory analysis.

7.9 Choosing Between Warehouse, Lake, and Lakehouse

The choice should be driven by workload and architectural requirements rather than technology fashion. A warehouse may be appropriate when the primary requirement is highly structured analytical reporting with strong semantic consistency and predictable workloads. A lake may be appropriate when the enterprise needs a highly flexible storage foundation for diverse data and exploratory processing. A lakehouse may be appropriate when the organization wants a common scalable data foundation that supports diverse analytical and AI workloads while providing stronger data management capabilities. In many enterprises, the answer is not one or the other.

All three architectural concepts may coexist. A mature enterprise architecture may contain:

• Operational databases
• Analytical warehouses
• Object storage
• Lakehouse environments
• Specialized databases
• Streaming platforms
• Search systems
• Vector stores

The Enterprise Data Architect’s role is to define the relationships among these components.

Architectural Decision Questions

Before selecting an architecture, the architect should ask:

1. What business problem is being solved?
2. What information types are involved?
3. What latency is required?
4. What analytical workloads are expected?
5. What AI workloads are expected?
6. What governance requirements apply?
7. What data quality is required?
8. What scale is expected?
9. What resilience is required?
10. What is the expected cost profile?
11. Who owns the data?
12. How will the architecture evolve?

These questions are more durable than choosing a specific technology.

Case study: Meridian Financial Group

7.10 Data Lakes, Warehouses, and Lakehouses at Meridian Financial Group

Meridian Financial Group historically relied heavily on enterprise data warehouses. The warehouse supported:

• Financial reporting
• Management reporting
• Regulatory analysis
• Customer analytics
• Operational performance reporting

The architecture was stable but increasingly difficult to scale for newer workloads. Meridian’s data environment began receiving large amounts of information from digital channels, applications, market systems, documents, and external sources. The organization initially responded by creating a large data lake. The lake successfully expanded storage capacity and enabled data science teams to experiment with new datasets. However, a new problem emerged. Different teams created their own datasets and transformations. Some datasets contained duplicate customer information. Some contained undocumented business rules. Some were no longer actively maintained.

Users often struggled to determine which dataset should be trusted. Meridian therefore began transitioning toward a lakehouse-oriented architecture. The goal was not to eliminate the existing warehouse. Instead, the organization established a broader analytical architecture. Structured financial reporting remained supported by governed warehouse capabilities. Large-scale and diverse data was retained in the scalable data foundation. Managed analytical tables provided stronger consistency and transactional behavior. Domain-oriented data products were introduced for important business areas. Metadata and lineage capabilities were strengthened. AI teams were provided controlled access to structured and unstructured enterprise information.

The most important change was organizational as much as technological. Meridian stopped thinking of the lakehouse as simply another platform. It began treating the analytical environment as an enterprise information capability. Business domains became responsible for the meaning and quality of their data. Platform teams became responsible for providing reliable technical capabilities. Enterprise architecture established common principles and integration patterns. Governance teams established policies. AI teams consumed trusted information through governed interfaces. The resulting architecture was more flexible because different workloads could use different processing patterns while still participating in a common enterprise information environment.

Practical Artifact: Enterprise Lake, Warehouse, and Lakehouse Assessment

The following assessment can be used when evaluating an organization’s current analytical architecture.

Step 1: Identify Analytical Workloads

Start with business requirements rather than technologies. Document workloads such as:

• Executive reporting
• Regulatory reporting
• Operational analytics
• Customer analytics
• Machine learning
• Data science
• AI
• Exploratory analysis

For each workload, document the required latency, scale, data types, and quality expectations.

Step 2: Map Existing Platforms

Create an inventory of:

• Warehouses
• Lakes
• Lakehouses
• Operational databases
• Streaming platforms
• Specialized data stores
• Search platforms
• AI data stores

Then identify which business domains and workloads depend on each platform.

Step 3: Assess Data Movement

Map how information moves through the architecture. For example:

Source → Ingestion → Storage → Transformation → Data Product → Consumer

Look for:

• Unnecessary copies
• Redundant pipelines
• Excessive transformations
• Uncontrolled movement
• Missing lineage

Step 4: Assess Governance

For each major dataset, determine whether the organization knows:

• The owner
• The definition
• The source
• The quality level
• The sensitivity
• The lineage
• The approved consumers

If these answers cannot be established, the technical platform may be functioning without sufficient information architecture.

Step 5: Assess AI Readiness

Evaluate whether the platform can support both structured and unstructured information. Ask:

• Can AI systems access governed enterprise data?
• Can documents be discovered and retrieved?
• Is metadata available?
• Can sensitive information be controlled?
• Can data lineage be established?
• Can information be connected to business context?

Step 6: Define the Target Architecture

The target architecture should describe:

• What should remain
• What should be modernized
• What should be consolidated
• What should be retired
• What new capabilities are required

The target should focus on architectural capabilities rather than prematurely selecting products.

Lakehouse Decision Matrix

The matrix is not a technology selection scorecard. It is a way to initiate architectural discussion. The actual decision should consider the enterprise’s requirements, existing investments, skills, regulatory environment, operating model, and long-term strategy.

Remember these

Key Takeaways

1. The data warehouse, data lake, and lakehouse represent different architectural approaches to managing enterprise information.
2. The data warehouse remains valuable for structured, governed, and highly consistent analytical workloads.
3. The data lake provides flexibility for large volumes and diverse forms of information.
4. A data lake without sufficient metadata, governance, ownership, and quality controls can become difficult to trust and use.
5. The lakehouse seeks to combine scalable and flexible storage with stronger analytical management capabilities.
6. Modern enterprises often need multiple analytical patterns rather than a single physical platform.
7. Batch and streaming architectures should be selected according to business latency requirements.
8. AI increases the importance of both structured and unstructured enterprise information.
9. Metadata, lineage, quality, ownership, and context become increasingly important as data platforms support AI.
10. The Enterprise Data Architect should design relationships among platforms rather than treating any individual platform as the complete enterprise architecture.

Think about your own organization

Architect’s Reflection

The evolution from warehouse to lake to lakehouse is sometimes presented as a sequence in which each new technology replaces the previous one. Enterprise architecture is rarely that simple. The warehouse did not become irrelevant because data lakes emerged. Data lakes did not become obsolete because lakehouses became popular. Each architectural approach addresses particular requirements. The deeper lesson is that enterprise data architecture evolves as the nature of information and business demand changes. The warehouse taught enterprises the importance of structure, integration, governance, and analytical consistency. The data lake expanded the enterprise’s ability to retain and explore diverse information. The lakehouse seeks to bring greater management and analytical discipline to a flexible data foundation. AI is now adding another dimension. The enterprise needs information not only for dashboards and reports, but also for retrieval, prediction, reasoning, automation, and intelligent decision support. This means the Enterprise Data Architect must think beyond storage technology. A data platform should be evaluated by the capabilities it provides to the enterprise.

• Can the organization discover information?
• Can it understand information?
• Can it trust information?
• Can it govern information?
• Can it share information?
• Can it analyze information?
• Can it provide information to AI systems safely and with sufficient context?

Those questions lead us back to the central architectural principle of this book:

The purpose of enterprise data architecture is not to build a data platform. It is to create an environment in which enterprise information can become a trusted and reusable strategic asset.

The lakehouse is one architectural response to that challenge. It is not the destination. The destination is an enterprise capable of turning information into intelligence.