Introduction

Data lakes have become the foundation of modern analytics, enabling organizations to store petabytes of structured and unstructured data in cloud object storage services such as Amazon S3, Azure Blob Storage, and Google Cloud Storage. Open table formats like Apache Iceberg, Delta Lake, and Apache Hudi provide a scalable, vendor-neutral approach that allows multiple analytics engines to access the same datasets.

However, querying data directly from remote storage is not always straightforward. Network latency, metadata lookups, and large Parquet file scans can increase query execution time compared to locally stored data, making interactive analytics more challenging.

ClickHouse 26.3 LTS introduces several enhancements designed to address these challenges, including improved parallel object storage reads, a built-in Parquet metadata cache, and asynchronous Iceberg metadata prefetching. Together, these optimizations reduce query latency and improve the performance of analytics workloads running directly on modern data lakes.

Why Data Lake Query Optimization Matters

Modern analytics platforms often separate storage from compute. Instead of copying data into multiple databases, organizations keep data in cloud object storage and allow different analytics engines to query it directly. While this reduces storage costs and simplifies data management, it also introduces challenges such as remote storage latency, metadata lookups, and large file scans. ClickHouse 26.3 addresses these challenges with several performance optimizations.

Understanding Data Lake Queries

A data lake is a centralized repository that stores both raw and processed data in open formats, allowing multiple analytics engines to access the same datasets without data duplication. Unlike traditional data warehouses, where data is often imported into proprietary storage, data lakes keep data in cloud object storage while enabling different tools to query it directly.

Common cloud object storage platforms include:

• Amazon S3
• Azure Blob Storage
• Google Cloud Storage

These storage platforms commonly use open table formats such as:

• Apache Iceberg
• Delta Lake
• Apache Hudi
• Parquet

Instead of ingesting data into native ClickHouse tables, ClickHouse can query these datasets directly from object storage. During query execution, ClickHouse reads the required metadata, identifies the relevant data files, applies optimizations such as parallel file reads and metadata caching, and returns the analytical results without moving or duplicating the data.

Object Storage
      │
      ▼
Parquet / Iceberg / Delta
      │
      ▼
ClickHouse
      │
      ▼
SQL Analytics

This approach eliminates data duplication while allowing organizations to use ClickHouse as a high-performance query engine over existing data lake storage.

Challenges of Querying Data Lakes

Although object storage is highly scalable and cost-effective, querying data directly presents several performance challenges.

Remote Storage Latency

Every query requires reading files across the network rather than from local disks.

Metadata Overhead

Open table formats maintain metadata describing snapshots, manifests, and data files. Accessing this metadata adds planning overhead.

Large File Scans

Without efficient pruning, queries may scan significantly more data than necessary.

Repeated Metadata Reads

Repeated dashboard queries often fetch the same metadata multiple times, increasing latency.

How ClickHouse Queries Data Lakes

ClickHouse connects directly to supported object storage services and open table formats, allowing SQL queries to execute without first importing data into native MergeTree tables. During query execution, ClickHouse reads only the required files and columns, minimizing unnecessary data movement and enabling efficient analytics over remote datasets.

ClickHouse supports direct access to several open data lake technologies, including:

• Apache Iceberg
• Delta Lake
• Apache Hudi
• Parquet files
• Object storage (Amazon S3 and compatible services)

This enables organizations to analyze data directly in their data lakes while avoiding the overhead of data duplication and complex ETL pipelines.

Data Lake Enhancements in ClickHouse 26.3 LTS

ClickHouse 26.3 LTS introduces multiple improvements that reduce latency when querying remote data lakes.

1. Faster Parallel Reading from Data Lakes

One of the most significant improvements is enhanced parallel reading of data stored in object storage.

When a query reads a relatively small number of large files, ClickHouse now distributes the work more efficiently across CPU cores. This optimization applies to:

• Apache Iceberg
• Delta Lake
• Apache Hudi
• Object storage reads

As a result, data lake queries can be tens of times faster on multi-core systems for these workloads.

2. Parquet Metadata Cache

Reading Parquet files requires accessing the file footer to understand schema and metadata.

Before 26.3, repeated queries often reread this metadata.

ClickHouse 26.3 introduces a Parquet metadata cache that stores Parquet footer information in memory.

Benefits include:

• Reduced metadata reads
• Lower remote I/O
• Faster repeated queries
• Improved dashboard performance

The cache is enabled by default and maintains consistency by tracking object changes using file ETags.

3. Iceberg Metadata Prefetching

Iceberg tables maintain metadata describing snapshots and manifests.

In previous versions, queries often waited while this metadata was fetched.

ClickHouse 26.3 introduces asynchronous Iceberg metadata prefetching.

Instead of downloading metadata during query execution, ClickHouse refreshes metadata in the background and serves queries from the cache when it is sufficiently fresh.

This reduces metadata lookup latency and improves response times for frequently queried Iceberg tables.

Architecture Overview

Example 1: Querying a Parquet Dataset

SELECT
    country,
    count() AS total_events
FROM s3(
    'https://my-bucket.s3.amazonaws.com/events/*.parquet'
)
WHERE event_date >= today() - 7
GROUP BY country;

In this query, ClickHouse reads the Parquet files directly from object storage. The WHERE clause helps reduce the amount of data scanned, while only the required columns are processed, improving query efficiency.

Example 2: Querying an Iceberg Table

SELECT
    product_category,
    SUM(revenue) AS total_sales
FROM iceberg('analytics.sales')
WHERE order_date >= today() - 30
GROUP BY product_category
ORDER BY total_sales DESC;

This example demonstrates querying an Iceberg-managed dataset directly. ClickHouse uses Iceberg metadata to locate the required files while the metadata cache and asynchronous prefetching help reduce planning overhead.

Real-World Example

Imagine an e-commerce company storing several years of clickstream data in Amazon S3 using Apache Iceberg.

Instead of copying terabytes of historical data into ClickHouse, analysts query the Iceberg tables directly.

With ClickHouse 26.3:

• Parallel reading speeds up remote file processing.
• The Parquet metadata cache avoids repeatedly reading metadata.
• Iceberg metadata prefetching reduces query planning time.

As a result, dashboard queries become significantly faster without moving data into ClickHouse.

Performance Benefits

The enhancements in ClickHouse 26.3 LTS provide several practical benefits for data lake analytics.

Together, these optimizations improve the performance of BI dashboards, exploratory analytics, and recurring queries against data stored in cloud object storage.

Best Practices

To get the best performance when querying data lakes with ClickHouse, consider the following recommendations:

• Store analytical data in Parquet format.
• Partition datasets based on common query filters.
• Select only the columns required by your queries.
• Apply filters early to reduce the amount of data scanned.
• Take advantage of the built-in Parquet metadata cache.
• Use asynchronous Iceberg metadata prefetching for frequently queried tables.
• Keep ClickHouse updated to benefit from ongoing data lake optimizations.

When Should You Use This?

ClickHouse 26.3 is particularly beneficial for organizations that store large analytical datasets in cloud object storage and require fast, interactive SQL analytics without duplicating data.

• Interactive analytics on cloud object storage
• Business intelligence dashboards
• Data exploration
• Querying Apache Iceberg tables
• Querying Delta Lake datasets
• Ad hoc SQL analysis over Parquet files
• Large-scale cloud analytics platforms

Key Takeaways

• Query cloud-hosted datasets without moving data.
• Parallel reading improves processing of remote files.
• Parquet metadata caching reduces repeated metadata access.
• Iceberg metadata prefetching lowers query planning latency.
• ClickHouse 26.3 makes interactive analytics over data lakes faster and more efficient.

Conclusion

As organizations continue adopting open table formats and cloud object storage, efficiently querying data lakes has become increasingly important for interactive analytics. ClickHouse already enables users to query data directly from formats such as Apache Iceberg, Delta Lake, Apache Hudi, and Parquet without duplicating data or building complex ETL pipelines.

ClickHouse 26.3 LTS further strengthens these capabilities through enhanced parallel object storage reads, a built-in Parquet metadata cache, and asynchronous Iceberg metadata prefetching. These improvements reduce metadata overhead, minimize remote I/O, and accelerate query execution, enabling faster and more efficient analytics on cloud-hosted datasets.

Whether you're building business intelligence dashboards, exploring large-scale datasets, or developing cloud-native analytics platforms, ClickHouse 26.3 LTS provides the performance and scalability needed to unlock faster insights from modern data lakes.

References

• ClickHouse v26.3 Release Overview
• ClickHouse is Data Lake Ready