Semi-structured data has become a standard part of modern analytics. Whether you're storing application logs, user attributes, telemetry data, or event metadata, it's common to encounter records where each row contains a different set of key-value pairs.

ClickHouse® offers the Map data type to handle these scenarios efficiently. A Map allows you to store dynamic attributes without creating hundreds of nullable columns or relying on JSON parsing during query execution.

However, while the Map data type is flexible, querying individual keys from large maps has traditionally been less efficient than querying regular columns. Prior to ClickHouse® 26.3, retrieving a single key often required scanning the entire serialized map for every row, even if only one value was needed.

ClickHouse® 26.3 introduces a new bucketed serialization format for the Map data type that significantly improves the performance of key lookups. By reorganizing how map data is stored on disk, ClickHouse® can reduce the amount of data read during queries while keeping inserts fast.

In this article, we'll explore:

• How Map serialization worked before 26.3
• The limitations of the previous approach
• The new bucketed serialization format
• Configuration settings
• Performance improvements
• When you should enable the feature

Understanding the Map Data Type

A Map stores key-value pairs inside a single column.

For example:

CREATE TABLE events
(
    event_time DateTime,
    user_id UInt64,
    attributes Map(String, String)
)
ENGINE = MergeTree
ORDER BY event_time;

A row might contain:

{
    "country": "India",
    "browser": "Chrome",
    "device": "Mobile",
    "campaign": "SummerSale"
}

This approach is particularly useful when different rows contain different attributes.

For example:

Instead of creating numerous nullable columns, all dynamic attributes remain inside a single Map.

How Map Serialization Worked Before 26.3

Internally, a Map is stored as two arrays:

• Keys
• Values

Conceptually:

Keys:
["country", "browser", "campaign"]
 
Values:
["India", "Chrome", "SummerSale"]

When a query requests:

SELECT attributes['browser']
FROM events;

ClickHouse® has to inspect the serialized map data to locate the requested key for each row. As the number of keys in each map grows, this process can require reading and processing more data than necessary because the storage layout isn't optimized for selective key access.

For workloads with many large maps and frequent lookups of individual keys, this extra work can become noticeable.

The Challenge

Imagine every row stores 100 different attributes.

If a dashboard only needs:

attributes['country']

the database still has to process the serialized map representation for each row to locate the requested key.

Across billions of rows, this additional work increases:

• Data read
• CPU usage
• Query latency

The larger each map becomes, the more pronounced the effect can be.

What's New in ClickHouse® 26.3?

ClickHouse® 26.3 introduces a new serialization format named with_buckets.

Instead of storing map entries as a single serialized unit, keys are distributed into multiple buckets based on their hash values when parts are merged.

Conceptually:

Bucket 1
--------
country
city
zip
 
Bucket 2
--------
browser
device
 
Bucket 3
--------
campaign
source
medium

When querying:

attributes['browser']

ClickHouse® can determine which bucket may contain the key and read only that bucket rather than processing the entire map representation.

This reduces the amount of data that needs to be read for selective key lookups.

How Bucketed Serialization Works

The optimization occurs during MergeTree background merges.

Newly inserted parts continue to use the existing serialization format. During background merges, ClickHouse® rewrites the data into the bucketed format according to the configured settings.

This design has two advantages:

• Inserts remain lightweight because no additional bucketing work is required during ingestion.
• The optimized storage layout is created asynchronously as parts are merged.

As a result, write performance is preserved while read performance improves for eligible queries.

Configuration Options

ClickHouse® 26.3 introduces several settings to control map serialization behavior.

map_serialization_version

Determines which serialization format should be used.

map_serialization_version_for_zero_level_parts

Controls the serialization format used for newly created (zero-level) parts before background merges occur.

max_buckets_in_map

Defines the maximum number of buckets available for map serialization.

Increasing the number of buckets can improve selective key access for larger maps, although it also increases metadata and storage management overhead.

map_buckets_strategy

Specifies how ClickHouse® determines the appropriate bucket count for maps.

map_buckets_min_avg_size

Defines the minimum average map size before bucketing becomes beneficial.

These settings allow administrators to balance storage efficiency and query performance based on workload characteristics.

Performance Improvements

According to the ClickHouse® 26.3 release information, bucketed serialization can substantially accelerate queries that retrieve individual keys from large Map columns.

The reported improvements vary depending on:

• Average number of keys per map
• Data distribution
• Query patterns
• Bucket configuration

For some workloads, single-key lookups were reported to be 2× to 49× faster compared to the previous serialization format.

The greatest benefits are observed when:

• Maps contain many keys.
• Queries repeatedly access a small subset of keys.
• Large volumes of data are scanned.

Example

Consider the following table:

CREATE TABLE logs
(
    timestamp DateTime,
    metadata Map(String, String)
)
ENGINE = MergeTree
ORDER BY timestamp;

Example query:

SELECT
    metadata['service'],
    count()
FROM logs
GROUP BY metadata['service'];

With bucketed serialization enabled and after background merges have rewritten parts, ClickHouse® can reduce the amount of map data that needs to be read for the service key, improving query efficiency.

When Should You Use It?

Bucketed serialization is most beneficial for workloads that:

• Store large Map columns.
• Frequently query individual keys.
• Process billions of rows.
• Handle log analytics.
• Collect observability or telemetry data.
• Analyze event metadata.

If maps are very small or most queries retrieve the entire map, the performance gains may be limited.

Things to Keep in Mind

A few important considerations:

• Existing data is not rewritten immediately after upgrading.
• Optimized serialization is applied as background merges create new parts.
• Insert performance remains largely unchanged because bucketing is not performed during initial writes.
• Query performance improvements depend on workload and access patterns.

Understanding these characteristics helps set realistic expectations when evaluating the feature.

Why This Matters

Many real-world analytics systems rely on dynamic attributes that evolve over time. The Map data type provides flexibility, but efficient access to individual keys is essential for interactive analytics.

By introducing bucketed serialization, ClickHouse® improves the storage layout for Map columns without changing the SQL interface. Applications can continue using the same queries while benefiting from reduced I/O and faster key lookups after data has been reorganized through background merges.

Conclusion

ClickHouse® 26.3 introduces a meaningful enhancement to the Map data type through the new bucketed serialization format. Rather than treating an entire map as a single serialized structure during selective key access, the database organizes map entries into buckets, allowing queries to read only the relevant portion of the data.

The optimization is applied during MergeTree background merges, preserving fast insert performance while improving read efficiency for many analytical workloads. Depending on data characteristics and query patterns, the ClickHouse® team reports significant speedups for single-key lookups.

For organizations working with logs, telemetry, event metadata, or other semi-structured datasets, this enhancement provides a practical way to improve query performance without changing existing SQL queries or data models. As with any storage optimization, it's worth evaluating the feature against your own workload, but for large Map columns with frequent key-based access, ClickHouse® 26.3 offers a compelling improvement.

References

clickhouse_26.3_bog