---
title: "Incremental Clustering"
sidebar_position: 2
---

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

# Incremental Clustering

Paimon currently supports ordering append tables using SFC (Space-Filling Curve)(see [sort compact](../maintenance/dedicated-compaction#sort-compact) for more info). 
The resulting data layout typically delivers better performance for queries that target clustering keys. 
However, with the current SortCompaction, even when neither the data nor the clustering keys have changed, 
each run still rewrites the entire dataset, which is extremely costly. 

To address this, Paimon introduced a more flexible, incremental clustering mechanism—Incremental Clustering. 
On each run, it selects only a specific subset of files to cluster, avoiding a full rewrite. This enables low-cost, 
sort-based optimization of the data layout and improves query performance. In addition, with Incremental Clustering, 
you can adjust clustering keys without rewriting existing data, the layout evolves dynamically as cluster runs and 
gradually converges to an optimal state, significantly reducing the decision-making complexity around data layout.


Incremental Clustering supports:
- Support incremental clustering; minimizing write amplification as possible.
- Support small-file compaction; during rewrites, respect target-file-size.
- Support changing clustering keys; newly ingested data is clustered according to the latest clustering keys.
- Provide a full mode; when selected, the entire dataset will be reclustered.

Incremental Clustering is supported for append tables in both unaware-bucket mode (`bucket = -1`) and
bucketed mode (`bucket > 0`). For bucketed append tables, additional requirements apply because
clustering gives up the ordered append guarantee within buckets.

## Enable Incremental Clustering

To enable Incremental Clustering, the following configuration needs to be set for the table:
<table className="table table-bordered">
    <thead>
    <tr>
      <th className="text-left" style={{width: "20%"}}>Option</th>
      <th className="text-left" style={{width: "10%"}}>Value</th>
      <th className="text-left" style={{width: "5%"}}>Required</th>
      <th className="text-left" style={{width: "10%"}}>Type</th>
      <th className="text-left" style={{width: "55%"}}>Description</th>
    </tr>
    </thead>
    <tbody>
    <tr>
      <td><h5>clustering.incremental</h5></td>
      <td>true</td>
      <td style={{wordWrap: "break-word"}}>Yes</td>
      <td>Boolean</td>
      <td>Must be set to true to enable incremental clustering. Default is false.</td>
    </tr>
    <tr>
      <td><h5>clustering.columns</h5></td>
      <td>'clustering-columns'</td>
      <td style={{wordWrap: "break-word"}}>Yes</td>
      <td>String</td>
      <td>The clustering columns, in the format 'columnName1,columnName2'. It is not recommended to use partition keys as clustering keys.</td>
    </tr>
    <tr>
      <td><h5>clustering.strategy</h5></td>
      <td>'zorder' or 'hilbert' or 'order'</td>
      <td style={{wordWrap: "break-word"}}>No</td>
      <td>String</td>
      <td>The ordering algorithm used for clustering. If not set, It'll decided from the number of clustering columns. 'order' is used for 1 column, 'zorder' for less than 5 columns, and 'hilbert' for 5 or more columns.</td>
    </tr>
    <tr>
      <td><h5>clustering.incremental.mode</h5></td>
      <td>'global-sort' or 'local-sort'</td>
      <td style={{wordWrap: "break-word"}}>No</td>
      <td>Enum</td>
      <td>The sort mode for incremental clustering compaction. Default is <code>global-sort</code>. <code>global-sort</code> performs a global range shuffle across tasks before local sorting, output files are globally ordered by the clustering columns at the cost of network shuffling. <code>local-sort</code> skips the global shuffle and sorts rows only within each compaction task independently, each output file is internally ordered but there is no global ordering across files, this mode is cheaper and sufficient for per-file Parquet lookup optimizations.</td>
    </tr>
    </tbody>

</table>

For bucketed append tables (`bucket > 0`), you must also set the following option:

<table className="table table-bordered">
    <thead>
    <tr>
      <th className="text-left" style={{width: "20%"}}>Option</th>
      <th className="text-left" style={{width: "10%"}}>Value</th>
      <th className="text-left" style={{width: "5%"}}>Required</th>
      <th className="text-left" style={{width: "10%"}}>Type</th>
      <th className="text-left" style={{width: "55%"}}>Description</th>
    </tr>
    </thead>
    <tbody>
    <tr>
      <td><h5>bucket-append-ordered</h5></td>
      <td>false</td>
      <td style={{wordWrap: "break-word"}}>Yes</td>
      <td>Boolean</td>
      <td>Must be set to false for bucketed append tables with incremental clustering.</td>
    </tr>
    </tbody>

</table>

Bucketed append tables with Incremental Clustering do not support `deletion-vectors.enabled = true`.

Example:

```sql
ALTER TABLE T SET (
    'bucket-append-ordered' = 'false',
    'clustering.incremental' = 'true',
    'clustering.columns' = 'event_time,user_id',
    'clustering.strategy' = 'zorder'
);
```

Once Incremental Clustering for a table is enabled, you can run Incremental Clustering in batch mode periodically 
to continuously optimizes data layout of the table and deliver better query performance.

**Note**: Since common compaction also rewrites files, it may disrupt the ordered data layout built by Incremental Clustering. 
Therefore, when Incremental Clustering is enabled, the table no longer supports write-time compaction or dedicated compaction; 
clustering and small-file merging must be performed exclusively via Incremental Clustering runs.

## Run Incremental Clustering
:::info

The following examples submit batch compact jobs. They are the recommended way to run Incremental Clustering explicitly.

:::

To run a Incremental Clustering job, follow these instructions. 

You don't need to specify any clustering-related parameters when running Incremental Clustering,
these options are already defined as table options. If you need to change clustering settings, please update the corresponding table options.

<Tabs groupId="incremental-clustering">

<TabItem value="spark-sql" label="Spark SQL">

Run the following sql:

```sql
--set the write parallelism, if too big, may generate a large number of small files.
SET spark.sql.shuffle.partitions=10;

-- run incremental clustering
CALL sys.compact(table => 'T')

-- run incremental clustering with full mode, this will recluster all data
CALL sys.compact(table => 'T', compact_strategy => 'full')

-- run incremental clustering with global-sort mode (default)
-- performs a global range shuffle across tasks, output files are globally ordered
CALL sys.compact(table => 'T', options => 'clustering.incremental.mode=global-sort')

-- run incremental clustering with local-sort mode
-- sorts rows only within each task, no global shuffle, cheaper and sufficient for Parquet lookup optimizations
CALL sys.compact(table => 'T', options => 'clustering.incremental.mode=local-sort')
```

</TabItem>

<TabItem value="flink-action" label="Flink Action">

Run the following command to submit a incremental clustering job for the table.

```bash
<FLINK_HOME>/bin/flink run \
    /path/to/paimon-flink-action-@@VERSION@@.jar \
    compact \
    --warehouse <warehouse-path> \
    --database <database-name> \
    --table <table-name> \
    [--compact_strategy <minor / full>] \
    [--table_conf <table_conf>] \
    [--catalog_conf <paimon-catalog-conf> [--catalog_conf <paimon-catalog-conf> ...]]
```

Example: run incremental clustering with global-sort mode (default), output files are globally ordered across all tasks.

```bash
<FLINK_HOME>/bin/flink run \
    /path/to/paimon-flink-action-@@VERSION@@.jar \
    compact \
    --warehouse s3:///path/to/warehouse \
    --database test_db \
    --table test_table \
    --table_conf sink.parallelism=2 \
    --table_conf clustering.incremental.mode=global-sort \
    --compact_strategy minor \
    --catalog_conf s3.endpoint=https://****.com \
    --catalog_conf s3.access-key=***** \
    --catalog_conf s3.secret-key=*****
```

Example: run incremental clustering with local-sort mode, sorts rows only within each task without global shuffle, cheaper and sufficient for Parquet lookup optimizations.

```bash
<FLINK_HOME>/bin/flink run \
    /path/to/paimon-flink-action-@@VERSION@@.jar \
    compact \
    --warehouse s3:///path/to/warehouse \
    --database test_db \
    --table test_table \
    --table_conf sink.parallelism=2 \
    --table_conf clustering.incremental.mode=local-sort \
    --compact_strategy minor \
    --catalog_conf s3.endpoint=https://****.com \
    --catalog_conf s3.access-key=***** \
    --catalog_conf s3.secret-key=*****
```
* `--compact_strategy` Determines how to pick files to be cluster, the default is `minor`.
    * `full` : All files will be selected for clustered.
    * `minor` : Pick the set of files that need to be clustered based on specified conditions.

Note: write parallelism is set by `sink.parallelism`, if too big, may generate a large number of small files.

You can use `-D execution.runtime-mode=batch` or `-yD execution.runtime-mode=batch` (for the ON-YARN scenario) to use batch mode.

</TabItem>

</Tabs>

## Auto-Clustering For Historical Partition
:::info

Auto-clustering for historical partitions currently applies only to unaware-bucket append tables (`bucket = -1`).
Bucketed append clustering does not use the `clustering.history-partition.*` table options.

:::

While performing incremental clustering on recently active partitions, Paimon can automatically detect historical and 
inactive partitions and evaluate whether their data layout has reached an optimal state. 
For those historical partitions that have not yet achieved optimal layout, Paimon will also perform full clustering on them 
during the same operation, thereby improving their query performance.

To enable auto-clustering for historical partitions, the following configuration needs to be set for the table:
<table className="table table-bordered">
    <thead>
    <tr>
      <th className="text-left" style={{width: "20%"}}>Option</th>
      <th className="text-left" style={{width: "10%"}}>Value</th>
      <th className="text-left" style={{width: "5%"}}>Required</th>
      <th className="text-left" style={{width: "10%"}}>Type</th>
      <th className="text-left" style={{width: "55%"}}>Description</th>
    </tr>
    </thead>
    <tbody>
    <tr>
      <td><h5>clustering.history-partition.idle-to-full-sort</h5></td>
      <td>3d</td>
      <td style={{wordWrap: "break-word"}}>Yes</td>
      <td>Duration</td>
      <td>The duration after which a partition without new updates is considered a historical partition. Default is null.</td>
    </tr>
    <tr>
      <td><h5>clustering.history-partition.limit</h5></td>
      <td>5</td>
      <td style={{wordWrap: "break-word"}}>Yes</td>
      <td>Integer</td>
      <td>The limit of history partition number for automatically performing full clustering. Default value is 5.</td>
    </tr>
    </tbody>

</table>


## Implement
To balance write amplification and sorting effectiveness, Paimon leverages the LSM Tree notion of levels to stratify data files 
and uses the Universal Compaction strategy to select files for clustering.
- Newly written data lands in level-0; files in level-0 are unclustered.
- All files in level-i are produced by sorting within the same sorting set.
- By analogy with Universal Compaction: in level-0, each file is a sorted run; in level-i, all files together constitute a single sorted run. During clustering, the sorted run is the basic unit of work.

By introducing more levels, we can control the amount of data processed in each clustering run. 
Data at higher levels is more stably clustered and less likely to be rewritten, thereby mitigating write amplification while maintaining good sorting effectiveness.
