---
title: "Global Index"
sidebar_position: 8
---

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

# Global Index

## Overview

Global Index is a powerful indexing mechanism for Data Evolution (append) tables. It enables efficient row-level lookups and filtering
without full-table scans. Paimon supports multiple global index types:

- **[BTree Index](./global-index/btree)**: A B-tree based index for scalar column lookups. Supports equality, IN, range predicates, and can be combined across multiple columns with AND/OR logic.
- **[Bitmap Index](./global-index/bitmap)**: A bitmap based index for enum-like scalar dimensions and tag columns. Supports equality, IN, prefix match on string columns, complement predicates, and null checks with compressed row-id bitmaps.
- **[Vector Index](./global-index/vector)**: An approximate nearest neighbor (ANN) index powered by Paimon's vector index library for vector similarity search.
- **[Full-Text Index](./global-index/full-text)**: A full-text search index backed by the native full-text engine for text retrieval. Supports term matching and relevance scoring.
- **[Hybrid Search](./global-index/hybrid-search)**: A multi-route search API that combines results from multiple vector routes, multiple full-text routes, or both before reading table rows.

| Index Type | Best For | Notes |
|---|---|---|
| BTree | Scalar filters on numeric, string, date, and timestamp columns | Best when predicates are selective, such as equality, IN, range, and null checks. |
| Bitmap | Enum-like dimensions and tag columns | Best for equality, IN, string prefix match, complement predicates, and null checks over compressed row-id bitmaps. |
| Vector | Top-K similarity search on embeddings | Uses ANN algorithms. Tune build-time and search-time options to balance recall, latency, and index size. |
| Full-Text | Keyword search over text columns | Uses full-text scoring and tokenizer configuration stored with each index file. |
| Hybrid Search | Combining multiple vector routes, multiple full-text routes, or vector and full-text retrieval together | Runs multiple scored routes and merges them with a ranker before reading rows. |

Global indexes work on top of Data Evolution tables. To use global indexes, your table **must** have:

- `'bucket' = '-1'` (unaware-bucket mode)
- `'row-tracking.enabled' = 'true'`
- `'data-evolution.enabled' = 'true'`

> Global index queries may not be exact when the index only covers part of the table data. If a query predicate matches the index, Paimon returns only the results from the indexed portion. Matching records in data that has not been indexed yet will not be returned.

## Prerequisites

Create a table with the required properties:

<Tabs groupId="global-index-create-table">

<TabItem value="sql" label="SQL">

```sql
CREATE TABLE my_table (
    id INT,
    name STRING,
    embedding ARRAY<FLOAT>,
    content STRING
) TBLPROPERTIES (
    'bucket' = '-1',
    'row-tracking.enabled' = 'true',
    'data-evolution.enabled' = 'true',
    'global-index.enabled' = 'true'
);
```

</TabItem>

<TabItem value="python-sdk" label="Python SDK">

```python
import pyarrow as pa

from pypaimon import Schema

schema = Schema.from_pyarrow_schema(
    pa.schema([
        pa.field("id", pa.int32()),
        pa.field("name", pa.string()),
        pa.field("embedding", pa.list_(pa.float32())),
        pa.field("content", pa.string()),
    ]),
    options={
        "bucket": "-1",
        "row-tracking.enabled": "true",
        "data-evolution.enabled": "true",
        "global-index.enabled": "true",
    },
)

catalog.create_table("db.my_table", schema, ignore_if_exists=False)
```

</TabItem>

</Tabs>

## Lifecycle

Create global indexes for all partitions or only selected partitions:

<Tabs groupId="global-index-build">

<TabItem value="sql" label="SQL">

```sql
CALL sys.create_global_index(
    table => 'db.my_table',
    index_column => 'name',
    index_type => 'btree'
);

CALL sys.create_global_index(
    table => 'db.my_table',
    index_column => 'name',
    index_type => 'btree',
    partitions => 'dt=2026-06-18;dt=2026-06-19'
);
```

</TabItem>

<TabItem value="python-sdk" label="Python SDK">

```python
table = catalog.get_table("db.my_table")

added_files = table.create_global_index("name")
print(added_files)
```

The API returns the number of committed index files. You can pass build options and restrict the
build to selected partitions:

```python
added_files = table.create_global_index(
    "name",
    index_type="btree",
    partitions=[{"dt": "2026-06-18"}, {"dt": "2026-06-19"}],
    options={"sorted-index.records-per-range": "10000000"},
)
```

</TabItem>

</Tabs>

PyPaimon global index build currently supports single-column BTree indexes,
single-column Bitmap indexes, single-column paimon-vindex vector indexes,
and single-column full-text indexes on tables with row tracking enabled.

Drop index files:

<Tabs groupId="global-index-drop">

<TabItem value="sql" label="SQL">

```sql
CALL sys.drop_global_index(
    table => 'db.my_table',
    index_column => 'name',
    index_type => 'btree'
);
```

</TabItem>

<TabItem value="python-sdk" label="Python SDK">

```python
table = catalog.get_table("db.my_table")

dropped_files = table.drop_global_index("name", index_type="btree")
print(dropped_files)
```

You can also restrict the drop to selected partitions, or count matched files without committing:

```python
matched_files = table.drop_global_index(
    "name",
    index_type="btree",
    partitions=[{"dt": "2026-06-18"}, {"dt": "2026-06-19"}],
    dry_run=True,
)
```

</TabItem>

</Tabs>

Global indexes are stored in index files and recorded in table metadata. To inspect index files and
their row-id coverage, query the `table_indexes` system table:

<Tabs groupId="global-index-table-indexes">

<TabItem value="sql" label="SQL">

```sql
SELECT index_type, index_field_name, row_range_start, row_range_end
FROM my_table$table_indexes
WHERE index_field_name IS NOT NULL;
```

</TabItem>

<TabItem value="python-sdk" label="Python SDK">

```python
import pyarrow.compute as pc

table_indexes = catalog.get_table("db.my_table$table_indexes")
read_builder = table_indexes.new_read_builder().with_projection([
    "index_type",
    "index_field_name",
    "row_range_start",
    "row_range_end",
])

pa_table = read_builder.new_read().to_arrow(
    read_builder.new_scan().plan().splits()
)
pa_table = pa_table.filter(pc.is_valid(pa_table["index_field_name"]))
print(pa_table)
```

</TabItem>

</Tabs>

You can also query `file_key_ranges` to inspect data file row-id ranges and diagnose coverage:

<Tabs groupId="global-index-file-key-ranges">

<TabItem value="sql" label="SQL">

```sql
SELECT file_path, first_row_id, record_count
FROM my_table$file_key_ranges;
```

</TabItem>

<TabItem value="python-sdk" label="Python SDK">

```python
file_key_ranges = catalog.get_table("db.my_table$file_key_ranges")
read_builder = file_key_ranges.new_read_builder().with_projection([
    "file_path",
    "first_row_id",
    "record_count",
])

pa_table = read_builder.new_read().to_arrow(
    read_builder.new_scan().plan().splits()
)
print(pa_table)
```

</TabItem>

</Tabs>

For workloads that need newly appended rows to become visible only after existing global indexes
cover them, enable the visibility callback:

<Tabs groupId="global-index-visibility-callback">

<TabItem value="sql" label="SQL">

```sql
ALTER TABLE my_table SET (
    'visibility-callback.enabled' = 'true',
    'visibility-callback.timeout' = '30 min'
);
```

</TabItem>

<TabItem value="python-sdk" label="Python SDK">

```python
from pypaimon.schema.schema_change import SchemaChange

catalog.alter_table(
    "db.my_table",
    [
        SchemaChange.set_option("visibility-callback.enabled", "true"),
        SchemaChange.set_option("visibility-callback.timeout", "30 min"),
    ],
)
```

</TabItem>

</Tabs>

## Coverage and Freshness

Global index files cover row-id ranges. If more rows are appended after an index is built, those
new rows are not automatically covered by the existing index files. Build the global index again
to create index files for newly uncovered data. Scalar queries default to `full` search, while
vector and full-text queries default to `fast` and only search indexed row ranges.

To improve freshness for query types that support raw-data search, set:

<Tabs groupId="global-index-search-mode">

<TabItem value="sql" label="SQL">

```sql
ALTER TABLE my_table SET ('vector-index.search-mode' = 'full');
```

</TabItem>

<TabItem value="python-sdk" label="Python SDK">

```python
from pypaimon.schema.schema_change import SchemaChange

catalog.alter_table(
    "db.my_table",
    [SchemaChange.set_option("vector-index.search-mode", "full")],
)
```

For a read-only override on an existing `Table` instance:

```python
full_table = table.copy({"vector-index.search-mode": "full"})
```

</TabItem>

</Tabs>

With `full` search, supported global-index queries first use the snapshot `nextRowId` and global
index row-id coverage to detect whether any row range is missing from the index. Raw data is scanned
only when such a gap exists. Use `detail` search when data files may have been rewritten or updated
after index creation; it scans data file metadata to find the exact unindexed row ranges and can
handle index invalidation caused by updates or rewrites.

Use `scalar-index.search-mode`, `vector-index.search-mode`, or
`full-text-index.search-mode` for one index family. The legacy
`global-index.search-mode` has no default and is used as a fallback when the corresponding
family-specific option is not explicitly configured.

To temporarily disable global-index scan acceleration while keeping the index files, set:

<Tabs groupId="global-index-enabled">

<TabItem value="sql" label="SQL">

```sql
ALTER TABLE my_table SET ('global-index.enabled' = 'false');
```

</TabItem>

<TabItem value="python-sdk" label="Python SDK">

```python
from pypaimon.schema.schema_change import SchemaChange

catalog.alter_table(
    "db.my_table",
    [SchemaChange.set_option("global-index.enabled", "false")],
)
```

</TabItem>

</Tabs>

Set it back to `true` to use global indexes during scans again.

## Build Options

These table options affect global index build and read behavior:

| Option | Default | Description |
|---|---|---|
| `global-index.enabled` | `true` | Whether scans can use global indexes. |
| `global-index.search-mode` | Not set | Legacy fallback search mode for global-index queries. Family-specific options take precedence. |
| `scalar-index.search-mode` | `fast` | Search mode for BTree and Bitmap queries. |
| `vector-index.search-mode` | `fast` | Search mode for vector queries. |
| `full-text-index.search-mode` | `fast` | Search mode for full-text queries. |
| `global-index.external-path` | Not set | Root directory for global index files. If not set, files are stored under the table index directory. |
| `global-index.column-update-action` | `THROW_ERROR` | Action for updates to indexed columns. `THROW_ERROR` rejects the update, `DROP_PARTITION_INDEX` drops affected partition indexes, and `IGNORE` keeps existing index files unchanged. Use `IGNORE` only when a stale index is acceptable; for example, a vector updated from `NULL` to a value remains invisible until the index is rebuilt. |
| `sorted-index.records-per-range` | `10000000` | Expected number of records per sorted global index file for BTree and Bitmap builds. |
| `sorted-index.build.max-parallelism` | `4096` | Maximum Flink or Spark parallelism for building sorted global indexes. |
| `global-index.row-count-per-shard` | `100000` | Target row count per shard for non-sorted global index builds such as vector and full-text indexes. |
| `global-index.build.max-shard` | `32` | Preferred maximum shard count for global index builds. |
| `global-index.build.max-parallelism` | `4096` | Maximum Flink or Spark parallelism for building non-sorted global indexes. |
| `global-index.thread-num` | `32` | Maximum number of concurrent threads for global index I/O. |
| `visibility-callback.enabled` | `false` | Whether batch or bounded-stream commits wait until existing global indexes cover newly added files. |
| `visibility-callback.timeout` | `30 min` | Maximum wait time for visibility callback. |

## BTree Index

Use BTree indexes for scalar column lookups and range predicates. See [BTree Index](./global-index/btree)
for build and query examples.

## Bitmap Index

Use Bitmap indexes for enum-like dimensions and tag columns queried by equality, IN, string prefix
match, complement, or null predicates. See [Bitmap Index](./global-index/bitmap) for build examples,
options, and file format details.

## Vector Index

Use Vector indexes for approximate nearest neighbor (ANN) search. See [Vector Index](./global-index/vector)
for supported index types, options, and search examples.

## Full-Text Index

Use Full-Text indexes for text retrieval backed by the native full-text engine. See [Full-Text Index](./global-index/full-text)
for tokenizer options and search examples.

## Hybrid Search

Use Hybrid Search to query multiple vector routes, multiple full-text routes, or a mix of both in one request. See
[Hybrid Search](./global-index/hybrid-search) for route configuration, rankers, and API examples.
