Key | Default | Type | Description
--- | --- | --- | ---
##### add-column-before-partition | false | Boolean | If true, when adding a new column without specifying a position, the column will be placed before the first partition column instead of at the end of the schema. This only takes effect for partitioned tables.
##### aggregation.remove-record-on-delete | false | Boolean | Whether to remove the whole row in aggregation engine when -D records are received.
##### alter-column-null-to-not-null.disabled | true | Boolean | If true, it disables altering column type from null to not null. Default is true. Users can disable this option to explicitly convert null column type to not null.
##### async-file-write | true | Boolean | Whether to enable asynchronous IO writing when writing files.
##### auto-create | false | Boolean | Whether to create underlying storage when reading and writing the table.
##### blob-as-descriptor | false | Boolean | Write blob field using blob descriptor rather than blob bytes.
##### blob-compaction.enabled | false | Boolean | Whether to compact blob files when compacting a data evolution table.
##### blob-descriptor-field | (none) | String | Comma-separated field names to treat as BLOB fields and store as serialized BlobDescriptor bytes inline in data files.
##### blob-descriptor.source-table | (none) | String | The source table whose FileIO is used to read descriptor-backed BLOB content and copy it into the target table's managed BLOB storage. The table must belong to the current catalog and can include a branch suffix, for example db.table$branch_rt. This option is not supported for target tables without a catalog loader, including external tables in REST catalogs. When set, other blob-descriptor.* FileIO options are ignored.
##### blob-field | (none) | String | Specifies column names that should be stored as blob type. This is used when you want to treat a BYTES column as a BLOB. Fields listed in blob-descriptor-field or blob-view-field are also treated as BLOB fields.
##### blob-view-field | (none) | String | Comma-separated field names to treat as BLOB fields and store as serialized BlobViewStruct bytes inline in data files and resolve from upstream tables at read time.
##### blob-view.resolve.enabled | true | Boolean | Whether to resolve blob-view-field values from upstream tables at read time. Set to false to preserve BlobViewStruct references when forwarding blob view values to another blob-view table.
##### blob-write-null-on-fetch-failure | false | Boolean | Whether to write NULL for a descriptor BLOB value when the referenced resource cannot be fetched during Flink writes (e.g. invalid URI or HTTP errors other than 404). HTTP 404 is handled by 'blob-write-null-on-missing-file'. When false, the write fails when the descriptor is read.
##### blob-write-null-on-missing-file | false | Boolean | Whether to write NULL for a descriptor BLOB value when the referenced file or HTTP resource does not exist during Flink writes. When false, the write fails when the descriptor is read.
##### blob.copy-buffer-size | 4 kb | MemorySize | Buffer size used when copying BLOB payloads into BLOB files.
##### blob.split-by-file-size | (none) | Boolean | Whether to consider blob file size as a factor when performing scan splitting.
##### blob.target-file-size | (none) | MemorySize | Target size of a blob file. Default is value of TARGET_FILE_SIZE.
##### bucket | -1 | Integer | Bucket number for file store. It should either be equal to -1 (dynamic bucket mode), -2 (postpone bucket mode), or it must be greater than 0 (fixed bucket mode).
##### bucket-append-ordered | true | Boolean | Whether to ignore the order of the buckets when reading data from an append-only table.
##### bucket-function.type | default | Enum | The bucket function for paimon bucket. Possible values: "default": The default bucket function which will use arithmetic: bucket_id = Math.abs(hash_bucket_binary_row % numBuckets) to get bucket. "mod": The modulus bucket function which will use modulus arithmetic: bucket_id = Math.floorMod(bucket_key_value, numBuckets) to get bucket. Note: the bucket key must be a single field of INT or BIGINT datatype. "hive": The hive bucket function which will use hive-compatible hash arithmetic to get bucket.
##### bucket-key | (none) | String | Specify the paimon distribution policy. Data is assigned to each bucket according to the hash value of bucket-key. If you specify multiple fields, delimiter is ','. If not specified, the primary key will be used; if there is no primary key, the full row will be used.
##### cache-page-size | 64 kb | MemorySize | Memory page size for caching.
##### chain-table.chain-partition-keys | (none) | String | Partition keys that participate in chain logic. Must be a contiguous suffix of the table's partition keys. Comma-separated. If not set, all partition keys participate in chain.
##### chain-table.enabled | false | Boolean | Whether enabled chain table.
##### chain-table.split.key-range-enabled | true | Boolean | If true, a batch chain-table scan splits each bucket's snapshot and delta files into multiple splits by key range to improve read parallelism. Files with intersecting key ranges always stay in the same split so that all versions of a key across the snapshot and delta branches are merged together. Set to false to fall back to one split per bucket.
##### chain-table.streaming.merge-snapshot | false | Boolean | If true, the starting phase of chain table streaming read performs anchor-based chain merging: for each group it merges the latest snapshot partition with delta partitions whose chain key is strictly greater than the snapshot chain key. This allows streaming readers to see cross-branch deletions and updates at the cost of a heavier startup scan. When false (default), the starting phase only reads the latest snapshot partition per group and later delta partitions as separate splits, which is lightweight but may not reflect cross-branch deletes.
##### changelog-file.compression | (none) | String | Changelog file compression.
##### changelog-file.format | (none) | String | Specify the message format of changelog files, currently parquet, avro and orc are supported.
##### changelog-file.prefix | "changelog-" | String | Specify the file name prefix of changelog files.
##### changelog-file.stats-mode | (none) | String | Changelog file metadata stats collection. none, counts, truncate(16), full is available.
##### changelog-producer | none | Enum | Whether to double write to a changelog file. This changelog file keeps the details of data changes, it can be read directly during stream reads. This can be applied to tables with primary keys. Possible values: "none": No changelog file. "input": Double write to a changelog file when flushing memory table, the changelog is from input. "full-compaction": Generate changelog files with each full compaction. "lookup": Generate changelog files through 'lookup' compaction.
##### changelog-producer.row-deduplicate | false | Boolean | Whether to generate -U, +U changelog for the same record. This configuration is only valid for the changelog-producer is lookup or full-compaction.
##### changelog-producer.row-deduplicate-ignore-fields | (none) | String | Fields that are ignored for comparison while generating -U, +U changelog for the same record. This configuration is only valid for the changelog-producer.row-deduplicate is true.
##### changelog.num-retained.max | (none) | Integer | The maximum number of completed changelog to retain. Should be greater than or equal to the minimum number.
##### changelog.num-retained.min | (none) | Integer | The minimum number of completed changelog to retain. Should be greater than or equal to 1.
##### changelog.time-retained | (none) | Duration | The maximum time of completed changelog to retain.
##### clustering.columns | (none) | String | Specifies the column name(s) used for comparison during range partitioning, in the format 'columnName1,columnName2'. If not set or set to an empty string, it indicates that the range partitioning feature is not enabled. This option will be effective only for append table without primary keys and batch execution mode.
##### clustering.history-partition.idle-to-full-sort | (none) | Duration | The duration after which a partition without new updates is considered a historical partition. Historical partitions will be automatically fully clustered during the cluster operation.
##### clustering.history-partition.limit | 5 | Integer | The limit of history partition number for automatically performing full clustering.
##### clustering.incremental | false | Boolean | Whether enable incremental clustering.
##### clustering.incremental.mode | global-sort | Enum | The sort mode for incremental clustering compaction. 'global-sort' (default) performs a global range shuffle so output files are globally ordered. 'local-sort' skips the global shuffle and only sorts rows within each compaction task, producing files that are internally ordered. 'local-sort' is cheaper and sufficient for Parquet lookup optimizations. Possible values: "global-sort": Perform global range shuffle and then local sort. Output files are globally ordered but require network shuffling. "local-sort": Sort rows only within each compaction task without global shuffle. Every output file is internally ordered.
##### clustering.incremental.optimize-write | false | Boolean | Whether enable perform clustering before write phase when incremental clustering is enabled.
##### clustering.strategy | "auto" | String | Specifies the comparison algorithm used for range partitioning, including 'zorder', 'hilbert', and 'order', corresponding to the z-order curve algorithm, hilbert curve algorithm, and basic type comparison algorithm, respectively. When not configured, it will automatically determine the algorithm based on the number of columns in 'clustering.by-columns'. 'order' is used for 1 column, 'zorder' for less than 5 columns, and 'hilbert' for 5 or more columns.
##### commit.callback.#.param | (none) | String | Parameter string for the constructor of class #. Callback class should parse the parameter by itself.
##### commit.callbacks | (none) | String | A list of commit callback classes to be called after a successful commit. Class names are connected with comma (example: com.test.CallbackA,com.sample.CallbackB).
##### commit.discard-duplicate-files | false | Boolean | Whether discard duplicate files in commit.
##### commit.force-compact | false | Boolean | Whether to force a compaction before commit.
##### commit.force-create-snapshot | false | Boolean | In streaming job, whether to force creating snapshot when there is no data in this write-commit phase.
##### commit.max-retries | 10 | Integer | Maximum number of retries when commit failed.
##### commit.max-retry-wait | 10 s | Duration | Max retry wait time when commit failed.
##### commit.min-retry-wait | 10 ms | Duration | Min retry wait time when commit failed.
##### commit.strict-mode.last-safe-snapshot | (none) | Long | If set, committer will check if there are other commit user's snapshot starting from the snapshot after this one. If found a COMPACT / OVERWRITE snapshot, or found a APPEND snapshot which committed files to fixed bucket, commit will be aborted.If the value of this option is -1, committer will not check for its first commit.
##### commit.timeout | (none) | Duration | Timeout duration of retry when commit failed.
##### commit.user-prefix | (none) | String | Specifies the commit user prefix.
##### compaction.delete-ratio-threshold | 0.2 | Double | Ratio of the deleted rows in a data file to be forced compacted for append-only table.
##### compaction.file-num-limit | 100000 | Integer | To avoid OOM caused by scanning compaction files, you can use this option to limit the for unaware-bucket append table compaction.
##### compaction.force-rewrite-all-files | false | Boolean | Whether to force pick all files for a full compaction. Usually seen in a compaction task to external paths.
##### compaction.force-up-level-0 | false | Boolean | If set to true, compaction strategy will always include all level 0 files in candidates.
##### compaction.incremental-size-threshold | (none) | MemorySize | When incremental size is bigger than this threshold, force a full compaction.
##### compaction.max-size-amplification-percent | 200 | Integer | The size amplification is defined as the amount (in percentage) of additional storage needed to store a single byte of data in the merge tree for changelog mode table.
##### compaction.min.file-num | 5 | Integer | For file set [f_0,...,f_N], the minimum file number to trigger a compaction for append-only table.
##### compaction.offpeak-ratio | 0 | Integer | Allows you to set a different (by default, more aggressive) percentage ratio for determining whether larger sorted run's size are included in compactions during off-peak hours. Works in the same way as compaction.size-ratio. Only applies if offpeak.start.hour and offpeak.end.hour are also enabled. For instance, if your cluster experiences low pressure between 2 AM and 6 PM , you can configure `compaction.offpeak.start.hour=2` and `compaction.offpeak.end.hour=18` to define this period as off-peak hours. During these hours, you can increase the off-peak compaction ratio (e.g. `compaction.offpeak-ratio=20`) to enable more aggressive data compaction
##### compaction.offpeak.end.hour | -1 | Integer | The end of off-peak hours, expressed as an integer between 0 and 23, exclusive. Set to -1 to disable off-peak.
##### compaction.offpeak.start.hour | -1 | Integer | The start of off-peak hours, expressed as an integer between 0 and 23, inclusive Set to -1 to disable off-peak
##### compaction.optimization-interval | (none) | Duration | Implying how often to perform an optimization compaction, this configuration is used to ensure the query timeliness of the read-optimized system table.
##### compaction.size-ratio | 1 | Integer | Percentage flexibility while comparing sorted run size for changelog mode table. If the candidate sorted run(s) size is 1% smaller than the next sorted run's size, then include next sorted run into this candidate set.
##### compaction.skip-expired-partitions | false | Boolean | Whether to skip compacting partitions that are already expired according to 'partition.expiration-time'. Only effective when 'partition.expiration-time' is set and 'partition.expiration-strategy' is 'values-time'. Note: even when this option is enabled, expired partitions may still be deleted during the compaction commit phase as a side effect of partition expiration triggered by committing the remaining active partitions.
##### compaction.small-file-ratio | 0.7 | Double | The ratio of target file size. Files whose size is smaller than target-file-size * compaction.small-file-ratio will be picked for compaction rewriting. This avoids compacting the same file repeatedly due to compression inaccuracy causing output files to be slightly smaller than the target size.
##### compaction.total-size-threshold | (none) | MemorySize | When total size is smaller than this threshold, force a full compaction.
##### consumer-id | (none) | String | Consumer id for recording the offset of consumption in the storage.
##### consumer.changelog-only | false | Boolean | If true, consumer will only affect changelog expiration and will not prevent snapshot from being expired.
##### consumer.expiration-time | (none) | Duration | The expiration interval of consumer files. A consumer file will be expired if it's lifetime after last modification is over this value.
##### consumer.ignore-progress | false | Boolean | Whether to ignore consumer progress for the newly started job.
##### consumer.mode | exactly-once | Enum | Specify the consumer consistency mode for table. Possible values: "exactly-once": Readers consume data at snapshot granularity, and strictly ensure that the snapshot-id recorded in the consumer is the snapshot-id + 1 that all readers have exactly consumed. "at-least-once": Each reader consumes snapshots at a different rate, and the snapshot with the slowest consumption progress among all readers will be recorded in the consumer.
##### continuous.discovery-interval | 10 s | Duration | The discovery interval of continuous reading.
##### cross-partition-upsert.bootstrap-parallelism | 10 | Integer | The parallelism for bootstrap in a single task for cross partition upsert.
##### cross-partition-upsert.index-ttl | (none) | Duration | The TTL in local index for cross partition upsert (primary keys not contain all partition fields), this can avoid maintaining too many indexes and lead to worse and worse performance, but please note that this may also cause data duplication.
##### data-evolution.compaction.rewrite-row-ids | false | Boolean | Whether data-evolution compaction may rewrite row IDs while physically applying deletion vectors. Enable only when callers do not rely on stable _ROW_ID; this invalidates row-id based references and drops global indexes for affected partitions.
##### data-evolution.enabled | false | Boolean | Whether enable data evolution for row tracking table.
##### data-evolution.merge-into.file-pruning | true | Boolean | If true, enables the file-level pruning step for MergeInto partial column update on data-evolution tables. Set this to false when most files in the target partition are expected to be updated, so that the overhead of collecting touched file IDs outweighs the benefit of pruning untouched files.
##### data-evolution.merge-into.source-persist | false | Boolean | Whether to persist source when process merge into action on data evolution table.
##### data-evolution.reassign.skip-contiguous-row-count | 1000000000 | Long | Strictly contiguous same-partition logical row-id runs containing more than this number of rows are excluded from row-id reassignment. Set to 0 to disable this filtering.
##### data-evolution.row-id-conflict-rewrite.max-size | 256 mb | MemorySize | Maximum total size of current data files whose row-id ranges PyPaimon may automatically rebase staged updates against when a concurrent compaction changes file boundaries. Set to 0 B to disable.
##### data-evolution.row-sidecar.enabled | false | Boolean | Whether to generate row-store sidecar files for normal data files on data evolution tables. The sidecar files are used to accelerate sparse row-id reads.
##### data-evolution.row-sidecar.max-selected-rows | 4096 | Long | Maximum selected row count for reading a row-store sidecar file. The sidecar is used only when the selected rows are no more than this value and the selected row ratio is no more than data-evolution.row-sidecar.max-selection-ratio.
##### data-evolution.row-sidecar.max-selection-ratio | 0.05 | Double | Maximum selected row ratio for reading a row-store sidecar file. The value must be in (0, 1]. The sidecar is used only when the selected row ratio is no more than this value and the selected row count is no more than data-evolution.row-sidecar.max-selected-rows.
##### data-file.external-paths | (none) | String | The external paths where the data of this table will be written, multiple elements separated by commas.
##### data-file.external-paths.specific-fs | (none) | String | The specific file system of the external path when data-file.external-paths.strategy is set to specific-fs, should be the prefix scheme of the external path, now supported are s3 and oss.
##### data-file.external-paths.strategy | none | Enum | The strategy of selecting an external path when writing data. Possible values: "none": Do not choose any external storage, data will still be written to the default warehouse path. "specific-fs": Select a specific file system as the external path. Currently supported are S3 and OSS. "round-robin": When writing a new file, a path is chosen from data-file.external-paths in turn. "entropy-inject": When writing a new file, a path is chosen based on the hash value of the file's content. "weight-robin": When writing a new file, a path is chosen based on configured weights.
##### data-file.external-paths.weights | (none) | String | The weights for external paths when data-file.external-paths.strategy is set to weight-robin. Format: 'weight1,weight2,...' with weights corresponding to paths in data-file.external-paths by order. Example: '10,5,15' means first path has weight 10, second 5, third 15. Weights must be positive integers.
##### data-file.path-directory | (none) | String | Specify the path directory of data files.
##### data-file.prefix | "data-" | String | Specify the file name prefix of data files.
##### data-file.thin-mode | false | Boolean | Enable data file thin mode to avoid duplicate columns storage.
##### delete.force-produce-changelog | false | Boolean | Force produce changelog in delete sql, or you can use 'streaming-read-overwrite' to read changelog from overwrite commit.
##### deletion-vector.index-file.target-size | 2 mb | MemorySize | The target size of deletion vector index file.
##### deletion-vectors.bitmap64 | false | Boolean | Enable 64 bit bitmap implementation. Note that only 64 bit bitmap implementation is compatible with Iceberg.
##### deletion-vectors.enabled | false | Boolean | Whether to enable deletion vectors mode. In this mode, index files containing deletion vectors are generated when data is written, which marks the data for deletion. During read operations, by applying these index files, merging can be avoided.
##### deletion-vectors.merge-on-read | false | Boolean | When deletion vectors are enabled, uncompacted files are not visible by default. Set this to true to enable merge-on-read, which makes uncompacted data visible at the cost of read performance. This option only affects batch scan visibility of DV level-0 files, it does not change streaming scan or changelog behavior.
##### deletion-vectors.modifiable | false | Boolean | Whether to enable modifying deletion vectors mode.
##### disable-explicit-type-casting | false | Boolean | If true, it disables explicit type casting. For ex: it disables converting LONG type to INT type. Users can enable this option to disable explicit type casting
##### dynamic-bucket.assigner-parallelism | (none) | Integer | Parallelism of assigner operator for dynamic bucket mode, it is related to the number of initialized bucket, too small will lead to insufficient processing speed of assigner.
##### dynamic-bucket.initial-buckets | (none) | Integer | Initial buckets for a partition in assigner operator for dynamic bucket mode.
##### dynamic-bucket.max-buckets | -1 | Integer | Max buckets for a partition in dynamic bucket mode, It should either be equal to -1 (unlimited), or it must be greater than 0 (fixed upper bound).
##### dynamic-bucket.target-row-num | 2000000 | Long | If the bucket is -1, for primary key table, is dynamic bucket mode, this option controls the target row number for one bucket.
##### dynamic-partition-overwrite | true | Boolean | Whether only overwrite dynamic partition when overwriting a partitioned table with dynamic partition columns. Works only when the table has partition keys.
##### end-input.check-partition-expire | false | Boolean | Optional endInput check partition expire used in case of batch mode or bounded stream.
##### fields.default-aggregate-function | (none) | String | Default aggregate function of all fields for partial-update and aggregate merge function.
##### file-index.in-manifest-threshold | 500 bytes | MemorySize | The threshold to store file index bytes in manifest.
##### file-index.read.enabled | true | Boolean | Whether enabled read file index.
##### file-operation.thread-num | (none) | Integer | The maximum number of concurrent file operations. By default is the number of processors available to the Java virtual machine.
##### file-reader-async-threshold | 10 mb | MemorySize | The threshold for read file async.
##### file.block-size | (none) | MemorySize | File block size of format, default value of orc stripe is 64 MB, and parquet row group is 128 MB.
##### file.compression | "zstd" | String | Default file compression. For faster read and write, it is recommended to use zstd.
##### file.compression.per.level |  | Map | Define different compression policies for different level, you can add the conf like this: 'file.compression.per.level' = '0:lz4,1:zstd'.
##### file.compression.zstd-level | 1 | Integer | Default file compression zstd level. For higher compression rates, it can be configured to 9, but the read and write speed will significantly decrease.
##### file.format | "parquet" | String | Specify the message format of data files, currently orc, parquet and avro are supported.
##### file.format.per.level |  | Map | Define different file format for different level, you can add the conf like this: 'file.format.per.level' = '0:avro,3:parquet', if the file format for level is not provided, the default format which set by `file.format` will be used.
##### file.suffix.include.compression | false | Boolean | Whether to add file compression type in the file name of data file and changelog file.
##### force-lookup | false | Boolean | Whether to force the use of lookup for compaction.
##### format-table.commit-hive-sync-url | (none) | String | Format table commit hive sync uri.
##### format-table.file.compression | (none) | String | Format table file compression.
##### format-table.implementation | paimon | Enum | Format table uses paimon or engine. Possible values: "paimon": Paimon format table implementation. "engine": Engine format table implementation.
##### format-table.partition-path-only-value | false | Boolean | Format table file path only contain partition value.
##### format-table.scan.list-parallelism | 64 | Integer | The parallelism of listing partition files during split planning for a Format Table with catalog-managed partitions.
##### full-compaction.delta-commits | (none) | Integer | For streaming write, full compaction will be constantly triggered after delta commits. For batch write, full compaction will be triggered with each commit as long as this value is greater than 0.
##### full-text-index.search-mode | fast | Enum | Search mode for full-text index queries. Possible values: "fast": Only search indexed data. "full": Use snapshot next row id and global index coverage to detect missing row ids, and scan raw data only when a gap exists. "detail": Scan data files to find exact unindexed rows. This can handle index invalidation caused by updates or rewrites.
##### global-index.build.max-parallelism | 4096 | Integer | The max parallelism of Flink/Spark for building global index.
##### global-index.build.max-shard | 32 | Integer | The preferred max number of shards for building global index. If the number of shards calculated by 'global-index.row-count-per-shard' exceeds this value, max-shard will be automatically increased to accommodate the data volume while keeping 'global-index.row-count-per-shard' unchanged.
##### global-index.column-update-action | THROW_ERROR | Enum | Defines the action to take when an update modifies columns that are covered by a global index. IGNORE leaves existing index files unchanged and may make the index stale. Possible values: "THROW_ERROR" "DROP_PARTITION_INDEX" "IGNORE"
##### global-index.enabled | true | Boolean | Whether to enable global index for scan.
##### global-index.external-path | (none) | String | Global index root directory, if not set, the global index files will be stored under the <table-root-directory>/index.
##### global-index.row-count-per-shard | 100000 | Long | Row count per shard for global index.
##### global-index.search-mode | (none) | Enum | Fallback search mode for global index queries. Possible values: "fast": Only search indexed data. "full": Use snapshot next row id and global index coverage to detect missing row ids, and scan raw data only when a gap exists. "detail": Scan data files to find exact unindexed rows. This can handle index invalidation caused by updates or rewrites.
##### global-index.thread-num | 32 | Integer | The maximum number of concurrent threads for global index I/O. Must be greater than 0.
##### ignore-delete | false | Boolean | Whether to ignore delete records.
##### ignore-update-before | false | Boolean | Whether to ignore update-before records.
##### incremental-between | (none) | String | Read incremental changes between start snapshot (exclusive) and end snapshot (inclusive), for example, '5,10' means changes between snapshot 5 and snapshot 10.
##### incremental-between-scan-mode | auto | Enum | Scan kind when Read incremental changes between start snapshot (exclusive) and end snapshot (inclusive). Possible values: "auto": Scan changelog files for the table which produces changelog files. Otherwise, scan newly changed files. "delta": Scan newly changed files between snapshots. "changelog": Scan changelog files between snapshots. "diff": Get diff by comparing data of end snapshot with data of start snapshot.
##### incremental-between-tag-to-snapshot | false | Boolean | Whether to read incremental changes between the snapshot corresponding to the tag.
##### incremental-between-timestamp | (none) | String | Read incremental changes between start timestamp (exclusive) and end timestamp (inclusive), for example, 't1,t2' means changes between timestamp t1 and timestamp t2.
##### incremental-to-auto-tag | (none) | String | Used to specify the end tag (inclusive), and Paimon will find an earlier tag and return changes between them. If the tag doesn't exist or the earlier tag doesn't exist, return empty. This option requires 'tag.creation-period' and 'tag.period-formatter' configured.
##### index-file-in-data-file-dir | false | Boolean | Whether index file in data file directory.
##### local-kv-db.block-size | 4 kb | MemorySize | Block size of the local key-value database.
##### local-merge-buffer-size | (none) | MemorySize | Local merge will buffer and merge input records before they're shuffled by bucket and written into sink. The buffer will be flushed when it is full. Mainly to resolve data skew on primary keys. We recommend starting with 64 mb when trying out this feature.
##### local-sort.max-num-file-handles | 128 | Integer | The maximal fan-in for external merge sort. It limits the number of file handles. If it is too small, may cause intermediate merging. But if it is too large, it will cause too many files opened at the same time, consume memory and lead to random reading.
##### lookup-compact | RADICAL | Enum | Lookup compact mode used for lookup compaction. Possible values: "RADICAL" "GENTLE"
##### lookup-compact.max-interval | (none) | Integer | The max interval for a gentle mode lookup compaction to be triggered. For every interval, a forced lookup compaction will be performed to flush L0 files to higher level. This option is only valid when lookup-compact mode is gentle.
##### lookup-wait | true | Boolean | When need to lookup, commit will wait for compaction by lookup.
##### lookup.blob-as-descriptor | false | Boolean | When enabled, the lookup join stores only the BlobDescriptor (a lightweight reference containing file URI, offset, and length) for BLOB fields instead of the full blob bytes. This dramatically reduces local disk and memory usage for tables with large BLOB columns (e.g., images, videos). The downstream consumer receives the serialized BlobDescriptor bytes and can resolve the actual blob content on demand.
##### lookup.cache-file-retention | 1 h | Duration | The cached files retention time for lookup. After the file expires, if there is a need for access, it will be re-read from the DFS to build an index on the local disk.
##### lookup.cache-max-disk-size | infinite | MemorySize | Max disk size for lookup cache, you can use this option to limit the use of local disks.
##### lookup.cache-max-memory-size | 256 mb | MemorySize | Max memory size for lookup cache.
##### lookup.cache-rows | 10000 | Long | The maximum number of rows to store in the cache.
##### lookup.cache-spill-compression | "zstd" | String | Spill compression for lookup cache, currently zstd, none, lz4 and lzo are supported.
##### lookup.cache.bloom.filter.enabled | true | Boolean | Whether to enable the bloom filter for lookup cache.
##### lookup.cache.bloom.filter.fpp | 0.05 | Double | Define the default false positive probability for lookup cache bloom filters.
##### lookup.cache.high-priority-pool-ratio | 0.25 | Double | The fraction of cache memory that is reserved for high-priority data like index, filter.
##### lookup.continuous.discovery-interval | (none) | Duration | The discovery interval of lookup continuous reading. This is used as an SQL hint. If it's not configured, the lookup function will fallback to 'continuous.discovery-interval'.
##### lookup.hash-load-factor | 0.75 | Float | The index load factor for lookup.
##### lookup.merge-buffer-size | 8 mb | MemorySize | Buffer memory size for one key merging in lookup.
##### lookup.merge-records-threshold | 1024 | Integer | Threshold for merging records to binary buffer in lookup.
##### lookup.remote-file.enabled | false | Boolean | Whether to enable the remote file for lookup.
##### lookup.remote-file.level-threshold | -2147483648 | Integer | Level threshold of lookup to generate remote lookup files. Level files below this threshold will not generate remote lookup files.
##### manifest-sort.enabled | false | Boolean | Whether to invoke manifest sort rewrite during commit. Note: enabling this changes the semantics of 'manifest.merge-min-count'. In the sort rewrite path, small manifest files within the rewrite budget are sorted and merged directly, so the minimum-count gate no longer prevents merging a small number of under-budget manifest files when full compaction is not triggered.
##### manifest-sort.max-rewrite-size | 256 mb | MemorySize | Maximum total size of manifest files to rewrite in a single sort rewrite pass. Sections exceeding this limit are skipped. Set to a larger value to allow more aggressive sort rewriting. The cap only limits the sorted rewrite portion and full/minor cleanup may still happen beyond it.
##### manifest-sort.partition-field | (none) | String | Partition field name to sort manifest entries by. Validated by schema validation, if not configured, defaults to the first partition field.
##### manifest.compression | "zstd" | String | Default file compression for manifest.
##### manifest.delete-file-drop-stats | false | Boolean | For DELETE manifest entry in manifest file, drop stats to reduce memory and storage. Default value is false only for compatibility of old reader.
##### manifest.format | "avro" | String | Specify the message format of manifest files.
##### manifest.full-compaction-threshold-size | 16 mb | MemorySize | The size threshold for triggering full compaction of manifest.
##### manifest.merge-min-count | 30 | Integer | To avoid frequent manifest merges, this parameter specifies the minimum number of ManifestFileMeta to merge. Note: when 'manifest-sort.enabled' is true, this minimum-count gate is only applied to the trailing sub-segment of a section that exceeds 'manifest-sort.max-rewrite-size'. Small under-budget sections are sorted and rewritten directly, so two small manifest files may be merged into one even when their count is below this threshold and full compaction is not triggered.
##### manifest.target-file-size | 8 mb | MemorySize | Suggested file size of a manifest file.
##### merge-engine | deduplicate | Enum | Specify the merge engine for table with primary key. Possible values: "deduplicate": De-duplicate and keep the last row. "partial-update": Partial update non-null fields. "aggregation": Aggregate fields with same primary key. "first-row": De-duplicate and keep the first row.
##### metadata.stats-dense-store | true | Boolean | Whether to store statistic densely in metadata (manifest files), which will significantly reduce the storage size of metadata when the none statistic mode is set. Note, when this mode is enabled with 'metadata.stats-mode:none', the Paimon sdk in reading engine requires at least version 0.9.1 or 1.0.0 or higher.
##### metadata.stats-keep-first-n-columns | -1 | Integer | Define how many columns' stats are kept in metadata file from front to end. Default value '-1' means ignoring this config.
##### metadata.stats-mode | "truncate(16)" | String | The mode of metadata stats collection. none, counts, truncate(16), full is available. "none": means disable the metadata stats collection. "counts" means only collect the null count. "full": means collect the null count, min/max value. "truncate(16)": means collect the null count, min/max value with truncated length of 16. Field level stats mode can be specified by fields.{field_name}.stats-mode
##### metadata.stats-mode.per.level |  | Map | Define different 'metadata.stats-mode' for different level, you can add the conf like this: 'metadata.stats-mode.per.level' = '0:none', if the metadata.stats-mode for level is not provided, the default mode which set by `metadata.stats-mode` will be used.
##### metastore.partitioned-table | false | Boolean | Whether to create this table as a partitioned table in metastore. For example, if you want to list all partitions of a Paimon table in Hive, you need to create this table as a partitioned table in Hive metastore. This config option does not affect the default filesystem metastore. For an internal format table in a REST catalog, it also makes the catalog own the table's partitions: a scan reads the partitions registered there and a write registers the ones it wrote, instead of listing the table directory.
##### metastore.tag-to-partition | (none) | String | Whether to create this table as a partitioned table for mapping non-partitioned table tags in metastore. This allows the Hive engine to view this table in a partitioned table view and use partitioning field to read specific partitions (specific tags).
##### metastore.tag-to-partition.preview | none | Enum | Whether to preview tag of generated snapshots in metastore. This allows the Hive engine to query specific tag before creation. Possible values: "none": No automatically created tags. "process-time": Based on the time of the machine, create TAG once the processing time passes period time plus delay. "watermark": Based on the watermark of the input, create TAG once the watermark passes period time plus delay. "batch": In the batch processing scenario, the tag corresponding to the current snapshot is generated after the task is completed.
##### num-levels | (none) | Integer | Total level number, for example, there are 3 levels, including 0,1,2 levels.
##### num-sorted-run.compaction-trigger | 5 | Integer | The sorted run number to trigger compaction. Includes level0 files (one file one sorted run) and high-level runs (one level one sorted run).
##### num-sorted-run.stop-trigger | (none) | Integer | The number of sorted runs that trigger the stopping of writes, the default value is 'num-sorted-run.compaction-trigger' + 3.
##### overwrite-upgrade | true | Boolean | Whether to try upgrading the data files after overwriting a primary key table.
##### page-size | 64 kb | MemorySize | Memory page size.
##### parquet.enable.dictionary | (none) | Integer | Turn off the dictionary encoding for all fields in parquet.
##### partial-update.remove-record-on-delete | false | Boolean | Whether to remove the whole row in partial-update engine when -D records are received.
##### partial-update.remove-record-on-sequence-group | (none) | String | When -D records of the given sequence groups are received, remove the whole row.
##### partition | (none) | String | Define partition by table options, cannot define partition on DDL and table options at the same time.
##### partition.default-name | "__DEFAULT_PARTITION__" | String | The default partition name in case the dynamic partition column value is null/empty string.
##### partition.end-input-to-done | false | Boolean | Whether mark the done status to indicate that the data is ready when end input.
##### partition.expiration-batch-size | (none) | Integer | The batch size of partition expiration. By default, all partitions to be expired will be expired together, which may cause a risk of out-of-memory. Use this parameter to divide partition expiration process and mitigate memory pressure.
##### partition.expiration-check-interval | 1 h | Duration | The check interval of partition expiration.
##### partition.expiration-max-num | 100 | Integer | The default deleted num of partition expiration.
##### partition.expiration-strategy | "values-time" | String | The strategy determines how to extract the partition time and compare it with the current time. "values-time": This strategy compares the time extracted from the partition value with the current time. "update-time": This strategy compares the last update time of the partition with the current time.
##### partition.expiration-time | (none) | Duration | The expiration interval of a partition. A partition will be expired if it's lifetime is over this value. Partition time is extracted from the partition value.
##### partition.idle-time-to-report-statistic | 0 ms | Duration | Set a time duration when a partition has no new data after this time duration, start to report the partition statistics to hms.
##### partition.legacy-name | true | Boolean | The legacy partition name is using `toString` for all types. If false, using cast to string for all types.
##### partition.mark-done-action | "success-file" | String | Action to mark a partition done is to notify the downstream application that the partition has finished writing, the partition is ready to be read. 1. 'success-file': add '_success' file to directory. 2. 'done-partition': add 'xxx.done' partition to metastore. 3. 'mark-event': mark partition event to metastore. 4. 'http-report': report partition mark done to remote http server. 5. 'custom': use policy class to create a mark-partition policy. Both can be configured at the same time: 'done-partition,success-file,mark-event,custom'.
##### partition.mark-done-action.custom.class | (none) | String | The partition mark done class for implement PartitionMarkDoneAction interface. Only work in custom mark-done-action.
##### partition.mark-done-action.http.params | (none) | String | Http client request parameters will be written to the request body, this can only be used by http-report partition mark done action.
##### partition.mark-done-action.http.url | (none) | String | Mark done action will reports the partition to the remote http server, this can only be used by http-report partition mark done action.
##### partition.sink-strategy | NONE | Enum | This is only for partitioned append table or postpone pk table, and the purpose is to reduce small files and improve write performance. Through this repartitioning strategy to reduce the number of partitions written by each task to as few as possible. none: Rebalanced or Forward partitioning, this is the default behavior, this strategy is suitable for the number of partitions you write in a batch is much smaller than write parallelism. hash: Hash the partitions value, this strategy is suitable for the number of partitions you write in a batch is greater equals than write parallelism. partition_dynamic: Dynamically adjusts shuffle strategy based on partition key traffic patterns. This mode monitors data distribution across partitions and rebalances load across downstream subtasks. Suitable when partition traffic is skewed and you want balanced write throughput. Possible values: "NONE" "HASH" "PARTITION_DYNAMIC"
##### partition.timestamp-format.strict | false | Boolean | When enabled, if a partition value does not match the 'partition.timestamp-formatter' or 'partition.timestamp-pattern' configuration, an error will be thrown during writing. This helps prevent dirty partition directories caused by incorrectly specified partition fields.
##### partition.timestamp-formatter | (none) | String | The formatter to format timestamp from string. It can be used with 'partition.timestamp-pattern' to create a formatter using the specified value. Default formatter is 'yyyy-MM-dd HH:mm:ss' and 'yyyy-MM-dd'. Supports multiple partition fields like '$year-$month-$day $hour:00:00'. The timestamp-formatter is compatible with Java's DateTimeFormatter.
##### partition.timestamp-pattern | (none) | String | You can specify a pattern to get a timestamp from partitions. The formatter pattern is defined by 'partition.timestamp-formatter'. By default, read from the first field. If the timestamp in the partition is a single field called 'dt', you can use '$dt'. If it is spread across multiple fields for year, month, day, and hour, you can use '$year-$month-$day $hour:00:00'. If the timestamp is in fields dt and hour, you can use '$dt $hour:00:00'.
##### pk-bitmap.index.columns | (none) | String | Comma-separated columns indexed by primary-key Bitmap indexes.
##### pk-btree.index.columns | (none) | String | Comma-separated columns indexed by primary-key BTree indexes.
##### pk-clustering-override | false | Boolean | Enables clustering by non-primary key fields. When set to true, the physical sort order of data files is determined by the configured 'clustering.columns' instead of the primary key, optimizing query performance for non-PK columns.
##### pk-full-text.index.columns | (none) | String | Comma-separated character columns indexed by primary-key full-text indexes. The first release supports exactly one column.
##### pk-vector.index.columns | (none) | String | Comma-separated VECTOR columns indexed by primary-key vector indexes. Each column owns one index and must define fields.<column>.pk-vector.index.type. Index options and distance metric are also field-scoped. The first release supports exactly one column.
##### postpone.batch-write-fixed-bucket | true | Boolean | Whether to write the data into fixed bucket for batch writing a postpone bucket table.
##### postpone.batch-write-fixed-bucket.max-parallelism | 2048 | Integer | The number of partitions for global index.
##### postpone.default-bucket-num | 1 | Integer | Bucket number for the partitions compacted for the first time in postpone bucket tables.
##### postpone.merge-on-read | false | Boolean | Whether to merge records in the postpone bucket with records in real buckets during batch reads. This requires an execution engine capable of routing and shuffling postpone records to their target real buckets.
##### postpone.target-row-num-per-bucket | (none) | Long | Target row number per bucket when batch writing fixed buckets or compacting postpone bucket files for a partition without real bucket data.
##### postpone.target-size-per-bucket | 1 gb | MemorySize | Target uncompressed serialized data size per bucket when Spark batch writes fixed buckets for a partition without real bucket data. This option is ignored when 'postpone.target-row-num-per-bucket' is configured.
##### primary-key | (none) | String | Define primary key by table options, cannot define primary key on DDL and table options at the same time.
##### query-auth.enabled | false | Boolean | Enable query auth to give Catalog the opportunity to perform column level and row level permission validation on queries.
##### read.batch-size | 1024 | Integer | Read batch size for any file format if it supports.
##### record-level.expire-time | (none) | Duration | Record level expire time for primary key table, expiration happens in compaction, there is no strong guarantee to expire records in time. You must specific 'record-level.time-field' too.
##### record-level.time-field | (none) | String | Time field for record level expire. It supports the following types: `timestamps in seconds with INT`,`timestamps in seconds with BIGINT`, `timestamps in milliseconds with BIGINT` or `timestamp`.
##### row-tracking.enabled | false | Boolean | Whether enable unique row id for append table.
##### row-tracking.partition-group-on-commit | true | Boolean | When row-tracking is enabled, whether to group new file metas by partition before commit, so that assigned row IDs are contiguous within each partition.This is useful if you want to build global indices on this table.
##### rowkind.field | (none) | String | The field that generates the row kind for primary key table, the row kind determines which data is '+I', '-U', '+U' or '-D'.
##### scalar-index.search-mode | fast | Enum | Search mode for scalar index queries. Possible values: "fast": Only search indexed data. "full": Use snapshot next row id and global index coverage to detect missing row ids, and scan raw data only when a gap exists. "detail": Scan data files to find exact unindexed rows. This can handle index invalidation caused by updates or rewrites.
##### scan.bounded.watermark | (none) | Long | End condition "watermark" for bounded streaming mode. Stream reading will end when a larger watermark snapshot is encountered.
##### scan.bucket | (none) | Integer | Specify a single bucket to scan. This option filters manifest entries and only plans splits for the given bucket. It is only supported for fixed-bucket primary key tables (bucket > 0). It cannot be used with postpone bucket tables.
##### scan.creation-time-millis | (none) | Long | Optional timestamp used in case of "from-creation-timestamp" scan mode.
##### scan.fallback-branch | (none) | String | When a batch job queries from a table, if a partition does not exist in the current branch, the reader will try to get this partition from this fallback branch.
##### scan.fallback-branch.read-fail-fast | false | Boolean | Whether to fail the read immediately when reading from a fallback branch throws. By default the failure is logged with the full stack trace and the reader falls through to the current branch, which can mask data issues. Set this to true to surface fallback branch errors to the caller instead.
##### scan.fallback-delta-branch | (none) | String | When a batch job queries from a chain table, if a partition does not exist in either main or snapshot branch, the reader will try to get this partition from chain snapshot and delta branch together.
##### scan.fallback-snapshot-branch | (none) | String | When a batch job queries from a chain table, if a partition does not exist in the main branch, the reader will try to get this partition from chain snapshot branch.
##### scan.file-creation-time-millis | (none) | Long | After configuring this time, only the data files created after this time will be read. It is independent of snapshots, but it is imprecise filtering (depending on whether or not compaction occurs).
##### scan.ignore-corrupt-files | false | Boolean | Ignore corrupt files while scanning.
##### scan.ignore-lost-files | false | Boolean | Ignore lost files while scanning.
##### scan.manifest.parallelism | (none) | Integer | The parallelism of scanning manifest files, default value is the size of cpu processor. Note: Scale-up this parameter will increase memory usage while scanning manifest files. We can consider downsize it when we encounter an out of memory exception while scanning
##### scan.max-splits-per-task | 10 | Integer | Max split size should be cached for one task while scanning. If splits size cached in enumerator are greater than tasks size multiply by this value, scanner will pause scanning.
##### scan.mode | default | Enum | Specify the scanning behavior of the source. Possible values: "default": Determines actual startup mode according to other table properties. If "scan.timestamp-millis" is set the actual startup mode will be "from-timestamp", and if "scan.snapshot-id" or "scan.tag-name" is set the actual startup mode will be "from-snapshot". Otherwise the actual startup mode will be "latest-full". "latest-full": For streaming sources, produces the latest snapshot on the table upon first startup, and continue to read the latest changes. For batch sources, just produce the latest snapshot but does not read new changes. "full": Deprecated. Same as "latest-full". "latest": For streaming sources, continuously reads latest changes without producing a snapshot at the beginning. For batch sources, behaves the same as the "latest-full" startup mode. "compacted-full": For streaming sources, produces a snapshot after the latest compaction on the table upon first startup, and continue to read the latest changes. For batch sources, just produce a snapshot after the latest compaction but does not read new changes. Snapshots of full compaction are picked when scheduled full-compaction is enabled. "from-timestamp": For streaming sources, continuously reads changes starting from timestamp specified by "scan.timestamp-millis", without producing a snapshot at the beginning. For batch sources, produces a snapshot at timestamp specified by "scan.timestamp-millis" but does not read new changes. "from-creation-timestamp": For streaming sources and batch sources, If timestamp specified by "scan.creation-time-millis" is during in the range of earliest snapshot and latest snapshot: mode is from-snapshot which snapshot is equal or later the timestamp. If timestamp is earlier than earliest snapshot or later than latest snapshot, mode is from-file-creation-time. "from-file-creation-time": For streaming and batch sources, consumes a snapshot and filters the data files by creation time. For streaming sources, upon first startup, and continue to read the latest changes. "from-snapshot": For streaming sources, continuously reads changes starting from snapshot specified by "scan.snapshot-id", without producing a snapshot at the beginning. For batch sources, produces a snapshot specified by "scan.snapshot-id" or "scan.tag-name" but does not read new changes. "from-snapshot-full": For streaming sources, produces from snapshot specified by "scan.snapshot-id" on the table upon first startup, and continuously reads changes. For batch sources, produces a snapshot specified by "scan.snapshot-id" but does not read new changes. "incremental": Read incremental changes between start and end snapshot or timestamp.
##### scan.plan-auto-tag-for-read.time-retained | (none) | Duration | When set, a temporary tag will be auto-created during batch scan planning to protect the read snapshot from expiration. The value specifies the tag's TTL. Should be longer than the longest expected batch read duration.
##### scan.plan-sort-partition | false | Boolean | Whether to sort plan files by partition fields, this allows you to read according to the partition order, even if your partition writes are out of order. It is recommended that you use this for streaming read of the 'append-only' table. By default, streaming read will read the full snapshot first. In order to avoid the disorder reading for partitions, you can open this option.
##### scan.primary-branch | (none) | String | When a batch job queries from a table, if a partition exists in the primary branch, the reader will read this partition from the primary branch. Otherwise, the reader will read this partition from the current branch.
##### scan.snapshot-id | (none) | Long | Optional snapshot id used in case of "from-snapshot" or "from-snapshot-full" scan mode
##### scan.tag-name | (none) | String | Optional tag name used in case of "from-snapshot" scan mode.
##### scan.timestamp | (none) | String | Optional timestamp used in case of "from-timestamp" scan mode, it will be automatically converted to timestamp in unix milliseconds, use local time zone
##### scan.timestamp-millis | (none) | Long | Optional timestamp used in case of "from-timestamp" scan mode. If there is no snapshot earlier than this time, the earliest snapshot will be chosen.
##### scan.watermark | (none) | Long | Optional watermark used in case of "from-snapshot" scan mode. If there is no snapshot later than this watermark, will throw an exceptions.
##### sequence.field | (none) | String | The field that generates the sequence number for primary key table, the sequence number determines which data is the most recent.
##### sequence.field.sort-order | ascending | Enum | Specify the order of sequence.field. Possible values: "ascending": specifies sequence.field sort order is ascending. "descending": specifies sequence.field sort order is descending.
##### sequence.snapshot-ordering | false | Boolean | When enabled, merge uses the commit snapshot id as the ordering key for primary-key conflicts: records from later snapshots always win. Designed for multi-writer scenarios on the same primary-key table where wall-clock sequence numbers cannot be globally ordered. The order of records within the same snapshot is not guaranteed. Mutually exclusive with sequence.field. Requires a primary-key table with write-only=true. Inline compaction is not allowed because snapshot ids are assigned only after commit. To compact such tables, run a dedicated compaction job/action with write-only=false.
##### sink.process-time-zone | (none) | String | The time zone to parse the long process time to TIMESTAMP value. The default value is JVM's default time zone. If you want to specify a time zone, you should either set a full name such as 'America/Los_Angeles' or a custom zone id such as 'GMT-08:00'. This option currently is used for extract tag name.
##### sink.watermark-time-zone | "UTC" | String | The time zone to parse the long watermark value to TIMESTAMP value. The default value is 'UTC', which means the watermark is defined on TIMESTAMP column or not defined. If the watermark is defined on TIMESTAMP_LTZ column, the time zone of watermark is user configured time zone, the value should be the user configured local time zone. The option value is either a full name such as 'America/Los_Angeles', or a custom timezone id such as 'GMT-08:00'.
##### snapshot.clean-empty-directories | false | Boolean | Whether to try to clean empty directories when expiring snapshots, if enabled, please note: hdfs: may print exceptions in NameNode. oss/s3: may cause performance issue.
##### snapshot.expire.execution-mode | sync | Enum | Specifies the execution mode of expire. Possible values: "sync": Execute expire synchronously. If there are too many files, it may take a long time and block stream processing. "async": Execute expire asynchronously. If the generation of snapshots is greater than the deletion, there will be a backlog of files.
##### snapshot.expire.limit | 50 | Integer | The maximum number of snapshots allowed to expire at a time.
##### snapshot.ignore-empty-commit | (none) | Boolean | Whether ignore empty commit.
##### snapshot.num-retained.max | infinite | Integer | The maximum number of completed snapshots to retain. Should be greater than or equal to the minimum number.
##### snapshot.num-retained.min | 10 | Integer | The minimum number of completed snapshots to retain. Should be greater than or equal to 1.
##### snapshot.time-retained | 1 h | Duration | The maximum time of completed snapshots to retain.
##### snapshot.watermark-idle-timeout | (none) | Duration | In watermarking, if a source remains idle beyond the specified timeout duration, it triggers snapshot advancement and facilitates tag creation.
##### sort-compaction.local-sample.magnification | 1000 | Integer | The magnification of local sample for sort-compaction.The size of local sample is sink parallelism * magnification.
##### sort-compaction.range-strategy | SIZE | Enum | The range strategy of sort compaction, the default value is quantity. If the data size allocated for the sorting task is uneven,which may lead to performance bottlenecks, the config can be set to size. Possible values: "SIZE" "QUANTITY"
##### sort-engine | loser-tree | Enum | Specify the sort engine for table with primary key. Possible values: "min-heap": Use min-heap for multiway sorting. "loser-tree": Use loser-tree for multiway sorting. Compared with heapsort, loser-tree has fewer comparisons and is more efficient.
##### sort-spill-buffer-size | 64 mb | MemorySize | Amount of data to spill records to disk in spilled sort.
##### sort-spill-threshold | (none) | Integer | If the maximum number of sort readers exceeds this value, a spill will be attempted. This prevents too many readers from consuming too much memory and causing OOM.
##### source.split.open-file-cost | 4 mb | MemorySize | Open file cost of a source file. It is used to avoid reading too many files with a source split, which can be very slow.
##### source.split.target-size | 128 mb | MemorySize | Target size of a source split when scanning a bucket.
##### spill-compression | "zstd" | String | Compression for spill, currently zstd, lzo and zstd are supported.
##### spill-compression.zstd-level | 1 | Integer | Default spill compression zstd level. For higher compression rates, it can be configured to 9, but the read and write speed will significantly decrease.
##### streaming-read-append-overwrite | false | Boolean | Whether to read the delta from append table's overwrite commit in streaming mode.
##### streaming-read-overwrite | false | Boolean | Whether to read the changes from overwrite in streaming mode. Cannot be set to true when changelog producer is full-compaction or lookup because it will read duplicated changes.
##### streaming.read.snapshot.delay | (none) | Duration | The delay duration of stream read when scan incremental snapshots.
##### table-read.sequence-number.enabled | false | Boolean | Whether to include the _SEQUENCE_NUMBER field when reading the audit_log or binlog system tables. This is only valid for primary key tables.
##### tag.automatic-completion | false | Boolean | Whether to automatically complete missing tags.
##### tag.automatic-creation | none | Enum | Whether to create tag automatically. And how to generate tags. Possible values: "none": No automatically created tags. "process-time": Based on the time of the machine, create TAG once the processing time passes period time plus delay. "watermark": Based on the watermark of the input, create TAG once the watermark passes period time plus delay. "batch": In the batch processing scenario, the tag corresponding to the current snapshot is generated after the task is completed.
##### tag.batch.customized-name | (none) | String | Use customized name when creating tags in Batch mode.
##### tag.callback.#.param | (none) | String | Parameter string for the constructor of class #. Callback class should parse the parameter by itself.
##### tag.callbacks | (none) | String | A list of commit callback classes to be called after a successful tag. Class names are connected with comma (example: com.test.CallbackA,com.sample.CallbackB).
##### tag.create-success-file | false | Boolean | Whether to create tag success file for new created tags.
##### tag.creation-delay | 0 ms | Duration | How long is the delay after the period ends before creating a tag. This can allow some late data to enter the Tag.
##### tag.creation-period | daily | Enum | What frequency is used to generate tags. Possible values: "daily": Generate a tag every day. "hourly": Generate a tag every hour. "two-hours": Generate a tag every two hours.
##### tag.creation-period-duration | (none) | Duration | The period duration for tag auto create periods.If user set it, tag.creation-period would be invalid.
##### tag.default-time-retained | (none) | Duration | The default maximum time retained for newly created tags. It affects both auto-created tags and manually created (by procedure) tags.
##### tag.num-retained-max | (none) | Integer | The maximum number of tags to retain. It only affects auto-created tags.
##### tag.period-formatter | with_dashes | Enum | The date format for tag periods. Possible values: "with_dashes": Dates and hours with dashes, e.g., 'yyyy-MM-dd HH' "without_dashes": Dates and hours without dashes, e.g., 'yyyyMMdd HH' "without_dashes_and_spaces": Dates and hours without dashes and spaces, e.g., 'yyyyMMddHH'
##### tag.time-expire-enabled | true | Boolean | Whether to enable tag expiration by retained time.
##### target-file-row-num | 9223372036854775807 | Long | Target number of rows per newly written data file; a file rolls when this or target-file-size is reached, whichever comes first. Enforced at bundle granularity, so a bundled write may exceed it by up to one bundle. Only constrains files at write time: compaction is size-based and may merge into larger files, and data-evolution compaction still produces a single file. Bounds per-file rows for wide columns to avoid data-evolution OOM. PyPaimon supports this for data-evolution append tables; its primary-key, blob and vector writers still fail fast when it is enabled. Disabled by default.
##### target-file-size | (none) | MemorySize | Target size of a file. primary key table: the default value is 128 MB. append table: the default value is 256 MB.
##### type | table | Enum | Type of the table. Possible values: "table": Normal Paimon table. "format-table": A file format table refers to a directory that contains multiple files of the same format. "materialized-table": A materialized table combines normal Paimon table and materialized SQL. "object-table": An object table combines normal Paimon table and object location. "lance-table": A lance table, see 'https://lancedb.github.io/lance/'. "iceberg-table": An iceberg table, see 'https://iceberg.apache.org/'.
##### upsert-key | (none) | String | Define upsert key to do MERGE INTO when executing INSERT INTO, cannot be defined with primary key.
##### variant.inferShreddingSchema | false | Boolean | Whether to automatically infer the shredding schema when writing Variant columns.
##### variant.shredding.adaptive.maxInferBufferRow | 256 | Integer | Maximum number of prefix rows sampled after the first file in an adaptive Variant shredding inference session.
##### variant.shredding.adaptive.retentionRatio | 0.05 | Double | Minimum combined presence ratio for retaining a Variant path selected in the previous file. This must not exceed 'variant.shredding.minFieldCardinalityRatio'.
##### variant.shredding.inferenceMode | per-file | Enum | The Variant shredding inference mode. PER_FILE infers each file independently. ADAPTIVE reuses bounded evidence within one rolling writer and samples a smaller prefix after the first file. Possible values: "per-file": Infer every file independently from its own prefix rows. "adaptive": Reuse bounded inference evidence within one rolling writer and correct it with a smaller prefix from each subsequent file.
##### variant.shredding.maxInferBufferRow | 4096 | Integer | Maximum number of rows to buffer for schema inference.
##### variant.shredding.maxSchemaDepth | 50 | Integer | Maximum traversal depth in Variant values during schema inference.
##### variant.shredding.maxSchemaWidth | 300 | Integer | Maximum number of shredded fields allowed in an inferred schema.
##### variant.shredding.minFieldCardinalityRatio | 0.1 | Double | Minimum fraction of rows that must contain a field for it to be shredded. Fields below this threshold will remain in the un-shredded Variant binary.
##### variant.shreddingSchema | (none) | String | The Variant shredding schema for writing.
##### vector-field | (none) | String | Specifies column names that should be stored as vector type. This is used when you want to treat a ARRAY column as a VECTOR.
##### vector-index.search-mode | fast | Enum | Search mode for vector index queries. Possible values: "fast": Only search indexed data. "full": Use snapshot next row id and global index coverage to detect missing row ids, and scan raw data only when a gap exists. "detail": Scan data files to find exact unindexed rows. This can handle index invalidation caused by updates or rewrites.
##### vector-search.distribute.enabled | false | Boolean | Whether to process distributed vector search.
##### vector-search.lateral-join.batch-size | 256 | Integer | The batch size for lateral vector search. Each batch executes vector topK search and table lookup for multiple query vectors.
##### vector.file.format | (none) | String | Specify the vector store file format.
##### vector.target-file-size | (none) | MemorySize | Target size of a vector-store file. Default is the same as TARGET_FILE_SIZE.
##### visibility-callback.check-interval | 10 s | Duration | The interval for checking visibility when visibility-callback enabled.
##### visibility-callback.enabled | false | Boolean | Whether to enable the visibility wait callback that waits for compaction or global index build to complete after commit. This is useful for primary key tables with deletion vectors or postpone bucket mode and row-tracking tables with global indexes to ensure data visibility, only used for batch mode or bounded stream.
##### visibility-callback.timeout | 30 min | Duration | The maximum time to wait for compaction or global index build to complete when visibility callback is enabled. If the timeout is reached, an exception will be thrown.
##### write-buffer-for-append | false | Boolean | This option only works for append-only table. Whether the write use write buffer to avoid out-of-memory error.
##### write-buffer-size | 256 mb | MemorySize | Amount of data to build up in memory before converting to a sorted on-disk file.
##### write-buffer-spill.max-disk-size | infinite | MemorySize | The max disk to use for write buffer spill. This only work when the write buffer spill is enabled
##### write-buffer-spillable | true | Boolean | Whether the write buffer can be spillable.
##### write-max-writers-to-spill | 10 | Integer | When in batch append inserting, if the writer number is greater than this option, we open the buffer cache and spill function to avoid out-of-memory.
##### write-only | false | Boolean | If set to true, compactions and snapshot expiration will be skipped. This option is used along with dedicated compact jobs.
##### write.batch-memory | 128 mb | MemorySize | Write batch memory for any file format if it supports.
##### write.batch-size | 1024 | Integer | Write batch size for any file format if it supports.
##### write.sequence-number-init-mode | scan | Enum | Specify how to initialize the next sequence number for primary key table writers. Possible values: "scan": initialize by scanning existing file metadata. "snapshot": initialize from the maximum sequence number recorded in snapshot properties, which can avoid scanning existing file metadata in write-only mode.
##### zorder.var-length-contribution | 8 | Integer | The bytes of types (CHAR, VARCHAR, BINARY, VARBINARY) devote to the zorder sort.