---
title: "Quick Start"
sidebar_position: 1
---

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import Stable from '@site/src/components/Stable';
import Unstable from '@site/src/components/Unstable';

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

# Quick Start

## Preparation

Paimon supports the following Spark versions with their respective Java and Scala compatibility. We recommend using the latest Spark version for a better experience.

- Spark 4.x (including 4.1, 4.0) : Pre-built with Java 17 and Scala 2.13

- Spark 3.x (including 3.5, 3.4, 3.3, 3.2) : Pre-built with Java 8 and Scala 2.12/2.13

Download the jar file with corresponding version.

<Stable>



| Version   | Jar (Scala 2.12)                                                                                                                                                                    | Jar (Scala 2.13)                                                                                                                                                                    |
|-----------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Spark 4.1 | -                                                                                                                                                                                   | [paimon-spark-4.1_2.13-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-4.1_2.13/@@VERSION@@/paimon-spark-4.1_2.13-@@VERSION@@.jar) |
| Spark 4.0 | -                                                                                                                                                                                   | [paimon-spark-4.0_2.13-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-4.0_2.13/@@VERSION@@/paimon-spark-4.0_2.13-@@VERSION@@.jar) |
| Spark 3.5 | [paimon-spark-3.5_2.12-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-3.5_2.12/@@VERSION@@/paimon-spark-3.5_2.12-@@VERSION@@.jar) | [paimon-spark-3.5_2.13-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-3.5_2.13/@@VERSION@@/paimon-spark-3.5_2.13-@@VERSION@@.jar) |
| Spark 3.4 | [paimon-spark-3.4_2.12-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-3.4_2.12/@@VERSION@@/paimon-spark-3.4_2.12-@@VERSION@@.jar) | [paimon-spark-3.4_2.13-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-3.4_2.13/@@VERSION@@/paimon-spark-3.4_2.13-@@VERSION@@.jar) |
| Spark 3.3 | [paimon-spark-3.3_2.12-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-3.3_2.12/@@VERSION@@/paimon-spark-3.3_2.12-@@VERSION@@.jar) | [paimon-spark-3.3_2.13-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-3.3_2.13/@@VERSION@@/paimon-spark-3.3_2.13-@@VERSION@@.jar) |
| Spark 3.2 | [paimon-spark-3.2_2.12-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-3.2_2.12/@@VERSION@@/paimon-spark-3.2_2.12-@@VERSION@@.jar) | [paimon-spark-3.2_2.13-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-3.2_2.13/@@VERSION@@/paimon-spark-3.2_2.13-@@VERSION@@.jar) |



</Stable>

<Unstable>



| Version   | Jar (Scala 2.12)                                                                                                                              | Jar (Scala 2.13)                                                                                                                              |
|-----------|-----------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------|
| Spark 4.1 | -                                                                                                                                             | [paimon-spark-4.1_2.13-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-4.1_2.13/@@VERSION@@/) |
| Spark 4.0 | -                                                                                                                                             | [paimon-spark-4.0_2.13-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-4.0_2.13/@@VERSION@@/) |
| Spark 3.5 | [paimon-spark-3.5_2.12-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-3.5_2.12/@@VERSION@@/) | [paimon-spark-3.5_2.13-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-3.5_2.13/@@VERSION@@/) |
| Spark 3.4 | [paimon-spark-3.4_2.12-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-3.4_2.12/@@VERSION@@/) | [paimon-spark-3.4_2.13-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-3.4_2.13/@@VERSION@@/) |
| Spark 3.3 | [paimon-spark-3.3_2.12-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-3.3_2.12/@@VERSION@@/) | [paimon-spark-3.3_2.13-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-3.3_2.13/@@VERSION@@/) |
| Spark 3.2 | [paimon-spark-3.2_2.12-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-3.2_2.12/@@VERSION@@/) | [paimon-spark-3.2_2.13-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-3.2_2.13/@@VERSION@@/) |



</Unstable>

You can also manually build bundled jar from the source code.

To build from source code, [clone the git repository](@@GITHUB_REPO@@), then build the bundled jar with the following command.

```bash
# build paimon spark 3.5 with scala 2.12
mvn clean package -DskipTests -pl paimon-spark/paimon-spark-3.5 -am

# build paimon spark 3.5 with scala 2.13
mvn clean package -DskipTests -pl paimon-spark/paimon-spark-3.5 -am -Pscala-2.13

# build paimon spark 4.0
mvn clean package -DskipTests -pl paimon-spark/paimon-spark-4.0 -am -Pspark4

# build paimon spark 4.1
mvn clean package -DskipTests -pl paimon-spark/paimon-spark-4.1 -am -Pspark4
```

For Spark 3.5, you can find the bundled jar in `./paimon-spark/paimon-spark-3.5/target/paimon-spark-3.5_2.12-@@VERSION@@.jar`.

## Setup

:::info

If you are using HDFS, make sure that the environment variable `HADOOP_HOME` or `HADOOP_CONF_DIR` is set.

:::

**Step 1: Specify Paimon Jar File**

Append path to paimon jar file to the `--jars` argument when starting `spark-sql`.

```bash
spark-sql ... --jars /path/to/paimon-spark-3.5_2.12-@@VERSION@@.jar
```

OR use the `--packages` option.

```bash
spark-sql ... --packages org.apache.paimon:paimon-spark-3.5_2.12:@@VERSION@@
```

Alternatively, you can copy `paimon-spark-3.5_2.12-@@VERSION@@.jar` under `spark/jars` in your Spark installation directory.

**Step 2: Specify Paimon Catalog**

<Tabs groupId="specify-paimon-catalog">

<TabItem value="catalog" label="Catalog">

When starting `spark-sql`, use the following command to register Paimon's Spark catalog with the name `paimon`. Table files of the warehouse is stored under `/tmp/paimon`.

```bash
spark-sql ... \
    --conf spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog \
    --conf spark.sql.catalog.paimon.warehouse=file:/tmp/paimon \
    --conf spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions
```

Catalogs are configured using properties under spark.sql.catalog.(catalog_name). In above case, 'paimon' is the
catalog name, you can change it to your own favorite catalog name.

After `spark-sql` command line has started, run the following SQL to create and switch to database `default`.

```sql
USE paimon;
USE default;
```

After switching to the catalog (`'USE paimon'`), Spark's existing tables will not be directly accessible, you
can use the `spark_catalog.${database_name}.${table_name}` to access Spark tables.

</TabItem>

<TabItem value="generic-catalog" label="Generic Catalog">

When starting `spark-sql`, use the following command to register Paimon's Spark Generic catalog to replace Spark
default catalog `spark_catalog`. (default warehouse is Spark `spark.sql.warehouse.dir`)

Currently, it is only recommended to use `SparkGenericCatalog` in the case of Hive metastore, Paimon will infer
Hive conf from Spark session, you just need to configure Spark's Hive conf.

```bash
spark-sql ... \
    --conf spark.sql.catalog.spark_catalog=org.apache.paimon.spark.SparkGenericCatalog \
    --conf spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions
```

Using `SparkGenericCatalog`, you can use Paimon tables in this Catalog or non-Paimon tables such as Spark's csv,
parquet, Hive tables, etc.

</TabItem>

</Tabs>

## Create Table

<Tabs groupId="create-paimon-table">

<TabItem value="catalog" label="Catalog">

```sql
create table my_table (
    k int,
    v string
) tblproperties (
    'primary-key' = 'k'
);
```

</TabItem>

<TabItem value="generic-catalog" label="Generic Catalog">

```sql
create table my_table (
    k int,
    v string
) USING paimon
tblproperties (
    'primary-key' = 'k'
);

```

</TabItem>

</Tabs>

## Insert Table

<Tabs groupId="insert-paimon-table">

<TabItem value="sql" label="SQL">

```sql
INSERT INTO my_table VALUES (1, 'Hi'), (2, 'Hello');
```

</TabItem>

<TabItem value="dataframe" label="DataFrame">

```scala
-- you can use
Seq((1, "Hi"), (2, "Hello")).toDF("k", "v")
  .write.format("paimon").mode("append").saveAsTable("my_table")

-- or
Seq((1, "Hi"), (2, "Hello")).toDF("k", "v")
  .write.format("paimon").mode("append").save("file:/tmp/paimon/default.db/my_table")
```

</TabItem>

</Tabs>

## Query Table

<Tabs groupId="query-paimon-table">

<TabItem value="sql" label="SQL">

```sql
SELECT * FROM my_table;

/*
1	Hi
2	Hello
*/
```

</TabItem>

<TabItem value="dataframe" label="DataFrame">

```scala
-- you can use
spark.read.format("paimon").table("my_table").show()

-- or
spark.read.format("paimon").load("file:/tmp/paimon/default.db/my_table").show()

/*
+---+------+
| k |     v|
+---+------+
|  1|    Hi|
|  2| Hello|
+---+------+
*/
```

</TabItem>

</Tabs>

## Spark Type Conversion

This section lists all supported type conversion between Spark and Paimon.
All Spark's data types are available in package `org.apache.spark.sql.types`.

<table className="table table-bordered">
    <thead>
    <tr>
      <th className="text-left" style={{width: "10%"}}>Spark Data Type</th>
      <th className="text-left" style={{width: "10%"}}>Paimon Data Type</th>
      <th className="text-left" style={{width: "5%"}}>Atomic Type</th>
    </tr>
    </thead>
    <tbody>
    <tr>
      <td><code>StructType</code></td>
      <td><code>RowType</code></td>
      <td>false</td>
    </tr>
    <tr>
      <td><code>MapType</code></td>
      <td><code>MapType</code></td>
      <td>false</td>
    </tr>
    <tr>
      <td><code>ArrayType</code></td>
      <td><code>ArrayType</code></td>
      <td>false</td>
    </tr>
    <tr>
      <td><code>BooleanType</code></td>
      <td><code>BooleanType</code></td>
      <td>true</td>
    </tr>
    <tr>
      <td><code>ByteType</code></td>
      <td><code>TinyIntType</code></td>
      <td>true</td>
    </tr>
    <tr>
      <td><code>ShortType</code></td>
      <td><code>SmallIntType</code></td>
      <td>true</td>
    </tr>
    <tr>
      <td><code>IntegerType</code></td>
      <td><code>IntType</code></td>
      <td>true</td>
    </tr>
    <tr>
      <td><code>LongType</code></td>
      <td><code>BigIntType</code></td>
      <td>true</td>
    </tr>
    <tr>
      <td><code>FloatType</code></td>
      <td><code>FloatType</code></td>
      <td>true</td>
    </tr>
    <tr>
      <td><code>DoubleType</code></td>
      <td><code>DoubleType</code></td>
      <td>true</td>
    </tr>
    <tr>
      <td><code>StringType</code></td>
      <td><code>VarCharType(Integer.MAX_VALUE)</code></td>
      <td>true</td>
    </tr>
    <tr>
      <td><code>VarCharType(length)</code></td>
      <td><code>VarCharType(length)</code></td>
      <td>true</td>
    </tr>
    <tr>
      <td><code>CharType(length)</code></td>
      <td><code>CharType(length)</code></td>
      <td>true</td>
    </tr>
    <tr>
      <td><code>DateType</code></td>
      <td><code>DateType</code></td>
      <td>true</td>
    </tr>
    <tr>
      <td><code>TimestampType</code></td>
      <td><code>LocalZonedTimestamp</code></td>
      <td>true</td>
    </tr>
    <tr>
      <td><code>TimestampNTZType(Spark3.4+)</code></td>
      <td><code>TimestampType</code></td>
      <td>true</td>
    </tr>
    <tr>
      <td><code>DecimalType(precision, scale)</code></td>
      <td><code>DecimalType(precision, scale)</code></td>
      <td>true</td>
    </tr>
    <tr>
      <td><code>BinaryType</code></td>
      <td><code>VarBinaryType</code>, <code>BinaryType</code></td>
      <td>true</td>
    </tr>
    <tr>
      <td><code>VariantType(Spark4.0+)</code></td>
      <td><code>VariantType</code></td>
      <td>true</td>
    </tr>
    </tbody>
</table>

:::warning

Due to the previous design, in Spark3.3 and below, Paimon will map both Paimon's TimestampType and LocalZonedTimestamp to Spark's TimestampType, and only correctly handle with TimestampType.

Therefore, when using Spark3.3 and below, reads Paimon table with LocalZonedTimestamp type written by other engines, such as Flink, the query result of LocalZonedTimestamp type will have time zone offset, which needs to be adjusted manually.

When using Spark3.4 and above, all timestamp types can be parsed correctly.

:::