Open-Source Spark

Prerequisites

The Xonai Accelerator is compatible with the Apache Spark open-source versions listed in the following table:

Spark Version

3.1.1

3.1.2

3.1.3

3.2.0

3.2.1

3.2.2

3.2.3

3.2.4

3.3.0

3.3.1

3.3.2

3.3.3

3.3.4

3.4.0

3.4.1

3.4.2

3.4.3

3.4.4

3.5.0

3.5.1

3.5.2

3.5.3

3.5.4

Coming soon

3.5.5

Coming soon

Note

Xonai does not yet support builds for Scala 2.13

Activation via spark-submit

The Xonai Accelerator can be activated on a per-application basis via Spark 3 configuration by adding the following properties:

--jars <scheme>://<path>/xonai-spark-plugin-<package>-<runtime>-<release>-<channel>-<platform>-<arch>.jar
--conf spark.plugins=com.xonai.spark.SQLPlugin

As an example, running Xonai with SparkPI program would be:

spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --jars <scheme>://<path>/xonai-spark-plugin-<package>-<runtime>-<release>-<channel>-<platform>-<arch>.jar \
  --conf spark.plugins=com.xonai.spark.SQLPlugin \
  $SPARK_HOME/examples/jars/spark-examples.jar 1000

Info

If you are defining explicit offheap or overhead memory, you may need to reconfigure property values depending on which Xonai memory mode is active.

Last update: May 21, 2025