Configuration Reference

This page documents all configuration properties specific to the Xonai Accelerator.

The properties can be activated just as any other Spark configuration property (see official documentation), following the --conf [key]=[value] format.

For example, adding --conf spark.xonai.sql.project.enabled=false to spark-submit:

$SPARK_HOME/bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --conf spark.xonai.sql.project.enabled=false \
  $SPARK_HOME/examples/jars/spark-examples.jar \
  1000

By convention, all Xonai Accelerator properties start with spark.xonai.

Configuration Property List

The following is a list of all configuration properties available for the Xonai Accelerator.

Property Key

Description

Default Value

spark.xonai.memory.dynamic.fallback.enabled

Whether the plugin should fallback to default Spark execution when the Xonai dynamic memory size is low.

true

spark.xonai.memory.dynamic.fallback.threshold

The value of Xonai dynamic memory size a executor must go below of to trigger execution fallback. By default, it is defined by numCores x 768M. Value in MiB unless otherwise specified.

spark.xonai.memory.mode

How the Xonai engine manages execution memory. Must be one of:

  • off-heap - Xonai uses the Spark off-heap management to acquire memory. Default if spark.memory.offHeap.size and spark.memory.offHeap.enabled are specified.

  • overhead - the Spark executor memory (JVM) must be reduced and the memory overhead increased for Xonai to use. Default if spark.executor.memoryOverhead is specified.

  • dynamic - Xonai uses memory not allocated by the JVM. Default if no other mode is selected. As the JVM grows in size the Xonai memory size decreases.

spark.xonai.memory.size

Amount of executor memory the Xonai engine can use. Default value is derived from spark.memory.offHeap.size in off-heap memory mode and from spark.executor.memoryOverhead in overhead and dynamic modes. In dynamic mode the value defines a lower-bound.

spark.xonai.memory.storageFraction

Amount of Xonai memory that can be used for storage and is immune to eviction, expressed as a fraction of spark.xonai.memory.size. The higher this value is the less memory may be available to execution that can lead to spilling to disk more often.

0.5

spark.xonai.report.enabled

Whether the Xonai plugin should write a report file with execution information.

false

spark.xonai.report.path

Directory where to write Xonai execution reports.

/tmp/xonai

spark.xonai.sql.broadcastHashJoin.enabled

Whether Xonai BroadcastHashJoin can be used.

true

spark.xonai.sql.coalesce.enabled

Whether Xonai Coalesce can be used.

true

spark.xonai.sql.enabled

Whether the Xonai Accelerator is enabled

true

spark.xonai.sql.expand.enabled

Whether Xonai Expand can be used.

true

spark.xonai.sql.filter.enabled

Whether Xonai Filter can be used.

true

spark.xonai.sql.generate.enabled

Whether Xonai Generate can be used.

true

spark.xonai.sql.hashAggregate.enabled

Whether Xonai HashAggregate can be used.

true

spark.xonai.sql.offload.mode

Strategy to offload operators to Xonai engine. Must be one of:

  • stage - entire stages are offloaded if supported

  • partial - contiguous supported operators at the start and end of stages are offloaded

  • all - all supported operators will be offloaded

stage

spark.xonai.sql.parquet.reader.enabled

Whether the Xonai Parquet reader should be used.

true

spark.xonai.sql.parquet.writer.enabled

Whether the Xonai Parquet writer should be used.

true

spark.xonai.sql.project.enabled

Whether Xonai Project can be used.

true

spark.xonai.sql.rowToColumnar.batchSize

Size of columnar batches when converting from row to columnar batches.

4096

spark.xonai.sql.shuffle.enabled

Whether data shuffle should use columnar format.

true

spark.xonai.sql.shuffledHashJoin.enabled

Whether Xonai ShuffledHashJoin can be used.

true

spark.xonai.sql.sort.enabled

Whether Xonai Sort can be used.

true

spark.xonai.sql.sortMergeJoin.enabled

Whether Xonai SortMergeJoin can be used.

true

spark.xonai.sql.takeOrderedAndProject.enabled

Whether Xonai TakeOrderedAndProject can be used.

true

spark.xonai.sql.union.enabled

Whether Xonai Union can be used.

true

Configuration Defaults

The Xonai Accelerator may modify the defaults of a few Spark properties as it is expected to perform better with these. Properties explicitly set take precedence over these defaults.

The modified Spark properties are as follows:

Property

Xonai Value

spark.sql.adaptive.advisoryPartitionSizeInBytes

32MB

spark.sql.cache.serializer

com.xonai.spark.CacheSerializer
Last update: Apr 02, 2025