Configuration Guide¶

This page documents all configuration properties specific to the Xonai Accelerator.

The properties can be activated just as any other Spark configuration property (see official documentation), following the --conf [key]=[value] format.

For example, adding --conf spark.xonai.sql.project.enabled=false to spark-submit:

$SPARK_HOME/bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --conf spark.xonai.sql.project.enabled=false \
  $SPARK_HOME/examples/jars/spark-examples.jar \
  1000

By convention, all Xonai Accelerator properties start with spark.xonai.

Configuration Property List¶

The following is a list of all configuration properties available for the Xonai Accelerator.

Property Key	Description	Default Value
spark.xonai.memory.dynamic.fallback.enabled	Whether the plugin should fallback to default Spark execution when the Xonai dynamic memory size is low.	true
spark.xonai.memory.dynamic.fallback.threshold	The value of Xonai dynamic memory size a executor must go below of to trigger execution fallback. By default, it is defined by `numCores` x `768M`. Value in MiB unless otherwise specified.
spark.xonai.memory.mode	How the Xonai engine manages execution memory. Must be one of: `off-heap` - Xonai uses the Spark off-heap management to acquire memory. Default if `spark.memory.offHeap.size` and `spark.memory.offHeap.enabled` are specified. `overhead` - the Spark executor memory (JVM) must be reduced and the memory overhead increased for Xonai to use. Default if `spark.executor.memoryOverhead` is specified. `dynamic` - Xonai uses memory not allocated by the JVM. Default if no other mode is selected. As the JVM grows in size the Xonai memory size decreases.
spark.xonai.memory.size	Amount of executor memory the Xonai engine can use. Default value is derived from `spark.memory.offHeap.size` in `off-heap` memory mode and from `spark.executor.memoryOverhead` in `overhead` and `dynamic` modes. In `dynamic` mode the value defines a lower-bound.
spark.xonai.memory.storageFraction	Amount of Xonai memory that can be used for storage and is immune to eviction, expressed as a fraction of `spark.xonai.memory.size`. The higher this value is the less memory may be available to execution that can lead to spilling to disk more often.	0.5
spark.xonai.report.enabled	Whether the Xonai plugin should write a report file with execution information.	false
spark.xonai.report.path	Directory where to write Xonai execution reports.	/tmp/xonai
spark.xonai.sql.broadcastHashJoin.enabled	Whether Xonai BroadcastHashJoin can be used.	true
spark.xonai.sql.coalesce.enabled	Whether Xonai Coalesce can be used.	true
spark.xonai.sql.enabled	Whether the Xonai Accelerator is enabled	true
spark.xonai.sql.expand.enabled	Whether Xonai Expand can be used.	true
spark.xonai.sql.filter.enabled	Whether Xonai Filter can be used.	true
spark.xonai.sql.generate.enabled	Whether Xonai Generate can be used.	true
spark.xonai.sql.hashAggregate.enabled	Whether Xonai HashAggregate can be used.	true
spark.xonai.sql.offload.mode	Strategy to offload operators to Xonai engine. Must be one of: `stage` - entire stages are offloaded if supported `partial` - contiguous supported operators at the start and end of stages are offloaded `all` - all supported operators are offloaded	stage
spark.xonai.sql.parquet.reader.enabled	Whether the Xonai Parquet reader should be used.	true
spark.xonai.sql.parquet.writer.enabled	Whether the Xonai Parquet writer should be used.	true
spark.xonai.sql.project.enabled	Whether Xonai Project can be used.	true
spark.xonai.sql.rowToColumnar.batchSize	Size of columnar batches when converting from row to columnar batches.	4096
spark.xonai.sql.shuffle.enabled	Whether data shuffle should use columnar format.	true
spark.xonai.sql.shuffledHashJoin.enabled	Whether Xonai ShuffledHashJoin can be used.	true
spark.xonai.sql.sort.enabled	Whether Xonai Sort can be used.	true
spark.xonai.sql.sortMergeJoin.enabled	Whether Xonai SortMergeJoin can be used.	true
spark.xonai.sql.takeOrderedAndProject.enabled	Whether Xonai TakeOrderedAndProject can be used.	true
spark.xonai.sql.union.enabled	Whether Xonai Union can be used.	true

Configuration Defaults¶

The Xonai Accelerator may modify the defaults of a few Spark properties as it is expected to perform better with these. Properties explicitly set take precedence over these defaults.

The modified Spark properties are as follows:

Property	Xonai Value
Property	Xonai Value			spark.sql.adaptive.advisoryPartitionSizeInBytes	32MB
spark.sql.cache.serializer	com.xonai.spark.CacheSerializer

Last update: Aug 15, 2025