Configuration Reference¶

This page documents all configuration properties specific to the Xonai Accelerator.

The properties can be activated just as any other Spark configuration property (see official documentation), folowing the --conf [key]=[value] format.

For example, adding --conf spark.xonai.sql.project.enabled=false to spark-submit:

$SPARK_HOME/bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --conf spark.xonai.sql.project.enabled=false \
  $SPARK_HOME/examples/jars/spark-examples.jar \
  1000

By convention, all Xonai Accelerator properties start with spark.xonai.

Configuration Property List¶

The following is a list of all configuration properties available for the Xonai Accelerator.

Property Key	Description	Default Value
spark.xonai.executor.memoryOverhead	The value of spark.executor.memoryOverhead specified without the plugin, if any. Value in MiB unless otherwise specified.
spark.xonai.memory.dynamic.enabled	Whether the Xonai memory size should be dynamic using memory not allocated by the JVM. Enabled by default if off-heap memory and memory overhead are not specified.	false
spark.xonai.memory.dynamic.fallback.enabled	Whether the plugin should fallback to default Spark execution when the Xonai dynamic memory size is low in a fraction of the executors.	true
spark.xonai.memory.dynamic.fallback.threshold	The value of Xonai dynamic memory size a executor must go below of to trigger execution fallback if spark.xonai.memory.dynamic.fallback.enabled is true. By default, it is defined by numCores x 768M. Value in MiB unless otherwise specified.
spark.xonai.memory.size	Amount of executor memory the Xonai plugin can use. If not specified this value is automatically computed from either spark.memory.offHeap.size or spark.executor.memoryOverhead.
spark.xonai.memory.storageFraction	Amount of Xonai memory that can be used for storage and is immune to eviction, expressed as a fraction of spark.xonai.memory.size. The higher this value is the less memory may be available to execution that can lead to spilling to disk more often.	0.5
spark.xonai.report.enabled	Whether the Xonai plugin should write a report file with execution information.	true
spark.xonai.report.path	Directory where to write Xonai execution reports.	/tmp/xonai
spark.xonai.sql.broadcastHashJoin.enabled	Enable (true) or disable (false) offloading BroadcastHashJoin to native engine.	true
spark.xonai.sql.coalesce.enabled	Enable (true) or disable (false) the Xonai implementation for Coalesce.	true
spark.xonai.sql.enabled	Enable (true) or disable (false) Xonai Spark SQL acceleration	true
spark.xonai.sql.expand.enabled	Enable (true) or disable (false) offloading Expand to native engine.	true
spark.xonai.sql.filter.enabled	Enable (true) or disable (false) offloading Filter to native engine.	true
spark.xonai.sql.generate.enabled	Enable (true) or disable (false) offloading Generate to native engine.	true
spark.xonai.sql.hashAggregate.enabled	Enable (true) or disable (false) offloading HashAggregate to native engine.	true
spark.xonai.sql.offload.mode	Strategy to offload operators to Xonai engine. Must be one of: stage - entire stages are offloaded if supported; partial - contiguous supported operators at the start and end of stages are offloaded; all - all supported operators will be offloaded.	stage
spark.xonai.sql.parquet.reader.enabled	Enable (true) or disable (false) the Xonai optimized Parquet reader.	true
spark.xonai.sql.parquet.writer.enabled	Enable (true) or disable (false) the Xonai optimized Parquet writer.	true
spark.xonai.sql.project.enabled	Enable (true) or disable (false) offloading Project to native engine.	true
spark.xonai.sql.rowToColumnar.batchSize	Size of columnar batches when converting from row to columnar batches.	4096
spark.xonai.sql.shuffle.enabled	Enable (true) or disable (false) shuffling data using columnar format.	true
spark.xonai.sql.shuffledHashJoin.enabled	Enable (true) or disable (false) offloading ShuffledHashJoin to native engine.	true
spark.xonai.sql.sort.enabled	Enable (true) or disable (false) offloading Sort to native engine.	true
spark.xonai.sql.sortMergeJoin.enabled	Enable (true) or disable (false) offloading SortMergeJoin to native engine.	true
spark.xonai.sql.takeOrderedAndProject.enabled	Enable (true) or disable (false) offloading TakeOrderedAndProject to native engine.	true
spark.xonai.sql.union.enabled	Enable (true) or disable (false) the Xonai implementation for Union.	true

Configuration Defaults¶

The Xonai Accelerator may modify the defaults of a few Spark properties as it is expected to perform better with these. Properties explicitly set take precedence over these defaults.

The modified Spark properties are as follows:

Property	Xonai Value
Property	Xonai Value			spark.sql.adaptive.advisoryPartitionSizeInBytes	32MB
spark.sql.cache.serializer	com.xonai.spark.CacheSerializer