Configuration Reference¶

This page documents all configuration properties specific to the Xonai Accelerator.

The properties can be activated just as any other Spark configuration property (see official documentation), folowing the --conf [key]=[value] format.

For example, adding --conf spark.xonai.sql.project.enabled=false to spark-submit:

$SPARK_HOME/bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --conf spark.xonai.sql.project.enabled=false \
  $SPARK_HOME/examples/jars/spark-examples.jar \
  1000

By convention, all Xonai Accelerator properties start with spark.xonai.

Configuration Property List¶

The following is a list of all configuration properties available for the Xonai Accelerator.

Property Key	Description	Default Value
Property Key	Description	Default Value	spark.xonai.sql.broadcastHashJoin.enabled	Enable (true) or disable (false) offloading BroadcastHashJoin to native engine.	true
spark.xonai.sql.compiler.enabled	Enable (true) or disable (false) offloading SQL operations to native engine.	true
spark.xonai.sql.expand.enabled	Enable (true) or disable (false) offloading Expand to native engine.	true
spark.xonai.sql.filter.enabled	Enable (true) or disable (false) offloading Filter to native engine.	true
spark.xonai.sql.hashAggregate.enabled	Enable (true) or disable (false) offloading HashAggregate to native engine.	true
spark.xonai.sql.inMemoryColumnarStorage.compression.codec	Codec to compress in memory data when spark.sql.inMemoryColumnarStorage.compressed is enabled. Supports same codecs as spark.io.compression.codec.	lz4
spark.xonai.sql.parquet.enabled	Enable (true) or disable (false) using Fusion optimized parquet reader.	true
spark.xonai.sql.parquet.lazyDecodingEnabled	Enable (true) or disable (false) lazy decoding when reading parquet columns.	true
spark.xonai.sql.project.enabled	Enable (true) or disable (false) offloading Project to native engine.	true
spark.xonai.sql.rowToColumnar.batchSize	Size of columnar batches when converting from row to columnar batches.	4096
spark.xonai.sql.shuffle.enabled	Enable (true) or disable (false) shuffling data using columnar format.	true
spark.xonai.sql.shuffledHashJoin.enabled	Enable (true) or disable (false) offloading ShuffledHashJoin to native engine.	true
spark.xonai.sql.sort.enabled	Enable (true) or disable (false) offloading Sort to native engine.	true
spark.xonai.sql.sortMergeJoin.enabled	Enable (true) or disable (false) offloading SortMergeJoin to native engine.	true
spark.xonai.sql.takeOrderedAndProject.enabled	Enable (true) or disable (false) offloading TakeOrderedAndProject to native engine.	true

Configuration Defaults¶

The Xonai Accelerator modifies the defaults of a few Spark properties as it is expected to perform better with these. Properties explicitly set take precedence over these defaults.

The modified Spark properties are as follows:

Property	Xonai Value
Property	Xonai Value	spark.executor.memoryOverheadFactor	0.1875
spark.sql.adaptive.advisoryPartitionSizeInBytes	32MB
spark.sql.cache.serializer	com.xonai.spark.CacheSerializer