Configuration

This page documents all configuration properties specific to FuseCore.

The properties can be activated just as any other Spark configuration property (see official documentation), following the --conf [key]=[value] format.

For example, adding --conf spark.xonai.sql.project.enabled=false to spark-submit:

$SPARK_HOME/bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --conf spark.xonai.sql.project.enabled=false \
  $SPARK_HOME/examples/jars/spark-examples.jar \
  1000

By convention, all FuseCore properties start with spark.xonai.

Configuration Property List

The following is a list of all configuration properties available for FuseCore.

Property Key

Description

Default Value

spark.xonai.memory.dynamic.fallback.enabled

Whether FuseCore should fallback to default Spark execution when dynamic memory size is low.

true

spark.xonai.memory.dynamic.fallback.threshold

Value of FuseCore dynamic memory size a executor must go below of to trigger execution fallback. By default, it is defined by numCores x 768M. Value in MiB unless otherwise specified.

spark.xonai.memory.mode

How FuseCore manages execution memory. Must be one of:

  • off-heap - FuseCore uses the Spark off-heap management to acquire memory. Default if spark.memory.offHeap.size and spark.memory.offHeap.enabled are specified.

  • overhead - the Spark executor memory (JVM) must be reduced and the memory overhead increased for FuseCore to use. Default if spark.executor.memoryOverhead is specified.

  • dynamic - FuseCore uses memory not allocated by the JVM. Default if no other mode is selected. As the JVM grows in size the FuseCore memory size decreases.

spark.xonai.memory.size

Amount of executor memory FuseCore can use. Default value is derived from spark.memory.offHeap.size in off-heap memory mode and from spark.executor.memoryOverhead in overhead and dynamic modes. In dynamic mode the value defines a lower-bound.

spark.xonai.memory.storageFraction

Amount of FuseCore memory that can be used for storage and is immune to eviction, expressed as a fraction of spark.xonai.memory.size. The higher this value is the less memory may be available to execution that can lead to spilling to disk more often.

0.5

spark.xonai.report.enabled

Whether the FuseCore should write a report file with execution information.

false

spark.xonai.report.path

Directory where to write FuseCore execution reports.

/tmp/fusecore

spark.xonai.sql.broadcastHashJoin.enabled

Whether FuseCore BroadcastHashJoin can be used.

true

spark.xonai.sql.coalesce.enabled

Whether FuseCore Coalesce can be used.

true

spark.xonai.sql.enabled

Whether FuseCore is enabled

true

spark.xonai.sql.expand.enabled

Whether FuseCore Expand can be used.

true

spark.xonai.sql.filter.enabled

Whether FuseCore Filter can be used.

true

spark.xonai.sql.generate.enabled

Whether FuseCore Generate can be used.

true

spark.xonai.sql.hashAggregate.enabled

Whether FuseCore HashAggregate can be used.

true

spark.xonai.sql.offload.mode

Strategy to offload operators to FuseCore. Must be one of:

  • stage - entire stages are offloaded if supported

  • partial - contiguous supported operators at the start and end of stages are offloaded

  • all - all supported operators are offloaded

stage

spark.xonai.sql.parquet.reader.cometEnabled

Whether the Comet Parquet reader should be used.

true

spark.xonai.sql.parquet.reader.enabled

Whether the FuseCore Parquet reader should be used.

true

spark.xonai.sql.parquet.writer.enabled

Whether the FuseCore Parquet writer should be used.

true

spark.xonai.sql.project.enabled

Whether FuseCore Project can be used.

true

spark.xonai.sql.rowToColumnar.batchSize

Size of columnar batches when converting from row to columnar batches.

4096

spark.xonai.sql.shuffle.enabled

Whether data shuffle should use columnar format.

true

spark.xonai.sql.shuffledHashJoin.enabled

Whether FuseCore ShuffledHashJoin can be used.

true

spark.xonai.sql.sort.enabled

Whether FuseCore Sort can be used.

true

spark.xonai.sql.sortMergeJoin.enabled

Whether FuseCore SortMergeJoin can be used.

true

spark.xonai.sql.takeOrderedAndProject.enabled

Whether FuseCore TakeOrderedAndProject can be used.

true

spark.xonai.sql.union.enabled

Whether FuseCore Union can be used.

true

spark.xonai.sql.window.enabled

Whether FuseCore Window can be used.

true

spark.xonai.sql.windowGroupLimit.enabled

Whether FuseCore WindowGroupLimit can be used.

true

Configuration Defaults

FuseCore may modify the defaults of a few Spark properties as it is expected to perform better with these. Properties explicitly set take precedence over these defaults.

The modified Spark properties are as follows:

Property

FuseCore Value

spark.sql.adaptive.advisoryPartitionSizeInBytes

32MB

spark.sql.cache.serializer

com.xonai.spark.CacheSerializer
Last update: Aug 17, 2026