Memory Modes¶

FuseCore uses off-heap memory to process data and it can operate in one of three modes: off-heap, overhead, and dynamic.

The default memory mode is selected based on existing properties and can be overridden with spark.xonai.memory.mode. The amount of memory FuseCore can use is also automatically computed and can be overridden with spark.xonai.memory.size.

Off-Heap¶

FuseCore uses the Spark off-heap management to acquire memory meaning that Spark and FuseCore share a memory pool. This mode is selected by default if spark.memory.offHeap.size and spark.memory.offHeap.enabled are specified.

The default usage looks as the following:

--conf spark.memory.offHeap.size=30g
--conf spark.memory.offHeap.enabled=true

Overhead¶

FuseCpre uses part of the executor memory overhead with a memory pool separate from Spark. This mode is selected by default if spark.executor.memoryOverhead is specified. The default executor memory overhead is 10% of spark.executor.memory (18% for EMR).

For FuseCore to run optimally, the executor memory (JVM) must be reduced and the memory overhead increased. For example, if an application has the following memory configuration:

--conf spark.executor.memory=30g
--conf spark.executor.memoryOverhead=5g

It can be modified to:

--conf spark.executor.memory=10g
--conf spark.executor.memoryOverhead=25g

By default, FuseCore memory size is around overhead - memory * overhead factor with a default factor of 0.2. For the example above the result is spark.xonai.memory.size=23g.

The overhead memory mode is recommended when most or all operations in an application are supported by FuseCore. Otherwise, non-supported operations may underperform due to limited memory.

Dynamic¶

FuseCore monitors memory allocation by the JVM and it uses non-allocated memory. This mode is selected by default if no other mode is selected. As the JVM grows in size the FuseCore memory size decreases.

This memory mode can be configured using:

--conf spark.xonai.memory.mode=dynamic
--conf spark.xonai.memory.size=1g

In this mode, the value of spark.xonai.memory.size defines a lower-bound and defaults to 50% of the executor memory overhead. If the dynamic memory size is low FuseCore will stop to run operations natively.

The dynamic memory mode is the most conservative and it is recommended when testing FuseCore for the first time or when using it in applications with frequent code changes.