How it works¶

The Xonai Accelerator plugs into Apache Spark 3.1+ via plugin interface, which allows injecting a custom backend for Spark APIs, such as SQL and DataFrame, and execute applications faster without requiring application code changes.

The Xonai Accelerator backend generates code for supported SQL operations with a custom DSL and MLIR-based compiler purpose-built to optimize data analytics programs. This replaces the default Java code generation and JVM compiler if the SQL operation is supported, otherwise it will fallback to the default Spark Catalyst engine and convert data between engines. Check your applications compatibility using Xonai Radar.

The Xonai Accelerator does not interfere with query planning, execution model or any other mechanism other than replacing the Catalyst engine at execution time only for the purpose of doing batch processing. This is a design principle in order to avoid breaking existing enterprise-grade applications or requiring tuning them for a custom backend.

Plugin Activation¶

The Xonai Accelerator can be activated on a per-application basis via Spark 3 configuration just by adding the following properties:

--jars <scheme>://<path>/xonai-spark-plugin-<package>-<runtime>-<release>-<platform>.jar
--conf spark.plugins=com.xonai.spark.SQLPlugin

For optimal performance, memory configuration may be required as specified in Memory Modes.