Prerequisites

Before getting started with the Xonai Accelerator for the open-source distribution of Apache Spark, you need to have a copy of the Xonai Accelerator JAR in a location that Spark jobs can access when submitted to the cluster.

The Xonai Accelerator supports Spark 3.x release versions listed in the following table, with the corresponding JAR artifact name. Note that the <distro> and <arch> tags need to be replaced with a Linux distribution and CPU label (see below).

Spark release

Xonai Accelerator JAR

3.3.1

xonai-spark-plugin-<distro>-3.3.1-<arch>.jar

3.3.0

xonai-spark-plugin-<distro>-3.3.0-<arch>.jar

3.2.1

xonai-spark-plugin-<distro>-3.2.1-<arch>.jar

3.2.0

xonai-spark-plugin-<distro>-3.2.0-<arch>.jar

3.1.2

xonai-spark-plugin-<distro>-3.1.2-<arch>.jar

3.1.1

xonai-spark-plugin-<distro>-3.1.1-<arch>.jar

The accelerator supports processors of x86-64 (e.g. Intel and AMD) and ARM 64-bit (e.g. AWS Graviton) architecture family. The label to be used is as follows:

Architecture

Label

x86-64

x86_64

ARM64

aarch64

The Xonai Accelerator is available for any commonly used Linux distribution in data centers.

Xonai Shuffle Manager Requirements

To use the Xonai Shuffle Manager, each executor node must hold a copy of the Xonai Accelerator JAR. This is required because the shuffle manager initialization stage comes before Spark 3 plugin initialization, and consequently, it cannot be set programmatically.

Check the activation guide to see how to launch Spark jobs with the Xonai Shuffle Manager.