Quick Setup

This page documents the one-time steps to set up the Xonai Accelerator for Amazon EMR using standard cluster bootstrap actions and configuration classifications.

Step 1: Create Bootstrap Script

Create a shell script xonai-activation.sh with the following contents and copy it to a location in S3 accessible from your EMR cluster.

#!/bin/bash
aws s3 cp s3://xonai-activation/install-xonai-emr.sh .
chmod +x ./install-xonai-emr.sh
nohup ./install-xonai-emr.sh &

This runs a script during EMR cluster bootstrapping that copies the Xonai Accelerator JAR to /home/hadoop/ and modify class paths as described here so that the Xonai Shuffle Manager can be activated.

Step 2: Create Default Configuration File

Create a file named xonai-properties.json with the following JSON content and copy it to a location in S3 accessible from your EMR cluster.

[
  {
    "Classification":"spark-defaults",
    "Properties":{
      "spark.jars":"/home/hadoop/xonai-spark-plugin.jar",
      "spark.plugins":"com.xonai.spark.SQLPlugin",
      "spark.shuffle.manager":"com.xonai.spark.ShuffleManager"
    }
  }
]

This will configure Spark applications to automatically activate the downloaded plugin JAR without needing to change spark-submit arguments.

Note

If any of the properties in the default configuration file is overridden in spark-submit or equivalent, the overridden ones need to be appended there.


Next Steps

Your EMR clusters can now launch Spark applications with the Xonai Accelerator, but you need to adjust executor memory properties as described in our guide.