How Xonai Works

Xonai is a petabyte-scale data infrastructure optimization platform that combines hardware acceleration with performance metering to enable real-time actionable insights for reducing data center costs.

The optimization process is achieved without needing to change the application code, data platform or execution environment.

The platform integrates with multiple Apache Spark 3 runtimes: open-source, EMR runtime and Databricks (beta). It delivers hardware acceleration to Spark applications to reduce the cost of running these, while also submitting usage and performance metrics to our backend.

The different components work together to drastically reduce infrastructure costs where Spark applications are a dominant cost driver.


Data Pipeline Acceleration

The core component responsible for reducing resource-intensive Spark application costs is the Xonai Accelerator, a plugin for Apache Spark 3 that is automatically deployed in Spark 3 applications after a one-time setup process to activate Xonai.

Hardware Acceleration

The Xonai Accelerator activates after Spark selects the final query execution plan, meaning that it preserves the same execution steps as in the original application to avoid unintended surprises in complex Spark applications.

The Catalyst engine code generation process for query plans is transparently replaced with Xonai’s custom compiler tailored for the optimization of data analytics programs, and these programs process incoming batches of data significantly faster than the JVM.

No Environment Changes

The Xonai Accelerator is environment-agnostic, meaning that it will not interfere with your resource manager (e.g. YARN or Kubernetes), hardware, data sources, data platform or any other third-party dependency connected to your Spark applications.

The scope of influence is limited to the internal dynamically generated programs on a per-stage basis that process incoming batches of data at run time.


Real-Time Insights

Note

The Xonai Account Console will be available soon.

Xonai acts as a centralized source of truth to optimize Big Data infrastructure with hardware acceleration and without interfering with established development workflows and operations.

The Xonai Account Console enables real-time actionable insights into how Xonai is optimizing your infrastructure as a whole and down to the individual applications.

Usage & Performance Metering

The Xonai Accelerator sends usage and relevant application performance and configuration metrics to our backend, such as task time, memory usage and configuration settings related to hardware resource allocation.

These are aggregated and enriched to provide accurate insights into how Spark is utilizing your infrastructure and how Xonai is impacting it as it is being rolled out.