Compatibility Reference¶

Apache Spark supports a large number of operations and data types for structured data processing. This page documents how data transformation results produced by the Xonai Accelerator are compatible with Spark and details the support status of each operation.

Compatibility of Results¶

The Xonai Accelerator produces the same data transformation results as Apache Spark, except for cases where Spark cannot itself guarantee to be deterministic or are left to the implementation to define.

In practice, this means that inconsistency between results is already present in Spark itself, even if the user is unaware of it. The following sections document specific cases where this occurs but is nevertheless expected.

Floating Point Arithmetic¶

Floating point calculations are never expected to be exact because of the fundamental limitation of representing continuous real numbers with a fixed set of bits.

The Xonai Accelerator supports the same 4-byte single-precision and 8-byte double-precision floating-point types as Spark, but both engines may produce results with very small discrepancies as the order of machine instructions is implementation-defined.

For example, Spark itself may produce inconsistent results for the same application if the JDK version is changed, as each JDK may produce instructions to do floating point arithmetic with a different order or because of changes in math-related builtins.

When maintaining arithmetic precision is critical, such as in currency calculations, the Spark Decimal type should be used instead.

Ordering of Results¶

Rows with the same sorting or grouping values may be returned in a different order than default Spark, but the SQL standard specification is always adhered to.

This means that the order of results of aggregations is inherently inconsistent, as the order of elements in the underlying hash table is implementation-defined, and this applies to other operations such as sort-merge join.

When sorting, both Spark and the Xonai Accelerator comply with the SQL standards specification and do not guarantee stable sorting, meaning the order of rows with equal sorting may not be the same between different Spark engines.

Backporting Bug Fixes¶

When the Xonai Accelerator supports a new Spark release version, it backports bug fixes that will also fix results-related bugs in previous Spark versions (see this as an example).

Operation Support¶

The Xonai Accelerator is regularly updated to support new operations, while components not yet supported will simply fall back to the default Spark execution engine.

This section documents all operation support status and is updated at every new release.

Support Status Symbols¶

The following table describes the meaning of each status support symbol.

Data Types¶

The following table describes the meaning of each abbreviated type name in the table and a brief description of the type. See the official documentation for more information.

SparkPlan or Executor Nodes¶

Expressions and SQL Functions¶

Aggregate Expressions¶

spark-alchemy Expressions¶

Arithmetic Expressions¶

Array Type Expressions¶

Bitwise Expressions¶

Core Expressions¶

Collection Expressions¶

Conditional Expressions¶

Constraint Expressions¶

CSV Expressions¶

Datetime Expressions¶