Compatibility Guide¶

This page documents how data transformation results produced by the Xonai Accelerator are compatible with Spark.

Compatibility of Results¶

The Xonai Accelerator produces the same data transformation results as Apache Spark, except for cases where Spark cannot itself guarantee to be deterministic or are left to the implementation to define.

In practice, this means that inconsistency between results is already present in Spark itself, even if the user is unaware of it. The following sections document specific cases where this occurs but is nevertheless expected.

Floating Point Arithmetic¶

Floating point calculations are never expected to be exact because of the fundamental limitation of representing continuous real numbers with a fixed set of bits.

The Xonai Accelerator supports the same 4-byte single-precision and 8-byte double-precision floating-point types as Spark, but both engines may produce results with very small discrepancies as the order of machine instructions is implementation-defined.

For example, Spark itself may produce inconsistent results for the same application if the JDK version is changed, as each JDK may produce instructions to do floating point arithmetic with a different order or because of changes in math-related builtins.

When maintaining arithmetic precision is critical, such as in currency calculations, the Spark Decimal type should be used instead.

Ordering of Results¶

Rows with the same sorting or grouping values may be returned in a different order than default Spark, but the SQL standard specification is always adhered to.

This means that the order of results of aggregations is inherently inconsistent, as the order of elements in the underlying hash table is implementation-defined, and this applies to other operations such as sort-merge join.

When sorting, both Spark and the Xonai Accelerator comply with the SQL standards specification and do not guarantee stable sorting, meaning the order of rows with equal sorting may not be the same between different Spark engines.

Bug Fixes¶

When the Xonai Accelerator supports a new Spark release version, it backports bug fixes that will also fix results-related bugs in previous Spark versions (see this as an example).

Additionally, the accelerator may fix bugs that are not publicly identified.

Exceptions¶

The following list describes exceptional scenarios in which the accelerator may differ slightly:

• Round and BRound expressions with float or double inputs will return NULL when the result cannot be represented in a 128-bit decimal format (equivalent to supporting up-to 38 digits).