Reporting Guide¶
The Xonai execution report file provides a simple and concise way to understand the execution of Spark applications. It includes non-sensitive data about the execution such as:
Xonai incompatibility details
SQL query and stage metrics, including the plan graph
With most information in one place, it should be easy to identify eventual performance bottlenecks or memory issues based on timing and peak execution memory metrics, respectively.
Xonai is continuously adding support for more operations but in the eventuality an operation is not supported the report will contains details about it.
Activation¶
The report can be activated with the configurations below. Xonai will create one file per Spark application named with the application id under the specified folder (in your file system of preference). Most information is written to the file when the application terminates.
--conf spark.xonai.report.enabled=true
--conf spark.xonai.report.path=YOUR_REPORT_DIR
Example¶
The example below shows in more detail the kind of information the report contains. The plan graph uses the DOT language, see Graphviz for more details.
{"failures":"unsupported_output_type","exec":"Filter","expr":"Cast","dataType":"string","childExpressions":["Remainder"],"childDataTypes":["long"]}
__XON_Query__
id 0
status COMPLETED
duration 1036 ms
xonai_task_time 123 ms
total_task_time 185 ms
peak_task_execution_memory 3244041 bytes
__XON_Stage__
id 0
status COMPLETE
xonai_task_time 123 ms
total_task_time 185 ms
peak_task_execution_memory 3244041 bytes
__XON_Graph__
digraph G {
0 [
label=<
<b>AdaptiveSparkPlan</b><br/><br/>
>
];
subgraph cluster1 {
isCluster="true";
label=<
<b>WholeStageCodegen (2)</b>
<br/>
<br/>duration: 132 ms<br/>
>;
2 [
label=<
<b>ColumnarToRow</b><br/><br/>
number of output rows: 1<br/>
number of input batches: 1
>
];
}
subgraph cluster3 {
isCluster="true";
label=<
<b>WholeStageCodegen (01)</b>
<br/>
<br/>duration: 141 ms<br/>
>;
4 [
label=<
<b>XonStats</b><br/><br/>
avg hash probes per key: 1<br/>
max input batch size: 500<br/>
max output batch size: 1<br/>
number of output batches: 1<br/>
number of output rows: 1<br/>
number of reallocations: 0<br/>
number of rehashes: 0<br/>
peak memory size: 3.1 MiB<br/>
prepare time: 123 ms<br/>
processing time: 0 ms<br/>
spill size: 0.0 B
>
];
5 [
label=<
<b>XonHashAggregate</b><br/><br/>
>
];
6 [
label=<
<b>XonProject</b><br/><br/>
>
];
}
7 [
label=<
<b>XonRowToColumnar</b><br/><br/>
number of input rows: 500<br/>
number of output batches: 1<br/>
build time: 2 ms<br/>
peak memory size: 32.0 KiB
>
];
subgraph cluster8 {
isCluster="true";
label=<
<b>WholeStageCodegen (1)</b>
<br/>
<br/>duration: 21 ms<br/>
>;
9 [
label=<
<b>Filter</b><br/><br/>
number of output rows: 500
>
];
10 [
label=<
<b>Range</b><br/><br/>
number of output rows: 1,000
>
];
}
2->0;
4->2;
5->4;
6->5;
7->6;
9->7;
10->9;
}