Native decimal AVG returns NULL for large sums at high scale, such as DECIMAL(38,18) above 1.7e16 #7007

Describe the bug

Native decimal AVG returns NULL when the sum is large for its scale. avg() in native/spark-expr/src/agg_funcs/avg_decimal.rs multiplies the unscaled i128 sum by 10^(result scale - sum scale), which is 10^4 for most inputs, before it divides by the count. It turns a checked_mul overflow into NULL. Spark divides with BigDecimal and only checks that the final average fits the result type.

For DECIMAL(38,18), Spark's type for Scala and Java BigDecimal fields, any group whose sum passes about 1.7e16 returns NULL. That covers global and grouped AVG, with ANSI on and off. The threshold is about 1.7 * 10^(34 - scale), so higher scales hit it sooner.

The same path also hides a real range error under ANSI. A single 100000000000000000 row in a DECIMAL(38,18) column averages to a value that doesn't fit DECIMAL(38,22). Spark raises NUMERIC_VALUE_OUT_OF_RANGE there, but Comet returns NULL.

Steps to reproduce

With Comet installed and Comet shuffle enabled:

val path = java.nio.file.Files.createTempDirectory("avg-scale").toString + "/data"
spark.conf.set("spark.comet.enabled", "false")
Seq((1, "9000000000000000"), (1, "9000000000000000"))
  .toDF("k", "raw")
  .selectExpr("k", "CAST(raw AS DECIMAL(38,18)) AS v")
  .coalesce(1)
  .write
  .parquet(path)
spark.conf.set("spark.comet.enabled", "true")
spark.read.parquet(path).createOrReplaceTempView("t")
spark.sql("SELECT avg(v) FROM t").show(false)
spark.sql("SELECT k, avg(v) FROM t GROUP BY k").show(false)

Both queries run as a native partial and a native final CometHashAggregateExec, and both return NULL.

Expected behavior

Both queries return 9000000000000000.0000000000000000000000, as Spark does, with ANSI on and off.

Additional context

Reproduced on main at 66c63bd with Spark 4.1.3. avg() is unchanged at 1.0.0 and 1.1.0, so both releases have it, and it isn't a regression.

#5420 widens an overflowing scaled numerator to i256 and raises NUMERIC_VALUE_OUT_OF_RANGE under ANSI, with grouped regressions for both rows. This issue tracks the defect on main apart from the AVG partial-buffer work in #5418.