Skip to content

Negative-scale decimal /, div, ceil and floor fall back to Spark while round dispatches #7023

Open

Description

@andygrove

Part of #5572. Follow-up from #5050.

With spark.sql.legacy.allowNegativeScaleOfDecimal=true, a negative-scale decimal input falls the whole projection back to Spark for four serdes:

CometRound returns Unsupported for the same input but mixes in CodegenDispatchFallback, so round stays in the Comet pipeline and runs Spark's own doGenCode (arithmetic.scala#L366-L382). round on negative-scale decimal in CometExpressionSuite covers it. The other four could do the same. The EPIC deferred ceil and floor as marginal, since only the legacy flag reaches them, and #5050 adds division to that group.

Fix: for CometCeil and CometFloor the negative scale is the only Unsupported case, so mixing in CodegenDispatchFallback is the whole change. For CometDivide and CometIntegralDivide it isn't a one-line change. The mixin would also send their other Unsupported results to the dispatcher, namely mathDataTypeSupportLevel's unsupported-datatype case and CometDivide's check that a decimal result has decimal operands. Each of those needs checking against the dispatcher's type gate, or these serdes need a way to dispatch only the negative-scale case.

Tests should cover a materialized negative-scale column, not only a cast literal. That input broke the dispatcher's decimal reader until #5050 made it read negative scales through BigDecimal.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions