Native Parquet scan fails on a duplicate field id in a column the query does not read #7006

Describe the bug

With spark.sql.parquet.fieldId.read.enabled=true, the native Parquet scan fails on a file in which two root columns carry the same field id whenever the table schema has a field with that id, even when the query never reads that field. Spark reads the file. ParquetReadSupport.clipParquetGroupFields only resolves the required schema, and matchIdField raises _LEGACY_ERROR_TEMP_2094 only for a required field whose id matches more than one file field.

Comet runs the check over the whole data schema instead. CometNativeScan serializes the relation's data schema (nativeDataSchema drops only variant columns the read does not request) together with a projection. init_datasource_exec builds the TableSchema from that data schema, and DataFusion's Parquet opener passes table_schema.file_schema() to SparkPhysicalExprAdapterFactory::create. remap_physical_schema then raises the duplicate id error while it resolves every logical root field. The ambiguous name check already waits for a column the read references (check_duplicate_columns), so only the id check fires early.

Steps to reproduce

In a CometTestBase suite on the default Spark 4.1.3 profile:

def withId(id: Int) =
  new MetadataBuilder().putLong(ParquetUtils.FIELD_ID_METADATA_KEY, id.toLong).build()
val writeSchema = new StructType()
  .add("p", IntegerType, true, withId(7))
  .add("q", IntegerType, true, withId(7))
  .add("b", IntegerType, true, withId(3))
spark
  .createDataFrame(spark.sparkContext.parallelize(Seq(Row(1, 2, 3), Row(4, 5, 6))), writeSchema)
  .write
  .parquet(path)
val readSchema = new StructType()
  .add("u", IntegerType, true, withId(7))
  .add("b", IntegerType, true, withId(3))
withSQLConf(SQLConf.PARQUET_FIELD_ID_READ_ENABLED.key -> "true") {
  spark.read.schema(readSchema).parquet(path).select("b").collect()
}

On main at 66c63bd:

Expected behavior

Comet should read b as Spark does, and raise the duplicate id error only when the read references a column whose id matches more than one file field, the way check_duplicate_columns already defers an ambiguous name.

Additional context

@comphead found this while reviewing #6990: #6990 (comment). branch-1.1 has the same check, and #6990 keeps the behavior. On top of #6990, one way to fix it is for FieldResolver::select to return every id match, as it already does for a name. resolve would then raise the error for a nested field, and the root would record the ambiguity for check_duplicate_columns to raise when the read references the column.

The shape needs a file that repeats a field id, so it should be rare in practice.