[SPARK-60386][SQL] Enforce FILE EXTERNAL in table columns and reject FILE where it would need an ordering - #59640

What changes were proposed in this pull request?

Part 2 of the FileType SPIP (SPARK-59132), following the Catalyst type added in #59305. The SPIP requires the EXTERNAL keyword when a FILE column is declared in a table, and gives FileType no defined ordering. This PR enforces both at analysis time.

Table columns:

• A FILE in a table column definition, including one nested in a struct, array or map, must be FILE EXTERNAL: CREATE TABLE, REPLACE TABLE, ALTER TABLE ADD COLUMNS and ALTER TABLE REPLACE COLUMNS fail with FILE_TYPE_MISSING_QUALIFIER. The check is in CheckAnalysis for V2 tables, in PreprocessTableCreation for V1 tables, and next to the existing failUnsupportedDataType call in AddColumns/ReplaceColumns.
• CREATE TABLE AS SELECT cannot store a FILE without a qualifier, as one produced by CAST(... AS FILE) (FILE_TYPE_UNKNOWN_WRITE).
• A FILE column cannot be a partition or clustering column, in CREATE TABLE for V1 and V2 tables (including V2 transforms such as bucket) and in ALTER TABLE ... CLUSTER BY (FILE_TYPE_UNSUPPORTED_COLUMN_USAGE.PARTITION_COLUMN and .CLUSTERING_COLUMN). A FILE-specific error is used rather than INVALID_PARTITION_COLUMN_DATA_TYPE, which V1 file sources raised before, so the error says why the column is rejected.

Routines:

• Functions declare a plain FILE, as the SPIP specifies. CREATE FUNCTION rejects FILE EXTERNAL in its parameter or return types (FILE_TYPE_IN_ROUTINE_SIGNATURE_UNSUPPORTED_QUALIFIER), and a FILE in a map key of its return type (FILE_TYPE_IN_ROUTINE_SIGNATURE_UNSUPPORTED_MAP_KEY).
• The same applies to the return type of UDFs, when they are registered through spark.udf.register/spark.udtf.register and, for UDFs that are not registered, when CheckAnalysis sees them in a query. A non-scalar Python UDF is skipped, as its data type is the schema of the rows it returns.

No ordering, following the existing checks for VARIANT:

• Map key: TypeUtils.checkForMapKeyType (DATATYPE_MISMATCH.INVALID_MAP_KEY_TYPE).
• Grouping: ExprUtils.checkValidGroupingExprs (GROUP_EXPRESSION_TYPE_IS_NOT_ORDERABLE).
• INTERSECT, EXCEPT and DISTINCT: CheckAnalysis and OperatorWithUncomparableTypeValidator of the single-pass resolver (UNSUPPORTED_FEATURE.SET_OPERATION_ON_FILE_TYPE).
• DISTRIBUTE BY/repartition: CheckAnalysis and RepartitionByExpressionResolver (UNSUPPORTED_FEATURE.PARTITION_BY_FILE).
• Hash functions: HashExpression.checkInputDataTypes (DATATYPE_MISMATCH.HASH_FILE_TYPE), which also guards hashes the planner inserts.
• Join keys, comparisons, sorting and window PARTITION BY/ORDER BY need no change: FileType is not orderable, so they already fail with DATATYPE_MISMATCH.INVALID_ORDERING_TYPE or EXPRESSION_TYPE_IS_NOT_ORDERABLE. ALTER COLUMN ... TYPE FILE already fails with NOT_SUPPORTED_CHANGE_COLUMN. This PR adds tests for these.
• Aggregates that deduplicate their input, such as count(DISTINCT f) and collect_set(f), are not rejected here. They cannot run before FILE has a physical representation, and whether to reject them is left to that part.

The FileType Scaladoc is updated to describe these rules as current rather than planned.

Why are the changes needed?

The SPIP defines a FILE column of a table as an external file reference, and a bare FILE (the unknown mode) as the type of an expression or a UDF result, so a table should never store a bare FILE. FileType has no defined ordering, so operations that compare, group or hash values must fail clearly rather than operate on the underlying struct.

Does this PR introduce any user-facing change?

No relative to released versions. The FILE type is only recognized when spark.sql.fileType.enabled is true, which is the default only in tests. Relative to master, the statements and operations listed above now fail with the errors listed above when the type is enabled.

How was this patch tested?

Added FileTypeRestrictionsSuite, covering each rule for V1 and V2 tables where both apply, nested FILE types, and the cases that are still accepted (FILE EXTERNAL columns, a FILE map value, a function declaring a plain FILE). Also ran SparkThrowableSuite, AnalysisErrorSuite, ComplexTypeSuite, HashExpressionsSuite, TypeUtilsSuite, UDFSuite, SQLFunctionSuite, VariantSuite, DataSourceV2SQLSuiteV1Filter, CreateTableAsSelectSuite, PartitionedWriteSuite and the V1 and V2 command suites under org.apache.spark.sql.execution.command.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Isaac

This pull request and its description were written by Isaac.