🌐 US-Proxy
class="logged-out env-production page-responsive" style="word-wrap: break-word;" >
Skip to content

[SPARK-60387][SQL][4.0] Keep the existing table when saveAsTable overwrite fails before writing - #59658

Closed
dongjoon-hyun wants to merge 1 commit into
apache:branch-4.0from
dongjoon-hyun:SPARK-60387-4.0
Closed

dongjoon-hyun wants to merge 1 commit into
apache:branch-4.0from
dongjoon-hyun:SPARK-60387-4.0

Conversation

@dongjoon-hyun

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

This PR aims to backport #59643 to branch-4.0.

This PR makes saveAsTable with SaveMode.Overwrite drop the existing V1 table only after the checks that run before writing, such as the analysis, the data schema check, and the write options.

Previously, DataFrameWriter.saveAsTable dropped the table before running the CTAS. Now the CTAS drops it:

  • CreateDataSourceTableAsSelectCommand drops it right before the write job starts, in the beforeSetupJob callback of FileFormatWriter.write (SPARK-60360), or right before CreatableRelationProvider.createRelation. If an extension replaces InsertIntoHadoopFsRelationCommand in the physical plan (e.g. a native writer), it drops the table before the write, as before.
  • CreateHiveTableAsSelectCommand drops it after the analysis, because it creates the new table before writing.

Note that this backport differs from the original patch in the following branch-4.0-specific points:

  • branch-4.0 has no SaveAsV1TableCommand, so the drop is removed from DataFrameWriter.saveAsTable instead.
  • branch-4.0 has no QueryExecution.runCommand (SPARK-54835), so DataSource.writeAndRead inlines the eager command execution, like the branch-4.1 backport.

Why are the changes needed?

When these checks fail, the table has already been dropped. A managed table loses its data too.

spark.range(3).write.saveAsTable("t")
spark.range(5).write.mode("overwrite").option("compression", "bogus").saveAsTable("t")
// SparkIllegalArgumentException: [CODEC_NOT_AVAILABLE.WITH_AVAILABLE_CODECS_SUGGESTION] ...
spark.table("t")
// AnalysisException: [TABLE_OR_VIEW_NOT_FOUND] The table or view `t` cannot be found. ...

The same happens with an unsupported data type (e.g. a map column in CSV) or a nonexistent partition column. This has been the behavior since Spark 1.3.0.

Does this PR introduce any user-facing change?

Yes. Such an overwrite keeps the existing table and its data. With format("hive"), only a failure in the analysis keeps the table, because CreateHiveTableAsSelectCommand creates the new table before writing. Valid writes are unchanged.

How was this patch tested?

Pass the CIs with the newly added tests in DataFrameReaderWriterSuite and MetastoreDataSourcesSuite.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

@dongjoon-hyun

Copy link
Copy Markdown
Member Author

Thank you, @HyukjinKwon !

dongjoon-hyun added a commit that referenced this pull request Oct 21, 2026
…erwrite fails before writing

### What changes were proposed in this pull request?

This PR aims to backport #59643 to `branch-4.0`.

This PR makes `saveAsTable` with `SaveMode.Overwrite` drop the existing V1 table only after the checks that run before writing, such as the analysis, the data schema check, and the write options.

Previously, `DataFrameWriter.saveAsTable` dropped the table before running the CTAS. Now the CTAS drops it:
- `CreateDataSourceTableAsSelectCommand` drops it right before the write job starts, in the `beforeSetupJob` callback of `FileFormatWriter.write` (SPARK-60360), or right before `CreatableRelationProvider.createRelation`. If an extension replaces `InsertIntoHadoopFsRelationCommand` in the physical plan (e.g. a native writer), it drops the table before the write, as before.
- `CreateHiveTableAsSelectCommand` drops it after the analysis, because it creates the new table before writing.

Note that this backport differs from the original patch in the following `branch-4.0`-specific points:
- `branch-4.0` has no `SaveAsV1TableCommand`, so the drop is removed from `DataFrameWriter.saveAsTable` instead.
- `branch-4.0` has no `QueryExecution.runCommand` (SPARK-54835), so `DataSource.writeAndRead` inlines the eager command execution, like the `branch-4.1` backport.

### Why are the changes needed?

When these checks fail, the table has already been dropped. A managed table loses its data too.

```scala
spark.range(3).write.saveAsTable("t")
spark.range(5).write.mode("overwrite").option("compression", "bogus").saveAsTable("t")
// SparkIllegalArgumentException: [CODEC_NOT_AVAILABLE.WITH_AVAILABLE_CODECS_SUGGESTION] ...
spark.table("t")
// AnalysisException: [TABLE_OR_VIEW_NOT_FOUND] The table or view `t` cannot be found. ...
```

The same happens with an unsupported data type (e.g. a map column in CSV) or a nonexistent partition column. This has been the behavior since Spark 1.3.0.

### Does this PR introduce _any_ user-facing change?

Yes. Such an overwrite keeps the existing table and its data. With `format("hive")`, only a failure in the analysis keeps the table, because `CreateHiveTableAsSelectCommand` creates the new table before writing. Valid writes are unchanged.

### How was this patch tested?

Pass the CIs with the newly added tests in `DataFrameReaderWriterSuite` and `MetastoreDataSourcesSuite`.

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

Closes #59658 from dongjoon-hyun/SPARK-60387-4.0.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
@dongjoon-hyun

Copy link
Copy Markdown
Member Author

Merge Summary:

Posted by merge_spark_pr.py

@dongjoon-hyun
dongjoon-hyun deleted the SPARK-60387-4.0 branch October 21, 2026 02:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants