[SPARK-57823][SQL] Support ORC predicate pushdown for nanosecond-precision timestamps by stevomitric · Pull Request #57681 · apache/spark

stevomitric · 2026-07-31T15:43:18Z

What changes were proposed in this pull request?

Enable ORC predicate pushdown for nanosecond-precision timestamp columns (TIMESTAMP_LTZ(p) / TIMESTAMP_NTZ(p), p in [7, 9]).

OrcFilters.getPredicateLeafType / castLiteralValue route framework types through OrcTypeOps, but both nanos ops inherited the default predicateLeafType = None. A pushed filter on a nanos column therefore reached the throwing arm of getPredicateLeafType and failed the query. This wires the two nanos ops (TimestampLTZNanosOrcOps / TimestampNTZNanosOrcOps) into the pushdown seam:

predicateLeafType now returns Some(PredicateLeaf.Type.TIMESTAMP), matching the physical ORC category the types are stored in (TIMESTAMP for NTZ, TIMESTAMP_INSTANT for LTZ).
castFilterLiteral converts the external filter literal to a plain java.sql.Timestamp whose local wall clock equals the value's nominal wall clock, so the pushed value compares consistently against the ORC min/max statistics:
- NTZ (java.time.LocalDateTime) -> Timestamp.valueOf(ldt).
- LTZ (java.time.Instant) -> Timestamp.valueOf(LocalDateTime.ofInstant(i, UTC)).

Two ORC-specific details drove the literal shape:

ORC's SearchArgument builder checks the literal's class by exact equality (value.getClass() == Type.getValueClass()), so it rejects the OrcTimestamp subclass used on the write path; the literal must be a plain java.sql.Timestamp.
ORC evaluates a TIMESTAMP / TIMESTAMP_INSTANT predicate against the stored value shifted into the JVM default zone (the default useUTCTimestamp=false read mode Spark uses), so the literal must carry the nominal wall clock in the local zone rather than the raw epoch instant. This keeps pushdown correct regardless of the JVM time zone.

Stale comments in TimestampNanosOrcOps / OrcTypeOps / OrcFilters that stated the nanos types had no pushdown are updated.

Why are the changes needed?

Sub-task of SPARK-56822. Without this, a filter on a nanosecond-precision timestamp ORC column throws unsupportedOperationForDataTypeError during planning instead of pushing the predicate down (or safely skipping it).

Does this PR introduce any user-facing change?

Yes. Predicates on nanosecond-precision timestamp ORC columns now push down to ORC search arguments (enabling stripe/row-group skipping) instead of failing.

How was this patch tested?

New OrcFilterSuite test "SPARK-57823: filter pushdown - nanosecond timestamp" asserts the expected PredicateLeaf.Operator is generated for all comparison operators, both nanos types, and all precisions.
New OrcQuerySuite test "SPARK-57823: ORC predicate pushdown returns correct results for nanos timestamps" verifies end-to-end that results are correct (checkAnswer) and that stripes are actually skipped (stripSparkFilter count), across V1/V2, vectorized/non-vectorized readers, both nanos types, and all precisions.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 4.8)

…ision timestamps ### What changes were proposed in this pull request? Enable ORC predicate pushdown for nanosecond-precision timestamp columns (`TIMESTAMP_LTZ(p)` / `TIMESTAMP_NTZ(p)`, `p` in [7, 9]). `OrcFilters.getPredicateLeafType` / `castLiteralValue` route framework types through `OrcTypeOps`, but both nanos ops inherited the default `predicateLeafType = None`. A pushed filter on a nanos column therefore reached the throwing arm of `getPredicateLeafType` and failed the query. This wires the two nanos ops (`TimestampLTZNanosOrcOps` / `TimestampNTZNanosOrcOps`) into the pushdown seam: - `predicateLeafType` now returns `Some(PredicateLeaf.Type.TIMESTAMP)`, matching the physical ORC category the types are stored in (`TIMESTAMP` for NTZ, `TIMESTAMP_INSTANT` for LTZ). - `castFilterLiteral` converts the external filter literal to a plain `java.sql.Timestamp` whose local wall clock equals the value's nominal wall clock, so the pushed value compares consistently against the ORC min/max statistics: - NTZ (`java.time.LocalDateTime`) -> `Timestamp.valueOf(ldt)`. - LTZ (`java.time.Instant`) -> `Timestamp.valueOf(LocalDateTime.ofInstant(i, UTC))`. Two ORC-specific details drove the literal shape: 1. ORC's `SearchArgument` builder checks the literal's class by exact equality (`value.getClass() == Type.getValueClass()`), so it rejects the `OrcTimestamp` subclass used on the write path; the literal must be a plain `java.sql.Timestamp`. 2. ORC evaluates a `TIMESTAMP` / `TIMESTAMP_INSTANT` predicate against the stored value shifted into the JVM default zone (the default `useUTCTimestamp=false` read mode Spark uses), so the literal must carry the nominal wall clock in the local zone rather than the raw epoch instant. This keeps pushdown correct regardless of the JVM time zone. Stale comments in `TimestampNanosOrcOps` / `OrcTypeOps` / `OrcFilters` that stated the nanos types had no pushdown are updated. ### Why are the changes needed? Sub-task of SPARK-56822. Without this, a filter on a nanosecond-precision timestamp ORC column throws `unsupportedOperationForDataTypeError` during planning instead of pushing the predicate down (or safely skipping it). ### Does this PR introduce any user-facing change? Yes. Predicates on nanosecond-precision timestamp ORC columns now push down to ORC search arguments (enabling stripe/row-group skipping) instead of failing. ### How was this patch tested? - New `OrcFilterSuite` test "SPARK-57823: filter pushdown - nanosecond timestamp" asserts the expected `PredicateLeaf.Operator` is generated for all comparison operators, both nanos types, and all precisions. - New `OrcQuerySuite` test "SPARK-57823: ORC predicate pushdown returns correct results for nanos timestamps" verifies end-to-end that results are correct (`checkAnswer`) and that stripes are actually skipped (`stripSparkFilter` count), across V1/V2, vectorized/non-vectorized readers, both nanos types, and all precisions. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 4.8) Co-authored-by: Isaac

The end-to-end nanos pushdown test in OrcQuerySuite ran only under the default (UTC) JVM time zone. Because the LTZ path stores and compares timestamps in the JVM default zone, a literal built in the wrong frame would diverge from the ORC stripe statistics only outside UTC, where this test never ran. Extract the test body into a helper and run it under UTC, America/Los_Angeles, and Asia/Tokyo (both offset signs) so timezone handling is actually exercised. No production code change; the existing pushdown implementation passes in all three zones. Co-authored-by: Isaac

stevomitric · 2026-08-01T20:42:55Z

cc @uros-b, @cloud-fan PTAL when you get a chance. Thanks.

Retrigger CI: the only failure was a flaky, unrelated Connect test (SparkConnectServiceInternalServerSuite hardcoded-port bind collision). Co-authored-by: Isaac

cloud-fan

0 blocking, 0 non-blocking, 2 nits.
The production behavior and coverage look sound; two test comments should be corrected so they describe the exercised path and timezone representation accurately.

Nits: 2 minor items (see inline comments).

Verification

I traced filter literals from OrcFilters.castLiteralValue through both nanos OrcTypeOps implementations and checked ORC 2.3.1's predicate-statistics path: with useUTCTimestamp=false, ORC compares against local timestamp statistics, which explains the LTZ wall-clock compensation. The new end-to-end test exercises that behavior under UTC, Los Angeles, and Tokyo.

dongjoon-hyun

+1, LGTM.

dongjoon-hyun · 2026-08-03T18:29:51Z

The three-zone e2e matrix (UTC / LA / JST under withDefaultTimeZone) validates the LTZ literal
frame for timestamps away from DST transitions — including UTC-ahead JST, which would have failed
checkAnswer if the literal were in the wrong frame wholesale. However, there is one untested
~1-hour-per-year window where the mapping is non-monotonic: Timestamp.valueOf shifts any wall
clock that falls into the JVM zone's spring-forward gap one hour ahead, colliding with (and
inverting against) the hour that follows. With the JVM in America/Los_Angeles, instants whose
UTC wall clock lies in 2020-03-08 02:00..03:00 hit that gap when re-interpreted in LA:
2020-03-08T02:30Z yields a literal epoch of local 03:30, while the later instant
2020-03-08T03:15Z yields local 03:15, so instant order and literal order disagree. If ORC's
stored-value shift for TIMESTAMP_INSTANT statistics does not reproduce exactly the same gap
behavior, a ts < lit boundary inside that window could prune a stripe containing rows the
residual filter would keep. (NTZ is not affected this way: its literal mapping is identical to the
write-path mapping, so pruning stays consistent with what the scan returns.)

Could you either confirm the ORC-side shift matches Timestamp.valueOf gap-for-gap, or extend the
e2e test with a case under withDefaultTimeZone(LA) whose data and boundary straddle such a window
(e.g. wall clocks 2020-03-08T01:30..03:30 with the boundary at 02:45)?

Two smaller notes:

An In pushdown case for the nanos types would cover the per-value castLiteralValue path in
buildLeafSearchArgument (values.map(...)); it is a one-line addition to the new
OrcFilterSuite test.
The existing micros TimestampNTZType path pushes a LONG leaf against ORC timestamp
statistics, which likely never prunes (type mismatch evaluates to YES_NO_NULL). The
TIMESTAMP-leaf approach taken here looks like the right template to align it with in a
follow-up.

HyukjinKwon

0 blocking, 1 non-blocking, 2 nits.
Correct ORC nanosecond pushdown with strong multi-zone coverage; two comment-accuracy nits (already raised) and one non-blocking DST-gap edge-case suggestion remain.

Already raised in existing discussion (2)

The test comment says 'a micro-only value would not exercise the nanos path.' Because the literal is explicitly typed as nanosType, a microsecond-aligned value still exercises the nanos pushdown path; the sub-microsecond digits exercise preservation/comparison beyond microsecond precision. Reword accordingly. -- existing discussion
The OrcQuerySuite comment says the LTZ column 'stores/compares timestamps in the JVM default zone.' Storage is the UTC instant; the JVM default zone enters only at comparison time (ORC compares the predicate against non-UTC timestamp statistics). Distinguish storage from comparison so it does not imply the on-disk value is zone-shifted. -- existing discussion

Suggestions (1)

General: LTZ pushdown has one untested edge: Timestamp.valueOf maps a wall clock in the JVM zone's spring-forward gap one hour ahead, so instant order and literal order can disagree in that ~1hr/year window (dongjoon's example: under America/Los_Angeles, 2020-03-08T02:30Z -> local 03:30 vs 03:15Z -> 03:15). If ORC's stored-value shift for TIMESTAMP_INSTANT stats does not reproduce the same gap behavior, a ts < lit boundary in that window could prune a stripe holding matching rows. Worth either confirming ORC's shift matches Timestamp.valueOf gap-for-gap, or adding an e2e case under withDefaultTimeZone(LA) straddling such a window. (NTZ is unaffected: its literal mapping equals the write-path mapping.) Non-blocking.

Verification

Traced castFilterLiteral for both nanos ops against the write-path serializer (NTZ value-equality) and against ORC's useUTCTimestamp=false local-stats comparison (LTZ wall-clock frame). Confirmed the OrcFilters framework-type fallback preserves the unsupported-type error. The added OrcFilterSuite (all operators x precisions x LTZ/NTZ) and multi-zone OrcQuerySuite e2e cover the correctness. Build/test: build/sbt 'sql/testOnly *OrcFilterSuite *OrcQuerySuite'. (Note: the contract-claim scanner failed once on a transient API error and succeeded on retry.)

…mment fixes Test-only follow-ups from PR review: - Reword the OrcFilterSuite comment: a microsecond-aligned value still exercises the nanos pushdown path (the literal is explicitly nanos-typed); the sub-microsecond digits exercise preservation and comparison beyond microsecond precision. - Reword the OrcQuerySuite comment to distinguish LTZ storage (the UTC instant) from comparison (the JVM default zone enters only when ORC evaluates the pushed predicate against the non-UTC timestamp statistics). - Add an In pushdown case to the OrcFilterSuite nanos test, covering the per-value castLiteralValue path (values.map(...)) in buildLeafSearchArgument. - Add a DST spring-forward-gap regression test to OrcQuerySuite: under America/Los_Angeles, LTZ nanos data straddling the 2020-03-08 02:00..03:00 gap with a boundary inside it. checkAnswer holds -- in the gap the pushed predicate only turns conservative, never pruning a stripe that holds matching rows. Co-authored-by: Isaac

stevomitric · 2026-08-10T21:25:46Z

Thanks! Addressed the two suggestions:

Added an In pushdown case to the OrcFilterSuite nanos test, covering the per-value castLiteralValue path (values.map(...)) in buildLeafSearchArgument.
For the DST spring-forward-gap edge: added a regression test under America/Los_Angeles with LTZ nanos data one minute apart straddling the 2020-03-08 02:00..03:00 gap and a boundary inside it.
cc @dongjoon-hyun

uros-b · 2026-08-11T10:46:11Z

Thank you for addressing all comments @stevomitric! LGTM. Also, thank you @cloud-fan, @dongjoon-hyun and @HyukjinKwon for review!

…ision timestamps ### What changes were proposed in this pull request? Enable ORC predicate pushdown for nanosecond-precision timestamp columns (`TIMESTAMP_LTZ(p)` / `TIMESTAMP_NTZ(p)`, `p` in [7, 9]). `OrcFilters.getPredicateLeafType` / `castLiteralValue` route framework types through `OrcTypeOps`, but both nanos ops inherited the default `predicateLeafType = None`. A pushed filter on a nanos column therefore reached the throwing arm of `getPredicateLeafType` and failed the query. This wires the two nanos ops (`TimestampLTZNanosOrcOps` / `TimestampNTZNanosOrcOps`) into the pushdown seam: - `predicateLeafType` now returns `Some(PredicateLeaf.Type.TIMESTAMP)`, matching the physical ORC category the types are stored in (`TIMESTAMP` for NTZ, `TIMESTAMP_INSTANT` for LTZ). - `castFilterLiteral` converts the external filter literal to a plain `java.sql.Timestamp` whose local wall clock equals the value's nominal wall clock, so the pushed value compares consistently against the ORC min/max statistics: - NTZ (`java.time.LocalDateTime`) -> `Timestamp.valueOf(ldt)`. - LTZ (`java.time.Instant`) -> `Timestamp.valueOf(LocalDateTime.ofInstant(i, UTC))`. Two ORC-specific details drove the literal shape: 1. ORC's `SearchArgument` builder checks the literal's class by exact equality (`value.getClass() == Type.getValueClass()`), so it rejects the `OrcTimestamp` subclass used on the write path; the literal must be a plain `java.sql.Timestamp`. 2. ORC evaluates a `TIMESTAMP` / `TIMESTAMP_INSTANT` predicate against the stored value shifted into the JVM default zone (the default `useUTCTimestamp=false` read mode Spark uses), so the literal must carry the nominal wall clock in the local zone rather than the raw epoch instant. This keeps pushdown correct regardless of the JVM time zone. Stale comments in `TimestampNanosOrcOps` / `OrcTypeOps` / `OrcFilters` that stated the nanos types had no pushdown are updated. ### Why are the changes needed? Sub-task of SPARK-56822. Without this, a filter on a nanosecond-precision timestamp ORC column throws `unsupportedOperationForDataTypeError` during planning instead of pushing the predicate down (or safely skipping it). ### Does this PR introduce any user-facing change? Yes. Predicates on nanosecond-precision timestamp ORC columns now push down to ORC search arguments (enabling stripe/row-group skipping) instead of failing. ### How was this patch tested? - New `OrcFilterSuite` test "SPARK-57823: filter pushdown - nanosecond timestamp" asserts the expected `PredicateLeaf.Operator` is generated for all comparison operators, both nanos types, and all precisions. - New `OrcQuerySuite` test "SPARK-57823: ORC predicate pushdown returns correct results for nanos timestamps" verifies end-to-end that results are correct (`checkAnswer`) and that stripes are actually skipped (`stripSparkFilter` count), across V1/V2, vectorized/non-vectorized readers, both nanos types, and all precisions. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 4.8) Closes #57681 from stevomitric/stevomitric/spark-57823-orc-nanos-pushdown. Authored-by: Stevo Mitric <stevomitric2000@gmail.com> Signed-off-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com> (cherry picked from commit 1a9fb51) Signed-off-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com>

uros-b · 2026-08-11T10:47:52Z

Merge Summary:

merged into master 1a9fb51
merged into branch-4.x 3deb9bb
merged into branch-4.3 55600bf

Posted by merge_spark_pr.py

…lumns read with a lower precision ### What changes were proposed in this pull request? This PR aims to fix the ORC filter pushdown on `TIMESTAMP_LTZ(p)` and `TIMESTAMP_NTZ(p)` columns when the ORC file was written with a higher precision than `p`. Since SPARK-57823 (#57681), ORC pushes down the predicates on these columns. However, the ORC reader truncates the stored nanoseconds to the read precision (SPARK-57455, #56557), while ORC evaluates a pushed down predicate against the stored values. Unlike Parquet (SPARK-60380, #59641), an ORC file schema has the written precision through the Catalyst type attribute (`spark.sql.catalyst.type`), and `OrcFilters.createFilter` builds a `SearchArgument` for each file from its file schema. So, this PR keeps the predicates with a value when the read precision is not lower than the written precision, and pushes down only `IsNull` and `IsNotNull` otherwise. - `OrcTypeOps.acceptsReadType` is added (default `true`), like `ParquetFilterOps.acceptsReadType` of SPARK-60358 (#59606). The ops of `TIMESTAMP_LTZ(p)` and `TIMESTAMP_NTZ(p)` return `true` only for `TIMESTAMP_LTZ(q)` and `TIMESTAMP_NTZ(q)` with `q >= p`, respectively. - `OrcFilters.createFilter` gets the read schema (`requiredSchema` in V1, `readDataSchema` in V2) and checks `acceptsReadType` before it converts a predicate with a value on a Types Framework field. A foreign ORC timestamp column without the attribute cannot be read with the nanosecond timestamp types (`UNSUPPORTED_FEATURE.ORC_TYPE_CAST`), so the written precision is always known here. Since it is known only for each file, the pushed filters in the query plan are not changed. ### Why are the changes needed? The query results are wrong. For example, `00:00:00.123456789` is read as `00:00:00.1234567` with `TIMESTAMP_NTZ(7)`, but the pushed down `=` compares the literal with the stored value. ```scala // spark.sql.timestampNanosTypes.enabled=true sql("SELECT CAST('1970-01-01 00:00:00.123456789' AS TIMESTAMP_NTZ(9)) AS ts UNION ALL " + "SELECT CAST('1970-01-01 00:00:01' AS TIMESTAMP_NTZ(9))").coalesce(1).write.orc("/tmp/nanos") val df = spark.read.schema("ts TIMESTAMP_NTZ(7)").orc("/tmp/nanos") df.collect() // Array([1970-01-01T00:00:00.123456700], [1970-01-01T00:00:01]) df.where("ts = CAST('1970-01-01 00:00:00.1234567' AS TIMESTAMP_NTZ(7))").collect() // Array() ``` `<=>`, `<=` and `IN` return no rows in the same way, with both V1 and V2, with both the vectorized and the row-based readers, and for `TIMESTAMP_LTZ` too. The expected result is `Array([1970-01-01T00:00:00.123456700])`, which is the result with `spark.sql.orc.filterPushdown=false`. ### Does this PR introduce _any_ user-facing change? Yes, this fixes the wrong query results in the unreleased branches (`master`, `branch-4.x` and `branch-4.3`), including [v4.3.0-rc1](https://github.com/apache/spark/releases/tag/v4.3.0-rc1) (2026-08-31) and [v4.3.0-rc2](https://github.com/apache/spark/releases/tag/v4.3.0-rc2) (2026-10-12). The predicates with a value are not pushed down anymore only for the files written with a higher precision than the read precision. The nanosecond timestamp types are a preview feature, which is disabled by default (`spark.sql.timestampNanosTypes.enabled`). ### How was this patch tested? Pass the CIs with the newly added test cases in `OrcFilterSuite` and `OrcV1FilterSuite`. The end-to-end test case fails without this fix. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 Closes #59654 from dongjoon-hyun/SPARK-60397. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…lumns read with a lower precision ### What changes were proposed in this pull request? This PR aims to fix the ORC filter pushdown on `TIMESTAMP_LTZ(p)` and `TIMESTAMP_NTZ(p)` columns when the ORC file was written with a higher precision than `p`. Since SPARK-57823 (#57681), ORC pushes down the predicates on these columns. However, the ORC reader truncates the stored nanoseconds to the read precision (SPARK-57455, #56557), while ORC evaluates a pushed down predicate against the stored values. Unlike Parquet (SPARK-60380, #59641), an ORC file schema has the written precision through the Catalyst type attribute (`spark.sql.catalyst.type`), and `OrcFilters.createFilter` builds a `SearchArgument` for each file from its file schema. So, this PR keeps the predicates with a value when the read precision is not lower than the written precision, and pushes down only `IsNull` and `IsNotNull` otherwise. - `OrcTypeOps.acceptsReadType` is added (default `true`), like `ParquetFilterOps.acceptsReadType` of SPARK-60358 (#59606). The ops of `TIMESTAMP_LTZ(p)` and `TIMESTAMP_NTZ(p)` return `true` only for `TIMESTAMP_LTZ(q)` and `TIMESTAMP_NTZ(q)` with `q >= p`, respectively. - `OrcFilters.createFilter` gets the read schema (`requiredSchema` in V1, `readDataSchema` in V2) and checks `acceptsReadType` before it converts a predicate with a value on a Types Framework field. A foreign ORC timestamp column without the attribute cannot be read with the nanosecond timestamp types (`UNSUPPORTED_FEATURE.ORC_TYPE_CAST`), so the written precision is always known here. Since it is known only for each file, the pushed filters in the query plan are not changed. ### Why are the changes needed? The query results are wrong. For example, `00:00:00.123456789` is read as `00:00:00.1234567` with `TIMESTAMP_NTZ(7)`, but the pushed down `=` compares the literal with the stored value. ```scala // spark.sql.timestampNanosTypes.enabled=true sql("SELECT CAST('1970-01-01 00:00:00.123456789' AS TIMESTAMP_NTZ(9)) AS ts UNION ALL " + "SELECT CAST('1970-01-01 00:00:01' AS TIMESTAMP_NTZ(9))").coalesce(1).write.orc("/tmp/nanos") val df = spark.read.schema("ts TIMESTAMP_NTZ(7)").orc("/tmp/nanos") df.collect() // Array([1970-01-01T00:00:00.123456700], [1970-01-01T00:00:01]) df.where("ts = CAST('1970-01-01 00:00:00.1234567' AS TIMESTAMP_NTZ(7))").collect() // Array() ``` `<=>`, `<=` and `IN` return no rows in the same way, with both V1 and V2, with both the vectorized and the row-based readers, and for `TIMESTAMP_LTZ` too. The expected result is `Array([1970-01-01T00:00:00.123456700])`, which is the result with `spark.sql.orc.filterPushdown=false`. ### Does this PR introduce _any_ user-facing change? Yes, this fixes the wrong query results in the unreleased branches (`master`, `branch-4.x` and `branch-4.3`), including [v4.3.0-rc1](https://github.com/apache/spark/releases/tag/v4.3.0-rc1) (2026-08-31) and [v4.3.0-rc2](https://github.com/apache/spark/releases/tag/v4.3.0-rc2) (2026-10-12). The predicates with a value are not pushed down anymore only for the files written with a higher precision than the read precision. The nanosecond timestamp types are a preview feature, which is disabled by default (`spark.sql.timestampNanosTypes.enabled`). ### How was this patch tested? Pass the CIs with the newly added test cases in `OrcFilterSuite` and `OrcV1FilterSuite`. The end-to-end test case fails without this fix. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 Closes #59654 from dongjoon-hyun/SPARK-60397. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit cd06735) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

stevomitric added 2 commits July 31, 2026 11:37

stevomitric changed the title ~~[WIP][SPARK-57823][SQL] Support ORC predicate pushdown for nanosecond-precision timestamps~~ [SPARK-57823][SQL] Support ORC predicate pushdown for nanosecond-precision timestamps Aug 2, 2026

[SPARK-57823][SQL] Empty commit to retrigger CI

6b1eccf

Retrigger CI: the only failure was a flaky, unrelated Connect test (SparkConnectServiceInternalServerSuite hardcoded-port bind collision). Co-authored-by: Isaac

uros-b reviewed Aug 2, 2026

View reviewed changes

Comment thread ...in/scala/org/apache/spark/sql/execution/datasources/orc/types/ops/TimestampNanosOrcOps.scala

uros-b reviewed Aug 2, 2026

View reviewed changes

Comment thread sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala

cloud-fan approved these changes Aug 3, 2026

View reviewed changes

Comment thread sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilterSuite.scala Outdated

Comment thread sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala Outdated

dongjoon-hyun approved these changes Aug 3, 2026

View reviewed changes

HyukjinKwon reviewed Aug 5, 2026

View reviewed changes

stevomitric requested review from HyukjinKwon and uros-b August 10, 2026 21:26

uros-b approved these changes Aug 11, 2026

View reviewed changes

uros-b closed this in 1a9fb51 Aug 11, 2026

dongjoon-hyun mentioned this pull request Oct 20, 2026

[SPARK-60397][SQL] Fix ORC filter pushdown on nanosecond timestamp columns read with a lower precision #59654

Closed

Navigation Menu

Repository navigation

[SPARK-57823][SQL] Support ORC predicate pushdown for nanosecond-precision timestamps - #57681

[SPARK-57823][SQL] Support ORC predicate pushdown for nanosecond-precision timestamps#57681
stevomitric wants to merge 4 commits into
apache:masterfrom
stevomitric:stevomitric/spark-57823-orc-nanos-pushdown

stevomitric commented Jul 31, 2026

Uh oh!

stevomitric commented Aug 1, 2026

Uh oh!

Uh oh!

Uh oh!

cloud-fan left a comment

Uh oh!

Uh oh!

Uh oh!

dongjoon-hyun left a comment

Uh oh!

dongjoon-hyun commented Aug 3, 2026

Uh oh!

HyukjinKwon left a comment

Uh oh!

stevomitric commented Aug 10, 2026

Uh oh!

uros-b commented Aug 11, 2026

Uh oh!

uros-b commented Aug 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

stevomitric commented Jul 31, 2026

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

stevomitric commented Aug 1, 2026

Uh oh!

Uh oh!

Uh oh!

cloud-fan left a comment

Choose a reason for hiding this comment

Verification

Uh oh!

Uh oh!

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Aug 3, 2026

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Already raised in existing discussion (2)

Suggestions (1)

Verification

Uh oh!

stevomitric commented Aug 10, 2026

Uh oh!

uros-b commented Aug 11, 2026

Uh oh!

uros-b commented Aug 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants