Skip to content

fix: coerce INT96 in two-level and legacy Parquet lists - #26464

Open
ranflarion wants to merge 1 commit into
apache:mainfrom
ranflarion:fix/int96-coercion-legacy-lists
Open

ranflarion wants to merge 1 commit into
apache:mainfrom
ranflarion:fix/int96-coercion-legacy-lists

Conversation

@ranflarion

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

With coerce_int96 set, INT96 timestamps inside Hive's two-level lists (how Hive writes array<timestamp>), bare repeated int96 fields and parquet-mr Avro's array lists stay Timestamp(Nanosecond, None), so values outside 1677-2262 overflow on decode, which is what coerce_int96 exists to prevent.

What changes are included in this PR?

Int96Coercer no longer rebuilds Parquet paths from Arrow field names, which assumed the three-level <list>.list.<element> layout. For each top-level field it records which of its Parquet leaf columns are INT96, in the depth-first order in which arrow-rs maps them to Arrow leaves, and walks the field's Arrow type in that order, so the list or map encoding no longer matters. Top-level fields are still matched by name, so a projected file schema keeps working. The walk now also descends LargeList and FixedSizeList, and replaces the explicit stack with a short recursion that keeps field metadata via field_with_new_type.

What is the testing strategy for this PR?

coerce_int96_to_resolution_with_two_level_and_legacy_lists covers a Hive two-level list, a bare repeated field, an Avro-style array list of structs, a three-level list, a map value and a non-INT96 nanosecond timestamp, and then a projected, reordered file schema. It fails on main for the first three. The existing nested-type and field-metadata tests pass unchanged.

Are there any user-facing changes?

No API changes. INT96 columns in those list layouts now coerce like the rest.

@github-actions github-actions Bot added the datasource Changes to the datasource crate label Oct 18, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.28571% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.89%. Comparing base (290ad15) to head (d563395).

Files with missing lines Patch % Lines
...tafusion/datasource-parquet/src/schema_coercion.rs 94.28% 8 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #26464      +/-   ##
==========================================
- Coverage   82.89%   82.89%   -0.01%     
==========================================
  Files        1154     1154              
  Lines      457656   457678      +22     
  Branches   457656   457678      +22     
==========================================
+ Hits       379391   379398       +7     
- Misses      55261    55272      +11     
- Partials    23004    23008       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

datasource Changes to the datasource crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

coerce_int96 leaves INT96 in two-level and legacy Parquet lists as nanoseconds

2 participants