Update arrow_datafusion (major)

[dashql-renovate]

This PR contains the following updates:

Package	Type	Update	Change
arrow	dependencies	major	55.2.0 -> 57.2.0
datafusion (source)	dependencies	major	48.0.1 -> 52.0.0
datafusion-common (source)	dependencies	major	48.0.1 -> 52.1.0
datafusion-datasource (source)	dependencies	major	48.0.1 -> 52.1.0
datafusion-execution (source)	dependencies	major	48.0.1 -> 52.1.0
datafusion-expr (source)	dependencies	major	48.0.1 -> 52.1.0
datafusion-functions (source)	dependencies	major	48.0.1 -> 52.1.0
datafusion-functions-aggregate (source)	dependencies	major	48.0.1 -> 52.1.0
datafusion-functions-nested (source)	dependencies	major	48.0.1 -> 52.1.0
datafusion-functions-window (source)	dependencies	major	48.0.1 -> 52.1.0
datafusion-optimizer (source)	dependencies	major	48.0.1 -> 52.1.0
datafusion-physical-expr (source)	dependencies	major	48.0.1 -> 52.1.0
datafusion-physical-plan (source)	dependencies	major	48.0.1 -> 52.1.0

---

Release Notes

apache/arrow-rs (arrow)

v57.2.0

Compare Source

Full Changelog

Breaking changes:

• Seal Array trait #​9092 [arrow] (tustvold)
• [Variant] Unify the CastOptions usage in parquet-variant-compute #​8984 (klion26)

Implemented enhancements:

• [parquet] further relax LevelInfoBuilder::types_compatible for ArrowWriter #​9098
• Update arrow-row documentation with Union encoding #​9084
• Add code examples for min and max compute functions #​9055
• Add append_n to bytes view builder API #​9034 [arrow]
• Move RunArray::get_physical_indices to RunEndBuffer #​9025 [arrow]
• Allow quote style in csv writer #​9003 [arrow]
• IPC support for ListView #​9002 [arrow]
• Implement BinaryArrayType for &FixedSizeBinaryArrays #​8992 [arrow]
• arrow-buffer: implement num-traits for i256 #​8976 [arrow]
• Support for Arc<str> in ParquetRecordWriter derive macro #​8972
• [arrow-avro] suggest switching from xz to liblzma #​8970 [arrow]
• arrow-buffer: add i256::trailing_zeros #​8968 [arrow]
• arrow-buffer: make i256::leading_zeros public #​8965 [arrow]
• Add spark like ignoreLeadingWhiteSpace and ignoreTrailingWhiteSpace options to the csv writer #​8961 [arrow]
• Add round trip benchmark for Parquet writer/reader #​8955 [parquet]
• Support performant interleave for List/LargeList #​8952 [arrow]
• [Variant] Support array access when parsing VariantPath #​8946
• Some panic!s could be represented as unimplemented!s #​8932 [arrow]
• [Variant] easier way to construct a shredded schema #​8922
• Support DataType::ListView and DataType::LargeListView in ArrayData::new_null #​8908 [arrow]
• Add GenericListViewArray::from_iter_primitive #​8906 [arrow]
• [Variant] Unify the cast option usage in ParquentVariant #​8873
• Blog post about efficient filter representation in Parquet filter pushdown #​8843 [parquet]
• Add comparison support for Union arrays in the cmp kernel #​8837 [arrow]
• [Variant] Support array shredding into List/LargeList/ListView/LargeListView #​8830
• Support Union data types for row format #​8828 [arrow]
• FFI support for ListView #​8819 [arrow]
• [Variant] Support more Arrow Datatypes from Variant primitive types #​8805
• FixedSizeBinaryBuilder supports append_array #​8750 [arrow]
• Implement special case zip with scalar for Utf8View #​8724 [arrow]
• [geometry] Wire up arrow reader/writer for GEOMETRY and GEOGRAPHY #​8717 [parquet]

Fixed bugs:

• Soundness Bug in try_binary when Array is implemented incorrectly in external crate #​9106
• casting Dict(_, LargeUtf8) to Utf8View (StringViewArray) panics #​9101
• wrong results for null count of nullif kernel #​9085 [parquet] [arrow]
• Empty first line in some code examples #​9063
• GenericByteViewArray::slice is not zero-copy but ought to be #​9014
• Regression in struct casting in 57.2.0 (not yet released) #​9005 [arrow]
• Fix panic when decoding multiple Union columns in RowConverter #​8999 [arrow]
• take_fixed_size_binary Does Not Consider NULL Indices #​8947 [arrow]
• [arrow-avro] RecordEncoder Bugs #​8934 [arrow]
• FixedSizeBinaryArray::try_new(...) Panics with Item Length of Zero #​8926 [arrow]
• cargo test -p arrow-cast fails on main #​8910 [arrow]
• GenericListViewArray::new_null ignores len and returns an empty array #​8904 [arrow]
• FixedSizeBinaryArray::new_null Does Not Properly Set the Length of the Values Buffer #​8900 [arrow]
• Struct casting requires same order of fields #​8870 [arrow]
• Cannot cast string dictionary to binary view #​8841 [arrow]

Documentation updates:

• Add Union encoding documentation #​9102 [arrow] (EduardAkhmetshin)
• docs: fix misleading reserve documentation #​9076 (WaterWhisperer)
• Fix headers and empty lines in code examples #​9064 (EduardAkhmetshin)
• Add examples for min and max functions #​9062 (EduardAkhmetshin)
• Improve arrow-buffer documentation #​9020 [arrow] (alamb)
• Move examples in arrow-csv to docstrings, polish up docs #​9001 [arrow] (alamb)
• Add example of parsing field names as VariantPath #​8945 (alamb)
• Improve documentation for `prep_null_mask_flter #​8722 [arrow] (alamb)

Performance improvements:

• [parquet] Avoid a clone while resolving the read strategy #​9056 [parquet] (alamb)
• perf: improve performance of encoding GenericByteArray by 8% #​9054 [arrow] (rluvaton)
• Speed up unary not kernel by 50%, add BooleanBuffer::from_bitwise_unary #​8996 [arrow] (alamb)
• arrow-select: improve dictionary interleave fallback performance #​8978 [arrow] (asubiotto)
• Add special implementation for zip for Utf8View/BinaryView scalars #​8963 [arrow] (mkleen)
• arrow-select: implement specialized interleave_list #​8953 [arrow] (asubiotto)

Closed issues:

• impl Index for UnionFields #​8958 [arrow]

Merged pull requests:

• Add DataType::is_decimal #​9100 (AdamGS)
• feat(parquet): relax type compatility check in parquet ArrowWriter #​9099 [parquet] (gruuya)
• [Variant] Move ArrayVariantToArrowRowBuilder to variant_to_arrow #​9094 (liamzwbao)
• chore: increase row count and batch size for more deterministic tests #​9088 [arrow] (Weijun-H)
• Fix nullif kernel #​9087 [arrow] (alamb)
• Add FlightInfo::with_endpoints method #​9075 [arrow] [arrow-flight] (lewiszlw)
• chore: run validation when debug assertion enabled and not only for test #​9073 [arrow] (rluvaton)
• Minor: make it clear cache array reader is not cloning arrays #​9057 [parquet] (alamb)
• Minor: avoid clone in RunArray row decoding via buffer stealing #​9052 [arrow] (lyang24)
• Minor: avoid some clones when reading parquet #​9048 [parquet] (alamb)
• fix: don't generate nulls for Decimal128 and Decimal256 when field is non-nullable and have non-zero null_density #​9046 [arrow] (rluvaton)
• fix: Rows size should use capacity and not len #​9044 [arrow] (rluvaton)
• fix: integration / Archery test With other arrows container ran out of space #​9043 (lyang24)
• feat: add new try_append_value_n() function to GenericByteViewBuilder #​9040 [arrow] (lyang24)
• Rename fields in BooleanBuffer for clarity #​9039 [arrow] (alamb)
• Allocate buffers before work in boolean_kernels benchmark #​9035 [arrow] (alamb)
• Move RunArray::get_physical_indices to RunEndBuffer #​9027 [arrow] (lyang24)
• Improve RunArray documentation #​9019 [arrow] (Jefffrey)
• Add BooleanArray tests for null and slice behavior #​9013 [arrow] (UtkarshSahay123)
• feat: support array indices in VariantPath dot notation #​9012 (foskey51)
• arrow-cast: Bring back in-order field casting for StructArray #​9007 [arrow] (brancz)
• arrow-ipc: Add ListView support #​9006 [arrow] (brancz)
• Add quote style to csv writer #​9004 [arrow] (xanderbailey)
• Fix row slice bug in Union column decoding with many columns #​9000 [arrow] (friendlymatthew)
• build(deps): bump actions/download-artifact from 6 to 7 #​8995 (dependabot[bot])
• minor: Add comment blocks to PR template #​8994 (Jefffrey)
• Implement BinaryArrayType for &FixedSizeBinaryArrays #​8993 [arrow] (Jefffrey)
• feat: impl BatchCoalescer::push_batch_with_indices #​8991 [arrow] (ClSlaid)
• [Arrow]Configure max deduplication length for StringView #​8990 [arrow] (lichuang)
• feat: implement append_array for FixedSizeBinaryBuilder #​8989 [arrow] (ClSlaid)
• Add benchmarks for Utf8View scalars for zip #​8988 [arrow] (mkleen)
• build(deps): bump actions/cache from 4 to 5 #​8986 (dependabot[bot])
• Take fsb null indices #​8981 [arrow] (alamb)
• Add List to interleave_kernels benchmark #​8980 [arrow] (alamb)
• Fix ipc errors for LargeList containing sliced StringViews #​8979 [arrow] (fabianmurariu)
• arrow-buffer: implement num-traits numeric operations #​8977 [arrow] (theirix)
• Update xz crate dependency to use liblzma in arrow-avro #​8975 [arrow] (jecsand838)
• arrow-data: avoid allocating in get_last_run_end #​8974 [arrow] (asubiotto)
• Support for Arc<str> in ParquetRecordWriter derive macro #​8973 (heilhead)
• feat: support casting Time32 to Int64 #​8971 [arrow] (tshauck)
• arrow-buffer: add i256::trailing_zeros #​8969 [arrow] (theirix)
• Perf: Vectorize check_bounds(2x speedup) #​8966 [arrow] (gstvg)
• arrow-buffer: make i256::leading_zeros public and tested #​8964 [arrow] (theirix)
• Add ignore leading and trailing white space to csv parser #​8960 [arrow] (xanderbailey)
• Access UnionFields elements by index #​8959 [arrow] (friendlymatthew)
• Add Parquet roundtrip benchmarks #​8956 [parquet] (etseidl)
• [Variant] Add variant to arrow for Date64/Timestamp(Second/Millisecond)/Time32/Time64 #​8950 (klion26)
• Let ArrowArrayStreamReader handle schema with attached metadata + do schema checking #​8944 [arrow] (jonded94)
• Adds ExtensionType for Parquet geospatial WKB arrays #​8943 [parquet] (BlakeOrth)
• Add builder to help create Schemas for shredding (ShreddedSchemaBuilder) #​8940 (XiangpengHao)
• build(deps): update criterion requirement from 0.7.0 to 0.8.0 #​8939 (dependabot[bot])
• fix: Resolve Avro RecordEncoder bugs related to nullable Struct fields and Union type ids #​8935 [arrow] (jecsand838)
• Some panic!s could more semantically be unimplemented! #​8933 [arrow] (abacef)
• fix: ipc decode panic with invalid data #​8931 [arrow] (leiysky)
• Allow creating zero-sized FixedSizeBinary arrays #​8927 [arrow] (tobixdev)
• Update test_variant_get_error_when_cast_failure... tests to uses a valid VariantArray #​8921 (alamb)
• Make flight sql client generic #​8915 [arrow] [arrow-flight] (lewiszlw)
• [minor] Name Magic Number "8" in FixedSizeBinaryArray::new_null #​8914 [arrow] (tobixdev)
• fix: cast Binary/String dictionary to view #​8912 [arrow] (Jefffrey)
• [8910]Fixed doc test with feature prettyprint #​8911 [arrow] (manishkr)
• feat: ArrayData::new_null for ListView / LargeListView #​8909 [arrow] (dqkqd)
• fead: add GenericListViewArray::from_iter_primitive #​8907 [arrow] (dqkqd)
• fix: GenericListViewArray::new_null returns empty array #​8905 [arrow] (dqkqd)
• Allocate a zeroed buffer for FixedSizeBinaryArray::null #​8901 [arrow] (tobixdev)
• build(deps): bump actions/checkout from 5 to 6 #​8899 (dependabot[bot])
• Add getters to UnionFields #​8895 [arrow] (friendlymatthew)
• Add validated constructors for UnionFields #​8891 [arrow] [arrow-flight] (friendlymatthew)
• Add bit width check #​8888 [parquet] (rambleraptor)
• [Variant] Improve variant_get performance on a perfect shredding #​8887 (XiangpengHao)
• Add UnionArray::fields #​8884 [arrow] (friendlymatthew)
• Struct casting field order #​8871 [arrow] (brancz)
• Add support for Union types in RowConverter #​8839 [arrow] (friendlymatthew)
• Add comparison support for Union arrays #​8838 [arrow] (friendlymatthew)
• [Variant] Support array shredding into List/LargeList/ListView/LargeListView #​8831 (liamzwbao)
• Add support for using ListView arrays and types through FFI #​8822 [arrow] (AdamGS)
• Add ability to skip or transform page encoding statistics in Parquet metadata #​8797 [parquet] (etseidl)
• Implement a Vec<RecordBatch> wrapper for pyarrow.Table convenience #​8790 (jonded94)
• Make Parquet SBBF serialize/deserialize helpers public for external reuse #​8762 [parquet] (RoseZhang123)
• Add cast support for (Large)ListView <-> (Large)List #​8735 [arrow] (vegarsti)

* This Changelog was automatically generated by github_changelog_generator

v57.1.0

Compare Source

Full Changelog

Breaking changes:

• Seal Array trait #​9092 [arrow] (tustvold)
• [Variant] Unify the CastOptions usage in parquet-variant-compute #​8984 (klion26)

Implemented enhancements:

• [parquet] further relax LevelInfoBuilder::types_compatible for ArrowWriter #​9098
• Update arrow-row documentation with Union encoding #​9084
• Add code examples for min and max compute functions #​9055
• Add append_n to bytes view builder API #​9034 [arrow]
• Move RunArray::get_physical_indices to RunEndBuffer #​9025 [arrow]
• Allow quote style in csv writer #​9003 [arrow]
• IPC support for ListView #​9002 [arrow]
• Implement BinaryArrayType for &FixedSizeBinaryArrays #​8992 [arrow]
• arrow-buffer: implement num-traits for i256 #​8976 [arrow]
• Support for Arc<str> in ParquetRecordWriter derive macro #​8972
• [arrow-avro] suggest switching from xz to liblzma #​8970 [arrow]
• arrow-buffer: add i256::trailing_zeros #​8968 [arrow]
• arrow-buffer: make i256::leading_zeros public #​8965 [arrow]
• Add spark like ignoreLeadingWhiteSpace and ignoreTrailingWhiteSpace options to the csv writer #​8961 [arrow]
• Add round trip benchmark for Parquet writer/reader #​8955 [parquet]
• Support performant interleave for List/LargeList #​8952 [arrow]
• [Variant] Support array access when parsing VariantPath #​8946
• Some panic!s could be represented as unimplemented!s #​8932 [arrow]
• [Variant] easier way to construct a shredded schema #​8922
• Support DataType::ListView and DataType::LargeListView in ArrayData::new_null #​8908 [arrow]
• Add GenericListViewArray::from_iter_primitive #​8906 [arrow]
• [Variant] Unify the cast option usage in ParquentVariant #​8873
• Blog post about efficient filter representation in Parquet filter pushdown #​8843 [parquet]
• Add comparison support for Union arrays in the cmp kernel #​8837 [arrow]
• [Variant] Support array shredding into List/LargeList/ListView/LargeListView #​8830
• Support Union data types for row format #​8828 [arrow]
• FFI support for ListView #​8819 [arrow]
• [Variant] Support more Arrow Datatypes from Variant primitive types #​8805
• FixedSizeBinaryBuilder supports append_array #​8750 [arrow]
• Implement special case zip with scalar for Utf8View #​8724 [arrow]
• [geometry] Wire up arrow reader/writer for GEOMETRY and GEOGRAPHY #​8717 [parquet]

Fixed bugs:

• Soundness Bug in try_binary when Array is implemented incorrectly in external crate #​9106
• casting Dict(_, LargeUtf8) to Utf8View (StringViewArray) panics #​9101
• wrong results for null count of nullif kernel #​9085 [parquet] [arrow]
• Empty first line in some code examples #​9063
• GenericByteViewArray::slice is not zero-copy but ought to be #​9014
• Regression in struct casting in 57.2.0 (not yet released) #​9005 [arrow]
• Fix panic when decoding multiple Union columns in RowConverter #​8999 [arrow]
• take_fixed_size_binary Does Not Consider NULL Indices #​8947 [arrow]
• [arrow-avro] RecordEncoder Bugs #​8934 [arrow]
• FixedSizeBinaryArray::try_new(...) Panics with Item Length of Zero #​8926 [arrow]
• cargo test -p arrow-cast fails on main #​8910 [arrow]
• GenericListViewArray::new_null ignores len and returns an empty array #​8904 [arrow]
• FixedSizeBinaryArray::new_null Does Not Properly Set the Length of the Values Buffer #​8900 [arrow]
• Struct casting requires same order of fields #​8870 [arrow]
• Cannot cast string dictionary to binary view #​8841 [arrow]

Documentation updates:

• Add Union encoding documentation #​9102 [arrow] (EduardAkhmetshin)
• docs: fix misleading reserve documentation #​9076 (WaterWhisperer)
• Fix headers and empty lines in code examples #​9064 (EduardAkhmetshin)
• Add examples for min and max functions #​9062 (EduardAkhmetshin)
• Improve arrow-buffer documentation #​9020 [arrow] (alamb)
• Move examples in arrow-csv to docstrings, polish up docs #​9001 [arrow] (alamb)
• Add example of parsing field names as VariantPath #​8945 (alamb)
• Improve documentation for `prep_null_mask_flter #​8722 [arrow] (alamb)

Performance improvements:

• [parquet] Avoid a clone while resolving the read strategy #​9056 [parquet] (alamb)
• perf: improve performance of encoding GenericByteArray by 8% #​9054 [arrow] (rluvaton)
• Speed up unary not kernel by 50%, add BooleanBuffer::from_bitwise_unary #​8996 [arrow] (alamb)
• arrow-select: improve dictionary interleave fallback performance #​8978 [arrow] (asubiotto)
• Add special implementation for zip for Utf8View/BinaryView scalars #​8963 [arrow] (mkleen)
• arrow-select: implement specialized interleave_list #​8953 [arrow] (asubiotto)

Closed issues:

• impl Index for UnionFields #​8958 [arrow]

Merged pull requests:

• Add DataType::is_decimal #​9100 (AdamGS)
• feat(parquet): relax type compatility check in parquet ArrowWriter #​9099 [parquet] (gruuya)
• [Variant] Move ArrayVariantToArrowRowBuilder to variant_to_arrow #​9094 (liamzwbao)
• chore: increase row count and batch size for more deterministic tests #​9088 [arrow] (Weijun-H)
• Fix nullif kernel #​9087 [arrow] (alamb)
• Add FlightInfo::with_endpoints method #​9075 [arrow] [arrow-flight] (lewiszlw)
• chore: run validation when debug assertion enabled and not only for test #​9073 [arrow] (rluvaton)
• Minor: make it clear cache array reader is not cloning arrays #​9057 [parquet] (alamb)
• Minor: avoid clone in RunArray row decoding via buffer stealing #​9052 [arrow] (lyang24)
• Minor: avoid some clones when reading parquet #​9048 [parquet] (alamb)
• fix: don't generate nulls for Decimal128 and Decimal256 when field is non-nullable and have non-zero null_density #​9046 [arrow] (rluvaton)
• fix: Rows size should use capacity and not len #​9044 [arrow] (rluvaton)
• fix: integration / Archery test With other arrows container ran out of space #​9043 (lyang24)
• feat: add new try_append_value_n() function to GenericByteViewBuilder #​9040 [arrow] (lyang24)
• Rename fields in BooleanBuffer for clarity #​9039 [arrow] (alamb)
• Allocate buffers before work in boolean_kernels benchmark #​9035 [arrow] (alamb)
• Move RunArray::get_physical_indices to RunEndBuffer #​9027 [arrow] (lyang24)
• Improve RunArray documentation #​9019 [arrow] (Jefffrey)
• Add BooleanArray tests for null and slice behavior #​9013 [arrow] (UtkarshSahay123)
• feat: support array indices in VariantPath dot notation #​9012 (foskey51)
• arrow-cast: Bring back in-order field casting for StructArray #​9007 [arrow] (brancz)
• arrow-ipc: Add ListView support #​9006 [arrow] (brancz)
• Add quote style to csv writer #​9004 [arrow] (xanderbailey)
• Fix row slice bug in Union column decoding with many columns #​9000 [arrow] (friendlymatthew)
• build(deps): bump actions/download-artifact from 6 to 7 #​8995 (dependabot[bot])
• minor: Add comment blocks to PR template #​8994 (Jefffrey)
• Implement BinaryArrayType for &FixedSizeBinaryArrays #​8993 [arrow] (Jefffrey)
• feat: impl BatchCoalescer::push_batch_with_indices #​8991 [arrow] (ClSlaid)
• [Arrow]Configure max deduplication length for StringView #​8990 [arrow] (lichuang)
• feat: implement append_array for FixedSizeBinaryBuilder #​8989 [arrow] (ClSlaid)
• Add benchmarks for Utf8View scalars for zip #​8988 [arrow] (mkleen)
• build(deps): bump actions/cache from 4 to 5 #​8986 (dependabot[bot])
• Take fsb null indices #​8981 [arrow] (alamb)
• Add List to interleave_kernels benchmark #​8980 [arrow] (alamb)
• Fix ipc errors for LargeList containing sliced StringViews #​8979 [arrow] (fabianmurariu)
• arrow-buffer: implement num-traits numeric operations #​8977 [arrow] (theirix)
• Update xz crate dependency to use liblzma in arrow-avro #​8975 [arrow] (jecsand838)
• arrow-data: avoid allocating in get_last_run_end #​8974 [arrow] (asubiotto)
• Support for Arc<str> in ParquetRecordWriter derive macro #​8973 (heilhead)
• feat: support casting Time32 to Int64 #​8971 [arrow] (tshauck)
• arrow-buffer: add i256::trailing_zeros #​8969 [arrow] (theirix)
• Perf: Vectorize check_bounds(2x speedup) #​8966 [arrow] (gstvg)
• arrow-buffer: make i256::leading_zeros public and tested #​8964 [arrow] (theirix)
• Add ignore leading and trailing white space to csv parser #​8960 [arrow] (xanderbailey)
• Access UnionFields elements by index #​8959 [arrow] (friendlymatthew)
• Add Parquet roundtrip benchmarks #​8956 [parquet] (etseidl)
• [Variant] Add variant to arrow for Date64/Timestamp(Second/Millisecond)/Time32/T

---

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Enabled.

♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

---

• If you want to rebase/retry this PR, check this box

---

This PR has been generated by Renovate Bot.