Skip to content

feat(aws_s3 sink): Add Apache Parquet encoding support #24706

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
szibis wants to merge 28 commits into from
Closed

feat(aws_s3 sink): Add Apache Parquet encoding support #24706

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
aebf74e
feat(codecs): add parquet crate dependency for columnar encoding
szibis Feb 21, 2026
66cbdec
feat(codecs): add Parquet batch serializer with compression and schem…
szibis Feb 21, 2026
4835a34
feat(aws_s3 sink): add Parquet batch encoding support
szibis Feb 21, 2026
ac0d96f
feat(codecs): add TOML-configurable schema for Parquet encoding
szibis Feb 21, 2026
f9b8a19
docs: add changelog entry for Parquet encoding in S3 sink
szibis Feb 21, 2026
7384ddf
style: fix clippy warnings and apply rustfmt
szibis Feb 21, 2026
84723c9
test: add Parquet integration test and benchmarks
szibis Feb 21, 2026
19a1e69
bench: add NDJSON+Snappy benchmark and fix ClickHouse match exhaustiv…
szibis Feb 21, 2026
4278868
perf(codecs): optimize Parquet serializer hot path
szibis Feb 21, 2026
2d882ff
docs: add CUE docs, configurable metadata, and YAML config example
szibis Feb 21, 2026
729b2ca
ci: add Trino to spelling allow list
szibis Feb 21, 2026
1245878
fix(codecs): strict schema mode and CUE doc corrections for Parquet
szibis Mar 2, 2026
2da70e2
feat(codecs): multi-schema Parquet encoding with edge case hardening
szibis Mar 12, 2026
e4acc53
docs(aws_s3): document all 6 Parquet schema input options in CUE
szibis Mar 12, 2026
15224cd
feat(codecs): add struct, list, map types to inline Parquet schema
szibis Mar 12, 2026
1a54bad
fix(docs): replace mypackage with logs in Protobuf schema example
szibis Mar 13, 2026
f437b16
fix(codecs): reject binary fields at config time in Parquet encoder
szibis Mar 14, 2026
6a6ce06
fix(aws_s3): auto-detect Content-Type and fix CUE docs for Parquet ba…
szibis Mar 15, 2026
5b4b3c7
test(aws_s3): add Parquet batch encoding config unit tests
szibis Mar 15, 2026
86db76f
fix(codecs): add missing Parquet content_type and update Cargo.lock a…
szibis Mar 19, 2026
35dffb5
fix: clarify test comment — timestamps null for null-handling test, n…
szibis Apr 3, 2026
39abe53
Include auto infer schema mode
petere-datadog Apr 7, 2026
7a0d2ef
added chrono-tz dependency
petere-datadog Apr 8, 2026
3e69504
Add auto infer schema tests
petere-datadog Apr 8, 2026
0d49117
Use a loop instead of recursion in next function to prevent stack ove…
petere-datadog Apr 8, 2026
4ce1f58
Prevent double serialization during encoding with auto infer schema mode
petere-datadog Apr 8, 2026
28fd3d2
rewrite try_normalize_schema to only need one passthrough.
petere-datadog Apr 9, 2026
2fb27c1
Merge pull request #1 from petere-datadog/parquet-encoding-v1
szibis Apr 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/actions/spelling/expect.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -603,6 +603,7 @@ transitioning
Trauring
Treemap
trialled
Trino
Trivago
trivy
trl
Expand Down
7 changes: 7 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ cargo-timing*.html
target
tests/data/wasm/*/target
bench_output.txt
rust-analyzer.toml

# Python
*.pyc
Expand Down Expand Up @@ -64,3 +65,9 @@ volumes/

# LLM tools
copilot-instructions.md

# local files not checked in
local/

# vscode
.vscode/
Loading
Loading