enhancement(codecs): replace serde_arrow with arrow-json

[benjamin-awd]

Summary

This PR replaces serde_arrow with arrow-json in the Arrow encoder - this removes a lot of the manual workarounds that we implemented i.e. custom timestamp conversion (which involves cloning LogEvents and looking for timestamp fields within the schema), post-processing and string-matching on error messages. arrow-json handles RFC 3339 timestamps natively and avoids using an extra dependency outside of the native arrow ecosystem.

Vector configuration

sinks:
  clickhouse:
    type: clickhouse
    inputs: [process]
    database: ...
    table: ...
    endpoint: ${CLICKHOUSE_HOST}:8123
    compression: zstd
    format: arrow_stream
    batch_encoding:
      codec: arrow_stream
      allow_nullable_fields: true

How did you test this PR?

Existing tests + ran locally

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

There are likely some differences, but

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

Related: #24124, #24409

Notes

• Please read our Vector contributor resources.
• Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
• Some CI checks run only after we manually approve them.
  • We recommend adding a pre-push hook, please see this template.
  • Alternatively, we recommend running the following locally before pushing to the remote branch:
    • make fmt
    • make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
    • make test
• After a review is requested, please avoid force pushes to help us review incrementally.
  • Feel free to push as many commits as you want. They will be squashed into one before merging.
  • For example, you can run git merge origin master and git push.
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
  run make build-licenses to regenerate the license inventory and commit the changes (if any). More details here.

[pront]

Thank you for addressing all review comments. The build_record_batch() implementation looks much better now!

[benjamin-awd]

The failing integration test is a bit of a weird one:

2026-02-23T16:39:49.215866Z ERROR codecs::internal_events: Schema constraint violation. error=Null value for non-nullable field 'required_field' error_code="encoding_null_constraint" error_type="encoder_failed" stage="sending"
2026-02-23T16:39:49.216060Z ERROR vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Schema constraint violation."
2026-02-23T16:39:49.216415Z ERROR vector::internal_events::common: Failed to build request. error=SchemaConstraintViolation(Null value for non-nullable field 'required_field') error_type="encoder_failed" stage="processing"

thread 'sinks::clickhouse::integration_tests::test_missing_required_field_emits_null_constraint_error' (36740) panicked at src/sinks/clickhouse/integration_tests.rs:1231:5:
assertion left == right failed
left: Ok(Delivered)
right: Ok(Rejected)

I could probably remove the test or modify it so that it doesn't check the BatchStatus, but that feels a bit hacky -- I suspect it's related to the emit call moving back to the encoder level.

Attempted a fix in 1426e1f -- let me know if that works.

[pront]

Checked this out locally and got:

error[E0433]: failed to resolve: use of unresolved module or unlinked crate `toml`
   --> lib/codecs/src/encoding/transformer.rs:303:13

[pront]

The failing integration test is a bit of a weird one:

2026-02-23T16:39:49.215866Z ERROR codecs::internal_events: Schema constraint violation. error=Null value for non-nullable field 'required_field' error_code="encoding_null_constraint" error_type="encoder_failed" stage="sending"
2026-02-23T16:39:49.216060Z ERROR vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Schema constraint violation."
2026-02-23T16:39:49.216415Z ERROR vector::internal_events::common: Failed to build request. error=SchemaConstraintViolation(Null value for non-nullable field 'required_field') error_type="encoder_failed" stage="processing"
thread 'sinks::clickhouse::integration_tests::test_missing_required_field_emits_null_constraint_error' (36740) panicked at src/sinks/clickhouse/integration_tests.rs:1231:5:
assertion left == right failed
left: Ok(Delivered)
right: Ok(Rejected)

I could probably remove the test or modify it so that it doesn't check the BatchStatus, but that feels a bit hacky -- I suspect it's related to the emit call moving back to the encoder level.

Attempted a fix in 1426e1f -- let me know if that works.

Hmm this requires some thought, I need to consider alternative designs. In the meantime, if you want to unblock this PR, you can create an issue for this (i.e. add ability to modify event finalizers when building requests), modify the test failing case and add this new issue link as a comment on the hacky test code.

[benjamin-awd]

Checked this out locally and got:

error[E0433]: failed to resolve: use of unresolved module or unlinked crate toml
--> lib/codecs/src/encoding/transformer.rs:303:13

I think this might be a pre-existing bug, I can reproduce by running cargo test -p codecs against master

[pront]

Checked this out locally and got:
error[E0433]: failed to resolve: use of unresolved module or unlinked crate toml
--> lib/codecs/src/encoding/transformer.rs:303:13

I think this might be a pre-existing bug, I can reproduce by running cargo test -p codecs against master

We submitted a fix: #24766
This PR can go in after the fix is merged.