feat(codecs): Support more avro types

[omwbennett]

Summary

Addresses #24773 by adding support for decoding Avro Date values, in addition to Fixed, TimeMillis, TimestampNanos, LocalTimestampNanos types.
Due to the way VRL stores integers, an extra step was added to the avro encoder path to coerce Avro types based on the schema field's LogicalType.

Vector configuration

sources:
  kafka_avro:
    type: kafka
    bootstrap_servers: "kafka:9092"
    group_id: "vector"
    topics:
      - test_topic
    decoding:
      codec: avro
      avro:
        schema: |
          {
            "type": "record",
            "name": "test",
            "fields": [
              {
                "name": "date_field",
                "type": { "type": "int", "logicalType": "date" }
              }
            ]
          }
        strip_schema_id_prefix: false

sinks:
  console_out:
    type: console
    inputs:
      - kafka_avro
    encoding:
      codec: json

Also used similar configs for the other Avro data types.

How did you test this PR?

Change Type

• Bug fix
• [x New feature
• Dependencies
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

Closes #24773

Notes

• Please read our Vector contributor resources.
• Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
• Some CI checks run only after we manually approve them.
  • We recommend adding a pre-push hook, please see this template.
  • Alternatively, we recommend running the following locally before pushing to the remote branch:
    • make fmt
    • make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
    • make test
• After a review is requested, please avoid force pushes to help us review incrementally.
  • Feel free to push as many commits as you want. They will be squashed into one before merging.
  • For example, you can run git merge origin master and git push.
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
  run make build-licenses to regenerate the license inventory and commit the changes (if any). More details here.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[omwbennett]

I have read the CLA Document and I hereby sign the CLA

[omwbennett]

Hey @vectordotdev/vector, appreciate if you could review this PR, thanks!

[pront]

@codex review

Hi @omwbennett, thanks for contributing to Vector. I enabled the workflows and requested a first pass from codex.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 857b4b3903

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve logicalType in date fixture schema

roundtrip_avro_fixtures reads this .avsc as the schema, but the new fixture stores date_field as plain int ({"type":{"type":"int"}}) rather than an Avro date logical type. That means decoding this fixture produces AvroValue::Int, so the new AvroValue::Date branch added in try_from is never exercised by tests; regressions in real logicalType: "date" inputs could pass CI unnoticed. Please keep the fixture schema with logicalType: "date" on the field type object so the test validates the intended path.

Useful? React with 👍 / 👎.

[pront]

@omwbennett this seems like a valid comment

[omwbennett]

This seems to be because the apache-avro dependency was updated to 0.20.0 which changed/fixed canonical_form() to remove logicalTypes, however the avro test fixtures were not re-generated until now.

I've updated the fixture generator to pass the schema as-is into the .avsc fixture file so we don't lose any parts of the schema.

This uncovered a bug in the Avro encoder where it's unable to encode a vrl Integer back to an Avro Date, so I've addressed this by making the encoder aware of the schema using a new method to_avro() in the encode path that maps different vrl values to Avro types based on the schema. I also went ahead and added support for a few other types as those looked straightforward.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cb2cbd735b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5336c62721

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[pront]

Hi @omwbennett, happy to review this once all codex comments are addressed (feel free to decline those with a justification if they do not make sense)

[omwbennett]

Hi @omwbennett, happy to review this once all codex comments are addressed (feel free to decline those with a justification if they do not make sense)

Thanks @pront, I have addressed all the codex review comments now, appreciate if you could take another look.