feat(new sink): new gcp_bigquery sink

[james9909]

Summary

This change adds a BigQuery sink that uses the Write API to log events with the default stream.

As called out in the original PR (#18886), future work could add support for the three other application-created write workflows (pending, committed, buffered) and could further optimize the write path.

Vector configuration

Source

"socket_input": {
  "decoding": {
    "codec": "json"
  },
  "framing": {
    "method": "bytes"
  },
  "log_namespace": true,
  "mode": "unix_datagram",
  "path": "/tmp/vector.sock",
  "socket_file_mode": 666,
  "type": "socket"
}

Sink

"bigquery_logs": {
  "batch": {
    "max_bytes": 9000000,
    "max_events": 10000,
    "timeout_secs": 1
  },
  "buffer": {
    "max_size": 2000000000,
    "type": "disk",
    "when_full": "drop_newest"
  },
  "dataset": "mydataset",
  "encoding": {
    "protobuf": {
      "desc_file": "/myproto.desc",
      "message_type": "myproto.MyMessage"
    }
  },
  "inputs": [
    "socket_input"
  ],
  "project": "myproject",
  "request": {
    "retry_max_duration_secs": 30
  },
  "table": "mytable",
  "type": "gcp_bigquery"
}

How did you test this PR?

PR is tested with an integration test using an emulator and with a real BigQuery stream. I have also tested this sink in real production systems handling tens of thousands of requests per second.

Testing against real BigQuery:

TEST_GCP_BIGQUERY_WRITE_STREAM="projects/my-project/datasets/my-dataset/tables/my-table/streams/_default" \
GOOGLE_APPLICATION_CREDENTIALS=/path/to/key.json \
cargo test \
    --no-default-features \
    --features gcp-bigquery-integration-tests \
    --lib sinks::gcp::bigquery::integration_tests::gcp_bigquery_sink \
    -- --nocapture

Change Type

• Bug fix
• New feature
• Dependencies
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

• Closes: Add a blackhole sink #246
• Related: feat(new sink): new gcp_bigquery sink #18886

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[james9909]

I have read the CLA Document and I hereby sign the CLA

[james9909]

hi @pront, I've taken a stab at updating the original bigquery PR here!

[rtrieu]

one minor find