Initial implementation of stateful gRPC stream for Logs agent by joyzhang-dd · Pull Request #42732 · DataDog/datadog-agent

joyzhang-dd · 2025-11-05T07:35:10Z

Initial implementation of stateful gRPC stream for Logs agent
Hooks gRPC stream to Logs agent pipeline

What does this PR do?

Motivation

Describe how you validated your changes

Additional Notes

agent-platform-auto-pr · 2025-11-05T07:55:53Z

Go Package Import Differences

Baseline: 5742833
Comparison: 1bf95e4

binary	os	arch	change
agent	linux	amd64	+6, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/automaton +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
agent	linux	arm64	+6, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/automaton +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
agent	windows	amd64	+6, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/automaton +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
agent	darwin	amd64	+6, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/automaton +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
agent	darwin	arm64	+6, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/automaton +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
iot-agent	linux	amd64	+6, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/automaton +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
iot-agent	linux	arm64	+6, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/automaton +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
heroku-agent	linux	amd64	+6, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/automaton +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
cluster-agent	linux	amd64	+6, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/automaton +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
cluster-agent	linux	arm64	+6, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/automaton +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
cluster-agent-cloudfoundry	linux	amd64	+1, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
cluster-agent-cloudfoundry	linux	arm64	+1, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
dogstatsd	linux	amd64	+1, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
dogstatsd	linux	arm64	+1, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
process-agent	linux	amd64	+1, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
process-agent	linux	arm64	+1, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
process-agent	windows	amd64	+1, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
process-agent	darwin	amd64	+1, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
process-agent	darwin	arm64	+1, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
heroku-process-agent	linux	amd64	+1, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
security-agent	linux	amd64	+7, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/automaton +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/DataDog/datadog-agent/pkg/trace/log
security-agent	linux	arm64	+7, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/automaton +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/DataDog/datadog-agent/pkg/trace/log
security-agent	windows	amd64	+7, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/automaton +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/DataDog/datadog-agent/pkg/trace/log
serverless	linux	amd64	+6, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/automaton +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
serverless	linux	arm64	+6, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/automaton +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
system-probe	linux	amd64	+7, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/automaton +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/DataDog/datadog-agent/pkg/trace/log
system-probe	linux	arm64	+7, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/automaton +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/DataDog/datadog-agent/pkg/trace/log
system-probe	windows	amd64	+7, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/automaton +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/DataDog/datadog-agent/pkg/trace/log
otel-agent	linux	amd64	+6, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/automaton +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
otel-agent	linux	arm64	+6, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/automaton +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb

Hooks gRPC stream to Logs agent pipeline

adapted the stateful transport implementation to import the new location

During flow-control/back-pressure, a slow/blocking Send won't block the supervisor loop

github-actions Bot added long review PR is complex, plan time to review it team/agent-log-pipelines labels Nov 5, 2025

joyzhang-dd force-pushed the joy.zhang/stateful-encoding-grpc-v2 branch 2 times, most recently from 8bb2b57 to a5526c1 Compare November 5, 2025 20:34

joyzhang-dd and others added 4 commits November 12, 2025 22:00

Initial implementation of stateful gRPC stream for Logs agent

9f98b91

Hooks gRPC stream to Logs agent pipeline

Handle state management and snapshot transmission

e0a7068

move stateful encoding proto to proper location

4177a6a

adapted the stateful transport implementation to import the new location

Implement backoff strategy at stream level covering common failures

bf41261

joyzhang-dd force-pushed the joy.zhang/stateful-encoding-grpc-v2 branch from 90e0d97 to bf41261 Compare November 14, 2025 15:15

modify proto

29feffc

joyzhang-dd force-pushed the joy.zhang/stateful-encoding-grpc-v2 branch from 888764c to 29feffc Compare November 14, 2025 19:31

joyzhang-dd and others added 2 commits November 15, 2025 22:08

Seperate out stream Send into its own goroutine

51c6b73

During flow-control/back-pressure, a slow/blocking Send won't block the supervisor loop

latest logs agent pattern and intake changes

1bf95e4

gh-worker-devtools-f43747 Bot closed this May 23, 2026

gh-worker-devtools-f43747 Bot deleted the joy.zhang/stateful-encoding-grpc-v2 branch May 23, 2026 03:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial implementation of stateful gRPC stream for Logs agent#42732

Initial implementation of stateful gRPC stream for Logs agent#42732
joyzhang-dd wants to merge 7 commits into
mainfrom
joy.zhang/stateful-encoding-grpc-v2

joyzhang-dd commented Nov 5, 2025

Uh oh!

agent-platform-auto-pr Bot commented Nov 5, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

joyzhang-dd commented Nov 5, 2025

What does this PR do?

Motivation

Describe how you validated your changes

Additional Notes

Uh oh!

agent-platform-auto-pr Bot commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Go Package Import Differences

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

agent-platform-auto-pr Bot commented Nov 5, 2025 •

edited

Loading