EventStream evaluation `DEADLINE_EXCEEDED` shouldn't be an error

[julianocosta89]

I may be wrong here, but I'd like to raise this to have a broader discussion.

On the OTel Demo we have a couple of services using flagd.evaluation.v*.Service/EventStream.
This creates a new span every X minutes.

• In Python we have spans that are 30 minutes long:
  • 
• In Java we have 10 minutes long spans:
  • 
• In Javascript we have 10 minutes long spans, all on the same trace:
  • 
  • This was reported here: EventStream flagd.evaluation.v2.Service not clearing trace context js-sdk-contrib#1522

The main issue to me is the span status ERROR.

If this connection is expected to end and reconnect, I don't think the span should be marked as error.

[toddbaert]

Hey @julianocosta89 , AFAIK, this is expected behavior from gRPC. Deadlines are actually recommended for streams (see https://grpc.io/docs/guides/deadlines/), and DEADLINE_EXCEEDED is the natural result when one fires. The stream reconnects and everything continues as normal.

So the "error" here is really just OTel gRPC instrumentation treating any non-OK gRPC status as a span error (which is reasonable from OTel's perspective). To be honest, DEADLINE_EXCEEDED always seemed strange to have as non-OK in the design of gRPC for the reasons above - it's non-exceptional. WDYT is the right fix? Should the providers suppress the error status for DEADLINE_EXCEEDED on streaming calls, or is this better handled at the OTel instrumentation layer? Is there perhaps some kind of OTel setting to avoid treating this as error?

(By they way, I moved this to flagd, because this is an issue specific to flagd (an OpenFeature implementation) not the OpenFeature spec itself). flagd is to OpenFeature what something like Jaeger is to Otel.

[Rul1an]

This seems like a good place to separate two things that often get collapsed in CI:

1. what the telemetry transport reports
2. what actually happened to the evaluated flag decision

If DEADLINE_EXCEEDED is expected for this stream lifecycle, then marking the span as an error may still be technically explainable, but it makes the CI/review surface noisy. The useful question becomes: did the stream reconnect, and did any evaluation result actually regress?

We have been using that split in Assay Harness for the OpenFeature path:

EvaluationDetails/result boundary -> decision receipt -> verified bundle -> Trust Basis diff -> Markdown/JUnit projection

So telemetry can stay operational, while CI reviewers get a smaller artifact that says what decision boundary changed.

Released recipe:
https://github.com/Rul1an/Assay-Harness/blob/v0.3.2/docs/OPENFEATURE_DECISION_RECEIPT_PIPELINE.md

[julianocosta89]

I'm by no means an expert on that topic in specific, but if something is expected, IMHO it is not an ERROR.
If OTel SemConv says it should be an ERROR, then we should discuss that in the SemConv.

[toddbaert]

I think it's less about OTel and more about gRPC - gRPC itself classifies this as a non-OK state, but also implies it's expected. OTel just goes along with that, which makes sense.

I really don't know what the best solution is here for us.

[Rul1an]

I think that split is probably the important design line here.

From a downstream consumer point of view, I would be careful not to let span status become the source of truth for flag-decision health. DEADLINE_EXCEEDED can be a transport-level non-OK result and still be part of a normal stream lifecycle if reconnect completed and evaluations kept working.

So the cleanest fix may not be to pretend the transport event did not happen, but to make the expected stream-rotation case distinguishable from a real broken stream. Then operational telemetry can stay honest, while reviewers and CI consumers do not have to infer evaluation regressions from a span error alone.

That keeps the two questions separate: did the stream lifecycle behave as expected, and did the evaluated flag result actually change.

[beeme1mr]

I'm by no means an expert on that topic in specific, but if something is expected, IMHO it is not an ERROR.
If OTel SemConv says it should be an ERROR, then we should discuss that in the SemConv.

Hey @julianocosta89, I'm wondering if it even makes sense for these long lived connections to even start a span. As far as I know, we're just using the standard gRPC sensor.

[dyladan]

The deadline is being used as a proxy for "is this stream still alive?" but it fires unconditionally, even when the stream is healthy. The real goal is to detect a hung stream:

• A stream with no recent events could be hung, or it could just have no flag changes to report
• HTTP/2 keepalive PINGs operate at the connection level, but multiple streams are multiplexed over a single connection, so a live connection doesn't guarantee a live stream. One connection could be serving both live and hung streams (in flagd case there is only 1, but in general).

Option A — Application-level heartbeat
flagd periodically sends a no-op/ping event on the stream. The SDK can then use absence of these heartbeats (rather than a deadline) to detect a hung stream. This would allow removing the deadline entirely and eliminate the spurious errors. Not sure exactly how the messages are formed and how the clients respond to them to determine if this could be done in a backwards compatible way.

Option A.2 — Application-level stream restarts
Create the request with a deadline of 10m on the client, and end the stream from the server after 8m with an OK status. Client re-establishes the connection as it already does, and there is no error code unless a stream actually hangs beyond its normal end time. 1 span per stream-segment is created using the existing OTel out of the box behavior.

Option B — Keep the deadline, suppress the noise in OTel
This would be handled on the OTel side, not here. But it's worth knowing that approach exists if Option A is a large lift.

Option C — Document and accept the current behavior
Add clear guidance that these reconnections are expected and show users how to filter them in their observability tooling.

One question we should be asking is what the telemetry would look like for flagd in an ideal world where the instrumentation was created specifically for this case. Do spans even make sense for server streams? If a stream never ends, the span will also never end and will never be exported by OpenTelemetry. flagd is using streaming responses as a way to make reverse-requests from the server to the client. From that perspective, it almost seems like the span should start when an event is received and end when it is finished processing. This span should actually be a child of a span on the server. Each event from the server kicks off some process on the client, but the current tracing doesn't show the causality that way at all. If that process did some async work what would its parent even be right now? It would probably be a sibling of the stream or even a root span, which is obviously not correct.

[Rul1an]

Yeah, this is close to how I’d slice it too.

To me the awkward part is that the deadline is doing two jobs at once: it is a liveness check, but it also produces the thing people see in tracing as the failure signal.

That makes the span error louder than it probably should be. The stream may have rotated exactly as designed, while the actual flag evaluation path stayed fine.

A planned restart with an OK end state feels cleaner to me for that reason. You still keep the operational fact visible: the stream segment ended and reconnected. But a real timeout still means something different.

I also agree that the more useful observability shape is probably closer to event-level. Stream telemetry can say whether the channel behaved. Each received flag or evaluation event can carry the smaller record people actually want to inspect later.

That keeps the transport story and the decision story from getting collapsed into one long-lived span.

[dyladan]

Created upstream semconv issues for 2 separate concerns:

1. It may not be appropriate to model gRPC streams as spans as they may never end. If the span never ends it is never flushed to exporters and provides little to no value. One possible solution is to end the span when the stream is established, then model each message separately as an event or a span. In the case of a server stream, the span on the client may even be a child of a span on the server. [gRPC] streaming request results in never-ending span open-telemetry/semantic-conventions#3703
2. If gRPC streams use a deadline the way flagd does to avoid hung streams, the stream will always show as many failed spans of whatever the deadline length is for the lifetime of the application. There should be a way to not treat these as failures because the gRPC community strongly encourages the use of deadlines on all requests (https://grpc.io/blog/deadlines/, https://grpc.io/docs/guides/keepalive/, https://grpc.io/docs/what-is-grpc/core-concepts/#deadlines, https://grpc.io/docs/guides/deadlines/#deadlines-on-the-client), which will always result in this behavior in a streaming service. Discussion: DEADLINE_EXCEEDED on streaming RPCs should not always be treated as a failure open-telemetry/semantic-conventions#3704

[julianocosta89]

@dyladan, really appreciate all the insights and the creation of the issue on the SemConv repo!
I'll continue the discussion over there