High frequency messages are missed [4.1.1]

[pmckenna555]

Hi, my SSE server emits a steady stream of events but will occasionally emit high frequency bursts. I have realised that when my SSE server emits events at a very high frequency that the SSE client misses these events.

if I put a sleep on my SSE server between each message emit then all the messages are received but I want to avoid this. Is this a bug or am I just missing a setting? I do not get any missed messages when consuming the events via a browser or terminal.

    fun subscribeToStream(streamId: String?, eventHandler: TestEventHandler?) {
        val streamURL = "$apiUrl/stream/$streamId"

        val httpConnectStrategy = HttpConnectStrategy.http(URI.create(streamURL))
        httpConnectStrategy
            .readTimeout(0, TimeUnit.SECONDS)
            .connectTimeout(0, TimeUnit.SECONDS) // no timeout
            .writeTimeout(0, TimeUnit.SECONDS)

        val builder = Builder(httpConnectStrategy)

        builder.retryDelay(3, TimeUnit.SECONDS);

        val backgroundEventSource = BackgroundEventSource.Builder(eventHandler, builder)
            .build()

        backgroundEventSource.start();
    }    

[kinyoklion]

Hello @pmckenna555,

Thank you for reporting the issue. Do you have any detail of the frequency and size for which you have issues? A selection of messages would be best if you can share them.

Aside from that I will see if I can reproduce any similar issue, and if so track down the cause.

Thank you,
Ryan

Filed internally as 234810

[pmckenna555]

Hey, thanks for getting back.

The high frequency events are sent as quick as possible with 0 delay between each and a simple method call to generate the message. It takes under 5 ms to generate 5000 of these messages within java (no SSE/network used). The messages I have been using are quite minimal.

    {
        "entity_name": "Customer",
        "data": {
            "id": 1
        }
    },
    {
        "entity_name": "Customer",
        "data": {
            "id": 2
        }
    },
    {
        "entity_name": "Customer",
        "data": {
            "id": 3
        }
    },
    {
        "entity_name": "Customer",
        "data": {
            "id": 4
        }
    },
    {
        "entity_name": "Customer",
        "data": {
            "id": 5
        }
    },
    {
        "entity_name": "Customer",
        "data": {
            "id": 6
        }
    },
    {
        "entity_name": "Customer",
        "data": {
            "id": 7
        }
    },
    {
        "entity_name": "Customer",
        "data": {
            "id": 8
        }
    },
    {
        "entity_name": "Customer",
        "data": {
            "id": 9
        }
    },
    {
        "entity_name": "Customer",
        "data": {
            "id": 10
        }
    },
    {
        "entity_name": "Customer",
        "data": {
            "id": 11
        }
    },
    {
        "entity_name": "Customer",
        "data": {
            "id": 12
        }
    },
    {
        "entity_name": "Customer",
        "data": {
            "id": 13
        }
    },
    {
        "entity_name": "Customer",
        "data": {
            "id": 14
        }
    },
    {
        "entity_name": "Customer",
        "data": {
            "id": 15
        }
    },
    {
        "entity_name": "Customer",
        "data": {
            "id": 16
        }
    },
    {
        "entity_name": "Customer",
        "data": {
            "id": 17
        }
    },
    {
        "entity_name": "Customer",
        "data": {
            "id": 18
        }
    },
    {
        "entity_name": "Customer",
        "data": {
            "id": 19
        }
    },
    {
        "entity_name": "Customer",
        "data": {
            "id": 20
        }
    },
    {
        "entity_name": "Customer",
        "data": {
            "id": 21
        }
    },
    {
        "entity_name": "Customer",
        "data": {
            "id": 22
        }
    },
    {
        "entity_name": "Customer",
        "data": {
            "id": 23
        }
    },
    {
        "entity_name": "Customer",
        "data": {
            "id": 24
        }
    },
    {
        "entity_name": "Customer",
        "data": {
            "id": 25
        }
    },
    {
        "entity_name": "Customer",
        "data": {
            "id": 26
        }
    },
    {
        "entity_name": "Customer",
        "data": {
            "id": 27
        }
    },
    {
        "entity_name": "Customer",
        "data": {
            "id": 28
        }
    }

[kinyoklion]

Hey @pmckenna555

Sorry for the delayed response, but we have been unable to reproduce this issue. We tried a number of things, sending hundreds of thousands of messages in rapid succession and didn't experience any problems receiving all messages. We also generally test messages spanning different numbers of chunks, and ran some specific tests with a rolling window of chunk overlap and didn't experience problems.

It is possible that it is something related to the specific way these messages are being sent. So if you could detail how that is implemented, technology/library/etc, then that could be helpful. In regards to a message sampling, if you could get the utf-8 that was sent over the wire when messages were missed (raw bytes is probably best), then we could feed that into the implementation for validation.

Thank you,
Ryan