Retrieving Event Statistics

[Kkoile]

We want to apply more sophisticated autoscaling of our worker instances based on the number of open and currently running events.

Unfortunately, looping through all tenants and looking into the event queue table on db is not sufficient, since we have a lot of tenants.

I guess you already have some kind of in memory queue, since you limit how many events are processed in parallel. Is it somehow possible to expose the numbers? We are interested in the amount of currently running events and especially open events which wait to be processed.

Thanks for your input.

[soccermax]

@Kkoile
yes you can access this currently via:

const eventQueue = require("@cap-js-community/event-queue");
eventQueue.WorkerQueue.instance;

// getter functions of worker queue
//  get queue() {
//    return this.#queue;
//  }
//
//  get runningLoad() {
//    return this.#runningLoad;
//  }

It's not perfect but you can see if you can work with this. However in the long run I would still wish for a solution where some central statistic is hold in redis cross all tenants to get this kind of data. Would that be something that would help you?

[Kkoile]

that's great! Do you think it would be possible to report these numbers as a dynatrace metric? If I understand it correctly, you already use open telemetry. So it would be very cool, if you expose it as a metric and we see it in dynatrace for our worker instances :-)

[soccermax]

Yes that should be possible. One thing to keep in mind is that the worker queue contains the same event in all instances. So meaning it does not represent the unique count of open events. So the long term solution would be to keep statistics in redis about open events and also for which tenant. But for a short term solution we can test to expose the current worker pool metrics via openTelemetry to DT and see if that already helps in your use case :)