Kubernetes performance against rivals

[TheOnlyArtz]

Hey Platformatic! first of all I wanted to say thank you for this awesome job, always fun to see competition in the scene.

We are relying heavily on Nest.js kafka's integration in my workplace and we are looking to migrate over. I'm conducting some benchmarks and stress tests to see what suits us better in a heavy throughput environment which requires low latency, sounds like this library suits the needs.

I'm benchmarking between this library and confluent's and the benchmarks I'm seeing in this repository don't add up, maybe I'm doing something wrong?

I'm stress testing the environment by sending 8000 1KB messages as fast as I can from my computer, in 40 messages per batch.

const TOTAL_MESSAGES = 8000
const BATCH_SIZE = 40
const IN_FLIGHT = 100
const LOG_EVERY = 1000
const ACKS = 1
const COMPRESSION = 'none'

(the stress test is using kafka.js tho its irrelevant for this issue)

Test environment

• AWS Kafka instance
• Stress test runs on my computer to the AWS (no local)
• 2 replicas of the microservice in a kubernetes environment (with the same kafka group id to share compute)
• Replicas are limited to 512MB of resources.
• Node 24 alpine23 docker image
• 512MB Cap for the pods

platformatic/kafka

While processing the messages fast enough and catching up, after only 3 minutes of running the pods gained too much memory and eventually killed by the orchestrator:

This is the way I've configured the platformatic consumer and producer:

const KAFKA_REQUEST_TIMEOUT_MS = 60000
const KAFKA_CONNECTION_TIMEOUT_MS = 10000
const KAFKA_RETRY_INITIAL_TIME_MS = 300
const KAFKA_RETRY_COUNT = 8
const KAFKA_MAX_IN_FLIGHTS = 5

const options: KafkaClientOptions = {
  bootstrapBrokers: brokers,
  groupId: `${kafkaConfig.groupId}-group`,
  tls,
  requestTimeout: KAFKA_REQUEST_TIMEOUT_MS,
  connectTimeout: KAFKA_CONNECTION_TIMEOUT_MS,
  retries: KAFKA_RETRY_COUNT,
  retryDelay: KAFKA_RETRY_INITIAL_TIME_MS,
  maxInflights: KAFKA_MAX_IN_FLIGHTS,
  producerClientId: `${kafkaConfig}-producer`,
  consumerClientId: `${kafkaConfig}-consumer`,
  adminClientId: `${kafkaConfig}-admin`,
}

// producer factory
new Producer<string, string, string, string>({
  ...toKafkaBaseOptions(options, options.producerClientId),
  serializers: stringSerializers,
  idempotent: false,
})

// consumer factory
return new Consumer({
  ...toKafkaBaseOptions(options, options.consumerClientId),
  autocommit: 5000,
  groupId: options.groupId,
})

// stream
this.stream = await this.consumer.consume({
  topics,
  mode: MessagesStreamModes.LATEST,
  fallbackMode: MessagesStreamModes.LATEST,
})

// consuming messages using rxJs concurrency
private async consumeMessages(
  stream: MessagesStream<Buffer, Buffer, Buffer, Buffer>,
): Promise<void> {
  const processingStream$ = from(stream).pipe(
    takeWhile(() => !this.shuttingDown),

    mergeMap((message) => this.processMessage(message), 50)
  );

  await lastValueFrom(processingStream$, {defaultValue: undefined});
}

confluent/kafka-javascript (rdkafka-node)

I felt like the experience with confluent was so much smoother, I could run the stress test forever and these were the results (after 10 minutes)

---

the platformatic is configured the same way as the confluent's client.
I'm really leaning towards platformatic but the benchmarks don't add up and I would love your input on that, maybe I'm just using it wrong.

[mcollina]

Can you upload your benchmarks to a repository? It's hard to tell otherwise.

At first glance, I would recommend never using rxjs under any circumstances, as it's a significant bottleneck in every application that uses it. I don't know if you are using it with Confluent as well, but it could explain the difference.

[TheOnlyArtz]

Can you upload your benchmarks to a repository? It's hard to tell otherwise.

At first glance, I would recommend never using rxjs under any circumstances, as it's a significant bottleneck in every application that uses it. I don't know if you are using it with Confluent as well, but it could explain the difference.

Actually I'm not using it with confluent, I'm just now trying to use the for await, I'll update the results.

I'm benchmarking against one of our microservices, I will try to extract it to a minimal reproducible repository of course! thank your for your fast response.

[TheOnlyArtz]

So it seems like rxjs was not the bottleneck, I'm working on creating a reproducible example and I'll send the link

[TheOnlyArtz]

https://github.com/TheOnlyArtz/platformic-kafka-min-demo @mcollina This is the exact way I'm utilizing the platformatic.

[mcollina]

That does not seem to contain a comparison benchmark.

[TheOnlyArtz]

That does not seem to contain a comparison benchmark.

Right, but this is the way I'm utilizing the library, confluent aside , maybe you can pin point issue which can lead to bad performance including rebalancing and crashing on high throughput.

It is going to be hard for me to put on a benchmark, also the behavior seems to persist in the k8s and not when running locally, probably has to do with stronger hardware environment locally (also in Platformatic benches)

[kibertoad]

@TheOnlyArtz can't you just ask Claude, or Cursor, or anything similar to generate a benchmark? it does so quite well

[TheOnlyArtz]

@TheOnlyArtz can't you just ask Claude, or Cursor, or anything similar to generate a benchmark? it does so quite well

The thing is that I can't seem to reproduce it locally. I will try once again

[TheOnlyArtz]

https://github.com/TheOnlyArtz/platformic-kafka-min-demo/tree/both
@mcollina @kibertoad This is the benchmark I came up with, I could NOT reproduce whats happening on the regular full service and the k8s environment, but even in this benchmark running the stress test for 20 minutes straight, confluent still seems to do better (?)

CPU:

MEM:

Just so I make it clear the original service gets messages, processes them and echos them back to another topic in the kafka, the benchmarks try to mimic that

[mcollina]

From the diagrams you posted, there seems to be a different memory management pattern between the two: the confluent driver is accumulating memory, while ours is essentially flat. GC activity can also explain the difference in CPU consumption. What would worry me, and I would recommend observing for a longer time, is whether the Confluent memory stabilizes or not.

At this low load, it's very hard to draw any conclusions regarding the performance of either driver.

[TheOnlyArtz]

From the diagrams you posted, there seems to be a different memory management pattern between the two: the confluent driver is accumulating memory, while ours is essentially flat. GC activity can also explain the difference in CPU consumption. What would worry me, and I would recommend observing for a longer time, is whether the Confluent memory stabilizes or not.

At this low load, it's very hard to draw any conclusions regarding the performance of either driver.

Thanks for the input, I have the confluent driver running for a few hours now in staging environment doing the regular service' work with simulative data and the RAM is very stable

Even stress testing the RAM rises to around 310 which is suspected but it stays there stably, this is what I could not achieve with the platformatic driver.

From the platformatic code in the benchmark, have you seen some miss-use? maybe the configuration or message processing can be better ?

[mcollina]

Even stress testing the RAM rises to around 310 which is suspected but it stays there stably, this is what I could not achieve with the platformatic driver.

What happens? Can you paste the error you are seeing? What's your k8s setup?

From the platformatic code in the benchmark, have you seen some miss-use? maybe the configuration or message processing can be better?

Remove all Nest, frankly, it's very hard to follow your setup.

[TheOnlyArtz]

Nest aside there are 3 relevant files in the Kafka directory..

• kafka.module.ts
• kafka-inbound-consumer
• Kafka producer

The said error is 700MB of ram usage while stress testing as you can see in the issue description.

The service just gets completely overwhelmed, rebalancing and dies from excessive RAM usage.

What's your k8s setup

What do you mean?
We are on AWS, the pods are capped at 512MB RAM

[mcollina]

Usually you configure K8s using some yml, which describes how your pods are set up.

[TheOnlyArtz]

Usually you configure K8s using some yml, which describes how your pods are set up.

I'm not devops
If you could tell me exactly what you need to know I'll happily provide