add proposal for multi namespace topic operator

[KyriosGN0]

This PR aims to introduce a multi namespace topic operator, its based on the proposal for UTO, and uses a different approach as opposed to strimzi/strimzi-kafka-operator#1206

[scholzj]

Thanks for the proposal. I left some comments about things that would not work or would be blockers for me in terms of backwards compatiblity etc.

But I do not think that this is the right direction. While it might create something that somehow works, I think it would be a hacky solution with a bad design and user experience. I think if we want to seriously go into the multi-namespace world, we might need to start from scratch and create a better match/respect for the Kubernetes namespace model.

[scholzj]

Tha naming here as well as below does not work because it is not unique. You need to create a unique naming mechanism (likely cluster name, namespace + some suffix).

[KyriosGN0]

Thanks, I updated the proposal accordingly to ensure uniqueness

[scholzj]

There is already an existing mechanim how to designate to which cluster the topic belongs through the strimzi.io/cluster label. Both from the user as well as from the TO perspective.

I do not understand why don't you use that? You will anyway need to extend it. It also has backwards compatibility implications that you would need to address.

[KyriosGN0]

Good Point, i updated the proposal accordingly

[scholzj]

I don't think the fix works.

• How does the operator know it is in single namespace or multi namespace mode?
• How does it know if there are more clusters (more TOs watching the same namespace) or not?

So I'm not sure this works or creates backwards compatile solution.

[KyriosGN0]

How does the operator know it is in single namespace or multi namespace mode?

according to the watchedNamespaces field, if its the plural its multi namespace, if its the singular, its single namespace

How does it know if there are more clusters (more TOs watching the same namespace) or not?

if its multi namespace it uses the strimzi.io/cluster = <namespace_of_to>/<kafka_name> label, so if you have 2 TO watching the same namespace. so if we create a topic in namespace a, its owned by TO of the namespace kafka, (due to multi namespace mode), then another TO tries to own it, it can't cuz its already managed by another TO (due to the label)

[scholzj]

according to the watchedNamespaces field, if its the plural its multi namespace, if its the singular, its single namespace

So, what if you have watchedNamespaces with a single namespace? What if you want to watch the whole cluster?

It relates to the point about deprecation as well, as you clearly need to move over from it if you want to deprecate it. But that would now impact the labels.

[KyriosGN0]

OK, i had a misunderstanding on my end about the strimzi.io/cluster label (i mistakenly thought its controlled by the operator)
since its controlled by the user, i updated the proposal to deprecate the short name (just cluster) and when switching from the singular to plural users should also migrate from the short to a fully name (namespace/kafka_name)
the TO will accept both of them, but it log a warning (or we can more forceful and also skip reconciliation when this situation occurs) that there is a possibility of a conflict

[scholzj]

I'm not sure it is so simple. If you just log a warning, how do you know which Topic Operator should manage it? And if you instead throw an error, you break backwards compatibility.

[KyriosGN0]

it should be the user's responsibility to know which topic relates to which Kafka during the migration

[scholzj]

That does not work because once you remove the namespace, it is beyond the operator's reach. So it cannot remove the finalizers.

[KyriosGN0]

I think you mean when someone else deletes the namespaces, and not when someone removes a namespace from the watchedNamespaces list, in the first case you are right and manual intervention is required, but if i remove a namespace from the operator's config i can perform action before i remove RBAC, i updated the proposal to make sure this is clear

[scholzj]

Sorry, maybe my comment was too confusing.

• Cluster Operator does not touch any topics. It should not remove any finalizers from them. (not 100% sure the CO is typo here or intention)
• Topic Operator would never know that some namespace was removed as it is removed through restart. So it starts with the new namespace configuration and has no idea a namespace was removed.
• And even if it knew about it, it would not have the RBAC rights to do anything with the topics because the RBAC rights are managed independently through CO, and there is no synchronization point.
• Plus we in general do not delete the user-facing custom resources.

I do not think you can really fix it to be honest. All what can be done here is that this is user's responsibility to either delete the topics first or manually later.

[scholzj]

How does this handle compatibility witht he existing watchNamespace field?

[KyriosGN0]

Updated the Proposal, i went with a route of depreacting this field in favor of the plural one

[scholzj]

Deprecation is fine. But we are not abandoning fields from one day to another. We do things in backwards compatible way.

[KyriosGN0]

no problem, what would you like to me to specify in regards to the deprecation ? is there a policy for announcing/managing deprecations ?

[scholzj]

Well, you need to explain how it will be handled when it will be used, how users will migrate, etc.

[scholzj]

If you don't manage it, you should not touch it. That will just cause tight reconciliation look as the two operators fight over the updates.

[KyriosGN0]

Addressed

[KyriosGN0]

Thanks @scholzj, I Have addressed your comments

1. Updated the RBAC names to be unique
2. added that the new plural watchNamespaces is deprecating watchNamespace
3. clarified that removing a namespace from watchNamespaces is different from deleting a namespace

[KyriosGN0]

Hey @scholzj, is there anything else i need to address? do i need to request other maintainers as reviewers for this proposal? (saw it on other proposals)

[scholzj]

Hey @scholzj, is there anything else i need to address? do i need to request other maintainers as reviewers for this proposal? (saw it on other proposals)

They should have notifications. But I requested their reviews for it explicitely.

[ppatierno]

I added @fvaleri as SME on the topic operator.

[im-konge]

Thanks for the proposal, I left few comments.

[im-konge]

I know that you talk about standalone deployment, but maybe it would be worth mentioning both modes here - so that UTO is deployed either as part of Entity Operator or it can be deployed standalone?

[KyriosGN0]

@im-konge updated the proposal accordingly

[im-konge]

So when UTO will have watchedNamespaces: ["kafka1", "kafka2"], the name of the Kafka cluster will by my-cluster and I will create KafkaTopics with label strimzi.io/cluster: my-cluster in both Namespaces, will they be reconciled by UTO? Or UTO will just take the KafkaTopics with such label from the Namespace it is running in only?

[KyriosGN0]

yes, they will both be reconciled (and if they have different configuration they will lead a cycle of conflicts), its the user's responsibility to make sure this doesn't happen, This is why we log a warning about using the unqualified form

[im-konge]

Yeah, cool. I didn't mean same KafkaTopics (like same topic names), but different topic names in different Namespaces without label having the <kafka-namespace>/<cluster-name>. But I think it's fine if they will be reconciled. Thanks

Maybe it would be worth mentioning? I don't remember if it was mentioned somewhere (possibly later in the proposal), but I think it would be worth to mention it here as well (something similar was in the previous section about watchNamespaces).

[im-konge]

This is correct for the mode when it is running in EntityOperator, but in standalone it should be created by user. So I guess we should change the files we are providing for standalone UTO, right?

[fvaleri]

Yes, we should also update the doc with the various deployment modes, similar to what we have for the Cluster Operator.

[KyriosGN0]

Yes, it sounds like the best way to avoid confusion (but we would need to announce their removal, and a docs upgrade)

[im-konge]

Yeah, I guess it should be mentioned in the proposal as here you are talking just about the EntityOperator configuration.

[fvaleri]

Hello @KyriosGN0, thanks for creating this proposal.

Kafka support for multi-tenancy is not great, and this is probably the main reason why we don't have multi-namespace support. The elephant in the room is the topic name mapping that you are not addressing here.

[fvaleri]

This is not correct. The watched namespace configuration allows the TO to watch a different namespace from where the Kafka cluster is deployed. The Kafka cluster's namespace is simply the default value.

I would suggest to rephrase to something like this:

Currently, the Topic Operator watches a single namespace, either the Kafka cluster's own namespace (default) or one alternative namespace configured via watchedNamespace field or STRIMZI_NAMESPACE environment variable. This is inconvenient for the following reasons:

[fvaleri]

What about this?

[KyriosGN0]

Will rephrase

[fvaleri]

Why do we always add TO's namespace implicitly? In order to be backwards compatible we only need to default to that, but if the user sets a custom list, then we only watch namespaces in that list.

[fvaleri]

What about this?

[KyriosGN0]

In my mind the watchedNamespaces list is not an override to the default behaviour but an extension of it, it doesn't make sense to me to remove the kafka cluster namespace when we omit it from the list

[fvaleri]

In most standard Kubernetes resources, the user's input completely replaces the default rather than extending it. If you want to do it differently, you should provide a clear motivation.

[fvaleri]

I would suggest to use CEL validation to reject the resource at admission time when both fields are set, making the conflict a hard error rather than a runtime precedence game.

When only the singular field is set, the TO behaves exactly as it does today — watching the Kafka cluster's own namespace plus the single additional namespace specified.

Not correct, as explained in another comment.

[fvaleri]

Yes, we should also update the doc with the various deployment modes, similar to what we have for the Cluster Operator.

[fvaleri]

In packaging we have strimzi-topic-operator as SA name for standalone mode.

[KyriosGN0]

This is an example for how the CO does creates the role binding, not the standalone manifests

[fvaleri]

That would be just a default, but the user should be able to customize the prefix.

[fvaleri]

If we claim multi-namespace support, users would not expect to be limited by topic names from other teams. In my mind, this is a prerequisite for the current proposal.

I imagine something like this:

apiVersion: kafka.strimzi.io/v1
kind: KafkaTopicPolicy
metadata:
  name: my-cluster-policy
  namespace: team-a
  labels:
    strimzi.io/cluster: kafka/my-cluster
spec:
  topicNamePrefix: "team-a."
  allowedConfigs:
    - retention.ms
    - cleanup.policy
  maxPartitions: 24
  maxReplicas: 3
  defaults:
    retention.ms: "604800000"

Platform teams manage policies per namespace, application teams manage their own topics. When there is no policy, the Topic Operator would apply some sensible default (e.g. namespace prefix). Of course the are many more details to discuss, so I agree that this should be a dedicated proposal.

[KyriosGN0]

so in your mind, a proposal about a KafkaTopicPolicy (or a proposal about central management of topic config) is a blocker for this proposal ?

[fvaleri]

o in your mind, a proposal about a KafkaTopicPolicy (or a proposal about central management of topic config) is a blocker for this proposal ?

This proposal would enable multi-tenancy on a shared Kafka cluster, so topic namespacing is critical IMO. Multi-tenancy is a much broader topic than multi-namespace support, but this is a fundamental piece.

[fvaleri]

It would be good to describe what happens with overlapping namespaces. The edge cases are: two or more TOs pointing to the same cluster or different clusters with the same name.

---

Another approach to solve this problem would be to leverage STRIMZI_RESOURCE_LABELS. When watchedNamespaces is used, the CO could add a label automatically:

  labels:                                                                                                                                                                                                                                                                                  
    strimzi.io/cluster: my-cluster
    strimzi.io/cluster-namespace: kafka # new

There are multiple benefits:

• The strimzi.io/cluster label stays unqualified. No deprecation, no migration of existing label values.
• Two TOs pointing to my-cluster in namespaces kafka and staging watching the same tenant namespace would have distinct selectors and never clash.
• Single-namespace deployments that don't set watchedNamespaces could keep working with just strimzi.io/cluster=my-cluster (no CR changes).
• Logs would not be flooded by unqualified label warnings.

[KyriosGN0]

Hey, @fvaleri @im-konge @scholzj, i have changed the solution in the proposal following #204 (comment) (as i like it more).
Thanks in Advance and sorry for the ping

[fvaleri]

What about my suggestion of adding the new label automatically? Did you reject that? Why? That would avoid the need for logging (which anyway should be in batches, not for every single KT).

[KyriosGN0]

@fvaleri you mean when the topic is managed via the strimzi.io/cluster=my-cluster but still doesn't have the strimzi.io/cluster-namespace label?

[fvaleri]

Yes, when watchedNamespaces is used, the CO should automatically add strimzi.io/cluster-namespace=<kafka-namespace> to the STRIMZI_RESOURCE_LABELS environment variable passed to the TO.

Main benefits:

1. The strimzi.io/cluster label stays unqualified. No deprecation, no migration of existing label values.
2. Single-namespace deployments that don't set watchedNamespaces could keep working with just strimzi.io/cluster=my-cluster (no CR changes). Logs would not be flooded by unqualified label warnings.

[fvaleri]

What about this?

[fvaleri]

What about this?

[fvaleri]

o in your mind, a proposal about a KafkaTopicPolicy (or a proposal about central management of topic config) is a blocker for this proposal ?

This proposal would enable multi-tenancy on a shared Kafka cluster, so topic namespacing is critical IMO. Multi-tenancy is a much broader topic than multi-namespace support, but this is a fundamental piece.

[fvaleri]

@KyriosGN0 hi, sorry for the late reply.

There are a few critical points that I think still need to be addressed before we can move forward with the proposal. I tried to clarify them, but feel free to ask me if anything is not clear.

[fvaleri]

In most standard Kubernetes resources, the user's input completely replaces the default rather than extending it. If you want to do it differently, you should provide a clear motivation.

[fvaleri]

Yes, when watchedNamespaces is used, the CO should automatically add strimzi.io/cluster-namespace=<kafka-namespace> to the STRIMZI_RESOURCE_LABELS environment variable passed to the TO.

Main benefits:

1. The strimzi.io/cluster label stays unqualified. No deprecation, no migration of existing label values.
2. Single-namespace deployments that don't set watchedNamespaces could keep working with just strimzi.io/cluster=my-cluster (no CR changes). Logs would not be flooded by unqualified label warnings.

[fvaleri]

This is another critical point for me (in addition to topic namespacing support), so we need more details on how this would work for the Topic Operator. The CO uses a thread-per-namespace approach where each thread has its own watch, and that helps to mitigate the noisy neighbor problem (a namespace producing a huge amount of events starving the others).

[KyriosGN0]

In most standard Kubernetes resources, the user's input completely replaces the default rather than extending it. If you want to do it differently, you should provide a clear motivation.

I Think in terms of User's expectation its really beneficial to always include the Kafka Cluster namespace, if it was me i would be surprised to find out if the kafka cluster namespace is not watched.

re: the 2nd comment (im sorry i just can't seem to comment) Agreed, i will update the proposal

This is another critical point for me (in addition to topic namespacing support), so we need more details on how this would work for the Topic Operator. The CO uses a thread-per-namespace approach where each thread has its own watch, and that helps to mitigate the noisy neighbor problem (a namespace producing a huge amount of events starving the others).

I tend to agree with the thread-per-namespace approach, but i also think that topics will be modifed/created/deleted in higher frequency, and with the amount of possible namespace, we possibly could reach some limitation around the number of threads (this is not an area i am familiar with, so i might be completely wrong)

thanks @fvaleri

[KyriosGN0]

@fvaleri gentle reminder 🙏

[fvaleri]

Thanks for the updates @KyriosGN0.

I went through the latest revision and there are still a few points from my earlier comments that I think need to be addressed before we can move forward. I would also wait for other maintainer's reviews, as you need at least two positive maintainer votes.

1. Topic namespacing. I consider this a prerequisite for the current proposal, not future work. If we claim multi-namespace support, users will expect topic name isolation between teams. The current answer of "run separate TO instances per team" undermines the purpose of shared multi-namespace watching. This does not mean we need to design KafkaTopicPolicy here, but the proposal should at least define a minimal default behavior (e.g. namespace prefix) and acknowledge it as a hard dependency rather than deferring it entirely.

2. Implicit namespace watching. The proposal still says the TO's own namespace is "always watched implicitly." In standard Kubernetes resources, user input replaces the default rather than extending it. If we want to deviate from that convention, we need a clear technical justification in the proposal. Defaulting to the TO's namespace when watchedNamespaces is empty is fine for backwards compatibility, but an explicit list should mean exactly that list.

3. Noisy neighbor mitigation. This is another critical point that currently has only a one-liner. The CO uses a thread-per-namespace approach where each thread has its own watch, which helps prevent a single namespace from starving the others. The proposal needs to describe how the TO would handle this concretely: architecture, threading/queuing model, and any resource isolation guarantees.

4. Information leakage in conflict messages. The proposal is currently self-contradictory. The conflict resolution section uses "Already managed by another topic operator", but the example section still shows "Managed by team-a/orders". The security section also describes leaking the winner's namespace/name as expected behavior. We wuldn't have this problem if point 1 wuole be properly addressed.

5. RBAC definitions. A few issues remain: The Role for explicit namespaces groups kafkatopics and kafkatopics/status with all verbs including delete on status, which is overly broad. We follow the principle of least privilege. The Role is missing kafkatopics/finalizers, which is present in the ClusterRole for wildcard mode. These should be consistent. There is a typo in the ClusterRole: [""kafkatopics/status"] (double opening quote).

6. Standalone ServiceAccount in examples. The text correctly mentions strimzi-topic-operator for standalone mode, but the RoleBinding example still uses my-cluster-entity-operator. It would help to either show both modes or add a note clarifying this is the Kafka-managed case only.