cannot list resource "pools" at the cluster scope

[chickenkiller]

What steps did you take and what happened?

I am in the process of deploying a garm-operator configuration in an AKS cluster, using ArgoCD.
I had to tweak a bit the sync waves to be able to create the admission webhook and its service first, and I could get rid of healthy check issues.
My Garm server seems to be responding fine and is running in another namespace in the same cluster.

I deployed an Organization, an Image and a Pool. All 3 custom resources are properly created in k8s, and reconciliation works for org and image (it seems). But an issue arises about the pools listing, which I can't understand properly:

garm-operator-controller-manager-7689ffcc46-dbzls manager E0605 13:42:06.703409       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.13/tools/cache/reflector.go:229: Failed to watch *v1beta1.Pool: failed to list *v1beta1.Pool: pools.garm-operator.mercedes-benz.com is forbidden: User "system:serviceaccount:garm-operator-system:garm-operator-controller-manager" cannot list resource "pools" in API group "garm-operator.mercedes-benz.com" at the cluster scope: Azure does not have opinion for this user.

What bothers me is the 'at the cluster scope'. Why does garm-operator need to list pool resources at the cluster scope, whereas they are declared in the proper garm-operator-system namespace? And the corresponding RoleBinding is not a ClusterRoleBinding...

What did you expect to happen?

I expect Pool list to be reconciled properly. One step at a time.

garm version

garm server: v0.1.5

garm-operator version

v0.4.1

Kubernetes version

Server Version: v1.30.11

Anything else you would like to add?

No response

[bavarianbidi]

Hey @chickenkiller,

i assume you've deployed the manifest from the release page (https://github.com/mercedes-benz/garm-operator/releases/tag/v0.4.1)?

just to double-check:

the following RoleBinding exists and the operator is having the corresponding servcieaccount attached:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    app.kubernetes.io/component: rbac
    app.kubernetes.io/created-by: garm-operator
    app.kubernetes.io/instance: manager-rolebinding
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/name: rolebinding
    app.kubernetes.io/part-of: garm-operator
  name: garm-operator-manager-rolebinding
  namespace: garm-operator-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: garm-operator-manager-role
subjects:
- kind: ServiceAccount
  name: garm-operator-controller-manager
  namespace: garm-operator-system

and at least the referenced role is having this defined

- apiGroups:
  - garm-operator.mercedes-benz.com
  resources:
  - pools
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch

[chickenkiller]

Hello @bavarianbidi,

yes I deployed using the release manifest. I just changed the controller command-line to use a config.yaml file instead of arguments.

And I confirm these objects are properly deployed:

RoleBinding garm-operator-manager-rolebinding

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  annotations:
    argocd.argoproj.io/tracking-id: garm-operator-dev:rbac.authorization.k8s.io/RoleBinding:garm-operator-system/garm-operator-manager-rolebinding
  labels:
    app: raw-configuration
    app.kubernetes.io/component: rbac
    app.kubernetes.io/created-by: garm-operator
    app.kubernetes.io/instance: manager-rolebinding
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/name: rolebinding
    app.kubernetes.io/part-of: garm-operator
    chart: raw-configuration-1.0.0
    heritage: Helm
    release: raw-configuration
  name: garm-operator-manager-rolebinding
  namespace: garm-operator-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: garm-operator-manager-role
subjects:
- kind: ServiceAccount
  name: garm-operator-controller-manager
  namespace: garm-operator-system

Role garm-operator-manager-role

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  annotations:
    argocd.argoproj.io/tracking-id: garm-operator-dev:rbac.authorization.k8s.io/Role:garm-operator-system/garm-operator-manager-role
  labels:
    app: raw-configuration
    chart: raw-configuration-1.0.0
    heritage: Helm
    release: raw-configuration
  name: garm-operator-manager-role
  namespace: garm-operator-system
rules:
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - garm-operator.mercedes-benz.com
  resources:
  - enterprises
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - garm-operator.mercedes-benz.com
  resources:
  - enterprises/finalizers
  verbs:
  - update
- apiGroups:
  - garm-operator.mercedes-benz.com
  resources:
  - enterprises/status
  verbs:
  - get
  - patch
  - update
- apiGroups:
  - garm-operator.mercedes-benz.com
  resources:
  - garmserverconfigs
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - garm-operator.mercedes-benz.com
  resources:
  - garmserverconfigs/finalizers
  verbs:
  - update
- apiGroups:
  - garm-operator.mercedes-benz.com
  resources:
  - garmserverconfigs/status
  verbs:
  - get
  - patch
  - update
- apiGroups:
  - garm-operator.mercedes-benz.com
  resources:
  - githubcredentials
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - garm-operator.mercedes-benz.com
  resources:
  - githubcredentials/finalizers
  verbs:
  - update
- apiGroups:
  - garm-operator.mercedes-benz.com
  resources:
  - githubcredentials/status
  verbs:
  - get
  - patch
  - update
- apiGroups:
  - garm-operator.mercedes-benz.com
  resources:
  - githubendpoints
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - garm-operator.mercedes-benz.com
  resources:
  - githubendpoints/finalizers
  verbs:
  - update
- apiGroups:
  - garm-operator.mercedes-benz.com
  resources:
  - githubendpoints/status
  verbs:
  - get
  - patch
  - update
- apiGroups:
  - garm-operator.mercedes-benz.com
  resources:
  - images
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - garm-operator.mercedes-benz.com
  resources:
  - organizations
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - garm-operator.mercedes-benz.com
  resources:
  - organizations/finalizers
  verbs:
  - update
- apiGroups:
  - garm-operator.mercedes-benz.com
  resources:
  - organizations/status
  verbs:
  - get
  - patch
  - update
- apiGroups:
  - garm-operator.mercedes-benz.com
  resources:
  - pools
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - garm-operator.mercedes-benz.com
  resources:
  - pools/status
  verbs:
  - get
  - patch
  - update
- apiGroups:
  - garm-operator.mercedes-benz.com
  resources:
  - repositories
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - garm-operator.mercedes-benz.com
  resources:
  - repositories/finalizers
  verbs:
  - update
- apiGroups:
  - garm-operator.mercedes-benz.com
  resources:
  - repositories/status
  verbs:
  - get
  - patch
  - update
- apiGroups:
  - garm-operator.mercedes-benz.com
  resources:
  - runners
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - garm-operator.mercedes-benz.com
  resources:
  - runners/finalizers
  verbs:
  - update
- apiGroups:
  - garm-operator.mercedes-benz.com
  resources:
  - runners/status
  verbs:
  - get
  - patch
  - update

[bavarianbidi]

hm, can you do me a favor and explicitly set the watchNamespace config option in the configFile to the namespace where your objects get applied to.

[chickenkiller]

hm, can you do me a favor and explicitly set the watchNamespace config option in the configFile to the namespace where your objects get applied to.

You're right, it works much better! I explicted it before but I had other issues at hand. Now the pool is created in garm! 🥳

[bavarianbidi]

Good to know, but it's definitely an issue on our side - need to investigate on this further.

[chickenkiller]

In that case I leave the issue open as a reminder and I let you close it when you have a fix for it? You can ask me to make attempts, I'll be happy to help.