Chapter04 - opendata-hub-operator-controller-manager crashloopbackoff

[LucianoBatista]

After apply the odh-subscription.yaml, I'm getting some errors on the pod. Could someone help me?

kubectl logs opendatahub-operator-controller-manager-bff48cb67-c6wpq -n operators -f

2023-05-13T16:31:40.886Z	INFO	controller.kfdef-controller	Starting Controller	{"reconciler group": "kfdef.apps.kubeflow.org", "reconciler kind": "KfDef"}
2023-05-13T16:31:40.987Z	INFO	controllers.KfDef	Adding finalizer	{"kfdef-finalizer.kfdef.apps.kubeflow.org": "ml-workshop/opendatahub-ml-workshop"}
2023-05-13T16:31:41.340Z	ERROR	controller-runtime.source	if kind is a CRD, it should be installed before calling Start	{"kind": "DeploymentConfig.apps.openshift.io", "error": "no matches for kind \"DeploymentConfig\" in version \"apps.openshift.io/v1\""}
2023-05-13T16:31:41.340Z	INFO	controller.secret-generator-controller	Starting workers	{"reconciler group": "", "reconciler kind": "Secret", "worker count": 1}
2023-05-13T16:31:41.346Z	INFO	controllers.KfDef	Adding finalizer	{"kfdef-finalizer.kfdef.apps.kubeflow.org": "ml-workshop/opendatahub-ml-workshop"}
I0513 16:31:42.488515       1 request.go:668] Waited for 1.048971907s due to client-side throttling, not priority and fairness, request: GET:https://10.96.0.1:443/apis/node.k8s.io/v1?timeout=32s
2023-05-13T16:31:43.289Z	ERROR	controller-runtime.source	if kind is a CRD, it should be installed before calling Start	{"kind": "BuildConfig.build.openshift.io", "error": "no matches for kind \"BuildConfig\" in version \"build.openshift.io/v1\""}
2023-05-13T16:31:43.289Z	ERROR	controller.kfdef-controller	Could not wait for Cache to sync	{"reconciler group": "kfdef.apps.kubeflow.org", "reconciler kind": "KfDef", "error": "failed to wait for kfdef-controller caches to sync: no matches for kind \"DeploymentConfig\" in version \"apps.openshift.io/v1\""}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.0/pkg/internal/controller/controller.go:234
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.0/pkg/manager/internal.go:696
2023-05-13T16:31:43.289Z	INFO	controller.secret-generator-controller	Shutdown signal received, waiting for all workers to finish	{"reconciler group": "", "reconciler kind": "Secret"}
2023-05-13T16:31:43.289Z	INFO	controller.secret-generator-controller	All workers finished	{"reconciler group": "", "reconciler kind": "Secret"}
2023-05-13T16:31:43.289Z	ERROR	error received after stop sequence was engaged	{"error": "leader election lost"}
runtime.goexit
	/usr/lib/golang/src/runtime/asm_amd64.s:1571
2023-05-13T16:31:43.290Z	ERROR	error received after stop sequence was engaged	{"error": "context canceled"}
runtime.goexit
	/usr/lib/golang/src/runtime/asm_amd64.s:1571
2023-05-13T16:31:43.290Z	ERROR	error received after stop sequence was engaged	{"error": "context canceled"}
runtime.goexit
	/usr/lib/golang/src/runtime/asm_amd64.s:1571
2023-05-13T16:31:43.290Z	ERROR	setup	problem running manager	{"error": "failed to wait for kfdef-controller caches to sync: no matches for kind \"DeploymentConfig\" in version \"apps.openshift.io/v1\""}
runtime.goexit
	/usr/lib/golang/src/runtime/asm_amd64.s:1571

[webmakaka]

I had the same error.
If you find solution, please share.

[LucianoBatista]

Versions that I'm using:

• minikube=v1.30.1
• kubernetes=1.22.4

All the rest of the configuration stays the same as the book.

[LucianoBatista]

@webmakaka, but did you were able to run all the rest of the services (airflow, mlflow, spark...)?

The only thing that appears on my operators namespace is:

[webmakaka]

I have problems with opendatahub-operator-controller-manager. It restarts. Two months ago everything worked for me.

If you do not have problems with it

Run

$ kubectl create ns ml-workshop

$ kubectl create -f manifests/kfdef/ml-platform.yaml -n ml-workshop

// Need to wait something about 15 minutes
$ watch kubectl get pods -n ml-workshop

Successfull result should be

// Need to wait something about 15 minutes
$ kubectl get pods -n ml-workshop
NAME                                           READY   STATUS      RESTARTS        AGE
app-aflow-airflow-scheduler-6ccc679fc6-6k9kv   2/2     Running     0               17m
app-aflow-airflow-web-55767cd99d-x9kh7         2/2     Running     0               9m41s
app-aflow-airflow-worker-0                     2/2     Running     1 (8m38s ago)   17m
app-aflow-postgresql-0                         1/1     Running     0               17m
app-aflow-redis-master-0                       1/1     Running     0               17m
flightsdatadb-0                                1/1     Running     0               15m
grafana-56594fd448-c76qd                       1/1     Running     0               17m
jupyterhub-6cff7d6cc5-62snc                    1/1     Running     0               17m
jupyterhub-db-0                                1/1     Running     0               17m
minio-ml-workshop-7fcc5dfd8-m57lv              1/1     Running     0               17m
minio-ml-workshop-lbfh5                        0/1     Completed   2               17m
mlflow-5fb4cb5d5d-89v74                        2/2     Running     0               17m
mlflow-db-0                                    1/1     Running     1 (16m ago)     17m
prometheus-odh-monitoring-0                    2/2     Running     0               13m
prometheus-operator-7869664bcc-f42d2           1/1     Running     0               15m
seldon-controller-manager-c5f59c56d-2rcc6      1/1     Running     0               17m
spark-operator-6bc4f8f5f8-rs2mk                1/1     Running     0               17m

[webmakaka]

I do not know how to fix

[LucianoBatista]

That's the problem, even when I run those commands, any service appears on the ml-workshop namespace.

[webmakaka]

They should appears in ml-workshop namespace

[LucianoBatista]

Is there some way to get a log of those services that aren't loading?

I have very little experience with K8s and just know get logs on specific pods...

[webmakaka]

You can setup k9s.
All microservices with errors will be red.

[webmakaka]

The solution to fix issue from first message is to stop auto updates for opendatahub-operator

$ cat << EOF | kubectl apply -f -
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: opendatahub-operator
  namespace: operators
spec:
  channel: stable
  # installPlanApproval: Automatic
  installPlanApproval: Manual
  name: opendatahub-operator
  source: community-operators-redhat
  sourceNamespace: olm
  #startingCSV: opendatahub-operator.v1.4.2
  startingCSV: opendatahub-operator.v1.1.1
EOF

Version 1.5 fails with error.

$ kubectl get installplan -A
NAMESPACE     NAME            CSV                           APPROVAL    APPROVED
ml-workshop   install-gtglv   prometheusoperator.0.56.3     Automatic   true
operators     install-45bp7   opendatahub-operator.v1.5.0   Manual      false
operators     install-6tmv4   opendatahub-operator.v1.4.2   Manual      true

[bataliero]

I have the same error:

2023-10-21T19:51:56.154Z        ERROR   controller-runtime.source       if kind is a CRD, it should be installed before calling Start   {"kind": "DeploymentConfig.apps.openshift.io", "error": "no matches for kind \"DeploymentConfig\" in version \"apps.openshift.io/v1\""}
I1021 19:51:57.302870       1 request.go:668] Waited for 1.048366166s due to client-side throttling, not priority and fairness, request: GET:https://10.96.0.1:443/apis/apps/v1?timeout=32s
2023-10-21T19:51:58.104Z        ERROR   controller-runtime.source       if kind is a CRD, it should be installed before calling Start   {"kind": "ImageStream.image.openshift.io", "error": "no matches for kind \"ImageStream\" in version \"image.openshift.io/v1\""}
2023-10-21T19:51:58.204Z        INFO    controller.secret-generator-controller  Starting workers        {"reconciler group": "", "reconciler kind": "Secret", "worker count": 1}
2023-10-21T19:52:00.054Z        ERROR   controller-runtime.source       if kind is a CRD, it should be installed before calling Start   {"kind": "BuildConfig.build.openshift.io", "error": "no matches for kind \"BuildConfig\" in version \"build.openshift.io/v1\""}
2023-10-21T19:52:00.157Z        ERROR   controller.kfdef-controller     Could not wait for Cache to sync        {"reconciler group": "kfdef.apps.kubeflow.org", "reconciler kind": "KfDef", "error": "failed to wait for kfdef-controller caches to sync: no matches for kind \"DeploymentConfig\" in version \"apps.openshift.io/v1\""}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
        /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.0/pkg/internal/controller/controller.go:234
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1
        /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.0/pkg/manager/internal.go:696
2023-10-21T19:52:00.157Z        INFO    controller.secret-generator-controller  Shutdown signal received, waiting for all workers to finish     {"reconciler group": "", "reconciler kind": "Secret"}
2023-10-21T19:52:00.157Z        INFO    controller.secret-generator-controller  All workers finished    {"reconciler group": "", "reconciler kind": "Secret"}
2023-10-21T19:52:00.157Z        ERROR   setup   problem running manager {"error": "failed to wait for kfdef-controller caches to sync: no matches for kind \"DeploymentConfig\" in version \"apps.openshift.io/v1\""}
runtime.goexit

Is there a solution?

[webmakaka]

One message above

[bataliero]

For reference- I am adding a instruction to have installplan approved:

kubectl patch installplan <plan> -n operators -p '{"spec":{"approved":true}}' --type merge

[ahmedelfaleh]

Hello Everyone,

I am still have the same issue, but I don't believe it is related to InstallPlan.
The problem is that there are manifests for OpenShift needs to be applied by the operators but their CRDs aren't existing in kubernetes.
I don't know how to resolve it, any help

[webmakaka]

Hi!
As i remember, you should disable auto updates to latest opendatahub-operator.