ORBOS 5.X constantly ensuring NGINX and keepalived

[privateVoit]

Hi there!

We're currently testing ORBOS 5.1.5 and discovered an issue with node agent. It is constantly reconciling and restarting nginx as well as keepalived.

Aug 26 09:07:15 master01 node-agent[101753]: ts="2021-08-26T09:07:15+02:00" msg="Ensuring software" software="[NGINX]"
Aug 26 09:07:15 master01 node-agent[101753]: ts="2021-08-26T09:07:15+02:00" msg="Ensuring dependency" dependency="NGINX" from="v1.18.0" to="v1.18.0"
Aug 26 09:07:16 master01 node-agent[101753]: ts="2021-08-26T09:07:16+02:00" evt="Dependency ensured" dependency="NGINX" from="v1.18.0" to="v1.18.0"
Aug 26 09:07:16 master01 node-agent[101753]: ts="2021-08-26T09:07:16+02:00" msg="Iteration done"
Aug 26 09:07:26 master01 node-agent[101753]: ts="2021-08-26T09:07:26+02:00" msg="Starting iteration"
Aug 26 09:07:34 master01 node-agent[101753]: ts="2021-08-26T09:07:34+02:00" msg="Repository pushed" repository="ssh://git@[REDACTED]/cluster-01/t01-cluster.git"
Aug 26 09:07:34 master01 node-agent[101753]: ts="2021-08-26T09:07:34+02:00" msg="Iteration done"
Aug 26 09:07:44 master01 node-agent[101753]: ts="2021-08-26T09:07:44+02:00" msg="Starting iteration"
Aug 26 09:07:53 master01 node-agent[101753]: ts="2021-08-26T09:07:53+02:00" msg="Repository pushed" repository="ssh://git@[REDACTED]/cluster-01/t01-cluster.git"
Aug 26 09:07:53 master01 node-agent[101753]: ts="2021-08-26T09:07:53+02:00" msg="Ensuring software" software="[NGINX]"
Aug 26 09:07:53 master01 node-agent[101753]: ts="2021-08-26T09:07:53+02:00" msg="Ensuring dependency" dependency="NGINX" from="v1.18.0" to="v1.18.0"
Aug 26 09:07:53 master01 node-agent[101753]: ts="2021-08-26T09:07:53+02:00" evt="Dependency ensured" dependency="NGINX" from="v1.18.0" to="v1.18.0"
Aug 26 09:07:53 master01 node-agent[101753]: ts="2021-08-26T09:07:53+02:00" msg="Iteration done"
Aug 26 09:08:03 master01 node-agent[101753]: ts="2021-08-26T09:08:03+02:00" msg="Starting iteration

We've seen this behaviour on every version starting with 5.0.

When comparing node-agents-current.yml and node-agents-desired.yml we can see follwing changes.

NGINX
The nginx.conf is constlancy changed. But i'ts only the order of the upstream ip's.
This:

        upstream httpsingress {
                server 10.73.244.82:30443; # workers
                server 10.73.244.94:30443; # workers
                server 10.73.244.101:30443; # workers
        }

Changes to this:

        upstream httpsingress {
                server 10.73.244.82:30443; # workers
                server 10.73.244.101:30443; # workers
                server 10.73.244.94:30443; # workers
        }

Keepalived
Keepalived is also constantly reconciled. It's is repeatedly chaging it's state from MASTER to BACKUP. Premsumably because of the repeated NGINX restarts?
This:

              	state MASTER
              	unicast_src_ip 10.73.244.103
              	unicast_peer {
              		10.73.244.111

Changes to this:

              	state BACKUP
              	unicast_src_ip 10.73.244.103
              	unicast_peer {
              		10.73.244.111

We've seen this issue on all cluster's that we've updated to 5.0 or higher.

What might cause this issue?

If you need any more informations, just ask!

[eliobischof]

could you please try again using ORBITER v5.1.10?

[privateVoit]

Good Morning!

I can't see any issues anymore in ORBOS 5.1.10. Thank you!