Goroutine terminates when DB fetch fail to provide account data #85
Description
Is your feature request related to a problem? Please describe.
This edge-case was observed some time ago which consists in a failure to process account data when the handler initialize the sync-up process check.
The logic implemented at the handler code, more specifically at the StartAccountHandler function, there is a verification step that happens at the very beginning, which consists in leveraging account data that would be use for the checklist (version diff, commithash, etc).
Currently, the decision made here is to terminate the Goroutine for this account when no account is found. The scenario covered is when users unsubscribe the Domain from the GitOps integration feature.
However, DB fetch may fail due to outages or systematic errors that wouldn't return a valid account at this particular step.
Describe the solution you'd like
We should have a recovery process that can be either systematic or scheduled by the application or user command.
Scheduled recover
This option would be defined as a separated Goroutine that could inspect all DB accounts in a certain interval and compare with running account handler Goroutines. That way, any issue related to DB can be recovered without any user action.
This solution would require changes only in switcher-gitops
User command
This option could allow user to fast refresh failed subscription to backup the operation when something went wrong with DB. This allows an immediate account recover if there is a need to timely sync-up an issued account.
This solution would require changes in switcher-gitops, switcher-api and switcher-management
Describe alternatives you've considered
Both solutions above could also be implemented as they deliver different expectations and covers a sum of value that could improve and fix any DB related issue.