Monitor system and notify admins when needed

[simonwacker]

We use systemd to manage services. For each service, the corresponding unit file is by default properly configured. Unit files for example specify if and how often a service restart should be done on failure. For some information on this see Set up self-healing services with systemd. If you are unfamiliar with systemd see 10 handy systemd commands: A reference for some basic commands.

• We could use systemd for monitoring as described in Using Systemd to monitor server’s processes instead of Monit and send emails on unit failures as descibred in Systemd status mail on unit failure but it comes with some drawbacks also mentioned in the articles. So instead monitor the system with Monit (see also Monit's documentation). This was done in Monitor system #17, in particular, 9cd8419
• Configure cron (see also https://wiki.debian.org/cron and https://man7.org/linux/man-pages/man8/cron.8.html) to send emails on errors. Use Chronic as a cure for Cron's chronic email problem. This was done in Monitor system #17, in particular, db21fa5
• Have cron not only send emails on errors but also write full logging output to journald like Docker does. This functionality was asked for in moreutils chronic - append full output to logfile? and implemented in An enhanced version of cronic. Alternatively, we could patch /usr/bin/chronic ourselves. This is the same as Keep cronjob logs in journald #15
• To monitor monit itself there is the paid service Cronitor. If we don't do this and monit dies for some reason, we won't know about it.
• To monitor the end-user experience use the paid service pingdom.

Sidenote: There is no native way to use variables from .env files in Ansible playbooks. For workarounds see https://graspingtech.com/ansible-environment-variables/ and https://gist.github.com/berkayunal/ccb1c3511f02d41b7654de17bced30b7