addrwatch: init script not working

[jesusangel]

Maintainer: @ondrej Caletka
Environment: arm_cortex-a9_vfpv3, 18.06.4

Description:

Hi,

I'm trying to start the addrwatch service through its init script using PROCD. I can run the program manually with this command:

/usr/sbin/addrwatch --output /var/log/addrwatch --hashsize 1024 --ratelimit -1 br-lan

But when I run /etc/init.d/addrwatch start nothing happens. The service isn't started. I have enabled PROCD_DEBUG and this is the output I get on launching the service:

# /etc/init.d/addrwatch start
{ "name": "addrwatch", "script": "\/etc\/init.d\/addrwatch", "instances": { "instance1": { "command": [ "\/usr\/sbin\/addrwatch", "--output", "\/var\/log\/addrwatch", "--hashsize", "1024", "--ratelimit", "-1", "br-lan" ], "netdev": [ "br-lan" ], "respawn": [ "3600", "5", "5" ], "triggers": [ [ "interface.*", [ "if", [ "eq", "interface", "lan" ], [ "run_script", "\/etc\/init.d\/addrwatch", "reload" ] ], 1000 ] ], "pidfile": "\/var\/run\/addrwatch.pid" } }, "triggers": [ [ "config.change", [ "if", [ "eq", "package", "addrwatch" ], [ "run_script", "\/etc\/init.d\/addrwatch", "reload" ] ], 1000 ] ], "data": { } }

It seems right to me, but the service isn't started. I can't see anything about addrwatch in syslog.

This is the config file for addrwatch:

config addrwatch
        option disabled '0'
        list interface 'lan'
        option syslog '0'
        option output '/var/log/addrwatch'
        option ratelimit '-1'

Do you know what's wrong? How can I start the service with the init script?

Kind regards,

[neheb]

ping @jefferyto since you edited the init script.

[jefferyto]

Environment: arm_cortex-a9_vfpv3, 18.06.4

My edit of the init script didn't go into 18.06 😝

It seems right to me, but the service isn't started. I can't see anything about addrwatch in syslog.

option syslog '0' means do not log to syslog, option output '/var/log/addrwatch' means log to /var/log/addrwatch.

Can you check if the process is running using ps and check if there is any output in /var/log/addrwatch?

You can also change your config to have option syslog '1' (and no output option) and see if anything appears in syslog.

I tried running this in armvirt-64 (in qemu); the process appears to have started correctly. I didn't see any log output (in either log file or syslog), but none of my network addresses changed inside qemu.

[jesusangel]

Hi @jefferyto ,

I want addrwatch to log to a separate file instead of using the system log, thats why I disabled the syslog option and configured the external log file. The problem is that the addrwatch isn't started when I run /etc/init.d/addrwatch start. I have checked it with ps. However, I can run it with /usr/sbin/addrwatch --output /var/log/addrwatch --hashsize 1024 --ratelimit -1 br-lan.

Kind regards,

[jefferyto]

Can you enable the verbose option in /etc/config/addrwatch and see if any debug/error messages appear in the log file and/or syslog?

[jesusangel]

Hello,

Sorry it has taken so long for me to anwser you. I enabled the verbose option in the config file and I ran /etc/init.d/addrwatch start as root, but nothing happens. There are not any messages in the messages log nor in the addrwatch log file. It seems to me the process is not working when I try to start it from the init script. However, it works if I launch it from the command line with:

/usr/sbin/addrwatch --output /mnt/sda1/log/addrwatch --ipv4-only --hashsize 1024 --ratelimit -1 --daemon br-lan

I enabled the PROCD debug and here is the output:

{ "name": "addrwatch", "script": "\/etc\/init.d\/addrwatch", "instances": { "instance1": { "command": [ "\/usr\/sbin\/addrwatch", "--output", "\/mnt\/sda1\/log\/addrwatch", "--verbose", "--ipv4-only", "--hashsize", "1024", "--ratelimit", "-1", "br-lan" ], "netdev": [ "br-lan" ], "respawn": [ "3600", "5", "5" ], "triggers": [ [ "interface.*", [ "if", [ "eq", "interface", "lan" ], [ "run_script", "\/etc\/init.d\/addrwatch", "reload" ] ], 1000 ] ], "pidfile": "\/var\/run\/addrwatch.pid" } }, "triggers": [ [ "config.change", [ "if", [ "eq", "package", "addrwatch" ], [ "run_script", "\/etc\/init.d\/addrwatch", "reload" ] ], 1000 ] ], "data": { } }

Everything seems right, but the process is not starting after running /etc/init.d/addrwatch start.

Regards,

Jesús Ángel.

[jesusangel]

Hello again,

Here you are my init script, just in case:

#!/bin/sh /etc/rc.common
# Copyright (C) 2016 OpenWrt.org

START=50
USE_PROCD=1
PROCD_DEBUG=1

validate_section_addrwatch() {
        uci_validate_section addrwatch addrwatch "${1}" \
                'disabled:bool:0' \
                'interface:list(string):lan' \
                'syslog:bool:1' \
                'output:string' \
                'quiet:bool:0' \
                'verbose:bool:0' \
                'ipv4only:bool:0' \
                'ipv6only:bool:0' \
                'blacklist:list(or(ip4addr,ip6addr))' \
                'hashsize:range(1,65536):1024'\
                'ratelimit:integer:3600'
        return $?
}

start_instance() {
        local cfg="$1"
        local disabled interface syslog output quiet verbose ipv4only ipv6only
        local blacklist hashsize ratelimit
        local netdevs=""

        validate_section_addrwatch "${cfg}" || {
                echo "validation of config $cfg failed"
                return 1
        }
        [ $disabled -ne 0 ] && return 1

        for iface in $interface; do
                local netdev
                network_get_physdev netdev "${iface}"
                append netdevs "${netdev}"
        done

        procd_open_instance
        procd_set_param command /usr/sbin/addrwatch
        [ "${syslog}" -eq 1 ] && procd_append_param command --syslog
        [ -n "${output}" ] && procd_append_param command --output "$output"
        [ "$quiet" -eq 1 ] && procd_append_param command --quiet
        [ "$verbose" -eq 1 ] && procd_append_param command --verbose
        [ "$ipv4only" -eq 1 ] && procd_append_param command --ipv4-only
        [ "$ipv6only" -eq 1 ] && procd_append_param command --ipv6-only
        [ -n "$hashsize" ] && procd_append_param command --hashsize "$hashsize"
        [ -n "$ratelimit" ] && procd_append_param command --ratelimit "$ratelimit"
        for blitem in $blacklist; do
                procd_append_param command "--blacklist=${blitem}"
        done
        procd_append_param command $netdevs
        procd_set_param netdev $netdevs
        procd_set_param respawn
        procd_open_trigger
        for iface in $interface; do
                procd_add_interface_trigger "interface.*" ${iface} /etc/init.d/addrwatch reload
        done
        procd_close_trigger
        procd_set_param pidfile /var/run/addrwatch.pid
        procd_close_instance
}

start_service() {
        . /lib/functions/network.sh
        config_load 'addrwatch'
        config_foreach start_instance 'addrwatch'
}

service_triggers() {
        procd_add_reload_trigger 'addrwatch'
}

[jefferyto]

Please enable both syslog and verbose, with no output and try again. Also please post the contents of /etc/config/addrwatch.

[t-w]

Hi,
I have just stepped on same (or very related) issue. I see it was left for a while so I look a bit myself and found something that may cause this.

If I understand well and the command

procd_append_param command --syslog

in the start_instance() adds the --syslog argument to the executed addrwatch command. I just had a look at the sources of addrwatch, and it seems to me that there is no --syslog option (see addrwatch.c. It was removed in this commit in 2013 and syslog is available in a different way(?).

So basically, the configuration and the init script for addrwatch seem broken. Setting syslog to 0 will probably solve the problem for the submitter - but the package does not seem to allow for sending logs to syslog (and to other machine, which is my case...). Any chance for fixing this?

Further comment is a bit on a side - I know this may not be the place to discuss this, but I guess it may help resolve this and similar issues:

Is there a way to see exactly the complete command (with options) that procd is executing starting a service? It seems necessary for debugging. The JSON stuff available after setting PROCD_DEBUG (as shown above) does not contain the complete command, only the overrides from the config file. For this case - syslog is set to 1 by default in the init script, so you do not see it there.

Also, complete output from the executed command would be useful when debugging - when you try:

# addrwatch --syslog

you can immediately see:

addrwatch: unrecognized option '--syslog'

If this was visible in the logs (at least after enabling some debug option) while starting the service, it would be helpful.

Is there a way to enable such debug mode?

[jefferyto]

@t-w

Setting syslog to 0 will probably solve the problem for the submitter

If the original poster installed addrwatch from the 18.06 packages feed, then they are using version 0.8 which does have a --syslog option. You can also see in their original comment that they have already set syslog to 0.

I just had a look at the sources of addrwatch, and it seems to me that there is no --syslog option

You are right that the --syslog option was removed in v1.0; this affects the package in 19.07 and master/21.02.

v1.0 switched to a modular architecture where monitoring output is placed in shared memory. Output modules then read this shared memory and output the information.

The commit you referenced removed the syslog logging of monitoring output from core. This functionality was moved into the addrwatch_syslog module.

This only affects the monitoring output though. Other logging messages continue to be sent to syslog.

The JSON stuff available after setting PROCD_DEBUG (as shown above) does not contain the complete command, only the overrides from the config file. For this case - syslog is set to 1 by default in the init script, so you do not see it there.

AFAIK the procd debug information contains the complete command line. procd has no knowledge of any defaults or overrides; it is the init file that reads the config file and adds command line options as necessary. If the command line options you see in the procd debug information are incorrect, you should check your config file and/or the init script.

[jefferyto]

@jesusangel What version of addrwatch are you using? Did you install it from the 18.06 packages feed or from somewhere else (e.g. compiled the package yourself)?

[t-w]

@jesusangel What version of addrwatch are you using? Did you install it from the 18.06 packages feed or from somewhere else (e.g. compiled the package yourself)?

(Thanks for quick replies!).

addrwatch 1.0.1-2, openwrt 19.07.7, I have just installed the package (I have just started with openwrt, so I haven't looked at building things for it, yet...).

Concerning the config. defaults - addrwatch is not starting (using its init script) unless I explicitly set option syslog '0' in my config - so there must be a default set somewhere. My guess is that it is done in init.d/addrwatch: validate_section_addrwatch(), because when I change the setting for syslog there then the addrwatch starts even when I comment-out this option in config. Otherwise it does not (as mentioned before). So it has to be at least one of the places concerned.

This is what is logged on a failed start:

Jun 17 12:07:09 OpenWrt procd: Instance addrwatch::instance1 s in a crash loop 6 crashes, 0 seconds since last crash

(so btw. there is no output from addrwatch - what would probably give some info about the reason of failure).

This is when addrwatch's syslog option is set to off:

==> /var/log/daemon.log <==
Jun 17 12:10:37 OpenWrt addrwatch: DEBUG: Ratelimiting duplicate entries to 1 per 3600 seconds
Jun 17 12:10:37 OpenWrt addrwatch: DEBUG: PROMISC mode enabled

==> /var/log/kern.log <==
Jun 17 12:10:37 OpenWrt kernel: [240477.533450] device br-lan entered promiscuous mode

==> /var/log/daemon.log <==
Jun 17 12:10:38 OpenWrt addrwatch: DEBUG: Opened interface br-lan (Ethernet)
Jun 17 12:10:38 OpenWrt addrwatch: DEBUG: Saving results to '/var/log/addrwatch' flat file

What I would like to have is the updates of addresses to be logged there as well.

Since the addrwatch architecture was changed (to that modular one) - is it possible to use the version 1.0.1 of the package with the syslog then? (by configuring it, not hacking in some non-standard way).

Btw I have noticed one more issue (I do not want throw in to much at the same time... but this is also config / init script for the package and maybe feedback on this will allow to improve it).
I want to monitor 2 interfaces (eth1.2 and br-lan) and despite having both defined in the config:

config addrwatch
[...]
	list interface 'br-lan'
	list interface 'eth1.2'
[...]

the ps w shows that the execution command line was:

18064 root      3404 S    /usr/sbin/addrwatch --output /var/log/addrwatch --verbose --hashsize 1024 --ratelimit 3600

and addrwatch seems to monitor only 1 interface (br-lan - as also seen in the logs above). From command-line, it can be done as follows:

# addrwatch eth1.2 br-lan

so it seems the interface list is not passed to the program.

My complete config:

config addrwatch
	option disabled '0'
	list interface 'br-lan'
	list interface 'eth1.2'
	option syslog '0'
	option verbose '1'
	option output '/var/log/addrwatch'

[jesusangel]

@jesusangel What version of addrwatch are you using? Did you install it from the 18.06 packages feed or from somewhere else (e.g. compiled the package yourself)?

Hi! I installed addrwatch from the 18.06 packages feed.

[jefferyto]

@t-w

addrwatch is not starting (using its init script) unless I explicitly set option syslog '0' in my config - so there must be a default set somewhere. My guess is that it is done in init.d/addrwatch: validate_section_addrwatch()

Yes, if the syslog option isn't specified it defaulted to being enabled (that's what the :1 at the end means).

Since the addrwatch architecture was changed (to that modular one) - is it possible to use the version 1.0.1 of the package with the syslog then? (by configuring it, not hacking in some non-standard way).

I don't think so. If #15899 is merged I plan on backporting it for 19.07, although there may still be issues with the syslog output.

I want to monitor 2 interfaces (eth1.2 and br-lan) and despite having both defined in the config:

The init script expects uci interface names (lan, wan, etc.) as set in /etc/config/network, not network device names.

[jefferyto]

@jesusangel I see that you marked your forum post as solved - does that mean you were able to get addrwatch running using the accepted solution?