phyreg hanging when PHY not properly initialized

[pepeki]

Hey Josh!
I have been testing your fix on our BBB setup and got to the point where it hangs in retrieving the registry values.

Launched directly from the command line - the output looks like this:

root@blackbone-board:~# phyreg test 18 13
Alive bits:0000-0000-0000-0100
First PHY found at address 2.
PHY=02 REG=18 : IDLE READ

And it never finishes...

When the PHY is initialized properly and ETH is working just fine - the phyreg command finishes and exits properly.

Any idea?

[bigjosh]

Based on the printout you have shown, it looks like it is getting stuck on this line...

while (*useraccessaddress & MDIO_USERACCESS0_GO_BIT);

This is waiting for the MDIO bus to complete the transaction and clear the GO bit. In your case for some reason the bit seams to not be getting cleared ever.

The specs for the AM335X inside the BBB do not mention the possibility of this outcome for a read...

I have never seen this before. Is there anything special about your setup that might give clues where to look for the problem? Have you seen this on multiple boards, or just one?

Thanks!

[pepeki]

Josh, first of: thank you for your very prompt response!!!
I have been testing this fix only on one board so far.

Well - about anything special: the OS on BB is not the official Debian distribution. We build our own OS image via Yocto 2.2 (morty) environment. It's based on linux-ti-staging 4.4 kernel, with some minor patches related to PRU.
https://github.com/AdrianPeniak/meta-blackbone

[bigjosh]

Ok, this looks like an issue with the 4.x kernels locking the MDIO registers.
I don't uses the 4.x kernels because of some bad experiences I've had with the new PRUSS subsystem (sounds like you are working around these?), but I expect someone will send me a PR with a patch to support these on phyreg soon. I'll report back when that happens!

[pepeki]

Mmm. Thanks for your explanation! We are stuck with 4.x for now... :(
We could leverage /sys/class/net/eth0/operstate with some intelligence for the failed state signaling.
However, we also gave a try to bbbrtc utility of yours for the reboot scheduling - and not much success there either.
The routine gets through w/o any apparant errors - but we get no sleep or wake out of it.
Do you think this is also related to the kernel version?
I believe this code uses different registers than phyreg. Am I right?
Thx!
Peter

[AdrianPeniak]

Hello @bigjosh ,
I have a different question. For something the following line while ( get32reg( base , RTC_STATUS_REG) & RTC_STATUS_RUN ) wait_counter++; is in my case endless. Therefore I limited overflow of counter, but now I see strange (random) printf related to program argument number. As you can see here:

# bbbrtc now 100
Opening /dev/mem...opened.
Mappng in 0x44e3e000...mapped at address 0xb6fe1000.
Stopping RTC...waiting for stop...took 4294967295 tries.
Checking RTC chip rev...checks good.
Setting NOW to 100...set.
WARNING: never does not really make sense here! Nothing done!
Reading NOW and printing to stdout...
1528816393
done.
Restarting RTC...waiting for it to start...took 0 tries.
Unmaping memory block...unmaped.
Closing fd...closed.

Do you have any suggestion why is loop endless and why I can't read-back real number from register?
PS: I using custom image based on 4.4-ti kernel.
Many thanks for help.
Adrian

[bigjosh]

This is almost certainly the same 4.x kernel locking issue. The reason that while() never exists is because the bits never got set, so putting a timeout on there does not help.

At least one person is working on figuring out how to get around this in the 4.x kernels, so maybe a solution soon. In the meantime I would really consider using a 3.x kernel on BBB unless there is something you really need that is only in the 4.x line.

[bigjosh]

@pepeki Yes it is likely that the 4.x is mucking with access to the RTC regs as well. This is easy to test, you could just try using bbbrtc to set one of the times, and then use it again to check that time and see if it was actually set. If not, then the register writes are not working - either because of 4.x kicker locking or devmem blocking or both.

[abferm]

@bigjosh

Interestingly I don't believe this is a kick register issue for phyreg. If I compile phyreg on a beaglebone (running a 4.x kernel on debian jessie) it appears to work in all situations. If I cross compile phyreg as is from an x86_64 machine line 209 of phyreg.c hangs every time, however if I add a body to the while loop with either a print or short sleep it works when cross compiled with a single execution of what is in the loop body. My best guess is that the native compiler automatically adds a delay and without it devmem locks up. The easiest solution is probably to change all of your while(...); statements to something like while(...){usleep(30); fprint(stderr, ".");}. Printing something to the stderr would help convey to the user that the program is waiting for something.

[bigjosh]

Interesting! Might be fun someday to look at the emitted ARM code and try to really pin down what the difference is that makes it hang.

[abferm]

Specifying -O0 also appears to clear up the issue for me without any changes to the code.

[romainreignier]

I have encountered the same issue on a 4.9-ti kernel built with buildroot so with -Os optimization.

Out of curiosity, I have used Compiler Explorer to get the assembly output: https://godbolt.org/z/7Q3rCm

With -O0:

        ldr     r3, [fp, #-16]
        ldr     r3, [r3]
        lsr     r3, r3, #31
        and     r3, r3, #255
        cmp     r3, #0
        beq     .L7
        b       .L8

With -Os:

        ldr     r3, [r5]
        cmp     r3, #0
        blt     .L6

So, from my basic understanding, the optimized version only compares the content of r5 instead of the data at the given address as -O0 does.

BUT, adding:

{
    usleep(30);
    fprintf(stderr, ".");
}

Does not seem to change anything in that logic.

Anyway, with O0, Os and with the usleep, none of the solutions works for me when the PHY is in bad state, while it works when everything is ok.

I might try with grepping dmesg for the PHY not found.
Reading /sys/class/net/eth0/operstate as @pepeki suggested might result in boot loops if Ethernet cable is disconnected.

[romainreignier]

Actually, I have the same issue that @AdrianPeniak and bigjosh/bbbrtc#1, even with bigjosh/bbbrtc@3867056

Calling bbb-long-reset the first time hangs. After a Ctrl+C and calling it a second time, the reboot works. From the logs, bbbrtc unlock is called before.

Manually calling bbbrtc unlock several times before does not help.

I know that you, @bigjosh , does not care about 4.x kernels, but it is more for other users that could have the same issue.

I use kernel 4.9.69 from TI.

[romainreignier]

For your information, after further tests, I have seen that on latest beagleboard.org image (2019-08-03), I have managed to make more than 300 successful boots while I usually had failures before 40 boots on my Buildroot system.

So I have used mainline uBoot 2019.10 which contains this patch and applied this patch to my ti 4.9.69 kernel.

With this combo, I have managed to get more than 3600 successful boots so it seems to me that I do not need bbbphyfix.

Anyway thank you very much @bigjosh for all your work and documentation to precisely explain the problem. I have learned a lot and it is very appreciated.