Client caused black screen BUG

When I start the client:
client -run 3 -gpu 0 -gpu 1 -report-gpu -report-host -user (name) -password (name)

it works fine for some minutes but then the client caused a black screen.
The machine is still running but no games are played.
No other things are possible and I need to restart the pc.

I have exactly the same gpus.
And no problems with fritz gui or chessbase 15 gui.

When running the client I see that it is using only one gpu and not both - how to fix this?
I can see it with msi afterburner and with gpu z.

Do I need parallelism?
Are there other things which I can also use with cmd?

Do we have something like logfile.txt when running the client?

[mooskagh]

As mentioned in Discord, that's most likely either too weak PSU for two GPUs, or poor cooling of GPUs.
Both is quite common in dual GPU systems.

I have a Corsair AX1600i PSU.
And lots of high end air cooler.
One GPU has only 74 degrees celsius.
The other one has 50.

The problem happens always, no matter if I use -gpu 0 -gpu 1 or only one gpu.

[borg323]

You didn't specify which gpu you have and what backend is being used. No matter, the client is unlikely to be causing this, but the way lc0 is called may be loading the gpu too much. Try lowering the parallelism to 4 (or less) to see if this help.
Finally, note that the client only uses one gpu by default. Adding a second -gpu to the command line just overrides the first. If you want to run on both gpus, the most efficient way is to run a second client instance, with a different -gpu number.

I have 2x the same RTX 2080 Ti.
I'm running the client with this:
client -run 3 -gpu 0 -gpu 1 -report-gpu -report-host -user (name) -password (name)
As you can see no backend and no parallelism.

Okay then I will use 2 clients instead of one. But I still have the same problem.
Also note that I think to most people don't know that the second gpu overrides the first gpu, when only one client is in use.
So this should be also fixed.

When using chess guis I use: (backend=cudnn-fp16,gpu=0),(backend=cudnn-fp16,gpu=1)
and roundrobin.

[borg323]

I asked about backend in case you were using the relatively new dx12, but I see you use cudnn-fp16 so this shouldn't be an issue.
Is this only happening on run 3?

No this happens also when running run 1 or 2.

I have the same problem when I use a power limit of 40%.

[borg323]

When you run the client, the output near the top contains the exact lc0 command line used. Can you try this on its own to confirm the client has nothing to do with this?
Example from an old log I had: /content/lc0/build/lc0 selfplay --backend-opts=backend=cudnn-fp16 --parallelism=32 --visits=10000 --cpuct=2.5 --cpuct-at-root=2.5 --root-has-own-cpuct-params=true --resign-percentage=4.0 --resign-playthrough=20 --temperature=0.90 --temp-endgame=0.75 --temp-cutoff-move=16 --temp-visit-offset=-0.8 --fpu-strategy=absolute --fpu-value=-1.0 --fpu-strategy-at-root=absolute --fpu-value-at-root=1.0 --minimum-kldgain-per-node=0.000012 --policy-softmax-temp=1.2 --resign-wdlstyle=true --training=true --weights=client-cache/3eb9d62ecc6aa2a84b7cdb789c50702a02477cf969949cf7ed788b71a3ea9cfa

@borg323
What exactly do you want me to do?

On my machine it looks like this:
Z:\LC0>client -run 3 -gpu 0 -report-gpu -report-host -user (name) -password (name)
Lc0 client version 26
2020/05/08 15:00:09 lc0_main.go:956: serverParams: [--visits=10000 --cpuct=1.32
--cpuct-at-root=1.9 --root-has-own-cpuct-params=true --resign-percentage=4.0 --r
esign-playthrough=20 --temperature=0.8 --temp-endgame=0.30 --temp-cutoff-move=60
--temp-visit-offset=-0.8 --fpu-strategy=reduction --fpu-value=0.23 --fpu-strate
gy-at-root=absolute --fpu-value-at-root=1.0 --minimum-kldgain-per-node=0.000040
--policy-softmax-temp=1.4 --resign-wdlstyle=true --noise-epsilon=0.1 --noise-alp
ha=0.12 --sticky-endgames=true --openings-pgn=books/960fen.pgn --openings-mode=s
huffled --moves-left-max-effect=0.2 --moves-left-threshold=0.0 --moves-left-slop
e=0.009 --moves-left-quadratic-factor=1.0 --moves-left-constant-factor=0.0]
Args: [Z:\LC0/lc0.exe selfplay --backend-opts=backend=cudnn-fp16,gpu=0 --paralle
lism=32 --visits=10000 --cpuct=1.32 --cpuct-at-root=1.9 --root-has-own-cpuct-par
ams=true --resign-percentage=4.0 --resign-playthrough=20 --temperature=0.8 --tem
p-endgame=0.30 --temp-cutoff-move=60 --temp-visit-offset=-0.8 --fpu-strategy=red
uction --fpu-value=0.23 --fpu-strategy-at-root=absolute --fpu-value-at-root=1.0
--minimum-kldgain-per-node=0.000040 --policy-softmax-temp=1.4 --resign-wdlstyle=
true --noise-epsilon=0.1 --noise-alpha=0.12 --sticky-endgames=true --openings-pg
n=books/960fen.pgn --openings-mode=shuffled --moves-left-max-effect=0.2 --moves-
left-threshold=0.0 --moves-left-slope=0.009 --moves-left-quadratic-factor=1.0 --
moves-left-constant-factor=0.0 --training=true --weights=client-cache\fdf4c93b57
96723fd1ec88b09dcc92474a727a582ebf028ece402eb6fe50c3a9]
_
| _ | |
|_ |_ |_| v0.25.1+git.69105b4 built Apr 30 2020
id name Lc0 v0.25.1+git.69105b4
id author The LCZero Authors.
Loading weights file from: client-cache\fdf4c93b5796723fd1ec88b09dcc92474a727a58
2ebf028ece402eb6fe50c3a9
Creating backend [multiplexing]...
Creating backend [cudnn-fp16]...
CUDA Runtime version: 10.0.0
Cudnn version: 7.4.2
Latest version of CUDA supported by the driver: 10.1.0
GPU: GeForce RTX 2080 Ti
GPU memory: 11 Gb
GPU clock frequency: 1545 MHz
GPU compute capability: 7.5
PGN: [FEN "bnrnkbqr/pppppppp/8/8/8/8/PPPPPPPP/BNRNKBQR w KQkq - 0 1"]

[borg323]

Then the command to run would be:
Z:\LC0\lc0.exe selfplay --backend-opts=backend=cudnn-fp16,gpu=0 --parallelism=32 --visits=10000 --cpuct=1.32 --cpuct-at-root=1.9 --root-has-own-cpuct-params=true --resign-percentage=4.0 --resign-playthrough=20 --temperature=0.8 --temp-endgame=0.30 --temp-cutoff-move=60 --temp-visit-offset=-0.8 --fpu-strategy=reduction --fpu-value=0.23 --fpu-strategy-at-root=absolute --fpu-value-at-root=1.0 --minimum-kldgain-per-node=0.000040 --policy-softmax-temp=1.4 --resign-wdlstyle=true --noise-epsilon=0.1 --noise-alpha=0.12 --sticky-endgames=true --openings-pgn=books/960fen.pgn --openings-mode=shuffled --moves-left-max-effect=0.2 --moves-left-threshold=0.0 --moves-left-slope=0.009 --moves-left-quadratic-factor=1.0 --moves-left-constant-factor=0.0 --training=true --weights=client-cache\fdf4c93b5796723fd1ec88b09dcc92474a727a582ebf028ece402eb6fe50c3a9

@borg323
I tried to run the command and got this:

Z:\LC0\lc0.exe selfplay --backend-opts=backend=cudnn-fp16,gpu=0 -
-parallelism=32 --visits=10000 --cpuct=1.32 --cpuct-at-root=1.9 --root-has-own-c
puct-params=true --resign-percentage=4.0 --resign-playthrough=20 --temperature=0
.8 --temp-endgame=0.30 --temp-cutoff-move=60 --temp-visit-offset=-0.8 --fpu-stra
tegy=reduction --fpu-value=0.23 --fpu-strategy-at-root=absolute --fpu-value-at-r
oot=1.0 --minimum-kldgain-per-node=0.000040 --policy-softmax-temp=1.4 --resign-w
dlstyle=true --noise-epsilon=0.1 --noise-alpha=0.12 --sticky-endgames=true --ope
nings-pgn=books/960fen.pgn --openings-mode=shuffled --moves-left-max-effect=0.2
--moves-left-threshold=0.0 --moves-left-slope=0.009 --moves-left-quadratic-facto
r=1.0 --moves-left-constant-factor=0.0 --training=true --weights=client-cache\fd
f4c93b5796723fd1ec88b09dcc92474a727a582ebf028ece402eb6fe50c3a9
_
| _ | |
|_ |_ |_| v0.25.1+git.69105b4 built Apr 30 2020
id name Lc0 v0.25.1+git.69105b4
id author The LCZero Authors.
Loading weights file from: client-cache\fdf4c93b5796723fd1ec88b09dcc92474a727a58
2ebf028ece402eb6fe50c3a9
Unhandled exception: Cannot read weights from client-cache\fdf4c93b5796723fd1ec8
8b09dcc92474a727a582ebf028ece402eb6fe50c3a9

Also I got a little taskmanager window with the information: lc0.exe doesn't work anymore.

[borg323]

Probably you are not running lc0 from the same directory the client (and lc0) are in. I assume this is Z:\LC0. There should be books and client-cache subdirectories, the first one containing 960fen.pgn and the second one containing fdf4c93b5796723fd1ec88b09dcc92474a727a582ebf028ece402eb6fe50c3a9.

@borg323

I have inside Z:\LC0
lc0.exe and client.exe and the other basic lc0 things. Also the books and client-cache subdirectories are there. 960fen.pgn is inside books and inside client-cache I have the right fdf4c93b5796723fd1ec88b09dcc92474a727a582ebf028ece402eb6fe50c3a9

I want to donate gpus to -run 3.
That's why I open an empty cmd and pasted inside what you have wrote me before.

[borg323]

We appreciate it, but first we need to figure out what is causing the problem. Here is the procedure:
Open a cmd window and then type:

Z:
CD \LC0

This will take you to the LC0 directory, and then run the command I gave earlier. I expect it will have the same problem We can then try to modify the command to see if we can isolate the issue.

@borg323

I have typed:
Z:
CD \LC0
and then the command to run, which I've got from you.
I will tell you later when it crashes again.
This is how it looks now:

Z:\LC0>Z:\LC0\lc0.exe selfplay --backend-opts=backend=cudnn-fp16,gpu=0 --paralle
lism=32 --visits=10000 --cpuct=1.32 --cpuct-at-root=1.9 --root-has-own-cpuct-par
ams=true --resign-percentage=4.0 --resign-playthrough=20 --temperature=0.8 --tem
p-endgame=0.30 --temp-cutoff-move=60 --temp-visit-offset=-0.8 --fpu-strategy=red
uction --fpu-value=0.23 --fpu-strategy-at-root=absolute --fpu-value-at-root=1.0
--minimum-kldgain-per-node=0.000040 --policy-softmax-temp=1.4 --resign-wdlstyle=
true --noise-epsilon=0.1 --noise-alpha=0.12 --sticky-endgames=true --openings-pg
n=books/960fen.pgn --openings-mode=shuffled --moves-left-max-effect=0.2 --moves-
left-threshold=0.0 --moves-left-slope=0.009 --moves-left-quadratic-factor=1.0 --
moves-left-constant-factor=0.0 --training=true --weights=client-cache\fdf4c93b57
96723fd1ec88b09dcc92474a727a582ebf028ece402eb6fe50c3a9
_
| _ | |
|_ |_ |_| v0.25.1+git.69105b4 built Apr 30 2020
id name Lc0 v0.25.1+git.69105b4
id author The LCZero Authors.
Loading weights file from: client-cache\fdf4c93b5796723fd1ec88b09dcc92474a727a58
2ebf028ece402eb6fe50c3a9
Creating backend [multiplexing]...
Creating backend [cudnn-fp16]...
CUDA Runtime version: 10.0.0
Cudnn version: 7.4.2
Latest version of CUDA supported by the driver: 10.1.0
GPU: GeForce RTX 2080 Ti
GPU memory: 11 Gb
GPU clock frequency: 1545 MHz
GPU compute capability: 7.5
gameready trainingfile Z:\LC0/data-hyepracghopp/game_000029.gz gameid 29 play_st
art_ply 0 player1 white result blackwon moves b2b3 g7g5 b1c3 f7f5 g2g4 f5f4 e2e3
b8c6 d2d4 e8g6 f1e2 e7e6 d1b2 d7d5 e2d2 c6b4 c1a1 g6c2 a2a3 c2d1 d2d1 b4c6 g1g3
from_fen rnknbqrb/pppppppp/8/8/8/8/PPPPPPPP/RNKNBQRB w KQkq - 0 1
tournamentstatus P1: +0 -1 =0 LOS: 15.87% P1-W: +0 -1 =0 P1-B: +0 -0 =0 npm 600.
875000 nodes 14421 moves 24
gameready trainingfile Z:\LC0/data-hyepracghopp/game_000001.gz gameid 1 play_sta
rt_ply 0 player1 white result whitewon moves d2d3 c7c6 f1g3 f8e6 b2b4 d8c7 e2e4
a7a5 b4b5 g8f6 e4e5 f6d5 a2a4 e8g8 g1f3 d7d6 e5d6 c7d6 e1g1 b7b6 f1e1 a8b7 g3f5
b7c7 e1e6 d6h2 from_fen qrbbknnr/pppppppp/8/8/8/8/PPPPPPPP/QRBBKNNR w KQkq - 0 1