Is the AlphaZero implementation reliable?

[Egiob]

Hey @sotetsuk, I wanted to run a series of experiments with AlphaZero and wanted to start from the implementation provided here.

However, I'm a little bit worried about the note in the README:

pgx/examples/alphazero/README.md

Lines 7 to 8 in 87278d2

	> [!NOTE]
	> This implementation of AlphaZero demonstrates sufficient learning performance in environments including 9x9 Go, but it has some slight differences in learning details compared to the original AlphaZero and Gumbel AlphaZero. An implementation that addresses these differences and focuses on enhanced efficiency is currently under development and is expected to be released shortly.

Is this still true or has the "corrected" version already been implemented? What are the potential discrepancies that you have in mind? I would like to make the baseline as reliable and strong as possible. Do you have recommendations for that?

Thanks a lot 😺

[Francesco215]

Yes it does! I've replicated it for Go 9x9 and Hex, although you probably should change the eval code. The current one is not complete

[Egiob]

Hello @Francesco215, what do you mean by replicated? Did you get good results against other implementations/agents or just improvement against previous checkpoints?

[Francesco215]

On Hex i've been able to reach optimal play. I've used MoHex as the optimal play baseline

On go 9x9 and gardner chess I just reproduced the results they claim against their non-optimal play baselines

I've made all of this in my private repository, but if your aim is just to reproduce their results, i suggest you to start with their example alphazero code

[Egiob]

Thanks for the answer. I'm interested to know the parameters you used for Hex ( 11x11 ?) and Go (9x9) if you don't mind to share !

[Francesco215]

For Hex you can follow this paper here https://arxiv.org/pdf/2104.03113

For Go I've followed the official pgx wandb report https://api.wandb.ai/links/sotetsuk/979hmps8

[Egiob]

Ok so you achieved the 50% win rate against MoHex on 9x9 Hex using the paper's params and pgx code? Do we have an idea of how good MoHex(9x9) is in a losing position? I made extensive experiments with MoHex(7x7) and it was playing not better than random when he was conviced to lose (he couldn't put up a good fight in losing positions although he would punish perfectly any mistake).

[Francesco215]

Ok so you achieved the 50% win rate against MoHex on 9x9 Hex using the paper's params and pgx code?

yes (btw you should change the eval code. The pgx one does not use search)

Do we have an idea of how good MoHex(9x9) is in a losing position? I made extensive experiments with MoHex(7x7) and it was playing not better than random when he was conviced to lose (he couldn't put up a good fight in losing positions although he would punish perfectly any mistake).

This is to be expected. An optimal play agent assumes that his opponent is an optimal play agent too. In a losing position he figures out that no matter what he does, he's going to loose anyways-- hence the random-looking moves.

In order to make the model put a good fight you should ask him to prolong the game as much as possible. As far as I know, Stockfish does something similar in forced-mate positions: if it cannot avoid losing, it chooses the line that delays mate the longest (hoping the opponent makes a mistake)

[Egiob]

yes, what I meant is that in the case of a "weak" oracle, the strong claim of achieving optimal play is somewhat weakened by the fact the oracle is not even trying to make the game as long as possible.

if you're okay to share your code and runs (metrics, params) it'd be very cool :)