Turn off game specific behaviors based on time#13
Turn off game specific behaviors based on time#13BartekCupial wants to merge 1 commit intoNetHack-LE:mainfrom
Conversation
|
Not a fan of just disabling it -- If people feel they want to have this part reproducible but setting the time to something static isn't possible in their setup (understandable on both counts) we'd need to add a hook to optionally override But also if the goal is to beat the real NetHack we should perhaps not dumb it down too much. |
|
Randomly pitching in (saw that NLE is getting some love again :) ). Agree with both here. The reliance on time is indeed very painful for running any experiments, and it is easy to miss unless you go deeper into Nethack functionality. This was a painful lesson during the NLE competition 😅 But also completely replacing it won't cut it, as it can be unexpected to people who know the logic. I would prefer informing users about this logic more in docs. Additionally, if "completing Nethack" is baked into Gym/Gymnsium-like environments, the environment should specify the same game-time with a hook heiner suggested. |
It turns out that the game is somehow different based on local time https://nethackwiki.com/wiki/Time. For reproducibility and evaluation reasons we shouldn't allow this changes. For example my agent score dropped from 5k to 3k just because it was full moon.
I turned off all four such cases. moon phases, friday 13th, midnight, and night.