About save states #55
NaotoEnjoyer
started this conversation in
General
Replies: 2 comments
-
|
it actually seems like second time resuming from newer second save state it display for some reason in the log the epoch number first state was saved as, like that second resume showed the 12 epoch it ''continues'' from that was the last trained epoch in very first save state. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Hello @NaotoEnjoyer , I'm not super familiar with how save states work (or don't) in kohya's. They do seem to act a bit weird, I'm not sure why. Would be something to ask about on the original repo to see if anyone knows how it works. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
hey so i wonder, do save states in your fork can be used multiple times over? i mean for example i wanted to do training that cant be finished in two google colab sessions, i did one training session, got the save state, resumed in the second session all loaded and looks fine, it resumes from last save epochs and shows correct total epochs to train and all, this session finishes, i load up the second save state in third session and this time it loads fine it seems but in training log it display completely wrong epoch it train from, for example i loaded 24 epoch but in resuming it shows it goes from 13/30 epoch instead and so on.. is it usual behavior for save states when you do multiple resumes one after another in same training?
Beta Was this translation helpful? Give feedback.
All reactions