CompassPlus Settings

[keezinv2]

Hey hey!

I recently moved from the main repo to this fork, I wanted to ask if any of you have good settings for CompassPlus or other fun settings to try out.
I mostly train style based LORAs.

So far I really like this fork and everything works great.
Thanks if you will be able to help me I really appreciate it a lot!

[67372a]

Hey @keezinv2 , in general we have moved past CompassPlus, though you are certainly free to experiment. Compass itself is still an effective option, and another one is SimplifiedAdEMAMix (better AdEMAMix, less vram use, more stable).

You can give these settings a whirl:
caloart_nvp_fftdescent_v0_1.toml.txt

Notable points:

• Setup for vpred + ztsnr
• GLoRA network
• Uses FFTDescent optimizer, new optimizer from Clybius
• Lora+ applied
• edm2 with importance weighting, a modified version of edm2 based on recent work by Madman

This seems like an effective configuration, make sure you have the latest code pulled, as have made some fixes recently.

ProdigyPlus is also a good optimizer, and should work with these settings, with LRs adjusted to 1 as is typical with prodigy, and considering applying use_speed=True on the network for greater vram savings and faster adaption (possible instability combined with weight decay).

[Darudado]

A 5060ti. Maybe some dependency issue.

[keezinv2]

I would also like other examples of config! I stole this and changed it a bit to work to learn styles, it's stable, but the learning could be stronger. Maybe it helps someone.

vpredeasyDoRALycorisSCORNMachina.toml.txt

Hey hey!

Thanks for sharing the settings, I get one error when running them, I tried with both Torch_compile=False and True, but I get the same error.
Do you know how to fix it?

W0810 19:56:10.997000 23828 venv\Lib\site-packages\torch\distributed\elastic\multiprocessing\redirects.py:29] NOTE: Redirects are currently not supported in Windows or MacOs.
W0810 19:56:15.762000 11988 venv\Lib\site-packages\torch\distributed\elastic\multiprocessing\redirects.py:29] NOTE: Redirects are currently not supported in Windows or MacOs.
Traceback (most recent call last):
File "C:\Data\LoRA_Easy_Training_Scripts\backend\sd_scripts\sdxl_train_network.py", line 213, in
trainer.train(args)
File "C:\Data\LoRA_Easy_Training_Scripts\backend\sd_scripts\train_network.py", line 2779, in train
optimizer.step()
File "C:\Data\LoRA_Easy_Training_Scripts\backend\sd_scripts\venv\Lib\site-packages\accelerate\optimizer.py", line 179, in step
self.optimizer.step(closure)
File "C:\Data\LoRA_Easy_Training_Scripts\backend\sd_scripts\venv\Lib\site-packages\LoraEasyCustomOptimizer\RexAnnealingWarmRestarts.py", line 60, in wrapper
return func.get(opt, opt.class)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Data\LoRA_Easy_Training_Scripts\backend\sd_scripts\venv\Lib\site-packages\torch\optim\optimizer.py", line 485, in wrapper
out = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Data\LoRA_Easy_Training_Scripts\backend\sd_scripts\venv\Lib\site-packages\torch\utils_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Data\LoRA_Easy_Training_Scripts\backend\sd_scripts\venv\Lib\site-packages\LoraEasyCustomOptimizer\scornmachina.py", line 566, in step
if group['torch_compile']:
~~~~~^^^^^^^^^^^^^^^^^
KeyError: 'torch_compile'

[Darudado]

I would also like other examples of config! I stole this and changed it a bit to work to learn styles, it's stable, but the learning could be stronger. Maybe it helps someone.
vpredeasyDoRALycorisSCORNMachina.toml.txt

Hey hey!

Thanks for sharing the settings, I get one error when running them, I tried with both Torch_compile=False and True, but I get the same error. Do you know how to fix it?

W0810 19:56:10.997000 23828 venv\Lib\site-packages\torch\distributed\elastic\multiprocessing\redirects.py:29] NOTE: Redirects are currently not supported in Windows or MacOs. W0810 19:56:15.762000 11988 venv\Lib\site-packages\torch\distributed\elastic\multiprocessing\redirects.py:29] NOTE: Redirects are currently not supported in Windows or MacOs. Traceback (most recent call last): File "C:\Data\LoRA_Easy_Training_Scripts\backend\sd_scripts\sdxl_train_network.py", line 213, in trainer.train(args) File "C:\Data\LoRA_Easy_Training_Scripts\backend\sd_scripts\train_network.py", line 2779, in train optimizer.step() File "C:\Data\LoRA_Easy_Training_Scripts\backend\sd_scripts\venv\Lib\site-packages\accelerate\optimizer.py", line 179, in step self.optimizer.step(closure) File "C:\Data\LoRA_Easy_Training_Scripts\backend\sd_scripts\venv\Lib\site-packages\LoraEasyCustomOptimizer\RexAnnealingWarmRestarts.py", line 60, in wrapper return func.get(opt, opt.class)(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Data\LoRA_Easy_Training_Scripts\backend\sd_scripts\venv\Lib\site-packages\torch\optim\optimizer.py", line 485, in wrapper out = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "C:\Data\LoRA_Easy_Training_Scripts\backend\sd_scripts\venv\Lib\site-packages\torch\utils_contextlib.py", line 116, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "C:\Data\LoRA_Easy_Training_Scripts\backend\sd_scripts\venv\Lib\site-packages\LoraEasyCustomOptimizer\scornmachina.py", line 566, in step if group['torch_compile']: ~~~~~^^^^^^^^^^^^^^^^^ KeyError: 'torch_compile'

Hello, that optimizer has an issue actually, which I completely forgot. I couldn't fix it completely as I'm not too familiar with the repo, but I patched it to just work. It doesn't seem like the arg 'torch_compile' is stored and retrieved correctly, so when the dictionary is read you get an error. I patched it up replacing:
if group['torch_compile']:
with
if group.get('torch_compile', False):

So even if it doesn't exist in the dictionary it won't crash.
"C:\Data\LoRA_Easy_Training_Scripts\backend\sd_scripts\venv\Lib\site-packages\LoraEasyCustomOptimizer\scornmachina.py"
The file for you is here.
Sorry.

[67372a]

I fixed that just now, added the missing parameter with default.

@Darudado try making sure torch_compile is disabled at the optimizer, network, and trainer levels under the extra args inputs for each.

[Darudado]

I fixed that just now, added the missing parameter with default.

@Darudado try making sure torch_compile is disabled at the optimizer, network, and trainer levels under the extra args inputs for each.

The issue with "OverflowError: Python int too large to convert to C long" I didn't learn how to fix, but whatever.
Meanwhile the issue with torch_compile is fixed.

Do share config if something seems to work well.