Support PiD (Pixel diffusion Decoder)

[anr2me]

Feature Idea

Project website https://research.nvidia.com/labs/sil/projects/pid/

TL;DR: PiD directly decodes latent representations into high-resolution images, replacing the decode–then–super-resolve cascade while achieving lower latency and higher visual quality.

Existing Solutions

No response

Other

No response

[kijai]

So I already did this and it works well, and even is good... then I noticed the license:
https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-internal-scientific-research-and-development-model-license/

This part is troublesome:
The Model and any Derivative Model may not be distributed, deployed, sublicensed, publicly displayed, publicly performed, or sublicensed by You.

This would currently mean we could only support using their .pth models, and also they restrict any commercial use as well.

Edit: License has been changed to bit more permissive nsclv1

[anr2me]

Hmm.. isn't the license a typical Apache License 2.0 🤔 https://github.com/nv-tlabs/PiD#license

Edit: Ah the model use a different license 😔 https://huggingface.co/nvidia/PiD#licenseterms-of-use

I guess users will need to get the models themselves (whether it's the original pth or safetensors converted by third-party).

[kijai]

Yeah, the T2I PixelDiT model uses the old license which does allow redistributing at least, so I can add support for it and repackage that model + text encoder that both models use, then let it be user's responsibility to deal with the PiD.

License was changed, models are here with (in my opinion) clearer naming:

https://huggingface.co/Comfy-Org/PixelDiT

[kijai]

Draft PR up:
#14103

[Antanas13]

Might be wrong here but I noticed the current version of PiD Conditioning node has Flux1 and sd3 latent decode selection but no Flux2 latent as a option at all and if i learned anything lately it's what Flux2 latents are VERY much not the same as Flux1 latents and should be treated with extreme care, especially if this thing extracts any sort of semantic or spatial information from latents it then uses in it's later processes - there's probably also a valid reason why Nvidia's guys included Flux2 latent alongside Flux1 and Sd3 latents in their official repo

[kijai]

Might be wrong here but I noticed the current version of PiD Conditioning node has Flux1 and sd3 latent decode selection but no Flux2 latent as a option at all and if i learned anything lately it's what Flux2 latents are VERY much not the same as Flux1 latents and should be treated with extreme care, especially if this thing extracts any sort of semantic or spatial information from latents it then uses in it's later processes - there's probably also a valid reason why Nvidia's guys included Flux2 latent alongside Flux1 and Sd3 latents in their official repo

Which is also why it's easy to autodetect (16 vs 128 channels) and no need to have separate selection, as mentioned in the tooltip on the node. Sd3 is 16 as well, so it needs to be selected, no way to differentiate it from latent only.

[Antanas13]

True enough on the autodetection side — my concern was more about whether Flux2 latents might need different handling semantically inside the PiD conditioning path itself, not the UI selection.

From my Flux2 VAE/tiled decode investigations lately, Flux2 latents seem extremely sensitive to interpolation and latent-space operations compared to older 16-channel latent formats. Even simple nearest/bilinear/bicubic latent resizing causes severe coherence and color degradation after decode.

That made me wonder whether the current processing and latent routing assumptions inside PiD conditioning are fully equivalent between Flux1 and Flux2, especially since NVIDIA separated Flux1 / Flux2 / SD3 handling explicitly in their official PiD repo, let alone trained specific models for each of these latents, instead of treating Flux2 as just “Flux with more channels” and making some adaptations for that.

I might be completely wrong here, but it feels like Flux2 latent structure may carry substantially denser spatial/semantic organization than earlier\legacy latent formats, so I figured it was worth asking.

[kijai]

True enough on the autodetection side — my concern was more about whether Flux2 latents might need different handling semantically inside the PiD conditioning path itself, not the UI selection.

From my Flux2 VAE/tiled decode investigations lately, Flux2 latents seem extremely sensitive to interpolation and latent-space operations compared to older 16-channel latent formats. Even simple nearest/bilinear/bicubic latent resizing causes severe coherence and color degradation after decode.

That made me wonder whether the current processing and latent routing assumptions inside PiD conditioning are fully equivalent between Flux1 and Flux2, especially since NVIDIA separated Flux1 / Flux2 / SD3 handling explicitly in their official PiD repo, let alone trained specific models for each of these latents, instead of treating Flux2 as just “Flux with more channels” and making some adaptations for that.

I might be completely wrong here, but it feels like Flux2 latent structure may carry substantially denser spatial/semantic organization than earlier\legacy latent formats, so I figured it was worth asking.

This is a question for the model devs, the implementation here matches their inference code. They did say they are releasing better flux2 model version already, so I'm not saying you're wrong, just that our implementation is correct.

[Antanas13]

Thanks for taking time to answer me - the PiD thing really piqued my interest and because i am primarily a Flux2 and it's derivatives (Klein Lens Ernie) user i became a bit over-obsessed over it because of the flux2 issues it currently has in this incarnation, so your reply will hopefully calm me down - for a time at least ))))