Support for Flux Controls + Flex.2 #692
Conversation
d8eadb7 to
22ab6f0
Compare
cosxl: smol cleanup CosXL: fix schedule choice Rename EDMVDenoiser Avoid inf for EDMVDenoiser + discrete schedule make parametrization flags public Fix CosXL with empty negative prompts Instruct-p2p support support 2 conditionings cfg Do not re-encode the exact same image twice pix2pix: fixes for 2-cfg Fix pix2pix latent inputs + improve inpainting a bit + fix naming prepare for other pix2pix-like models Support sdxl ip2p CoxXL edit: fix reference image embeddings Support 2-cond cfg properly in cli fix typo in help Support masks for ip2p models
Might be specific to Flex.2-preview unfortunately: |
The same trick seems to work well with flux canny too, even though in the diffusers implementation, there's no way to set control strength. Not sure how Flux depth will handle scaled latents though, I haven't tested it yet, but I don't think it will work. Edit: it seems to work too. |
|
Will this PR make it easier to support SDXL control LoRAs in the future? |
Yes, some of the changes from this PR will help support the control LoRAs (both for SDXL and flux), but there's still quite a lot of work on the LoRA implementation side for these to function (need support for Lora bias, allowing mismatched dimensions between Lora and model weights... Plus the Loras are loaded late into the pipeline, after the control inputs are prepared) |
cont #679
Models
Flux Controls:
https://huggingface.co/black-forest-labs/FLUX.1-Canny-dev
https://huggingface.co/black-forest-labs/FLUX.1-Depth-dev
Flux models with built-in controlnet-like conditionning suport (depth or canny depending on the model)
Flex.2 (preview)
https://huggingface.co/ostris/Flex.2-preview
Pruned and de-distilled opensource Flux model with support for inpainting and controlnet-like conditionning (line, pose and depth all supported by a single model).
Usage
To enable control image conditionning, use the
--control-imageargument (same as when using a control net). You can also use--control-strengthto change how much effect the control image should have.