Add Timestep shift, SGM Uniform and Simple scheduler and support for NitroFusion #675
+240
−44
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds support for the timestep-shift technique required for inference with NitroFusion models and Diff-Instruct* and other one-step models. It also adds support for two schedulers: SGM Uniform and Simple, because the existing scheduler, for a mysterious reason, fails at step 2 calculation and produces an output similar to step 1.
NitroFusion is one of the best models for single-step inference, making it useful for inference on compute-constrained devices like mobile phones or CPUs.
Example command:
./bin/sd -m nitrosd-realism_f16.gguf -v -p "cute cat" --cfg-scale 1 --steps 1 --timestep-shift 250 -H 1024 -W 1024 --seed 2024 --schedule sgm_uniform
The recommended timestep-shift values by the authors are 250 for NitroSD-Realism and 500 for NitroSD-Vibrant and 400 for Diff-Instruct*.
I created GGUF versions of NitroFusion that already include the fixed SDXL VAE, available for download here.
The authors mentioned it's possible to extract LoRA weights from these models and apply them to other checkpoints. I’ll try to do that in the future.
EDIT : Just add Diff-Instruct* GGUF
References:
Timestep-shift implementation: node.py
SGM Uniform: sd_schedulers
Simple: samplers.py