ControlNet now available in the WebUI!

[ThereforeGames]

2/15 EDIT: If you run into any issues with this shortcode, I would encourage you to try Mikubill's dedicated ControlNet extension. Their implementation differs from my own in that theirs hooks directly into the A1111 pipeline whereas mine is more of a "wrapper" for the original ControlNet code (including its own external DDIM sampler.)

---

Hi folks,

Good news! I managed to get the incredible ControlNet script running in our favorite WebUI.

It is available via the Unprompted extension.

If you already have Unprompted, all you have to do is fetch the latest update through the Extensions menu. If you don't have Unprompted yet, then... I don't know what to tell you. I guess you don't like me very much.

Jokes aside, I only got this working a few minutes ago, so it's limited to the pose2image model for now. But it should be trivial to add support for the remaining models (knock on wood.) Might have a few more of them ready tonight! (Which ones do you want first?)

EDIT - Currently supported models: Pose, Scribble, M-LSD, Depth Map, Normal Map

Setup

Step 1) After installing the Unprompted extension, you need to grab a few files from ControlNet's HuggingFace page:

https://huggingface.co/lllyasviel/ControlNet

You can git the whole thing if you want, but it's something like 50+ GB. For the Pose model, all you need are the following:

• models/control_sd15_openpose.pth
• annotator/ckpts/body_pose_model.pth
• annotator/ckpts/hand_pose_model.pth

Step 2) Place the control_sd15_openpose.pth file into your WebUI's models/Stable-diffusion directory. Rename it from .pth to .ckpt.

Step 3) Download the cldm_v15.yaml file from the ControlNet repository and place it next to the ckpt:

https://github.com/lllyasviel/ControlNet/tree/main/models

Step 4) Rename it to control_sd15_openpose.yaml.

Step 5) Now, place the two annotator files here:

extensions\unprompted\lib_unprompted\stable_diffusion\controlnet\annotator\ckpts

Step 6) Boot up the WebUI. In the img2img tab, load an initial image and use the [controlnet] shortcode. Here's the prompt for my example above:

[controlnet]walter white dancing in a suit and red tie, blue sky background, best quality

Press generate. If the resulting image doesn't adhere to the pose, press generate again. There's currently a bug where the first result sometimes doesn't work.

Important: Please do not attempt to load the ControlNet model from your normal WebUI dropdown. Just let the shortcode do its thing.

Tip: Try using the save_memory argument if you encounter any memory problems.

Known Issues

• If the Openpose model can't detect a human pose, the script will throw an error and you'll get a black square as your result.
• The "save" button doesn't work at the moment.
• ControlNet is only compatible with the DDIM sampler.
• Inpainting is not currently supported.
• Token length must be kept at 77 tokens or less.
• Potentially conflicts with some other extensions. Will investigate soon.

Have fun!

[ThereforeGames]

Unprompted v7.5.1 Update:

I've rewritten the model loading mechanism using built-in A1111 functions. If you tried the previous version, you'll need to move and rename the following files:

1. extensions/unprompted/lib_unprompted/stable_diffusion/controlnet/models/control_sd15_openpose.pth to...
   models/Stable-diffusion/control_sd15_openpose.ckpt (note the file extension)

2. extensions/unprompted/lib_unprompted/stable_diffusion/controlnet/models/cldm_v15.yaml to...
   models/Stable-diffusion/control_sd15_openpose.yaml (the config name must match the checkpoint name)

Now the [controlnet] shortcode won't have to re-load the whole darn thing every time you generate an image. :)

Important: Please do not attempt to load the ControlNet model from the normal WebUI dropdown. Just let the shortcode do its thing.

Known Issues:

• The first image you generate may not adhere to the ControlNet pose. All generations afterwards seem to work fine. Currently investigating why this is the case.
• If you want to return to normal img2img or txt2img after using ControlNet, you'll need to manually load another model from your dropdown menu.
• The ControlNet models don't seem to work with half-precision (which is one reason why you shouldn't attempt to load it through the dropdown menu.) The [controlnet] shortcode forces the WebUI to temporarily use full precision.

[enzyme69]

Testing it on macOS, M2 Pro machine, finding error:

Model loaded in 53.7s (calculate hash: 3.0s, load weights from disk: 2.3s, create model: 0.8s, apply weights to model: 3.5s, move model to device: 43.8s, load textual inversion embeddings: 0.2s).
Error running postprocess: /Users/jimmygunawan/stable-diffusion-webui/extensions/unprompted/scripts/unprompted.py
Traceback (most recent call last):
  File "/Users/jimmygunawan/stable-diffusion-webui/modules/scripts.py", line 404, in postprocess
    script.postprocess(p, processed, *script_args)
  File "/Users/jimmygunawan/stable-diffusion-webui/extensions/unprompted/scripts/unprompted.py", line 453, in postprocess
    Unprompted.shortcode_objects[i].after(p,processed)
  File "/Users/jimmygunawan/stable-diffusion-webui/extensions/unprompted/shortcodes/stable_diffusion/controlnet.py", line 96, in after
    sd_model = sd_models.load_model(info,None,None).cuda()
  File "/Users/jimmygunawan/stable-diffusion-webui/venv/lib/python3.10/site-packages/pytorch_lightning/core/mixins/device_dtype_mixin.py", line 128, in cuda
    device = torch.device("cuda", torch.cuda.current_device())
  File "/Users/jimmygunawan/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/cuda/__init__.py", line 482, in current_device
    _lazy_init()
  File "/Users/jimmygunawan/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/cuda/__init__.py", line 211, in _lazy_init
    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

[marvin-dent]

I am getting the same after trying to install ControlNet on Macbook Pro M1. Any advice?

[pablonaj]

Yess, thanks for that!
Could you add ControlNet with M-LSD Lines and scribble maps?

[ThereforeGames]

Sure thing! I'll do those next.

[BonzoIPNMadrid]

Do we need a cldm_v15.yaml file renamed for each of the other ckpts in /models/stable-diffusion/ i.e. control_sd15_mlsd.yaml to pair with control_sd15_mlsd.ckpt ?

[syddharth]

Hey @ThereforeGames !
Firstly, thanks for the great plugin and awesome updates. I find it very useful and it has become a part of my worflow on webUI.

I have successfully downloaded and installed the files in the mentioned locations, I get the following error when I try to use the walter white command:

Error running postprocess: G:\stablediffusion\stable-diffusion-webui\extensions\unprompted\scripts\unprompted.py
Traceback (most recent call last):
File "G:\stablediffusion\stable-diffusion-webui\modules\scripts.py", line 404, in postprocess
script.postprocess(p, processed, *script_args)
File "G:\stablediffusion\stable-diffusion-webui\extensions\unprompted\scripts\unprompted.py", line 459, in postprocess
Unprompted.shortcode_objects[i].after(p,processed)
File "G:\stablediffusion\stable-diffusion-webui\extensions\unprompted/shortcodes\stable_diffusion\controlnet.py", line 44, in after
from lib_unprompted.stable_diffusion.controlnet.annotator.openpose import apply_openpose
File "G:\stablediffusion\stable-diffusion-webui\extensions\unprompted\lib_unprompted\stable_diffusion\controlnet\annotator\openpose_init_.py", line 11, in
body_estimation = Body(this_path + '/../ckpts/body_pose_model.pth')
File "G:\stablediffusion\stable-diffusion-webui\extensions\unprompted\lib_unprompted\stable_diffusion\controlnet\annotator\openpose\body.py", line 20, in init
model_dict = util.transfer(self.model, torch.load(model_path))
File "G:\stablediffusion\stable-diffusion-webui\extensions\sd_dreambooth_extension\reallysafe.py", line 117, in load
return load_with_extra(filename, *args, **kwargs)
File "G:\stablediffusion\stable-diffusion-webui\extensions\sd_dreambooth_extension\reallysafe.py", line 164, in load_with_extra
return unsafe_torch_load(filename, *args, **kwargs)
File "G:\stablediffusion\stable-diffusion-webui\modules\safe.py", line 106, in load
return load_with_extra(filename, extra_handler=global_extra_handler, *args, **kwargs)
File "G:\stablediffusion\stable-diffusion-webui\modules\safe.py", line 151, in load_with_extra
return unsafe_torch_load(filename, *args, **kwargs)
File "G:\stablediffusion\stable-diffusion-webui\venv\lib\site-packages\torch\serialization.py", line 771, in load
with _open_file_like(f, 'rb') as opened_file:
File "G:\stablediffusion\stable-diffusion-webui\venv\lib\site-packages\torch\serialization.py", line 270, in _open_file_like
return _open_file(name_or_buffer, mode)
File "G:\stablediffusion\stable-diffusion-webui\venv\lib\site-packages\torch\serialization.py", line 251, in init
super(_open_file, self).init(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: 'G:\stablediffusion\stable-diffusion-webui\extensions\unprompted\lib_unprompted\stable_diffusion\controlnet\annotator\openpose/../ckpts/body_pose_model.pth'

What would be a way around this?

[ThereforeGames]

Hmm, can you confirm that you have the two annotator .pth files in the ckpts folder as shown below?

If so, please provide me with the following diagnostic info:

• Operating system
• Python version

Thanks!

[syddharth]

Had the files in '\extensions\unprompted\lib\stable_diffusion\controlnet\annotator\ckpts'
moved them to '\extensions\unprompted\lib_unprompted\stable_diffusion\controlnet\annotator\ckpts'

It works now, though only with DDIM and throws an error when token length is more than 77.

PS: Also it doesn't seem to work with inpaint mode, can that be supported?

[ThereforeGames]

Glad to hear you got it working.

Inpainting should be relatively simple to add, but I'll need to do some research on supporting other samplers & extending the token limit. For what it's worth, the original codebase does not provide those features.

[ThereforeGames]

Unprompted v7.5.2 Update:

The Scribble and M-LSD models are now supported! Specify which model you'd like to use with the model argument - do not include the file extension.

For example, here's how you might reference the Scribble model:

[controlnet model="control_sd15_scribble"]my prompt goes here

Remember, with each model you need to follow the steps outlined in the original post. TL;DR rename the .pth file to .ckpt, download associated annotator files, and make a copy of the yaml file.

In the near future, I'll hook up this shortcode to the Wizard UI. My priority right now is adapting the remaining models.

Have fun!

[ghost]

amazing work bro, do you have discord?

[MarcusRobbins]

wonderful. Being a demanding user, what Id would really love is to be able to drag the pose model around!

[2blackbar]

OK , got this
`Error executing callback after_component_callback for F:\sd\extensions-builtin\roll-artist\scripts\roll-artist.py
Traceback (most recent call last):
File "F:\sd\modules\script_callbacks.py", line 164, in after_component_callback
c.callback(component, **kwargs)
File "F:\sd\extensions-builtin\roll-artist\scripts\roll-artist.py", line 47, in after_component
add_roll_button(global_prompt)
File "F:\sd\extensions-builtin\roll-artist\scripts\roll-artist.py", line 19, in add_roll_button
roll = gr.Button(value=art_symbol, elem_id="roll", visible=len(shared.artist_db.artists) > 0)
AttributeError: module 'modules.shared' has no attribute 'artist_db'

Running on local URL: http://127.0.0.1:7860

To create a public link, set share=True in launch().
100%|████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 2.43it/s]
Calculating sha256 for F:\sd\models\Stable-diffusion\control_sd15_openpose.ckpt: d19ffffeeaff6d9feb2204b234c3e1b9aec039ab3e63fca07f4fe5646f2ef591
Loading weights [d19ffffeea] from F:\sd\models\Stable-diffusion\control_sd15_openpose.ckpt
Creating model from config: F:\sd\models\Stable-diffusion\control_sd15_openpose.yaml
ControlLDM: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Loading VAE weights specified in settings: F:\sd\models\Stable-diffusion\vae-ft-mse-840000-ema-pruned.vae.pt
Applying cross attention optimization (Doggettx).
Model loaded in 63.8s (create model: 0.9s, apply weights to model: 33.7s, apply dtype to VAE: 0.3s, load VAE: 12.3s, move model to device: 13.5s, load textual inversion embeddings: 2.9s).
cuda
cuda
Data shape for DDIM sampling is (1, 4, 80, 64), eta 0.75
Running DDIM Sampling with 20 timesteps
DDIM Sampler: 0%| | 0/20 [00:01<?, ?it/s]
Error running postprocess: F:\sd\extensions\unprompted\scripts\unprompted.py
Traceback (most recent call last):
File "F:\sd\modules\scripts.py", line 404, in postprocess
script.postprocess(p, processed, *script_args)
File "F:\sd\extensions\unprompted\scripts\unprompted.py", line 459, in postprocess
Unprompted.shortcode_objects[i].after(p,processed)
File "F:\sd\extensions\unprompted/shortcodes\stable_diffusion\controlnet.py", line 110, in after
samples, intermediates = ddim_sampler.sample(self.steps, num_samples,
File "F:\sd\venv\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "F:\sd\extensions\unprompted\lib_unprompted\stable_diffusion\controlnet\ldm\models\diffusion\ddim.py", line 103, in sample
samples, intermediates = self.ddim_sampling(conditioning, size,
File "F:\sd\venv\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "F:\sd\extensions\unprompted\lib_unprompted\stable_diffusion\controlnet\ldm\models\diffusion\ddim.py", line 163, in ddim_sampling
outs = self.p_sample_ddim(img, cond, ts, index=index, use_original_steps=ddim_use_original_steps,
File "F:\sd\venv\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "F:\sd\extensions\unprompted\lib_unprompted\stable_diffusion\controlnet\ldm\models\diffusion\ddim.py", line 211, in p_sample_ddim
model_uncond, model_t = self.model.apply_model(x_in, t_in, c_in).chunk(2)
File "F:\sd\modules\sd_hijack_utils.py", line 17, in
setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
File "F:\sd\modules\sd_hijack_utils.py", line 28, in call
return self.__orig_func(*args, **kwargs)
File "F:\sd\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 858, in apply_model
x_recon = self.model(x_noisy, t, **cond)
File "F:\sd\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "F:\sd\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 1329, in forward
out = self.diffusion_model(x, t, context=cc)
File "F:\sd\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "F:\sd\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 768, in forward
emb = self.time_embed(t_emb)
File "F:\sd\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "F:\sd\venv\lib\site-packages\torch\nn\modules\container.py", line 139, in forward
input = module(input)
File "F:\sd\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "F:\sd\extensions-builtin\Lora\lora.py", line 178, in lora_Linear_forward
return lora_forward(self, input, torch.nn.Linear_forward_before_lora(self, input))
File "F:\sd\venv\lib\site-packages\torch\nn\modules\linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: expected scalar type Half but found Float

0%| | 0/1 [00:00<?, ?it/s]
Error completing request
Arguments: ('task(l667ujil0cqq1uz)', 0, '[controlnet]man in forest by bajki', '', [], <PIL.Image.Image image mode=RGBA size=1000x1200 at 0x166E1AACEE0>, None, None, None, None, None, None, 20, 1, 4, 0, 1, False, False, 1, 1, 7, 1.5, 0.75, -1.0, -1.0, 0, 0, 0, False, 512, 512, 0, 0, 32, 0, '', '', '', [], 0, True, -1.0, '

\n
• CFG Scale should be 2 or lower.
\n

\n', True, True, '', '', True, 50, True, 1, 0, False, 4, 1, '

Recommended settings: Sampling Steps: 80-100, Sampler: Euler a, Denoising strength: 0.8

', 128, 8, ['left', 'right', 'up', 'down'], 1, 0.05, 128, 4, 0, ['left', 'right', 'up', 'down'], False, False, 'positive', 'comma', 0, False, False, '', '

Will upscale the image by the selected scale factor; use width and height sliders to set tile size

', 64, 0, 2, 1, '', 0, '', True, False, False, 1, '', 0, '', 0, '', True, False, False, False, 0, True, True, '', False, 1, 'Both ▦', False, '', False, True, False, False, False, False, 0, False, '', '', '', 'generateMasksTab', 1, 4, 2.5, 30, 1.03, 1, 1, 5, 0.5, 5, False, True, '

Will upscale the image depending on the selected target size type

', 512, 8, 32, 64, 0.35, 32, 0, True, 0, False, 8, 0, 0, 2048, 2048, 2) {}
Traceback (most recent call last):
File "F:\sd\modules\call_queue.py", line 56, in f
res = list(func(*args, **kwargs))
File "F:\sd\modules\call_queue.py", line 37, in f
res = func(*args, **kwargs)
File "F:\sd\modules\img2img.py", line 169, in img2img
processed = process_images(p)
File "F:\sd\modules\processing.py", line 486, in process_images
res = process_images_inner(p)
File "F:\sd\modules\processing.py", line 628, in process_images_inner
samples_ddim = p.sample(conditioning=c, unconditional_conditioning=uc, seeds=seeds, subseeds=subseeds, subseed_strength=p.subseed_strength, prompts=prompts)
File "F:\sd\modules\processing.py", line 1044, in sample
samples = self.sampler.sample_img2img(self, self.init_latent, x, conditioning, unconditional_conditioning, image_conditioning=self.image_conditioning)
File "F:\sd\modules\sd_samplers_kdiffusion.py", line 302, in sample_img2img
samples = self.launch_sampling(t_enc + 1, lambda: self.func(self.model_wrap_cfg, xi, extra_args=extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
File "F:\sd\modules\sd_samplers_kdiffusion.py", line 221, in launch_sampling
return func()
File "F:\sd\modules\sd_samplers_kdiffusion.py", line 302, in
samples = self.launch_sampling(t_enc + 1, lambda: self.func(self.model_wrap_cfg, xi, extra_args=extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
File "F:\sd\venv\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "F:\sd\repositories\k-diffusion\k_diffusion\sampling.py", line 128, in sample_euler
denoised = model(x, sigma_hat * s_in, **extra_args)
File "F:\sd\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "F:\sd\modules\sd_samplers_kdiffusion.py", line 116, in forward
x_out = self.inner_model(x_in, sigma_in, cond={"c_crossattn": [cond_in], "c_concat": [image_cond_in]})
File "F:\sd\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "F:\sd\repositories\k-diffusion\k_diffusion\external.py", line 112, in forward
eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
File "F:\sd\repositories\k-diffusion\k_diffusion\external.py", line 138, in get_eps
return self.inner_model.apply_model(*args, **kwargs)
File "F:\sd\extensions\unprompted/lib_unprompted/stable_diffusion/controlnet\cldm\cldm.py", line 332, in apply_model
control = self.control_model(x=x_noisy, hint=cond_hint, timesteps=t, context=cond_txt)
File "F:\sd\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "F:\sd\extensions\unprompted/lib_unprompted/stable_diffusion/controlnet\cldm\cldm.py", line 287, in forward
guided_hint = self.input_hint_block(hint, emb, context)
File "F:\sd\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "F:\sd\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 86, in forward
x = layer(x)
File "F:\sd\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "F:\sd\extensions-builtin\Lora\lora.py", line 182, in lora_Conv2d_forward
return lora_forward(self, input, torch.nn.Conv2d_forward_before_lora(self, input))
File "F:\sd\venv\lib\site-packages\torch\nn\modules\conv.py", line 457, in forward
return self._conv_forward(input, self.weight, self.bias)
File "F:\sd\venv\lib\site-packages\torch\nn\modules\conv.py", line 453, in _conv_forward
return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Given groups=1, weight of size [16, 3, 3, 3], expected input[2, 5, 1, 1] to have 3 channels, but got 5 channels instead`

[pablonaj]

same here, I'm getting the same error (I think), I'm running:
python: 3.10.6  •  torch: 1.13.1+cu117  •  xformers: 0.0.16rc425  •  gradio: 3.16.2  •  commit: 3715ece0  •  checkpoint: [27a4ac756c]

The log says:
Error completing request
Arguments: ('task(4qqpn5bonxom0y8)', 0, '[controlnet model="control_sd15_openpose"] a roman marble statue', '', [], <PIL.Image.Image image mode=RGBA size=537x611 at 0x2AC433E5960>, None, None, None, None, None, None, 70, 11, 4, 0, 1, False, False, 1, 1, 1.5, 1.5, 0.75, -1.0, -1.0, 0, 0, 0, False, 512, 512, 0, 0, 32, 0, '', '', '', [], 0, 0, 0, 0, 0, 0.25, False, 7, 100, 'Constant', 0, 'Constant', 0, 4, False, False, 'LoRA', 'None', 1, 1, 'LoRA', 'None', 1, 1, 'LoRA', 'None', 1, 1, 'LoRA', 'None', 1, 1, 'LoRA', 'None', 1, 1, 'Refresh models', 0.9, 5, '0.0001', False, 'None', '', 0.1, False, '', False, False, False, False, 'Auto', 0.5, 1, False, False, True, -1.0, '

\n
• CFG Scale should be 2 or lower.
\n

\n', True, True, '', '', True, 50, True, 1, 0, False, 4, 1, '

Recommended settings: Sampling Steps: 80-100, Sampler: Euler a, Denoising strength: 0.8

', 128, 8, ['left', 'right', 'up', 'down'], 1, 0.05, 128, 4, 0, ['left', 'right', 'up', 'down'], False, False, 'positive', 'comma', 0, False, False, '', '

Will upscale the image by the selected scale factor; use width and height sliders to set tile size

', 64, 0, 2, '

Will upscale the image by the selected scale factor; use width and height sliders to set tile size

', 64, 0, 2, False, False, False, False, False, False, False, 1, '', 0, '', 0, '', True, False, False, False, 0, None, True, None, None, False, '

Will upscale the image depending on the selected target size type

', 512, 0, 8, 32, 64, 0.35, 32, 0, True, 0, False, 8, 0, 0, 2048, 2048, 2) {}
Traceback (most recent call last):
File "G:\Stable_Diffusion\Automatic1111_v3\stable-diffusion-webui\modules\call_queue.py", line 56, in f
res = list(func(*args, **kwargs))
File "G:\Stable_Diffusion\Automatic1111_v3\stable-diffusion-webui\modules\call_queue.py", line 37, in f
res = func(*args, **kwargs)
File "G:\Stable_Diffusion\Automatic1111_v3\stable-diffusion-webui\modules\img2img.py", line 117, in img2img
p = StableDiffusionProcessingImg2Img(
File "G:\Stable_Diffusion\Automatic1111_v3\stable-diffusion-webui\modules\processing.py", line 912, in init
self.image_cfg_scale: float = image_cfg_scale if shared.sd_model.cond_stage_key == "edit" else None
AttributeError: 'NoneType' object has no attribute 'cond_stage_key'

[bakaEC]

Same error.

[bbecausereasonss]

Can anyone describe what makes this better than just using IMG2IMG or Pix2Pix?

[ljleb]

For example, you can be very precise with the pose of generated people or characters using the openpose controlnet model. No more (((hands up))) or any head scratching what-word-was-it-already, you can let an image do the work for you and it leaves out more prompt space for other aspects of the idea you want to bring to life.

[FurkanGozukara]

Wow so fast

I made a tutorial for their native script I guess I should make one for this one now

https://youtu.be/YJebdQ30UZQ

[XeonG]

Running DDIM Sampling with 20 timesteps
DDIM Sampler:   0%|                                                                                                                                                   | 0/20 [00:00<?, ?it/s]
Error running postprocess: C:\Projects\Stable_Diffusion\stable-diffusion-webui\extensions\unprompted\scripts\unprompted.py
Traceback (most recent call last):
  File "C:\Projects\Stable_Diffusion\stable-diffusion-webui\modules\scripts.py", line 404, in postprocess
    script.postprocess(p, processed, *script_args)
  File "C:\Projects\Stable_Diffusion\stable-diffusion-webui\extensions\unprompted\scripts\unprompted.py", line 459, in postprocess
    Unprompted.shortcode_objects[i].after(p,processed)
  File "C:\Projects\Stable_Diffusion\stable-diffusion-webui\extensions\unprompted/shortcodes\stable_diffusion\controlnet.py", line 110, in after
    samples, intermediates = ddim_sampler.sample(self.steps, num_samples,
  File "C:\Projects\Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "C:\Projects\Stable_Diffusion\stable-diffusion-webui\extensions\unprompted\lib_unprompted\stable_diffusion\controlnet\ldm\models\diffusion\ddim.py", line 103, in sample
    samples, intermediates = self.ddim_sampling(conditioning, size,
  File "C:\Projects\Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "C:\Projects\Stable_Diffusion\stable-diffusion-webui\extensions\unprompted\lib_unprompted\stable_diffusion\controlnet\ldm\models\diffusion\ddim.py", line 163, in ddim_sampling
    outs = self.p_sample_ddim(img, cond, ts, index=index, use_original_steps=ddim_use_original_steps,
  File "C:\Projects\Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "C:\Projects\Stable_Diffusion\stable-diffusion-webui\extensions\unprompted\lib_unprompted\stable_diffusion\controlnet\ldm\models\diffusion\ddim.py", line 211, in p_sample_ddim
    model_uncond, model_t = self.model.apply_model(x_in, t_in, c_in).chunk(2)
  File "C:\Projects\Stable_Diffusion\stable-diffusion-webui\modules\sd_hijack_utils.py", line 17, in <lambda>
    setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
  File "C:\Projects\Stable_Diffusion\stable-diffusion-webui\modules\sd_hijack_utils.py", line 28, in __call__
    return self.__orig_func(*args, **kwargs)
  File "C:\Projects\Stable_Diffusion\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 858, in apply_model
    x_recon = self.model(x_noisy, t, **cond)
  File "C:\Projects\Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Projects\Stable_Diffusion\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 1329, in forward
    out = self.diffusion_model(x, t, context=cc)
  File "C:\Projects\Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Projects\Stable_Diffusion\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 768, in forward
    emb = self.time_embed(t_emb)
  File "C:\Projects\Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Projects\Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\container.py", line 204, in forward
    input = module(input)
  File "C:\Projects\Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Projects\Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 must have the same dtype

[LaZoRBear]

Ever got it to work? I'm in the same boat currently.

[Pscgylotti]

if your gpu is gtx 16XX , try launching with args:
--no-half --precision full

[am576]

I'm guessing you missed a yaml file step from the installtion process, as I did. Got the same error, after downloading the yaml file, it works

[BlipOnNobodysRadar]

Looks amazing, but unfortunately, I can't seem to use it. I get this issue at step 6

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 8.00 GiB total capacity; 7.23 GiB already allocated; 0 bytes free; 7.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I tried again with save_memory = True in config.py, but no luck. Is this just not usable with 8gb VRAM? What's the minimum?

[FurkanGozukara]

Looks amazing, but unfortunately, I can't seem to use it. I get this issue at step 6

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 8.00 GiB total capacity; 7.23 GiB already allocated; 0 bytes free; 7.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I tried again with save_memory = True in config.py, but no luck. Is this just not usable with 8gb VRAM? What's the minimum?

i think native scripts work better

i saw people running

here my tutorial : Transform Your Sketches into Masterpieces with Stable Diffusion ControlNet AI - How To Use Tutorial

[osmannakka]

real

[FurkanGozukara]

@ThereforeGames

not working on a fresh installation. i have started recording a tutorial video but not working even on a fresh installation

0 extensions installed other than yours

fresh latest version - python 3.10.8

windows 10

here all the settings and error

Error completing request
Arguments: ('task(iwcokxjsiyqnake)', 0, '[controlnet model="control_sd15_mlsd"] awesome fantastic dragon', '', [], <PIL.Image.Image image mode=RGBA size=866x684 at 0x1ED186C35B0>, None, None, None, None, None, None, 20, 0, 4, 0, 1, False, False, 1, 1, 7, 1.5, 0.75, -1.0, -1.0, 0, 0, 0, False, 512, 512, 0, 0, 32, 0, '', '', '', [], 0, True, -1.0, '<ul>\n<li><code>CFG Scale</code> should be 2 or lower.</li>\n</ul>\n', True, True, '', '', True, 50, True, 1, 0, False, 4, 1, '<p style="margin-bottom:0.75em">Recommended settings: Sampling Steps: 80-100, Sampler: Euler a, Denoising strength: 0.8</p>', 128, 8, ['left', 'right', 'up', 'down'], 1, 0.05, 128, 4, 0, ['left', 'right', 'up', 'down'], False, False, 'positive', 'comma', 0, False, False, '', '<p style="margin-bottom:0.75em">Will upscale the image by the selected scale factor; use width and height sliders to set tile size</p>', 64, 0, 2, 1, '', 0, '', 0, '', True, False, False, False, 0) {}
Traceback (most recent call last):
  File "C:\control net automatic\stable-diffusion-webui\modules\call_queue.py", line 56, in f
    res = list(func(*args, **kwargs))
  File "C:\control net automatic\stable-diffusion-webui\modules\call_queue.py", line 37, in f
    res = func(*args, **kwargs)
  File "C:\control net automatic\stable-diffusion-webui\modules\img2img.py", line 169, in img2img
    processed = process_images(p)
  File "C:\control net automatic\stable-diffusion-webui\modules\processing.py", line 486, in process_images
    res = process_images_inner(p)
  File "C:\control net automatic\stable-diffusion-webui\modules\processing.py", line 628, in process_images_inner
    samples_ddim = p.sample(conditioning=c, unconditional_conditioning=uc, seeds=seeds, subseeds=subseeds, subseed_strength=p.subseed_strength, prompts=prompts)
  File "C:\control net automatic\stable-diffusion-webui\modules\processing.py", line 1044, in sample
    samples = self.sampler.sample_img2img(self, self.init_latent, x, conditioning, unconditional_conditioning, image_conditioning=self.image_conditioning)
  File "C:\control net automatic\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 302, in sample_img2img
    samples = self.launch_sampling(t_enc + 1, lambda: self.func(self.model_wrap_cfg, xi, extra_args=extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
  File "C:\control net automatic\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 221, in launch_sampling
    return func()
  File "C:\control net automatic\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 302, in <lambda>
    samples = self.launch_sampling(t_enc + 1, lambda: self.func(self.model_wrap_cfg, xi, extra_args=extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
  File "C:\control net automatic\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "C:\control net automatic\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py", line 145, in sample_euler_ancestral
    denoised = model(x, sigmas[i] * s_in, **extra_args)
  File "C:\control net automatic\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\control net automatic\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 116, in forward
    x_out = self.inner_model(x_in, sigma_in, cond={"c_crossattn": [cond_in], "c_concat": [image_cond_in]})
  File "C:\control net automatic\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\control net automatic\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 112, in forward
    eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
  File "C:\control net automatic\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 138, in get_eps
    return self.inner_model.apply_model(*args, **kwargs)
  File "C:\control net automatic\stable-diffusion-webui\extensions\unprompted/lib_unprompted/stable_diffusion/controlnet\cldm\cldm.py", line 332, in apply_model
    control = self.control_model(x=x_noisy, hint=cond_hint, timesteps=t, context=cond_txt)
  File "C:\control net automatic\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\control net automatic\stable-diffusion-webui\extensions\unprompted/lib_unprompted/stable_diffusion/controlnet\cldm\cldm.py", line 287, in forward
    guided_hint = self.input_hint_block(hint, emb, context)
  File "C:\control net automatic\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\control net automatic\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 86, in forward
    x = layer(x)
  File "C:\control net automatic\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\control net automatic\stable-diffusion-webui\extensions-builtin\Lora\lora.py", line 182, in lora_Conv2d_forward
    return lora_forward(self, input, torch.nn.Conv2d_forward_before_lora(self, input))
  File "C:\control net automatic\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\conv.py", line 463, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "C:\control net automatic\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\conv.py", line 459, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Given groups=1, weight of size [16, 3, 3, 3], expected input[2, 5, 1, 1] to have 3 channels, but got 5 channels instead

[nick888888888]

Exact same problem here

[enternalsaga]

in my case, the generated image didnt capture the pose

[GreenLandisaLie]

Do you have a 'control_sd15_openpose.yaml' file next to 'control_sd15_openpose.ckpt'? Seems like you might have missed a step because yours didn't load at all.

[enternalsaga]

it all there

[ljleb]

The generated pose seems about right. Is the result the same on different seeds?

[FurkanGozukara]

ok this extension seriously wasted my 1.5 hours
there is another extension that just works flawlessly

finished recording hopefully will be on my channel tomorrow : https://www.youtube.com/@SECourses

[GreenLandisaLie]

Would it be possible to run this on the cpu when using --use-cpu=all? Right now it doesn't work but hopefully that might be something easy to fix.

[JasonGilholme]

Not sure if you solved this already, but I found that the implementation of modules.devices.get_device_for(task) doesn't take into account the 'all' option that you should be able to provide for the --use-cpu flag.

After changing that function to take that into account things started working as expected for me.

[Rokurolize]

WebUI could not be started.

Loading Unprompted v7.5.5 by Therefore Games
(SETUP) Initializing Unprompted object...
(SETUP) Loading configuration files...
(SETUP) Debug mode is False
Loading weights [d19ffffeea] from /storage/stable-diffusion/stable-diffusion-webui/models/Stable-diffusion/control_sd15_openpose.ckpt
Creating model from config: /storage/stable-diffusion/stable-diffusion-webui/models/Stable-diffusion/control_sd15_openpose.yaml
Failed to create model quickly; will retry using slow method.
loading stable diffusion model: ModuleNotFoundError
Traceback (most recent call last):
  File "/storage/stable-diffusion/stable-diffusion-webui/webui.py", line 111, in initialize
    modules.sd_models.load_model()
  File "/storage/stable-diffusion/stable-diffusion-webui/modules/sd_models.py", line 400, in load_model
    sd_model = instantiate_from_config(sd_config.model)
  File "/storage/stable-diffusion/stable-diffusion-webui/repositories/stable-diffusion-stability-ai/ldm/util.py", line 79, in instantiate_from_config
    return get_obj_from_str(config["target"])(**config.get("params", dict()))
  File "/storage/stable-diffusion/stable-diffusion-webui/repositories/stable-diffusion-stability-ai/ldm/util.py", line 87, in get_obj_from_str
    return getattr(importlib.import_module(module, package=None), cls)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'cldm'


Stable diffusion model failed to load, exiting
root@ngwnqj59lz:/notebooks# 

[Rokurolize]

Thank you, it worked successfully.

[ffedev]

This can occur when the last loaded model was ControlNet. Sometimes simply restarting the UI will fix it, but if not, you'll need to temporarily rename or move the model that it's failing to load (in this case control_sd15_openpose.ckpt).

I got a similiar issue, It wouldn't generate images, and when I re-started my Webui it stopped working.

I deleted the Extension from the /extensions folder and the openpose.ckpt and .yaml from the /models/stable-diffusion folder but the Webui still won't launch.

is there a procedure to "clean uninstall" extensions?

Installing requirements for Web UI Launching Web UI with arguments: --medvram --xformers Traceback (most recent call last): File "D:\_Stable_Diffusion\stable-diffusion-webui\launch.py", line 361, in <module> start() File "D:\_Stable_Diffusion\stable-diffusion-webui\launch.py", line 352, in start import webui File "D:\_Stable_Diffusion\stable-diffusion-webui\webui.py", line 15, in <module> from modules import import_hook, errors, extra_networks, ui_extra_networks_checkpoints File "D:\_Stable_Diffusion\stable-diffusion-webui\modules\ui_extra_networks_checkpoints.py", line 6, in <module> from modules import shared, ui_extra_networks, sd_models File "D:\_Stable_Diffusion\stable-diffusion-webui\modules\shared.py", line 12, in <module> import modules.interrogate File "D:\_Stable_Diffusion\stable-diffusion-webui\modules\interrogate.py", line 15, in <module> from modules import devices, paths, shared, lowvram, modelloader, errors File "D:\_Stable_Diffusion\stable-diffusion-webui\modules\modelloader.py", line 7, in <module> from basicsr.utils.download_util import load_file_from_url File "D:\_Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\basicsr\__init__.py", line 3, in <module> from .archs import * File "D:\_Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\basicsr\archs\__init__.py", line 5, in <module> from basicsr.utils import get_root_logger, scandir File "D:\_Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\basicsr\utils\__init__.py", line 4, in <module> from .img_process_util import USMSharp, usm_sharp File "D:\_Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\basicsr\utils\img_process_util.py", line 1, in <module> import cv2 File "D:\_Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\cv2\__init__.py", line 8, in <module> from .cv2 import * ImportError: Bindings generation error. Submodule name should always start with a parent module name. Parent name: cv2.cv2. Submodule name: cv2

[feelingok]

same problem, tried fresh installs multiple times but always shows this error

[m3ngyang]

Same issue after trying the controlnet.

[m3ngyang]

still failed to start the WebUI.

Launching Web UI with arguments: --share --enable-insecure-extension-access
No module 'xformers'. Proceeding without it.
Loading Unprompted v7.9.1 by Therefore Games
(SETUP) Initializing Unprompted object...
(SETUP) Loading configuration files...
(SETUP) Debug mode is False
Loading weights [d19ffffeea] from /projects/stable-diffusion-webui/models/Stable-diffusion/control_sd15_openpose.ckpt
Creating model from config: /projects/stable-diffusion-webui/models/Stable-diffusion/control_sd15_openpose.yaml
Failed to create model quickly; will retry using slow method.
loading stable diffusion model: ModuleNotFoundError
Traceback (most recent call last):
  File "/projects/stable-diffusion-webui/webui.py", line 139, in initialize
    modules.sd_models.load_model()
  File "/projects/stable-diffusion-webui/modules/sd_models.py", line 438, in load_model
    sd_model = instantiate_from_config(sd_config.model)
  File "/projects/stable-diffusion-webui/repositories/stable-diffusion-stability-ai/ldm/util.py", line 89, in instantiate_from_config
    return get_obj_from_str(config["target"])(**config.get("params", dict()))
  File "/projects/stable-diffusion-webui/repositories/stable-diffusion-stability-ai/ldm/util.py", line 97, in get_obj_from_str
    return getattr(importlib.import_module(module, package=None), cls)
  File "/work/miniconda3/envs/diffusion/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'cldm'


Stable diffusion model failed to load, exiting

[2blackbar]

This one works guys! This allows even for 960x960 resolution and no doubles on 1.5 model, i cant believe this, its really a revolution for SD, this piece of code is pretty incredible, same pose, any res you want, set weight to 0.5
When i check the results i laugh and scream at how good it is, this is a must and needs to be in webui by default, no extension
https://github.com/Mikubill/sd-webui-controlnet

[FurkanGozukara]

Made a tutorial for the flawlessly working extension

sorry ThereforeGames i have spent 1.5 hours on your extension yesterday while making the video. it also caused me like 1 hours when post editing

16.) Automatic1111 Web UI
Sketches into Epic Art with 1 Click: A Guide to Stable Diffusion ControlNet in Automatic1111 Web UI

[LonicaMewinsky]

With much love and appreciation to @Mikubill and @ThereforeGames for the extension work, IMO this seems like something that should be integrated directly as a feature.

[westmancurtis]

Especially for users who rely on the API. :)

[timoncool]

Hey there! Amazing script, but I find some issue in batch process, creating only one image from butch after it I have this error:
Will process 112 images, creating 1 new images for each. 100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [00:13<00:00, 2.28it/s] Error completing request | 30/3360 [00:12<24:18, 2.28it/s] Arguments: ('task(so77emzgdxv56x7)', 5, 'fullbody portrait of girl 33 years old, show fuck sign, curly blond hair. Very smart face, glasses, snub nose. Blue detailed eyes. Huge breasts, little chubby. \nWearing in pink t-shirt and black mini skirt.\nDancing on the floor.\noutlined, black border, pastel colors, sticker, neko, anime character high detail high quality intricate details beautiful, acrylic painting, trending on pixiv fanbox, palette knife and brush strokes, style of makoto shinkai jamie wyeth james gilleard edward hopper greg rutkowski studio ghibli genshin impact', '(censored:1.3), (SFW:1.3), (worst quality:1.4), (low quality:1.4), (monochrome:1.1), bad_prompt_version2, bad_artist_anime, (loli: 1.5), (shota:1.5), (child:1.4), ((disfigured)), ((bad art)), vignette, cinematic, grayscale, bokeh, blurred, depth of field, (bad-hands-5:1.2), ', [], <PIL.Image.Image image mode=RGBA size=512x1280 at 0x16766432B90>, None, None, None, None, None, None, 30, 1, 4, 0, 1, False, False, 1, 1, 7, 1.5, 1, 3698233669.0, -1.0, 0, 0, 0, False, 1280, 512, 0, 1, 156, 0, 'G:\\AI\\stable-diffusion-webui\\outputs\\dance', 'G:\\AI\\stable-diffusion-webui\\outputs\\dance\\1', '', [], 0, True, 'keyword prompt', 'keyword1, keyword2', 'None', 'textual inversion first', True, False, 1, False, False, False, 1.1, 1.5, 100, 0.7, False, False, True, False, False, 0, 'Gustavosta/MagicPrompt-Stable-Diffusion', '', False, 'x264', 'mci', 10, 0, False, True, True, True, 'intermediate', 'animation', True, 'openpose', 'control_sd15_openpose [fef5e48e]', 1, None, False, 'Scale to Fit (Inner Fit)', False, False, False, 'Denoised', 5.0, 0.0, 0.0, False, 'mp4', 'h264', 2.0, 0.0, 0.0, False, 0.0, True, True, False, 100, 0.6, 20, 0.0, 0.0, '', True, True, '', 10, 40, 'VP9 (webm)', '', True, 20, '<ul>\n<li><code>CFG Scale</code> should be 2 or lower.</li>\n</ul>\n', True, True, '', '', True, 50, True, 1, 0, False, 4, 1, '<p style="margin-bottom:0.75em">Recommended settings: Sampling Steps: 80-100, Sampler: Euler a, Denoising strength: 0.8</p>', 128, 8, ['left', 'right', 'up', 'down'], 1, 0.05, 128, 4, 0, ['left', 'right', 'up', 'down'], False, None, None, '', '', '', '', 'Auto rename', {'label': 'Upload avatars config'}, 'Open outputs directory', 'Export to WebUI style', True, {'label': 'Presets'}, {'label': 'QC preview'}, '', [], 'Select', 'QC scan', 'Show pics', None, False, False, 'positive', 'comma', 0, False, False, '', '<p style="margin-bottom:0.75em">Will upscale the image by the selected scale factor; use width and height sliders to set tile size</p>', 64, 0, 2, 1, '', 0, '', 0, '', True, False, False, False, 0, 'Not set', True, True, '', '', '', '', '', 1.3, 'Not set', 'Not set', 1.3, 'Not set', 1.3, 'Not set', 1.3, 1.3, 'Not set', 1.3, 'Not set', 1.3, 'Not set', 1.3, 'Not set', 1.3, 'Not set', 1.3, 'Not set', False, 'None', False, 0, True, 384, 384, False, 2, True, True, False, False, 0, 0, 512, 512, False, False, True, True, True, False, False, 1, False, False, 2.5, 4, 0, False, 0, 1, False, False, 'u2net', False, False, False, False, '', '', False, '', 127, False, 30, 9999, 1, 10, 0.25, True, False, '1', '', True, '', '', '', '', '', '', '', '', '', '', '', 'None', 0.3, 60) {} Traceback (most recent call last): File "G:\AI\stable-diffusion-webui\modules\call_queue.py", line 56, in f res = list(func(*args, **kwargs)) File "G:\AI\stable-diffusion-webui\modules\call_queue.py", line 37, in f res = func(*args, **kwargs) File "G:\AI\stable-diffusion-webui\modules\img2img.py", line 163, in img2img process_batch(p, img2img_batch_input_dir, img2img_batch_output_dir, img2img_batch_inpaint_mask_dir, args) File "G:\AI\stable-diffusion-webui\modules\img2img.py", line 76, in process_batch processed_image.save(os.path.join(output_dir, filename)) AttributeError: 'numpy.ndarray' object has no attribute 'save'

[ljleb]

see fix here: Mikubill/sd-webui-controlnet#111 (comment)

In settings tab, tick the "Do not append map to output" checkbox.

[ljleb]

As a side note, please report issues related to the extension in the issue tracker for the extension, instead of here. Other people may already have solved the problem you are encountering, or at least may have encountered it already. Also, if it's an original issue, the author will be able to be aware of it faster if you post there instead of here.

[winthropharvey]

I'm using @Mikubill's extension, and it works flawlessly except for openpose which always produced completely blank black images for the mask while throwing no error. Anyone else have this issue?

Edit: Literally as I went back to test it more it just started working... I didn't change anything. The only thing I can think of is that perhaps there was something still downloading or installing in the background that hadn't finished, though I'm baffled why it wouldn't throw an error in that case. Anyway, it's working now.

[Gojo42]

did you ever find any particular issue with that, i am having the same problem and it's not fixing.

[ljleb]

Just in case it could help someone here, if using an openpose pose directly as controlnet image, the preprocessor should be set to none, not openpose. Set it to openpose when you want to use a real image and infer the pose with a model.

[Khayoon]

thank you @ljleb the problem me and @Gojo42 are having is making the openpose pose (changing the skeleton and producing completely blank black images in the extension tab) Any help is appreciated

[am576]

well, It's in steps 3 and 4 in the description Step 3) Download the cldm_v15.yaml file from the ControlNet repository and place it next to the ckpt: https://github.com/lllyasviel/ControlNet/tree/main/models Step 4) Rename it to control_sd15_openpose.yaml