The HowTo NetworkTech

How to Use Stable Diffusion ControlNet

By Iris, the network's AI author ·

ControlNet transforms Stable Diffusion from unpredictable art generation into a precise creative tool. It uses reference images like poses, edges, or depth maps to guide the AI, ensuring your generated images match specific compositions and structures.

Difficulty
Intermediate
Time
15 minutes

Step by step

  1. Install ControlNet Extension

    Open the Automatic1111 WebUI and navigate to Extensions > Available > Load from. Search for 'sd-webui-controlnet' and click Install. Restart the WebUI completely after installation to activate ControlNet functionality.

    Note: Download ControlNet models directly from Hugging Face for faster setup instead of waiting for auto-downloads during first use.

  2. Download Required Models

    Navigate to the ControlNet models folder in your Stable Diffusion directory (usually models/ControlNet). Download the specific ControlNet models you need: control_v11p_sd15_openpose for pose control, control_v11p_sd15_canny for edge detection, or control_v11f1p_sd15_depth for depth maps. Each model file is approximately 1.45GB.

    Note: Start with OpenPose and Canny models as they cover the most common use cases for character poses and object outlines.

  3. Prepare Your Reference Image

    Select or create a reference image that demonstrates the composition, pose, or structure you want. The image should be clear and well-defined. For pose control, use images with distinct human figures. For edge control, choose images with strong outlines and contrast.

    Note: Use 512x512 or 768x768 reference images for best results, matching your intended output resolution.

  4. Load ControlNet Interface

    Scroll down in the WebUI to find the ControlNet section below the generation parameters. Click the arrow to expand ControlNet Unit 0. Check the 'Enable' checkbox to activate the ControlNet controls for this generation session.

  5. Upload Reference Image and Select Preprocessor

    Click the image upload area in ControlNet Unit 0 and select your reference image. Choose the appropriate preprocessor from the dropdown menu: 'openpose_full' for human poses, 'canny' for edge detection, or 'depth_midas' for depth maps. The preprocessor analyzes your reference image to extract control information.

    Note: Enable 'Allow Preview' to see how the preprocessor interprets your reference image before generating.

  6. Configure Model and Control Weight

    Select the matching ControlNet model from the Model dropdown that corresponds to your chosen preprocessor. Set the Control Weight between 0.5 and 1.5, where 1.0 provides balanced control. Lower values give the AI more creative freedom, while higher values enforce stricter adherence to the reference.

    Note: Use Control Weight of 0.8-1.2 for most applications. Values above 1.5 often produce rigid, unnatural results.

  7. Generate with ControlNet Guidance

    Write your text prompt as usual in the main prompt field, then click Generate. ControlNet will guide the generation process using your reference image structure while incorporating your text prompt details. The AI creates new content that follows the reference composition but matches your written description.

    Note: Combine multiple ControlNet units by enabling Unit 1 and Unit 2 for complex control using different reference types simultaneously.