ModelListedExternal

Pixeldit ControlNet

ControlNet scribble conditioning and IP-Adapter style transfer for PixelDiT-1300M.

Sourcehuggingface.co/madtune/pixeldit-controlnet/tree/db700ca47ed83d368f8e981299cee1478376922d

KindControlNetBase modelmadtune/pixeldit-diffusersVersionBgrade

Model source

Kind: ControlNet
File size: 1.44 GB
Base model: madtune/pixeldit-diffusers
Version: vcontrolnet
Updated: Oct 6, 2026
Downloads: 50
Source: Hugging Face

Source description

PixelDiT ControlNet + IP-Adapter

ControlNet scribble conditioning and IP-Adapter style transfer for PixelDiT-1300M.

Note: PixelDiT-1300M is a model by NVIDIA Research. This repo contains trained adapters only — we are not affiliated with NVIDIA.

Files

File	Description
`controlnet.safetensors`	Combined ControlNet (7 blocks) + IP-Adapter weights
`ip_adapter.safetensors`	IP-Adapter weights only
`hed_detector.safetensors`	HED edge detector (Apache-2.0, VGG-based)
`config.json`	Model config
`train.py`	Joint ControlNet + IP-Adapter training script
`precompute_wd_tags.py`	Run WD tagger on dataset → `wd_tags.json`
`precompute_embeddings.py`	Encode images with SigLIP + Gemma → memmap files
`precompute_hed.py`	Precompute HED edge maps for a dataset
`control_maps.py`	Edge map post-processing utilities
`hed.py`	HED model definition
`convert_to_safetensors.py`	Convert .pt checkpoints to safetensors

Usage

from diffusers.pipelines.pixeldit import PixelDiTStyledPipeline
from huggingface_hub import hf_hub_download
from PIL import Image
import torch

pipe = PixelDiTStyledPipeline.from_pretrained_styled(
    "madtune/pixeldit-diffusers",
    controlnet_path=hf_hub_download("madtune/pixeldit-controlnet", "controlnet.safetensors"),
    ip_adapter_path=hf_hub_download("madtune/pixeldit-controlnet", "ip_adapter.safetensors"),
    hed_ckpt_path=hf_hub_download("madtune/pixeldit-controlnet", "hed_detector.safetensors"),
    torch_dtype=torch.bfloat16,
)
pipe.enable_model_cpu_offload(gpu_id=1)

out = pipe(
    image=Image.open("style_ref.jpg"),
    prompt="gothic pale woman, dramatic rim lighting",
    variation_strength=0.85,
    ctrl_strength=0.25,
    ip_strength=0.85,
    flow_shift=8.0,
    guidance_scale=4.5,
    num_inference_steps=50,
).images[0]
out.save("output.jpg")

Recommended settings

Mode	`ctrl_strength`	`ip_strength`	`variation_strength`
Pure variation	0.0	0.0	0.65–0.85
ControlNet only	0.25	0.0	0.85
IP-Adapter only	0.0	0.85	0.85
Full combo (best)	0.25	0.35–0.85	0.85

flow_shift=8.0 + guidance_scale=3.0–3.5 works well at 768px+. 4.5 is valid but produces oversaturated colours.

Sources

1 source

Hugging FacePrimaryVersion db700ca47ed83d368f8e981299cee1478376922d

huggingface.co/madtune/pixeldit-controlnet/tree/db700ca47ed83d368f8e981299cee1478376922d

Verified Oct 6

Model artifacts

1 artifact

controlnet.safetensors

safetensors · 1.44 GB · SHA-256 24b191b8ae1d…d26a · Hugging Face

Download file

Source excerpts

3 excerpts

ReadmeJul 29huggingface.co/madtune/pixeldit-controlnet

--- license: apache-2.0 base_model: madtune/pixeldit-diffusers tags: - controlnet - ip-adapter - pixeldit - diffusers - image-to-image - style-transfer --- # PixelDiT ControlNet + IP-Adapter ControlNet scribble conditioning and IP-Adapter style transfer for [PixelDiT-1300M](https://huggingface.co/madtune/pixeldit-diffusers). > **Note:** PixelDiT-1300M is a model by [NVIDIA Research](https://research.nvidia.com/). This repo contains trained adapters only — we are not affiliated with NVIDIA. ## Files | File | Description | |---|---| | `controlnet.safetensors` | Combined ControlNet (7 blocks) + IP-Adapter weights | | `ip_adapter.safetensors` | IP-Adapter weights only | | `hed_detector.safetensors` | HED edge detector (Apache-2.0, VGG-based) | | `config.json` | Model config | | `train.py` | Joint ControlNet + IP-Adapter training script | | `precompute_wd_tags.py` | Run WD tagger on dataset → `wd_tags.json` | | `precompute_embeddings.py` | Encode images with SigLIP + Gemma → memmap files | | `precompute_hed.py` | Precompute HED edge maps for a dataset | | `control_maps.py` | Edge map post-processing utilities | | `hed.py` | HED model definition | | `convert_to_safetensors.py` | Convert .pt checkpoints to safetensors | ## Usage ```python from diffusers.pipelines.pixeldit import PixelDiTStyledPipeline from huggingface_hub import hf_hub_download from PIL import Image import torch pipe = PixelDiTStyledPipeline.from_pretrained_styled( "madtune/pixeldit-diffusers", controlnet_path=hf_hub_download("madtune/pixeldit-controlnet", "controlnet.safetensors"), ip_adapter_path=hf_hub_download("madtune/pixeldit-controlnet", "ip_adapter.safetensors"), hed_ckpt_path=hf_hub_download("madtune/pixeldit-controlnet", "hed_detector.safetensors"), torch_dtype=torch.bfloat16, ) pipe.enable_model_cpu_offload(gpu_id=1) out = pipe( image=Image.open("style_ref.jpg"), prompt="gothic pale woman, dramatic rim lighting", variation_strength=0.85, ctrl_strength=0.25, ip_strength=0.85, flow_shift=8.0, guidance_scale=4.5, num_inference_steps=50, ).images[0] out.save("output.jpg") ``` ## Recommended settings | Mode | `ctrl_strength` | `ip_strength` | `variation_strength` | |---|---|---|---| | Pure variation | 0.0 | 0.0 | 0.65–0.85 | | ControlNet only | 0.25 | 0.0 | 0.85 | | IP-Adapter only | 0.0 | 0.85 | 0.85 | | Full combo (best) | 0.25 | 0.35–0.85 | 0.85 | `flow_sh...

NoteOct 6huggingface.co/madtune/pixeldit-controlnet

Source context: 50 downloads · 1 likes · Pipeline image-to-image · Library diffusers · Repo madtune/pixeldit-controlnet

NoteJul 29huggingface.co/madtune/pixeldit-controlnet

Source context: 23 downloads · 0 likes · Pipeline image-to-image · Library diffusers · Repo madtune/pixeldit-controlnet