Regional Prompting with Regional LoRAs support — SDXL v1.0

About v1.1

ControlNet integration within regions can be tricky because ControlNet maps the reference image to the entire output image space, not to the mask space. This means that if the reference image contains a character in the center, the character will also be generated in the center of the output image, regardless of the mask.

To address this, you can use one of the following approaches:

Switch set_cond_area in the Regional Subgraph from the default setting to mask bounds (note: this may negatively affect interactions between regions),

Or use the newly added "Pad image inside the mask" subgraph, which repositions and pads the reference image so it fits within the mask area.

Also was added "Person Mask" subgraph, which allows extraction of a person mask from the input image for more precise region mapping.

This subgraph uses a YOLO segmentation model, which should be placed in:

/models/ultralytics/segm

(you can download model here )

Known Issues with the Workflow

Full mask coverage is required Masks must collectively cover the entire image. The combined masks from all regions should span the full image area. If any part of the image is not covered by at least one mask, that area will not be processed correctly during regional sampling and may appear empty, flat, or incorrectly rendered. If it is not possible to cover the entire image with masks, you can merge all masks into one, invert it, and use the result as an additional (background) region.

Gradient (soft) masks are not supported The workflow does not work correctly with gradient (soft) masks. If a mask contains values below 100% (partial transparency), the sampler cannot properly denoise those areas. This results in visible noise, artifacts, or unstable textures. In some cases, you may reduce noise from non-binary masks by adding an additional KSampler with low CFG, step count, and denoise values—but this is not guaranteed to fully fix the issue.

The workflow in theory should work with any base model regardless of architecture (for now it has been tested on SDXL models, Chroma, Flux, and ZImage).

The workflow uses the recent ComfyUI subgraph feature , which allows extending the setup from 2 regions to any number of regions by simply duplicating the Regional Subgraph (and corresponding Regional FaceDetailer if needed).

Regional Subgraph structure

Each Regional Subgraph represents one independent region of the image and contains everything required to define a single character or element.

The inputs of the Regional Subgraph are:

region mask Defines where this region is applied in the image. This can come from any mask generation method (rect masks, segmentation, depth, manual paint, etc.).

regional positive conditioning The prompt that describes the character or object inside this region.

regional negative conditioning Required, but can be either global negative prompt or region-specific.

model (branched) This is a key part of the workflow. Instead of using the base model directly, the model is first branched and modified (for example by applying LoRAs) before being passed into the Regional Subgraph.

Model branching and region-specific LoRAs

Each region has its own model branch. This is done by taking the base model and applying region-specific LoRAs before feeding it into the Regional Subgraph.

Typical setup:

Base model with global LoRAs → shared for the whole image and passed directly into Regional Sample

For each region:

duplicate the model path

apply one or more LoRAs specific to that region

pass the modified model into the Regional Subgraph model input

This allows each region to have completely different:

characters

styles

LoRAs

visual identity

Importantly, this avoids the common problem where LoRAs are applied globally and interfere with each other.

Extending the workflow

To add more regions:

Copy an existing Regional Subgraph

Connect:

a new mask

a new prompt

a new model branch (with its own LoRAs)

Add the output to:

Combine Conditioning

CombineRegionalPrompts

This makes the system fully scalable without restructuring the graph.

Customization

Everything except the following can be replaced depending on user preference or task:

Regional Subgraphs and Regional Sampler (structure must remain for modularity)

Combine Conditioning

CombineRegionalPrompts

Users are free to replace:

Mask generation (rect masks, segmentation, ControlNet, etc.)

Prompt encoding pipeline

LoRA setup

FaceDetailer or any post-processing

This allows adapting the workflow to:

character scenes

object composition

style mixing

inpainting pipelines

animation setups

Important note: Base steps vs Regional sampling

One of the most important parameters in this workflow is base_only_steps , which controls how the generation process is split between:

Base sampling stage

Regional sampling stage

How it works

The total number of steps is defined by the sampler (for example 30 or 50 steps).

These steps are divided into two phases:

Base sampling (base_only_steps)

Only the global prompt is applied

No regional prompts or masks are used yet

This stage defines:

composition

pose

camera angle

general structure of the image

Regional sampling (remaining steps)

Regional prompts and masks are applied

Each region modifies its assigned part of the image

This stage refines:

character identity

local details

LoRA-specific features

Why this matters

The base stage essentially “locks in” the structure of the image.

If too many steps are spent in base sampling:

the image becomes stable and coherent

but regions have less ability to change it

LoRAs may appear weak or not recognizable

If too few steps are spent in base sampling:

regions have strong influence

characters become more accurate

but overall composition may become unstable or inconsistent

Recommended default

A good starting point is:

base_only_steps = 50% of total steps

This provides a balance between:

stable composition

effective regional control

When to increase base_only_steps

Increase base steps if:

composition is broken or inconsistent

characters are not aligned properly

perspective or layout is unstable

This gives the base stage more time to establish a solid structure.

When to decrease base_only_steps

Decrease base steps if:

region-specific LoRAs are weak

characters are not recognizable

regional prompts have little effect

This allows the regional stage to have more influence over the final image. However prefer increasing total steps first before decreasing base_only_steps

Practical intuition

Base stage = “what the image looks like”

Regional stage = “who/what is inside each part”

Balancing these two is key to getting both:

strong composition

strong character identity