Selective Gradient Diffusion for Localized Text-Guided Image Augmentation

Diffusion models have demonstrated strong capabilities in text-guided image editing; however, most existing approaches update the global parameters of the generative network, leading to unintended modifications in non-target regions and reduced structural fidelity. This work introduces a Selective Gradient Diffusion (SGD) framework for text-guided image-toimage augmentation, designed to achieve localized modifications while preserving the integrity of the surrounding content. The proposed architecture leverages a latent diffusion backbone in which low-rank adapters (LoRA) are embedded within crossattention layers of the U-Net to enable parameter-efficient finetuning. To further constrain edits, a region-weighted noise prediction loss emphasizes modifications within specified masks, and a gradient-masking strategy restricts weight updates to selected neurons. This combination ensures that edits driven by natural language prompts are confined to semantically relevant regions without disturbing unrelated pixels. Experiments conducted on interior design datasets demonstrate that the proposed method achieves higher edit precision, improved structural preservation, and reduced parameter overhead compared to state-of-the-art diffusion-based editing approaches. The results suggest that selective neuron updating in diffusion models offers an effective direction for controllable and efficient textguided image augmentation. The supportive code is available at: https://sucharithasu.github.io/SGDWeb/,