Color-Aware Re-Ranking for Fashion Image Retrieval

Color plays a central role in fashion perception and user preference, yet modern embedding-based image retrieval systems primarily optimize for semantic and structural similarity, often neglecting chromatic coherence. As a result, retrieved garments may be category-correct but visually inconsistent in color. This paper introduces CA-Rank, a segmentation-driven, model-agnostic post-retrieval framework that enhances fashion image retrieval by explicitly incorporating perceptual color similarity.Each image is first processed by a class-agnostic segmentation model to isolate garment pixels, ensuring that both embeddings and color descriptors are computed on garment-only regions. CA-Rank evaluates two color similarity strategies:Weighted Color Matching (WCM), which compares discrete HSV-based color compositions, and Weighted Chamfer Similarity (WCS), which operates in the CIE-LAB color space on quantized palettes using a symmetric, weight-aware Chamfer formulation. While WCM provides a simple baseline, WCS demonstrates substantially higher robustness to illumination, mixed hues, and fine-grained chromatic variation; therefore, the final re-ranking relies on WCS to refine the top-K candidates without any additional training or modification of the underlying retrieval model. Experiments on a segmented subset of the In-Shop Clothes Retrieval benchmark show consistent improvements in Hit@K across diverse backbones, with the largest gains at top ranks where users are most sensitive to color mismatches. Qualitative analyses further indicate that CA-Rank produces retrieval results that are both semantically accurate and perceptually color-consistent, offering a simple and effective mechanism for improving the visual coherence of fashion search systems.