Automated cell segmentation and spot detection in fluorescence in situ hybridization staining to assess HER2 status in breast cancer

Fluorescence in situ hybridization (FISH) approach is constituted of a pair of complementary techniques for precisely detecting gene amplification and over-expression which are regarded as signs of cancer in patients. Signal detection of FISH whole slides is extremely significant as enables to detect amplification situation. However, nuclei detection and segmentation of FISH slides through microscopic images is tedious and time-consuming for pathologists to evaluate. Furthermore, FISH specimen slides provided at pathological laboratories are frequently noisy and not analyzable entirely. Therefore, traditional visual methods require more time due to fact that they are exceedingly reliant on human view. They require more time and attention in the evaluation process by pathologists. Nowadays, computer-aided FISH solutions bring radical remedies in this particular problematic area of pathology. Although computer-aided FISH solutions have many advantages, they have some drawbacks due to the variability of staining images. In this study, we present an accurate cell nuclei segmentation and signal detection methodology to detect red and green spots localized in segmented cells. The problems in the visual experiments as well as the assessment of amplification state of the evaluated cases are presented, which corresponds to the visual scoring of pathologists.