A MODEL FOR EVALUATION OF IN-SITU HYBRIDIZATION SPOT-COUNT DISTRIBUTIONS IN TISSUE-SECTIONS

The interpretation of in situ hybridization (ISH) spot-count distribut ions, obtained from evaluation of ISH signals in tissue sections, is c omplicated by the unknown impact of nuclear truncation and of the loca lization of ISH spots within the nuclei. In this study, a mathematical model was developed to investigate the effects of nuclear truncation and of the distribution of ISH spots within the nucleus on the ISH spo t-count distribution in tissue sections. In this model, it was assumed that nuclei are spherical and of constant diameter and that ISH spots have negligible size and are distributed randomly within the nucleus (''volume model'') or along the nuclear membrane (''surface model''). A minimal nuclear profile diameter was introduced in order to study th e effect of rejecting small nuclear fragments for spot-count evaluatio n. Given the section thickness, the nuclear size, the minimal nuclear profile diameter, and the true number of ISH spots per nucleus and the ir spatial distribution within the nucleus, the model predicts the pro portion of nuclei observable in the section with a specific number of ISH spots. A program that performs the model calculations was develope d for PC and is available upon request. For section thickness greater than 50% of the nuclear diameter, the main effect of increasing sectio n thickness on spot-count distributions was the increase of the propor tion of nuclei with the true chromosome copy number of spots. For lowe r section thickness, the total distribution shifted towards lower spot frequencies. The influence of the minimal profile diameter was most n otable for values close to the nuclear diameter. The effect of the loc alization of ISH spots within the nucleus was shown to be prominent, e specially for sections with thickness smaller than the nuclear diamete r. Good correspondence between model-predicted distributions and measu red distributions was obtained using the volume model and taking into account only large nuclear profiles. (C) 1995 Wiley-Liss, Inc.