Mapping the melatonin receptor. 6. Melatonin agonists and antagonists derived from 6H-isoindolo[2,1-a]indoles, 5,6-dihydroindolo[2,1-a]isoquinolines,and 6,7-dihydro-5H-benzo[c]azepino[2,1-a]indoles

6H-Isoindolo[2,1-a]indoles (5, 7, 10, 13), 5,6-dihydroindolo[2,1-a]isoquino lines (20, 21), and 6,7-dihydro-5H-benzo[c]azepino[2,1-a]indoles (23, 25, 2 7, 30) have been prepared as melatonin analogues to investigate the nature of the binding site of the melatonin receptor. The affinity of analogues wa s determined in a radioligand binding assay using cloned human mt(1) and MT 2 receptor subtypes expressed in NIH 3T3 cells. Agonist and antagonist pote ncy was measured using the pigment aggregation response of a clonal line of Xenopus laevis melanophores. The 2-methoxyisoindolo[2,1-a]indoles (7a-d) s howed much higher binding affinities than the parent isoindoles (5a-e), and whereas 7a-c were agonists in the functional assay, 7d and 5a-e were antag onists. The 2-ethoxyisoindolo[2,1-a]indoles (10a-d) showed reduced binding affinities compared to their methoxy analogues, while the 5-chloro derivati ve 13 showed a considerable reduction in binding affinity and potency compa red to 7a. The 10-methoxy-5,6-dihydroindolo[2,1-a]isoquinolines (21a-c) had higher binding affinities than the corresponding parent indoloisoquinoline s (20a-c) in the human receptor subtypes, and the parent compounds were ant agonists whereas the 10-methoxy derivatives were agonists in the functional assay. The N-cyclobutanecarbonyl derivatives of both the parent (20d) and 10-methoxyl (21d) series had similar binding affinities and were both antag onists with similar potencies. The 11-methoxy-6,7-5H-benzo[c]azepino[2,1-a] indoles (25a-d) had higher binding affinities than the corresponding parent compounds (23a-d) at the MT2 receptor but similar affinities at the mt(1) site; all of the compounds were antagonists in the functional assay. Changi ng 11-methoxy for 11-ethoxy decreased the binding affinity slightly, and th is was more evident at the MT2 receptor. All of the derivatives investigate d had either the same or a greater affinity for the human MT2 receptor comp ared to the mt(1) receptor (range 1:1-1:132). This suggests that the mt(1) and MT2 receptor pockets differ in their ability to accommodate alkyl group s in the indole nitrogen region of the melatonin molecule. Two compounds (7 c and 25c) were tested in functional assays on recombinant mt(1) and MT2 me latonin receptors. Compound 7c is a potent agonist with some selectivity (4 4-fold) for the MT2 receptor, while 25c is an MT2-preferring antagonist. In creasing the carbon chain length between N-1 of indole and the 2-phenyl gro up from n = 1 through n = 3 leads to a fairly regular decrease in the bindi ng affinity, but, remarkably, when n = 3, it converts the methoxy compounds from melatonin agonists to antagonists. The Xenopus melatonin receptor thu s cannot accommodate an N-n-alkyl chain attached to a 2-phenyl substituent with n > 2 in the required orientation to induce or stabilize the active re ceptor conformation.