Recently, discrete Fourier transform (DFT)-based discrete multitone modulat
ion (DMT) systems have been widely applied to various applications. In this
paper, we study a broader class of DMT systems using more general unitary
matrices instead of DFT matrices. For this class, we will show how to desig
n the optimal DMT systems over frequency-selective channels with colored no
ise. In addition, asymptotical performance of DFT-based and optimal DMT sys
tems will be studied and shown to be equivalent, However, for a moderate nu
mber of bands, the optimal DMT system offers significant gain over the DFT-
based DMT system, as will be demonstrated by examples.