We study the unusual behavior of the incommensurate lattice formed Ly Li on
the anisotropic trough-like surface of Mo(112). Starting at a coverage of
0.66, we found a continuous transformation of a rectangular into an oblique
unit cell. This oblique cell transforms back into a rectangular cell at co
verages higher than 0.85. Both transitions are explained by an instability
in the shear modulus of the layers at the respective critical coverages. It
leads to the development of smectic-like fluctuations and finally to a mel
ting transition. For the oblique phase below melting, long-range correlatio
ns along the troughs, but correlations over less than 10 lattice constants
normal to the troughs are observed in experiment. The experimental results
are in reasonable agreement with the theoretical model proposed.