When a thin binary layer grows epitaxially on an elemental substrate, the c
omposition often modulates in the plane of the layer. The layer may even de
mix when the bulk of the same composition is miscible. Sometimes the layer
separates into two phases, forming periodic stripes or other regular patter
ns. The size of the separated phases may be in the range 1-100 nm, and stab
le against coarsening on annealing. This paper develops a thermodynamic fra
mework to study these remarkable phenomena. For an epilayer less than a few
monolayers thick, the excess energy cannot be attributed to individual sou
rces of superficial misfit. Instead, we lump the epilayer and adjacent mono
layers of the substrate into a single superficial object, and specify the e
xcess surface energy for the object. The variation of the surface energy de
nsity with the variation of the strain defines surface stress. When the com
position modulates in the epilayer, the surface stress is nonuniform, defor
ms the substrate, and reduces the total energy. Consequently, the compositi
on-dependent surface stress tends to refine phases, resisting coarsening. I
n a stability analysis, we perturb a uniform concentration field into a sin
usoidal field with a small amplitude and an arbitrary wavelength. The entro
py of mixing stabilizes the uniform layer for long wavelengths. To stabiliz
e the uniform layer for short wavelengths, we assume that the surface energ
y density also depends on concentration-gradient. A stability condition sho
ws the combined elastic, entropic, and gradient effects. We also outline a
dynamical system, which can be used to study pattern emergence and evolutio
n during annealing. (C) 2000 Elsevier Science Ltd. All rights reserved.