A theory of selective reflection (SR) of a light beam from the interfa
ce of a dielectric medium and a dense atomic vapor is presented. Follo
wing another earlier work on this subject [J. Guo et al., Opt. Commun.
110, 732 (1994)], we calculate the atomic density matrix and the tran
smitted field in the vapor self-consistently. We extend our previous t
heory to include an attractive atom-dielectric wall interaction and mu
ltiple atomic transitions. We examine the limits at which the conventi
onal SR theories, which are based on the assumption of a plane-wave fi
eld in the vapor, start to fail. It is shown that at or above vapor de
nsities of order 10(17)/cm(3), the frequency shifts of the SR line sha
pe due to the local-field correction (Lorentz-Lorenz shift), the atom-
wall interaction, and the nonexponential attenuation of the field in t
he vapor ran have comparable orders of magnitude, and the proper deter
mination of all these effects is essential for calculating the correct
SR signals. A comparison of the theoretical SR spectra with an experi
mental result for a Rb vapor with a density of order 10(17)/cm(3) is a
lso presented.