Application of nonlinear magneto-optical (Faraday) rotation to magnetometry
is investigated. Our experimental setup consists of a modulation polarimet
er that measures rotation of the polarization plane of a laser beam resonan
t with transitions in Rb. Rb vapor is contained in an evacuated cell with a
ntirelaxation coating that enables atomic ground-state polarization to surv
ive many thousand wall collisions. This leads to ultranarrow features (simi
lar to 10(-6) G) in the magnetic-field dependence of optical rotation. The
potential sensitivity of this scheme to sub-mu G magnetic fields as a funct
ion of atomic density, light intensity, and light frequency is investigated
near the D1 and D2 lines of Rb-85. It is shown that through an appropriate
choice of parameters the shot-noise-limited sensitivity to small magnetic
fields can reach 3x10(-12) G/root Hz.