Three-dimensional particle-in-cell simulations of the interaction of an ult
ra-intense linearly-polarized laser light with an overdense plasma are pres
ented. Intense laser radiation is shown to be unstable against modulation b
oth in the direction of the laser propagation direction and in the directio
n perpendicular to the polarization direction. Growth rate of the instabili
ty has a maximum when laser frequency is of the order of the plasma frequen
cy modified due to the relativistic increase of electron mass in the laser
field. Analytical description of the instability is also presented.