Systems of nonlinear differential equations with periodic coefficients, whi
ch include Hill's and Mathieu's equations as examples in the linear limit,
an important from a practical point of view. Nonlinear Hill's equations mod
el a variety of dynamical systems of interest to physics and engineering, i
n which perturbations enter as periodic modulations of their linear frequen
cies. As is well known, the stability properties of some fundamental period
ic solutions of these systems is often an essential problem. The main purpo
se of this paper is to concentrate on one such class of nonlinear Hill's eq
uations and study the stability properties of some of their simplest period
ic solutions analytically as well as numerically. To accomplish this task,
we first use an extension of the generalized averaging method to approximat
e these solutions and then apply the technique of multiple scaling to perfo
rm the stability analysis. A three-particle system with free-free boundary
conditions is studied as an example. The accuracy of our results is tested,
within the limits of first-order perturbation theory, and is found to be w
ell confirmed by numerical experiments. The stability analysis of these sim
ple periodic solutions, though local in itself, can yield considerable info
rmation about more global properties of the dynamics, since it is in the vi
cinity of such solutions that the largest regions of regular or chaotic mot
ion are usually observed, depending on whether the periodic solution is, re
spectively, stable or unstable. (C) 2000 Elsevier Science B.V. All rights r
eserved.