This paper is devoted to the concept of instability in dynamical syste
ms with the main emphasis on orbital, Hadamard, and Reynolds instabili
ties. It demonstrates that the requirement about differentiability in
dynamics in some cases is not consistent with the physical nature of m
otions, and may lead to unrealistic solutions. Special attention is pa
id to the fact that instability is not an invariant of motion: it depe
nds upon frames of reference, the metric of configuration space, and c
lasses of functions selected for mathematical models of physical pheno
mena. This leads to the possibility of elimination of certain types of
instabilities (in particular, those which lead to chaos and turbulenc
e) by enlarging the class of functions using the Reynolds-type transfo
rmation in combination with the stabilization principle: the additiona
l terms (the so-called Reynolds stresses) are found from the condition
s that they suppress the original instability. Based upon these ideas,
a new approach to chaos and turbulence as well as a new mathematical
formalism for nonlinear dynamics are discussed.