We discuss the long term maintenance of acquired memory in synaptic co
nnections of a perpetually learning electronic device. This is affecte
d by ascribing each synapse a finite number of stable states in which
it can maintain for indefinitely long periods. Learning uncorrelated s
timuli is expressed as a stochastic process produced by the neural act
ivities on the synapses. In several interesting cases the stochastic p
rocess can be analyzed in detail, leading to a clarification of the pe
rformance of the network, as an associative memory, during the process
of uninterrupted learning. The stochastic nature of the process and t
he existence of an asymptotic distribution for the synaptic values in
the network imply generically that the memory is a palimpsest but capa
city is as low as log N for a network of N neurons. The only way we fi
nd for avoiding this tight constraint is to allow the parameters gover
ning the learning process (the coding level of the stimuli; the transi
tion probabilities for potentiation and depression and the number of s
table synaptic levels) to depend on the number of neurons. It is shown
that a network with synapses that have two stable states can dynamica
lly learn with optimal storage efficiency, be a palimpsest, and mainta
in its (associative) memory for an indefinitely long time provided the
coding level is low and depression is equilibrated against potentiati
on. We suggest that an option so easily implementable in material devi
ces would not have been overlooked by biology. Finally we discuss the
stochastic learning on synapses with variable number of stable synapti
c states.