In this paper, we address the problem of separation of mutually independent
sources in nonlinear mixtures. First, we propose theoretical results and p
rove that in the general case, it is not possible to separate the sources w
ithout nonlinear distortion. Therefore, we focus our work on specific nonli
near mixtures known as post-nonlinear mixtures. These mixtures constituted
by a linear instantaneous mixture (linear memoryless channel) followed by a
n unknown and invertible memoryless nonlinear distortion, are realistic mod
els in many situations and emphasize interesting properties i.e., in such n
onlinear mixtures, sources can be estimated with the same indeterminacies a
s in instantaneous linear mixtures, The separation structure of nonlinear m
ixtures is a two-stage system, namely, a nonlinear stage followed by a line
ar stage, the parameters of which are updated to minimize an output indepen
dence criterion expressed as a mutual information criterion, The minimizati
on of this criterion requires knowledge or estimation of source densities o
r of their log-derivatives. A first algorithm based on a Gram-Charlier expa
nsion of densities is proposed, Unfortunately, it fails for hard nonlinear
mixtures. A second algorithm based on an adaptive estimation of the log-der
ivative of densities leads to very good performance, even with hard nonline
arities. Experiments are proposed to illustrate these results.