Minimum relative entropy and probabilistic inversion in groundwater hydrology

The similarity between maximum entropy (MaxEnt) and minimum relative entrop y (MRE) allows recent advances in probabilistic inversion to obviate some o f the shortcomings in the former method. The purpose of this paper is to re view and extend the theory and practice of minimum relative entropy. In thi s regard, we illustrate important philosophies on inversion and the similar ly and differences between maximum entropy, minimum relative entropy, class ical smallest model (SVD) and Bayesian solutions for inverse problems. MaxE nt is applicable when we are determining a function that can be regarded as a probability distribution. The approach can be extended to the case of th e general linear problem and is interpreted as the model which fits all the constraints and is the one model which has the greatest multiplicity or "s preadout" that can be realized in the greatest number of ways. The MRE solu tion to the inverse problem differs from the maximum entropy viewpoint as n oted above. The relative entropy formulation provides the advantage of allo wing for non-positive models, a prior bias in the estimated pdf and 'hard' bounds if desired. We outline how MRE can be used as a measure of resolutio n in linear inversion and show that MRE provides us with a method to explor e the limits of model space. The Bayesian methodology readily lends itself to the problem of updating prior probabilities based on uncertain field mea surements, and whose truth follows from the theorems of total and compound probabilities. In the Bayesian approach information is complete and Bayes' theorem gives a unique posterior pdf. In comparing the results of the class ical, MaxEnt, MRE and Bayesian approaches we notice that the approaches pro duce different results. In comparing MaxEnt with MRE for Jayne's die proble m we see excellent comparisons between the results. We compare MaxEnt, smal lest model and MRE approaches for the density distribution of an equivalent spherically-symmetric earth and for the contaminant plume-source problem. Theoretical comparisons between MRE and Bayesian solutions for the case of the linear model and Gaussian priors may show different results. The Bayesi an expected-value solution approaches that of MRE and that of the smallest model as the prior distribution becomes uniform, but the Bayesian maximum a posteriori (MAP) solution may not exist for an underdetermined case with a uniform prior.