Tests of statistical methods for estimating galaxy luminosity function andapplications to the Hubble Deep Field

We study statistical methods for the estimation of the luminosity function (LF) of galaxies. We focus on four nonparametric estimators: the 1/V-max es timator, the stepwise maximum-likelihood estimator, Choloniewski's estimato r, and the improved Lynden-Bell estimator. The performance of the 1/V-max e stimator has recently been questioned, especially for the faint-end estimat ion of the LF. We improve these estimators for studies of the distant unive rse, and examine their performances for various classes of functional forms by Monte Carlo simulations. We also apply these estimation methods to the mock Two-Degree Field (2dF) redshift survey catalog prepared by Cole et al. We find that the 1/V-max estimator yields a completely unbiased result if there is no inhomogeneity, but is not robust against clusters or voids. Thi s is consistent with the well-known results, and we do not confirm the bias trend of 1/V-max claimed by Willmer in the case of a homogeneous sample. W e also find that the other three maximum-likelihood type estimators are qui te robust and give consistent results with each other. In practice, we reco mmend Choloniewski's estimator for two reasons: (1) it simultaneously provi des the shape and normalization of the LF; and (2) it is the fastest among these four estimators, because of its algorithmic simplicity. We then analy ze the photometric redshift data of the Hubble Deep Field prepared by Ferna ndez-Soto et al, using the above four methods. We also derive the luminosit y density, rho (L), at the B and I bands. Our B-band estimation is roughly consistent with that of Sawicki, Lin, & Yee, but it is a few times lower at 2.0 < z < 3.0. The evolution of rho (L)(I) is found to be less prominent.