Empirical Bayes method

In statistics, empirical Bayes methods involve:

• An "underlying" probability distribution of some unobservable quantity assigned to each member of a statistical population. This quantity is a random variable if a member of the population is chosen at random. The probability distribution of this random variable is not known, and is thought of as a property of the population.

• An observable quantity assigned to each member of the population. When a random sample is taken from the population, it is desired first to estimate the "underlying" probability distribution, and then to estimate the value of the unobservable quantity assigned to each member of the sample.

This is probably incomprehensible without concrete examples.

Contents

1 Examples 1.1 The original example, introduced by Herbert Robbins in 1956 1.2 An example involving the normal distribution 2 References

Examples

The original example, introduced by Herbert Robbins in 1956

Each customer of an insurance company has an "accident rate" Θ and is insured against "accidents"; the probability distribution of Θ is the "underlying" distribution, and is unknown. The number of "accidents" suffered by each customer in a specified baseline time period has a Poisson distribution whose expected value is the particular customer's "accident rate". That number of "accidents" is the observable quantity. A crude way to estimate the underlying probability distribution of the "accident rate" Θ is to estimate the proportion of members of the whole population suffering 0, 1, 2, 3, ... accidents during the specified time period to be equal to the corresponding proportion in the observed random sample. Having done so, it is then desired the "accident rate" of each customer in the sample. One may use the conditional expected value of the "accident rate" Θ given the observed number X of "accidents" during the baseline period. Given the assumed Poisson distribution of accidents, one can show that

<math>E(\Theta\mid X=x)={(x+1)P(X=x+1) \over P(X=x)}.\qquad\qquad\qquad (*)<math>

The quantities P(X = x + 1) and P(X = x) must be estimated based on the sample. That is why the word empirical appears in the name of this concept. The conditional expected value of Θ given the observed value X = x is found by using Bayes' theorem. That is why the word Bayes appears.

Thus, if a customer suffers six "accidents" during the baseline period, that customer's estimated "accident rate" is 7 × [the proportion of the sample who suffered 7 "accidents"] / [the proportion of the sample who suffered 6 "accidents"].

Proof of the identity labeled with the asterisk above:

Denote the probability density function of the underlying "accident rate" <math>\Theta<math> by <math>f_\Theta(\theta)<math> (as is often done in probability theory, we use capital letters for random variables and corresponding lower-case letters for the dummy variables in the density or mass functions). Denote the conditional probability mass function of the number of accidents suffered by a randomly chosen customer during the baseline period, given that that customer's "accident rate" is <math>\theta<math>, by <math>f_{X\mid\Theta=\theta}(x)<math>. This conditional distribution was assumed to be a Poisson distribution; therefore we have

<math>f_{X\mid\Theta=\theta}(\theta)={\theta^x e^{-\theta} \over x!}.<math>

By Bayes' theorem, the conditional probability density function of <math>\theta<math> given the event X = x is

<math>{\theta^x e^{-\theta} \over x!}f_{X\mid\Theta=\theta}(x)/[\mbox{normalizing}\ \mbox{constant}].<math>

The normalizing constant by which we divide is the integral with respect to <math>\theta<math> from 0 to ∞ of the function in the numerator. Consequently, the conditional expected value of <math>\Theta<math> given the observed number x of accidents is

<math>{\int_0^\infty \theta f_\Theta(\theta)f_{X\mid\Theta=\theta}(x)\,d\theta \over \int_0^\infty f_\Theta(\theta)f_{X\mid\Theta=\theta}(x)\,d\theta}={\int_0^\infty \theta f_\Theta(\theta){\theta^x e^{-\theta} \over x!}\,d\theta \over \int_0^\infty f_\Theta(\theta){\theta^x e^{-\theta} \over x!}\,d\theta}={\int_0^\infty \theta^{x+1} e^{-\theta} f_\Theta(\theta)\,d\theta \over \int_0^\infty \theta^x e^{-\theta} f_\Theta(\theta)\,d\theta}<math>

<math>={E\left(\Theta^{x+1} e^{-\Theta}\right) \over E\left(\Theta^x e^{-\Theta}\right)}={(x+1)! E(P(X=x+1\mid \Theta)) \over x! E(P(X=x\mid \Theta))}.<math>

By the law of total probability (and the routine cancellation of factorials), this is equal to

<math>{(x+1) P(X=x+1) \over P(X=x)}.<math>

An example involving the normal distribution

Suppose the weights of a large population of 35-year-old men are normally distributed with expected value μ and standard deviation σ. A crude measuring instrument measures a man's weight with a measurement error that is normally distributed with expected value 0 and standard deviation τ. The man's true weight is not observable; his weight measured with error is observed. The conditional probability distribution of a randomly chosen man's true weight, given his weight-measured-with-error, can be found by using Bayes' theorem, and then the conditional expected value can be used as an estimate of his true weight, provided that the values of μ, σ, and τ are known. But they are not. One may use the data to estimate the standard deviation of the measurement errors by measuring each man multiple times. One may similarly estimate the population average weight and the population standard deviation of weights by weighing multiple men. These estimates of parameters based on the data are the occasion for the use of the word empirical. Finally, one may then estimate the aforementioned conditional expected true weight by using Bayes' theorem.

Mathematical details are still to be added here.

References

• Herbert Robbins, An Empirical Bayes Approach to Statistics, Proceeding of the Third Berkeley Symposium on Mathematical Statistics, volume 1, pages 157-163, University of California Press, Berkeley, 1956.