Score (statistics)
|
|
In statistics, the score is the derivative, with respect to some parameter <math>\theta<math>, of the logarithm (commonly the natural logarithm) of the likelihood function. If the observation is <math>X<math>, then the score <math>V<math> can be found through the chain rule:
- <math>
V = \frac{d}{d\theta} \log L(\theta;X) = \frac{1}{L(\theta;X)} \frac{d L(\theta;X)}{d\theta}. <math>
Note that <math>V<math> is a function of <math>\theta<math> and the observation <math>X<math>. The score <math>V<math> is a sufficient statistic for <math>\theta<math>.
The expected value of <math>V<math>, written <math>\mathrm{E}(V)<math>, is zero. To see this, rewrite the definition of expectation, using the fact that the probability of observing <math>x<math> is just <math>L(\theta; x)<math>, which is conventionally denoted by <math>f(x; \theta)<math> (in which the dependence on <math>x<math> is more explicit). With this change of notation and writing <math>f(x; \theta)<math> for the derivative with respect to <math>\theta<math>,
- <math>
\mathrm{E}(V) = \int
\frac{f'(x; \theta)}{f(x; \theta)} f(x; \theta)
dx = \int \frac{d f(x; \theta)}{d\theta} \, dx <math>
where the integral runs over the whole of the probability space and a dash denotes differentiation with respect to <math>\theta<math>. If certain differentiability conditions are met, the integral may be rewritten as
- <math>
\frac{d}{d\theta} \int f(x; \theta) \, dx = \frac{d}{d\theta}1 = 0 <math>.
It is worth restating the above result in words: the expected value of the score is zero. Thus, if one were to repeatedly sample from some distribution, and repeatedly calculate the score, then the mean value of the scores would tend to zero as the number of repeat samples approached infinity.
The variance of the score is known as the Fisher information and is written <math>\mathcal{I}(\theta)<math>. Because the expectation of the score is zero, this may be written as
- <math>
\mathcal{I}(\theta) = \mathrm{E} \left[
\left[
\frac{d}{d \theta} \log L(\theta;X)
\right]^2
\right] <math>.
Note that information, as defined above, is not a function of a particular observation, as the random variable <math>X<math> has been averaged out. This concept of information is useful when comparing two methods of observation of some random process.