Receiver operating characteristic
|
|
In signal detection theory, a receiver operating characteristic (ROC) is a graphical plot of the sensitivity (fraction of true positives - TP) vs. 1-specificity (the fraction of false positives - FP) for a binary classifier system as its discrimination threshold is varied. The usage receiver operator characteristic is also common.
ROC curves are used to evaluate the results of a prediction and were first employed in the study of discriminator systems for the detection of radio signals in the presence of noise in the 1940s. In the 1960s they began to be used in psychophysics, to assess human (and occasionally animal) detection of weak signals. They also proved to be useful for the evaluation of machine learning results, such as the evaluation of Internet search engines. They are also used extensively in epidemiology and medical research.
The best possible prediction method would yield a graph that was a point in the upper left corner of the ROC space, i.e. 100% sensitivity (all true positives are found) and 100% specificity (no false positives are found). A completely random predictor would give a straight line at an angle of 45 degrees from the horizontal, from bottom left to top right: this is because, as the threshold is raised, equal numbers of true and false positives would be let in. Results below this no-discrimination line would suggest a detector that gave wrong results consistently, and could therefore be simply used to make a detector that gave useful results by inverting its decisions.
Sometimes, the ROC is used to generate a summary statistic. Three common versions are:
- the intercept of the ROC curve with the line at 90 degrees to the no-discrimination line
- the area between the ROC curve and the no-discrimination line
- d' (pronounced "d-prime"), the distance between the mean of the distribution of activity in the system under noise-alone conditions and its distribution under signal plus noise conditions, divided by their standard deviation, under the assumption that both these distributions are normal with the same standard deviation.
However, any attempt to summarize the ROC curve into a single number loses information about the pattern of tradeoffs of the particular discriminator algorithm.
In engineering, the area between the ROC curve and the line is most frequently used statistic, because of its useful mathematical properties as a non-parametric statistic. This area is often simply known as the discrimination. In psychophysics, d' is the commonest measure.
Illustration of the use of ROC graphs for discrimination. TP - true positives, FP - false p., TN - true negatives, FN - false n.. Starting from two distributions of positives (red) and negatives (blue) one can apply a threshold criterion (vertical line) to arbitrarily separate the two. For overlapping distributions, there is always a tradeoff between sensitivity (TP) and specificity (1-FP). TP and FN as well as TN and FP both add up to 1. Sliding the threshold line towards the distribution of positives will result in a decreased probability for true positive detection P(TP) and FPs, which is equivalent to moving the ROC curve (dashed) downwards. If the two distributions overlap completely, the ROC curve will be the diagonal shown as the dot-dashed curve.
See also
External links
- Receiver Operating Characteristic bibliography (http://splweb.bwh.harvard.edu:8000/pages/ppl/zou/roc.html)
- A simple example of a ROC curve (http://gim.unmc.edu/dxtests/roc2.htm)
- applets (http://www.anaesthetist.com/mnm/stats/roc/)
- A more thorough treatment of ROC curves and signal detection theory (http://www-psych.stanford.edu/~lera/psych115s/notes/signal/)