Talk:Normal distribution

Every reference describing normal distributions ultimately says this, and yet I have never seen a single normal distribution that I would describe first and foremost as bell-shaped. Some of them do perhaps mildly resemble some quite odd-shaped bells, but I think even these cases are a stretch. Does anyone else find this terminology silly, or, even better, know its historical development? --Ryguasu 10:22 Dec 2, 2002 (UTC)

pdf, cdf, plots, tables

Why is there a table of the Gaussian cumulative distribution function here?

Wikipedia is not a repository of source texts,
it is not useful as an authoritative source (as anyone can edit), and
a graph would be much more informative about its properties.

-- Anon.

occurence of the normal distribution

challenges to conventional wisdom

I don't like the examples at all. If a species shows sexual dimorphism, the size of specimens won't be a gaussian, just like the text points out about human blood pressure. Also, test scores are basically an example of the Gaussian limit of Binomials, and GPAs certainly do not follow a gaussian distribution because of grade inflation and limited range of grade points.

To me, these examples smack of the fallacy that "everything is gaussian". See Zipfs law

My rewrite of this page is still under way, in any case. -- Miguel

You could add those counter examples to the list of variables that don't follow the Normal distribution.

I don't understand the point about IQ scores and binomials, though. Which binomials are you thinking of? AxelBoldt

Test scores and the binomial-to-gaussian limit

It's a bit more complicated than I made it sound, but the point is that the test score is basically a count of the number of correct answers, and therefore is a discrete variable like a Binomial and not a continuous variable like a Normal, so there is a Binomial-to-Normal limit involved. Here's the actual argument, written out:

Take a test that is composed of N True/False questions. Characterize a test-taker by their probability p of getting a right answer. Then their score will be a binomial B(N,p).

Now, consider a population of test takers. There will be a function F(p) on [0,1] which is the probability density that a randomly selected test-taker will have probability p of getting the right answer. Then, when you administer the test to a sample of test-takers, the probability distribution of the number of correct answers will be the convolution of B(N,p) and F(p), or

P(n) = int_[0,1] P(B(N,p)=n) F(p) dp

-- Miguel

If N is big enough, then the distribution of n/N should be pretty much the same as the distribution of p though (is that right?). So the truly interesting question is then: Why is p approximately normally distributed (or is it?). I would claim that it is, because of the central limit theorem (p pretty much describes the "intelligence" of a person, which is the result of many small mostly indepedent additive effects). AxelBoldt

If N is big enough we can use the central limit theorem to replace P(B(N,p)=n) by the density of a N(Np,Np(1-p)). In other words,

P(n/N=x) = int_[0,1] exp(-N(p-x)²/2p(1-p)) (N/2πp(1-p))^1/2 F(p) dp

This is not exactly a gaussian convolution, but it comes close. The observed P(n/N) is a smoothed-out version of F(p). The two features of this that I want to stress are 1) we did use the theorem of de Moivre-Laplace; 2) the gaussian has variance p(1-p)/N, which means that the test performs best with very good or very bad test-takers, for whom the test is unnecessary anyway; 3) in the limit of infinite N you recover F(p) exactly.

Now, I don't know what the distribution of F(p) should be, but it has to be on [0,1]. The natural family of distributions on [0,1] is the Beta distribution, but that doesn't mean that it has to be a Beta.

I'll create an entry for the Beta shortly. -- Miguel

Normals and lognormals in biology

I was just reading

Huxley, Julian: Problems of Relative Growth (1932)

and the overwhelming biological evidence is that (as pointed out in the text) growth processes proceed by multiplicative increments, and that therefore body size should follow a lognormal rather than normal distribution. The size of plants and animals is approximately lognormal.

Of course, a sum of very many small lognormal increments approaches a normal, but except in the growth of inert appendages such as shells, hair, nails, claws and teeth, growth is best modeled by a multitude of random multiplicative increments.

One should not expect human height distributions in humans to be normal, but lognormal, and the usual statement that they are normally distributed is not supported by an application of the Central Limit Theorem.

--Miguel

statistical tests of the claims

How is it determined that IQ scores, heights, etc. are "approximately normal"? Does someone just collect a very large sample, plot a histogram, and go "wow - it looks like a bell!" I assume there are more formal methods. Also, does anyone have references for the studies concluding the approximate normality of the variables discussed in the article? --Ryguasu 15:35 Dec 10, 2002 (UTC)

The answer is that until someone comes up with a causation model that explain why it should be normal, it is just a guess. It is a common fallacy that "everything is Gaussian". -- Miguel

The tests to check whether a given distribution is normal (or lognormal etc.) are called "goodness of fit" tests. One simple minded approach is to divide the variable's range into subintervals, let the theoretical distribution predict the probabilities of the various subintervals, and compare those predictions to the observed frequencies with a chi-square test. I don't have references for the relevant studies. Once you have empirically verified that a given distribution is approximately normal, you can of course dream up all sorts of explanations, typically that the given variable can be seen as the result of many small additive influences. AxelBoldt 23:36 Dec 10, 2002 (UTC)

It's not clear to me that tests for normality indicate anything useful. Specifically I am referring to chi-square tests and Kolmogorov-Smirov tests; these and related tests are conventional null-hypothesis significance tests, and share the problems common to such tests. In particular, since it is known a priori that there are upper and lower limits to human height (for example), the distribution of human height cannot be normal, nor can it be log-normal. Thus any failure to reject the null hypothesis can be due only to lack of data. A test which tells you "you have a big data set" isn't very useful; it certainly isn't any more useful than looking at a histogram and saying, "Hmm, yes, looks normal to me". If we need to mention such tests, I'd direct the reader elsewhere. Disclaimer: I am a Bayesian; be that as it may, please respond to technical issues. Wile E. Heresiarch 20:28, 5 Jan 2004 (UTC)

Several people have indicated to me that there are indeed articles and/or books out there somewhere that go through real-life data in a number of domains, showing many instances of the normal distribution being a good approximate fit. Unfortunately, nobody seems to remember where they might have seen these discussions. Has anyone seen one? I would really appreciate seeing some real data, rather than just being told what seems to be roughly normal and what doesn't. --Ryguasu 21:14 Feb 10, 2003 (UTC)

Funny you should ask. I have exactly the same question. Every statistics textbook claims this, but they never seem to back it up with references. I posted the same question to usenet, and here (http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&threadm=40200384.0302071614.438fe80c%40posting.google.com&rnum=1&prev=/groups%3Fq%3Dauthor:axel%2Bauthor:boldt%26hl%3Den%26lr%3D%26ie%3DUTF-8%26scoring%3Dd%26selm%3D40200384.0302071614.438fe80c%2540posting.google.com%26rnum%3D1) is what I got. AxelBoldt 21:55 Feb 10, 2003 (UTC)

Somebody pointed me to History of Statistics books by Stigler, and they look promising. AxelBoldt 15:51 Feb 12, 2003 (UTC)

Axel, what did you find in Stigler's books? -- Miguel

Maximum-entropy argument

Isn't it true that the normal distribution N(μ,σ²) is the distribution with the largest entropy among all distributions with mean μ and variance σ²? That would make it the "default choice" in a certain sense. AxelBoldt

Yes, you can show that the distribution with the largest entropy given <X> and <X²> has density proportional to exp(-ax-bx²).

The entropy of the density f is the expected value of log f. Finding the maximum entropy given <x> and <x^2> is an easy variational problem with Lagrange multipliers. — Miguel 05:15, 2004 Apr 20 (UTC)

Proposal to reorder the examples

I'd like to suggest that the examples section be put in this order: (1) central limit theorem and approximation of other distributions by norml distribution; (2) exactly normal distributions found in nature; (3) close approximations to normal distributions found in nature; (4) non-normal distributions found in nature. I'd further suggest that we strike the IQ example. The examples section shouldn't contain distributions that you might possibly consider normal, if you're feeling charitable and you are willing to overlook some serious difficulties: the examples should be solid examples. Wile E. Heresiarch 20:28, 5 Jan 2004 (UTC)

I've reordered the examples as described above. To further improve the examples, I would suggest (1) merging the bullet list with the named sections; I don't think it buys us anything to have both, and (2) for each example, give a reference to a work in the appropriate field -- e.g., physics, economics, etc, not a statistics book. Wile E. Heresiarch 19:27, 11 Jan 2004 (UTC)

Part of the problem is that, other than mathematical applications of the central limit theorem, we have not been able to substantiate the common claim that all kinds of things are approximately normal. Axel actually undertook a search for actual successful tests of normality in the literature and found only claims without references. I would strike out the IQ example, too, but given the common association of IQ with "the Bell curve", we might have to spend some time debunking it. I replaced the claim that sizes of biological specimens are normal with the discussion of growth, which has been standard biology since the 1930's but still has not made it to statistics textbooks! I agree on the reorganization, but I would add (5) common misconceptions about normality. -- Miguel 12:11 6 Jan 2004

normals and multivariate random variables

I have a question: is the sum of two normal variables always normal, even if the two variables are not independent? (Let's treat constant variables as normal for now, with σ=0.) --AxelBoldt

I just got the answer from the EFnet irc channel #math: the sum does not have to be normal. To quote:

 <not_cub> How about two normals dependent in the following way, (means are 0). If X<0,

           choose Y in the middle 50-th percentiles. If X>0 choose Y outside. Then

           clearly X+Y is not even symmetric

AxelBoldt

No. One can find two normals that are uncorrelated, i.e., their covariance is zero, the sum of which is not normal. Let X be a standard normal. Let Y be X or -X according as |X| is small or large; "large" means bigger than a specified number c. If c is big enough then cov(X,Y) < 0; if c is close to 0 then cov(X,Y) > 0; since cov(X,Y) depends continuously on c, there is an intermediate value of c for which cov(X,Y) = 0. X and Y are both standard normals.

Two random variables X, Y have a joint normal distribution if their joint distribution is such that for any two reals a and b the linear combination aX+bY is normal. A similar definition applies to more than two normals. The distribution of a vector of joint normals is determined by the means and the covariances (including the variances). The whole matrix of covariances is sometimes called the covariance matrix, but I prefer to call it the variance, since it's the natural higher-dimensional analog of the variance. It is always a non-negative-definite matrix. Michael Hardy 23:44 Jan 15, 2003 (UTC)

non-normal distributions

QUESTION: can some give me an example with nonnormal distribution and explain why it is not normal? (mia)

There are infinitely many such distributions. You might check out the binomial distribution. (It's not the best example, because with some parameters it can resemble a normal distribution. But it's relatively easy to understand.) Note that you should take care that "normal" here has two meanings:

pertaining to the "Normal distribution"
typical, standard, normal, etc..

The "Normal distribution" is normal is the only one that is normal in the first sense. Several different distributions (including the binomial distribution) can be considered normal in the second sense. I don't know if this is clear in the article. --Ryguasu 02:42 Jan 30, 2003 (UTC)

I advocate renaming the article from "Normal" to "Gaussian". We are perpetuating a 19th-century fallacy by sticking to the name "normal". Moreover, in generalizations the name "gaussian" is more common - nobody talks about normal stochastic processes, but gaussian stochastic processes. Also, people talk about non-gaussianity of their data when they do scientific work, possibly because "non-normality" sounds too similar to "abnormality" and should be de-emphasized. -- Miguel

I disagree, because the expression "Normal distribution" is, as far as I know, more common than "Gaussian distribution". Of course, "Gaussian" is a better name (as the article explains), but that is irrelevant. -- Jitse Niesen 22:01, 17 Aug 2003 (UTC)

I have a PhD in statistics, and I seldom see "Gaussian" and I see "normal" every day, and "non-normality" is the usual locution, whereas "non-gaussianity" is one I've never heard nor read. "Gaussian" is a misnomer since de Moivre would be the appropriate eponym. Michael Hardy 04:03, 18 Aug 2003 (UTC)

"Gaussian" has become the more common term in mathematical physics, which is where I come from. And I agree, that de Moivre would be the right name to use. By the way, I read the Doctrine of Chances for a term paper a long time ago, and it's great fun. Did you know that the law of large numbers proves the existence of God? -- Miguel

I never heard of the theological implications; how do those work? Physicists sometimes seem out of touch when they talk about probability. For example, it seems some of them define the "error function" as

<math>\mbox{erf}(x)=\int_{-\infty}^x e^{-u^2}\,du<math>

rather than as

<math>\int_{-\infty}^x e^{-u^2/2}\,du.<math>

Michael Hardy 14:22, 18 Aug 2003 (UTC)

Yes, it's rather hard to be a physicist and care about precision in probability theory, you quickly feel like an outcast... As for the Theological implications, let me quote the man himself:

What we have said is also applicable to a Ratio of Inequality, as appears from our 9th corollary. And thus in all cases it will be found that altho' Chance produces Irregularities, still the Odds will be infinitely great, that in the process of Time, those Irregularities will bear no proportion to the recurrency of that Order which naturally results from ORIGINAL DESIGN.

...Again, as it is thus demonstrable that there are, in the constitution of things, certain laws according to which Events happen, it is no less evident from Observation, that those Laws serve to wise, useful and beneficent purposes: to preserve the stedfast Order of the Universe, to propagate the several Species of beings, and furnish to the sentient Kind such degrees of happiness as are suited to their State.

But such Laws, as well as the original Design and Purpose of their Establishment, must all be from without; the Inertia of matter, and the nature of all created Beings, rendering it impossible that any thing should modify its own essence, or give to itself, or to anything else, an original determination or propensity. And hence, if we blind not ourselves with metaphysical dust, we shall be led, by a short and obvious way, to the acknowledgement of the great MAKER and GOVERNOR of all; Himself all-wise, all-powerful and good.

If I paraphrased it, I couldn't do it justice. -- Miguel

A Google search gives "gaussian distribution" 90,000 hits, and "normal distribution" 250,000. It seems to me that the top "gaussian" hits are more authoritative than the top "normal" hits, but that's hardly a compelling argument. -- Miguel

Thank you for your reply. I need an example for non normal distribution. It is for a project at college.We are on a chapter in Statistics about Gauss and how to get from discrete to continous variables and about the bell curve and normal distribution. Our teacher said that there are only 3 cases of not normal distribution and we shouldn't look in astronomy cause there everything is normal distributed. To help you understand better what I need: like the IQ scores is normally distributed on the bell curve. when we put the mean in the midlle 50% is on the left side and 50% on the right. and the bell curve is symmetrical. Well I need exactly the opposite. I need to write about something which is not normal distibuted and why it is not.from what i understood it will be a thing like temperature, height, IQ scores, etc. In the page about normal distribution it has 3 not normal cases but I don't understand why they are not normal and I don't know if they are correct. I hope I made more clear what I need and I hope you can help me on that. Thank you. (mia)

Oh, boy, your professor needs a clue. -- Miguel

Some context would be need before that "3 cases" comment can be understood. The waiting time until the arrival of the next phone call at a swithboard is usually modelled by a memoryless exponential distribution. That can serve as another example. An exponentially distributed random variable is always positive and is memoryless; normally distributed r.v.s do not have those properties. Michael Hardy 18:47 Jan 30, 2003 (UTC)

New comments not related to discussions above

I have made several edits, none major. Here I'll called out the ones that might be considered slightly controversial.

throughout: Gaussian/gaussian distribution -> normal distribution (use consistent terminology throughout article)

para 1: strike "Among people... (notably physics)" (lots of people in lots of fields use Gaussian or normal as the fancy strikes them -- there's no clean division and it's not necessary in this article to sort it all out)

under "Probability density function": strike (the "68 - 95.5 - 99.7 rule") (the quote marks suggest that this is a well known phrase, but I've never seen such a rule in a couple of decades of reading, and Google doesn't know about it either)

abbreviated "cumulative distribution function" to "cdf" throughout (I hope it's clearer this way)

reorganized discussion of cdf of standard and general cases

under "Generating Gaussian random variables": strike "beautiful" (the Box-Muller transform is not particularly striking; it certainly isn't any more so than any of the other results cited in this article; if anything here is beautiful it's the central limit theorem)

rephrased summary of central limit theorem

It's a good topic and I believe the technical part of the article is in good shape. I think the applications section needs more work, though. Have at it, Wile E. Heresiarch 18:41, 4 Jan 2004 (UTC)

Names of the distribution in different fields

Hello. At present the intro paragraph reads "Many physicists and engineers call it the Gaussian distribution, but normal distribution is the name by which it is known to probabilists and statisticians." If I'm not mistaken, both names are in general use in many fields. Can we really pin down who calls it what, and do we need to? -- How about if it reads "The normal distribution is also called the Gaussian distribution, especially in physics and engineering." Other sugggestions? Happy editing, Wile E. Heresiarch 22:02, 2 Apr 2004 (UTC)

Yes, I prefer your suggestion to the present text. --Zero 05:24, 3 Apr 2004 (UTC)

Well, I went ahead and changed the text as I suggested above. I really think that we needn't worry about who calls it what; maybe someone will be inspired to remove "physics and engineering" entirely. Wile E. Heresiarch 03:11, 8 Apr 2004 (UTC)

Everything is Gaussian?

What does 'everything is Gaussian' mean as used on this page? A Google search turns up... er... this page. I don't think it is at all clear from the article what it actually means. --Mysteronald 18:07, 19 Sep 2004 (UTC)

Well, what's intended is that the most common statistical tests work as advertised, to the extent that they work at all, if the distributions involved are exactly Gaussian, and approximately otherwise. (I'm thinking of the t-test and the F-test here.) Even though we really know that the distributions involved are never or maybe almost never Gaussian, people still use the same tests. I agree that statement about "everything is Gaussian" is confusing; maybe you or I can think of a better way to word it. Regards & happy editing, Wile E. Heresiarch 04:18, 21 Sep 2004 (UTC)

IQ discussion

I propose that the large IQ section be replaced by a link to somewhere else. It is barely relevant to this page. Comments? --Zero 11:59, 5 Feb 2005 (UTC)

OK, if you have an objection please state it now as I'm planning to come back in a couple of days to delete that whole section. --Zero 09:05, 22 Mar 2005 (UTC)

Just for the record: no objection if what you're proposing is to slice out the IQ section and merge it into a more appropriate article. Make sure you also move the references that apply to the IQ section and remove those that are no longer needed from this page. --MarkSweep 09:59, 22 Mar 2005 (UTC)

No objections if it's moved elsewhere, but it is somewhat relevant to the normal distribution. It's often used as an example of what is a normal distribution, and unlike the other examples, the entire test is formed/graded/whatever you want to call it to give a normal distribution. It'd be nice if you could leave a summary of the IQ section in this article though. --jag123 01:55, 25 Mar 2005 (UTC)

Peer review

I've listed this article for peer review. It's pretty good already and could perhaps soon become a featured article. If it improves significantly, someone should nominate it as a featured article candidate. --MarkSweep 05:56, 22 Mar 2005 (UTC)

In use

You jumped the gun way to fast there MarkSweep. :) Cburnett 08:57, 22 Mar 2005 (UTC)

I guess we both were editing it before the servers went dead, and then when they came back – boom. Anyway, I'm done with the merges now.

About my changes: There is some general disagreement about how to handle math inside regular text, using either <math> or low-level formatting and HTML entities. I prefer the former, since it leaves more choice to the user. Still, the consensus seems to be that mixing images and text should be avoided (readers can still ask for it by setting their preferences). My math rendering preferences are set to "HTML if possible or else PNG", and I tend to format things in such a way that <math> in regular paragraphs will render as HTML. For displayed formulas, I still prefer PNG for anything "large" (sums, integrals, fractions, etc., which look ugly and/or plain confusing in pure HTML).

Another thing: Is there a consensus about what the second parameter of the normal distribution is? We seem to be going back and forth between "variance" and "standard deviation". I've seen encountered three options: consistently use variance, consistently use sd, or inconsistently use both. Given a choice, I'd prefer being consistent. I've changed the text so that it says "variance" almost everywhere, except when talking about the "scale parameter", which is the standard deviation. --MarkSweep 09:39, 22 Mar 2005 (UTC)

I prefer PNG always so I change everything to math tags so that I get my PNG. :)

I've always seen variance as the second parameter and seems more natural because the multivariate has the covariance matrix (I've *never* heard of a standard deviation matrix). Cburnett 15:49, 22 Mar 2005 (UTC)

You can get PNG by setting your math rendering preference to "Always render PNG". I only set my preference to "HTML if possible" for purposes of editing.

About standard deviation as the second parameter, Gelman et al. describe σ as the scale parameter, but then define N in terms of σ² and (of course) use the variance in most cases. The normal distribution functions of R are parameterized in terms of the standard deviation. In the GNU Scientific Library, the bivariate Gaussian pdf is parameterized in terms of the two standard deviations and the correlation coefficient. But your point about the general multivariate case is very convincing, so I guess we should use "variance" throughout. --MarkSweep 20:37, 22 Mar 2005 (UTC)

Correct, except I can't get PNG rendered formulas if some are entered as raw HTML like numerous articles do. So I convert them all to tex and go from there.

I don't have it with me at the moment, but I'm fairly sure the Statistical Inference by Casella & Berger primarily use variance. But, still, the univariate can be derived from the multivariate and the multivariate uses a covariance matrix. I don't know that a square root matrix of a symmetric matrix always exists. Meaning that A is a square root matrix of B if A*A=B. I know it will always hold for a hermitian matrix but you can't generalize that to a generic, symmetric matrix. Anywho... Cburnett 00:15, 23 Mar 2005 (UTC)

Certainly such a square root exists when the matrix is symmetric and the entries are real; see spectral theorem. But I've always seen variance rather than SD used as the parameter. Michael Hardy 01:25, 23 Mar 2005 (UTC)

Uncorrelated implies independent?

From various discussions and searches (including one at Talk:Kalman filter) it appears to be widely believed and even published that if X and Y are both normally distributed and uncorrelated, then they are independent. I'm sure this is false and have created a simple example in which |X| = |Y| but they are uncorrelated because the sign is random. The misconception seems to be widespread enough that it would be worth adding something to this page clarifying the mistake. — ciphergoth 13:43, 2005 Apr 28 (UTC)

Well the answer from correlation is no. That article notes:

If the variables are independent then the correlation is 0, but the converse is not true because the correlation coefficient detects only linear dependencies between two variables. Here is an example: Suppose the random variable X is uniformly distributed on the interval from −1 to 1, and Y = X². Then Y is completely determined by X, so that X and Y are as far from being independent as two random variables can be, but their correlation is zero; they are uncorrelated. However, in the special case when X and Y are jointly normal, independence is equivalent to uncorrelatedness.

Which is similar to what you said, but even easier to explain I think. Though my mastery of the mechanics of this has lost me, so I can't recall how to show why X and Y have 0 correlation. - Taxman 14:42, Apr 28, 2005 (UTC)

Thanks for this. Unfortunately it almost perfectly fails to answer the question. It asserts that in general, uncorrelated =!=> independent, and then states a special case involving normally distributed variables in which uncorrelated => independent. The variables in the example are not normally distributed either. It thus doesn't do anything to answer the question posed, which is: if X and Y are normally distributed and uncorrelated, are they necessarily independent?

Jointly normal is in this case a very important difference. It would have been clearer had I simply removed the sentence about the special case, which was a bit off track for the question. The rest is a clear example to answer your original question. The only difference is the example happened to discuss uniform distributed RV's instead of normal. In any case, I think the answer to your original question is now clear to you, so this is moot. - Taxman 21:33, Apr 28, 2005 (UTC)

I think the answer is that just because both X and Y are normal does not imply that they are jointly normal, and that uncorrelated => independent only where they are jointly normal. In fact, I think I have constructed an example in which X and Y are both normal, and uncorrelated, but not jointly normal and not independent. It would be good if this were spelled out somewhere, since it would answer a common misconception. — ciphergoth 16:02, 2005 Apr 28 (UTC)

uncorrelated + normal ---/---> independent.
uncorrelated + jointly normal ------> independent.

This is widely known. Michael Hardy 17:36, 28 Apr 2005 (UTC)

The misconception arises from the subtle difference between the marginal distributions being normal and the joint distribution being a bivariate normal. See multivariate normal distribution for details, where it is explained that a joint normal distribution requires that every linear combination of X and Y be normal, not just X and Y. This is not the case when |X|=|Y|. — Miguel 21:02, 2005 Apr 28 (UTC)

As I thought. It seems to be quite a widespread misconception, judging by the references User:Orderud found It's good to have this clarified - thanks! — ciphergoth 21:15, 2005 Apr 28 (UTC)

Retrieved from "https://academickids.com:443/encyclopedia/index.php/Talk:Normal_distribution"