Doubtful Significance: Can an Amorphous Cloud of Points Really Illustrate a Significant Relationship?

49 Pages Posted: 10 Aug 2012 Last revised: 13 Aug 2012

See all articles by G. M. Swann

G. M. Swann

Nottingham University Business School

Date Written: August 9, 2012

Abstract

Can an amorphous cloud of points really illustrate a statistically significant relationship between two variables? We examine one particular scatter-plot from 402 observations which appears to show no relationship whatever between the Y and X variables, and yet, the conventional OLS regression finds a t-statistic of 3.0. To explain this puzzle, we derive a formula showing that the conventional t-statistic is the product of a signal-to-noise ratio and the square root of the regression degrees of freedom. This signal-to-noise ratio is a convenient measure of the visual clarity of any relationship in the scatter-plot. If the ratio is 1, then the relationship is reasonably clear to the eye. If the ratio is 10, then it is very clear indeed. But if the ratio is 0.1, or less, then the relationship is more or less invisible to the eye. However, even if the signal-to-noise ratio is very low, our formula shows that we can still obtain a large t-statistic, so long as we have enough observations. But how it can make sense to say that an amorphous cloud of points is ‘significant’? The standard independence assumption of regression analysis is absolutely critical here, and robustness checks show that without it, the relationship between y and x could be almost anything. Finally, we examine a sample of 2220 parameter estimates taken from recent articles in 20 leading journals, and find that for most of these – even those that are reported as statistically significant – the implicit signal to noise ratio is very low indeed (median value 0.033). The implication is that in the absence of the independence assumption, these parameter estimates tell us very little indeed. We conclude with a brief reexamination of this critical independence assumption and whether it is really justified.

Keywords: signal-to-noise ratio, t-statistics, statistical significance

JEL Classification: C12, C13, B41

Suggested Citation

Swann, G. M., Doubtful Significance: Can an Amorphous Cloud of Points Really Illustrate a Significant Relationship? (August 9, 2012). Nottingham University Business School Research Paper No. 2012-08, Available at SSRN: https://ssrn.com/abstract=2127179 or http://dx.doi.org/10.2139/ssrn.2127179

G. M. Swann (Contact Author)

Nottingham University Business School ( email )

Jubilee Campus
Wollaton Road
Nottingham, NG8 1BB
United Kingdom

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
316
Abstract Views
2,980
Rank
175,064
PlumX Metrics