tag:blogger.com,1999:blog-8205333.post5130623042356278138..comments2023-10-16T11:28:03.544+00:00Comments on Anomaly UK: Devaluation of SignificanceAnomaly UKhttp://www.blogger.com/profile/04780148789321563441noreply@blogger.comBlogger2125tag:blogger.com,1999:blog-8205333.post-5951602618928580292016-09-27T16:31:09.281+00:002016-09-27T16:31:09.281+00:00..And this is the response quoting my comment:
No.....And this is the response quoting my comment:<br /><br />November 25, 2009<br />The Ghastly Specter of Monty Hall!<br />Anomaly UK responds to my Random Observations . He was heading not entirely down my path, but he makes a good point that’s worth repeating. Andrew references the Monty Hall problem .<br /><br />Your paper will include a calculation of significance. This is essentially an estimate of the probability that a correlation as strong as the one you found would exist purely as a result of randomness in the data, even if your theory is false.<br />This calculation assumes the "proper" sequence of events. You have a theory, and you test the data for a correlation. Since you in fact poked around for correlations, then came up with a theory, the significance calculation is not valid. The true significance depends on the probability that, having found a randomly-caused correlation somewhere, you can then invent a theory to explain it. That probability is very difficult to estimate, but is probably much greater - meaning that the significance of the correlation is much smaller.<br /><br />As you say, theories aren't formed in a vacuum, and so there is not such a clear division between the "right" way of doing it and the "wrong" way of doing it. Nobody is completely ignorant of the data when they start to theorize. That is a real problem with nearly all statistics-based results that are published today. They are all presented with significance calculations based on the assumption that the forming of the theory was independent of the data - an assumption that is very unlikely to be completely true. Therefore nearly every significance estimate published is an overestimate.<br /><br />I believe the issue Anomaly mentions to be very nearly universal. In order to publish, I must read and reference the related literature. That action, alone, nearly guarantees that I am NOT ignorant of the data.<br /><br />In light of this fact, I am occasionally dismayed when reviewers demand a particular type of “purity” in the scientific process, which, I suppose, was the point of my original post.<br /><br />I also find it blackly amusing when we bring such arguments to bear against research that contradicts our preferences, but tend not to use such arguments against research the conforms to out preferences. Yes, I’m guilty, too.<br />Posted by Noel Campbell at 10:38 AMAnomaly UKhttps://www.blogger.com/profile/04780148789321563441noreply@blogger.comtag:blogger.com,1999:blog-8205333.post-22199756568928053632016-09-27T16:28:37.398+00:002016-09-27T16:28:37.398+00:00Division of Labour no longer exists: This is the o...Division of Labour no longer exists: This is the original post referred to:<br /><br />November 23, 2009<br />Random Observations<br />1. Suppose I run a regression out of curiosity, just to see what I find. 2. The results puzzle me mightily. 3. I get an idea, and tinker some more, pretty much confirming the idea. 4. Now I write it up. Here's where it gets interesting.<br /><br />A. I write up my explanation as if I was never confused and as if I expected to find what I found. Reviewers consider this to be good theorizing and the paper is accepted.<br /><br />B. I write up my explanation totally honestly, and in chronological order, as above. Reviewers dismiss my work as ad hoc theorizing and the paper is rejected.<br /><br />Agreed, if I behave as outlined in the opening comments, I would not conform to the ideal scientific method. Then again, scientific theorizing does not occur in a vacuum. I must know there's a research question out there before I try to answer it. If I consider the scientific method to begin when I formulate my research question, then I start at #2, and my behavior is ethical and consistent with the ideal scientific method.<br /><br />Does the problem lie with me or the review process?<br /><br />Posted by Noel Campbell at 10:58 AMAnomaly UKhttps://www.blogger.com/profile/04780148789321563441noreply@blogger.com