Did Ronald Fisher ever say anything on varying the threshold of significance level?

Question

There has been a growing chorus against the conventional NHST (Null Hypothesis Significance Testing). One thing is the blind usage of a monolithic significance level $5\%.$

In a recent thread at CV, user Peter Flom claimed that Fisher had asserted somewhere that he advocated changing the threshold.

While Peter couldn't remember the exact quote and the source, it was supposed to be like:

... no sane researcher uses the same significance level in all cases.

My question is, did Ronald Fisher ever express anything on this issue? Would appreciate if someone is able to pin down the source where he made such statement(s).

Answer 1

Jerry Dallal collected some of Fisher's quotes from various works. And Fisher said too much, first setting the P=0.05 cutoff in 1925, then not following it himself, and, finally, openly advocating flexibility in 1956 in response to Neyman and Pearson taking it too far. However, even then he advocated not setting any fixed cutoff at all, not changing it, this is what the passage alluded to in the OP is about. On more charitable reading, Fisher always meant 0.05 only as a vague threshold, not a cutoff. For more on Fisher's vs Neyman-Pearson's approaches see Hypothesis testing: Fisher vs. Popper vs. Bayes.

The P=0.05 level comes from Fisher's Statistical Methods for Research Workers (1925), albeit with some wiggle room:

"The value for which P=0.05, or 1 in 20, is 1.96 or nearly 2; it is convenient to take this point as a limit in judging whether a deviation ought to be considered significant or not. [...] If P is between .1 and .9 there is certainly no reason to suspect the hypothesis tested. If it is below .02 it is strongly indicated that the hypothesis fails to account for the whole of the facts. Belief in the hypothesis as an accurate representation of the population sampled is confronted by the logical disjunction: Either the hypothesis is untrue, or the value of $\chi^2$ has attained by chance an exceptionally high value. The actual value of P obtainable from the table by interpolation indicates the strength of the evidence against the hypothesis. A value of $\chi^2$ exceeding the 5 per cent. point is seldom to be disregarded."

Despite these declarations, he was more flexible in practice in the same book, or, less charitably, inconsistent:

"The results of t shows that P is between .02 and .05. The result must be judged significant, though barely so [...] we find... t=1.844 [with 13 df, P = 0.088]. The difference between the regression coefficients, though relatively large, cannot be regarded as significant."

Says Dallal:"Part of the reason for the apparent inconsistency is the way Fisher viewed P values. When Neyman and Pearson proposed using P values as absolute cutoffs in their style of fixed-level testing, Fisher disagreed strenuously. Fisher viewed P values more as measures of the evidence against a hypotheses..." It is only then, in Statistical Methods and Scientific Inference (1956), that Fisher elevated his practiced flexibility and attention to context to a principle:

"The attempts that have been made to explain the cogency of tests of significance in scientific research, by reference to hypothetical frequencies of possible statements, based on them, being right or wrong, thus seem to miss the essential nature of such tests. A man who "rejects" a hypothesis provisionally, as a matter of habitual practice, when the significance is at the 1% level or higher, will certainly be mistaken in not more than 1% of such decisions. For when the hypothesis is correct he will be mistaken in just 1% of these cases, and when it is incorrect he will never be mistaken in rejection. This inequality statement can therefore be made. However, the calculation is absurdly academic, for in fact no scientific worker has a fixed level of significance at which from year to year, and in all circumstances, he rejects hypotheses; he rather gives his mind to each particular case in the light of his evidence and his ideas. Further, the calculation is based solely on a hypothesis, which, in the light of the evidence, is often not believed to be true at all, so that the actual probability of erroneous decision, supposing such a phrase to have any meaning, may be much less than the frequency specifying the level of significance."

Answer 2

Seems to me that it is difficult to understand Fisher's writings by examining short quoted sections. @Conifold's answer is excellent, but I would point out a couple of things for consideration.

First, it is all too easy to find apparent or real discrepancies between things that he wrote, particularly given that his writings extend from before to after the invention of statistical methods. For example, while working on his 1925 book he was not in a position to know that it would be important to point out that 0.05 should not be used as a fixed cutoff until after the publication by Neyman & Pearson in 1933!

Fisher was not a supporter of the one-and-done approach that follows from Neytman–Pearsonian hypothesis testing, and his lack of support was expressed more clearly and forcefully in his work that was later than the Neyman & Pearson papers that first proposed their framework. I would say that it is not a "charitable reading" that Fisher meant 0.05 as "a vague threshold, not a cutoff", but an exact reading. His philosophy was never that a singular statistical result led to an inference about the real world.

Some of the discrepancies that are apparent but lacking in reality stem from the fact that Fisher mostly worked from pre-tabulated tables of critical points from the distributions of test statistics. He did not have access to a computer, or even to an electronic calculator for most of his career. When he says that 0.05 is "convenient" part of that convenience is that it is one of the columns in his table!

Second, it is difficult for anyone to write clearly about general usage of statistical methods in forming scientific inferences without occasionally making it sound like a statistical result drives the inference. It just takes too many words that the resulting paragraphs turn into stodge. (Try it yourself if you have any doubt about that.) That difficulty is doubled for most of us who have been educated on a mixture of Fisherian and Neyman–Pearsonian philosophies because it is a difficulty of writing clearly and a difficulty of reading clearly! It is all too easy to incorrectly assume that there is some direct path between a Fisherian p-value of less than, say, 0.05 and inference.