# Has correlation led to major discoveries?

We like to repeat that correlation does not imply casuation. My gut feeling is however that (source: xkcd):

Correlation doesn't imply causation, but it does waggle its eyebrows suggestively and gesture furtively while mouthing 'look over there'.

Were important discoveries rather the effect of

• mathematical explorations ("I will take this and that equation, combine them together and test if this is true in the real world")
• or physical experiments ("The more I pull on this spring, the more resistance there is. Oh look, the relationship is linear - I will look at physics to understand why") - in that case correlation was the lead mechanism.

This is obviously oversimplified, my question is about the predominance of each of these techniques in history. I guess that the second case was much more common initially, then with a more elaborate mathematical apparatus purely theoretical explorations were made possible (or easier).

• This is a very, very broad question. Most of science (even the physical sciences) are based on correlation rather than causation. Commented Jun 27, 2016 at 15:31

I agree with David Hammen's comment that most science is based on correlation rather than causation. Also agree that it's awfully broad. So broad I'm not sure of any definitive way to answer it. But there are many fun examples that illustrate the point. So here are a few. (Incidentially, I'm aware of the irony that I'm trying to answer the question with examples, which itself is correlation data.)

Edward Tufte, in his 1997 Visual Explanations, describes how John Snow used correlation between cholera incidence and location to deduce that the London cholera epidemic of 1854 was explained by drinking water from a single contaminated well. From this correlation data, he then designed an experiment to test his theory: He had the handle of the well pump removed, and cholera incidence dropped. Tufte's books are not available online, but this story is well-known. A google search on John Snow and cholera will give you dozens of different versions to read. For example: http://www.ph.ucla.edu/epi/snow/snowcricketarticle.html.

Indeed, the entire field of epidemiology is fundamentally concerned with looking for statistical patterns. (See https://en.wikipedia.org/wiki/Epidemiology#Causal_inference.) In medicine, clinical trials often compare different treatment options. These clinical trials may eventually lead to research that uncovers details of drugs mechanism of action, and to how the body often fails to respond to them. But most of modern medicine is based on correlation. The statistuical tests answer the question, "If the data is truly random, what is the probability that we would see the patterns that we do?" If the probability is low enough, the studies assume the results are not random. But that's about all they truly say. Much of the Evidence Based Medicine curriculum has to do with understanding the statistical tests that are used to evaluate the results of different studies. (See https://en.wikipedia.org/wiki/Evidence-based_medicine.)

The recent discovery of gravity waves is an example of your first bullet. While I'm far from an expert, my understanding is the theoretical physicists (Einstein in particular) built a mathematical model of what gravity waves would look like, if they did actually exists. That gave experimental physicists some idea of what to look for. See http://www.space.com/31922-gravitational-waves-detection-what-it-means.html. The discovery of the Higgs Boson followed the same pattern. Researchers knew they had found it, only because the signals their equipment gave them matched those predicated by theoretical mathematical models. See https://en.wikipedia.org/wiki/Higgs_boson.

I'm going to make a broad generalization that I'll leave it to others to judge: When there is a large body of well-tested theory, and mechanisms of action are fairly well known (such as in physics), then research is driven by mathematical theories (because they are relatively cheap to produce and to weed out the inconsistent ones) and confirmed by experiment (because they are relatively expensive to perform). When mechanisms of action are not that well understood, and even the best theories can't explain the many, real-world exceptions (such as biology & medicine), then research tends to run physical experiments (because the theories are too weak for good predictive power), and them mathematics is used to find patterns in the data (that can help to refine the theory).

The field of chemistry is a nice example of the way these two approaches complement each other. In the days of alchemy, there was no coherent theory of chemistry. So research progressed mostly through experimentation. But that experimentation eventually led to the atomic theory of matter, and statistical mechanics, and for a while research was guided at proving the predictions made by those theories. But atoms can combine in very complicated ways that were mathematically too complicated to compute. So even though the theories seemed correct in simple situations, for real-world applications, research was mostly guided by experimentation. But now that computing power is relatively cheap and plentiful, we do see chemical research being guided by theory again: for example, "designer drugs" and computer modeling of macro-molecules. Now, instead of weeding out ineffective drugs through experiment, many can be eliminated based on computer modeling alone. See http://wyss.harvard.edu/viewpressrelease/179/computer-model-sets-new-precedent-in-drug-discovery.

The book The Structure of Scientific Revolutions (Thomas Kuhn, 1962) introduced the term "paradigm shift." It was a struggle for me to read (and I should probably read it again), but it describes this progression from immature sciences (that are guided by correlation because they have no coherent theory), to more mature sciences that are guided by theory and mathematical modeling. Then, as experimental techniques get more sophisticated and accurate, shortcomings in the accepted theory become discovered. This leads to a period of research led by experimentation again, and researchers look for patterns in the exceptions that are found to the original theory. Eventually, a "paradigm shift" happens, the old theory is replaced by a new one that can account for the exceptions, and the field once again moves to a period of research guided by mathematics and theory.