The Scientific Method: Or Why I Loath Metastudies

Every so often I am glancing through news headlines, or do not get the mute button pressed fast enough after the evening weather report, and read or hear a breathless headline about some massive medical or scientific discovery or research result. And 99% of the time, if I read or listen farther, it is the result of a single study with limited participants, or of a gigantic computer-correlated meta-analysis of hundreds of research papers that shows a two percent change in X. And never lists the starting percentage of cases of X, or the original numbers, or what the search parameters were. Or it is a computer modeled study. I have not hurled the remote through the screen, or punched my laptop, yet, but it’s tempting. Bad science seems to encourage my violent streak.

I grew up surrounded by science: chemistry, physics, biology, geology, and a little civil engineering on the side. So I learned about Occam’s Razor, and the scientific method, and the importance of the null hypothesis. Both my masters and PhD work were based on using multiple sources to prove or disprove a hypothesis.

(The Null Hypothesis states that X will cause no change in Y. So you design your experiment to show that X causes no change. If it does cause a change, and a repeatable change, you have disproven the Null hypothesis. This is very important in medicine, because you should be trying to prove that X procedure or drug will not affect a certain disease or condition.)

I was taught that you start with a theory or an observation. You try to disprove the theory, eliminating possible causes by testing them. Those tests must produce reproducible results. You should also try to find every possible probable cause that might explain your observation, prior to devising an odd, unusual, or novel explanation for what happened. Then you publish the results and invite other people to test them. If they disprove your hypothesis, so goes it. Learn and go on. If their tests support your work, congrats. That’s science.

Alas, once large sums of money become available, and politics meets research, the process seems to twist out of recognition. I’ll say up front that I have a little more faith in corporate-funded research, because I know that there is a bias from the beginning. It is a known influence and I can weigh what I read with that in mind. Sort of like reading economic policy suggestions from the Communist Party USA. But government-funded research has become a major bugaboo, especially in climatology and medicine. Fads and causes seem to be driving far too many of the experiments and interpretations. “Draw your curve then plot your points” used to be a joke: now it seems to have become a mission statement. Anthropogenic Climate Change is one (in)famous example, but the 40 year push for a low-fat, high carb diet is another example, as is the focus on prion tangles in Alzheimer’s research. At least in the latter two cases, entire economic systems were not devised and advocated based on computer models and corrupted data sets, although the long-term harm of the low-fat diet to public health may be considerable.

The modern political-scientific method appears to be as follows. 1) Find out what the government is giving grants for. 2) Read the journals to see what the preferred results appear to be. 3) Create model or compose algorithm. 4) Run study. 5) Check results. If results do not line up with grant-producing hypothesis, adjust model or algorithm. 6) Repeat 4 and 5 until desired results produced. 7) Publish and apply for next grant.

I’m aware that politics has always shaped scientific research, going back to the days of phlogiston theory and Lord Kelvin’s declaration about having reached the sum possible knowledge. There’s a reason alchemists searched for gold as well as enlightenment. But in certain fields the ties have become too close, and “science” is serving as a justification for policies that, if carried through to their logical ends, will cause far more harm to a lot of people than they could possibly cure. As Bjorn Lomborg and others have pointed out, using science as a cover for political change is a very bad thing. If someone wants world socialism and wealth redistribution, it should not be disguised as Saving the Planet.

If science writers and honest researchers bemoaned the public’s loss of faith in science and the research process before 1990, just wait until political science comes crashing down around our ears. All research will be tarred with the same brush.

I support the separation of science and state (money). We’ll never get away from advocacy research, because let’s face it, people need research $$ and very few donors give to support pure curiosity. But I want policy-driven advocacy science eliminated, or very, very clearly labeled as such. And I want all data made available to all researchers, or options presented for those who use proprietary data sets (corporate information, or patented materials, for example) so that real tests can be done. And I want the data to stop being massaged, adjusted, tweaked, “corrected,” and otherwise manipulated so that the past stops changing.

Bring back the null hypothesis!

One thought on “The Scientific Method: Or Why I Loath Metastudies

Comments are closed.