In science, as in life, we all like to celebrate the big news.
We confirmed the existence of black holes by the ripples they create in space time. We photographed the shadow of a black hole. We figured out how to edit DNA. We found the Higgs boson!
What we don't usually hear about is the years of back-breaking, painstaking hard work that delivers inconclusive results, appearing to provide no evidence for the questions scientists ask – the incremental application of constraints that bring us ever closer to finding answers and making discoveries.
Yet without non-detections – what we call the null result – the progress of science would often be slowed and stymied. Null results drive us forward. They keep us from repeating the same errors, and shape the direction of future studies.
There is, in fact, much that we can learn from nothing.
Often, however, null results don't make it to scientific publications. This not only can generate significant inefficiencies in the way science is done, it's an indicator of potentially bigger problems in the current scientific publication processes.
"We know that there is this powerfully distorting effect of failing to publish null results," psychologist Marcus Munafò of the University of Bristol told ScienceAlert.
"Solving the problem isn't straightforward, because it's very easy to generate a null result by running a bad experiment. If we were to flood the literature with even more null results by generating poor-quality studies, that wouldn't necessarily help the fundamental problem, which is, ultimately, to get the right answers to important questions."
Defining the problem
The null hypothesis defines the parameters under which the results of a study would be indistinguishable from background noise. Gravitational wave interferometry is a nice, neat example: The signals produced by gravitational waves are very faint, and there are many sources of noise that can affect LIGO's sensors. A confirmed detection could only be made once those sources were conclusively ruled out.
If those sources cannot be ruled out, that is what's called a null result. That doesn't mean gravitational waves were not detected; it just means we can't determine that we've made a detection with any certainty.
An aerial photo of the Virgo interferometer, which detects gravitational waves. (The Virgo collaboration/CCO 1.0)
This can be really useful, and in some fields – like cosmology, and gravitational wave astronomy – the publication of null results helps scientists to adjust the parameters of future experiments.
In other fields, where results can be more qualitative than quantitative, null results are less valued.
"Part of the problem in a lot of behavioral and medical science is that we can't make quantitative predictions," Munafò explained.
"So, we're just looking for evidence that there is an effect or an association, irrespective of its magnitude, which then creates this problem when, if we fail to find evidence that there is an effect, we haven't put any parameters on whether or not an effect that small would actually matter – biologically, theoretically, clinically. We can't do anything with it."
An extraordinary nothing
When wielded correctly, a null result can yield some extraordinary findings.
One of the most famous examples is the Michelson-Morley experiment, conducted by physicists Albert A. Michelson and Edward W. Morley in 1887. The pair were attempting to detect the velocity of our planet with respect to 'luminiferous aether' – the medium through which light was thought to travel, much as waves travel through water.
As Earth moved through space, they hypothesized, oncoming waves of light rippling through a perfectly still, Universe-wide ocean of aether should move at a slightly different speed to those rippling out at right angles to it. Their experiments were ingenious and painstaking, but of course, they detected nothing of the sort. The null result showed that the speed of light was constant in all reference frames, which Einstein would go on to explain with his special theory of relativity.
Michelson's 1881 inteferometer, designed to detect aether. (Albert Abraham Michelson, public domain)
In other instances, null results can help us to design instrumentation and future experiments. The detection of colliding black holes via gravitational waves only took place after years of null detections allowed for improvements to the design of the gravitational wave interferometer. While at CERN, physicists have so far made no detection of a dark matter signal in particle collision experiments, which has allowed constraints to be placed on what it could be.
"Null experiments are just a part of the full range of observations," astrophysicist George Smoot III of UC Berkeley told ScienceAlert. "Sometimes you see something new and amazing and sometimes you see there is not."
When it's all about hard numbers, null results are often easier to interpret. In other fields, there can be little incentive to publish.
The implications of non-detection aren't always clear, and studies that do make a significant finding receive more attention, more funding, and are more likely to be cited. Clinical trials with positive results are more likely to be published than those with negative or null results. When it comes to deciding who is going to get a research grant, these things matter.
Scientists, too, are very busy people, with many potential lines of inquiry they could be pursuing. Why chase the null hypothesis when you could be using your time doing research that is more likely to be seen and lead to further research opportunities?
To publish or null
As well as leaving out important context that could help us learn something new about our world, the non-publication of null results can also lead to inefficiency – and, worse, could even discourage young scientists from pursuing a career, as Munafò found first-hand. As a young PhD student, he set about replicating an experiment that had found a certain effect, and thought his results would naturally be the same.
"And it didn't work. I didn't find that effect in my experiment. So as an early career researcher, you think, well, I must have done something wrong, maybe I'm not cut out for this," he said.
"I was fortunate enough to bump into a senior academic who said, 'Oh, yeah, no one could replicate that finding'. If you've been in the field long enough, you find out about this stuff through conversations at conferences, and your own experiences, and so on. But you have to stay in the field long enough to find that out. If you aren't lucky enough to have that person tell you that it's not your fault, it's just the fact that the finding itself is pretty flaky, you might end up leaving the field."
Academic publishing has been grappling with this problem, too. In 2002, a unique project – the Journal of Negative Results in BioMedicine – was established to encourage the publication of results that might not otherwise see the light of day. It closed in 2017, claiming to have succeeded in its mission, as many other journals had followed its lead in publishing more articles with negative or null results.
However, encouraging scientists to bring their negative results to light may sometimes prove close to fruitless. On the one hand, there's the potential for a glut of poorly conceived, poorly designed, poorly conducted studies. But the opposite is possible, too.
In 2014, the Journal of Business and Psychology published a null results special issue, and received surprisingly few submissions. This, the editors deduced, could be because scientists themselves are conditioned to believe that null results are worthless. In 2019, the Berlin Institute of Health announced a reward for replication studies, explicitly welcoming null results, yet only received 22 applications.
These attitudes could change. We've seen that it can happen; Smoot, for instance, has gleaned a great deal of insight from null detections.
The Crab Nebula, a known source of cosmic rays. (NASA, ESA, J. Hester and A. Loll/Arizona State University)
"Search for antimatter in the cosmic rays – that was a null experiment and convinced me that there was no serious amount of antimatter in our galaxy and likely on a much larger scale, even though there was a great symmetry between matter and antimatter," he said.
"Next null experiment was testing for violation of angular momentum and rotation of the Universe. While it is conceivable, the null result is very important to our worldview and cosmology view and it was the initial motivation for me to use the cosmic microwave background radiation to observe and measure the Universe. That led to more null results, but also some major discoveries."
Ultimately, it might be a slow process. Publication needs to incentivize not null results in and of themselves, but studies designed in such a way that these results can be interpreted and published in their appropriate context. By no means a trivial ask, but one crucial to scientific progress.
"Getting the right answer to the right question matters," Munafò said.
"And sometimes that will mean null results. But I think we need to be careful not to make the publication of a null result an end in itself; it's a means to an end, if it helps us get to the right answer, but it needs more than just the publication of null results to get there.
"Ultimately, what we need are better formulated questions and better designed studies so that our results are solid and informative, irrespective of what they are."