Should you believe the findings of scientific studies? Amid current concerns about the public’s trust in science, old arguments are resurfacing that can sow confusion.As a statistician involved in research for many years, I know the care that goes into designing a good study capable of coming up with meaningful results. Understanding what the results of a particular study are and are not saying can help you sift through what you see in the news or on social media.Let me walk you through the scientific process, from investigation to publication. The research results you hear about crucially depend on the way scientists formulate the questions they’re investigating.The scientific method and the null hypothesisResearchers in all kinds of fields use the scientific method to investigate the questions they’re interested in.AdvertisementAdvertisementAdvertisementAdvertisementFirst, a scientist formulates a new claim – what’s called a hypothesis. For example, is having some genetic mutations in BRCA genes related to a higher risk of breast cancer? Then they gather data relevant to the hypothesis and decide, based on the data, whether that initial claim was correct or not.It’s intuitive to think that this decision is cleanly dichotomous – that the researcher decides the hypothesis is either true or false. But of course, just because you decide something doesn’t mean you’re right.If the claim is really false but the researcher decides, based on the evidence, it’s true – a false positive – they commit what’s called a Type 1 error. If the claim is really true but the researcher fails to see that – a false-negative conclusion – then they commit a Type 2 error.Moreover, in the real world, it gets a little messier. It’s really hard to decide about the truth or falsity of a claim just based on what’s observed.AdvertisementAdvertisementAdvertisementAdvertisementFor that reason, most scientists employ what is called the null hypothesis significance testing framework. Here’s how it works: A researcher first states a “null hypothesis,” something that’s contrary to what they want to prove. For instance, in our example the null hypothesis is that BRCA genetic mutations are not associated with increased breast cancer occurrence.The scientist still gathers data and makes a decision, but the decision is not about whether the null is true. Instead, a researcher decides whether there’s enough evidence to reject the null hypothesis or not.What rejecting the null does and doesn’t meanUnderstanding this distinction is crucial. Rejecting the null is equivalent in practice to acting as though it is false – in the example, rejecting the null means claiming that those with some BRCA gene mutations do have a higher risk of breast cancer. Along with other evidence, such as the size of the increased risk, this outcome can justify recommending early breast cancer screening for people with the identified BRCA mutations.But failing to reject the null hypothesis doesn’t imply that it’s true – in this case, it doesn’t mean there is no association between the BRCA mutations and breast cancer. Rather, such a result is inconclusive; there’s not enough evidence …
