How to recognise a bad study?

How to recognise a bad study?

Every year, thousands of scientific studies are published.

Some provide valuable information. Others have significant limitations. A few even lead to misleading conclusions when misinterpreted.

For the general public, however, it is difficult to tell the difference. After all, a study published in a scientific journal often seems credible at first glance.

This is precisely what explains why so many appealing claims circulate on social media. An influencer cites a study. A media outlet repeats the result. That conclusion is then presented as scientific truth.

Yet not all studies have the same level of reliability.

A low-quality study does not become solid simply because it was published. Conversely, a well-designed study can sometimes be poorly summarised or taken out of context.

The real question, then, is not whether a study is “good” or “bad”, but which elements allow us to assess how much confidence it deserves.

A low-quality study does not become solid simply because it was published.

Why are not all studies equal?

The word study actually covers very different methods.

Some research simply follows populations in order to observe associations. Others directly test an intervention under controlled conditions. Some include several thousand participants. Others rely on only a few dozen. Some are replicated by several independent teams. Others remain isolated. All these differences directly influence the strength of the conclusions.

This is why researchers never judge a study solely on its result. They first look at how that result was obtained.

Is the type of study suited to the question?

The first question is to identify the type of study.

An observational study mainly reveals associations. It generally does not demonstrate a cause-and-effect relationship — a distinction detailed in “Correlation or causation: why the difference matters”.

By contrast, a randomised controlled trial often allows the effect of an intervention to be studied more directly by limiting certain biases. This does not mean all observational studies are bad. They simply answer different questions.

Understanding the type of study is therefore the first step before interpreting its conclusions.

What the scientific literature actually shows

Researchers assess a study's quality according to several criteria: the methodology used, the sample size, the presence of a control group where relevant, randomisation, statistical methods, transparency of analyses, and the ability to reproduce the results.

The more rigorous these elements are, the more reliable the conclusions generally become. Conversely, a study with several methodological weaknesses should be interpreted with greater caution. It is also worth remembering that a very well-conducted study can produce a different result from another equally solid study.

This is precisely why researchers favour systematic reviews and meta-analyses, which allow the full body of available evidence to be assessed rather than a single publication — a point developed in “Why do scientific studies contradict each other?”.

Does sample size matter?

Yes.

A study of twenty people generally offers less certainty than a study including several hundred or several thousand participants. Small samples show more variability. They increase the risk that observed results are due to chance.

This does not mean a small study is useless. It can provide interesting information or open up new lines of research. However, its conclusions often need to be confirmed by larger studies before being considered solid.

The most common mistakes

One of the most common mistakes is focusing solely on the result presented in the study's abstract. A striking result naturally attracts attention. Yet it is often the details of the methodology that determine how much confidence it deserves.

Another mistake is ignoring the limitations acknowledged by the authors. In most scientific publications, an entire section is devoted to the study's weaknesses. Researchers explain there the elements that may limit the interpretation of the results.

A study that openly acknowledges its limitations is not necessarily less credible. On the contrary, this transparency is generally a sign of scientific rigour.

Finally, many people consider a published study to be automatically correct. Yet publication is an important step, but it does not guarantee that the results will be confirmed by future research — which is also why “science changes its mind”.

Do conflicts of interest invalidate a study?

Not necessarily.

Some research is funded by companies, universities, public bodies or associations. Funding does not automatically negate a study's quality. However, it is important that these conflicts of interest are clearly disclosed. Researchers and readers alike can then interpret the results with greater transparency.

Industry-funded studies are therefore not systematically bad. They simply need to be assessed using the same methodological criteria as any other. The quality of the method remains the main criterion for evaluation.

How do researchers assess the strength of a study?

Before placing great confidence in a publication, several questions are generally asked.

Is the type of study suited to the question being asked?

Is the sample large enough?

Were participants allocated appropriately where necessary?

Are the statistical methods appropriate?

Do the authors clearly describe their limitations?

Are the results consistent with other available research?

The more favourably a study answers these criteria, the more credible its conclusions become. No scientific work is perfect. The goal is therefore to assess the level of confidence one can reasonably place in it.

Practical recommendations

To interpret a study with more critical thinking, a few simple habits can help.

Do not rely solely on the title or the abstract.

Identify the type of study before interpreting its conclusions.

Look at the sample size.

Check whether the authors acknowledge the limitations of their work.

Always compare a study with other available research on the same topic.

Place more trust in systematic reviews and meta-analyses than in a single study.

Conclusion

Recognising a quality study does not mean looking only at its result.

It also requires examining the methodology used, the sample size, the limitations acknowledged by the authors and the consistency with other available research.

Researchers never base their conclusions on a single publication.

They gradually assess the full body of evidence in order to build an increasingly solid level of confidence.

Ultimately, reading a study critically does not mean trying to discredit it.

It simply means understanding what it actually allows us to conclude… and what it does not yet allow us to claim.