Doing research is hard. Understanding research—and the choices that researchers make in their design, reporting of results, and interpretations make—is really hard. Fortunately, some simple strategies and heuristics can help provide meaning and context to a study’s takeaways.
Like many things in life, doing research is a series of decisions with tradeoffs. Sometimes researchers trade convenience and cost for accuracy. Sometimes they use a larger number of subjects not well characterized or controlled, while other times, they use fewer subjects with rigorous controls. Sometimes the subject population includes a wide range of ages, demographics, and pre-existing conditions; other times, the population is narrow.
If there is an overall theme to decoding nutrition research, it’s understanding how to think about those trade-offs and how they impact the results. Here are 10 questions or heuristics I think about when reading nutritional science.
1. What is the big question addressed in this study?
The rationale for a given study is usually in the Introduction section of the paper (typically the first section, below the Abstract, which is a summary of the entire paper). The Introduction should provide context, leading the reader from a broad perspective of the field to the specific question this paper wants to answer. Sometimes the Introduction will end with either the hypothesis or the explicit purpose of the study and may even include a summary of the results. Getting a clear understanding of the big question the study is trying to answer can help you get your bearings before diving into the more specific aspects of the paper.
2. What is the design of the study, and why is it designed that way?
While randomized controlled trials (RCTs) with a double-blind, placebo-controlled, crossover approach might be the “gold standard,” there are plenty of legitimate reasons not to do these studies or to be skeptical of such an approach. So when looking at the design of the study, keep an open mind and try to understand why the researchers used that design.
For instance, a published paper on nutrition may be one part of a larger randomized clinical trial and analyzed separately to the study’s primary outcomes. An observational design approach may have nutritional interventions as the primary outcome, leading to more robust conclusions than an RCT. Other times, a cross-sectional study design takes a snapshot of what people report they eat and some clinical or disease markers. Each approach has its strengths and weaknesses. Understanding the study design can help you evaluate the applicability, level of evidence and provide a framework for how well the study was done and what limitations might be essential to pay extra attention to.
3. How was the nutrition data collected?
There are many different ways to collect nutrition data, and some are certainly better than others. For example, a survey that asks you to recall the number of times you ate a specific food in the last few weeks is not as accurate as a laboratory providing every meal throughout the study. However, a survey is a lot easier and less expensive to use, so you may be able to study a larger population. But a survey may also have specific biases based on the language, culture, and original application it was intended for. In most studies that use a survey, you can usually find a link to it in the study so you can see for yourself. If you have ever seen a paper that proposes a specific nutrient will lower or increase your risk for a particular disease by some percentage or amount, you are probably reading a study that used a survey.
4. What were the significant outcomes (primary and secondary) of the study?
Good clinical studies are designed specifically to look at one primary-outcome-based specific intervention. A primary outcome might be weight loss, fasting glucose levels, diabetes incidence, or death in response to a particular diet. In addition to primary outcomes, secondary outcomes related to the primary outcomes are also likely measured and reported, such as body composition, oral glucose tolerance test, use of diabetes medication, or healthspan for the examples above. The major outcomes that occur in response to the specific intervention are likely to be the most robust findings. Secondary or exploratory analysis might provide interesting insights to form the basis of future studies but might not hold up when replicated by others.
5. Are there biases at play?
The obvious one here is industry funding as a source of bias, and yes, many nutrition studies received funding from food industries, which is a problem. The integrity of the data might be subject to the researchers’ bias to promote their research or personal agenda with the goal of future funding. Identifying bias is not necessarily a reason to dismiss a study—we all have biases—but should give you additional context for understanding the findings. If you read a study funded by an industry trade group, for example, look for other studies on the same subject with different funding sources to see if the outcomes vary.
6. What are the confounding factors?
Confounding factors are alternative explanations that might explain the data. They may exist because of an oversight by the researchers or a lack of resources that didn’t allow the researchers to measure or control for every factor. Nutritional studies, in particular, can have a lot of confounding factors.
Examples include diet composition, age, physical activity levels, socio-economic factors, sleep, stress, time of year the study was done. It’s pretty much impossible not to have any confounders, and often the best we can do is use statistical approaches to “control” them and be aware of alternative explanations to the findings. Researchers are usually well aware of these limitations or become aware of them via the peer-review process. There are two places I look for these types of constraints. Often the reason a particular study is being done is because of limitations in previous research, which may be explained in the Introduction. Similarly, researchers will often acknowledge limitations towards the end of the discussion, some of which may be potential confounders that could influence the results.
7. Do these findings extrapolate to other populations?
It’s easy to see a study about how some diet might result in weight loss and improvement in metabolic disease risk factors, and think to yourself, “That’s the diet for ME!” It might be … if you are representative of the subject population for that study.
We often conduct research with metabolically unhealthy subjects (i.e., people with diabetes, cardiovascular disease, or obesity) or likely to develop a metabolic disease (i.e., people with prediabetes or known risk factors). That makes sense: those are the patients we want to help. However, how someone with a pre-existing metabolic disease responds to a diet might not be the same as someone young and healthy would respond, or the degree of improvement might not be as dramatic.
8. Do these findings extrapolate to other clinical measures?
Just as you can’t necessarily extend a finding from a population with diabetes to one without, you also can’t say that an outcome of one type necessarily leads to improvements in another. For example, improvements in a risk factor for cardiovascular disease might not translate to increased longevity or lower risks of diabetes or increased healthspan—even though these diseases are all related. Given that many of these diseases have common or similar causes, it’s tempting to say that if we improve one metabolic disease, we will improve others. But such interpretations should be made carefully and always need validation studies.
9. Is it statistical significance for clinical significance?
As researchers, we are trained that an intervention that does not achieve statistical significance does not work (which may or may not be correct). However, it’s important to remember that even if a result does achieve the bar of “statistical significance,” it’s possible that the effect is so small in absolute terms it’s not clinically relevant. Some examples include a slight decrease in body weight or a small increase in aerobic capacity. Sure, these clinical variables might move in the right direction, but how much benefit does someone get from losing 2 pounds when they are 50 pounds overweight?
This is particularly true when looking at associations between two variables in a large number of patients. It’s easy to find significant results with minimal effects (reported at the strength of correlation). As someone who is not a clinician, I am acutely aware that I can easily fall into this trap, so I often ask clinicians whether statistically significant changes are meaningful to them. Look at the actual absolute effects in the results and don’t be seduced by the phrase “statistically significant.”
10. Science is noisy and iterative but bends towards innovation and truth.
As exciting as any one study is, we must remember that science moves forward slowly; sometimes, three steps forward, two steps back. Often science is not testing the latest gadget or diet or training technique until well after it has gained acceptance by the general public. Scientists are skeptical by nature and generally move slowly and carefully, as our goal is broad applicability rather than individualization. Individuals can move fast, test a variety of interventions on themselves and learn some critical insights about themselves. While those results are valid for that person, they are not science. And it’s important not to confuse the two as they each provide their own value.
As with any skill, reading and understanding science takes practice, but it’s a skill that can unlock a fascinating world if you stick with it. It can help you understand the science behind the headlines to develop a nuanced understanding that allows a more realistic and practical implementation of scientific findings in your own life.
For a deeper dive into examining scientific studies, check out Peter Attia’s in-depth guide, Studying Studies.
