1.C.1
Critically evaluating research findings
Énoncé de compétence 1.C.1 d’ASPPB : Critically evaluate research findings (e.g., methodology, interpretation, assumptions, quality of evidence, researcher positionality)
This statement asks psychologists to judge whether a study's conclusions follow from its methods: how participants were assigned and measured, what the analysis assumes, how strong the whole body of evidence is and how the researchers' own position may have shaped the work. Published findings vary widely in quality, and selective reporting can make weak evidence look strong. Careful appraisal decides how much weight a finding deserves.
Notions clés
- Risk of bias
- Systematic error that pushes an estimate away from the truth. Cochrane's RoB 2 tool for randomized trials examines five domains: the randomization process, deviations from intended interventions, missing outcome data, measurement of the outcome and selection of the reported result.
- Selective outcome reporting
- Choosing which outcomes to report according to their results. Comparing protocols with publications, Chan and colleagues found that 62 percent of trials had at least one primary outcome changed, introduced or omitted, and that significant outcomes were more often fully reported.
- Certainty of a body of evidence
- GRADE rates certainty for a body of evidence, not a single study. Randomized trials start as high certainty and nonrandomized studies as low, and certainty can be rated down for risk of bias, inconsistency, indirectness, imprecision or publication bias.
- Analytic flexibility
- Undisclosed choices, such as which measures to report, when to stop collecting data or which covariates to include, can raise the false-positive rate far above the nominal .05. Full disclosure of these choices lets readers judge how much flexibility shaped a result.
- Researcher positionality
- Researchers' backgrounds, assumptions and relationships with participants can shape the questions asked and how data are read. Qualitative reporting standards ask authors to describe their perspectives and how they managed their influence, so readers can judge the analysis.
Exemple commenté
A published trial of an online program for insomnia states in its abstract that the program reduced daytime fatigue. The trial's registry entry, posted before enrollment, named insomnia severity at 8 weeks as the primary outcome and did not list fatigue. Insomnia severity appears only in a supplementary table, where the group difference was not significant.
- Compare the registry with the report: the registered primary outcome has been moved out of view and an unregistered outcome promoted to the abstract.
- Recognize the pattern: selective outcome reporting is common, and discrepancies between registered and published outcomes tend to favor statistically significant results.
- Judge the fatigue finding: it may be one of several outcomes examined, so it carries a higher risk of being a false positive and is best treated as exploratory.
- Weigh the trial: its main question showed no significant benefit, and the switch raises concern under RoB 2's domain on selection of the reported result.
Checking a trial's registered outcomes against its report can reveal selective reporting that the abstract hides.
Vérifiez vos acquis
A systematic review pools six randomized trials of a parenting program. Three found a clear benefit, two found no effect and one found that children did slightly worse, and the reviewers could not explain the differences by population, dose or methods. In GRADE terms, which concern most applies?
Sources
- Sterne, Savovic, Page and colleagues (2019), RoB 2: a revised tool for assessing risk of bias in randomised trials, BMJ (PMID 31462531), The five bias domains (main text)
- Chan, Hrobjartsson, Haahr, Gotzsche and Altman (2004), Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles, JAMA (PMID 15161896), Abstract, Results
- Mathieu, Boutron, Moher, Altman and Ravaud (2009), Comparison of registered and published primary outcomes in randomized controlled trials, JAMA (PMID 19724045), Abstract, Results and Conclusion
- Balshem, Helfand, Schunemann and colleagues (2011), GRADE guidelines: 3. Rating the quality of evidence, Journal of Clinical Epidemiology (PMID 21208779), Abstract
- Guyatt, Wang, Eachempati and colleagues (2025), Core GRADE 4: rating certainty of evidence: risk of bias, publication bias, and reasons for rating up certainty, BMJ (PMID 40360206), Abstract: randomized trials start high, nonrandomized studies low
- Guyatt, Schandelmaier, Brignardello-Petersen and colleagues (2025), Core GRADE 3: rating certainty of evidence: assessing inconsistency, BMJ (PMID 40328467), Abstract: inconsistency as unexplained variability across studies
- Guyatt, Oxman, Kunz and colleagues (2011), GRADE guidelines: 7. Rating the quality of evidence: inconsistency, Journal of Clinical Epidemiology (PMID 21803546), Abstract: rating down when some studies show benefit and others no effect or harm
- Simmons, Nelson and Simonsohn (2011), False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant, Psychological Science (PMID 22006061), Abstract
- Levitt, Bamberg, Creswell, Frost, Josselson and Suarez-Orozco (2018), Journal article reporting standards for qualitative primary, qualitative meta-analytic, and mixed methods research in psychology, American Psychologist (PMID 29345485), JARS-Qual: researcher description and managing perspectives