#50: How to read the medical literature like a journal editor

July 27, 2017 | By Matthew Watto, MD

Master this practical approach to reading the medical literature (*No statistics needed!) with expert tips from Dr. Christine Laine, Editor in Chief, Annals of Internal Medicine, and Dr. Darren Taichman, Executive Deputy Editor, Annals of Internal Medicine. They teach us what we should be reading, and detail their thought processes as they appraise an article. Topics covered include: Is 3 minute critical appraisal possible? What’s the deal with P-values? What are common sources of bias? How does the approach differ with clinical trials versus observational studies versus meta-analyses?

*Minimal statistics needed ; )

Join our newsletter mailing list. Rate us on iTunes, recommend a guest or topic and give feedback at thecurbsiders@gmail.com.

Critical Appraisal Pearls:

1. “Most of critical appraisal comes before statistical analysis!” -Dr. Christine Laine
2. Step 1: What question is the article trying to answer? Are these endpoints clinically relevant to my patients/practice?
3. Step 2: What interventions are being compared?
4. Step 3: Who was included in this study? Would my patient meet certain inclusion, or exclusion criteria?
5. Step 4: Was there a difference in outcomes between groups? If not, then was the study adequately powered to detect a difference?
6. PICO acronym: Patient- age, sex, and other important demographic info. Intervention- can be diagnostic, therapeutic, etc. Comparison- To what intervention is it compared? Outcome- What endpoints are studied?
7. Power: The ability of a study to detect a difference between groups (e.g. if power is too low, then study may erroneously conclude no difference between groups). Questions to ask yourself: How many patients were in the study (sample size)? Is the magnitude of the difference between the groups meaningful or clinically significant? What is the frequency of each endpoint (e.g. death, hospitalizations, etc.) being studied?
8. P-values: Don’t just look for a “significant” p-value of <0.05. Look at the confidence interval (CI) and its lower and upper bounds to assess for overlap between groups being measured.
9. 95 percent confidence interval: Tells you w/95% confidence that the true result fell within a given range e.g. A broad 95% CI of (1.5 to 95) tells us with 95% certainty that the true result is between 1.5 and 95. A narrower 95% CI of (1.5 to 3.5) for the same endpoint would give a more precise estimate of where the true value lies.
10. “Clinical” versus “statistical” significance: A p-value of <0.05 does not always mean “clinical” significance. It’s more important to assess the importance of the endpoint being studied and the confidence intervals (see above). E.g. If group A has 60% mortality, group B has 60.1% mortality and p-value is <0.001, then this study showed a “statistically”, but not “clinically” significant difference.
11. Bias: Common sources include industry funding, design flaws, incomplete blinding. Researchers may have intellectual biases of their own that are hard to uncover. Industry funding is most troublesome when backing opinion pieces where methods, and data are not fully disclosed.
12. Intention to treat (ITT): Definition varies by author. Ideally an ITT analysis includes all patients who participated in a study regardless of whether or not they completed the planned protocol. Benefits = preserves randomization and minimizes confounding.
13. Per protocol analysis: Analysis includes data only from patients who followed protocol based on the planned/assigned intervention. If lots of patients drop out, the benefits of randomization can be lost leading to more of an observational study with higher risk of bias.
14. Observational study: Bias is much higher and also more difficult to detect. Observational studies are sometimes the best evidence available if no clinical trial has been done in your patient population. Types include: case-control, cohort, cross-sectional, and longitudinal studies
15. Systematic review: Must be performed prior to a meta-analysis. Includes everything that is out there. More rigorous than a “narrative review”, which doesn’t necessarily use all relevant studies. Often, the conclusion is that the evidence cannot answer this question. As a reader must evaluate search strategy used, and quality of included studies.
16. Meta-analysis: A statistical analysis that combines information from multiple research studies. Must follow a systematic review. Sometimes it is not appropriate to perform a meta-analysis if the studies included have a high degree of heterogeneity (excessive variation in outcomes between studies).

Goal: Listeners will perform rapid and effective appraisal and be familiar with common terminology when reading medical literature.

Learning objectives:
After listening to this episode listeners will…

1. Develop a 3 minute process to appraise any scientific study
2. Identify study type
3. Recognize common sources of bias in scientific studies
4. Evaluate clinical relevance of end-points
5. Interpret common statistical terminology e.g. p-value, power
6. Define intention-to-treat
7. Define per protocol analysis and recall its limitations
8. Evaluate a study for statistical power to detect a difference between groups
9. Explain the benefits and risks of using composite endpoints
10. Utilize PICO to formulate a clinical question, or to assist with critical appraisal
11. Recall the difference between a systematic and narrative review.

Disclosures:
Dr. Christine Laine and Dr. Darren Taichman are both editors of the Annals of Internal Medicine and employees of the American College of Physicians.

Time Stamps
00:00 Intro
01:13 Listener comment on Entresto
02:50 Picks of the week
09:36 Getting to know our guests
14:00 How to stay up on the medical literature
17:15 Three minute critical appraisal
19:15 Step 1: Assess the outcome being studied
20:50 Statistical versus clinical significance
22:33 Evaluating composite endpoints
24:47 Statistical power
28:58 Evaluating for bias
34:40 Recap of what we’ve learned so far
36:33 Is PICO useful?
39:01 Observational studies and bias
41:09 Evaluating a meta-analysis
46:05 Take home points
50:35 The Curbsiders recap the episode
53:02 Outro

Links from the show:

1. Creed (film) by Ryan Coogler
2. Designing Clinical Research 4th Ed (book) by Stephen Hulley
3. The Black Swan (book) by Nassim Nicholas Taleb
4. The Black Swan (film) by Darren Aronofsky
5. Wonder (book) by R.J. Palacio
6. JAMA User’s Guide to the Medical Literature (book) by Gordon Guyatt et al.
7. Join the American College of Physicians (ACP) here
8. Annals of Internal Medicine is free for ACP members
9. JournalWise is free for ACP members: It is customizable and sends email alerts w/links to important articles along with expert commentary.