That's a great question: what happens when we can't perform large trials?

Welcome to installment #2 of That’s a Great Question. Every month or so, I’ll highlight a thoughtful reader question that reflects a common concern or confusion—and answer it in depth here. Have a question you’d like considered? Comment below or email me at kristen@youcanknowthings.com.

Earlier this month I published “The Scale of Flawed Science” — a revamped version of the hierarchy of evidence for the age of memes, AI hallucinations, and fraud. Strong meta-analyses, randomized controlled trials, and well-designed observational studies are at the top (trust these), animal studies, small observational studies, and weak meta-analyses are in the middle (need more data), and screenshots, fraud, and claims that defy the laws of physics are at the bottom (unreliable, ignore).

Overall the feedback was positive, but one reader found the scale discouraging — for a very understandable reason. Here is his question (and story):

“I know this wasn’t your intention, but as a parent of a child with an ultra-rare disease (Barth syndrome), the Scale of Flawed Science is deeply discouraging. Since there are only about 140 patients in the US with Barth syndrome, trials for treatment would never move beyond “Meh, need more data” on the scale.

Do you have any thoughts about how research on rare and ultra-rare disease treatments fit within the Scale of Flawed Science?”

This question comes from Andrew Dryden, a science teacher who shared the story of his 6-month-old son Gilbert. Baby Gilbert has a genetic disorder called Barth Syndrome — a rare disease caused by a mutation that impairs mitochondria, the ‘powerhouse of the cell.’ The mutation starves the cells of energy, leading to heart failure, muscle weakness, and a weakened immune system. Tragically, many children with Barth Syndrome die before the age of 4.

Baby Gilbert has been taking a treatment called elamipretide which has shown promise for Barth Syndrome, but approval for this treatment has been challenging. Because Barth Syndrome is so rare—affecting only about 1 in 300,000—it’s hard to study. While smaller studies and case reports show promising results for elamipretide (and so far, Baby Gilbert is doing well on the treatment), there’s no large randomized trial. But in order to conduct a large trial, you need a large number of subjects with the disease. When the disease is extremely rare, it makes designing these ‘high quality’ studies very challenging.

This creates a conundrum for rare disease research, with parents like Gilbert’s left fighting for a potentially life-saving treatment for their child, and reviewers asking for studies that are difficult to perform due to the rarity (and severity) of the disease. You can read more about baby Gilbert’s story and elamipretide here.

Baby Gilbert’s story raises a broader question — what happens when we don’t have those “strong studies” on the top of the scale? Do we always wait, or do we act? Here is how I would approach it.

If the high quality studies exist, go with those. This was the main purpose of the scale — to help people rank quality of evidence. If you have a small observational study saying a drug doesn’t work, and multiple well-designed randomized-controlled trials saying it does, the randomized trials win.

Lower quality studies aren’t useless, but they should be interpreted with caution. Sometimes, we don’t have those larger trials — maybe the trials are in process, but we don’t have the data yet. Or maybe it’s not feasible to run a giant trial, because there aren’t enough subjects (as is the case for rare diseases like Barth Syndrome). Or perhaps it’s theoretically possible to run the trial, but no one wants to fund it. In all these scenarios, it’s very reasonable to use the lower quality data that we do have to guide decision making, understanding the limitations. Think of it like navigating an unknown city — a detailed map is best (top tier studies), but a partial map can still be useful (lower tier studies), and is typically much better than no map at all (no studies).

Ultimately, it comes down to a risk-benefit calculation. The goal of clinical research is to help us understand which treatments work — and which don’t — so we can make better medical decisions. But in reality, 100% certainty is rare, and we often have to act with incomplete information. When those beautiful, large randomized-controlled trials aren’t available, it’s reasonable for doctors to rely on the best evidence available, weighing the risks and benefits of treatment based on what is known — the drug’s safety profile, the benefits shown in the (imperfect) studies, and the overall quality of the data.

I like how Andrew, Baby Gilbert’s father, summarized it: “the hierarchy of evidence should not be an altar at which we sacrifice the lives of patients in pursuit of the ideal clinical trial. Instead, it should be a tool that empowers us to make thoughtful decisions that improve patient outcomes.”

Learn more about baby Gilbert and his parents’ fight for elamipretide approval here:

Kristen Panthagani, MD, PhD, is completing a combined emergency medicine residency and research fellowship focusing on health literacy and communication. In her free time, she is the creator of the medical blog You Can Know Things, available on Substack and youcanknowthings.com. You can also find her on Instagram and Threads. Views expressed belong to KP, not her employer.