To RCT or not to RCT?

While interventional, randomized double blind placebo-controlled studies are the gold standard used to assess efficacy of interventions [1], answering other scientific questions does not necessarily require an active intervention.

My point is not to dis experimental studies in any way, as experimental studies are considered to be more reliable than observational studies because the process can be controlled, and randomization can eliminate bias and ensure comparable study groups in experimental studies [2]. Having said this, these types of research studies have their limitations.

-

Some shortcomings of RCTs

RCTs can be highly resource-intensive to conduct, for example to examine very long-term outcomes, which may be more important to patients [3]. Findings from a systematic review of costs in randomized controlled trials (RCT) reported that the median costs per recruited patient were USD 409 (ranging from USD 41 to 6,990). Additionally, overall costs of an RCT ranged from USD 43 to 103,254 per patient, and USD 0.2 to 611.5 million per RCT (although the methodology of gathering these overall estimates was not clear in 75% of the studies) [4].

Some interventions could be difficult to implement or could be considered unethical. This could happen for instance due to the absence of clinical equipoise [3], or because only a group is exposed to the intervention that could be advantageous to both groups.

Another disadvantage to using RCTs in population health research is the lack of generalizability, or low external validity [5]. This occurs for a number of reasons as explained by Sanson Fisher et al, the group, organization, or population that participates may be unrepresentative, such as, for example, a particular set of teaching hospitals.

In any case, what I want to be the most important message of this article is that RCTs are a form of study design and this design is not appropriate for all forms of research needs [6]. For example, diagnostic accuracy studies are best analyzed using a case-based, case-control design. Rare diseases are best studied using case-control designs [6].

-

Evidence based medicine

Evidence-Based Medicine is the integration of best research evidence with clinical expertise and patient values [7]. The classic Evidence-Based Pyramid [8] (Figure 1) depicts a triangular representation of the levels of evidence with the strongest evidence at the top and progressing down through evidence with decreasing strength. At the top of the pyramid are systematic reviews, which are the strongest form of evidence. Under these are lesser strength formats for filtered information including critically-appraised topics (evidence synthesis) and critically-appraised individual articles (article synopses) [9]. Below we can find unfiltered information, the top form being RCTs. Successive lower levels include cohort studies, case-control studies, case reports and background Information/Expert Opinion [9].

-

 

Figure 1. Evidence Based Medicine Pyramid (image by Andy Puro, 2014) [8]

-

For some years now, the Center for Evidence Based Medicine from Oxford University [10] has developed different levels of evidence based on the research question to be tackled. RCTs and specially, systematic reviews of RCTs are the highest order of evidence for assessing treatment benefits (efficacy), how well screening tools perform and treatment harms (both rare and common). On the other hand, random sample surveys are the appropriate way to assess prevalence of a given disease/condition, systematic reviews of cross-sectional studies are the best way to assess diagnostic method performance, and inception cohorts and their systematic reviews are the highest order of evidence for prognosis research (see Table 1).

-

Table 1. Oxford Centre for Evidence-Based Medicine 2011 Levels of Evidence [11]

 

* Level may be graded down on the basis of study quality, imprecision, indirectness (study PICO – patient, intervention, comparator, outcome- does not match questions PICO), because of inconsistency between studies, or because the absolute effect size is very small; Level may be graded up if there is a large or very large effect size.

** As always, a systematic review is generally better than an individual study.

-

Take home message

RCTs are not the answer for every scientific or research question. The use of observational study designs could be more fit for purpose, cheaper and more generalizable (when done appropriately).

-

---

 

References

1. Hariton E, Locascio JJ. Randomised controlled trials - the gold standard for effectiveness research: Study design: randomised controlled trials. BJOG. 2018;125(13):1716.
2. Kang H. Appropriate design of research and statistical analyses: observational versus experimental studies. Korean J Anesthesiol. 2013;65(2):105-7.
3. Saldanha IJ, Skelly AC, Ley KV, Wang Z, Berliner E, Bass EB, et al. Inclusion of Nonrandomized Studies of Interventions in Systematic Reviews of Intervention Effectiveness: An Update. AHRQ Methods for Effective Health Care. Rockville (MD)2022.
4. Speich B, von Niederhausern B, Schur N, Hemkens LG, Furst T, Bhatnagar N, et al. Systematic review on costs and resource use of randomized clinical trials shows a lack of transparent and comprehensive data. J Clin Epidemiol. 2018;96:1-11.
5. Sanson-Fisher RW, Bonevski B, Green LW, D'Este C. Limitations of the randomized controlled trial in evaluating population-based health interventions. Am J Prev Med. 2007;33(2):155-61.
6. Cook CE, Thigpen CA. Five good reasons to be disappointed with randomized trials. J Man Manip Ther. 2019;27(2):63-5.
7. Sackett DL SS, Richardson WS. Evidence-based medicine: how to practice and teach EBM. 2nd ed. Edinburgh: Churchill Livingstone; 2000.
8. Library HHS. Evidence-Based Medicine: Types of Studies 2022 [Available from: https://guides.himmelfarb.gwu.edu/ebm/studytypes.
9. Pittsburgh Uo. Evidence Based Medicine Resources 2025 [Available from: https://hsls.libguides.com/pyramid.
10. Medicine UoOCfE-B. OCEBM Levels of Evidence 2025 [Available from: https://www.cebm.ox.ac.uk/resources/levels-of-evidence/ocebm-levels-of-evidence.
11. OCEBM Levels of Evidence Working Group* JH, Iain Chalmers (James Lind Library), Paul Glasziou, Trish Greenhalgh, Carl Heneghan, Alessandro Liberati, Ivan Moschetti, Bob Phillips, Hazel Thornton, Olive Goddard and Mary Hodgkinson. The Oxford 2011 Levels of Evidence2011. Available from: http://www.cebm.net/index.aspx?o=5653.