Generalisability
Item 21 - Generalisability (external validity, applicability) of the trial findings
Examples
“As the intervention was implemented for both sexes, all ages, all types of sports, and at different levels of sports, the results indicate that the entire range of athletes, from young elite to intermediate and recreational senior athletes, would benefit from using the presented training programme for the prevention of recurrences of ankle sprain. By including non-medically treated and medically treated athletes, we covered a broad spectrum of injury severity. This suggests that the present training programme can be implemented in the treatment of all athletes. Furthermore, as it is reasonable to assume that ankle sprains not related to sports are comparable with those in sports, the programme could benefit the general population.”(260)
“This replicates and extends the work of Clarke and colleagues and demonstrates that this CB (cognitive behavioural) prevention program can be reliably and effectively delivered in different settings by clinicians outside of the group who originally developed the intervention. The effect size was consistent with those of previously reported, single-site, indicated depression prevention studies and was robust across sites with respect to both depressive disorders and symptoms … In this generalisability trial, we chose a comparison condition that is relevant to public health—usual care … The sample also was predominantly working class to middle class with access to health insurance. Given evidence that CB therapy can be more efficacious for adolescents from homes with higher incomes, it will be important to test the effects of this prevention program with more economically and ethnically diverse samples.”(261)
Explanation
External validity, also called generalisability or applicability, is the extent to which the results of a study can be generalised to other circumstances.(262) Internal validity, the extent to which the design and conduct of the trial eliminate the possibility of bias, is a prerequisite for external validity: the results of a flawed trial are invalid and the question of its external validity becomes irrelevant. There is no absolute external validity; the term is meaningful only with regard to clearly specified conditions that were not directly examined in the trial. Can results be generalised to an individual participant or groups that differ from those enrolled in the trial with regard to age, sex, severity of disease, and comorbid conditions? Are the results applicable to other drugs within a class of similar drugs, to a different dose, timing, and route of administration, and to different concomitant therapies? Can similar results be expected at the primary, secondary, and tertiary levels of care? What about the effect on related outcomes that were not assessed in the trial, and the importance of length of follow-up and duration of treatment, especially with respect to harms?(263)
External validity is a matter of judgment and depends on the characteristics of the participants included in the trial, the trial setting, the treatment regimens tested, and the outcomes assessed.(5) (136) It is therefore crucial that adequate information be described about eligibility criteria and the setting and location (see item 4b), the interventions and how they were administered (see item 5), the definition of outcomes (see item 6), and the period of recruitment and follow-up (see item 14). The proportion of control group participants in whom the outcome develops (control group risk) is also important. The proportion of eligible participants who refuse to enter the trial as indicated on the flowchart (see item 13) is relevant for the generalisability of the trial, as it may indicate preferences for or acceptability of an intervention. Similar considerations may apply to clinician preferences.(264) (265)
Several issues are important when results of a trial are applied to an individual patient.(266) (267) (268) Although some variation in treatment response between an individual patient and the patients in a trial or systematic review is to be expected, the differences tend to be in magnitude rather than direction.
Although there are important exceptions,(268) therapies (especially drugs (269)) found to be beneficial in a narrow range of patients generally have broader application in actual practice. Frameworks for the evaluation of external validity have been proposed, including qualitative studies, such as in integral “process evaluations”(270) and checklists.(271) Measures that incorporate baseline risk when calculating therapeutic effects, such as the number needed to treat to obtain one additional favourable outcome and the number needed to treat to produce one adverse effect, are helpful in assessing the benefit-to-risk balance in an individual patient or group with characteristics that differ from the typical trial participant.(268) (272) (273) Finally, after deriving patient centred estimates for the potential benefit and harm from an intervention, the clinician must integrate them with the patient’s values and preferences for therapy. Similar considerations apply when assessing the generalisability of results to different settings and interventions.
Page last edited: 24 March 2010
