External validation of multivariable prediction models

A systematic review of methodological conduct and reporting

Gary S. Collins, Joris A. De Groot, Susan Dutton, Omar Omar, Milensu Shanyinde, Abdelouahid Tajar, Merryn Voysey, Rose Wharton, Ly Mee Yu, Karel G. Moons, Douglas G. Altman

Research output: Contribution to journalReview article

157 Citations (Scopus)

Abstract

Background: Before considering whether to use a multivariable (diagnostic or prognostic) prediction model, it is essential that its performance be evaluated in data that were not used to develop the model (referred to as external validation). We critically appraised the methodological conduct and reporting of external validation studies of multivariable prediction models. Methods. We conducted a systematic review of articles describing some form of external validation of one or more multivariable prediction models indexed in PubMed core clinical journals published in 2010. Study data were extracted in duplicate on design, sample size, handling of missing data, reference to the original study developing the prediction models and predictive performance measures. Results: 11,826 articles were identified and 78 were included for full review, which described the evaluation of 120 prediction models. in participant data that were not used to develop the model. Thirty-three articles described both the development of a prediction model and an evaluation of its performance on a separate dataset, and 45 articles described only the evaluation of an existing published prediction model on another dataset. Fifty-seven percent of the prediction models were presented and evaluated as simplified scoring systems. Sixteen percent of articles failed to report the number of outcome events in the validation datasets. Fifty-four percent of studies made no explicit mention of missing data. Sixty-seven percent did not report evaluating model calibration whilst most studies evaluated model discrimination. It was often unclear whether the reported performance measures were for the full regression model or for the simplified models. Conclusions: The vast majority of studies describing some form of external validation of a multivariable prediction model were poorly reported with key details frequently not presented. The validation studies were characterised by poor design, inappropriate handling and acknowledgement of missing data and one of the most key performance measures of prediction models i.e. calibration often omitted from the publication. It may therefore not be surprising that an overwhelming majority of developed prediction models are not used in practice, when there is a dearth of well-conducted and clearly reported (external validation) studies describing their performance on independent participant data.

Original languageEnglish
Article number40
JournalBMC Medical Research Methodology
Volume14
Issue number1
DOIs
Publication statusPublished - 19 Mar 2014
Externally publishedYes

Fingerprint

Validation Studies
Calibration
PubMed
Sample Size
Publications
Datasets

ASJC Scopus subject areas

  • Epidemiology
  • Health Informatics

Cite this

External validation of multivariable prediction models : A systematic review of methodological conduct and reporting. / Collins, Gary S.; De Groot, Joris A.; Dutton, Susan; Omar, Omar; Shanyinde, Milensu; Tajar, Abdelouahid; Voysey, Merryn; Wharton, Rose; Yu, Ly Mee; Moons, Karel G.; Altman, Douglas G.

In: BMC Medical Research Methodology, Vol. 14, No. 1, 40, 19.03.2014.

Research output: Contribution to journalReview article

Collins, GS, De Groot, JA, Dutton, S, Omar, O, Shanyinde, M, Tajar, A, Voysey, M, Wharton, R, Yu, LM, Moons, KG & Altman, DG 2014, 'External validation of multivariable prediction models: A systematic review of methodological conduct and reporting', BMC Medical Research Methodology, vol. 14, no. 1, 40. https://doi.org/10.1186/1471-2288-14-40
Collins, Gary S. ; De Groot, Joris A. ; Dutton, Susan ; Omar, Omar ; Shanyinde, Milensu ; Tajar, Abdelouahid ; Voysey, Merryn ; Wharton, Rose ; Yu, Ly Mee ; Moons, Karel G. ; Altman, Douglas G. / External validation of multivariable prediction models : A systematic review of methodological conduct and reporting. In: BMC Medical Research Methodology. 2014 ; Vol. 14, No. 1.
@article{b8750ce1895a4c4b8466b515dc9d804d,
title = "External validation of multivariable prediction models: A systematic review of methodological conduct and reporting",
abstract = "Background: Before considering whether to use a multivariable (diagnostic or prognostic) prediction model, it is essential that its performance be evaluated in data that were not used to develop the model (referred to as external validation). We critically appraised the methodological conduct and reporting of external validation studies of multivariable prediction models. Methods. We conducted a systematic review of articles describing some form of external validation of one or more multivariable prediction models indexed in PubMed core clinical journals published in 2010. Study data were extracted in duplicate on design, sample size, handling of missing data, reference to the original study developing the prediction models and predictive performance measures. Results: 11,826 articles were identified and 78 were included for full review, which described the evaluation of 120 prediction models. in participant data that were not used to develop the model. Thirty-three articles described both the development of a prediction model and an evaluation of its performance on a separate dataset, and 45 articles described only the evaluation of an existing published prediction model on another dataset. Fifty-seven percent of the prediction models were presented and evaluated as simplified scoring systems. Sixteen percent of articles failed to report the number of outcome events in the validation datasets. Fifty-four percent of studies made no explicit mention of missing data. Sixty-seven percent did not report evaluating model calibration whilst most studies evaluated model discrimination. It was often unclear whether the reported performance measures were for the full regression model or for the simplified models. Conclusions: The vast majority of studies describing some form of external validation of a multivariable prediction model were poorly reported with key details frequently not presented. The validation studies were characterised by poor design, inappropriate handling and acknowledgement of missing data and one of the most key performance measures of prediction models i.e. calibration often omitted from the publication. It may therefore not be surprising that an overwhelming majority of developed prediction models are not used in practice, when there is a dearth of well-conducted and clearly reported (external validation) studies describing their performance on independent participant data.",
author = "Collins, {Gary S.} and {De Groot}, {Joris A.} and Susan Dutton and Omar Omar and Milensu Shanyinde and Abdelouahid Tajar and Merryn Voysey and Rose Wharton and Yu, {Ly Mee} and Moons, {Karel G.} and Altman, {Douglas G.}",
year = "2014",
month = "3",
day = "19",
doi = "10.1186/1471-2288-14-40",
language = "English",
volume = "14",
journal = "BMC Medical Research Methodology",
issn = "1471-2288",
publisher = "BioMed Central",
number = "1",

}

TY - JOUR

T1 - External validation of multivariable prediction models

T2 - A systematic review of methodological conduct and reporting

AU - Collins, Gary S.

AU - De Groot, Joris A.

AU - Dutton, Susan

AU - Omar, Omar

AU - Shanyinde, Milensu

AU - Tajar, Abdelouahid

AU - Voysey, Merryn

AU - Wharton, Rose

AU - Yu, Ly Mee

AU - Moons, Karel G.

AU - Altman, Douglas G.

PY - 2014/3/19

Y1 - 2014/3/19

N2 - Background: Before considering whether to use a multivariable (diagnostic or prognostic) prediction model, it is essential that its performance be evaluated in data that were not used to develop the model (referred to as external validation). We critically appraised the methodological conduct and reporting of external validation studies of multivariable prediction models. Methods. We conducted a systematic review of articles describing some form of external validation of one or more multivariable prediction models indexed in PubMed core clinical journals published in 2010. Study data were extracted in duplicate on design, sample size, handling of missing data, reference to the original study developing the prediction models and predictive performance measures. Results: 11,826 articles were identified and 78 were included for full review, which described the evaluation of 120 prediction models. in participant data that were not used to develop the model. Thirty-three articles described both the development of a prediction model and an evaluation of its performance on a separate dataset, and 45 articles described only the evaluation of an existing published prediction model on another dataset. Fifty-seven percent of the prediction models were presented and evaluated as simplified scoring systems. Sixteen percent of articles failed to report the number of outcome events in the validation datasets. Fifty-four percent of studies made no explicit mention of missing data. Sixty-seven percent did not report evaluating model calibration whilst most studies evaluated model discrimination. It was often unclear whether the reported performance measures were for the full regression model or for the simplified models. Conclusions: The vast majority of studies describing some form of external validation of a multivariable prediction model were poorly reported with key details frequently not presented. The validation studies were characterised by poor design, inappropriate handling and acknowledgement of missing data and one of the most key performance measures of prediction models i.e. calibration often omitted from the publication. It may therefore not be surprising that an overwhelming majority of developed prediction models are not used in practice, when there is a dearth of well-conducted and clearly reported (external validation) studies describing their performance on independent participant data.

AB - Background: Before considering whether to use a multivariable (diagnostic or prognostic) prediction model, it is essential that its performance be evaluated in data that were not used to develop the model (referred to as external validation). We critically appraised the methodological conduct and reporting of external validation studies of multivariable prediction models. Methods. We conducted a systematic review of articles describing some form of external validation of one or more multivariable prediction models indexed in PubMed core clinical journals published in 2010. Study data were extracted in duplicate on design, sample size, handling of missing data, reference to the original study developing the prediction models and predictive performance measures. Results: 11,826 articles were identified and 78 were included for full review, which described the evaluation of 120 prediction models. in participant data that were not used to develop the model. Thirty-three articles described both the development of a prediction model and an evaluation of its performance on a separate dataset, and 45 articles described only the evaluation of an existing published prediction model on another dataset. Fifty-seven percent of the prediction models were presented and evaluated as simplified scoring systems. Sixteen percent of articles failed to report the number of outcome events in the validation datasets. Fifty-four percent of studies made no explicit mention of missing data. Sixty-seven percent did not report evaluating model calibration whilst most studies evaluated model discrimination. It was often unclear whether the reported performance measures were for the full regression model or for the simplified models. Conclusions: The vast majority of studies describing some form of external validation of a multivariable prediction model were poorly reported with key details frequently not presented. The validation studies were characterised by poor design, inappropriate handling and acknowledgement of missing data and one of the most key performance measures of prediction models i.e. calibration often omitted from the publication. It may therefore not be surprising that an overwhelming majority of developed prediction models are not used in practice, when there is a dearth of well-conducted and clearly reported (external validation) studies describing their performance on independent participant data.

UR - http://www.scopus.com/inward/record.url?scp=84899459258&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84899459258&partnerID=8YFLogxK

U2 - 10.1186/1471-2288-14-40

DO - 10.1186/1471-2288-14-40

M3 - Review article

VL - 14

JO - BMC Medical Research Methodology

JF - BMC Medical Research Methodology

SN - 1471-2288

IS - 1

M1 - 40

ER -