Skip to content
Back to outputs

A review of measurement practice in studies of clinical decision support systems 1998-2017

Research output: Contribution to journalArticlepeer-review

Standard

A review of measurement practice in studies of clinical decision support systems 1998-2017. / Scott, Philip J.; Brown, Angela W.; Adedeji, Taiwo; Wyatt, Jeremy C; Georgiou, Andrew; Eisenstein, Eric L.; Friedman, Charles P.

In: Journal of American Medical Informatics Association, 16.04.2019.

Research output: Contribution to journalArticlepeer-review

Harvard

Scott, PJ, Brown, AW, Adedeji, T, Wyatt, JC, Georgiou, A, Eisenstein, EL & Friedman, CP 2019, 'A review of measurement practice in studies of clinical decision support systems 1998-2017', Journal of American Medical Informatics Association. https://doi.org/10.1093/jamia/ocz035

APA

Scott, P. J., Brown, A. W., Adedeji, T., Wyatt, J. C., Georgiou, A., Eisenstein, E. L., & Friedman, C. P. (2019). A review of measurement practice in studies of clinical decision support systems 1998-2017. Journal of American Medical Informatics Association. https://doi.org/10.1093/jamia/ocz035

Vancouver

Scott PJ, Brown AW, Adedeji T, Wyatt JC, Georgiou A, Eisenstein EL et al. A review of measurement practice in studies of clinical decision support systems 1998-2017. Journal of American Medical Informatics Association. 2019 Apr 16. https://doi.org/10.1093/jamia/ocz035

Author

Scott, Philip J. ; Brown, Angela W. ; Adedeji, Taiwo ; Wyatt, Jeremy C ; Georgiou, Andrew ; Eisenstein, Eric L. ; Friedman, Charles P. / A review of measurement practice in studies of clinical decision support systems 1998-2017. In: Journal of American Medical Informatics Association. 2019.

Bibtex

@article{568ff40154ec49a0940b8c6affbfdc47,
title = "A review of measurement practice in studies of clinical decision support systems 1998-2017",
abstract = "Objective - To assess measurement practice in clinical decision support evaluation studies.Materials and methods - We identified empirical studies evaluating clinical decision support systems published from 1998 to 2017. We reviewed titles, abstracts, and full paper contents for evidence of attention to measurement validity, reliability, or reuse. We used Friedman and Wyatt's typology to categorize the studies.Results - There were 391 studies that met the inclusion criteria. Study types in this cohort were primarily field user effect studies (n = 210) or problem impact studies (n = 150). Of those, 280 studies (72%) had no evidence of attention to measurement methodology, and 111 (28%) had some evidence with 33 (8%) offering validity evidence; 45 (12%) offering reliability evidence; and 61 (16%) reporting measurement artefact reuse.Discussion - Only 5 studies offered validity assessment within the study. Valid measures were predominantly observed in problem impact studies with the majority of measures being clinical or patient reported outcomes with validity measured elsewhere.Conclusion - Measurement methodology is frequently ignored in empirical studies of clinical decision support systems and particularly so in field user effect studies. Authors may in fact be attending to measurement considerations and not reporting this or employing methods of unknown validity and reliability in their studies. In the latter case, reported study results may be biased and effect sizes misleading. We argue that replication studies to strengthen the evidence base require greater attention to measurement practice in health informatics research.",
author = "Scott, {Philip J.} and Brown, {Angela W.} and Taiwo Adedeji and Wyatt, {Jeremy C} and Andrew Georgiou and Eisenstein, {Eric L.} and Friedman, {Charles P.}",
note = "{\textcopyright} The Author(s) 2019. Published by Oxford University Press on behalf of the American Medical Informatics Association.",
year = "2019",
month = apr,
day = "16",
doi = "10.1093/jamia/ocz035",
language = "English",
journal = "Journal of American Medical Informatics Association",
issn = "1067-5027",
publisher = "Oxford University Press",

}

RIS

TY - JOUR

T1 - A review of measurement practice in studies of clinical decision support systems 1998-2017

AU - Scott, Philip J.

AU - Brown, Angela W.

AU - Adedeji, Taiwo

AU - Wyatt, Jeremy C

AU - Georgiou, Andrew

AU - Eisenstein, Eric L.

AU - Friedman, Charles P.

N1 - © The Author(s) 2019. Published by Oxford University Press on behalf of the American Medical Informatics Association.

PY - 2019/4/16

Y1 - 2019/4/16

N2 - Objective - To assess measurement practice in clinical decision support evaluation studies.Materials and methods - We identified empirical studies evaluating clinical decision support systems published from 1998 to 2017. We reviewed titles, abstracts, and full paper contents for evidence of attention to measurement validity, reliability, or reuse. We used Friedman and Wyatt's typology to categorize the studies.Results - There were 391 studies that met the inclusion criteria. Study types in this cohort were primarily field user effect studies (n = 210) or problem impact studies (n = 150). Of those, 280 studies (72%) had no evidence of attention to measurement methodology, and 111 (28%) had some evidence with 33 (8%) offering validity evidence; 45 (12%) offering reliability evidence; and 61 (16%) reporting measurement artefact reuse.Discussion - Only 5 studies offered validity assessment within the study. Valid measures were predominantly observed in problem impact studies with the majority of measures being clinical or patient reported outcomes with validity measured elsewhere.Conclusion - Measurement methodology is frequently ignored in empirical studies of clinical decision support systems and particularly so in field user effect studies. Authors may in fact be attending to measurement considerations and not reporting this or employing methods of unknown validity and reliability in their studies. In the latter case, reported study results may be biased and effect sizes misleading. We argue that replication studies to strengthen the evidence base require greater attention to measurement practice in health informatics research.

AB - Objective - To assess measurement practice in clinical decision support evaluation studies.Materials and methods - We identified empirical studies evaluating clinical decision support systems published from 1998 to 2017. We reviewed titles, abstracts, and full paper contents for evidence of attention to measurement validity, reliability, or reuse. We used Friedman and Wyatt's typology to categorize the studies.Results - There were 391 studies that met the inclusion criteria. Study types in this cohort were primarily field user effect studies (n = 210) or problem impact studies (n = 150). Of those, 280 studies (72%) had no evidence of attention to measurement methodology, and 111 (28%) had some evidence with 33 (8%) offering validity evidence; 45 (12%) offering reliability evidence; and 61 (16%) reporting measurement artefact reuse.Discussion - Only 5 studies offered validity assessment within the study. Valid measures were predominantly observed in problem impact studies with the majority of measures being clinical or patient reported outcomes with validity measured elsewhere.Conclusion - Measurement methodology is frequently ignored in empirical studies of clinical decision support systems and particularly so in field user effect studies. Authors may in fact be attending to measurement considerations and not reporting this or employing methods of unknown validity and reliability in their studies. In the latter case, reported study results may be biased and effect sizes misleading. We argue that replication studies to strengthen the evidence base require greater attention to measurement practice in health informatics research.

U2 - 10.1093/jamia/ocz035

DO - 10.1093/jamia/ocz035

M3 - Article

C2 - 30990522

JO - Journal of American Medical Informatics Association

JF - Journal of American Medical Informatics Association

SN - 1067-5027

ER -

ID: 13855599