Reliability analysis for a proposed critical appraisal tool demonstrated value for diverse research designs
Crowe, Michael, Sheppard, Lorraine, and Campbell, Alistair (2012) Reliability analysis for a proposed critical appraisal tool demonstrated value for diverse research designs. Journal of Clinical Epidemiology, 65 (4). pp. 375-383.
|PDF (Published Version) - Repository staff only - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader|
|PDF (Supporting Online Material) - Repository staff only - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader|
View at Publisher Website: http://dx.doi.org/10.1016/j.jclinepi.201...
Objective: To examine the reliability of scores obtained from a proposed critical appraisal tool (CAT).
Study Design and Setting: Based on a random sample of 24 health-related research papers, the scores from the proposed CAT were examined using intraclass correlation coefficients (ICCs), generalizability theory, and participants' feedback.
Results: The ICC for all research papers was 0.83 (consistency) and 0.74 (absolute agreement) for four participants. For individual research designs, the highest ICC (consistency) was for qualitative research (0.91) and the lowest was for descriptive, exploratory and observational research (0.64). The G study showed a moderate research design effect (32%) for scores averaged across all papers. The research design effect was mainly in the Sampling, Results, and Discussion categories (44%, 36%, and 34%, respectively). The scores for research designs showed a majority paper effect for each (53–70%), with small to moderate rater or paper x rater interaction effects (0–27%).
Conclusions: Possible reasons for the research design effect were that the participants were unfamiliar with some of the research designs and that papers were not matched to participants' expertise. Even so, the proposed CAT showed great promise as a tool that can be used across a wide range of research designs.
|Item Type:||Article (Refereed Research - C1)|
|Keywords:||critical appraisal; methodology; research; validation; reliability; evidence-based practice; research design|
|FoR Codes:||16 STUDIES IN HUMAN SOCIETY > 1608 Sociology > 160807 Sociological Methodology and Research Methods @ 50%|
11 MEDICAL AND HEALTH SCIENCES > 1117 Public Health and Health Services > 111799 Public Health and Health Services not elsewhere classified @ 50%
|SEO Codes:||92 HEALTH > 9299 Other Health > 929999 Health not elsewhere classified @ 20%|
97 EXPANDING KNOWLEDGE > 970111 Expanding Knowledge in the Medical and Health Sciences @ 60%
97 EXPANDING KNOWLEDGE > 970116 Expanding Knowledge through Studies of Human Society @ 20%
|Deposited On:||05 Apr 2012 14:08|
|Last Modified:||01 May 2013 11:13|
Last 12 Months: 0
|Citation Counts with External Providers:||Web of Science: 2|
Repository Staff Only: item control page