Assessing the quality of research needs to go beyond scoring: commentary on Crowe and Sheppard (2011): authors' response

Abstract

[Extract] To evaluate the quality of research is one of the most challenging steps in review processes, either as part of systematic reviews or peer review. Throughout the last decades, a number of criteria have been discussed and various instruments have been proposed.

The critical appraisal tool (CAT) newly developed by Crowe and Sheppard (2011) contributes to this increasing methodological inventory. It was developed based on a critical review of more than 40 CATs and reporting guidelines for different study designs of primary and secondary research (Crowe and Sheppard, 2011a). Its structure follows that of reporting guidelines. The tool contains eight sections following the study outline, e.g. introduction, design, sample, or findings. Each section is divided into sub-sections containing different descriptors, which focus either on reporting issues or aspects of research conduct and aim to guide the reviewer through the appraisal process. Based on the judgements made for each applicable descriptor, the reviewer has to assign a score between 0 and 5 to each section of the study report.

Although the CAT was developed and validated systematically, there are limitations which, in my view, require further discussion.