A psychometric study found AMSTAR 2 to be a valid and moderately reliable appraisal tool.

Abstract

OBJECTIVES\nTo determine the inter-rater reliability (IRR) of AMSTAR 2 for reviews of pharmacological or psychological interventions for the treatment of major depression, to compare it to that of AMSTAR and ROBIS and to assess the convergent validity between the appraisal tools.\n\n\nSTUDY DESIGN AND SETTING\nTwo groups of four raters were each assigned one of two samples of 30 systematic reviews (SRs). All eight raters applied AMSTAR 2 to their sample. Each group also applied either AMSTAR or ROBIS. Fleiss kappa and Gwet s AC1 were calculated and agreement between the tools assessed.\n\n\nRESULTS\nThe median kappa values as a measure of IRR indicated a moderate agreement for AMSTAR 2 (Median=0.51), a substantial agreement for AMSTAR (Median=0.62), and a fair agreement for ROBIS (Median=0.27). Validity results showed a positive association for AMSTAR and AMSTAR 2 (r=0.91) as well as ROBIS and AMSTAR 2 (r=0.84). For the overall rating, AMSTAR 2 showed a high concordance with ROBIS and a lower concordance with AMSTAR.\n\n\nCONCLUSION\nThe IRR of AMSTAR 2 was found to be slightly lower than the IRR of AMSTAR and higher than the IRR of ROBIS. Validity measurements indicate that AMSTAR 2 is closely related to both ROBIS and AMSTAR.