Insufficient comprehensiveness of study collection and leaping to the conclusion

Abstract

To the Editor, We have read with interest the article by Ota et al.1 We applaud the authors’ efforts in conducting a systematic review on the efficacy of shakuyakukanzoto (SKT) for the treatment of muscle cramps. However, we have three concerns. First, the systematic review may have been insufficiently comprehensive.2 The authors did not search major literature and clinical trial registration databases, such as Embase, the International Clinical Trials Registry Platform, and ClinicalTrials.gov. Three randomizedcontrolled trials were included in their review, but a more extensive search could have yielded a more comprehensive collection of studies. The use of clinical trial registration databases to collect unpublished studies could have reduced publication bias. Second, the authors should clarify whether they included herbal medicines from countries other than Japan in their review, because herbal medicines such as SKT are frequently used in China and Korea. If herbal medicines from other countries were included, then databases containing Chineseor Koreanlanguage articles should also have been included. For example, one systematic review on the effect of herbal medicine on poststroke spasticity by Cai et al utilized Chinese databases and clinical trial registration websites.3 In China, SKT is known as “shaoyaogancaotang”; the randomizedcontrolled trials reviewed by Cai et al frequently utilized shaoyaogancaotang and its two ingredients, “bai shao” and “gan cao,” as interventions. If the authors reviewed studies on Chinese herbal medicines, they should have included “shaoyaogancaotang,” “bai shao,” and “gan cao” in their search. Finally, the authors’ conclusion in the abstract, “SKT might show efficacy in treating muscle cramps in patients with cirrhosis or lumbar spinal stenosis,” was a leap that may lead readers to overestimate the review s results. Of the three trials analyzed, two were at high risk of bias and one was of concern; moreover, a metaanalysis was not performed because of high heterogeneity. These issues raise doubts about the certainty of the evidence, making it difficult to conclude anything about direction of the effect.