Readers' guide for review articles: Why worry about methods?
ACP J Club. 1991 July-Aug;115:A12. doi:10.7326/ACPJC-1991-115-1-A12
Related Content in the Archives
• Editorial: Meta-statistics: help or hindrance?
Readers of medical journals would be astounded to find an original article that did not include a description of the research methods. On the other hand, they are not surprised to find review articles that do not report the methods used to prepare the review. In fact, until recently, almost no reviews included methods sections. Why, then, do the criteria for selecting review articles for inclusion in ACP Journal Club articles insist on a methods section?
Indeed, some of our readers might ask why we include review articles at all. We do so for 2 reasons. First, it is sometimes not feasible to test an innovation definitively. Second, the completed studies on a given topic may not provide a clear answer; reviews of inconclusive or conflicting studies can sometimes help sort out these situations. In fact, because few definitive studies exist on any clinical topic, scientifically sound reviews (or “overviews”) are needed for most clinical questions.
The reason for insisting on a methods section for review articles is the same as that for the report of an original investigation: Without knowing how a conclusion was reached, the reader has difficulty deciding whether to believe it (1). Readers of reviews that do not report the methods of review must accept the conclusions of the authors on faith.
Many types of review articles exist, but those addressing practical clinical questions should be, in fact, scientific surveys of the literature. The same principles that apply to epidemiologic surveys apply to reviews: A question must be posed, a target population of information sources must be identified and accessed, a reproducible method must be used to select articles for detailed review, appropriate information must be extracted from those articles in an unbiased way, and conclusions must be derived that are supported by the evidence presented. In meta-analyses (quantitative overviews), formal statistical analysis should be used to help derive conclusions.
As with other research, several methodologic decisions must be made when preparing a review, and each one involves potential threats to the validity of the ultimate conclusion. These threats include how studies are identified and selected for inclusion, how the quality of studies is assessed, how much weight is given each study, and how results are combined and interpreted. If authors do not describe how these decisions were made, readers cannot assess the credibility of the authors' conclusions.
Unfortunately, reviews of the same topic often disagree, and, if the authors and peer reviewers do not pay enough attention to review methodology, neither group can guarantee the validity of a review. Experts preparing reviews risk being unduly influenced by their own research, and their understandably strong opinions about their area can lead them to judge other evidence more on its support for their convictions than on its methodologic strengths.
ACP Journal Club's criteria for selecting review articles begin by insisting that they have a methods section. This section enables readers to decide for themselves whether the review has been conducted using reproducible methods and with safeguards against bias in the selecting and interpreting of studies. Readers can then make informed decisions about using the review's conclusions as the basis for clinical decisions that will do more good than harm.
Andrew D. Oxman, MD, MSc