ADVICE FROM LIBRARIANS
Key considerations for conducting reviews
developed by Robin Parker & Lindsey Sikora
The PIECES framework proposed by Foster and Jewell (2017) is a useful tool to guide a researcher through the overarching phases of a review.
The PIECES acronym stands for:
Planning the review
Evaluating and appraising the evidence
Collecting and combining data;
Explaining the synthesis;
Summarizing the findings.
Putting the PIECES together
PLANNING THE REVIEW
The essential groundwork of a successful review involves developing a clear research question, along with inclusion and exclusion criteria. The research question must align with the review type, with some narrow in scope, defined before data collection, and precise (e.g., systematic reviews); others are broad in scope, evolve over the course of the data collection and analysis, and become precise during the review process (e.g., state-of-the-art reviews). In general, the research question should be both comprehensive and clear, with details about the key concepts: the population or problem; intervention, innovation, or exposure of interest; and particular learner, organizational, or health outcomes.
Whether or not researchers write a formal review protocol or research proposal, as is strongly recommended for systematic and scoping reviews, authors should prepare for a significant planning stage to set themselves up for success. This planning starts with developing and refining an appropriate research question (or questions), building a review team, and selecting the suitable type of synthesis methodology.
During the planning stage, scholars should also engage in a preliminary search into the literature to confirm the need for a review, as well as to get an initial sense of the types and volume of data sources that may address the research question. Keep in mind that the initial scan of the literature and refining the review question may be an iterative process, where the former can influence modifications to the latter.
A part of the planning process to consider early on, alongside the formation of the review team, is determining the resources and technologies available to assist with the synthesis process. In addition to bibliographic databases and journal subscriptions, citation management software, review management software, and data analysis and reporting tools should also be considered.
Finally, once the review type and question have been finalized, the review team will need to discuss and decide on a list of inclusion and exclusion criteria. In other words, how will reviewers know whether or not a given article or other data source should be retained for the synthesis? The types of evidence to be included and the characteristics of that evidence will be determined by the fit to address the review question(s) and the type of analysis planned.
Protip: Determining the ideal venue for publication (e.g. open access options) and identifying the target audience should be considered during the planning phase to ensure alignment with the research question and purpose.
IDENTIFYING STUDIES & RESOURCES
Identifying studies begins by searching the appropriate databases, which are determined based on the research question. Once the initial database searches are complete, it is important to supplement the search with studies that may not have been included in the databases. This often means conducting a supplemental search looking through conference abstracts, technical reports, association websites and other grey literature resources (research produced by organizations outside of the traditional commercial or academic publishing and distribution channels). Once a rigorous search through the databases and supplemental search resources has been completed, the studies can then be evaluated.
It is important to work with an information specialist (IS), such as a librarian, to determine the most appropriate sources to search and to find the most efficient approaches to identifying potentially relevant evidence. The IS can advise on, or even help to develop, the search strategy using best practices for conducting and reporting searches. We recommend inviting an IS to join the review team or to serve as a consultant on this important part of the review process (Morris, Boruff & Gore, 2016; Spencer & Eldridge, 2018).
Protip: For reviews on topics in postgraduate medical education, don’t forget to search education-specific databases such as Education Resources Information Center (ERIC), Education Research Complete, or others at your institutional library.
EVALUATING & APPRAISING THE EVIDENCE
Evaluating and appraising the evidence includes screening the retrieved studies or other data sources to determine eligibility, first based on fit to the topic (relevance). For some review types, this may consist of a systematic screening of all studies to select those that meet the inclusion and exclusion criteria. For others, researchers may use a more strategic and purposive means of sampling the data to address the review question. In some cases, eligibility may also include meeting certain methodological quality standards (rigor). Some review methods will include a formalized appraisal process, such as using a checklist to determine the risk of bias within the data source (e.g., Cochrane Risk of Bias tool or Medical Education Research Study Quality Inventory (Cook & Reed, 2015)), whereas other reviews may look for fit with the theoretical or conceptual framework of the issue being examined.
Protips: 1) Pilot your inclusion criteria with a sample (for example 10% of the full set of retrieved records) to ensure the entire review team is clear on what should be included and how the criteria are operationalized. This will help improve the inter-rater reliability of the full project. 2) Consider using review management software to handle the large number of citations and facilitate collaborative screening.
COLLECTING & COMBINING DATA
Each review type will have different approaches for collecting and combining data, which involves pulling the relevant pieces of information out of the selected evidence. Some review types (e.g.systematic and scoping reviews) include extracting specific variables into forms or tables using a data extraction template. Other types of review (e.g. state-of-the-art reviews) will use inductive or interpretive approaches to generate themes, codes, or other types of new data out of the texts of the included data sources. Regardless of the review type, the common aspect to this stage is distilling the primary data to the elements that will be used to answer the research question.
Protip: Again, piloting the data extraction form with a handful of included studies will help to catch omissions in the data collection process. Refer back to the review question and the planned analysis when considering the variables that should be extracted from each study. This will also help improve the inter-rater reliability of the full project.
EXPLAINING THE SYNTHESIS
At this stage, researchers will bring the results from the individual studies together for analysis, and then explain what they have identified through the synthesis. While this step will look different for each review type, it is a distinguishing feature of all synthesis methodologies. Whether the synthesis process is quantitative, qualitative, conceptually-guided, or some combination thereof, this step is what makes a review an actual form of knowledge synthesis. It is at this step that researchers transparently and systematically begin drawing together data from disparate sources to address the research question, leading to possible recommendations or other outcomes. Whatever form this synthesis may take, the important commonality is that the synthesis process, itself, is clearly explained with key decisions made by the team explicitly described for the reader.
Protip: For some review types, there are software options that can assist with analysis, such as RevMan for meta-analysis or qualitative data analysis software for qualitative analyses.
SUMMARIZING THE FINDINGS
Once the findings have been synthesized and the review team is ready for dissemination, several factors need to be considered. Visualizations, such as figures and tables, together with the text of the report, are key for disseminating reviews. Pulling these parts into a coherent narrative with key findings highlighted in engaging ways will lead to successful publication, especially after following the other steps according to the guidelines and standards appropriate for the review type. Researchers can look for reporting standards for their specific review methodology as a guide. Keeping the purpose and desired audience in mind, researchers can format their report into a manuscript suitable for publication.
ROBIN PARKER, PHD(C)
Robin Parker is the Evidence Synthesis Librarian for Dalhousie Libraries in Nova Scotia, Canada where she also supports research and learning for Dalhousie University’s Faculty of Medicine. Robin is an Interdisciplinary PhD candidate at Dalhousie and is researching how academic librarians teach evidence synthesis methods to learners in health and medicine programs. Robin has supported hundreds of review projects and is co-author on over a dozen published systematic and scoping reviews.
LINDSEY SIKORA, PHD(C)
Lindsey Sikora is the Head for Health Sciences, Medicine, Science, Technology, Engineering and Mathematics (HMSTEM) at the University of Ottawa Library in Ottawa, Ontario, Canada. She obtained her Masters of Information Studies from the University of Toronto, in Toronto, Ontario, Canada, and her Bachelors of Science (Hons.) in Behavioural Neuroscience from Laurentian University in Sudbury, Ontario, Canada. She is currently a PhD candidate at the University of Ottawa, Faculty of Education, in the Health Professions Education stream. Lindsey has vast experience working collaboratively with many research groups on knowledge synthesis within the areas of medicine, health sciences, education and librarianship.