If you’ve spent much time digging through the research literature in your field of study, you’ve undoubtedly come across systematic review and meta-analysis studies. For many of our dissertation assistance clients, conducting a review study is one path they can choose for their graduate research. So, what exactly is a systematic review and meta-analysis? In this article, we’ll go over the basics of this form of research, including the role of this form of research and also the steps involved. This should help you to decide whether completing a systematic review is a good choice for your own dissertation research.

What Is the Role of Systematic Reviews and Meta-Analyses?

Systematic review and meta-analysis studies hold the very top spot on the hierarchy of evidence-based research, meaning that they provide the strongest form of evidence available. When you hear terms like “evidence-based practice” or “evidence-based policy,” this evidence is comprised largely of systematic review and meta-analysis studies. Clearly these studies make very important contributions to the research literature, which many of our dissertation assistance clients find appealing—it’s an ambitious and prestigious form of research to undertake! In fact, researchers and practitioners often specifically seek out systematic reviews to help them gain a sense of the state of current knowledge on specific topics. 

As you have delved into the literature in your own field, you’ve surely noticed how vast the research literature can be, and you’ve probably also noticed that different studies—using both quantitative and qualitative analysis —can come to different conclusions on the same or similar questions. The chore of making sense of large amounts of research is what makes dissertation topic development and literature reviews so time-consuming! Now, imagine that you are a practitioner seeking information on the most effective treatments for a patient’s illness—would you want to spend that amount of time sifting through the literature to get an answer to your question?

Definitely not, and this is where the systematic review and meta-analysis can help. Because systematic reviews provide such strong evidence, practitioners in health and therapeutic fields rely on this type of research for keeping up to date on effectiveness of treatments and therapies for various health conditions. Many of our dissertation consulting clients are working toward graduate degrees in health sciences fields, and for them, conducting a systematic review and meta-analysis provides the opportunity to contribute directly to healthcare practice. That’s pretty exciting!

Systematic reviews are not just appropriate for health treatment research, however. Although the approach originated in health-related fields of research, this form of study has now spread throughout many different fields. As you might imagine, systematic reviews are especially important for topics where there has already been a ton of research. The reason for this is that they help to analyze the various results of multiple studies—which may include both statistical analysis and qualitative analysis—and condense these down to a much more concise rendering of the findings. 

For example, transformational leadership is a construct that many of our dissertation assistance clients know quite well, as it is heavily researched in relation to workplace outcomes. To help make sense of the massive amount of research on this leadership style, researchers have conducted systematic reviews and meta-analyses to answer such questions as how transformational leadership generalizes across cultures (Crede et al., 2019) and how it affects employee creativity (Koh et al., 2019).

In addition to providing the state of knowledge on a topic, systematic reviews also help to identify areas of conflict or research gaps. If you have begun searching for a research gap as you develop the problem statement for your own dissertation, then you know just how valuable this is! These are the types of contributions you could make if you chose to conduct a systematic review for your dissertation or thesis. Next, to give you a clearer idea of what systematic reviews and meta-analyses entail, let’s discuss each of these in more detail.

What Is a Systematic Review?

The signature features of the systematic review are (a) an intensive process that reduces bias at every stage of the study and (b) transparent reporting that allows other researchers to assess the study’s rigor. As indicated previously, the overall goal of the systematic review is to locate and critically appraise all studies on a given topic, providing a synthesis of the study’s findings to answer one or more guiding research questions. Including all studies on the topic relevant to the research question(s) supports the quality of the systematic review because it eliminates the potential for selection bias (i.e., only picking out certain studies to include in the review).

 To meet commonly accepted standards of rigor, a systematic review should be designed to conform to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. In our work with dissertation assistance clients, we always follow the PRISMA method for systematic reviews, which specifies the aspects of the study you need to report upon thoroughly to meet standards of transparency. This includes developing and reporting a priori processes for key steps like (a) sample study search criteria, (b) inclusion and exclusion criteria, (c) quality evaluation, and (d) data extraction and analysis. 

You might be wondering why we are defining systematic review and meta-analysis separately, as these terms are often considered synonymous. But, the two are actually not the same. Meta-analysis is a form of statistical analysis that is typically but not always included within a systematic review. A meta-analysis can only be conducted when two or more studies chosen for the systematic review have very similar variables. Studies with populations and/or variables that are too different cannot be run through the statistical analysis processes associated with meta-analysis; instead, they are summarized in a table, with comparisons of outcomes discussed in a qualitative sense. 

What Is a Meta-Analysis?

In the course of our dissertation assistance and statistical consulting with clients, we find that many are under the impression that systematic review and meta-analysis are one and the same; however, it’s more accurate to say that meta-analysis is a subset of systematic review. As pointed out above, not all systematic reviews include meta-analysis, but for the sake of quality, all meta-analysis studies should be conducted within studies that follow a systematic review design. This is because “systematic review” refers to the larger, transparent process detailed in the PRISMA guidelines, which are the elements that make systematic reviews such high-quality studies.  

Although systematic review refers to the research approach more broadly, meta-analysis is much more specific. Meta-analysis refers to statistical analysis techniques that are used to analyze the pooled results of multiple quantitative studies that have the same topic and focus (i.e., same population, independent variables, dependent variables). To conduct this statistical analysis, you need to transform the results of the studies in the sample so that it is possible to determine overall effect size, confidence intervals, and degree of heterogeneity. When assisting our statistical consulting clients, our statisticians also provide forest plots to visualize the results.

Steps for Conducting Systematic Review and Meta-Analysis

Again, the step-by-step, transparent process of the systematic review is what lends it such quality, and in our work with dissertation consulting clients, we progress through a series of steps to generate a viable idea for a study and to bring the study to fruition. For systematic reviews and meta-analyses, we generally move through the following phases:

Phase 1: Develop Review Question(s)

Systematic reviews are constructed around one or more specific research questions, and so the first step is to develop clear questions that can be addressed by the existing research. For example, Koh et al. (2019) built their review around the central question of how transformational leadership influences creativity. Sometimes our dissertation assistance clients have very specific ideas such as this for research questions, but other times they might have a general area they would like to investigate. In either case, it is important to do some checking around in research databases to make sure that you’re narrowing in on a feasible topic. 

As you sort through the literature on your topic, consider: Have there been multiple studies conducted on your topic of interest? What research questions were used in previous studies? Is there a recent systematic review on your topic already? These types of questions will help you to determine the need for a systematic review, whether there is sufficient research to include in a review study, and the possible research questions you might consider to guide your systematic review. 

For our dissertation consulting clients in health treatment fields, systematic reviews and meta-analyses typically address research questions that target a specific population, intervention, comparison, outcomes, and study design (PICOS). For example, Kvam et al. (2016) conducted a meta-analysis to examine exercise as a treatment for depression, and their PICOS dimensions were:

P = Adults with depression

I = Exercise

C = Placebo, antidepressant medication, treatment as usual, no treatment

O = Scores on validated depression instrument

S = Randomized control trials

Specifying the PICOS dimensions as you develop research questions ensures that your review study remains focused and that you can make valid comparisons between the outcomes of the studies you ultimately include in your sample. If you have your heart set on conducting meta-analysis as part of your review study, make sure statistical analysis pertinent to your PICOS dimensions was conducted in at least two existing studies in the literature.

Phase 2: Create Inclusion and Exclusion Criteria

Once you clearly define your research question(s), the next step is to establish inclusion and exclusion criteria for studies in your review. It is essential that the studies you include in your sample address your research question(s), and using your key variables or PICOS dimensions to decide which studies you keep for your dissertation sample and which you discard will help you to meet this requirement. In other words, you use your specific population and variables from your research question(s) to define the inclusion and exclusion criteria for your review. To provide an example, Kvam et al. (2016) set inclusion criteria around adult status and official diagnosis of unipolar depression, and they set exclusion criteria that barred use of studies including persons with seasonal forms of depression. 

Phase 3: Create Search Strategy and Locate Studies 

The next step is to jump into the literature to locate studies for your review. We always encourage our dissertation assistance clients to follow PRISMA guidelines, which suggest that a thorough search strategy be established before you begin the process of search and retrieval of studies. In your refinement and editing of your search strategy, keep in mind that it must be specific enough that your results could be reproduced by anyone else who followed the strategy. 

 If you choose to complete a systematic review for your dissertation, it will definitely help to keep in mind that the quality of the study hinges upon reducing bias at each step of the study. It may be tedious work, but your search strategy must be thorough enough to find every study on your topic—this offsets the risk of selection bias. So, your strategy will need to include an exhaustive list of keywords and index terms to ensure that no relevant studies are missed. This includes all terms that relate to your PICOS dimensions or key variables. For example, in their examination of transformational leadership’s effects cross-culturally, Crede et al. (2019) used the following search terms: transformational leadership in combination with organizational citizenship behavior, OCBs, volitional behaviors, contextual behaviors, and job performance. 

In addition to your search terms, your search strategy should specify the research databases you will search to find studies for your sample. We always suggest to our dissertation consulting clients that they also include a hand-search of journals that are especially relevant for their topic in their search strategy. It may also help to go over the references lists of studies that are particularly relevant to your review study. Once this search strategy is complete, you then follow it perfectly, documenting the results precisely (i.e., number of sources retrieved) for every step of your search. 

Phase 4: Select Studies and Extract Data

Now that you have found all of the studies that might fit in your review, you go back to the inclusion and exclusion criteria that you created in phase 2. Again, transparency of process is a core aspect of quality in a systematic review, so you will need to document the outcomes of your study selection process, with clear reasons given for exclusion of studies. When we assist our dissertation consulting clients with systematic reviews, we always recommend using the PRISMA flow diagram at this step. This diagram helps by offering a visual representation of the total number of studies retrieved, the number of studies excluded for different reasons, and the number of studies that will ultimately form the sample for your dissertation.

After selecting the final group of studies that will form the sample for your systematic review, you can get going on data extraction. You’re probably picking up on this pattern by now, but this step needs to be documented in great detail. This involves development of a table for tracking key information from each study, including authors, publication year, details regarding participants, the study’s design, and findings or outcomes. To prepare for statistical analysis, you can also code data from the sampled studies on the data extraction form. This process requires that you first develop a coding schema and pilot test it before completing the data extraction process—this is something that our statisticians can help with if you’d like!  

Phase 5: Critical Appraisal of Study Quality

A big part of the appeal of conducting a systematic review for many of our dissertation assistance clients is the prestige of publishing a study that is regarded as such high-quality evidence. Critically appraising the studies in your sample is an important step in attaining that quality. When considering the cumulative evidence on a particular topic, it is important to consider the strength of the individual studies that comprise that evidence. That is why, for this next step, you will need to assess risk of bias in each study in your sample, tracking this assessment on the table you created for data extraction. Make sure you also assess any potential bias that may affect the overall quality of the evidence, such as publication bias or selective reporting.

When we assist our dissertation consulting clients with this quality assessment, we follow established protocols or checklists to evaluate each study on various aspects of their study design, methods, and procedures. For example, the Cochrane risk of bias tool is well regarded by researchers and is often used to guide our quality evaluations of sample studies for systematic reviews. By answering a series of questions about each study, you arrive at a risk classification for each.

Based on your assessment of your sample studies’ quality, you can then confidently present a reliable account of the strength of the evidence in your review. This quality review also allows for assessment of heterogeneity across studies and will help you to determine whether meta-analysis is feasible for your dissertation. It may help to keep in mind that if you choose to exclude any studies due to risk of bias, this exclusion needs to be consistent with your a priori inclusion/exclusion criteria. For this reason, aspects of study quality (e.g., randomization to conditions) should always be considered when you establish your criteria for inclusion or exclusion.

Phase 6: Data Synthesis  

After you have extracted the data from the studies in your sample, you can finally get to the good part: data synthesis! This is where you use the data from the studies in your sample to answer your research question(s). This will always include a qualitative synthesis of your sample studies’ data, and if possible, it will include statistical analysis (i.e., meta-analysis) and in some cases qualitative analysis.

The qualitative synthesis portion of this step involves tabulation of study characteristics and outcomes for all included studies. In addition to presenting this in table form, you need to also discuss the different studies’ findings in narrative form to: (a) compare and contrast study characteristics and outcomes, (b) summarize the studies’ cumulative findings with regard to the research question(s), and (c) recognize any risks of bias and how these affect interpretation. If multiple studies using qualitative research methods are included, you may also conduct a secondary qualitative analysis of the pooled data.

Recall from earlier that meta-analysis is only possible if you have at least two studies in your sample that have the same or very similar variables. This is because the meta-analysis involves pooling the data of multiple studies to allow for a secondary statistical analysis. If meta-analysis is possible, then this is the phase where you would conduct this statistical analysis, including calculating overall effect sizes, confidence intervals, and degree of heterogeneity. As with your qualitative synthesis, you should write a narrative description of those statistical analysis results. Forest plots are also typically used, as they provide a visual depiction of the data.

Phase 7: Presenting Results

After completing your data synthesis, it’s time to write up the full research manuscript, and we definitely suggest following the PRISMA guidelines as we do with our dissertation assistance clients. Your introduction will provide the rationale and objectives of the study, and then you follow that up with a highly detailed description of the methods and procedures as you executed them (e.g., search criteria, search results, data extraction). Remember that transparency of process is a major contributor to the quality of systematic reviews, so be detailed! 

Following your extremely detailed recounting of your methods and procedures, present the results of your data synthesis in tables, graphs, and diagrams. As we discussed just above, you’ll need to write a narrative description of your results so that it’s clear to the reader how these results answer your research question(s). Also make sure that you provide a description of how you appraised the quality of the studies in your sample, with discussion of how any risk of bias identified might create limitations on inferences that can be made based on the results. Finally, you’ll need to provide a discussion of the study’s findings relative to the context of current research on the topic, which includes implications for practice and recommendations for future research to strengthen the related evidence.

Conclusion

If you’re like our more ambitious dissertation consulting clients, by now you’re probably excited at the prospect of using your dissertation to contribute to the official evidence base! As the previous steps probably suggest, this form of research is as challenging as it is satisfying, and there is nothing wrong with reaching out for help from the experts with a massive undertaking such as this. From what our dissertation assistance clients tell us, publishing a systematic review and meta-analysis is definitely something to be proud of! We’re here to assist with statistical consulting or more comprehensive help if you need it, and we hope your review study is a great success!

References

Crede, M., Jong, J., & Harms, P. (2019). The generalizability of transformational leadership across cultures: A meta-analysis. Journal of Managerial Psychology, 34(3), 139-155. https://doi.org/10.1108/JMP-11-2018-0506

Koh, D., Lee, K., & Joshi, K. (2019). Transformational leadership and creativity: A meta-analytic review and identification of an integrated model. Journal of Organizational Behavior, 40(6), 625-650. https://doi.org/10.1002/job.2335

Kvam, S., Kleppe, C. L., Nordhus, I. H., & Hovland, A. (2016). Exercise as a treatment for depression: A meta-analysis. Journal of Affective Disorders, 202, 67-86. https://doi.org/10.1016/j.jad.2016.03.063