Systematic Reviews in Sociological Research

Abstract

Systematic review is a rigorous selection of a set of studies or state-of-the-art research relevant to a specific research question or to a specific practice or intervention. Review follows a carefully designed methodological framework of analysis to eliminate bias and error. Systematic reviews were first created in the biomedical field and adopted into the social sciences decades later. These are particularly helpful in the field of sociology, although they are also prone to problems that hinder implementation in some branches of sociology. Nevertheless, measures can be taken to limit these problems and help develop a successful systematic review in sociology.

Overview

A systematic review, as its name indicates, is the methodological, orderly, and standardized overview of primary research done or studies published in a discipline or field. However, to review all that has been investigated and published in any discipline would be too broad and unwieldy a task; therefore, researchers must focus narrowly on a research question that he or she wishes to investigate. A systematic review is similar to a literature review, yet with significant differences. A literature review summarizes the knowledge on a specific topic in a more subjective manner than a systematic review, without it being less scholarly. A literature review, for instance, can include a wide scope of papers, as large as a hundred or more. Moreover, it does not need to be strictly guided by a research question. A systematic review, on the other hand, is guided by a research question, which serves to narrow down the scope of publications or trials under review. Being more specific, systematic reviews usually review a much smaller number of studies and these, in turn, are often based on primary research.

To undertake a systematic review, it is important to understand the difference between primary and secondary research and what a research question is. Primary research is based on direct testimony or evidence—for instance, on testimonies from research participants who have witnessed or experienced a specific event, or from primary sources such as original documents, artifacts, physical samples or data collected directly from sources. Secondary research, on the other hand, is a process of review on the scientific and scholarly literature available, which may be qualitative or quantitative. The final purpose is to synthesis the research reviewed in order to find a solution to a problem or answer to a question. Secondary research also serves to guide the development of social processes, such as policies or interventions.

The research question is a crucial part of a systematic review. Before the start of a research project, investigators will identify a question that will guide the review and determine the methodology and resources required. In social sciences, such as sociology, psychology, and communications, research questions may combine two main types of methods: quantitative and qualitative.

Quantitative data is based on numerical or countable information, while qualitative data is information that is not numeric, even though it may be made quantifiable. Qualitative information, for instance, is based on context or different opinions about something. A researcher may ask a group of people how they feel about a specific product or event and the answers tabulated according to different categories to find out how many people feel positively or negatively about the issue. The numeric data from an investigation is often considered primary research in the social sciences and may include data from censuses, surveys, and so on. The combination of both these methods, quantitative and qualitative, is more common to the social sciences, for instance, than the humanities or the hard sciences.

A systematic review in the social sciences is, then, an overview of research done on a specific topic, delineated in the research question. After formulating the research question, the reviewer will select, evaluate, and finally synthesize—or combine and summarize—all the research relevant to the research question. A systematic review falls within the purview of what is known as “high-level research”, which is a narrow summary or general overview, presentation, and explanation of a specific subject. As such, it focuses on a few specific standardized points in common among all the research projects overviewed, which relate directly to the research question, obviating details that may be redundant or irrelevant.

A systematic review may be done by an individual or by an expert panel. It may include research that has been published as well as unpublished research. In any case, systematic reviews are considered an objective way of organizing knowledge. In fact, systematic reviews originated in the medical sciences and were later adopted into the social sciences as a way to analyze, incorporate, and standardize in sociology the evidence-based practices of the biomedical field.

Some experts argue that a systematic review is basically a literature review guided by a research question, aimed at finding and synthesizing the best or most high-quality research relevant to that question. In other words, literature reviews assess the “state of the art” for a topic of the humanities, but a systematic review takes it further and more formally, following well-delineated criteria. Both types of review, however, create a synthesis to account for the current state of scholarly or scientific knowledge about the research question. While the literature review provides a more general overview, then, a systematic review seeks to develop a deep and narrow overview on one specific question.

Systematic reviews have been adopted in the social sciences, in all disciplines that collect and analyze data. Sociology has many branches in which the collection and assessment of high-quality research is important, because the discipline often develops and applies methods for application and practice in social fields that range from public health and policy, to crime and delinquency, social work, collective behaviors, management, welfare, and social security.

Systematic reviews in sociology became to be incorporated in the 1990s and tended to focus mainly on methods of primary research, although it later opened to other types of research. The move reflected the need for more evidence-based methods and practices; in time, the use of systematic reviews expanded to other areas of the social sciences. Systematic reviews prevent duplication of research and help illuminate the quality of current research, enabling a deeper understanding of an issue by comparing results and methods from new research. Systematic reviews have bettered the quality of policies and practices in the social services. Critics, however, often question the aptness of applying systematic review methods to sociology and other social sciences, arguing that it would be more appropriate to develop new methodologies for these disciplines.

There are three different approaches to systematic reviews: Evidence-based, theory-driven, and mixed methods. Evidence-based approaches are the original method and derive from those used in the medical field, where the interest is usually centered on synthesizing data. Proponents of theory-driven approaches consider that evidence-based approaches offer a “one size fits all” system too limited for the social sciences; in response, alternative review methods were developed with a different perspective about how knowledge accumulates. For example, a systematic review may develop a theory that seeks to understand the impact of different factors based on variations between individuals or groups. Such systematic reviews, proponents argue, look at data based on context and have wider applications. They are of greater benefit for policymakers than evidence-based reviews and are helpful to users and practitioners of different social services. This system can be applied to diverse situations, such as understanding why individuals use—or not—some services, prefer some products over others, or identify new needs and potential resources. Moreover, theory-driven approaches are more flexible and better able to adapt to contextual situations that require cultural sensitivity and care.

Finally, a systematic review in sociology may have a mixed approach, that is, combine methods from evidence-based and theory-driven systems. Although all systematic reviews are guided by a research question, the latter has even greater impact on the applicability of this approach. In other words, a mixed approach is not suitable for all research questions. Among its benefits, however, is that it provides higher adaptability to different data and useful insights, illuminating connections that might be overlooked using just one of the previous methods.

Applications

Because they tend to examine broad social issues, sociologists often confront a problem of excess information or information overload. In fact, most researchers must confront, at some point, the realization that there is more information than what human capabilities can manage or may be viably include in a single research project. Much of the available information may be contradictory, noncritical, insufficiently reviewed, use inconsistent methods, lack an explanation of how researchers developed their theory or methodology, show bias, and so on.

A systematic review helps prevent these problems; furthermore, a proper systematic review must make explicit the methods used in the review so that others can assess them. There are several steps to developing a systematic review.

The first stage is identifying the research question that will guide the review. The question leads the reviewer (or panel of reviewers) to design the protocol, that is, the standards and procedure that will be followed to select, gather, and analyze the research papers to be reviewed. Adhering to a formal protocol helps avoid falling into bias and error. After these elements have been laid down and described, a search is carried out for high-quality research relevant to the research question, which can be done through databases, bibliographies, scholarly or scientific articles, specialized libraries, and so on. After carefully selecting a set of publications to be used, researchers analyze each one by one, following the same checklist and rigorous criteria, in order to decide which qualify for the review.

Once the publications are selected, they are read and the pertinent information or data retrieved and tabulated. Each is described and, if necessary, depicted through graphs, diagrams, or tables. The latter make for a clearer understanding of the comparison process.

The next stage is to compile and summarize the findings. This is known as synthesis. A synthesis draws conclusions from the results and must be in direct relation to the research question. By synthesizing a large body of empirical research, a systematic review becomes a very useful instrument for the practice and implementation of social policies, interventions, and services. It makes credible research accessible to individuals and institutions who do not have the means or resources to access and study large bodies of research, and the review can be used to analyze trends, correlations, variations, rates, preferences, and many other social phenomena. Moreover, its rigorous systematic framework helps minimize error and bias at every step of the way, ensuring that results, conclusions, and recommendations are comprehensive and sound.

For research heavy on quantitative data, experts recommend the development of a meta-analysis, that is, a statistical process that pools all the data. Data pooling offers many benefits, such as providing access to diverse data, aggregating a larger amount of data which, in turn, leads to more accurate results and, sometimes, it makes access to data more affordable.

Another system used for systematic reviews in social sciences, especially for quantitative reviews, is a meta-analysis. A meta-analysis combines or pools the data from the set of articles analyzed, with the purpose of doing a statistical analysis. It is a procedure specific to systematic reviews, even though not all systematic reviews include a meta-analysis. In other words, a systematic review is the whole process of selecting, analyzing, and synthesizing all the evidence available, and meta-analysis is a subset of the review linked to the pooled data gathered from the systematic review. A meta-analysis is common to the health field and social sciences and used when the numbers of subjects and/or the results are larger than usual; or when they present greater variation. However, experts warn that a meta-analysis is not appropriate for all types of systematic reviews, especially when combining data from very different types of sources.

Issues

Meta-analyses have become common in the social and biomedical sciences. However, some challenge the validity of meta-analyses, arguing that combining data from studies that are too dissimilar produce biased or unsound results. Others argue that many new applications fall under the umbrella term of meta-analysis and call for a careful recalibration of what methodologies or strategies merit to be considered a meta-analysis. In fact, it is difficult even for seasoned researchers to keep abreast of all new technologies and methodological trends, as well as the latest findings, especially when disciplines intersect with others, such as in some interdisciplinary projects. There is currently some lack of consensus on how to solve some of the issues that have arisen in the arena of systematic reviews in sociological research.

One of the critiques against qualitative studies in social sciences is that they are less structured than hard sciences and biomedical disciplines; they argue that social science research is more prone to subjectivity, that is, to personal bias or theoretical and methodological error. In fact, bias and error are risks common to all fields of study. Moreover, there are other systems besides meta-analysis that can be used to deal with error and bias.

Some, for instance, recommend the use of mixed methods when appropriate. Others recommend adhering to protocols, which helps limit bias and error, and reviewing findings with peers and participants, using multiple people to interpret the data and verify data sources—called triangulation. For these to be implemented, systematic research review designers need to ensure they have the necessary the funds and resources to form a proper research team.

One of the most important ways to avoid the pitfalls of error and bias is to do the necessary legwork before beginning the systematic review. Reviews often fall into bias when the data analysis strategy is created after gathering the information, or when the reviewer has selected a set of studies inadequate for the research question, leading the reviewer to survey the wrong data. Moreover, a reviewer may have selected the wrong methodology for the research question. All of these are steps that need to be clarified and delineated before commencing a systematic review, particularly in the social sciences.

According to experts, sociologists should adopt systematic reviews because they map out the current state of research and knowledge in the field and can help lay out overall standards for research programs. This has applicability in the theoretical as well as methodological arenas. Systematic reviews serve the common good by making information available to policymakers in the fields of social care and public health programs and can be adapted to match the needs of institutions, communities, as well as individual practitioners.

Terms & Concepts

Bias: An inclination, partiality, or prejudice in favor or against one person, result, thing, or group.

Evidence-Based: Practices based on empirical methods and objective evidence.

Qualitative: The quality, factors, or context of something that is not numerically quantifiable.

Quantitative: Something that is measurable by its quantity or numbers.

Synthesis: The composition and combination, summarized, of ideas to create a new system, process, or theory.

Systemic Causes: The inherent and implicit roots of a problem, which are embedded in a whole system, rather than dependent upon specific, isolated factors.

Bibliography

Caird, J., Sutcliffe, K., Kwan, I., Dickson, K., & Thomas, J. (2015). Mediating policy-relevant evidence at speed: Are systematic reviews of systematic reviews a useful approach? Evidence & Policy: A Journal of Research, Debate & Practice, 11(1), 81–97. Retrieved January 1, 2019 from EBSCO Online Database Sociology Source Ultimate. http://search.ebscohost.com/login.aspx?direct=true&db=sxi&AN=100635423&site=ehost-live

Campbell, A., Taylor, B., Bates, J., & O’Connor-Bones, U. (2018). Developing and applying a protocol for a systematic review in the social sciences. New Review of Academic Librarianship, 24(1), 1–22.

Cornish, F. (2015). Evidence synthesis in international development: A critique of systematic reviews and a pragmatist alternative. Anthropology & Medicine, 22(3), 263–277. Retrieved January 1, 2019 from EBSCO Online Database Sociology Source Ultimate. http://search.ebscohost.com/login.aspx?direct=true&db=sxi&AN=112293735&site=ehost-live

Farrington, D. P., & Jolliffe, D. (2017). Special issue on systematic reviews in criminology. Aggression & Violent Behavior, 33, 1–3. Retrieved January 1, 2019 from EBSCO Online Database Sociology Source Ultimate. http://search.ebscohost.com/login.aspx?direct=true&db=sxi&AN=121637447&site=ehost-live

Halter, M., Boiko, O., Pelone, F., Beighton, C., Harris, R., Gale, J., & Drennan, V. (2017). The determinants and consequences of adult nursing staff turnover: A systematic review of systematic reviews. BMC Health Services Research, 17, 1–20.

Mackenzie, K., Such, E., Norman, P., & Goyder, E. (2018). The development, implementation and evaluation of interventions to reduce workplace sitting: A qualitative systematic review and evidence-based operational framework. BMC Public Health, 19(1), n.p.

Papaioannou, D., Sutton, A., Carroll, C., Booth, A., & Wong, R. (2010). Literature searching for social science systematic reviews: Consideration of a range of search techniques. Health Information & Libraries Journal, 27(2), 114–122.

Pickering, C., Grignon, J., Steven, R., Guitart, D., & Byrne, J. (2015). Publishing not perishing: How research students transition from novice to knowledgeable using systematic quantitative literature reviews. Studies in Higher Education, 40(10), 1756–1769.

Rigterink, A. S., & Schomerus, M. (2016). Probing for proof, plausibility, principle and possibility: A new approach to assessing evidence in a systematic evidence review. Development Policy Review, 34(1), 5–27.

Victor, L. (2008). Systematic reviewing in the social sciences: Outcome and explanation. Enquire 1(1), 32–46.

Suggested Reading

Boland, A., Cherry, G., & Dickson, R. (2017). Doing a systematic review: A student’s guide. Thousand Oaks, CA: Sage Publications.

Gough, D., Oliver, S., & Thomas, J. (2017). An Introduction to Systematic Reviews. Thousand Oaks, CA: Sage Publications.

Maldonado Castañeda, O. J. (2019). (Un)contested evidence: scientific literature, systematic reviews and the politics of evidence in the introduction of HPV vaccines in Colombia. Sociology of Health & Illness, 41(1), 81–94. Retrieved January 1, 2019 from EBSCO Online Database Sociology Source Ultimate. http://search.ebscohost.com/login.aspx?direct=true&db=sxi&AN=133988463&site=ehost-live

Miyahara, M., Lagisz, M., Nakagawa, S., & Henderson, S. E. (2017). A narrative meta-review of a series of systematic and meta-analytic reviews on the intervention outcome for children with developmental co-ordination disorder. Child: Care, Health & Development, 43(5), 733–742. Retrieved January 1, 2019 from EBSCO Online Database Sociology Source Ultimate. http://search.ebscohost.com/login.aspx?direct=true&db=sxi&AN=124504983&site=ehost-live

Neimann Rasmussen, L., & Montgomery, P. (2018). The prevalence of and factors associated with inclusion of non-English language studies in Campbell systematic reviews: A survey and meta-epidemiological study. Systematic Reviews, 7(1), n.p.

Vaidya, N., Thota, A. B., Proia, K. K., Jamieson, S., Mercer, S. L., Elder, R. W., … Zaza, S. (2017). Practice-based evidence in community guide systematic reviews. American Journal of Public Health, 107(3), 413–420. Retrieved January 1, 2019 from EBSCO Online Database Sociology Source Ultimate. http://search.ebscohost.com/login.aspx?direct=true&db=sxi&AN=121193365&site=ehost-live