Survey Research in Education

Abstract

Survey research aims to provide a comprehensive, representative summary of specific characteristics, beliefs, attitudes, opinions, or behavior patterns of a population. There are a number of different methods of collecting survey information, such as in-person, phone, and email questionnaires and interviews. Surveys are not simply exhaustive collections of statistics about specific traits of a population. Surveys are always conducted in response to particular research questions, generally qualitative in nature, and thus aim to collect only information that might be relevant to the study at hand. Survey study design includes the articulation of the research question, of the scope of the study, and of the targeted population, the subdivision into themes or sub-questions of the original problem, the selection of survey items, pilot testing, and finally, administration, collection of information, and data analysis. Survey research methodology is the most commonly used approach in the social sciences, including education research, accounting for about 70% of all studies.

Overview

Introduction to Survey Research. Survey research aims to provide a comprehensive, representative summary of specific characteristics, beliefs, attitudes, opinions, or behavior patterns of a population. There are a number of different methods of collecting survey information, such as in-person, phone, and e-mail questionnaires and interviews. Surveys have been used for over two thousand years (Di Iorio, 2005); the first recorded use was around 150 BCE, when Hipparchus surveyed the night sky and recorded the positions of stars as well as their relative brightness as measured by a six-point scale. Today, surveys may be found in all aspects of public life: they are used to collect information about voting practices and political preferences, about satisfaction with a particular product or experience, and about personal habits and behaviors, to name but a few. Survey research methodology is the most commonly used approach in the social sciences, accounting for about 70% of all studies (Lodico et al, 2006).

Surveys are not simply exhaustive collections of statistics about specific traits of a population. Surveys are always conducted in response to particular research questions, generally qualitative in nature, and thus aim to collect only information that might be relevant to the study at hand. In educational research, surveys have been used to gather information on test scores in order to identify patterns of low achievement, to form impressions of new teachers' attitudes toward teaching, and to identify trends in student interests, among many other applications. For example, Mellard, Patterson, and Prewett (2007) used survey research to collect information about adult students' reading patterns and about students' traits in order to identify ways to encourage adults to read. Richardson, Slater, and Wilson (2007) used survey research to collect information on university students in the UK.

Use in Quantitative Research. Surveys, though primarily quantitative in design and implementation, are always developed in response to a qualitative inquiry, unlike other forms of quantitative research (Lodico et al, 2006). Surveys are not experimental, so data is not collected to test a hypothesis, but rather, to describe—both qualitatively and quantitatively, but primarily quantitatively—existing conditions and attitudes. Surveys can take the form of questionnaires or of interviews, though neither format implies specific types of questions: interviews might require survey participants to choose their answers between previously determined categories, while questionnaires may ask for open ended responses. Di Iorio (2005) suggests using open ended questions for gathering information of a personal or sensitive nature, and multiple choice questions for easily quantifiable information. Sometimes both questionnaires and interviews are used within a single survey study. For example, Mamlock-Naamam et al (2007) evaluated a workshop designed to encourage science teachers to create their own curricula through questionnaires—to get a rough quantitative approximation to teacher beliefs—as well as through interviews—to gauge teachers' initial reactions and to probe quantitative answers in more detail.

Applications

Designing a Survey. The first step in the design of a survey study is the articulation of the research question, of the scope of the study, and of the targeted population. Next, the research question is subdivided into themes or sub-questions that reflect different dimensions of the problem. Survey items are then formulated and selected using scaling, a method of generating questions and statements that adequately measure—quantitatively—the qualitative aspects of the attitude or belief under investigation.

After survey items are selected, and after the questionnaire or interview format are completed, a survey must undergo pilot testing before it can be used to collect and generalize information about the targeted population.

Scaling. Because surveys answer qualitative questions through an analysis of correlations between quantitative measurements, scaling—the process of transforming qualitative statements into meaningful quantitative measures—is critical to the design of a survey study. Surveys may employ open-ended questions; however, these are rarely used outside of allowances for "additional" or "other" comments at the end, as these types of responses are not easily processed or analyzed (Di Iorio, 2005).

Scaling is not simply a process of assigning numerical values to qualitative statements, such as a score of 5 for "highly agree" or of 1 for "highly disagree." Scaling is the intricate, nuanced technique of developing qualitative statements that gauge respondents' beliefs about a particular issue—thus scaling research always begins with the clear identification of a research question or aim. Experimental research is then conducted to determine appropriate qualitative statements for use on surveys that adequately gauge the desired features in the population of interest. There are several formalized conceptual and experimental frameworks commonly used in scaling research; the most relevant to educational researchers are:

• Thurstone Scaling,
• Likert Scaling, and
• Guttman Scaling.

All of these methods are unidimensional—they only measure one aspect, or dimension, of a variable. For example, if the study aims to gauge the attitudes of various constituencies in a particular school district toward school choice initiatives, a unidimensional scale might be used to measure attitudes as (very and somewhat) favorable or unfavorable (Lodico et al, 2006).

Thurstone Scaling. Louis Leon Thurstone (1887–1955) was a pioneer in psychometrics, the discipline concerned with measurement of psychological traits such as intelligence, personality, values, and attitudes. Thurstone developed three scaling methods: those of equal-appearing, paired comparisons, and successive intervals. While Thurstone scaling is rare today (Di Iorio, 2005), it presented a revolutionary breakthrough in measurement in the early 1900s and forms the foundation for thinking about all other methods of scaling (Allen, 1994).

After the identification of the research question or aim, Thurstone scaling calls for the generation of statements that gauge beliefs about the content matter of the study; a large pool, of about 70+ statements, is appropriate. The questions generated should be suited in tone, language, and content for the targeted population, and should be similar in structure (Lodico et al, 2006).

A team of experts in the subject matter of the survey should then be called upon to rate these items on a scale that may have any number of "points"—a five point scale, for example, would allow one to rate a statement on a scale from 1 to 5. A Thurstone scale may have an odd or even number of points; an odd number of points allows for a "neutral" answer, while an even number of points does not (see Figure 1).

At this phase of the scaling, the experts do not rate the items to reflect their own opinions, but on how much each statement reflects a favorable or unfavorable attitude. For example, in a study to gauge attitudes on school choice, generated statements might include "School vouchers would give each child an opportunity to find an environment s/he can best learn in" and "School choice would lead to large discrepancies between those who have the resources to locate, apply to, and travel to another school and those who do not." The first of these items would be rated as favorable (very, moderately, somewhat), while the second would be rated as unfavorable.

After each item has been rated by several content-matter experts, scores of all statements are tallied, tabulated, and ordered. The score of a statement is composed of three parts: the median and the first and third quartile. The median is the rating below which 50% of responses and above which 50% of responses fall. The first quartile is the rating below which 25% of responses fall, while the third quartile is the rating below which 75% fall. For each item, the difference between the inter-quartile range, or the difference between the first and third quartile ratings, is computed (see Figure 2).

Thurstone scoring for the statement "School vouchers give each child an opportunity to find an environment s/he can best learn in." If there are 8 judges, and they rate this statement as shown above, the statement will have a score of: 4 for the first quartile, 4.5 for the median, 5 for the second quartile, and 1 for the interquartile range.

Statements for the survey are then chosen; items picked should equally represent all medians, so that the survey consists of an equal number of statements that reflect favorable and unfavorable attitudes. When electing between items with the same median score, the ones with the smallest inter-quartile range should be chosen, as these reflect statements that have the least variability between respondents and are thus the most reliable.

The Thurstone method described above is that of equal-appearing scaling; however, Thurstone also proposed others that are commonly used in the social sciences: paired comparison and successive interval scaling. In the previous example, the subject matter experts rate each item independently (hence, each is "equal-appearing"), while in paired comparison (Allen, 1994) and successive interval (Adams & Messick, 1958) scaling items are compared with each other before any score is given, and are then chosen among based on the degree to which each reflects the desired response (favorable or unfavorable, for example).

Likert (Summative) Scaling. Toward the mid part of the twentieth century, as scientific methodologies became increasingly employed in the study of social and psychological aspects of individuals and communities, the field of psychometrics gained momentum, attracting more researchers such as Rensis Likert (1903–1981), an educator and psychologist. Likert, inspired by Thurstone, developed what became the single most used scaling method in the social sciences. Because of its popularity, Likert scaling is one of the most well developed methods in psychometrics, with the greatest capacity to measure nuance and to detect inconsistencies (Lodico et al., 2006).

The process of Likert scaling is similar to that of Thurstone scaling and begins the same way: a question and the population are defined, a large set of statements specifically related to the matter of the study are generated, and experts are asked to rate these items on a point-scale measuring the degree to which a statement reflects a positive or negative attitude. However, unlike a Thurstone scale, which may rely on several unrelated questions to measure attitudes, a Likert scale is summative and must consist of a series of items, the responses to which must be calibrated through a relatively equal distribution of gradations between ratings (DiIorio, 2005). For example, in Figure 3(b), the respondent is asked how strongly s/he disagrees with the statement "I am generally comfortable around a lot of people." These qualitative statements are given, as well as symbolic values representing them. Calibration is the process of choosing statements from the previously generated pool to reflect equal gradations between ratings, such that the symbolic values translate directly (linearly) to the numeric values used to rate the survey.

The Thurstone method had been previously criticized for not calibrating the rating scale to the symbolic scale given to the participants. For example, the statement "I believe all people have a right to life" would be more likely to be agreed to by most people, so responding "Agree" in this case is not of the same magnitude as responding "Disagree"—those who disagree are farther from the norm (Allen, 1994).

A Likert scale is cumulative and must add the ratings between all items, because items are correlated. Cumulative scales allow for more nuanced descriptions than a simple scale of independent items can offer (Lodico et al., 2006). For example, a scale used to measure intro/extroversion might ask a series of related questions such as "I have a hard time talking with people I do not know," "I enjoy going to large parties," or "I am generally comfortable around a lot of people." These statements, though worded differently, all measure the degree to which an individual is "introverted" or "extroverted," and the degree to which one agrees or disagrees with these statements as considered together (summatively) reflects the participant's personality type.

In Likert scaling, the method of selecting between the generated items is more sophisticated than in Thurstone scaling and consists essentially of a comparison and determination of correlations between all items used to measure a particular attribute. Because the scale is cumulative, the correlations between items are reflected in the rating and thus must be adequately designed and accounted for.

Guttman Scaling. Louis Guttman (1916–1987) developed a scale that is cumulative (as opposed to summative). Guttman survey responses have a multiple-choice implementation, where the agreed upon response also implies other previously mentioned responses (see Figure 4).

Guttman scales are appropriate for obtaining information that is already quantitative, such as is depicted in 4(a). If one answers "less than 7 miles per hour," the researcher can imply that the participant can also run less than 5 or less than 6 miles per hour. Using Guttman scales for ranking qualitative statements is more difficult and involves many judgment calls (Robinson, 1973). Example 4(b) attempts to measure the extent to which a parent would allow a child to socialize. The example assumes that if a parent would allow a child to attend a dance at school, she or he would also allow the child to attend a party at a friend's house. Choosing an answer choice implies one agrees with the previous statements.

Sampling the Population. The ideal survey would be a census—a collection of information about all members of a population (Di Iorio, 2005). If the population is small, say, all students in the fifth grade or the group of all teachers at an elementary school, a census might be possible; in this case, the sample would be identical to the population. However, getting information from all members of a community or group is often difficult, so a technique through which a representative sample is selected—sampling—is necessary.

The sampling methods chosen and parameters defined depend on a number of population features. Sampling may be random, stratified, convenient, or may take on a number of other forms. In random sampling methods, each individual in the population has an equal chance of being chosen. In stratified sampling, the sample is chosen to represent all sub-groups of a particular type—such as race, age, gender—of the population. Ideally, a stratified sample reflects similar proportions of sub-groups members as are present in the targeted population. Convenience sampling, a method that is not recommended in most cases, is a process of choosing the most accessible subgroup of the population (Lodico et al, 2006).

Pilot Testing. Pilot testing a survey is not always possible by study design. For example, pilot testing might not be feasible if a study is made on reactions or attitudes toward a particular event. If a researcher wants to survey the experiences of parents at a back-to-school night, they have only one chance to administer the survey. In cases like this, it is useful to revise the survey from event to event in order to account for previously observed discrepancies or errors.

When it is possible, rigorous preliminary testing is recommended. Collins (2003) states that pilot testing is necessary to determine whether the ways in which respondents understand questions are relatively similar across the group, whether the information asked for is easily accessible to participants, to ensure that the phrasing of survey items is such that responses correlate to what the study intends to measure. When interviews are used to administer a survey, Collins (2003) also recommends not altering the exact phrasing of the previously agreed upon questions to ensure that delivery is not a factor in differences between individual responses. Delivery may include factors that are not related to the wording of the question, however, such as the mood or temper of the administrator. Errors are thus introduced into interview studies—essentially, through personal interaction between the administrator of the survey and the study participant.

Survey Considerations. There are several considerations that must be appraised throughout a survey study: validity, errors, and ethics are some of the most critical.

Validity. Validity refers to the extent to which the constructed instrument measures what it was intended to in its proper scope, and to which results are stable across different samples of the population. There are, loosely, four types of validity conceptualizations:

• Statistical conclusion,
• Internal,
• Construct, and
• External validity (Calder et al, 1982).

Statistical conclusion validity refers to the extent to which there are variations in the data that cannot be explained by the variables of the study. Internal validity refers to the extent to which two variables can be said to be causally connected (Calder et al, 1982). Construct validity refers to whether or not quantitative measurements chosen to reflect qualitative aspects of the attitude under examination adequately do so (Peter, 1981). External validity refers to the extent to which survey results are generalizable to the targeted population. Generalizability takes on one of two primary forms: that of generalizing results to represent the entirety of a population, or that of generalizing results to understand other populations, settings, or contexts. The latter form is used predominately in comparative studies.

External validity is primarily affected by the methods used to select samples used in the study. Adequate sample selection ensures that results can be generalized to the population or comparatively to other populations.

In addition to the previously mentioned validity forms, some survey researchers suggest another: content validity (Lodico, 2006). Content validity refers to the extent to which the content of the items on the survey adequately cover the breadth and depth of the problem in question. Content validity has two dimensions: item (depth) and sampling (breadth). The item consideration is a measurement of the logicality and inclusivity of the sequence and organization of items, and the sampling consideration is a measure of the extent to which the items chosen best reflect the idea they are trying to convey (as opposed to other items expressing the idea in different ways).

Errors. Errors are events or conditions of the study that negatively affect the validity of the study. There are several forms of errors, including bias and measurement, random, and systematic errors. Bias refers to the inherent inclinations for particular views or constructs that affect survey design. In the early twentieth century, for example, discriminatory ideals drove the development of intelligence tests (Birgham, 1923; Vincent, 1991). Measurement errors occur when the instrument, or survey, is not adequately calibrated and therefore does not measure what it aims to accurately. Random errors are those that occur sporadically and cannot be explained by existing constructs and variables. Systematic errors occur when certain results are constantly distorted (Di Iorio, 2005).

Ethics. Because surveying requires researchers to work with a large number of participants and to collect personal or private information about their attitudes and beliefs, ethical considerations are of central importance in survey research. Participants must be informed of all researcher intentions and study methods. Consent must be obtained before a survey can be administered—an individual cannot be forced to participate in a survey. Issues of confidentiality and anonymity must be clarified, and a trust established that might encourage participants to be more open and cooperative with the study, a critical factor in the collection of accurate information on personal attitudes (Lodico et al, 2006).

Survey researchers are sometimes involved in situations in which ethical considerations conflict with what they think would be best in the situation. The Youth Risk Behavior Study, for example, was used to collect information on sensitive topics related to students' lives and was analyzed by many researchers for various trends. Edwards and Magel (2007) used the Behavior Survey to study fitness and nutrition habits among students in a Midwestern school district; Niyonsenga and Hlaing (2007) used it to examine the differences in risky sexual behavior engagement between females and males; and Howard et al. (2007) studied patterns of abuse in girls who experienced date violence. If a researcher uncovers that a student is engaging in harmful behavior, s/he is not required to report it to authorities, unless it is a direct threat to the life of an individual. However, the researcher might be compelled to want to help the student. In this case, personal discretion is advised. Considerations might include the relationship between the researcher and the student, between the student and the family and community network, or the students' traits.

Viewpoints

Criticisms of the Survey Method. Critiques of the survey method center primarily around the scaling process (Furfey & Daly, 1937). Scaling a measurement of attitude is essentially a process of factor analysis, or of quantifying qualitative statements. Factor analysis extracts essential features of attitudes and beliefs and molds all nuances into small sets of ratings.

Solutions offered to these critiques of the survey method include suggestions to limit survey use to the measurement of specific attitudes, or of attitudes toward specific events or ideas (Ostrom, 1971–1972). Limiting the scope of the survey through this process of clearly defining a precise point of interest improves its validity. Despite the critiques, survey research continues to permeate research and to offer solutions to many forms of questions in the social sciences.

Terms & Concepts

Cumulative Scale: A scale in which choosing an answer implies the respondent agrees with all answers preceding the one chosen.

Factor Analysis: A method of analyzing qualitative information and extracting essential features that can be quantized and used to represent the information.

Generalizability: The extent to which results can be generalized to represent the entirety of a population, or to understand other populations, settings, or contexts.

Scaling: A method of generating questions and statements that adequately measure—quantitatively—the qualitative aspects of the attitude or belief under investigation.

Scaling Gradation: The relative difference between scale ratings.

Summative Scale: A scale that is rated by adding the scores of all items together; summative scales take into account correlations between the different items.

Validity: The extent to which the constructed instrument measures what it was intended for in its proper scope, and to which results are stable across different samples of the population. There are loosely five types of validity conceptualizations: statistical conclusion, internal, construct, content, and external validity.

Bibliography

Adams, E., & Messick, S. (1958). An axiomatic formulation and generalization of successive intervals scaling. Psychometrika, 23, 355–368.

Allen, M. (1994). Thurstone's method of paired comparisons: Review of an old but still-useful measurement protocol. Antonio, TX: Annual Meeting of the Southwest Educational Research Association. (ERIC Document Reproduction Service No. ED367704). Retrieved December 14, 2007 from EBSCO Online Education Research Database. http://www.eric.ed.gov/ERICDocs/data/ericdocs2sql/content%5Fstorage%5F01/0000019b/80/15/5c/c2.pdf

Bangura, A. (1992). The limitations of survey research methods in assessing the problem of minority student retention in higher education. Lewiston, NJ: Mellen University Press.

Brigham, C. (1923). A study of American intelligence. Princeton: Princeton University Press.

Calder, B., Phillips, L., & Tybout, A. (1982). The concept of external validity. The Journal of Consumer Research, 49 , 240–244.

Collins, D. (2003). Pretesting survey instruments: An overview of cognitive methods. Quality of Life Research, 12, 229–238. Retrieved November 10, 2007 from EBSCO Online Database Academic Search Premier. http://search.ebscohost.com/ login.aspx?direct=true&db=aph&AN=10838539&loginpage=Login.asp&site=ehost-live

Croasmun, J. T., & Ostrom, L. (2011). Using Likert-type scales in the social sciences. MPAEA Journal of Adult Education, 40, 19–22. Retrieved December 6, 2013 from EBSCO Online Database Education Research Complete. http://search.ebscohost.com/login.aspx?direct=true&db=ehh&AN=85944039&site=ehost-live

Di Iorio, C. (2005). Measurement in health behavior: methods for research and education. San Francisco, CA: Jossey-Bass.

Edwards, J., & Magel, R. (2007). Use of the youth risk behavior survey to monitor trends for nutrition and physical activity in a midwest city school district. Journal of School Health, 77, 351–358.

Ericksen, S. (1948). A skeptical note on the use of attitude scales toward war. Journal of Social Psychology, 27 , 79–90. Retrieved November 10, 2007 from EBSCO Online Database Education Research Complete. http://search.ebscohost.com/login.aspx?direct=true&db=ehh&AN=16510116&loginpage=Login.asp&site=ehost-live

Fincham, J. E., & Draugalis, J. R. (2013). The importance of survey research standards. American Journal of Pharmaceutical Education, 77 , 1–4. Retrieved December 6, 2013 from EBSCO Online Database Education Research Complete. http://search.ebscohost.com/login.aspx?direct=true&db=ehh&AN=87614113&site=ehost-live

Fulton, B. R. (2018). Organizations and survey research. Sociological Methods & Research, 47(2), 240–276. Retrieved October 25, 2018 form EBSCO Online Database Education Source. http://search.ebscohost.com/login.aspx?direct=true&db=eue&AN=128168106&site=ehost-live&scope=site

Furfey, P., & Daly, J. (1937). A criticism of factor analysis as a technique of social research. American Sociological Review, 2, 178–186.

Garvey, J. C. (2017). Considerations for queer as a sexual identity classification in education survey research. Journal of College Student Development, 58(7), 1113–1118. Retrieved October 25, 2018 from EBSCO Online Database Education Source. http://search.ebscohost.com/login.aspx?direct=true&db=eue&AN=125817938&site=ehost-live&scope=site

Hansen, K. (2007). The effects of incentives, interview length, and interviewer characteristics on response rates in a CATI-Study. International Journal of Public Opinion Research, 19, 112–121.

Lodico, M., Spaulding, D., & Voegtle, K. (2006). Methods in educational research: From theory to practice. San Francisco, CA: Jossey-Bass.

Mamlok-Naaman, R., Hofstein, A., & Penick, J. Involving science teachers in the development and implementation of assessment tools for "science for all" type curricula. Journal of Science Teacher Education, 18, 497–524.

Mellard, D., Patterson, M., & Prewett, S. (2007). Reading practices among adult education participants. Reading Research Quarterly, 42, 188–213.

Niyonsenga, T., & Hlaing, W. (2007). Prevalence and correlates of sexual risk behaviors by gender among multi-ethnic adolescents. Annals of Epidemiology, 17, 742–743.

Ostrom, T. (1971-1972). Item construction in attitude measurement. The Public Opinion Quarterly, 35, 593–600.

Peter, P. (1981). Construct validity: a review of basic issues and marketing practices. Journal of Marketing Research, 18, 133–145.

Phillips, D., & Schweisfurth, M. (2007). Comparative and international education : An introduction to theory, method, and practice. New York: Continuum Publishing.

Richardson, J., Slater, J., & Wilson, J. (2007). The National Student Survey: Development, findings and implications. Studies in Higher Education, 32, 557–580.

Robinson, J. (1973). Toward a more appropriate use of Guttman scaling. Public Opinion Quarterly, 37, 260–267.

Ryu, E., Couper, M., & Marans, R. (2006). Survey incentives: Cash vs. in-kind; Face-to-face. International Journal of Public Opinion Research, 18, 89–106.

Shields, L. (2007). Using semantic differentials in fieldwork. Journal of Evaluation in Clinical Practice, 13, 116–119.

Saleh, A., & Bista, K. (2017). Examining factors impacting online survey response rates in educational research: Perceptions of graduate students. Journal of MultiDisciplinary Evaluation, 13(29), 63–74. Retrieved October 25, 2018 from EBSCO Online Database Education Source. http://search.ebscohost.com/login.aspx?direct=true&db=eue&AN=125574736&site=ehost-live&scope=site

Sharkness, J., & DeAngelo, L. (2011). Measuring student involvement: A comparison of classical test Theory and item response theory in the construction of scales from student surveys. Research in Higher Education, 52, 480–507. Retrieved December 6, 2013 from EBSCO Online Database Education Research Complete. http://search.ebscohost.com/login.aspx?direct=true&db=ehh&AN=61930910&site=ehost-live

Spooren, P, Mortelmans, D., & Denekens, J. (2007). Student evaluation of teaching quality in higher education: development of an instrument based on 10 Likert-scales. Assessment & Evaluation in Higher Education, 32 , 667-679.

Suggested Reading

Ary, D., Jacobs, L. C., Sorensen, C. K., & Walker, D. A. (2019). Introduction to Research in Education. 10th ed. Boston, MA: Cengage.

Barge, S., & Gehlbach, H. (2012). Using the theory of satisficing to evaluate the quality of survey data. Research in Higher Education, 53, 182–200. Retrieved December 6, 2013 from EBSCO Online Database Education Research Complete. http://search.ebscohost.com/login.aspx?direct=true&db=ehh&AN=70842547&site=ehost-live

Carroll, J., & Schweiker, R. (1951). Factor analysis in educational research. Review of Educational Research, 21, 368–388.

Dailey, J. (1953). Development and applications of tests of educational achievement outside the schools. Review of Educational Research, 23, 102–109.

Huebner, R. A. (2013). A survey of educational data-mining research. Research in Higher Education Journal, 19 1–13. Retrieved December 6, 2013 from EBSCO Online Database Education Research Complete. http://search.ebscohost.com/login.aspx?direct=true&db=ehh&AN=90440426&site=ehost-live

Lynch, J. (1983). The role of external validity in theoretical research. The Journal of Consumer Research, 10 , 109–111.