Test Bias

Since the enactment of the No Child Left Behind Act of 2001, standardized testing has been at the center of a heated debate: critics claim that the tests betray unfair biases that limit minority students and have serious repercussions on their academic success as well as on the performance of teachers, school districts, and state educational efforts. Test bias can be a tricky subject because it is often difficult to determine whether different scores among test takers are caused by differences in ability or by a test bias. The times are long past when a person could easily spot the most obvious biases in testing instruments, such as referring to minorities using derogatory terms. However, there can still be a hidden bias in a testing instrument or test item that favors majority over minority populations.

Keywords Content Validity; Cultural Bias; Gender Bias; High-Stakes Test; Item Bias; Language Bias; No Child Left Behind Act of 2001 (NCLB); Norm-Referenced Test; Socioeconomic Bias; Standardized Tests; Test Bias; Validity

Testing & Evaluation > Test Bias

Overview

Test bias occurs when a test item or entire test causes students of similar abilities to perform differently because of their ethnic, cultural, religious, or gender differences. For a test to be valid, it must measure student achievement regardless of their divergent backgrounds (Jorgensen, 2005). Because standardized tests have come to have long-range effects on students and school districts, it is important to be aware of and avoid all forms of test bias so that standardized tests accurately measure achievement of all test takers.

Test bias can be a tricky subject because it is often difficult to determine whether different scores among test takers are caused by differences in ability or by a test bias. The times are long past when a person could easily spot the most obvious biases in testing instruments, such as referring to minorities using derogatory terms. However, there can still be a hidden bias in a testing instrument or test item that favors majority over minority populations.

There are many types of testing bias, and a testing instrument only needs to have one to be considered biased and, therefore, invalid. Among the types of possible test bias are cultural, socioeconomic, and gender bias, item bias, construct bias, sampling, language bias, and examiner bias:

• Cultural, socioeconomic, and gender bias occurs when a test item favors one gender, cultural, or socioeconomic group over another, uses terms that may be derogatory toward a group, or uses terms that may be more familiar to one group than another.

• Item bias occurs when a test item requires test takers to have secondary abilities, experiences, or knowledge to accurately respond to the test item.

• Construct bias occurs when a test is structured in a way that requires test takers to have secondary abilities, experiences, or knowledge for the test to accurately measure their achievement. Intelligence tests have come under scrutiny, with critics claiming that they do not measure the inherent aptitudes of minority populations, but rather, how well minorities share White, middle-class values and knowledge (Mercer, 1979, as cited in Skiba, Knesting & Bush, 2002).

• Sampling becomes a potential hotbed when discussing test bias, too. When subpopulations are sampled proportionally, the test will be biased against any minority population since, by definition, minority populations are in the minority. But if minority populations are overrepresented on a testing instrument, the test may end up being biased against the majority population. Therefore, when random sampling—which is considered statistically valid—occurs in the development of a testing instrument, the test should favor the group that comprises the largest proportion of the defined sample.

• Language occurs when designated subgroups of interest within a population are not equally familiar with test vocabulary or when the meaning of a word is not the same across subgroups.

• Examiner bias occurs if the examiner is not of the same culture or race as the students being tested (Skiba et al., 2002).

Applications

Bias

As stated above, there can be many different forms of test bias. When evaluating a test item or an entire testing instrument for bias, there are at least three issues that should be considered: fairness, bias, and stereotyping. Some questions that can be asked that address test item fairness include:

• Does the item give a positive representation of designated subgroups of interest?

• Is the test item material equally familiar to every designated subgroup of interest?

• Are designated subgroups of interest represented in relation to their presence in the general population being tested?

• Is there a greater opportunity for members of one group to have prior knowledge of the vocabulary used? (A potentially unfair item might reference a regatta, a word that would be known to test takers who live or vacation near water or who own a boat but that might be unfamiliar to those who reside primarily in urban settings.)

• Is there a greater opportunity for members of one group to have experience with a test item reference or become familiar with the method that the items present? (A potentially unfair question might reference a European or cross-country vacation, which some socioeconomic groups may not have experienced, or to reference plowing a field, an activity with which many urban or suburban students may not be familiar) (Hambleton & Rodgers, 1995).

A test item “may be biased if it contains language that is differently familiar to subgroups of test takers, or if the item structure or format is differently difficult for subgroups of test takers” (Hambleton & Rodgers, 1995, p. 2).

A test item “may be language biased if it uses terms that are not used across the tested population or if it uses terms that have different connotations among groups of the tested population” (Hambleton & Rodgers, 1995, p. 2). An example of language bias against African American students occurred in a study in which students had to recognize and then name an object that began with the same sound as hand. One correct response would have been heart, but many African American students chose car instead because in the slang they use, a car is known as a hog, which also has the same sound as hand. Therefore, although the African American students had understood the concept on which they were being tested, their answers were incorrect because of language differences (Scheuneman, 1982, as cited in Hambleton & Rodgers, 1995).

A test item may possess content bias if it refers to experiences or information that are not common across the tested population. A vocabulary or reading comprehension test that predominantly refers to rural experiences when knowledge of rural life is not being assessed is an example of a test that is biased against urban students.

Some questions that can be asked to detect bias include:

• Does the item contain content that is differently familiar to designated subgroups of interest? (content bias)

• Will members of designated subgroups of interest answer a test item differently for reasons not based on the ability being measured? (content bias)

• Does a test item require the test taker to have information or skills that cannot be expected to be within the educational background of all the test takers? (content bias)

• Does the test item contain words that have different or unfamiliar meanings for designated subgroups of interest? (language bias)

• Is the test item free of needlessly difficult vocabulary? (language bias)

• Is the test item free of group-specific language, vocabulary, or reference pronouns? (language bias)

• Does a test item contain any clues that would help one group and not another? (structure and format bias)

• Are there any inadequacies or ambiguities in the test instructions? (structure and format bias)

• Do the test instructions tend to differently confuse members of designated subgroups of interest? (structure and format bias)

Questions such as these can help test evaluators determine if there is content, language, and/or item structure and format bias (Hambleton & Rodgers, 1995, p. 4).

Stereotyping and inadequate representation of minorities are other forms of test bias. While this type of bias may not make a test item any harder for test takers, it can cause undue stress, which can prevent test takers from doing their best.

Examples of this type of bias might be test items that reinforce negative stereotypes, imply that a certain subgroup is inferior, or use derogatory terms such as housewife, Chinaman, colored people, red man, and lower class. For the sake of gender neutrality, other terms that should be avoided include those job designations ending in man. Instead of using policeman, test item writers should use "police officer"; and in place of fireman, "fire fighter" should be used. Depicting members of designated subgroups of interest as having stereotypical occupations, such as a Chinese launderer, should also be avoided. (Hambleton & Rodgers, 1995).

Some questions used to detect stereotyping might include:

• Does the item represent designated subgroups of interest in a positive way?

• Are designated subgroups of interest referred to in the same way with respect to using names and titles? (An example of being unfair would be using the title "Mr." for all males and referring to women by their first name.)

Validity

Test bias can be considered a validity issue. The four aspects of validity that are most commonly addressed are:

• Content validity

• Construct validity

• Predictive validity

• Consequential validity

Content Validity

Content bias can occur if test developers assume that all students have been exposed to the same concepts, lessons, vocabulary, and life experiences. As the United States becomes increasingly ethnically and racially diverse, this assumption becomes more of a problem for test developers. Students who come from culturally and linguistically diverse backgrounds may not perform as well as other students because of their different life experiences, how they were socialized, and what they have been exposed to throughout their lives (Stockman, 2000, as cited in Laing & Kamhi, 2003). To assure content validity, or that a test actually measures what it purports to measure, test publishers must make sure that a test is not biased.

One way to determine content validity is to have a panel of experts from diverse backgrounds examine each test item to try to detect potential bias.

Test items can also be checked for bias using differential item functioning. Differential item performance between subgroups occurs when unexpected differences between groups' scores remain once groups are matched for ability. If subgroups of equal ability perform differently on a test or test item, the test or test item may be considered biased. For example, a differential item functioning would occur if, when rural and urban students with the same overall test score are compared, the urban students score higher on a test item involving a subway. These items are thought to be differential in relation to their function because, among students of equal ability levels, the probability of a correct answer is not the same for both groups. A test item may be considered biased if there is an unexpected differential in performance on the item between two subgroups of a tested population (Dorans, 1989, as cited in Anderson & DeMars, 2002).

However, higher or lower scores by a particular group on any given item is not usually considered sufficient evidence that an item is biased either for or against a group. Rather than being a fault of the test, disproportionately high or low scores could be caused by differences in ability rather than differences among gender, ethnic, or socioeconomic groups. So, for a test item to be eliminated, “a group's performance on the item must be either better or worse than the group's performance on the testing instrument as a whole” (Schellenberg, 2004, p. 10).

Construct Validity

Construct validity occurs when the structure of a test makes the test similarly difficult across all groups of a tested population. Construct bias, on the other hand, happens when a test measures different abilities when different populations are tested. An example of a lack of construct validity can easily occur in mathematics. If a mathematics assessment includes several complex word problems, among most test takers it will measure their mathematical abilities. But for ESL students, a word problem is also a vocabulary and reading comprehension test. If a test displays a construct bias, it can be impossible to determine whether a test taker has actually mastered the skill being tested because the test is dependent upon the test taker also mastering a secondary skill. Therefore, in the case of word problems, the assessment may measure the same abilities for every group within the population that is taking the test, and it therefore lacks construct validity. (Schellenberg, 2004).

Predictive Validity

Predictive validity refers to how accurately a testing instrument predicts an outcome measure. This includes whether or not achievement tests are valid and precise indicators of academic success in the future. Predictive validity can be a difficult measurement because, at least in academics, future achievements are invariably measured by another assessment. This leads the initial assessment to be little more than a predictor of how well the test taker will score on future assessments rather than a predictor of actual future learning or achievement.

Consequential Validity

Consequential validity concerns whether the consequences that arise from test results are appropriate to the test takers' abilities. Consequential validity used to be more strongly associated with psychological and workplace testing, where the results could mean the difference between a job offer or promotion, making the consequences of the testing instrument truly life changing. However, when dealing with students, the consequential validity of achievement tests comes into play when students are referred for remedial assistance, selected for gifted programs, or denied or accepted into programs with limited enrollment. Students’ lives could even be altered depending on whether they are encouraged to participate in certain academic paths and if they have the resources to do so.

Consequential validity used to be mostly concerned with individual students, but the No Child Left Behind Act of 2001 has altered that. The act has transformed standardized testing into a high-stakes process in which schools, school districts, and states can face dire consequences if their students do not make adequate progress each year (Schellenberg, 2004).

Inherent Biases

Detractors of psychological tests contend that ethnic group differences on psychological tests are caused by inherent biases embedded in the tests through flawed psychometric methodology, rather than through differences in the actual psychology of ethnic groups. They argue that group differences are caused by characteristics of the tests and are unrelated to differences in the psychological trait being measured.

Those who believe the tests are inherently biased claim that

• The content of the test is unfamiliar to and inappropriate for minority test takers

• The standardization representative sampling of tests has an insufficient number of minorities for them to have a significant voice in test item selection

• The examiner and language bias are present since most psychologists are Caucasian and only speak standard English, which can be intimidating and confusing for minority test takers

• The tests measure different attributes when used with minority students

• The tests do not predict outcomes or future behaviors for minority children because they are not valid (Reynolds, 1983)

Further Insights

Test Bias & the Achievement Gap

Since all types of potential test bias are more commonly addressed in the twenty-first century, the question that faces test developers, educators, and lawmakers is whether achievement gaps in this country are a result of test biases or actual differences in achievement. Several methods used for detecting and eliminating bias in testing instruments have been mentioned here, but despite all the attention test bias has received and all the money spent on trying to eradicate test bias, the achievement gap still exists. In an ideal world, an item design would enable assessments to not only measure student academic achievement, but also provide detailed information about what learning outcomes students have yet to master (Jorgensen, 2005).

Substantial research efforts have not supported the hypothesis that standardized intelligence and achievement tests contain inherent cultural bias (Brown et al., 1999; Cole, 1981; Jensen, 1980; Roesenbach & Mowder, 1981; Suzuki & Valencia, 1997, as cited in Skiba et al., 2002). The possibility of bias has not been conclusively dismissed, but cultural biases do not appear to be the reason that minorities have lower test scores and a disproportionate placement in special education classes. Standardized tests appear to be reasonably accurate in assessing individual aptitude without bias, and they also show that there are discrepancies between populations. But using test results to determine individual aptitude and ignoring cultural and educational factors that potentially depress minority performance, can lead to inaccurate interpretations even if the test itself is not considered biased (Skiba et al., 2002).

Non-instructional factors can also explain many of the differences that occur from one test score to another when evaluating tests between school districts. On the 1992 National Assessment of Education Progress, a study discovered that math scores differed by 89 percent due to a combination of four things: how many parents lived at home, what their highest educational achievements were, what type of community they lived in, and the level of poverty they were in, if at all. State tests results are similar, with their percentages varying slightly depending on which socioeconomic variables are considered.

Standardized Tests

Since norm-referenced tests are not intended to measure the quality of learning or teaching, they should not be used to assess learning or teaching. Norm-referenced tests are designed so that half the students will score above the norm and the other half will score below the norm. In addition, the Stanford Achievement Test (SAT), Iowa Test of Basic Skills (ITBS), Metropolitan Achievement Test (MAT), California Achievement Test (CAT), and Comprehensive Test of Basic Skills (CTBS) are designed so that only about half of all students will be able to correctly answer most questions. Therefore, students who do not answer most questions correctly, or score above the norm, should not be penalized, nor should the school, school district, or state.

In 2012, civil rights and educational groups filed a federal complaint in New York alleging that African Americans and Hispanics were unfairly excluded from several of New York City’s high schools due to an admission test they believed was racially discriminatory (Baker, 2012). Reasons as to why vary, but lower income and African American and Latino students across the United States consistently score below higher income, White, or Asian students in achievement tests and college admission tests (Rooks, 2012). These issues continued in education in the early 2020s, though statistical methods of determining which items may be biased, called differential item functioning, aided test makers in avoiding bias (Bandalos, 2018).

Critics also point out that standardized tests tend to measure superficial thinking rather than critical thinking. One study classified elementary students “as actively engaged in learning if they asked questions while they read, or tried to connect what they were learning to past lessons. The study then classified those students who just copied down answers, guessed a lot, or skipped over difficult parts” as being only 'superficially' engaged (Kohn, 2000, p. 1). However, it was the superficial learners who tended to score highest on the Comprehensive Test of Basic Skills and the Metropolitan Achievement Test. Other discoveries have proved to be similar, having come from studies of middle and high school students in which middle school students took the Comprehensive Test of Basic Skills and high school students took the College Board's SAT test. So, while there are many students who are actively engaged in learning who do well on standardized tests, there is still a positive correlation between standardized test results and a shallow, less engaged approach to learning (Kohn, 2000).

Conclusion

When testing instruments are used to make far-reaching decisions, it is important to ensure that the instrument is not biased. However, short of taking every student in the nation that fits the testing profile and then randomly selecting the sampling group from them, it would be impossible to create the perfect representative group on which to test an assessment for bias. Even if it were possible, detractors would still claim that even with minority representation in any category—ethnicity, religion, economic status, etc.—a bias against the minority and for the majority would still exist, simply because the minority is in the minority. There will probably always be a seemingly harmless word such as escalator that passes the item bias test despite the many students living in rural areas who have never seen an escalator before and do not know what one is. Therefore, it is important that anyone involved in high-stakes testing consider using the chosen instrument in conjunction with other forms of assessment in order to mitigate any possible bias that may exist in the testing instrument.

Terms & Concepts

Content Validity: Content validity is the extent to which a test measures only what it is purported to measure.

Cultural Bias: Cultural bias in testing occurs when test items may favor or discriminate against a particular subgroup of the testing population based on race or ethnicity.

Gender Bias: Gender bias in testing occurs when test items may favor or discriminate against based on a student's gender.

High-Stakes Tests: High-stakes tests are when those test scores are used to make decisions that have important consequences for students, schools, school districts, and/or states and can include high school graduation, promotion to the next grade, resource allocation, and instructor retention.

Language Bias: Language bias in testing occurs when test items may favor or discriminate against a particular subgroup of the testing population based on the language used on the test. Language bias can also occur if the language used by the test administrator is different from the test taker's native language or type of language they use—such as African American vocabulary, standard English, English vernacular, etc.

No Child Left Behind Act of 2001 (NCLB): The No Child Left Behind Act of 2001 is the reauthorization and major overhaul of the Elementary and Secondary Education Act of 1965, the federal law regarding K–12 education.

Norm-Referenced Test: Norm-referenced tests are assessments administered to students to determine how well they perform in comparison to other students taking the same assessment.

Socioeconomic Bias: Socioeconomic bias in testing occurs when test items may favor or discriminate against a particular subgroup of the testing population based on social and/or economic factors.

Standardized Tests: Standardized tests are exams that are administered and graded in a uniform way, and the tests are created so that each question can be interpreted in the same way and remain consistent.

Test Bias: Test bias occurs when provable and systematic differences in the results of students taking the test are discernible based on group membership, such as gender, socioeconomic standing, race, or ethnic group.

Bibliography

Anderson, R. & DeMars, C. (2002). Differential item functioning: Investigating item bias. Assessment Update, 14, 12. Retrieved July 24, 2007, from EBSCO Online Database Education Research Complete. http://search.ebscohost.com/login.aspx?direct=true&db=ehh&AN=10349987&site=ehost-live

Baker, Al. (2012, September 27). Charges of bias in admission test policy at eight elite public high schools. New York Times. Retrieved December 12, 2013, from http://www.nytimes.com/2012/09/28/nyregion/specialized-high-school-admissions-test-is-racially-discriminatory-complaint-says.html

Bandalos D. L. (2018). Measurement theory and applications for the social sciences. Guilford Press.

Fischer, F. T., Schult, J., & Hell, B. (2013). Sex-specific differential prediction of college admission tests: A meta-analysis. Journal of Educational Psychology, 105, 478–488. Retrieved October 9, 2014, from EBSCO Online Database Education Research Complete. http://search.ebscohost.com/login.aspx?direct=true&db=ehh&AN=87508966&site=ehost-live

Ford, D. Y., & Helms, J. E. (2012). Overview and introduction: Testing and assessing African Americans: 'Unbiased' tests are still unfair. Journal of Negro Education, 81, 186–189. Retrieved December 13, 2013, from EBSCO Online Database Education Research Complete. http://search.ebscohost.com/login.aspx?direct=true&db=ehh&AN=83853077

Fraire, J. (2014). Why your college should dump the SAT. Chronicle of Higher Education, 60, A44. Retrieved October 9, 2014, from EBSCO Online Database Education Research Complete. http://search.ebscohost.com/login.aspx?direct=true&db=ehh&AN=95785703&site=ehost-live

Hambleton, R., & Rodgers, J. (1995). Item bias review (ERIC Digest No. ED398241). Retrieved July 24, 2007, from the Education Resources Information Center website: http://eric.ed.gov/?id=ED398241

Jorgensen, M. A. (2005, September 1). Test bias or real differences? THE Journal. Retrieved May 13, 2023, from http://thejournal.com/articles/2005/09/01/test-bias-or-real-differences.aspx

Kohn, A. (2000). Standardized testing and its victims. Education Week, 20, 60. Retrieved July 24, 2007, from EBSCO Online Database Education Research Complete. http://search.ebscohost.com/login.aspx?direct=true&db=ehh&AN=3730038&site=ehost-live

Laing, S., & Kamhi, A. (2003). Alternative assessment of language and literacy in culturally and linguistically diverse populations. Language, Speech, & Hearing Services in Schools, 34, 44–55. Retrieved July 4, 2007, from EBSCO Online Database Education Research Complete. http://search.ebscohost.com/login.aspx?direct=true&db=ehh&AN=8841126&site=ehost-live

Lee, T. C., Graham, J. R., Sellbom, M., & Gervais, R. O. (2012). Examining the potential for gender bias in the prediction of symptom validity test failure by MMPI-2 symptom validity scale scores. Psychological Assessment, 24, 618–627. Retrieved December 13, 2013, from EBSCO Online Database Education Research Complete. http://search.ebscohost.com/login.aspx?direct=true&db=ehh&AN=79755726

Reynolds, C. (1983). Test bias: In God we trust; all others must have data. Journal of Special Education, 17, 241–258. Retrieved July 24, 2007, from EBSCO Online Database Education Research Complete. http://search.ebscohost.com/login.aspx?direct=true&db=ehh&AN=4727754&site=ehost-live

Rooks, Noliwe M. (2012, October 11). Why it's time to get rid of standardized tests. Time. Retrieved May 13, 2023, from https://ideas.time.com/2012/10/11/why-its-time-to-get-rid-of-standardized-tests

Rosales, J. R. and Walker, T. (2021, March 20). The racist beginnings of standardized testing. NEA. https://www.nea.org/advocating-for-change/new-from-nea/racist-beginnings-standardized-testing

Schellenberg, S. (2004). Test bias or cultural bias: Have we really learned anything? Retrieved July 24, 2007, from National Association of Test Directors website: http://www.natd.org/2004Proceedings.pdf

Skiba, R., Knesting, K., & Bush, L. (2002). Culturally competent assessment: More than nonbiased tests. Journal of Child & Family Studies, 11, 61–78. Retrieved July 24, 2007, from EBSCO Online Database Education Research Complete. http://search.ebscohost.com/login.aspx?direct=true&db=ehh&AN=6768458&site=ehost-live

Thompson, G. L., & Allen, T. G. (2012). Four effects of the high-stakes testing movement on African American K-12 students. Journal of Negro Education, 81, 218–227. Retrieved December 13, 2013, from EBSCO Online Database Education Research Complete. http://search.ebscohost.com/login.aspx?direct=true&db=ehh&AN=83853080

Suggested Reading

Berk, R. (Ed.). (1982). Handbook of methods for detecting test bias. The Johns Hopkins University Press.

Camilli, G., & Shepard, L. (1994). Methods for identifying biased test items. Sage Publications.

Gipps, C., & Murphy P. (1994). A fair test? Assessment, achievement and equity. Open University Press.

Hamayan, E., & Damico, J. (1994). Limiting bias in the assessment of bilingual students. PRO-ED, Inc.

Osterlind, S. (1983). Test item bias. Sage Publications.

Ryan, T. G. (2012). Ontario educators’ perceptions of barriers to the identification of gifted children from economically disadvantaged and limited English proficient backgrounds. Journal of the International Association of Special Education, 13, 16–27. Retrieved October 9, 2014, from EBSCO Online Database Education Research Complete. http://search.ebscohost.com/login.aspx?direct=true&db=ehh&AN=86730009&site=ehost-live

Toldson, I. A. (2012). Editor's comment: When standardized tests miss the mark. Journal of Negro Education, 81, 181–185. Retrieved December 13, 2013, from EBSCO Online Database Education Research Complete. http://search.ebscohost.com/login.aspx?direct=true&db=ehh&AN=83853076

White, G., Jr. (2012). 'I am teaching some of the boys': Chaplain Robert Boston Dokes and army testing of Black soldiers in World War II. Journal of Negro Education, 81, 200–217. Retrieved December 13, 2013, from EBSCO Online Database Education Research Complete. http://search.ebscohost.com/login.aspx?direct=true&db=ehh&AN=83853079