Integrated Learning Systems

Integrated Learning Systems (ILSs) are packages of interactive software created to teach a concept. While often used in large corporations for employee training, ILSs have emerged as a comprehensive way to instruct, remediate, and accommodate students of varying ages in public schools across the globe. Educational programs such as SuccessMaker(r) , Plato(r), and Kerzweil 3000(r) are only some of the software packages available to school districts, and each package tends to promote ideal resources. ILS creators have produced documentation that their products are effective when used in classrooms and that their software increases the skill levels of the students who use them. Some educational researchers, however, argue that the increase in skill level is either inconsistent from study to study or is short lived, in that students whose achievement scores do increase tend to find that increase leveling off over time.

Keywords Accommodation; Computer Assisted Instruction (CAI); Computer Assisted Learning (CAL); Dysgraphia; Dyslexia; Holistic Instruction; Integrated Learning Systems (ILS); Individuals with Disabilities Act (IDEA); Individualized Education Plan (IEP); Kurzweil 3000(r); No Child Left Behind Act of 2001 (NCLB); Phonemic Awareness; Phonics; Plato(r); Reading First; SuccessMaker(r)

Teaching Methods > Integrated Learning Systems

Overview

Now that computers are competing with television for the attention of school-aged children, there has been increased consideration to incorporate technology-based products into the classroom. Even in the poorest of U.S. school districts computers can be found. While some schools have entire labs dedicated to computers, many classrooms have sections of their rooms devoted to computerized learning. In many schools, some form of computer assisted instruction (CAI) is part of the curriculum as more and more students become comfortable with the technology. For many, maneuvering from screen to screen is as easy as figuring out which of several remote controls will adjust the volume of their televisions. In response to the greater awareness among school-age children of a computer's capabilities, computer program companies have developed entire software packages to offer tutorial assistance to students while keeping track of their achievement. These packages, called Integrated Learning Systems (ILSs), contain content (e.g., specific subjects or topics of focus), record-keeping databases, and a management system on which the program runs. ILS products, such as SuccessMaker(r), Plato(r), and Kurzweil(r), are considered standard components in classroom instruction in many school districts across the United States, the United Kingdom, Australia, and other developed countries.

How Integrated Learning Systems Work

The lure of ILS programs is that they supplement learning on an individualized level in basic skills areas, such as reading and math; other programs focus on specific fields of study, such as thermodynamics, or specific populations of learners, such as students with a learning disability or those who know English as a second language. Unlike some course curricula, the software programs are designed to enhance learning based on the skill level of each student who works with them. For example, if a third grader is testing at a first-grade reading level, the instruction provided by SuccessMaker(r) will begin at the first grade level and move in increments as the student demonstrates improvement in reading ability. Thus, if a student lacks understanding of a concept, the program does not move forward as would a class lesson; as a result, students don't fall behind, feel inferior, and/or give up. As noted in their work from 2001, Jervis & Gkolia (2005) indicate that instructors who have used an ILS in the classroom have seen an increase in motivation and confidence in their academically weaker students.

The Influence of No Child Left Behind

Before concepts from the No Child Left Behind Act were incorporated into public school retention strategies in 2001, ILSs were already available to educators, allowing them to create innovative ways to assist students in becoming successful learners (Jervis & Gkolia 2005). However, even though computer- assisted learning was available prior to the No Child Left Behind legislation, the legislation as well as the improvement in technological capabilities in the past fifteen years has made ILSs a logical addition to classroom learning. Once used as a supplement in the 1970s and 1980s, ILSs are now considered an intrinsic part of a daily learning schedule for many students in the United States, the United Kingdom, and Australia. Academically weak students, students with disabilities, students considered "at risk," and non-native English speakers now have the opportunity to spend time on individualized lessons that track their progress, test and retest material, and supply them with attention-grabbing graphics and narration.

Reading First legislation, a section within the No Child Left Behind Act, created five specific skill areas for which education systems would be accountable with regard to standardized testing through the third grade. Course content was modified according to these five areas:

• Vocabulary,

• Reading fluency (both silent and aloud),

• Reading comprehension,

• Phonics, and

• Phonemic awareness.

Districts became accountable for basic literacy skill instruction. As early childhood educators know, learning is foundational: as students develop an adequate level of literacy in their early years, so too will they be successful in literacy achievement later in school (and on later tests). ILS producers created their literacy software programs with a focus on these five components to assist classroom instruction.

Applications

Implications for Classroom Use

By providing specific information regarding each student, the teacher can set up an ILS to focus a child's learning directly where it is needed, be it three levels above the rest of the class or two levels behind. Specific curricula (content and learning level) can be entered and saved for each student, allowing for students to not only continue their learning where they left off the last time they used the system, but also allowing for review of the last material studied. If different concepts are presented in one lesson time (the time a student spends utilizing an ILS during a school day), a review and testing mechanism provides the student with the capability to learn material, review it, and move on to another concept. The learned material is embedded in later instruction; thus review and assessment are continuous, and an instructor can easily identify areas of strength and weakness in each student's learning.

Using the System

For initial application (or placement), an instructor has to input each student's skill level based on the material provided by the software. The course lessons then begin in the middle of a student's achievement level, so that material is neither too difficult nor too easy. As skill levels increase, so can the level at which a student is expected to succeed. This adjustment is made by the instructor based both on a student's performance with the material already provided and on the course/lesson goals and objectives. More specifically, for each goal or objective, the software creates exercises of equal difficulty that come randomly from a set of variables already incorporated into the computer program. Once material is mastered, the program presents information for an incrementally greater skill level.

The program can also forecast the student’s progress based on that student's strongest and weakest skills. This forecast can be beneficial to both the teacher and the student because classroom instruction can focus on identified areas of weakness, and within the program itself, students are not left wondering what will come next. They can anticipate possible coursework based on what they have been accomplishing. As with any content and regardless of the bells and whistles attached with any learning application, students will achieve at differing paces and according to differing material. Administrators and teachers who are accountable to state achievement standards can utilize the forecasting capabilities of SuccessMaker(r) to alter goals, objectives, course material and presentation as well as assessment methods before students find themselves in academic difficulty and behind the standard. SuccessMaker(r) has created its software to correlate with each state's standards, which makes the implementation of specific teaching strategies a condition of practice rather than catch up for school districts (Thrall & Tingey 2003).

ILS for Students with Disabilities

In 1975, the Individuals with Disabilities Act (IDEA), required all public school systems to grant equal access to education to all of its students. As a result, each student attending a public education institution who has either a learning or a physical disability must be granted an Individualized Education Plan (IEP) - a plan to create a learning environment in which student number one has the same access to being a successful learner as students number two, three, four, and five.

The Kurzweil 3000(r) is a computerized learning system used in many schools to assist students who have a disability with regard to learning academic concepts. These disabilities can be learning, as with students with dyslexia or dysgraphia, or physical, as in students with a visual or brain functioning difficulty. The system capabilities are vast when considering the length of time it can take a student with a learning disability to read, assess, and comprehend material from a text. One of the ways a student can gain assistance with this product is through the combination of the computer software and a scanner. A text can be scanned into a computer and read aloud to the student with the text appearing either in large print, in different colors (from one word to another) and/or with each word being highlighted as it is being presented vocally. For students with visual disorders, attention issues, and decoding problems, this is greatly improves the access they have to the same material as their peers. And, as accommodations are required of school systems (including those in higher education), Kurzweil 3000(r) is at the top of its field with regard to options, capabilities, and availability. In addition, when a technical issue arises (computer, scanner, speakers, content, etc.,) a technical representative is available to offer assistance - talking a user through a problem - during most of the day (Ludlow & Foshay, 2006, p. 80).

Advantages of Kurzweil 3000(r)

Ludlow & Foshay (2006) note that the Kurzweil 3000(r) program offers "access to written material for students with diverse special needs that include learning disabilities, visual impairments, Asperger syndrome, and problems with attention and concentration" (p. 81). Using a scanner equipped with a digital converter, students have the ability to manipulate text in a variety of ways. According to the authors,

The program permits the user to customize text with enlargement up to 500% of its original size and to increase or decrease the voice output speed to match the listener's comprehension abilities. It also enables students to take notes, both vocal and written, on the computer directly in the digital/scanned-in text. Students can highlight key points and bookmark areas they feel are important; further, notes and highlights are easily copied and pasted to text files for later review. Either instructors or students may create audio CDs to facilitate listening to books "on the go." The program may also be used by students for writing assignments. For example, students can type their ideas into the program, use the spell-check function, and have the program read back the text so that they can self-edit their work … Spell-checking is available in multiple languages (Ludlow & Foshay, 2006, p. 79).

Another constructive component of the program is that it can help students who know English as a second language or who find reading (and learning) difficult for reasons other than a disability. Kurzweil 3000(r) facilitates "active reading through highlighting, bookmarks, and voice notes for the purpose of creating outlines, study guides and word lists" (Ludlow & Foshay, 2006, p. 79). In addition, Web resources like online books, picture dictionaries (an aid to students who have difficulty reading pages of text), and definitions and synonyms from a dictionary (aids to non-native speakers of English) are available to Kurzweil 3000(r) users.

Disadvantages of Kurzweil 3000(r)

However, like many good ideas, the availability and use of Kurzweil 3000(r) is most often limited to students with disabilities because school districts and colleges can avert the cost of the program with grants and waivers specifically for the disabled population. In effect, school districts must offer students with disabilities the tools with which to succeed. Yet, they are not required to offer such assistance to English language learners or students who are slower at achieving academic success.

One concern that might prevent the widespread use of the Kurzweil 3000 is that its $1400.00 price tag may be prohibitive. However, according to Cullen, Keesey, Alber-Morgan, and Wheaton (2013), “if Kurzweil 3000 is not available to teachers who would like to use this intervention, similar activities may be developed with other computer programs available within school districts, or teachers can seek outside funding to obtain these programs. When developing appropriate and motivating computer-assisted instruction, the key is to include individualized response prompts, frequent opportunities to respond, and corrective feedback or reinforcement for each response” (Cullen et al., 2013, p. 102).

Viewpoints: Do ILS Improve Learning?

Inconsistent Research

There are several issues concerning ILS use: The first is the research itself. According to many researchers, the biggest problem with ILSs is that there is little evidence that shows they successfully assist student learning. O'Byrne, Securro, Jones & Cadle (2006) note that "these programs are especially challenging to assess because of the lack of systematic research demonstrating their effectiveness in helping students learn."

Plato Learning, Inc. (2004) assessed studies on the use of some of their own products and came to a similar conclusion.

There is no straightforward relationship between patterns of PLATO use and achievement, because of complexities of placement, progress, and various learner variables. As a result, we report utilization patterns and correlations of module mastery and time on task with achievement as a means of describing the program, but do not attribute causal significance to them (p. 5).

However, according to an evaluation of PLATO(r) curricula in several Tennessee counties in 1992-1993, the curricula offered to school systems is one of the PLATO Learning System's greatest strengths. PLATO Learning Inc. (1998) notes that

Instructors using the courseware appreciated the depth of PLATO courseware in a variety of subject areas and PLATO's compatibility with their specific program objectives as well as standardized tests such as the ABLE (Adult Basic Learning Examination), GED (General Educational Development ) and the ACT (American College Test). "The curriculum is well rounded," said one Board of Education representative. "I have yet to find any material which repeats itself. We could not be happier with the software … because it reinforces classroom instruction, offers variety [of graphics, animation, skill level, lessons] and immediate feedback, and affords students the opportunity to learn in a non-threatening, non-judgmental environment" ("Teamwork West Tennessee," 1998, p. 1)

From an administrative perspective, Gary Bloom (1999), associate director of the New Teacher Center at the University of California at Santa Cruz and a former superintendent, discusses the controversy among educators with regard to the use of ILS and other software programs in classrooms. Putting it bluntly, Bloom states that imbedded in the controversy is the research itself. "The only consensus is that there is no true consensus about the effectiveness of educational technology" (Para. 4). Specifically, Bloom notes that,

Heavily marketed Integrated Learning Systems have not produced evidence of student achievement that would justify their high costs. Some research shows that the advantages of these systems disappear when the same amount of time is spent with paper, pencils and equivalent print materials … In the only study to date that compared the cost effectiveness of technology-based instruction to other methodologies, including reduced class size, peer tutoring was found to be more cost-effective (Bloom, 1999, par. 5).

Furthermore, Wood (1998) notes that studies in the UK looking at SuccessMaker(r) and its effect on reading were inconsistent and sometimes showed the program's use to be connected with a lower achievement in students. Additionally, Becker's (1992) research shows that ILS use was more effective with students who needed to improve their skills and students who were already considered successful but showed no impact on students holding in the academic middle ground.

In a ten-year comparison study of student expectations regarding technology-enhanced learning, Jackson, Helms, Jackson, and Gum, (2011) found that students continue to show a strong desire for lecture-dominated classes. In the ideal classroom environment envisioned by the students in the study, class discussions and exercises, written handouts, and outlines share space with DVDs, presentation software, email, computer projects, simulations, and the Internet.

Lack of Learning Context

The second issue is the context by which a student learns information, including the ability to collaborate with peers and an instructor or tutor. Without a context for application a student will put the new material aside in lieu of information that makes sense in his "real world" experiences. An ILS that tests vocabulary, for example, will only be useful if the student is required to use the vocabulary in his everyday life. And, it's possible for a classroom disconnect as well, as traditional classroom learning areas (with carpets, bookshelves, and desks) are compared to computer stations that include fun, graphic-filled activities separate from the rest of the class. The perception of two separate entities with two separate goals may result, and this difference could make the transition of ideas from one space of learning to another difficult. The teacher's role would be to take the activity (or objective) from the ILS lesson and create other activities within the rest of the classroom. The difficulty here would be incorporating the activity of each of the students' ILS experience when each could be working on something different during their computer time.

PLATO(r) mathematical products address this concern. According to a study by the PLATO Learning Network (2004), the activities in the math curricula have been set up to be both collaborative and contextual.

They are open-ended problem-solving activities, based on real-world scenarios for the use of mathematics, in which learners work together to plan their strategy and execute it--and if necessary, try again, depending on the outcome … there is extensive combination of math tasks and topics cross-strand, and problems provide compelling, authentic scenarios within which learners can connect what they have learned to what they know, as they solve problems. There is a tool-rich work environment, and instructors can use the open-ended environment to adapt the tasks within each project to the needs of their learners and their curriculum (Plato Learning, Inc., 2004, p. 72).

Additionally, it has been proven that when students collaborate and build on what they already know, using that knowledge as a basis for new classroom learning, they become stronger achievers academically (Moll, 1989; Moll & Diaz, 1986; Palincsar & Brown, 1989; Palinscar, Ramson, & Derber, 1988/89; Brown, Palincsar, & Purcell, 1986).

The 'One Size Fits All' Model

The third concern is the "one size fits all" model program. While research on the whole does show that ILSs can be beneficial to students, Hoffmann (1985) cautions that no one software program or system will be beneficial for every learner or in all learning situations. In his book, Frames of Mind, Howard Gardner (1993) shows that students not only learn differently and at different levels, but that they have different intelligences altogether, meaning that while some people are natural learners when it comes to math (logical intelligence) and writing (linguistic intelligence), the same can be said for some people with regard to athletics or dance (kinesthetic intelligence; Humphries, Bidner, & Edwards, 2011). Just as a class lecture will be more captivating to an auditory learner than a visual learner, one software program may not show the same benefits (e.g., academic success) from one student to another.

Cost

Finally, ILS packages are expensive, sometimes running over $2000 (or more) to set up one computer. When the utilization of an ILS relies solely on the technology strength of a teacher and the interest level of his students - two distinctly exclusive and variable attributes - the practicality of such a purchase depends on tech support, additional time in the life of an already busy instructor, and the assumption that students will want to use the devices simply because they seem "cool." If a third grade classroom is functioning well - with some children working in the reading corner of the room, others receiving instruction from the teacher referencing a project on which they must work, and still others working with an ILS in the computer section of the room - it could be very disruptive to have a technical problem occur on one of the computers. For students who value working with their teacher, the computer section may not be appealing; low interest could very well lead to little improvement. Furthermore, as research yields product assessment, ILS manufacturers will continue to improve their products, making the need for teachers to "keep up."

Discussion

As the general public knows, computers are not a passing fad. The best use, it would seem, for ILSs in an educational environment would consist of a dual approach in which the software supplements the academic experience. Students will continue to learn at different levels and in different ways, so if an approach in addition to the one-on-one assistance of a teacher can be implemented in the daily schedule of a classroom, it should be. ILSs are adaptive, their capabilities flexible, and more than anything else, they encourage computer use for educational purposes. However, the reviews about ILS use remain mixed, and there doesn't seem to be a level of compromise at this time. The research that is available shows effectiveness in motivation and confidence for students who use an ILS. If nothing else, a student being motivated to learn and confident in his ability to do so is a positive thing. And, with a teacher to take ILS learning into the classroom, building on the learned material, making it collaborative, continuous, and meaningful, the combination of traditional and computerized teaching methods can't be seen as anything other than integrative and holistic. When helping students learn is the goal, there should be no debate regarding combining ILS and teacher interaction.

Terms & Concepts

Accommodation: A modification of learning provided for students with a documented learning or physical disability (e.g., the provision of extended time for an in-class exam).

Computer Assisted Instruction (CAI): A method of teaching that incorporates the use of a computer program into a curriculum.

Computer Assisted Learning (CAL): Supplemental instruction offered through interactive computer software.

Dysgraphia: Writing disorder not linked to intelligence that typically stems from brain trauma.

Dyslexia: Reading disorder not linked to intelligence that causes faulty identification of letters and words.

Holistic Instruction: Instruction provided in a variety of modes (discussion, lecture, writing assignment, group collaboration) to achieve a skill or piece of knowledge.

Individualized Education Plan (IEP): A written strategy created by educators and/or administrators to assist students with learning or physical disabilities in achieving educational success

Individuals with Disabilities Education Act (IDEA): The law ensuring that anyone with a physical or learning disability has equal access to education.

Integrated Learning Systems (ILS): Packages of software containing an areas of content (reading, mathematics, social studies), a database with which to maintain individual student records, and a management system to run the software program.

No Child Left Behind Act of 2001: Legislation enacted to ensure that all students (regardless of ethnicity, socioeconomic status, or disability) have access to instructional approaches that have been proven to be successful.

Phonemic Awareness: The ability to hear and manipulate different sounds when developing the skill of reading.

Phonics: Instructional term describing how students are taught to connect sounds to letters or groups of letters as an instrumental way of learning how to read.

Reading First: Legislation embedded into the No Child Left Behind Act that focuses instruction methodology on five specific areas: vocabulary, reading comprehension, fluency of reading (both aloud and silent), phonics, and phonemic awareness.

Bibliography

Becker J.H. (1992). A model for improving the performance of integrated learning systems: mixed individualized/group/whole class lessons, cooperative learning and organizing time for teacher-led remediation of small groups. Educational Technology 2, 65-79.

Bloom, G. (1999). Doorstops… or deliverance? Thrust for Educational Leadership, 28 , 24. Retrieved September 12, 2007 from EBSCO Online Database Education Research Complete. http://search.ebscohost.com/login.aspx?direct=true&db=ehh&AN=1715892&site=ehost-live

Brophy, J. (Ed.). Teaching for understanding and self-regulated learning 1, 35-71. Greenwich, CT: JAI Press.

Brown, A. L., Palinscar, A. S., & Purcell, L. (1986). Poor readers: Teach, don't label. In U. Neisser (Ed.), The academic performance of minority children: New perspectives (pp. 105-143). Hillsdale, NJ: Lawrence Erlbaum Associates.

Cullen, J., Keesey, S., Alber-Morgan, S. R., & Wheaton, J. (2013). The effects of computer-assisted instruction using Kurzweil 3000 on sight word acquisition for students with mild disabilities. Education & Treatment of Children, 36, 87-103. Retrieved December 27, 2013, from EBSCO Online Database Education Research Complete. http://search.ebscohost.com/login.aspx?direct=true&db=ehh&AN=87600226&site=ehost-live

Garcia, E. & Padilla R., eds. (1985). Advances in bilingual education research. Tucson: The University of Arizona Press.

Gardner, H. (1993). Frames of Mind. HarperCollins: New York.

Hoffmann R. (1985). Educational software: Evaluation? No! Utility? Yes! Journal of Learning Disabilities 18 , 358-360. Retrieved September 12, 2007 from EBSCO Online Database Academic Search Premier. http://search.ebscohost.com/login.aspx?direct=true&db=ehh&AN=4738022&site=ehost-live

Humphries, C. A., Bidner, S., & Edwards, C. (2011). Integrated learning with physical education and music. Clearing House, 84, 174-179. doi:10.1080/00098655.2010.543192 Retrieved December 27, 2013, from EBSCO Online Database Education Research Complete. http://search.ebscohost.com/login.aspx?direct=true&db=ehh&AN=64320471&site=ehost-live

Jackson, M., Helms, M. M., Jackson, W. T., & Gum, J. R. (2011). Student expectations of technology-enhanced pedagogy: A ten-year comparison. Journal of Education for Business, 86, 294-301. doi:10.1080/08832323.2010.518648. Retrieved December 27, 2013, from EBSCO Online Database Education Research Complete. http://search.ebscohost.com/login.aspx?direct=true&db=ehh&AN=61460053&site=ehost-live

Jervis, A. & Gkolia, C. (2005). "The machine stops": One school's rejection of integrated learning systems. Education & Information Technologies, 10 , 305-321. Retrieved September 12, 2007,from EBSCO Online Database Education Research Complete. http://search.ebscohost.com/login.aspx?direct=true&db=ehh&AN=18171839&site=ehost-live

Johnson, D. (1998). Customizing education. Technology Connection, 4 , 48. Retrieved September 11, 2007 from EBSCO Online Database Education Research Complete. http://search.ebscohost.com/login.aspx?direct=true&db=ehh&AN=173810&site=ehost-live

Kemp, S. (1997). Software characteristics can affect student achievement. Curriculum Administrator, 32 , 48.

Ludlow, B., Foshay, J. (2006). Kurzweil 3000. Journal of Special Education Technology, 21 , 79-81. Retrieved September 15, 2007 from EBSCO Online Database Education Research Complete. http://search.ebscohost.com/login.aspx?direct=true&db=ehh&AN=22291460&site=ehost-live

Moll, L. C. (1989). Teaching second language students: A Vygotskian approach. In D. Johnson & D. Roen (Eds.), Richness in writing: Empowering ESL students (pp. 55-69). New York: Longman.

Moll, L. C., & Diaz, S. (1986). Ethnographic pedagogy: Promoting effective bilingual instruction. In E.Garcia & R. Padilla (Eds.), Advances in bilingual education research (pp. 127-149). Tucson: The University of Arizona Press.

National Council of Teachers of Mathematics. (1989). Curriculum and evaluation standards for school mathematics. Reston, VA: Author.

O'Byrne, B., Securro, S., Jones, J. & Cadle, C. (2006). Making the cut: the impact of an integrated learning system on low achieving middle school students. Journal of computer assisted learning, 22 , 218-228. Retrieved September 11, 2007 from EBSCO Online Database Education Research Complete. http://search.ebscohost.com/login.aspx?direct=true&db=ehh&AN=20779358&site=ehost-live

Palinscar, A. S. & Brown, A. L. (1989). Classroom dialogues to promote self-regulated comprehension. In J.Brophy (Ed.), Teaching for understanding and self-regulated learning (pp. 35-71). Greenwich, CT: JAI Press.

Palinscar, A. S., Ramson, K., & Derber, S. (1988/1989). Collaborative research and development of reciprocal teaching. Educational Leadership, 46 , 37-40.

development of reciprocal teaching. Educational Leadership, 46 , 37-40.

Plato Learning, Inc. (2004). Technical paper #12: An overview of the research base of PLATO(r) (Revised Version 2). Minnesota, MN: PLATO Learning, Inc. Retrieved September 11, 2007, from PLATO Learning Systems http://www.plato.com/media/Technical-White%20Papers/1/An%20Overview%20of%20the%20Research%20Base%20of%20PLATO%20Software.pdf

Teamwork West Tennessee: An Evaluation of the PLATO Computer System in Carroll, Fayette, Hardin, Haywood, and Madison Counties (1992-1993). (1998). PLATO Learning Systems, Inc. Retrieved September 10, 2007, from PLATO Learning Systems http://www.plato.com/media/Evaluation%20Studies/T/Teamwork%20West%20Tennessee.pdf

Thrall, T. & Tingey, B. (2003.) SuccessMaker(r) Motion: A Research Summary. Pearson Digital Learning Technical Note TN030409. Retrieved September 13, 2007 from Pearson Digital Learning http://www.pearsondigital.com/pdfs/whitepapers/SuccessMaker_Motion-A_Research_Summary.pdf

Suggested Reading

Ainsley, J., Banks D. & Fleming M. (2002). The influence of IT: perspectives from five Australian schools. Journal of Computer Assisted Learning, 18, 395-40

Brown, J. (1997). When is a system an ILS? In J. Underwood & J. Brown (Eds) Integrated learning systems in UK schools (pp. 6-14). London: Heinemann.

Buckleitner, W. (1999). The state of children's software evaluation - yesterday, today, and in the 21st century. Information Technology in Childhood Education Annual, 211-220.

Case, C. & Truscott D. (1999). The lure of bells and whistles: choosing the best software to support reading and writing. Reading and Writing Quarterly, 15, 361-370.

Furner, J.M. & Daigle, D. (2004). The educational software/website effectiveness survey. International Journal of Instructional Media, 31, 61-77.

George middle school increases student achievement with comprehensive learning system (2002). THE Journal, 29, 52-53.

Goyne, J., McDonough, S. K. & Padgett, D. D. (2000). Practical guidelines for evaluating educational software. Educational Software, 73, 345-348.

Grehan, A. & Ross, S. (2004). An evaluation of the effects of FOCUS on first grade reading achievement in a title I elementary school. Retrieved October 19, 2007 from http://www.plato.com/media/Technical-White%20Papers/2/Piedmont%20Elementary%20Evaluation%20of%20PLATO%20Focus.pdf

Isernhagen, J.C. (1999). Technology: a major catalyst for increasing learning. THE Journal, 27, 30-34.

Jones, H.T. & Paolucci, R. (1999). Research framework and dimensions for evaluating the effectiveness of educational technology systems on learning outcomes. Journal of Research on Computers in Education, 31, 17-27.

Jones, J., Staats, W., Bowling, N., Bickel, R., Cunningham, M. & Cadle, C. (2004-2005). An evaluation of the Merit reading and software program in the Calhoun county (WV) middle/high school. Journal of Research on Technology in Education, 37, 177-195.

Lewis, A. (1999). Integrated learning systems and pupils with low attainments in reading. British Journal of Special Education, 26, 153-157.

Ligas, M. R. (2002). Evaluation of Broward county alliance of quality schools project. Journal of Education for Students Placed at Risk, 7, 117-139.

Lynch L., Fawcett A.J. & Nicholson R.I. (2000). Computer assisted reading instruction in a secondary school: an evaluation study. British Journal of Educational Technology, 31, 333-48.

Maddox, L. (1991). Integration versus computer labs: an either/or proposition? Educational Technology, 31, 36-43.

Mann D., Shakeshaft C., Becker J. & Kottkamp R. (1999). West Virginia's Basic Skills/Computer Education Program: An analysis of student achievement. Santa Monica, CA: Milken Family Foundation.

NCLB dominates at FETC (2003). Technology and Learning, 23 .

Papert, S. (1993). The Children's Machine: Rethinking School in the Age of the Computer. New York: Basic Books.

Popular programs will be available to use with mobile devices. (2013). Electronic Education Report, 20, 1-2. Retrieved December 27, 2013, from EBSCO Online Database Education Research Complete. http://search.ebscohost.com/login.aspx?direct=true&db=ehh&AN=89650206&site=ehost-live

Shade, D.D. & Watson J.A. (1990). Computers in early education: issues put to rest, theoretical links to sound practice and the potential contribution of microworlds. Journal of Educational Computing Research, 6, 375-392.

Texas middle school adopts software for reading assessment and instruction (1995). THE Journal, 22, 69-70.

Traynor, P. L. (2003). Effects of computer-assisted-instruction on different learners.

Journal of Instructional Psychology, 30, 137-143.

Truett, C. (1984). Field-testing education software: are publishers making the effort? Educational Technology, 24, 7-12.

Underwood, J. (1996). Paper No. 159, Proposal to CAL97, March 23-26. Exeter, UK: School of Education, University of Exeter.

Underwood, J. D. (2000). A comparison of two types of computer support for reading development. Journal of Research in Reading, 23, 136-148.

Underwood, J. D. & Brown, J. (Eds.) (1997). Integrated Learning Systems: Potential into Practice. Oxford, UK: NCET/Heinemann.

Weaver, G. C. (2000). An examination of the national educational longitudinal study database to probe correlation between computer use in school and improvement in test scores. Journal of Science and Technology, 9, 121-133.

Wood, D. (1998). The UK ILS Evaluations - final report. Coventry, UK: British Educational Communications and Technology Agency. Wood, D., Underwood, J. & Avi, P. (1999). Integrated learning systems in the classroom. Computers and Education, 33, 91-108.