Information theory
Information theory is a mathematical framework established in 1948 by Claude E. Shannon that focuses on the transmission, storage, and retrieval of information. It encompasses various fields, including biology, engineering, psychology, communications, and sociology. The core concept of information theory is that information can be quantified and treated as a physical entity, measured in units known as bits, which represent binary states (on/off or yes/no). A crucial component of the theory is entropy, which reflects the uncertainty or randomness contained within a message. Shannon's work demonstrated how messages could be transmitted accurately despite noise and distortion, emphasizing the importance of encoding messages with self-checking features. Information theory also analyzes general communication systems, detailing the roles of information sources, channels, receivers, and the impact of noise on message transmission. As it evolves, the theory continues to inform various applications, including coding theory, which focuses on the design of error-correcting codes for reliable communication. Overall, information theory provides a foundational understanding of how information is conveyed and processed across diverse domains.
On this Page
Information theory
Information theory is a branch of mathematics that extends into biology, engineering, psychology, medical science, communications, and sociology. The theory has also been applied to linguistics, cryptography, and phonetics. Additionally, many communication engineers apply the theory in their work.
Claude E. Shannon founded information theory in 1948. The theory involves the communication of information and the behavior of information as it is transmitted, stored, and retrieved. Numerous elements come into play with information theory, including entropy, a source of information, a channel, a receiving device, a destination, and noise. Furthermore, the theory discusses the fundamental unit of information. Information theory also describes the elements and the quantities of general communications systems and explains the concept of uncertainty as it relates to information.
Overview
Information theory was founded in 1948 when a mathematician named Claude Shannon, who was working at Bell Telephone Laboratories, published a paper titled “A Mathematical Theory of Communication.” The paper focused on the communication of information and the most efficient way of transmitting it. Mathematicians and scientists throughout the world showed interest in Shannon’s paper, and the information theory discipline was soon born.
Information theory involves several elements. Perhaps the most basic of these elements is that information can be measured as a physical quantity. In other words, information can be handled as if it has mass or density. One of the most significant elements of the theory is entropy. Shannon explained entropy is equivalent to a shortage in a message’s information content. This means the message contains a degree of uncertainty or randomness. Shannon explained what happens in a noisy conversation. He proved that signals could be sent without distortion and could be received with accuracy, as long as the message was encoded with a self-checking feature.
Information theory also focuses on the fundamental unit of information. This unit, which is called a bit, is a yes/no or on/off situation. It can be expressed by the binary digits 1 and 0, which represent Boolean two-value binary algebra. A 1 means the power is on, while a 0 means the power is off. Combinations of bits allow for more complicated information. For example, something seen by the human eye may be measured in bits, as the retina records either light (yes/on) or dark (no/off). The combination of these light/dark situations comprises the complete picture.
Some of information theory’s other elements involve general communications systems. It states that these systems include a source of information that serves as a transmitting device and turns the information (the message) into a form that can be transmitted by a certain means. Furthermore, these systems include the channel that transmits the message. General communications systems also have a receiving device that decodes the message, turning it back into a form that is close to the original. The systems include the destination of the message or the intended recipient. Lastly, general communications systems have a source of noise, which is distortion or interference. This noise changes the message during transmission.
The theory further explains the quantities that general communications systems yield. The theory suggests that information is a degree of order, or nonrandomness. This nonrandomness can be measured mathematically, and so a mathematical characterization of general communications systems yields certain quantities. These include the rate that information is generated at the source; the channel’s capacity for handling information; and the average amount of information in the message.
The theory also describes uncertainty as it relates to information. Uncertainty is the process of selecting objects from a set of objects. For example, if a device produces three symbols (A, B, or C), an individual is uncertain which symbol will be produced next. Uncertainty decreases when the symbol appears; the individual has received information. In this sense, information is a decrease in uncertainty. Furthermore, uncertainty may be measured. In this example, the device has an uncertainty of three symbols. However, this kind of measurement does not work if a second device is combined with the first one. For instance, if a second device produces two symbols (1 and 2), it would have an uncertainty of two symbols. But if the second device were combined with the first device, the combined device would then have six possibilities—A1, A2, B1, B2, C1, and C2. The device would now have an uncertainty of six symbols. The average person, however, does not view information in this way. For example, if an individual receives one book and then receives another book, he or she will add the amount of information that has been received. In other words, once a person receives the second book, the individual would state that he or she received twice as much information, because one book is added to another.
Many of the techniques used with information theory are derived from the science of probability. For example, probability is used when given estimates of an information transmission’s accuracy under noise interference. Furthermore, probability is used with encoding and decoding approaches to reduce uncertainty or error.
Other disciplines have developed from information theory, including coding theory. Coding theory, which is also called algebraic coding theory, involves the design of codes that correct errors and allow for reliable information transmission over noisy channels. Coding theory utilizes several algebraic techniques that deal with group theory, finite fields, and polynomial algebra. Information theory continues to evolve, as many of today’s experimentalists and theorists still show interest in the theory.
Bibliography
New York University. “Claude Shannon.” New York University. New York University. Web. 24 Nov. 2014. <http://www.nyu.edu/pages/linguistics/courses/v610003/shan.html>
Rouse, Margaret. “Information Theory.” TechTarget. TechTarget. Web. 24 Nov. 2014. <http://searchnetworking.techtarget.com/definition/information-theory>
Weisstein, Eric W. “Coding Theory.” Wolfram MathWorld. Wolfram Research, Inc. Web. 24 Nov. 2014. <http://mathworld.wolfram.com/CodingTheory.html>