Artificial neural networks

SUMMARY: Artificial neural networks use sophisticated mathematical algorithms and computational functions to simulate biological neural networks.

The term “neural networks” is generally applied to the systems of biological or artificial neurons. More often, it is used in application to artificial neural networks that are designed to reproduce some human brain functions, such as information processing, memory, and pattern recognition. However, this term is also used for biological neural networks, for which the term “neural system” is more common. The beginning of modern neural network research is credited to neuroscientist Warren McCulloch and mathematician Walter Pitts in 1943. McCulloch had spent decades pondering the logic of the nervous system (for example, what allows people to think or feel) before beginning his collaboration with Pitts. He specifically credited Pitts’s knowledge of modular arithmetic for the success of their joint work, which produced the McCulloch–Pitts Theory of Formal Neural Networks. Their research suggests that any computable function can be completely realized by a McCulloch–Pitts artificial neural network, though some such networks would be impractically large.

Artificial Neural Networks

Artificial neural networks are mathematical tools or physical devices that function similarly to biological neural systems. They consist of building blocks, called “artificial neurons,” which resemble the structure of real neurons. Each biological neuron includes three major parts: dendrites, soma, and axon (see Figure 1A). Correspondingly, each artificial neuron also consists of three major parts: inputs (or “dendrites”), transformation function (“soma”), and output (“axon”) (see Figure 1B). The terminology that is generally used for biological neurons is also often applied to artificial neurons.

Modern neural networks use data analysis and non-linear statistical methods to model complex relationships between inputs and outputs or to find patterns. Bayesian methods of inference, named for Thomas Bayes, are increasingly employed. Graph theory and geometry are also very useful for mapping neural networks, assessing their capabilities, and studying pattern classification. Artificial neural networks are applied to a variety of problems in science, industry, and finance in which people must draw conclusions and make decisions from noisy and incomplete data. They can perform pattern recognition and pattern classification, time series analysis and prediction, function approximation, and signal processing. Several types of artificial neural networks were developed for the specific problems for which they can find the best solution. The most famous of them are single- and multi-layer perceptrons; Hopfield neural networks, named for John Hopfield; self-organizing Kohonen maps, named for Tuevo Kohonen; and Boltzmann machines, named for the Ludwig Boltzmann distribution. Regardless of the type of neural network or the problem it is designed to solve, the output is some mathematical function of the inputs, often involving probability distributions. As examples, consider functions of the three types of the artificial neural networks represented in Figure 2.

The first, single-layer perceptron consists of one layer of artificial neurons and was designed for pattern recognition and classification problems (see Figure 2A). The input pattern of signals si is fed to each neuron in the perceptron with different weights, wij. Then the signals are added in each jth neuron to form a weighted sum Σiwijsi, which is processed by a transformation (nonlinear) function, resulting in a pattern of the output signals oj. Thus, the pattern of output signals oj is determined by the set of weights wij, and this set of weights forms a memory in the neural network. To obtain the desired response pattern dj to a given input pattern si, the perceptron is required to be “trained.” Training (or learning) procedures consists of the method that adjusts neural network weights wij that form desired output pattern dj.

Because of limited capability of single-layer perceptrons (for example, they cannot reproduce “exclusive OR” logical operations), the multilayer perceptrons (see Figure 2B) became very popular for different problems in pattern recognition and classification. Inclusion of one or more “hidden” layers into the neural networks increased their learning capability and performance. Multilayer perceptrons are learned by so-called backpropagation algorithm that changes weights wij in all layers to ensure desired output in the last layer.

Both single-layer and multilayer perceptrons belong to a class of feedforward neural networks, as connections between the neurons do not form closed loops (see Figures 2A and 2B), and information transfers only in one direction, from the input to the output. A Hopfield neural network is a representative of another class, recurrent artificial neural networks, with bidirectional flow of information (see Figure 2C). Each neuron in this network is connected to the others with symmetric bidirectional connections, and its output is calculated in a way similar to that for perceptrons. A Hopfield neural network runs by cycles. During one cycle, the output of each neuron is calculated using external inputs and neural outputs from the previous cycle. These neuronal outputs become their inputs, with corresponding weights and transformation function, during the next cycle. Neural outputs are recalculated for each cycle until the system reaches a steady state. This steady state pattern of neural outputs represents a stored pattern in the Hopfield neural network. Information in Hopfield neural networks, as in perceptrons, is stored in the weights, wij.

Bibliography

Coolen, A. C. C. “A Beginner’s Guide to the Mathematics of Neural Networks.” In Concepts for Neural Networks. Edited by L. J. Landau and J. G. Taylor. Springer, 1998.

Haykin, Simon. Neural Networks: A Comprehensive Foundation. 2nd ed. Prentice Hall, 1999.

Kandel, Eric, James Schwartz, and Thomas Jessell. Principles of Neural Science. 4th ed. McGraw-Hill, 2000.

Rumelhart, David E., James L. McClelland, and the PDP Research Group. Parallel Distributed Processing. Vol. 1. Foundations. MIT Press, 1986.

"What Is a Neural Network?" IBM, www.ibm.com/think/topics/neural-networks. Accessed 3 Feb. 2025.