This document has been provided to you by 123seminarsonly.com
1111101010110000001101000101110011110000001100001001110011110110100111100011101001010010110111101011111010010000011100000101110000011011000111110
1101101010110010100111100111011001110000000110001011110011010100100111001001100001010010011111100001011010111110111110101011000000110100010111001111000000110000100111001111011010011110001110100101001011011110101111101001000001110000010111000011011000111110110110101011001010011110011101100111000000011000101111001101010010011100100110000101001001111110000101101011111011111010101100000011010001011100
1111000000110000100111001111011010011110001110100101001011011110101111101001000001110000010111000001101100011111011011010101100101001111001110110
0111000000011000101111001101010010011100100110000101001001111110000101101011111011111010101100000011010001011100111100000011000010011100111101101001111000111010010100101101111010111110100100000111000001011100001101100011111011011010101100101001111001110110011100000001100010111100110101001001110010011000010100100111111000010110101111101111101010110000001101000101110011110000001100001001110011110110
1001111000111010010100101101111010111110100100000111000001011100000110110001111101101101010110010100111100111011001110000000110001011110011010100
1001110010011000010100100111111000010110101111101111101010110000001101000101110011110000001100001001110011110110100111100011101001010010110111101011111010010000011100000101110000110110001111101101101010110010100111100111011001110000000110001011110011010100100111001001100001010010011111100001011010111110111110101011000000110100010111001111000000110000100111001111011010011110001110100101001011011110
1110110100010001000001110110111100000111100000111100101111000111001001010000110101100001101001011010110100100100110010110010101100000111010011011011110110000001001001010100110100000111101010110100001110100111100011110010111101100001000010011010110111000101111011010001000100000111011011110000011110000011110010111100011100100101000011010110000110100101101011010010010011001011001010110000011101001101
1011110110000001001001010100110100000111101010110100001110100111100011110010111101100001000010011010110111000101111011010001000100000111011011110000011110000011110010111100011100100101000011010110000110100101101011010010010011001011001010110000011101001101101111011000000100100101010011010000011110101011010000111010011110001111001011110110000100001001101011011100010111101101000100010000011101101111
0000011110000011110010111100011100100101000011010110000110100101101011010010010011001011001010110000011101001101101111011000000100100101010011010000011110101011010000111010011110001111001011110110000100001001101011011100010111101101000100010000011101101111000001111000001111001011110001110010010100001101011000011010010110101101001001001100101100101011000001110100110110111101100000010010010101001101
0000011110101011010000111010011110001111001011110110000100001001101011011100010111101101000100010000011101101111000001111000001111001011110001110010010100001101011000011010010110101101001001001100101100101011000001110100110110111101100000010010010101001101000001111010101101000011101001111000111100101111011000010000100110101101110001011110110100010001000001110110111100000111100000111100101111000111
0010010100001101011000011010010110101101001001001100101100101011000001110100110110111101100000010010010101001101000001111010101101000011101001111000111100101111011000010000100110101101110001011110110100010001000001110110111100000111100000111100101111000111001001010000110101100001101001011010110100100100110010110010101100000111010011011011110110000001001001010100110100000111101010110100001110100111
1000111100101111011000010000100110101101110001011110110100010001000001110110111100000111100000111100101111000111001001010000110101100001101001011010110100100100110010110010101100000111010011011011110110000001001001010100110100000111101010110100001110100111100011110010111101100001000010011010110111000101111011010001000100000111011011110000011110000011110010111100011100100101000011010110000110100101
1010110100100100110010110010101100000111010011011011110110000001001001010100110100000111101010110100001110100111100011110010111101100001000010011010110111000101111011010001000100000111011011110000011110000011110010111100011100100101000011010110000110100101101011010010010011001011001010110000011101001101101111011000000100100101010011010000011110101011010000111010011110001111001011110110000100001001
Neural Networks
1. INTRODUCTION
The power and speed of modern digital computers is truly astounding. No human can ever hope to compute a million operations a second. However, there are some tasks for which even the most powerful computers cannot compete with the human brain, perhaps not even with the intelligence of an earthworm. Imagine the power of the machine which has the abilities of both computers and humans. It would be the most remarkable thing ever. And all humans can live happily ever after (or will they?).
Before discussing the specifics of artificial neural nets though, let us examine what makes real neural nets - brains - function the way they do. Perhaps the single most important concept in neural net research is the idea of connection strength. Neuroscience has given us good evidence for the idea that connection strengths - that is, how strongly one neuron influences those neurons connected to it – are the real information holders in the brain. Learning, repetition of a task, even exposure to a new or continuing stimulus can cause the brain's connection strengths to change, some synaptic connections becoming reinforced and new ones are being created, others weakening or in some cases disappearing altogether.
The second essential element of neural connectivity is the excitation/inhibition distinction. In human brains, each neuron is
either excitatory or inhibitory, which is to say that its activation will either increase the firing rates of connected neurons, or decrease the rate, respectively. The amount of excitation or inhibition produced is of course, dependent on the connection strength - a stronger connection means more inhibition or excitation, a weaker connection means less.
The third important component in determining a neuron's response is called the transfer function. Without getting into more technical detail, the transfer function describes how a neuron's firing rate varies with the input it receives. A very sensitive neuron may fire with very little input, for example. A neuron may have a threshold, and fire rarely below threshold, and vigorously above it.
Each of these behaviors can be represented mathematically, and that representation is called the transfer function. It is often convenient to forget the transfer function, and think of the neurons as being simple addition machines, more activity in equals more activity out. This is not really accurate though, and to develop a good understanding of an artificial neural network, the transfer function must be taken into account. Armed with these three concepts: Connection Strength, Inhibition/Excitation, and the Transfer Function, we can now look at how artificial neural nets are constructed. In theory, an artificial neuron (often called a 'node') captures all the important elements of a
biological one. Nodes are connected to each other and the strength of that connection is normally given a numeric value between -1.0 for maximum inhibition, to +1.0 for maximum excitation. All values between the two are acceptable, with higher magnitude values indicating stronger connection strength. The transfer function in artificial neurons whether in a computer simulation, or actual microchips wired together, is typically built right into the nodes' design.
1111101010110000001101000101110011110000001100001001110011110110100111100011101001010010110111101011111010010000011100000101110000011011000111110
1101101010110010100111100111011001110000000110001011110011010100100111001001100001010010011111100001011010111110111110101011000000110100010111001111000000110000100111001111011010011110001110100101001011011110101111101001000001110000010111000011011000111110110110101011001010011110011101100111000000011000101111001101010010011100100110000101001001111110000101101011111011111010101100000011010001011100
1111000000110000100111001111011010011110001110100101001011011110101111101001000001110000010111000001101100011111011011010101100101001111001110110
0111000000011000101111001101010010011100100110000101001001111110000101101011111011111010101100000011010001011100111100000011000010011100111101101001111000111010010100101101111010111110100100000111000001011100001101100011111011011010101100101001111001110110011100000001100010111100110101001001110010011000010100100111111000010110101111101111101010110000001101000101110011110000001100001001110011110110
1001111000111010010100101101111010111110100100000111000001011100000110110001111101101101010110010100111100111011001110000000110001011110011010100
1001110010011000010100100111111000010110101111101111101010110000001101000101110011110000001100001001110011110110100111100011101001010010110111101011111010010000011100000101110000110110001111101101101010110010100111100111011001110000000110001011110011010100100111001001100001010010011111100001011010111110111110101011000000110100010111001111000000110000100111001111011010011110001110100101001011011110
1011111010010000011100000101110000011011000111110110110101011001010011110011101100111000000011000101111001101010010011100100110000101001001111110
0001011010111110111110101011000000110100010111001111000000110000100111001111011010011110001110100101001011011110101111101001000001110000010111000
Neural Networks approaches this problem by trying to mimic the structure and function of our nervous system. Many researchers believe that AI (Artificial Intelligence) and neural networks are completely opposite in their approach. Conventional AI is based on the symbol system hypothesis. Loosely speaking, a symbol system consists of indivisible entities called symbols, which can form more complex entities, by simple rules. The hypothesis then states that such a system is capable of and is necessary for intelligence.
The general belief is that Neural Networks is a sub-symbolic science. Before symbols themselves are recognized, some thing must be done so that conventional AI can then manipulate those symbols. To
make this point clear, consider symbols such as cow, grass, house etc. Once these symbols and the "simple rules" which govern them are known, conventional AI can perform miracles. But to discover that something is a cow is not trivial. It can perhaps be done using conventional AI and symbols such as - white, legs, etc. But it would be tedious and certainly, when you see a cow, you instantly recognize it to be so, without counting its legs. Progress in this area can be made only by breaking this line of distinction between AI and Neural Networks, and combining the results obtained in both, towards a unified framework.
2. WHAT IS A NEURON?
Most neurons like the one shown here, consist of a cell body plus an axon and many dendrites. The axon is a protuberance that delivers the neuron’s output to connections with other neurons. Dendrites are
1111101010110000001101000101110011110000001100001001110011110110100111100011101001010010110111101011111010010000011100000101110000011011000111110
1101101010110010100111100111011001110000000110001011110011010100100111001001100001010010011111100001011010111110111110101011000000110100010111001111000000110000100111001111011010011110001110100101001011011110101111101001000001110000010111000011011000111110110110101011001010011110011101100111000000011000101111001101010010011100100110000101001001111110000101101011111011111010101100000011010001011100
1111000000110000100111001111011010011110001110100101001011011110101111101001000001110000010111000001101100011111011011010101100101001111001110110
0111000000011000101111001101010010011100100110000101001001111110000101101011111011111010101100000011010001011100111100000011000010011100111101101001111000111010010100101101111010111110100100000111000001011100001101100011111011011010101100101001111001110110011100000001100010111100110101001001110010011000010100100111111000010110101111101111101010110000001101000101110011110000001100001001110011110110
1001111000111010010100101101111010111110100100000111000001011100000110110001111101101101010110010100111100111011001110000000110001011110011010100
1001110010011000010100100111111000010110101111101111101010110000001101000101110011110000001100001001110011110110100111100011101001010010110111101011111010010000011100000101110000110110001111101101101010110010100111100111011001110000000110001011110011010100100111001001100001010010011111100001011010111110111110101011000000110100010111001111000000110000100111001111011010011110001110100101001011011110
1011111010010000011100000101110000011011000111110110110101011001010011110011101100111000000011000101111001101010010011100100110000101001001111110
0001011010111110111110101011000000110100010111001111000000110000100111001111011010011110001110100101001011011110101111101001000001110000010111000
0001101100011111011011010101100101001111001110110011100000001100010111100110101001001110010011000010100100111111000010110101111101111101010110000
0011010001011100111100000011000010011100111101101001111000111010010100101101111010111110100100000111000001011100000110110001111101101101010110010
0110000100001001101011011100010111101101000100010000011101101111000001111000001111001011110001110010010100001101011000011010010110101101001001001100101100101011000001110100110110111101100000010010010101001101000001111010101101000011101001111000111100101111011000010000100110101101110001011110110100010001000001110110111100000111100000111100101111000111001001010000110101100001101001011010110100100100
1100101100101011000001110100110110111101100000010010010101001101000001111010101101000011101001111000111100101111011000010000100110101101110001011110110100010001000001110110111100000111100000111100101111000111001001010000110101100001101001011010110100100100110010110010101100000111010011011011110110000001001001010100110100000111101010110100001110100111100011110010111101100001000010011010110111000101
1110110100010001000001110110111100000111100000111100101111000111001001010000110101100001101001011010110100100100110010110010101100000111010011011011110110000001001001010100110100000111101010110100001110100111100011110010111101100001000010011010110111000101111011010001000100000111011011110000011110000011110010111100011100100101000011010110000110100101101011010010010011001011001010110000011101001101
1011110110000001001001010100110100000111101010110100001110100111100011110010111101100001000010011010110111000101111011010001000100000111011011110000011110000011110010111100011100100101000011010110000110100101101011010010010011001011001010110000011101001101101111011000000100100101010011010000011110101011010000111010011110001111001011110110000100001001101011011100010111101101000100010000011101101111
0000011110000011110010111100011100100101000011010110000110100101101011010010010011001011001010110000011101001101101111011000000100100101010011010000011110101011010000111010011110001111001011110110000100001001101011011100010111101101000100010000011101101111000001111000001111001011110001110010010100001101011000011010010110101101001001001100101100101011000001110100110110111101100000010010010101001101
0000011110101011010000111010011110001111001011110110000100001001101011011100010111101101000100010000011101101111000001111000001111001011110001110010010100001101011000011010010110101101001001001100101100101011000001110100110110111101100000010010010101001101000001111010101101000011101001111000111100101111011000010000100110101101110001011110110100010001000001110110111100000111100000111100101111000111
0010010100001101011000011010010110101101001001001100101100101011000001110100110110111101100000010010010101001101000001111010101101000011101001111000111100101111011000010000100110101101110001011110110100010001000001110110111100000111100000111100101111000111001001010000110101100001101001011010110100100100110010110010101100000111010011011011110110000001001001010100110100000111101010110100001110100111
1000111100101111011000010000100110101101110001011110110100010001000001110110111100000111100000111100101111000111001001010000110101100001101001011010110100100100110010110010101100000111010011011011110110000001001001010100110100000111101010110100001110100111100011110010111101100001000010011010110111000101111011010001000100000111011011110000011110000011110010111100011100100101000011010110000110100101
1010110100100100110010110010101100000111010011011011110110000001001001010100110100000111101010110100001110100111100011110010111101100001000010011010110111000101111011010001000100000111011011110000011110000011110010111100011100100101000011010110000110100101101011010010010011001011001010110000011101001101101111011000000100100101010011010000011110101011010000111010011110001111001011110110000100001001
protuberances that provide plenty of surface area, facilitating connection with the axons of other neurons. Dendrites often divide a great deal, forming extremely bushy dendritic trees. Axons divide to some extent but far less than dendrites.
A neuron does nothing unless the collective influence of all its inputs reaches a threshold level. Whenever that threshold level is reached the neuron produces a full-strength output in the form of a narrow pulse that proceeds from the cell body, down the axon and into the axon’s branches. Whenever this happens, the neuron is said to fire. Because a neuron either fires or does nothing it is said to bean all-or-none device.
Axons influence dendrites over narrow gaps called synapses. Stimulation at some synapses encourages neurons to fire. Stimulation at others discourages neurons from firing. There is
mounting evidence that learning takes place in the vicinity of synapses and has something to do with the degree to which synapses translate the pulse traveling down one neuron’s axon into excitation or inhibition of the next neuron. Through those connections electrical pulses are transmitted, and information is carried in the timing and the frequency with which these pulses are emitted. So, our neuron receives information from other neurons, processes it and then relays this information to other neurons. A question, which immediately arises, is: of what form does this processing take? Clearly the neuron must generate some kind of output based on the cumulative input. We still don't know the exact answer to the question as to what happens in a biological neuron. However, we do know that our neuron integrates the pulses that arrive and when this integration exceeds a certain limit, our neuron in turn emits a pulse.
Finally, one more thing that you should know is that dendrites modify the amplitude of the pulses traveling through them. This modification varies with time, as the network `learns'. So we could assume that
when a connection (dendrite) is very strong, the importance of the neuron from which this connection come has an important role in the network, and on the other hand, when a connection is very narrow, the importance of the neuron from which the connection comes from is less high. Thus the neural network stores information in the pattern of connection weights.
The nervous system may be viewed as a three-stage system, as depicted in the block diagram. Control to the system in the brain, represented by the neural (nerve) net, which continually receives information, perceives it, and make appropriate decisions. Two sets of arrows are shown in the figure. Those pointing from left to right indicate the forward transmission of information-bearing signals through the system. The arrows pointing from right to left signify the presence of feedback in the system. The receptors convert stimuli from the human body or the external environment into electrical impulses that convey information to the neural net (brain). The effectors convert electrical impulses generated by the neural net into
discernible responses as system outputs.
The number of neurons in the human brain is staggering. Current estimates suggests there maybe of the order of 1011 neurons per person. If the number of neurons is staggering, the number of synapses must be toppling. In the cerebellum- that part of the brain that is crucial to motor coordination- a single neuron may receive inputs from as many as 105 synapses. In as much as most of the neurons in the brain are in cerebellum, each brain has in the order of 1016 synapses.
Compared to the biological brain, a typical artificial neural network (ANN) is not likely to have more than 1,000 artificial neurons. The artificial neurons we use to built our neural networks are truly primitive in comparison to those found in the brain .The neural networks we are presently able to design are just as primitive compared to the local circuits and the interregional circuits in the brain. What is really satisfying, is the remarkable progress that we have made on so many fronts during the past two decades.
Functioning of the Nervous System
The nature of interconnections between 2 neurons can be such that - one neuron can either stimulate or inhibit the other. An interaction can take place only if there is an edge between 2 neurons. If neuron A is connected to neuron B as below with a weight w, then if A is stimulated sufficiently, it sends a signal to B. The signal depends on the weight w, and the nature of the signal, whether it is stimulating or inhibiting. This depends on whether w is positive or negative. If sufficiently strong signals are sent, B may become stimulated.
Note that A will send a signal only if it is stimulated sufficiently, that is, if its stimulation is more than its threshold. Also if it sends a signal, it will send it to all nodes to which it is connected. The threshold for different neurons may be different. If many neurons send signals to A, the combined stimulus may be more than the threshold.
Next if B is stimulated sufficiently, it may trigger a signal to all neurons to which it is connected. Depending on the complexity of the structure, the overall functioning may be very complex but the functioning of individual neurons is as simple as this. Because of this we may dare to try to simulate this using software or even special purpose hardware.
4. ARTIFICIAL NEURON.
A simple neuron
An artificial neuron is a device with many inputs and one output. The neuron has two modes of operation; the training mode and the using mode. In the training mode, the neuron can be trained to fire (or not),
for particular input patterns. In the using mode, when a taught input pattern is detected at the input, its associated output becomes the current output. If the input pattern does not belong in the taught list of input patterns, the firing rule is used to determine whether to fire or not.
1111101010110000001101000101110011110000001100001001110011110110100111100011101001010010110111101011111010010000011100000101110000011011000111110
1101101010110010100111100111011001110000000110001011110011010100100111001001100001010010011111100001011010111110111110101011000000110100010111001111000000110000100111001111011010011110001110100101001011011110101111101001000001110000010111000011011000111110110110101011001010011110011101100111000000011000101111001101010010011100100110000101001001111110000101101011111011111010101100000011010001011100
1111000000110000100111001111011010011110001110100101001011011110101111101001000001110000010111000001101100011111011011010101100101001111001110110
0111000000011000101111001101010010011100100110000101001001111110000101101011111011111010101100000011010001011100111100000011000010011100111101101001111000111010010100101101111010111110100100000111000001011100001101100011111011011010101100101001111001110110011100000001100010111100110101001001110010011000010100100111111000010110101111101111101010110000001101000101110011110000001100001001110011110110
1000111100101111011000010000100110101101110001011110110100010001000001110110111100000111100000111100101111000111001001010000110101100001101001011010110100100100110010110010101100000111010011011011110110000001001001010100110100000111101010110100001110100111100011110010111101100001000010011010110111000101111011010001000100000111011011110000011110000011110010111100011100100101000011010110000110100101
1010110100100100110010110010101100000111010011011011110110000001001001010100110100000111101010110100001110100111100011110010111101100001000010011010110111000101111011010001000100000111011011110000011110000011110010111100011100100101000011010110000110100101101011010010010011001011001010110000011101001101101111011000000100100101010011010000011110101011010000111010011110001111001011110110000100001001
1111101010110000001101000101110011110000001100001001110011110110100111100011101001010010110111101011111010010000011100000101110000011011000111110
1101101010110010100111100111011001110000000110001011110011010100100111001001100001010010011111100001011010111110111110101011000000110100010111001111000000110000100111001111011010011110001110100101001011011110101111101001000001110000010111000011011000111110110110101011001010011110011101100111000000011000101111001101010010011100100110000101001001111110000101101011111011111010101100000011010001011100
0000011110000011110010111100011100100101000011010110000110100101101011010010010011001011001010110000011101001101101111011000000100100101010011010000011110101011010000111010011110001111001011110110000100001001101011011100010111101101000100010000011101101111000001111000001111001011110001110010010100001101011000011010010110101101001001001100101100101011000001110100110110111101100000010010010101001101
0000011110101011010000111010011110001111001011110110000100001001101011011100010111101101000100010000011101101111000001111000001111001011110001110010010100001101011000011010010110101101001001001100101100101011000001110100110110111101100000010010010101001101000001111010101101000011101001111000111100101111011000010000100110101101110001011110110100010001000001110110111100000111100000111100101111000111
0010010100001101011000011010010110101101001001001100101100101011000001110100110110111101100000010010010101001101000001111010101101000011101001111000111100101111011000010000100110101101110001011110110100010001000001110110111100000111100000111100101111000111001001010000110101100001101001011010110100100100110010110010101100000111010011011011110110000001001001010100110100000111101010110100001110100111
1000111100101111011000010000100110101101110001011110110100010001000001110110111100000111100000111100101111000111001001010000110101100001101001011010110100100100110010110010101100000111010011011011110110000001001001010100110100000111101010110100001110100111100011110010111101100001000010011010110111000101111011010001000100000111011011110000011110000011110010111100011100100101000011010110000110100101
1010110100100100110010110010101100000111010011011011110110000001001001010100110100000111101010110100001110100111100011110010111101100001000010011010110111000101111011010001000100000111011011110000011110000011110010111100011100100101000011010110000110100101101011010010010011001011001010110000011101001101101111011000000100100101010011010000011110101011010000111010011110001111001011110110000100001001
Firing Rules
The firing rule is an important concept in neural networks and accounts for their high flexibility. A firing rule determines how one calculates whether a neuron should fire for any input pattern. It relates to all the input patterns, not only the ones on which the node was trained.
A simple firing rule can be implemented by using Hamming distance technique. The rule goes as follows:
Take a collection of training patterns for a node, some of which cause it to fire (the 1-taught set of patterns) and others, which prevent it from doing so (the 0-taught set). Then the patterns not in the collection cause the node to fire if, on comparison , they have more input elements in common with the 'nearest' pattern in the 1-taught set than with the 'nearest' pattern in the 0-taught set. If there is a tie, then the pattern remains in the undefined state.
For example, a 3-input neuron is taught to output 1 when the input (X1,X2 and X3) is 111 or 101 and to output 0 when the input is 000 or 001. Then, before applying the firing rule, the truth table is;
X1 / 0 / 0 / 0 / 0 / 1 / 1 / 1 / 1X2: / 0 / 0 / 1 / 1 / 0 / 0 / 1 / 1
X3: / 0 / 1 / 0 / 1 / 0 / 1 / 0 / 1
OUT: / 0 / 0 / 0/1 / 0/1 / 0/1 / 1 / 0/1 / 1
As an example of the way the firing rule is applied, take the pattern 010. It differs from 000 in 1 element, from 001 in 2 elements, from
1111101010110000001101000101110011110000001100001001110011110110100111100011101001010010110111101011111010010000011100000101110000011011000111110
1101101010110010100111100111011001110000000110001011110011010100100111001001100001010010011111100001011010111110111110101011000000110100010111001111000000110000100111001111011010011110001110100101001011011110101111101001000001110000010111000011011000111110110110101011001010011110011101100111000000011000101111001101010010011100100110000101001001111110000101101011111011111010101100000011010001011100
1111000000110000100111001111011010011110001110100101001011011110101111101001000001110000010111000001101100011111011011010101100101001111001110110
0111000000011000101111001101010010011100100110000101001001111110000101101011111011111010101100000011010001011100111100000011000010011100111101101001111000111010010100101101111010111110100100000111000001011100001101100011111011011010101100101001111001110110011100000001100010111100110101001001110010011000010100100111111000010110101111101111101010110000001101000101110011110000001100001001110011110110
1001111000111010010100101101111010111110100100000111000001011100000110110001111101101101010110010100111100111011001110000000110001011110011010100
1001110010011000010100100111111000010110101111101111101010110000001101000101110011110000001100001001110011110110100111100011101001010010110111101011111010010000011100000101110000110110001111101101101010110010100111100111011001110000000110001011110011010100100111001001100001010010011111100001011010111110111110101011000000110100010111001111000000110000100111001111011010011110001110100101001011011110
1011111010010000011100000101110000011011000111110110110101011001010011110011101100111000000011000101111001101010010011100100110000101001001111110
0001011010111110111110101011000000110100010111001111000000110000100111001111011010011110001110100101001011011110101111101001000001110000010111000
0001101100011111011011010101100101001111001110110011100000001100010111100110101001001110010011000010100100111111000010110101111101111101010110000
0011010001011100111100000011000010011100111101101001111000111010010100101101111010111110100100000111000001011100000110110001111101101101010110010
1001111001110110011100000001100010111100110101001001110010011000010100100111111000010110101111101111101010110000001101000101110011110000001100001001110011110110100111100011101001010010110111101011111010010000011100000101110000110110001111101101101010110010100111100111011001110000000110001011110011010100100111001001100001010010011111100001011010111110111110101011000000110100010111001111000000110000
1001110011110110100111100011101001010010110111101011111010010000011100000101110000011011000111110110110101011001010011110011101100111000000011000
1011110011010100100111001001100001010010011111100001011010111110111110101011000000110100010111001111001101101111000001111000001111001011110001110010010100001101011000011010010110101101001001001100101100101011000001110100110110111101100000010010010101001101000001111010101101000011101001111000111100101111011000010000100110101101110001011110110100010001000001110110111100000111100000111100101111000111
0010010100001101011000011010010110101101001001001100101100101011000001110100110110111101100000010010010101001101000001111010101101000011101001111000111100101111011000010000100110101101110001011110110100010001000001110110111100000111100000111100101111000111001001010000110101100001101001011010110100100100110010110010101100000111010011011011110110000001001001010100110100000111101010110100001110100111
1100101100101011000001110100110110111101100000010010010101001101000001111010101101000011101001111000111100101111011000010000100110101101110001011110110100010001000001110110111100000111100000111100101111000111001001010000110101100001101001011010110100100100110010110010101100000111010011011011110110000001001001010100110100000111101010110100001110100111100011110010111101100001000010011010110111000101
1110110100010001000001110110111100000111100000111100101111000111001001010000110101100001101001011010110100100100110010110010101100000111010011011011110110000001001001010100110100000111101010110100001110100111100011110010111101100001000010011010110111000101111011010001000100000111011011110000011110000011110010111100011100100101000011010110000110100101101011010010010011001011001010110000011101001101
1011110110000001001001010100110100000111101010110100001110100111100011110010111101100001000010011010110111000101111011010001000100000111011011110000011110000011110010111100011100100101000011010110000110100101101011010010010011001011001010110000011101001101101111011000000100100101010011010000011110101011010000111010011110001111001011110110000100001001101011011100010111101101000100010000011101101111
0000011110000011110010111100011100100101000011010110000110100101101011010010010011001011001010110000011101001101101111011000000100100101010011010000011110101011010000111010011110001111001011110110000100001001101011011100010111101101000100010000011101101111000001111000001111001011110001110010010100001101011000011010010110101101001001001100101100101011000001110100110110111101100000010010010101001101
0000011110101011010000111010011110001111001011110110000100001001101011011100010111101101000100010000011101101111000001111000001111001011110001110010010100001101011000011010010110101101001001001100101100101011000001110100110110111101100000010010010101001101000001111010101101000011101001111000111100101111011000010000100110101101110001011110110100010001000001110110111100000111100000111100101111000111
0010010100001101011000011010010110101101001001001100101100101011000001110100110110111101100000010010010101001101000001111010101101000011101001111000111100101111011000010000100110101101110001011110110100010001000001110110111100000111100000111100101111000111001001010000110101100001101001011010110100100100110010110010101100000111010011011011110110000001001001010100110100000111101010110100001110100111
1000111100101111011000010000100110101101110001011110110100010001000001110110111100000111100000111100101111000111001001010000110101100001101001011010110100100100110010110010101100000111010011011011110110000001001001010100110100000111101010110100001110100111100011110010111101100001000010011010110111000101111011010001000100000111011011110000011110000011110010111100011100100101000011010110000110100101
1010110100100100110010110010101100000111010011011011110110000001001001010100110100000111101010110100001110100111100011110010111101100001000010011010110111000101111011010001000100000111011011110000011110000011110010111100011100100101000011010110000110100101101011010010010011001011001010110000011101001101101111011000000100100101010011010000011110101011010000111010011110001111001011110110000100001001
101 in 3 elements and from 111 in 2 elements. Therefore, the 'nearest' pattern is 000, which belongs, in the 0-taught set. Thus the firing rule requires that the neuron should not fire when the input is 001. On the other hand, 011 is equally distant from two taught patterns that have different outputs and thus the output stays undefined (0/1). By applying the firing in every column the following truth table is obtained;
X1: / 0 / 0 / 0 / 0 / 1 / 1 / 1 / 1X2: / 0 / 0 / 1 / 1 / 0 / 0 / 1 / 1
X3: / 0 / 1 / 0 / 1 / 0 / 1 / 0 / 1
OUT: / 0 / 0 / 0 / 0/1 / 0/1 / 1 / 1 / 1
The difference between the two truth tables is called the generalization of the neuron. Therefore the firing rule gives the neuron a sense of similarity and enables it to respond 'sensibly' to patterns not seen during training.
Pattern Recognition.
An important application of neural networks is pattern recognition. Pattern recognition can be implemented by using a feed-forward (figure 1) neural network that has been trained accordingly. During training, the network is trained to associate outputs with input
patterns. When the network is used, it identifies the input pattern and tries to output the associated output pattern. The power of neural networks comes to life when a pattern that has no output associated with it, is given as an input. In this case, the network gives the output that corresponds to a taught input pattern that is least different from the given pattern.
Figure 1.
For example:
The network of figure 1 is trained to recognise the patterns T and H. The associated patterns are all black and all white respectively as shown below.
If we represent black squares with 0 and white squares with 1 then the truth tables for the 3 neurons after generalization are:
X11: / 0 / 0 / 0 / 0 / 1 / 1 / 1 / 1X12: / 0 / 0 / 1 / 1 / 0 / 0 / 1 / 1
X13: / 0 / 1 / 0 / 1 / 0 / 1 / 0 / 1
OUT: / 0 / 0 / 1 / 1 / 0 / 0 / 1 / 1
1111101010110000001101000101110011110000001100001001110011110110100111100011101001010010110111101011111010010000011100000101110000011011000111110
1101101010110010100111100111011001110000000110001011110011010100100111001001100001010010011111100001011010111110111110101011000000110100010111001111000000110000100111001111011010011110001110100101001011011110101111101001000001110000010111000011011000111110110110101011001010011110011101100111000000011000101111001101010010011100100110000101001001111110000101101011111011111010101100000011010001011100
1111000000110000100111001111011010011110001110100101001011011110101111101001000001110000010111000001101100011111011011010101100101001111001110110
0111000000011000101111001101010010011100100110000101001001111110000101101011111011111010101100000011010001011100111100000011000010011100111101101001111000111010010100101101111010111110100100000111000001011100001101100011111011011010101100101001111001110110011100000001100010111100110101001001110010011000010100100111111000010110101111101111101010110000001101000101110011110000001100001001110011110110
1001111000111010010100101101111010111110100100000111000001011100000110110001111101101101010110010100111100111011001110000000110001011110011010100
1001110010011000010100100111111000010110101111101111101010110000001101000101110011110000001100001001110011110110100111100011101001010010110111101011111010010000011100000101110000110110001111101101101010110010100111100111011001110000000110001011110011010100100111001001100001010010011111100001011010111110111110101011000000110100010111001111000000110000100111001111011010011110001110100101001011011110
1011111010010000011100000101110000011011000111110110110101011001010011110011101100111000000011000101111001101010010011100100110000101001001111110
0001011010111110111110101011000000110100010111001111000000110000100111001111011010011110001110100101001011011110101111101001000001110000010111000
00011011000111110110110101011001010011110011101100111000000011000101111001101010010011100100110000101001001111110000101101011111011111010101100000011010001011100111100000011000010011100111101101001111000111010010100101101111010111110100100000111000001011100000110110001111101101101010110010
1001111001110110011100000001100010111100110101001001110010011000010100100111111000010110101111101111101010110000001101000101110011110000001100001001110011110110100111100011101001010010110111101011111010010000011100000101110000110110001111101101101010110010100111100111011001110000000110001011110011010100100111001001100001010010011111100001011010111110111110101011000000110100010111001111000000110000
1001110011110110100111100011101001010010110111101011111010010000011100000101110000011011000111110110110101011001010011110011101100111000000011000
1011110011010100100111001001100001010010011111100001011010111110111110101011000000110100010111001111001101101111000001111000001111001011110001110010010100001101011000011010010110101101001001001100101100101011000001110100110110111101100000010010010101001101000001111010101101000011101001111000111100101111011000010000100110101101110001011110110100010001000001110110111100000111100000111100101111000111
0010010100001101011000011010010110101101001001001100101100101011000001110100110110111101100000010010010101001101000001111010101101000011101001111000111100101111011000010000100110101101110001011110110100010001000001110110111100000111100000111100101111000111001001010000110101100001101001011010110100100100110010110010101100000111010011011011110110000001001001010100110100000111101010110100001110100111
1000111100101111011000010000100110101101110001011110110100010001000001110110111100000111100000111100101111000111001001010000110101100001101001011010110100100100110010110010101100000111010011011011110110000001001001010100110100000111101010110100001110100111100011110010111101100001000010011010110111000101111011010001000100000111011011110000011110000011110010111100011100100101000011010110000110100101
1010110100100100110010110010101100000111010011011011110110000001001001010100110100000111101010110100001110100111100011110010111101100001000010011010110111000101111011010001000100000111011011110000011110000011110010111100011100100101000011010110000110100101101011010010010011001011001010110000011101001101101111011000000100100101010011010000011110101011010000111010011110001111001011110110000100001001
Top neuron
X21: / 0 / 0 / 0 / 0 / 1 / 1 / 1 / 1X22: / 0 / 0 / 1 / 1 / 0 / 0 / 1 / 1
X23: / 0 / 1 / 0 / 1 / 0 / 1 / 0 / 1
OUT: / 1 / 0/1 / 1 / 0/1 / 0/1 / 0 / 0/1 / 0
Middle neuron
X21: / 0 / 0 / 0 / 0 / 1 / 1 / 1 / 1X22: / 0 / 0 / 1 / 1 / 0 / 0 / 1 / 1
X23: / 0 / 1 / 0 / 1 / 0 / 1 / 0 / 1
OUT: / 1 / 0 / 1 / 1 / 0 / 0 / 1 / 0
Bottom neuron
From the tables it can be seen the following associations can be extracted:
In this case, it is obvious that the output should be all blacks since the input pattern is almost the same as the 'T' pattern.
Here also, it is obvious that the output should be all whites since the input pattern is almost the same as the 'H' pattern.
Here, the top row is 2 errors away from the a T and 3 from an H. So the top output is black. The middle row is 1 error away from both T and H so the output is random. The bottom row is 1 error away from T and 2 away from H. Therefore the output is black. The total output of the network is still in favor of the T shape.
5. ARCHITECTURE OF NEURAL NETWORKS.
Feed-forward networks
Feed-forward ANNs allow signals to travel one way only; from input to output. There is no feedback (loops) i.e. the output of any layer does
not affect that same layer. Feed-forward ANNs tend to be straightforward networks that associate inputs with outputs. They are extensively used in pattern recognition. This type of organization is also referred to as bottom-up or top-down.
1111101010110000001101000101110011110000001100001001110011110110100111100011101001010010110111101011111010010000011100000101110000011011000111110
1101101010110010100111100111011001110000000110001011110011010100100111001001100001010010011111100001011010111110111110101011000000110100010111001111000000110000100111001111011010011110001110100101001011011110101111101001000001110000010111000011011000111110110110101011001010011110011101100111000000011000101111001101010010011100100110000101001001111110000101101011111011111010101100000011010001011100
1111000000110000100111001111011010011110001110100101001011011110101111101001000001110000010111000001101100011111011011010101100101001111001110110
0111000000011000101111001101010010011100100110000101001001111110000101101011111011111010101100000011010001011100111100000011000010011100111101101001111000111010010100101101111010111110100100000111000001011100001101100011111011011010101100101001111001110110011100000001100010111100110101001001110010011000010100100111111000010110101111101111101010110000001101000101110011110000001100001001110011110110
1001111000111010010100101101111010111110100100000111000001011100000110110001111101101101010110010100111100111011001110000000110001011110011010100
1001110010011000010100100111111000010110101111101111101010110000001101000101110011110000001100001001110011110110100111100011101001010010110111101011111010010000011100000101110000110110001111101101101010110010100111100111011001110000000110001011110011010100100111001001100001010010011111100001011010111110111110101011000000110100010111001111000000110000100111001111011010011110001110100101001011011110
1011111010010000011100000101110000011011000111110110110101011001010011110011101100111000000011000101111001101010010011100100110000101001001111110
0001011010111110111110101011000000110100010111001111000000110000100111001111011010011110001110100101001011011110101111101001000001110000010111000
0001101100011111011011010101100101001111001110110011100000001100010111100110101001001110010011000010100100111111000010110101111101111101010110000
0011010001011100111100000011000010011100111101101001111000111010010100101101111010111110100100000111000001011100000110110001111101101101010110010
1001111001110110011100000001100010111100110101001001110010011000010100100111111000010110101111101111101010110000001101000101110011110000001100001001110011110110100111100011101001010010110111101011111010010000011100000101110000110110001111101101101010110010100111100111011001110000000110001011110011010100100111001001100001010010011111100001011010111110111110101011000000110100010111001111000000110000
1001110011110110100111100011101001010010110111101011111010010000011100000101110000011011000111110110110101011001010011110011101100111000000011000
Feedback networks
Feedback networks can have signals traveling in both directions by introducing loops in the network. Feedback networks are very powerful and can get extremely complicated. Feedback networks are dynamic; their 'state' is changing continuously until they reach an equilibrium point. They remain at the equilibrium point until the input changes and a new equilibrium needs to be found. Feedback architectures are also referred to as interactive or recurrent, although the latter term is often used to denote feedback connections in single-layer organizations.
1111101010110000001101000101110011110000001100001001110011110110100111100011101001010010110111101011111010010000011100000101110000011011000111110
1101101010110010100111100111011001110000000110001011110011010100100111001001100001010010011111100001011010111110111110101011000000110100010111001111000000110000100111001111011010011110001110100101001011011110101111101001000001110000010111000011011000111110110110101011001010011110011101100111000000011000101111001101010010011100100110000101001001111110000101101011111011111010101100000011010001011100
1111000000110000100111001111011010011110001110100101001011011110101111101001000001110000010111000001101100011111011011010101100101001111001110110
0111000000011000101111001101010010011100100110000101001001111110000101101011111011111010101100000011010001011100111100000011000010011100111101101001111000111010010100101101111010111110100100000111000001011100001101100011111011011010101100101001111001110110011100000001100010111100110101001001110010011000010100100111111000010110101111101111101010110000001101000101110011110000001100001001110011110110
1001111000111010010100101101111010111110100100000111000001011100000110110001111101101101010110010100111100111011001110000000110001011110011010100
1001110010011000010100100111111000010110101111101111101010110000001101000101110011110000001100001001110011110110100111100011101001010010110111101011111010010000011100000101110000110110001111101101101010110010100111100111011001110000000110001011110011010100100111001001100001010010011111100001011010111110111110101011000000110100010111001111000000110000100111001111011010011110001110100101001011011110
1011111010010000011100000101110000011011000111110110110101011001010011110011101100111000000011000101111001101010010011100100110000101001001111110
1010110100100100110010110010101100000111010011011011110110000001001001010100110100000111101010110100001110100111100011110010111101100001000010011010110111000101111011010001000100000111011011110000011110000011110010111100011100100101000011010110000110100101101011010010010011001011001010110000011101001101101111011000000100100101010011010000011110101011010000111010011110001111001011110110000100001001
Network layers
The commonest type of artificial neural network consists of three groups, or layers, of units: a layer of "input" units is connected to a layer of "hidden" units, which is connected to a layer of "output" units. (see the figure)
-The activity of the input units represents the raw information that is fed into the network.
-The activity of each hidden unit is determined by the activities of the input units and the weights on the connections between the input and the hidden units.
-The behavior of the output units depends on the activity of the hidden units and the weights between the hidden and output units.
This simple type of network is interesting because the hidden units are free to construct their own representations of the input. The weights between the input and hidden units determine when each hidden unit is active, and so by modifying these weights, a hidden unit can choose what it represents. We also distinguish single-layer and multi-layer architectures. The single-layer organization, in which all units are connected to one another, constitutes the most general case and is of more potential computational power than hierarchically structured multi-layer organizations. In multi-layer networks, units are often numbered by layer, instead of following a global numbering.
Perceptrons
This is a very simple model and consists of a single `trainable' neuron. Trainable means that its threshold and input weights are modifiable. Inputs are presented to the neuron and each input has a
desired output (determined by us). If the neuron doesn't give the desired output, then it has made a mistake. To rectify this, its threshold and/or input weights must be changed. How this change is to be calculated is determined by the learning algorithm. The output of the perceptron is constrained to Boolean values - (true, false), (1,0), (1,-1) or whatever. This is not a limitation because if the output of the perceptron were to be the input for something else, then the output edge could be made to have a weight. Then the output would be dependant on this weight.
The perceptron looks like -
x1, x2, ..., xn are inputs, these could be real numbers or Boolean values depending on the problem.