ANALYSIS OF SYNAPTIC INTERACTION FOR ARTIFICIAL NEURON HARDWARE IMPLEMENTATION

O.H. Petrosyan1, V.P. Grigoryants2, G.A. Karapetyan3 and A.R. Grigoryan4

Faculty of Cybernetics

State Engineering University of Armenia

1

2

4

Abstract:Analysis of synaptic interaction is performed. In the result of analysis a method is developed, which helps to evaluate full functionality of artificial neuron. Analysis of logic gates is performed as a synaptic interaction. In the result of analysis “AND” logic gate is suggested as a synaptic interaction type in terms of simplicity, power consumption and area. A method is developed, which allows transferring from one synaptic interaction to another.

Keywords: artificial neuron, synaptic interaction, weights, logic gates, hardware implementation.

  1. INTRODUCTION

Artificial neural networks (ANN) are computational models inspired by the intricately interconnected collection of neurons that work together in a biological brain. The importance of artificial neural networks is apparent in their versatility, as they are being used for computation and data analysis in a wide range of applications. The concept arose out of neurologists in the 1940's using electric circuits to attempt to model the activity of neurons in the brain. The use of artificial neural networks as a computing device was overshadowed by the now-standard von Neumann architecture, and it was only in the 1970s that research and development of neural networks resumed, allowing them to extend over all areas of science in their processing abilities. The use of neural networks offers the following useful properties and capabilities: nonlinearity, input-output mapping, adaptivity, evidential response, contextual information, fault tolerance, VLSI implementability, uniformity of analysis and design, neurobiological analogy. Also the investigation of an artificial neural networks helps to understand how the brain is working on. Thus it will help to cure brain diseases, such as Alzheimer’s, Parkinson's, schizophrenia, brain damage, amnesia, etc. [1-3]

There are 2 approaches to build neural networks, one is software, and the other one is hardware. The first one has wide-spread use and well-known, but for complex problems, software implementation of ANN requires huge resources, and will not optimally work with standard architecture processors (such as Von Neumann, Harvard, ARM, etc.) [3-6]. Also for solving complex problems mentioned above with ANN is usually possible with supercomputers, which have thousands of processors. Of course, this kind of resource usage is not the best solution. Unlike the first approach, hardware implementation of ANN don’t have wide-spread use, and isn’t well-known due to basic logic gates (artificial neuron) of ANN hard implementations [6-9]. It’s important to mention, that ANN hardware approach don’t contradict with the software one, as of course, in hardware implementation we will also have a software, which will work on that platform (ANN hardware). Hardware implementation of ANN opens the new sphere in computational engineering as well. It’s known that ANN can perform information parallel, which will help to build a new processor (neural-processor) with new architecture and higher performance.

  1. ARTIFICIAL NEURON MODEL

ANN consistsof an interconnected group of artificialneurons (AN). There are a lot of models of AN, which differs from each other by activation functions and type of synaptic interaction (input interaction) [3,5,10].

We will discuss AN with sign activation function, because it provides digital output, thus it can be implemented by digital hardware [3,5,10]. Other activation functions are analog, thus the use of them in CMOS VLSI is limited, due to low noise immune. The first model of AN was simple threshold logic [10,11]. But as it is known threshold logic has lack in the functionality, i.e. it is impossible to implement “XOR” and “XNOR” functions with single threshold logic gate [3,5,12,13]. Then the model of AN was enhanced and synaptic interactions were added, which helps single AN to implement all Boolean functions and single AN becomes so called full functional [5]. Analytically AN with sign activation function describes as follows:

wherei is i-th synaptic weight, xi is i-th input value,  is threshold, and sign function becomes 0 when its argument is negative and 1 otherwise [3,5,10,11].Graphical view of AN shown in Figure 1.

Figure 1. Graphical view of AN

In Figure 1 f is function of synaptic interaction.To synthesize anAN it is necessary to construct a characteristic system of inequalities for all possible combinations of inputs, and solve it according to the given Boolean function. The given Boolean function is a function which should be implemented by AN. Characteristic system of AN is a system of inequalities, which is constructed according to all combinations of inputs. Each row of the system shows weighted sum of corresponding input combination [10]. If the inequality of the any row is true then AN generates 1 at corresponding input combination of that row, otherwise if the inequality is false then at the output of AN will be 0. As total amount of input combinations for n inputs is 2n, thus we should have 2n independent inequalities, and each inequality of characteristic system should describe each combination. So to have full functional AN with n inputs it is necessary and sufficient to have a characteristic system with 2n independent inequalities.

There are two types of synaptic inputs, the first one is connected directly to the input of AN, and the second one connected to the output of synaptic interactions. The total amount of the first type of synapsis for AN with n inputs equals to n, and the total amount of the second type of synapsis equals to 2n-n-1. Thus the total amount of synapses should be 2n-1, otherwise we will have either none full functional AN or there will be excessive synapses, which influence can be extrapolated to the influence of threshold.

  1. A METHOD TO CHOOSE SYNAPTIC INTERACTION TYPE

A question arises, which Boolean functions can be used for AN as a synaptic interaction, and which one to choice in terms of area, power consumption and simplicity perspectives without suffering performance and functionality?

To analyze which synaptic interaction to use for AN hardware implementationlets start to discuss AN with 2 inputs (Figure 2), and then the gained results will be spread to AN with more than 2 inputs.

Figure 2. AN with 2 input

In this case as an input interaction can be 16 Boolean functions, which are shown in Table1.

Table 1.

x1 / x2 / f0 / f1 / f2 / f3 / f4 / f5 / f6 / f7 / f8 / f9 / f10 / f11 / f12 / f13 / f14 / f15
0 / 0 / 0 / 0 / 0 / 0 / 0 / 0 / 0 / 0 / 1 / 1 / 1 / 1 / 1 / 1 / 1 / 1
0 / 1 / 0 / 0 / 0 / 0 / 1 / 1 / 1 / 1 / 0 / 0 / 0 / 0 / 1 / 1 / 1 / 1
1 / 0 / 0 / 0 / 1 / 1 / 0 / 0 / 1 / 1 / 0 / 0 / 1 / 1 / 0 / 0 / 1 / 1
1 / 1 / 0 / 1 / 0 / 1 / 0 / 1 / 0 / 1 / 0 / 1 / 0 / 1 / 0 / 1 / 0 / 1

Let’s assign the value of function fi at any j input combination tofij. Namely fi0=fi(0,0), fi1=fi(0,1), fi0=fi(1,0), fi0=fi(1,1). For example f23=f2(1,0)=1, it means that 3rd combination of 2nd function has value 1. So the first index of fij shows function number and the second one shows combination number (i-function number, j-combination number). For n inputs i{0,-1} and j {0,2n--1}.

For two inputs AN characteristic system is shown below.

(Eq. 1)

The first row of the system corresponds to {0,0} input combination, the second row corresponds to {0,1} input combination and so on. For example, if the first row is true then as it has already been mentioned we will have 1 at the output of AN in {0,0} input combination, otherwise we will have 0. The same reasoning is true for remaining inequalities of Eq. 1. Now we should find out which fij values make the inequalities independent from each other to have full functional AN.

It is obvious that constant 0 (f0), constant 1 (f15), (f3), (f12), (f5), (f10) functions cannot be used as an input interaction, because amount of independent inequalities becomes less than 2n. To be sure of that we can add 1st row with 4th one and 2nd row with 3rd one of Eq. 1.In the result we will get the following:

Thus we will have the same inequalities if

fi0+fi3=fi1+fi2(Eq. 2)

Please note that in adding operation we should understand arithmetic adding and not logical. As fij{0,1}, then Eq. 2 will be true for following conditions (see Table 2).

Table 2.

x1 / x2 / f0 / f3 / f5 / f10 / f12 / f15
0 / 0 / 0 / 0 / 0 / 1 / 1 / 1
0 / 1 / 0 / 0 / 1 / 0 / 1 / 1
1 / 0 / 0 / 1 / 0 / 1 / 0 / 1
1 / 1 / 0 / 1 / 1 / 0 / 0 / 1

So Table 2 shows the functions, which cannot be used as a synaptic interaction.

Now let’s discuss input interaction with none 0 preserving functions, i.e. all functions which fi0=1 [14]. That functions are from f8 to f15. In this case the characteristic system will be following.

Similarly if we have fi1+fi2=fi3, then the AN will be none full functional. Thus fi1+fi2≠fi3 condition should beverified for full functional AN.In this case we have 4 conditions either fi1+fi2=0 and fi3=1, or fi1+fi2=1 and fi3=0, or fi1+fi2=2 and fi3=1, or fi1+fi2=2 and fi3=0. Let’s discuss those 4 conditions separately.

  1. When fi1+fi2=0 and fi3=1, then
  1. When fi1+fi2=1 and fi3=0, then
  1. When fi1+fi2=2 and fi3=1, then
  1. When fi1+fi2=2 and fi3=0, then

As you can see in these4 casesaccording to axioms of inequalities [15] we have got dependent inequalities, thus AN will be none full functional. Therefore none 0 preserving functions cannot be used as input interaction of AN. So the remaining functions are f1, f2, f4, f6 and f7, i.e. “AND”, “abjunction”, “XOR” and “OR” functions. Let’s verify that they provide a characteristic system of independent inequalities.

  • For “AND” synaptic interaction characteristic system is following.

(Eq. 3)

As you can see inequalities are independent.

  • For “abjunction” synaptic interaction characteristic system is following.

or (Eq. 4)

In both cases we have a system with independent inequalities.

  • For “XOR” synaptic interaction characteristic system is following.

(Eq. 5)

In this case we have also a system with independent inequalities.

  • For “OR” synaptic interaction characteristic system is following.

(Eq. 6)

In this case we have a system of independent inequalities as well. So “AND”, “abjunction”, “XOR” and “OR” gates can be used as synaptic interaction for full functional AN.

One more question arises, which one of these functions to choose in terms of simplicity, power consumption and area?

“AND” and “OR” logic gates in CMOS process are normally implemented by at least 6 transistors, whereas the logic gates implementing “abjunction” or “XOR” functions consist of more then 6 transistors (for “XOR” it is normally to use 12 transistors, and for abjunction it is normally to use 8 transistors) [16-18]. Therefore in terms of simplicity “AND” and “OR” logic gates are more rational choice. In terms of area of course “AND” gate has normally less area in layout then “OR”, “XOR” and “abjunction” gates [16-18].

Now let’s discuss which logic gate to use in terms of power consumption. It is known that dynamic power consumption of logic gate equals:

Pdyn=01fCLV2DD

Where Pdyn is dynamic power consumption of logic gate, VDD is voltage supply value, 01 is switching activity, is the probability that a clock event results in a 01 (or power-consuming) event at the output of the gate, f the maximum possible event rate of the inputs [16]. Now let’s consider 2 inputs “AND”, “OR”, “XOR” and “abjunction” logic gates with the inputs A and B. Let pA and pB be the probabilities that the inputs A and B are one. Thus switching activity equations per logic gates are shown in the table and the graphics in the Figure 3 and Figure 4.

Table 3

Logic gate / 01
AND / (1-pApB)pApB
OR / (1-pA)(1-pB)[1-(1-pA) (1-pB)]
XOR / [1-(pA+pB-2pApB)](pA+pB-2pApB)
abjunction / [1-pA(1-pB)]pA(1-pB)

As you can see from the graphics (Figure 3 and Figure 4) maximum value of switching activity for “AND”, “OR” and “abjunction” is 0.25, but for “XOR” it is 1, thus their power consumptions in average will be less than “XOR” one.

Figure 3.Switching activity of “AND”, “OR”, and “abjunction” gates

Figure 4.Switching activity of “XOR” gate

In case of uniform input distribution the probability that the output of logic gate equals one for “AND” and abjunction gates is 1/4, for “OR” the probability equals 3/4, and for “XOR” it is 1/2. Thus for “AND” and “abjunction” input interaction synapses (synaptic weights) in average will be less active compared with “OR” and “XOR” gates, which will reduce power consumption caused by the synapses. Taking into account these two facts power consumption of input interaction logic gate and synapse, the rational choice is to use “AND” gate as an input interaction of AN.

So summarizing the results the most rational choice is to use “AND” logic gate as a synaptic interaction in terms of all aspects.

The same reasoning is true for AN with more then 2 inputs, and this method is applicable for more then 2 inputs. For more then 2 inputs in terms of simplicity, power consumption and area it is obvious to use read once functions [19] as a synaptic interaction. Thus the use of “AND” logic gate as synaptic interaction is still reasonable for AN with more then 2 inputs.

  1. CONVERSION OF SYNAPTIC INTERACTIONS

Let’s say that AN should implement some y Boolean function, which has following values y={y0, y1, y2, …, yi, …, }. The index of yi shows the number of input combination, at which y=yi. If y0=0, then the first row of AN characteristic system is false. Thus taking into account Eq.3-6 the threshold value should be more then 0. From low distribution of parameters and optimality point of view let’s set a threshold to be equal to possible minimal discrete positive value: =1. If y0=1, then the first row of ANcharacteristic system is true. Similarly threshold should be: =0. Thus we can write following expression: . If y1=0, then the second row of AN characteristic system is false. For AN with “AND”synaptic interaction (Eq. 3) 1 should be less than . Otherwise if y1=1, then the second row of AN characteristic system is true. Thus for AN with “AND”synaptic interaction (Eq. 3) 1 should be less than . So similarly from low distribution of parameters and optimality point of view we can write following expression for 1 of AN with “AND” synaptic interaction (Eq. 3) . By subtraction we should understand arithmetic subtraction and not logic.

Continuing in the same way for remaining rows and remaining input interactions we will get following for 2 input AN:

  • For “AND” interaction:

(Eq. 7)

  • For “abjunction”interaction:

(Eq. 8)

or

(Eq. 9)

  • For “XOR” interaction:

(Eq. 10)

  • For “OR” interaction:

(Eq. 11)

Please note that the same reasoning is true for AN more than 2 inputs, and the characteristic systems are constructed in the same way.

Solving Eq.7-11 systems we can switch from one type of input interaction to another. For example if we have AN with “OR”synaptic interaction, by using Eq.7 and Eq.11, we can get AN with “AND”synaptic interaction, which implements the same function as AN with “OR”synaptic interaction does.

Choosing as a reference AN with “AND”synaptic interaction and solving the systems for other interactions we can get the values of weights and threshold for that synaptic interactions dependently on AN with “AND”synaptic interaction (see Table 4). Doing the same thing for other synaptic interaction by choosing as a reference “abjunction” (ABJ), “XOR” and “OR” we get Table 5, Table 6, and Table 7 correspondingly. So Table 4-7 help us to transfer from AN with one type of synaptic interaction to another one, if the weights and threshold of reference AN is known.

Table 4.

AND / 1 / 2 / 3 / 
ABJ / 1+3 / 2 / -3 / 
XOR / 1+3/2 / 2+3/2 / -3/2 / 
OR / 1+3 / 2+3 / -3 / 

Table 5.

ABJ / 1 / 2 / 3 / 
AND / 1+3 / 2 / -3 / 
XOR / 1+3/2 / 2-3/2 / 3/2 / 
OR / 1 / 2-3 / 3 / 

Table 6.

XOR / 1 / 2 / 3 / 
AND / 1+3 / 2+3 / -23 / 
ABJ / 1-3 / 2+3 / 23 / 
OR / 1-3 / 2-3 / 23 / 

Table 7.

OR / 1 / 2 / 3 / 
AND / 1+3 / 2+3 / -3 / 
ABJ / 1 / 2+3 / 3 / 
XOR / 1+3/2 / 2+3/2 / 3/2 / 

The same reasoning is true for AN with more than 2 inputs. So by using the developed method of switching from AN with one synaptic interaction to another, we can calculated the weights and threshold corresponding to the reference AN.

  1. CONCLUSION

Analysis of AN synaptic interaction was performed. It was proved that to have full functional AN with n inputs it is necessary and sufficient to have 2n-1 weights, a threshold and synaptic interaction should be one of following Boolean functions: “AND”, “abjunction”, “XOR”, “OR”. Comparing analysis was performed for“AND”, “abjunction”, “XOR”, “OR” gates in terms of simplicity, area and power consumption. As a result of the analysis the best choice is to use “AND” logic gate as asynaptic interaction.

Besides, a method was developed, which helps to design an ANby using another known AN. The method is universal and can be used for any amount of inputs.

  1. REFERENCES
  1. John G. Nicholls, A. Robert Martin, Paul A. Fuchs, et al. - From Neuron to Brain, Fifth edition. November 7, 2011, 621p.
  2. HaykinSimon - Neural Networks. A Comprehensive Foundation. Second Edition.-Published by Pearson Education. July 16, 1998, 823p.
  3. Ali Kattan, Rosni Abdullah and Zong Woo Geem – Artificial Neural Network Training and Software Implementation Techniques. University Sains Malaysia. Pub. Date:2011.
  4. Jain, A.K - Artificial neural networks: a tutorial. IEEE Computer (Volume: 29, Issue 3), 1996. - 31-44pp.
  5. R. Wojtyna and T. Talaska -Transresistance CMOS neuron for adaptive neural networks implemented in hardware. Bulletin of the Polish Academy of Sciences Technical Sciences Vol. 54, No. 4, 2006
  6. Kwok Cheung Li. Neuron – MOS enhanced Differential Current-Switch Threshold Logic // Mathematics andComputer Science: MSc thesis, Delft University of Technology Faculty of Electrical Engineering, 2006. – 47 p.
  7. V. Beiu, J.M. Quintana, M.J. Avedilo, R. Andonie – Differential implementations of threshold logic gates. Signals, Circuits and Systems, 2003. SCS 2003. International Symposium on(Volume:2 ). 2003.
  8. V. Beiu – VLSI Implementations of Threshold Logic. A Comprehensive Survey. IEEE Transactionon Neural Networks, Vol. 14, N0 5, September 2003.
  9. Мкртчян С.О., Мкртчян - А.С. Основы цифровой нейроинформатики: Учебноепособие. – Ереван: Изд-воГИУА “Чартарагет”, 2007. – 387 с.
  10. Warren S. McCulloch and Walter Pitts.- A logical calculus of the ideas immanent nervous activity//Bulletin of Mathematical Biophysics.-1943.- Vol. 5.-P.115-133.
  11. Richard Bland – Learning XOR: exploring the space of a classic problem. University of Striling. Department of Computer Science and Mathematics. Computing Science Technical Report, 1998. – 53p.
  12. Jan M. Rabaey,Anantha Chandra-kasan,BorivojeNikolic– DigitalIntegrated Circuits (2nd Edition), Prentice Hall, 2003, -761p.
  13. John P. Uyemura – CMOS Logic Circuit Design, Springer 1999 edition, 528p.