Connectionist Modelling of Language and Cognitive Processes
Practical with OXlearn
Gert Westermann August 2011
In order to understand and appreciate connectionist modelling it is very useful to learn how to use neural networks in practice. This is what this practical is about. It is a mixture of enabling you to set up simple neural network models, train and test them, and at the same time to deepen your understanding of neural modelling.
The practical handout contains tasks and questions which form part of the assessment for this course. Some of these are revision questions from the seminars and/or the texts on the reading list, but most will require you to use the OXlearn simulator and build your own neural network models.
Please enter your answers to the questions in this sheet in bold. Where you are asked to provide graphs of your results, please paste the pictures directly into the sheet (use .jpg or .gif format to limit the size of your document). You can either take screenshots or extract the figures from OXlearn (using the ‘extract’ button and File -> Save As).
For each question, the number in parenthesis indicates the maximum number of points you can get for answering correctly. Answering the more advanced FoodForThought questions is not strictly part of the coursework. However, I would be delighted if you had a go at (some of) these and I might be tempted to allocate extra points for answers that are extraordinarily concise or informed.
Note: In the practical sessions, you can work in small groups if you wish. However, if you choose to do so you need to indicate in your portfolio who you have been working with in a particular session. Also, you still need to formulate your answers in your own words, even if you have discussed them as a group.
1. Understanding how activation is sent through a neural network.
Question 1: What are the two tasks performed by each individual node (or unit) in a neural network? (1)
Question 2: Why are input units functionally different from all the other units in a network? (1)
Question 3: x1 = 3; x2 = - 2; x3 = 1.5; x4 = 0.7. What is ? (1)
Imagine a minimal neural network as drawn here (A), whose output unit uses the sigmoidal (or logistic) activation function shown on the right hand side (B).
Question 4: Given the input values and connection weights are as shown above, what is the net input received by the output unit? (1)
Question 5: Given the sigmoidal activation function (B), what is the rough activation value of the output unit (look it up in the graph)? (1)
Question 6: If both connection weights were 0, what would the activation value of the output unit be? (1)
Question 7: And if the weights had the values 1.2 and 0.7 and both inputs were 0, what would be the resulting output activation? (1)
Question 8: Name three possible activation functions for neural network units. For each of these functions, (a) indicate if the function is linear or not and (b) describe verbally how the function works (in which way the net input is transformed into an activation value). (2)
Bias units: A bias node is an extra node in many neural networks that always (automatically) has an activation value of 1 (it is always on). The bias is usually connected to all units in the network, and its weights are adjusted in the same manner as all other weights in the network (they can end up being negative or positive).
Question 9: Explain why a bias node might be necessary for a network to perform correctly (maybe with an example?). (2)
FoodForThought 1: Is there a simple way of describing the effect of the bias node on a unit’s activation function?
FoodForThought 2: Is there an equivalent to the bias node’s functionality in biological neurons?
2. Setting up your first neural network with OXlearn
This practical introduces you to the OXlearn simulation software that you will be using in this practical. OXlearn is a high-level neural network simulation program that enables you to easily define connectionist models and run simulations with them. OXlearn is actually a MATLAB program, which means that it only works when you have a recent version (7.3 or better) of MATLAB installed on your computer (we have it installed here). However, you do not need to do any programming yourself; all the functionality of OXlearn is accessible via its graphical user interface - basically by using buttons, dropdown menus, etc.
OXlearn is the successor of the t-learn software which is used in the recommended books for this module (McLeod et al, 1998; and Plunkett & Elman, 1997; see reading list). All the exercises contained in these books can also be done with OXlearn, and it has become a lot easier, too.
Today’s instructions are exceedingly detailed (you can skip what you know already) in order to give you a chance to familiarize yourself with OXlearn. However, after today you will be expected to know your way around and you will not get step-by-step instructions any more. If you have problems using OXlearn, please consult the user’s manual, and if this doesn’t help speak to me. There will be a second supervised OXlearn session in week 2 of the course.
Simulating an AND network in OXlearn
In this task you will use OXlearn to set up and train a neural network to learn the AND function. Similar to all simulation projects, this requires setting up the simulation (defining the train patterns, the network architecture and the training options) and then training the network and analysing its performance (during training or subsequent testing).
The very first thing you’ll have to do is download the OXlearn program onto your personal USB drive. You can find the OXlearn folder at http://psych.brookes.ac.uk/oxlearn/ (you might need to unzip the folder). This page also has the OXlearn manual.
Next you have to start MATLAB, just click on the MATLAB icon on your desktop. Once it has opened, you’ll need to load the OXlearn program. To do so, click on File->Open and then browse to the OXlearn folder, and double click on OXlearn.m. The OXlearn interface will open.
The first thing you will notice is the simulation overview display which has opened in the OXlearn interface. Most of the panels will be empty and the red attention signs at the left indicate that the simulation is not yet set up properly to be trained or tested. This is not surprising, as we are currently dealing with an empty simulation.
The first thing to do is to assign a name to your new simulation project. Go to File -> Save Simulation As and browse to a suitable location on your USB drive, for example you could create a new folder called “practical1” in the default “Simulations” folder. Also, you have to provide a name, e.g. “myAND”, to which the extension ‘.mat’ will be attached automatically. To add subsequent changes to this simulation file you just choose File -> Save Simulation or press CTRL + S, as in most programs.
Note: Please make sure to save all simulation files that you are producing on your USB drive! When you log out, all data will be deleted from the computer..
Now you are ready to add content to your simulation. There are three essential parts of a simulation that need to be set up properly before you can train the network. These three parts concern the train patterns, the network architecture and the training options, all accessible from the Set-up menu in the menu bar at the top of the OXlearn window - or directly from the simulation overview (Inspect -> Simulation) by clicking on the corresponding edit button (the blue pen).
Table 1: the AND function
Patterns / Input / Output targetunit 1 / unit 2 / unit 1
Pattern 1 / 0 / 0 / 0
Pattern 2 / 0 / 1 / 0
Pattern 3 / 1 / 0 / 0
Pattern 4 / 1 / 1 / 1
Note: In binary problems like this you can also read a value of 1 as “on”, ”true” or “given”, while a value of 0 represents “off”, “false” or “absent”.
Go to Set-up -> Train Patterns. In this window you can define the Input and Target Patterns you want the network to learn from, essentially feeding the program with the content of Table 2. There are, in fact, several ways of doing so (see the OXlearn User Manual), only one will be described here. First, you have to choose an appropriate number of input and output (target) units, as well as the number of different patterns that you are going to define. This can be done by clicking on the “change size of patterns” button on the left hand side of the display (use the tooltip information that is shown when the mouse pointer hovers over a button). Once you have entered these numbers (4 patterns, 2 input units, 1 target unit) you can see that the two graphs have adapted their size. However, you still need to enter the values of the desired input and target activation. This is simple: just right-click on the appropriate area of a graph and enter the value (0 or 1). You might also want to change the pattern labels (right click on them to edit) to something more descriptive. For now you are done, you can close the Set-up -> Train Patterns window by clicking on OK.
Note: In fact, changes in all set-up windows are applied immediately, not only when the window is closed.
Go to Set-up -> Network. In this window you can define the exact network architecture by choosing a network type from a dropdown menu and defining the appropriate number of units for each layer, as well as whether you want a bias node or not. For now we want a “2-layer feed-forward” network with a bias unit and a sigmoidal activation function in the output layer. The input and output layer size should correspond with the train patterns.
Go to Set-up -> Training Options. In this window you define all the parameters pertaining to the exact way in which the network is trained, e.g. which learning algorithm is used, for how long the network should be trained, in which order the individual patterns are presented during training, as well as some options to control which information is logged during the training process. For the current simulation, you can just leave all these parameters at their default values and close the window.
Note: You will realize that the status of the train options in the simulation overview display has changed, it now shows a green tick instead of the red attention sign. The reason is this: whenever you open a set-up window, all the corresponding parameters that are not yet defined will be created with pre-defined default values. Thus, even if you have not changed any of the values, the act of opening the window has lead to the creation of all the relevant parameters.
Note: The grey ticks in front of the bottom most four elements (weights, train performance, verify performance and test performance) indicates that all is well, but the corresponding parameters are still empty. The reason is simple: the corresponding parameters are obtained by training, verifying or testing the network. Once you have done this, the ticks will turn green.
At a later stage, you might also encounter yellow attention signs here. They indicate that the set-up parameters (the topmost five) and the results parameters (the last four) might be inconsistent. This can happen when, in a simulation that has already been trained, you change the set-up (e.g. if you choose a different learning rate). Because the weights, train performance, etc., are the outcome of training with the previous set-up, your simulation is temporarily inconsistent. However, you don’t need to worry: training the network with the novel set-up will replace the results parameters and thus make the simulation consistent again.
The first three elements in the status panel (in the simulation overview display) should now all be ticked, which means that you are ready to train the network. Before doing so, however, you should take the time to familiarize yourself with some of the other displays provided by OXlearn, all of which let you inspect different aspects of a simulation in more detail. You can change displays by selecting the options under the Inspect menu – the first entry (Inspect -> Simulation) corresponds to the currently shown simulation overview display. Because much of the information you might want to display concerns the network’s performance during and after training, the displays will of course be more informative once you have trained the network.
And this is what you’ll be doing now. Switch to the performance display (Inspect -> Performance), and then go to Run -> Train Network. Provided you have set up everything correctly, this will lead to the network being trained for the given number of sweeps (1000, as per default setting).
Note: A sweep is the processing of one individual training pattern, where the identity of the pattern is determined by the chosen presentation order (you might want to bring up the Training Options window again to check). If you have left this parameter at “sequential”, this means that the first pattern is presented in the first training sweep, then the second pattern, then the third, and so on, starting over with the first pattern again after the last pattern (here the forth) has been presented. The other options either randomly shuffle the order of presenting each pattern once (“random without replacement”) or draw a pattern at random for each individual sweep (random with replacement”). A related term is an epoch, which usually means one pass through all the training patterns. Thus, in the current simulation, four sweeps make one epoch.