COURSE ANNOUNCEMENT -- Spring 2004
DSES-4810-01 Intro to COMPUTATIONAL INTELLIGENCE
& SOFT COMPUTING
With ever increasing computer power readily available novel engineering methods based on “soft computing” are emerging at a rapid rate. This course provides the students with a working knowledge of computational intelligence (CI) covering the basics of fuzzy logic, neural networks, genetic algorithms and evolutionary computing, simulated annealing, wavelet analysis, artificial life and chaos. Applications in control, forecasting, optimization, data mining, fractal image compression and time series analysis are illustrated with engineering case studies.
This course provides a hands-on introduction to the fascinating discipline of computational intelligence (i.e. the synergistic interplay of fuzzy logic, genetic algorithms, neural networks, and other soft computing techniques). The students will develop the skills to solve engineering problems with computational intelligence paradigms. The course requires a CI-related project in the student’s area of interest.
Instructor: Prof. Mark J. Embrechts (x 4009 )
Office Hours: Thursday 10-11 am (CII5217)
Or by appointment.
Class Time: Monday/Thursday: 8:30 – 9:50 am (Amos Eaton Hall 216)
Text (optionakl): J. S. Jang, C. T. Sun, E. Mizutani, “Neuro-Fuzzy and Soft Computing,” Prentice Hall, 1996 (1998). ISBN 0-13-261066-3
Course is open to graduate students and seniors of all disciplines.
GRADING:
Tests 10%
5 Homework Projects 35%
Course Project 40%
Presentation 15%
ATTENDANCE POLICY
Course attendance is mandatory, a make-up project is required for each missed class. A missed class without make-up results in the loss of half a grade point.
ACADEMIC HONESTY
Homework Projects are individual exercises. You can discuss assignments with your peers, but not copy. Course project may be in groups of 2.
COMPUTATIONAL INTELLIGENCE - COURSE OUTLINE
1. INTRODUCTION TO ARTIFICIAL NEURAL NETWORKS (ANNs)
1.1 History
1.2 Philosophy of neural nets
1.3 Overview neural nets
2. INTRODUCTION TO FUZZY LOGIC
2.1 History
2.2 Philosophy of Fuzzy Logic
2.3 Terminology and definitions
3. INTRODUCTION TO EVOLUTIONARY COMPUTING
3.1 Introduction to Genetic Algorithms
3.2 Evolutionary Computing/ Evolutionary programming/ Genetic Programming
3.3 Terminology and definitions
4. NEURAL NETWORK APPLICATIONS/DATAMINING WITH ANNs
4.1 Case study: time series forecasting (population forecasting)
4.2 Case study: automated discovery of novel pharmaceuticals (Part I)
4.3 Data mining with neural networks
5. FUZZY LOGIC APPLICATIONS/FUZZY EXPERT SYSTEMS
5.1 Fuzzy logic case study: tipping
5.2 Fuzzy expert systems
6. SIMULATED ANNEALING/GENETIC ALGORITHM APPLICATIONS
6.1 Simulated annealing
6.2 Supervised clustering with GAs
6.3 Case study: automated discovery of novel pharmaceuticals (Part II)
7. DATA VISUALIZATION WITH SELF-ORGANIZING MAPS
7.1 The Kohonen feature map
7.2 Case study: visual explorations for novel pharmaceuticals (Part III)
7. ARTIFICIAL LIFE
7.1 Cellular automata
7.2 Self-organized criticality
7.3 Case study: highway traffic jam simulation
8. FRACTALS and CHAOS
8.1 Fractal Dimension
8.2 Introduction to Chaos
8.3 Iterated Function Systems
9. WAVELETS
Monday January 12, 2004
DSES-4810-01 Intro to COMPUTATIONAL INTELLIGENCE
& SOFT COMPUTING
Instructor: Prof. Mark J. Embrechts (x 4009 or 371-4562) ()
Office Hours: Tuesday 10-12 am (CII5217)
Or by appointment.
Class Time: Monday/Thursday: 8:30-9:50 (Amos Eaton Hall 216)
TEXT (optional): J. S. Jang, C. T. Sun, E. Mizutani, “Neuro-Fuzzy and Soft Computing,” Prentice Hall, 1996. (1998) ISBN 0-13-261066-3
LECTURES #1-3: INTRO to Neural Networks
The purpose of the first two lectures is to expose an overview of the philosophy of artificial neural networks. Today's lecture will provide a brief history of neural network development and inspire the idea of training a neural network. We will introduce a neural network as a framework to generate a map from an input space to an output space. Three basic premises will be discussed to explain artificial neural networks:
(1) A problem can be formulated and represented as a map from a m-dimensional space Rm to a n-dimensional space Rn, or Rm -> Rn.
(2) Such a map can be realized by setting up an equivalent artificial framework of basic building blocks of McCulloch-Pitts artificial neurons. This collection of artificial neurons forms an artificial neural network or ANN.
(3) The neural net can be trained to conform to the map based on samples of the map and will reasonably generalize to new cases it has not encountered before.
Handouts:
1. Mark J. Embrechts, "Problem Solving with Artificial Neural Networks."
2. Course outline and policies.
Tasks:
Start thinking about project topic, meet with me during office hours or by appointment.
3
3
PROJECT DEADLINES:
January 22 Homework Project #0 (web page summary)
January 29 Project proposal (2 typed pages, title, references,
Motivation, deliverable, evaluation criteria)
WHAT IS EXPECTED FROM THE CLASS PROJECT?
· Prepare a monologue about a course related subject (15 to 20 written pages and supporting material in appendices).
· Prepare a 20 minute lecture about your project and give presentation. Hand in a hard copy of your slides.
· A project starts in the library. Prepare to spend at least a full day in the library over the course of the project. Meticulously write down all the relevant references, and attach a copy of the most important references to your report.
· The idea for the lecture and the monologue is that you spend the maximum amount of effort to allow a third party to present that same material, based on your preparation, with a minimal amount of effort.
· The project should be a finished and self-consistent document where you meticulously digest the prerequisite material, give a brief introduction to your work, and motivate the relevance of the material. Hands-on program development and personal expansions of and reflections on the literature are strongly encouraged. If your project involves programming, hand in a working version of the program (with source code) and document the program with a user’s manual and sample problems.
· It is expected that you spend on average 6 hours/week on the class project.
PROJECT PROPOSAL
· A project proposal should be a fluent text of at least 2 full pages, where you are trying to sell the idea for a research project in a professional way. Therefore the proposal should contain a clear background and motivation.
· The proposal should define a clear set of goals, deliverables, and time table.
· Identify how you would consider your project successful and address evaluation criteria
· Make sure you select a title (acronyms and logos are suggested as well), and add a list of references to your proposal.
3
3
PROBLEM SOLVING WITH ARTIFICIAL NEURAL NETWORKS
Mark J. Embrechts
1. INTRODUCTION TO NEURAL NETWORKS
1.1 Artificial neural networks in a nutshell
This introduction to artificial neural networks explains as briefly as possible what is commonly understood by an artificial neural network and how they can be applied to solve data mining problems. Only the most popular type of neural networks will be discussed here: i.e., feedforward neural networks (usually trained with the popular backpropagation algorithm). Neural nets emerged from psychology as a learning paradigm, which mimics how the brain learns. There are many different types of neural networks, training algorithms, and different ways to interpret how and why a neural network operates. A neural network problem is viewed in this write-up as a parameter free implementation of a map and it is silently assumed that most data mining problems can be framed as a map. This is a very limited view, which does not fully cover the power of artificial neural networks. However, this view leads to a intuitive basic understanding of the neural network approach for problem solving with a minimum of otherwise necessary introductory material.
Three basic premises will be discussed in order to explain artificial neural networks:
(1) A problem can be formulated and represented as a map from a m-dimensional space Rm to a n-dimensional space Rn, or Rm -> Rn.
(2) Such a map can be implemented by constructing an artificial framework of basic building blocks of McCulloch-Pitts artificial neurons. This collection of artificial neurons forms an artificial neural network (ANN).
(3) The neural net can be trained to conform to the map based on samples of the map and will reasonable generalize to new cases it has not encountered before.
The next sections expand on these premises and explain a map, McCulloch-Pitts neuron, artificial neural network or ANN, training and generalization.
1.2 Framing an equivalent map for a problem
Let us start by considering a token problem and reformulate this problem as a map. The token problem involves deciding whether a seven-bit binary number is odd or even. To restate this problem as a map two spaces are considered: a seven-dimensional input space containing all the seven-bit binary numbers, and a one-dimensional output space with just two elements (or classes): odd or even, which will be symbolically represented by a one or a zero. Such a map can be interpreted as a transformation from Rm to Rn, or Rm -> Rn (with m = 7 and n = 2). A map for the seven-bit parity problem is illustrated in figure 1.1.
Figure 1.1 The seven-bit parity problem posed as a mapping problem.
The seven-bit parity problem was just framed as a formal mapping problem. The specific details of the map are yet to be determined: all we have so far is that we hope that a precise function can be formulated that transfers the seven bit binary input space to a 1-dimensional 1-bit output space which solves the seven-bit parity problem. We hope that eventually we can specify a green-box that formally could be implemented as a subroutine in a C-code, where the subroutine would have a header of the type:
void Parity_Mapping(VECTOR sample, int *decision) {
code line 1;
...
line of code;
*decision = ... ;
} // end of subroutine
In other words: given a seven-bit binary vector as an input to this subroutine (e.g. {1, 0, 1, 1, 0, 0, 1}), we expect the subroutine to return an integer nicknamed "decision.” The value for decision will turn out to be unity or zero, depending on whether the seven-bit input vector is odd or even.
We call this methodology a green-box approach to problem solving to imply that we only hope that such a function can eventually be realized, but that so far, we are clueless about how exactly we are going to fill the body of that green box. Of course, you probably guessed by now that somehow artificial neural networks will be applied to do this job for us. Before elaborating on neural networks we still have to discuss a subtle but important point related to our way of solving the seven-bit parity problem. Implicitly it is assumed for this problem that all seven-bit binary numbers are available and that the parity of each seven-bit binary number is known.
Let us complicate the seven-bit parity problem by specifying that we know for the time being the correct parity for only 120 of the 128 possible seven-bit binary numbers. We want to specify a map for these 120 seven-bit binary numbers such that the map will correctly identify the eight remaining binary numbers. This is a much more difficult problem than mapping the seven-bit parity problem based on all the possible samples, and whether an answer exists and can be found for this type of problem is often not clear at all from the onset. The methodology for learning what has to go in the green box for this problem will divide the available samples for this map in a training set -- a subset of the known samples -- and a test set. The test set will be used only for evaluating the goodness of the green box implementation to the map.
Let us introduce a second example to illustrate how a regression problem can be reformulated as a mapping problem. Consider a collection of images of circles: all 64x64 black-and-white (B&W) pixel images. The problem here is to infer the radii of these circles based on the pixel values. Figure 1.2 illustrates how to formulate this problem as a formal map. A 64x64 image could be scanned row by row and be represented by a string of zeros and ones depending whether the pixel is white or black. This input space has 64x64 or 4096 binary elements and can therefore be considered as a space with 4096 dimensions. The output space is a one-dimensional number, being the radius of the circle in the appropriate units.
We generally would not expect for this problem to have access to all possible 64x64 B&W images of circles to determine the mapping function. We therefore would only consider a representative sample of circle images, somehow use a neural network to fill out the green box to specify the map, and hope that it will give the correct circle radius within a certain tolerance for future out-of-sample 64x64 B&W image of circles. It actually turns out that the formal mapping procedure as described so far would yield lousy estimates for the radius. Some ingenious form of preprocessing on the image data (e.g., considering selected frequencies of a 2-D Fourier transform) will be necessary to reduce the dimensionality of the input space.
Most problems can be formulated in multiple ways as a map of the type: Rm -> Rn. However, not all problems can be elegantly transformed into a map, and some formal mapping representations might be betters than others for a particular problem. Often ingenuity, experimentation, and common sense are called for to frame an appropriate map that can adequately be represented by artificial neural networks.
Figure 1.2 Determining the radius of a 64x64 B&W image of a circle, posed as a formal mapping problem.
1.3 The McCulloch-Pitts neuron and artificial neural networks
The first neural network premise states that most problems can be formulated as an equivalent formal mapping problem. The second premise states that such a map can be represented by an artificial neural network (or ANN): i.e., a framework of basic building blocks, the so-called McCulloch-Pitts artificial neurons.
The McCulloch-Pitts neuron was first proposed in 1943 by Warren McCulloch and Walter Pitts, a psychologist and a mathematician, in a paper illustrating how simple artificial representations of neurons could in principle represent any arithmetic function. How to actually implement such a function was first addressed by the psychologist Donald Hebb in 1949 in his book "The organization of behavior." The McCulloch-Pitts neuron can easily be understood as a simple mathematical operator. This operator has several inputs and one output and performs two elementary operations on the inputs: first it makes a weighted sum of all the inputs, and then it applies a functional transform to that sum which will be send to the output. Assume that there are N inputs {x1, x2, ... , xN}, or an input vector and consider the output y. The output y can be expressed as a function of its inputs according to the following equations: