Classification

Regression
Weights
Decision Surface
Generalization
Features
Training Data
Testing Data
Activation function
Threshold and Bias

Activation functions
f ( net )

2
1 e

( net )

1 bipolar continuous

1, net 0
f ( net ) sgn( net )
bipolar bina
1, net 0
f ( net )

1
1 e

( net )

unipolar continuous

1, net 0
f ( net ) sgn( net )
unipolar bin
0 , net 0

a=logsig(n) = 1 / (1 + exp(-n))

1
0.8
0.6

n = -5:0.1:5;
a = logsig(n);
plot(n,a)

0.4
0.2
0
-0.2
-0.4
-0.6
-0.8
-1
-5

-4

-3

-2

-1

1
0.9
0.8

a = tansig(n) = 2/(1+exp(-2*n))-1

0.7
0.6
0.5

n = -5:0.1:5;
a = tansig(n);
plot(n,a)

0.4
0.3
0.2
0.1
0
-5

-4

-3

-2

-1

Learning and Generalization

Learning is a relatively permanent change in behaviour brought about by
experience.
Learning The network must learn decision surfaces from a set of training
patterns so that these training patterns are classified correctly.
Generalization After training, the network must also be able to generalize,
i.e.correctly classify test patterns it has never seen before.
Usually we want our neural networks to learn well, and also to generalize
well.
Sometimes, the training data may contain errors (e.g. noise in the
experimental determination of the input values, or incorrect
classifications). In this case, learning the training data perfectly may
make the generalization worse. There is an important tradeoff between
learning and generalization that arises quite generally.

Generalization in Classification

Generalization in Function Approximation

Training a Neural Network

Whether our neural network is a simple Perceptron, or a much more
complicated multilayer network with special activation functions,
we need to develop a systematic procedure for determining
appropriate connection weights.
The general procedure is to have the network learn the appropriate
weights from a representative set of training data.
In all but the simplest cases, however, direct computation of the
weights is intractable.
Instead, we usually start off with random initial weights and adjust
them in small steps until the required outputs are produced.
Trained networks are expected to generalize, i.e. deal appropriately
with input data they were not trained on.

Perceptron Learning

Convergence of Perceptron Learning

Problem: Perceptron Learning Rule

Learning rate= 0.1
X1=[1 2 0 1]1, X2=[0 1.5 -.5 -1]1, X3=[-1 1 .5 -1]1
d1 = -1
d 2 = -1
d3 = 1

x1

w1=[1 -1 0 .5];
Net1=net1=[1 -1 0 .5]*[1 -2 0 -1]'=2.5

x2

w2=w1+.1*(-1-1)*x1
w2 =[ 0.8000 -0.6000

0.7000]

x3

net2=w2*x2'= -1.6000

x4

w3=w2
net3=w2*x3'=-2.1
w4=w2+.1*(1--1)*x3=[0.6000 -0.4000

0.1000

0.5000]

0.1000

0.7000];

net4=w4*x1'=0.9
w5=w4+.1*(-1-1)*x1=[0.4000
net6=w5*x3'=-1.05

0.0000

w1
w2

w3
w4