Perceptron

Frank Rosenblatt devised Perceptron that operated much in the

same way as the human mind.
Perceptron could "learn" - and that was the breakthrough needed
to pioneer today’s current neural network
The adjusting of weights to produce a particular output is called
the "training" of the network which is the mechanism that
allows the network to learn

Rosenblatt, Frank (1958), The Perceptron: A Probabilistic Model for

Information Storage and Organization in the Brain, Cornell Aeronautical
Laboratory, Psychological Review, v65, No. 6, pp. 386-408.

Perceptron can not realize an XOR gate. We need more

complex network or use different transfer functions.
 Decision Boundaries
The decision boundary is the surface at which the output of the unit is
precisely equal to the threshold
On one side of this surface, the output Y will be 0, and on the other side it will be 1.
In two dimension the surface is w1x1 + w2x2 = θ
Which we can write x2 = (-w1/w2)x1+ θ/w2
which is the equation of a line of slope -w1/w2 with intercept θ/w2

x1
w1

1 w1
 slope = 
y w2
2 w2

w2

0 2
x2

W1=1, w2=2,   2
 Learning and Generalization
Classification: The ability to assign an input observation to a
category
Learning is a relatively permanent change in behaviour brought
about by experience.
Learning: The network must learn decision surfaces from a set of
training patterns so that these training patterns are classified
correctly.
Generalization: After training, the network must also be able to
generalize, i.e.correctly classify test patterns it has never seen
before.
Usually we want our neural networks to learn well, and also to
generalize well.
Sometimes, the training data may contain errors (e.g. noise in the
experimental determination of the input values, or incorrect
classifications). In this case, learning the training data perfectly
may make the generalization worse. There is an important
tradeoff between learning and generalization that arises quite
generally.
 Blue lines show the decision boundaries of two neural networks to solve a
two class classification problem. Which is a better neural network?

x2

2 1

x1
Generalization in Classification
Generalization in Function Approximation
 Training a Neural Network

Whether our neural network is a simple Perceptron, or a much

more complicated multilayer network with special activation
functions, we need to develop a systematic procedure for
determining appropriate connection weights.
The general procedure is to have the network learn the
appropriate weights from a representative set of training data.
In all but the simplest cases, however, direct computation of the
weights is intractable.
Instead, we usually start off with random initial weights and adjust
them in small steps until the required outputs are produced.
Trained networks are expected to generalize, i.e. deal
appropriately with input data they were not trained on.
 Perceptron Learning
• How does a perceptron acquire its knowledge?
• The question really is: How does a perceptron learn the
appropriate weights?
• Since it’s starting with no knowledge, it is going to make
mistakes. When it makes a mistake, we are going to adjust
the weights to make that mistake less likely in the future
• When we adjust the weights, we’re going to take relatively
small steps to be sure we don’t over-correct and create new
problems.
Perceptron Learning
Convergence of Perceptron Learning
 new ol d 1.0 + 1 = 2.0
1w = 1w + p1 =
– 0.8 2 1.2

Type nnd4db in MATLAB command window.

Type nnd4pr
What is the capability of the following network? We added one more
layer with one neuron.

-1
x1
1
3
1
-1 1
x2
1 1
1
-4 -2.5
x3=1

This is a two layer network. This has one hidden layer.

Each neuron in the first layer divides the plane into two half- planes. The
half-planes where the neurons’ response s are +1 have been marked with
arrows pointing toward the positive response. The fourth neuron responds
+1 if and only if the first3 neurons’ output =1
x1-3=0
x2
3

x1-x2>0

3-x1>0
2

x1+x2-4>0

x1-x2=0 x1+x2-4=0

1 2 3
x1
Illustration of some possible decision boundaries which can
be generated by networks having threshold activation
functions and various numbers of layers (if activation
function is step function)
Design a single output two layer network which classifies
the shaded region in Fig. from the other region.

(1,3)

(3, 2)

(1,1)
 The following general learning rule is adopted in neural network studies
The weight increases in proportion to the product of input x

w    r  x
and learning signal r

x1=[1 0 -2 -1]';
where η is called learning rate.
w1=[-1 2 0 .5]';
For perceptron r =(t-o) t=target
net1=w1'*x1

A perceptron using bipolar binary activation function is net1 = -1.5000

shown in Fig. 1. x1=[ 1 0 -2 -1]t and x2=[ -1 2 0 -1]t . The w2=w1+0.1*(1- sign(net1))*x1
target for x1 is 1 and for x2 is -1. The initial weight vector w2 = -0.8000
is w=[ -1 2 0 0.5]t Apply perceptron learning rule to 2.0000
find the weight w3 after one epoch. η = 0.1. -0.4000

0.3000
Fig. 1 x2=[-1 2 0 -1]';
w1
net2=w2'*x2
w2 net2 = 4.5000

w3=w2+0.1*(-1-sign(net2))*x2

w3 w3 = 0.6000

1.6000

w4 -0.4000

0.5000