Documentos de Académico
Documentos de Profesional
Documentos de Cultura
Soma
Synapse
Dendrites
Axon
Soma
Dendrites
Synapse
Input Signals
Middle Layer
Input Layer
Output Layer
Weights
Output Signals
x1
w1
x2
w2
Neuron
wn
xn
X = xi wi
i =1
+ 1, if X
Y =
1, if X <
Sign function
Sigmoid function
Y
+1
0
-1
Y
+1
0
-1
0, if X < 0
1, if X < 0
Y
+1
0
-1
1, if X 0
+1, if X 0 sigmoid
step
sign
Y
=
Y
=
Y
=
Linear function
0
-1
1
1+ e X
Y linear= X
Single--layer twoSingle
two-input perceptron
Inputs
x1
w1
Linear
Combiner
Hard
Limiter
w2
x2
Output
Y
Threshold
The Perceptron
The operation of Rosenblatts perceptron is based
on the McCulloch and Pitts neuron model.
model. The
model consists of a linear combiner followed by a
hard limiter.
The weighted sum of the inputs is applied to the
hard limiter, which produces an output equal to +1
if its input is positive and 1 if it is negative.
xi wi = 0
i =1
x2
Class A 1
1
x1
Class A 2
x1
2
x1w 1 + x2w 2 = 0
(a) Two-input perceptron.
x3
x1 w 1 + x 2 w 2 + x 3 w 3 = 0
e( p) = Yd ( p) Y ( p)
where p = 1, 2, 3, . . .
Y ( p ) = step x i ( p ) w i ( p )
i = 1
wi ( p ) = xi ( p ) e( p )
Step 4:
4: Iteration
Increase iteration p by one, go back to Step 2 and
repeat the process until convergence.
Indian Institute of Information Technology, Allahabad
Desired
output
Initial
weights
Actual
output
Error
Final
weights
x1
x2
Yd
w1
w2
w1
w2
0
0
1
1
0
1
0
1
0
0
0
1
0.3
0.3
0.3
0.2
0.1
0.1
0.1
0.1
0
0
1
0
0
0
1
1
0.3
0.3
0.2
0.3
0.1
0.1
0.1
0.0
0
0
1
1
0
1
0
1
0
0
0
1
0.3
0.3
0.3
0.2
0.0
0.0
0.0
0.0
0
0
1
1
0
0
1
0
0.3
0.3
0.2
0.2
0.0
0.0
0.0
0.0
0
0
1
1
0
1
0
1
0
0
0
1
0.2
0.2
0.2
0.1
0.0
0.0
0.0
0.0
0
0
1
0
0
0
1
1
0.2
0.2
0.1
0.2
0.0
0.0
0.0
0.1
0
0
1
1
0
1
0
1
0
0
0
1
0.2
0.2
0.2
0.1
0.1
0.1
0.1
0.1
0
0
1
1
0
0
1
0
0.2
0.2
0.1
0.1
0.1
0.1
0.1
0.1
0
0
1
1
0
1
0
1
0
0
0
1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0
0
0
1
0
0
0
0
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
x2
x2
1
x1
x1
0
(a) AND (x 1 x 2 )
x1
(b) OR (x 1 x 2 )
(c) Exclusive-OR
(x 1 x 2 )
Input Signals
Input
layer
First
hidden
layer
Second
hidden
layer
Output
layer
Three--layer backThree
back-propagation neural network
Input signals
1
x1
x2
xi
y1
y2
yk
yl
wij
wjk
m
n
xn
Input
layer
Hidden
layer
Output
layer
Error signals
, +
Fi
Fi
Step 2:
2: Activation
Activate the backback-propagation neural network by
applying inputs x1(p), x2(p),, xn(p) and desired
outputs yd,1(p), yd,2(p),, yd,n(p).
(a) Calculate the actual outputs of the neurons in
the hidden layer:
n
y j ( p ) = sigmoid x i ( p ) w ij ( p ) j
i =1
Step 2:
2: Activation (continued)
(b) Calculate the actual outputs of the neurons in
the output layer:
m
y k ( p ) = sigmoid x jk ( p ) w jk ( p ) k
j =1
Step 3:
3: Weight training
Update the weights in the backback-propagation network
propagating backward the errors associated with
output neurons.
(a) Calculate the error gradient for the neurons in the
output layer:
k ( p) = yk ( p ) [1 yk ( p)] ek ( p)
where
ek ( p ) = y d , k ( p ) y k ( p )
Step 3:
3: Weight training (continued)
(b) Calculate the error gradient for the neurons in
the hidden layer:
l
j ( p ) = y j ( p ) [1 y j ( p )] k ( p ) w jk ( p )
k =1
Step 4:
4: Iteration
Increase iteration p by one, go back to Step 2 and
repeat the process until the selected error criterion
is satisfied.
As an example, we may consider the threethree-layer
back--propagation network. Suppose that the
back
network is required to perform logical operation
Exclusive--OR
Exclusive
OR.. Recall that a singlesingle-layer perceptron
could not do this operation. Now we will apply the
three--layer net.
three
3
x1
w13
1
w35
w23
5
5
w24
x2
w45
4
w24
Input
layer
4
1
Hidden layer
Output
layer
y5
[
) = 1 / [1 + e
]
] = 0.8808
Sum-Squared Error
10
-1
10
-2
10
10
-3
10
-4
50
100
Epoch
150
200
Desired
output
x1
x2
yd
1
0
1
0
1
1
0
0
0
1
1
0
Actual
output
y5
Y
0.0155
0.9849
0.9849
0.0175
Error
e
0.0155
0.0151
0.0151
0.0175
Sum of
squared
errors
0.0010
x1
+1.0
1
2.0
+1.0
+0.5
5
+1.0
x2
+1.0
+1.0
4
+0.5
1
y5
Decision boundaries
x2
x2
x2
x 1 + x 2 1.5 = 0
x 1 + x 2 0.5 = 0
1
x1
x1
0
1
(a)
1
(b)
x1
0
1
(c)
tan h
2a
1+ e
bX
w jk ( p ) = w jk ( p 1) + y j ( p ) k ( p )
where is a positive number (0 < 1) called the
momentum constant.
constant. Typically, the momentum
constant is set to 0.95.
This equation is called the generalised delta rule.
rule.
10
101
100
10-1
10-2
10-3
10-4
20
40
60
Epoch
80
100
120
Learning Rate
1.5
1
0.5
0
-0.5
-1
20
40
60
80
Epoch
100
120
140
Heuristic 1
If the change of the sum of squared errors has the same
algebraic sign for several consequent epochs, then the
learning rate parameter, , should be increased.
Heuristic 2
If the algebraic sign of the change of the sum of
squared errors alternates for several consequent
epochs, then the learning rate parameter, , should be
decreased.
Indian Institute of Information Technology, Allahabad
10
101
100
10-1
10-2
10-3
10-4
10
20
30
40
50
60
Epoch
70
80
90
100
Learning Rate
1
0.8
0.6
0.4
0.2
0
20
40
60
Epoch
80
100
120
10
101
100
10-1
10-2
10-3
10-4
10
10
20
30
40
50
Epoch
60
70
80
Learning Rate
2.5
2
1.5
1
0.5
0
20
30
40
50
Epoch
60
70
80
90
Multilayer neural networks trained with the backbackpropagation algorithm are used for pattern
recognition problems. However, to emulate the
human memorys associative characteristics we
need a different type of network: a recurrent
neural network.
network.
A recurrent neural network has feedback loops
from its outputs to its inputs. The presence of
such loops has a profound impact on the learning
capability of the network.
x1
y1
x2
y2
xi
yi
xn
yn
Output Signals
Input Signals
Y =
M
y n
T
Y
Y
m m M I
m=1
(1, 1, 1)
(1, 1, 1)
(1, 1, 1)
(1, 1, 1)
y1
0
(1, 1, 1)
(1, 1, 1)
(1, 1, 1)
(1, 1, 1)
y3
Indian Institute of Information Technology, Allahabad
1
Y2 = 1
1
T
T
Y
=
[
1
1
1
]
Y
or 1
2 = [ 1 1 1]
2
0
2
2
2
0
Y1 = sign 2
2
Y2 = sign 2
2
2
0
0
2
2 1 0 1
2 1 0 = 1
0 1 0 1
2 1 0 1
2 1 0 = 1
0 1 0 1
Indian Institute of Information Technology, Allahabad
BAM operation
x1(p)
x1(p+1)
1
1
x2(p)
xi(p)
xn(p)
Input
layer
y1(p)
y2(p)
yj(p)
ym(p)
x2(p+1)
xi(p+1)
xn(p+1)
Output
layer
1
1
y1(p)
y2(p)
yj(p)
ym(p)
Input
layer
Output
layer
T
W=
X m Ym
m=1