Sensitivity Analysis:

a Validation and
Verification Tool


Terry Bahill
Systems and Industrial Engineering
University of Arizona
Tucson, AZ 85721-0020
terry@sie.arizona.edu
Copyright , 1993-2009 Bahill
This file is located at
http://www.sie.arizona.edu/sysengr/slides/

4/12/2012
2
References
Smith, E. D., Szidarovszky, F., Karnavas, W. J. and
Bahill, A. T., Sensitivity analysis, a powerful system
validation technique, The Open Cybernetics and Systemics
Journal,
http://www.bentham.org/open/tocsj/openaccess2.htm,
2: 39-56, 2008, doi: 10.2174/1874110X00802010039

W. J. Karnavas, P. Sanchez and A. T. Bahill, Sensitivity
analyses of continuous and discrete systems in the time
and frequency domains, IEEE Trans. Syst. Man.
Cybernetics, SMC-23(2), 488-501, 1993.

2009 Bahill
4/12/2012
3
You should perform a
sensitivity analysis anytime you
- create a model
- write a set of requirements
- design a system
- make a decision
- do a tradeoff study
- originate a risk analysis
- want to discover the cost drivers

2009 Bahill
4/12/2012
4
In a sensitivity analysis
- change
the values of
inputs
parameters
architectural features
- measure changes in
outputs
performance indices
2009 Bahill
4/12/2012
5
A sensitivity analysis can be used to
- validate a model,
- warn of unrealistic model behavior,
- point out important assumptions,
- help formulate model structure,
- simplify a model,
- suggest new experiments,
- guide future data collection efforts,
- suggest accuracy for calculating parameters,
- adjust numerical values of parameters,
- choose an operating point,
- allocate resources,
- detect critical criteria,
- suggest manufacturing tolerances,
- identify cost drivers.
2009 Bahill
4/12/2012
6
History: the earliest sensitivity analyses
- The genetics studies on the pea by Gregor
Mendel, 1865.
- The statistics studies on the Irish hops crops by
Gosset (reported under the pseudonym
Student), ca 1890.
2009 Bahill
4/12/2012
7
Classes of sensitivity functions
- Analytic
for well defined systems
usually partial derivatives
- Empirical
show sensitivity to parameters
observe system changes when
parameters are changed
works for an unmodeled system

2009 Bahill
4/12/2012
8
Types of sensitivity functions
- Analytic
absolute
relative
semirelative
- Empirical
direct observation
sinusoidal variation of parameters
design of experiments
Excel, using what-if analysis

2009 Bahill
4/12/2012
9
The absolute-sensitivity function
The absolute-sensitivity of the
function F to variations in the
parameter o is



It should be evaluated at the
normal operating point
(NOP).
NOP
F
F
S
o
o
c
=
c
2009 Bahill
4/12/2012
10
Examples
- Absolute-sensitivity functions are used
to calculate changes in the output due to changes
in the inputs or system parameters
to see when a parameter has its greatest effect
in adaptive control systems
2009 Bahill
4/12/2012
11
A process model example

x and y are inputs and A to F model system
parameters. The output z is love potion number 9.
The normal operating point is




What is the easiest way to increase
the quantity of z?
This sounds like a problem for absolute-sensitivity
functions.
2 2
z Ax By Cxy Dx Ey F = + + + + +
0 0 0 0
0 0 0 0
( , ) (1,1), 1, 2,
3, 5, 7, 8,
x y A B
C D E F
= = =
= = = =

0
2. z =
2009 Bahill
Absolute-sensitivity functions*
4/12/2012 12 2009 Bahill
0 0 0 0 0
NOP
0 0 0 0 0
NOP
2 0,
2 0.
z
x
z
y
z
S A x C y D
x
z
S B y C x E
y
c
= = + + =
c
c
= = + + =
c
2
0
NOP
2
0
NOP
0 0
NOP
0
NOP
0
NOP
NOP
1,
1,
1,
1,
1,
1,
z
A
z
B
z
C
z
D
z
E
z
F
z
S x
A
z
S y
B
z
S x y
C
z
S x
D
z
S y
E
z
S
F
c
= = =
c
c
= = =
c
c
= = =
c
c
= = =
c
c
= = =
c
c
= =
c
4/12/2012
13
What about interactions?
- Change two parameters at the same time.
- Interactions can be bigger than the first-order effects.
- Non-steroidal, anti-inflammatory drugs (NSAID) such
as Ibupropren and Aleve have dangerous interactions
with angiotensin converting enzymes, which effect
the kidneys and lower blood pressure. No
pharmacists would allow you to take both.
- My mother once cleaned the toilet with sodium
hypochlorite (Clorox bleach) and ammonia. It
produced chorine gas.
- Alcohol and barbiturates are much more dangerous
if mixed.



2009 Bahill
Cooperation
- The performance of a system could be greater than
the sum of its subsystems (cooperation).
- Alone, neither a human nor a knife can slice bread.
- Together a blind person and a Seeing Eye dog do
better than either alone.
- A pair of chopsticks performs more than twice as
well as an individual chopstick.
- Two lions chasing a Thompkins Gazelle are more
than twice as likely to catch it, than a single lion.

4/12/2012 2009 Bahill 14
4/12/2012
15
Interactions in the process control model
- Change two parameters at the same time.
- Mixed partial derivatives can be bigger than first-
order partial derivatives.
- Of the 64 possible second-partial derivatives, only
the following are nonzero.
2009 Bahill
Interactions*
4/12/2012 16 2009 Bahill
2
2
2
NOP
2
0
NOP
2
0
2
NOP
2
0
2
NOP
1,
3,
2 2,
2 4.
z
y E
z
x y
z
x
z
y
z
S
y E
z
S C
x y
z
S A
x
z
S B
y

c
= =
c c
c
= = =
c c
c
= = =
c
c
= = =
c
2
0
NOP
2
0
NOP
2
NOP
2
0
NOP
2
0
NOP
2 2,
1,
1,
2 2,
1,
z
x A
z
x C
z
x D
z
y B
z
y C
z
S x
x A
z
S y
x C
z
S
x D
z
S y
y B
z
S x
y C

c
= = =
c c
c
= = =
c c
c
= =
c c
c
= = =
c c
c
= = =
c c
4/12/2012 2009 Bahill 17

Table 1. Effects of Individual and Combined Parameter Changes
for some second-order interaction terms with delta of 1.0, where
0
x x x = + A etc.
Functions
Normal
values
Values
increased
by one
unit
New z
values
z A
Total
change
in z
( , ) f x A
A=1
x=1
A=2
x=2
7 5 5
0
( , ) f x A
A=1 A=2 3 1
2 z A =

0
( , ) f x A
x=1 x=2 3 1
0 0
( , ) f x A
A=1
x=1
2 0

( , ) f y B
B=2
y=1
B=3
y=2
8 6 6
0
( , ) f y B
B=2 B=3 3 1
3 z A =

0
( , ) f y B
y=1 y=2 4 2
0 0
( , ) f y B
B=2
y=1
2 0

The purpose of
this slide is to
show the affects
of interactions,
without using
mathematics.

We will now use
these data to
estimate the value
of one of the
mixed-partial
derivatives
Estimate the
mixed-second-partial derivative
4/12/2012 2009 Bahill 18
2
0 0 0 0 0 0
( , ) ( , ) ( , ) ( , ) ( , ) f f f f f o | o | o | o | o |
o | o |
c +
~
c c A A

2
7 3 3 2
3
1
z
x A
c +
~ =
c c

This is the wrong answer.
Analytically we found that the correct value is 2.
A smaller step size, 0.01
4/12/2012 2009 Bahill 19
Use the following general equation from Smith, Szidarovszky, Karnavas and Bahill [2008].
{ }
0 0 0 0 0 0 0 0
2 2
0 0 0 0
( , ) ( , ) ( , )( ) ( , )( )
1
( , )( ) 2 ( , )( )( ) ( , )( )
2!
x y
xx xy yy
f x y f x y f x y x x f x y y y
f x x f x x y y f y y , q , q , q
' ' = +
'' '' '' + + +

Converting to find the value if we change x and A yields
{ }
0 0 0 0 0 0 0 0
2 2
0 0 0 0
( , ) ( , ) ( , )( ) ( , )( )
1
( , )( ) 2 ( , )( )( ) ( , )( )
2!
x A
xx xA AA
f x A f x A f x A x x f x A A A
f x x f x x A A f A A , q , q , q
' ' = +
'' '' '' + + +

Now, using the symbols that we used in our absolute sensitivity functions, we can write
{ } 2 2
0 0 0 0
2 2 2
( , ) ( , )
1
( , ) 2 ( , ) ( , )
2!
z z
x A
z z z
x A
x A
z S x A S x A
S S S , q , q , q

A = A + A
+ A + A + A

Inserting numbers we get
{ }
1
0*0.01 1*0.01 2*0.0001 2*2*0.0001 0.01 3 0 0
2
z A = + + + + =
This delta z will now be put into row 2 column 5 of the following table.
z A
4/12/2012 2009 Bahill 20
Table 2. Effects of Individual and Combined Parameter Changes for
some second-order interaction terms with delta of 0.01, where
0
x x x = + A etc.
Functions
Normal
values
Values
increased
by delta
New z
values
z A
Total change
in z
( , ) f x A
A=1
x=1
A=1.01
x=1.01
2.0103 0.0103 0.0103
0
( , ) f x A
A=1 A=1.01 2.0100 0.0100
0.0101 z A =

0
( , ) f x A
x=1 x=1.01 2.0001 0.0001
0 0
( , ) f x A
A=1
x=1
2.0000 0

( , ) f y B
B=2
y=1
B=2.01
y=2.01
2.0104 0.0104 0.0104
0
( , ) f y B
B=2 B=2.01 2.0100 0.0100
0.0102 z A =

0
( , ) f y B
y=1 y=2.01 2.0002 0.0002
0 0
( , ) f y B
B=2
y=1
2.0000 0

4/12/2012 2009 Bahill 21
Estimate the mixed-second-partial derivative, according to this formula from Smith,
Szidarovszky, Karnavas and Bahill [2008].
2
0 0 0 0 0 0
( , ) ( , ) ( , ) ( , ) ( , ) f f f f f o | o | o | o | o |
o | o |
c +
~
c c A A

Table 3. Values to be used in estimating the second
partial derivative
Terms
Parameter values with a
0.01 step size
Function
values
( , ) f o |
A=1.01
x =1.01
2.0103
0
( , ) f o |
A=1.00
x =1.01
2.0100
0
( , ) f o |
A=1.01
x =1.00
2.0001
0 0
( , ) f o |
A=1.00
x =1.00
2.0000
2
2.0103 2.0100 2.0001 2.0000 0.0002
2
0.01*0.01 0.0001
z
x A
c +
~ = =
c c

This is the same value that we computed analytically.
4/12/2012 2009 Bahill 22
Table 4. Effects of Individual and Combined Parameter Changes for some third-
order interaction terms, delta = 0.001
Functions
Normal
values
Values
increased
by delta
New z
values
z A
Total change in z
( , ) f x A
A=1
x=1
A=1.001
x=1.002
2.0108 0.0108 0.0108
0
( , ) f x A
A=1 A=1.001 2.0100 0.0100
0.0102 z A =

0
( , ) f x A
x=1 x=1.001 2.0001 0.0001
0
( , ) f x A
x=1 x=1.001 2.0001 0.0001
0 0
( , ) f x A
A=1
x=1
2.0000 0

( , ) f y B
B=2
y=1
B=2.001
y=1.002
2.0112 0.0112 0.0112
0
( , ) f y B
B=2 B=2.001 2.0100 0.0100
0.0104 z A =

0
( , ) f y B
y=1 y=1.001 2.0002 0.0002
0
( , ) f y B
y=1 y=1.001 2.0002 0.0002
0 0
( , ) f y B
B=2
y=1
2.0000 0

4/12/2012
23
Third-order partial derivatives
- Once again the interaction affect is larger than the
sum of the individual changes.
- But at least the third-order terms are smaller than
the first and second-order terms.
- Three of the third-order partial derivatives are
greater than zero.
- All of the fourth-order partial derivatives are zero.
2009 Bahill
Minisummary
- The purposes of this section were
to show the bad affects of too large of a step size
to show how to calculate derivatives analytically
and to estimate derivatives numerically
to show that interactions are important
to show how to consider third- and forth-order
derivatives.


4/12/2012 2009 Bahill 24
4/12/2012
25
A pendulum clock
1
I have a grandfather clock in Tucson that I would
like to move to a cabin up on Mount Lemmon.
But Ive been told that the changes in
temperature and altitude will make it inaccurate.
Which will be the bigger culprit?
The period of oscillation of a pendulum is

A one-meter pendulum has a two-second
period. If the temperature changes by AT, then
the length becomes

Use the absolute-sensitivity function of P with
respect to T to calculate how many seconds per
day the clock will gain.
g l P / 2t =
0
(1 )
T
l l k T = + A
2009 Bahill
4/12/2012
26
A pendulum clock
2
The absolute-sensitivity function is



The coefficient of expansion of a brass rod is

At the normal operating point AT=0 and l=l
0,
so


Mt. Lemmon is 2000 meters higher than Tucson and temperature
changes 5C per 1000 m. So AT=-10C.
Therefore, the change in period is

The pendulum will gain 8.6 seconds per day*.
NOP NOP
2 /
(1
P
T
T
T
l g l k
T
g l k T
S
t t c
= =
c
+ A
5
0
0
/ 2 10 sec/ C
P
T
T
k g
l
S
t

= =

5
2 10 /C
T
k

=
4
2 10 sec
P
T
P S T

A = A =
2009 Bahill
4/12/2012
27
A pendulum clock
3
However, the gravitational acceleration constant depends on
altitude (H)

Therefore the period becomes



and the absolute-sensitivity of P with respect to H is



For this equation the normal operating point is sea level, so
H=0 and g
0
=9.78. So,


6
0
3 10 , where is in meters. g g H H

=

0
2
H
l
P
g k H
t
=

( )
3
0
NOP
P
H
H
H
k l
S
g k H
t
=

2009 Bahill
4/12/2012
28
A pendulum clock
4



Going from Tucson to Mount Lemmon AH=2000, so


The clock loses 26 seconds per day.

Although changes due to temperature and altitude are in the
opposite direction, they do not cancel each other out, because
changes due to altitude are bigger.
0 7
3
0
3 10 s/m
H P
H
k l
S
g
t

= =
4
6 10 sec
P
H
P S H

A = A =
2009 Bahill
4/12/2012
29
Single pole with time delay
1



Use an absolute-sensitivity function to find when the
parameter K has the greatest effect on the step
response of the system. The step response is
( )
( )
( ) 1
s
Y s Ke
M s
R s s
u
t

= =
+
( )
( )
1
s
sr
Ke
s
Y
s s
u
t

=
+
2009 Bahill
4/12/2012
30
Single pole with time delay
2
The absolute-sensitivity function of the step-response
with respect to K is



which transforms into


K has its greatest effect when the response reaches
steady-state.

( ) 1
) (
0

0
+
=

s s
e
s
Y
s
K
S
sr
0 0
( ) /
( ) 1
sr
t
K
y
t e
S
u t
=
2009 Bahill
4/12/2012
31
Types of sensitivity functions
- Analytic
absolute
relative
semirelative
- Empirical
direct observation
sinusoidal variation of parameters
design of experiments
Excel, using what-if analysis

2009 Bahill
4/12/2012
32
The relative-sensitivity function
The relative-sensitivity of the function F to
variations in the parameter o is






Relative-sensitivity functions are used to
compare parameters.
%change in
%change in
F
F F F
S
o
o o o
A
= =
A
0
0
NOP
F
F
F
S
o
o
o
c
=
c
2009 Bahill
4/12/2012
33
Process control model
What is the easiest way (smallest percent change in an
operating point parameter) to increase the quantity of
z that is being produced? Now this problem seems
appropriate for relative-sensitivity functions.

2009 Bahill
Relative-sensitivity functions
4/12/2012 34 2009 Bahill
( )
( )
0 0
0
NOP 0 0
0 0
NOP 0 0
0 0
0 0 0 0 0
NOP 0 0
0 0
0 0 0 0 0
0 0
NOP
3.5,
4,
2 0,
2 0.
z
E
z
F
z
x
z
y
z E E
S y
E z z
z F F
S
F z z
z x x
S A x C y D
x z z
z y y
S B y C x E
y z z
c
= = =
c
c
= = =
c
c
= = + + =
c
c
= = + + =
c
2
0 0
0
NOP 0 0
2
0 0
0
NOP 0 0
0 0
0 0
NOP 0 0
0 0
0
NOP 0 0
0.5,
1,
1.5,
2.5,
z
A
z
B
z
C
z
D
z A A
S x
A z z
z B B
S y
B z z
z C C
S x y
C z z
z D D
S x
D z z
c
= = =
c
c
= = =
c
c
= = =
c
c
= = =
c
4/12/2012
35
F is most important
- Therefore, we should increase F if we wish to
increase z.
- What about interactions? Could we do better by
changing two parameters at the same time?
2009 Bahill
Interactions*
4/12/2012 36 2009 Bahill
2 2 2
0 0 0 0
2 2 2
0 0
NOP
2
0 0 0 0 0
2 2
0 0
NOP
2
0 0
2
0
NOP
2 2
0 0 0 0
2 2
0 0
NOP
2
0 0 0 0 0
2 2
0 0
NOP
2 2*1 *1
0.5,
2
0.75,
1.25,
2
1.0,
0.75,
z
x A
z
x C
z
x D
z
y B
z
y C
z x A x A
S
x A z z
z x C x y C
S
x C z z
z x D
S
x D z
z y B y B
S
y B z z
z y C x y C
S
y C z z

c
= = = =
c c
c
= = =
c c
c
= =
c c
c
= = =
c c
c
= = =
c c
2
2
2
0 0
2
0
NOP
2
0 0 0 0 0
2 2
0 0
NOP
2 2 2
0 0 0
2 2 2
0 0
NOP
2 2 2
0 0 0
2 2 2
0 0
NOP
1.75,
0.75,
2
0.5,
2
1.0.
z
y E
z
x y
z
x
z
y
z y E
S
y E z
z x y x y C
S
x y z z
z x x A
S
x z z
z y y B
S
y z z

c
= =
c c
c
= = =
c c
c
= = =
c
c
= = =
c
4/12/2012
37
Mini-summary
- Using absolute-sensitivity functions, the second- and
third-order terms, e. g.


were the most important, but using relative-
sensitivity functions, F was the most important
parameter.
- The absolute-sensitivity functions show the most
important parameters for a fixed size change in the
parameters
- The relative-sensitivity functions show the most
important parameters for a certain percent change in
the parameters.
3
2
and
z
x A
c
c c
2
2
z
y
c
c
2009 Bahill
4/12/2012
38
Relatively the most important
0 0
NOP 0 0
4
z
F
F F z
S
F z z
c
= = =
c
2009 Bahill
4/12/2012
39
The operating point*
- In the process control example, if the operating point
is changed from (1, 1) to (10, 10), then the output z
becomes most sensitive (relatively) to the input y.
- At the operating point (1, 1), the output is not
sensitive to the inputs, which means we could
twiddle with the inputs forever and not be able to
control the output. Therefore, this is not a desirable
operating point.*
2009 Bahill
More examples of
relative sensitivity functions
4/12/2012 2009 Bahill 40
4/12/2012
41
Double pole with time-delay
Which of the parameters is
most important?
2
( )
( )
( )
( 1)
s
Y s Ke
M s
R s
s
u
= =
+
0
0
NOP
1
M
K
M
K
M
K
S
c
= =
c
0
0
0

NOP

M
M
s
M
S
c
= =
c
0
0
0
NOP 0
2

1
M s M
M
s
S
t
t
c
= =
c +
2009 Bahill
4/12/2012
42
Frequency domain output
4/12/2012
43
Results
- For low frequencies, K is biggest
- For mid-frequencies, t is biggest
- For high frequencies, u is biggest
2009 Bahill
A simple closed loop system
4/12/2012 44 2009 Bahill
Transfer function
( )
( )
( )
Y s K
M s
R s s K
= =
+

Time-domain step-response
( ) 1
Kt
sr
y t e

=
Time-domain relative-sensitivity function
0
0 0
0 0
1 1
sr
K t
y Kt
K
K t K t
NoP
K K te
S te
e e

= =


Frequency-domain step-response
( )
( )
sr
K
Y s
s s K
=
+

Frequency-domain relative-sensitivity function
0

( )
sr
Y
K
s
S s
s K
=
+

Take the inverse Laplace transform
0
0

( )
sr
Y K t
K
S t K e o

=
where o is the unit impulse.
4/12/2012 45 2009 Bahill
Time domain output
4/12/2012
46
An expert system
Sensitivity functions need not be functions of
time or frequency.
If premise1 = true (CF
1
)
and premise2 = true (CF
2
)
or premise3 = true (CF
3
)
then conclusion = true CF
4
.
The certainty of the rule uses the minimum of
the certainties of the AND clauses.
2009 Bahill
Certainty factor domain output
4/12/2012 47 2009 Bahill
If CF
1
< CF
2
, then the final certainty factor
becomes
3 4 1 4 1 4
100
100 100 10, 000
f
CFCF CFCF CFCF
CF
| |
| |
= +
| |
\ .
\ .

The relative-sensitivity functions are
1
3 4 10 4
0
NOP
1
10, 000 100
f
CF
CF
f
CFCF CF CF
S
CF
| |
=
|
\ .

2
0
f
CF
CF
S =
3
30 1 4 4
0
NOP
1
10, 000 100
f
CF
CF
f
CF CFCF CF
S
CF
| |
=
|
\ .

4
3 1 3 4 40 1
0
NOP
2
100 100 1, 000, 000
f
CF
CF
f
CF CFCFCF CF CF
S
CF
= +
4/12/2012 48 2009 Bahill
Assume
1 3 4 2
80 and 81 CF CF CF CF = = = =
Then
0
87
f
CF = and
1
0.26
f
CF
CF
S =
2
0
f
CF
CF
S =
3
0.26
f
CF
CF
S =
4
0.53
f
CF
CF
S =
Changes in CF
4
are twice as important
as changes in CF
1
or CF
3

and increases in CF
2
have no effect.
4/12/2012
49
Limitations of the
relative-sensitivity function
( ) ( ) ( ) ( )
L x t y t Lx t Ly t = (


( )
0
NOP 0
( ) ( ) ( )
( )
f t
f t f t f t
S
f t
o
o
o o o
c c
= =
c c

( )
0
NOP 0
( ) ( ) ( )

( )
F s
F s F s F s
S
F s
o
o
o o o
c c
= =
c c

Because
( ) ( )

f t F s
S S
o o
= we have two functions
2009 Bahill
4/12/2012
50
Disadvantages of
relative-sensitivity functions
- different in time and frequency domains
- cannot use Laplace transforms to get time-
domain solution
- division by zero problem
0
0
NOP
F
F
F
S
o
o
o
c
=
c
2009 Bahill
4/12/2012
51
Types of sensitivity functions
- Analytic
absolute
relative
semirelative
- Empirical
direct observation
sinusoidal variation of parameters
design of experiments
Excel, using what-if analysis

2009 Bahill
4/12/2012
52
Double pole with time delay

2009 Bahill
4/12/2012
53
The semirelative-sensitivity function
The semirelative-sensitivity of the function F to
variations in the parameter o is
0
NOP
F
F
S
o
o
o
c
=
c
2009 Bahill
Tradeoff study
4/12/2012 54 2009 Bahill
A Generic Tradeoff Study
Criteria
Weight of
Importance
Alternative-1 Alternative-2
Criterion-1 Wt
1
S
11
S
12

Criterion-2 Wt
2
S
21
S
22

Alternative
Rating
Sum
1
Sum
2


A Numeric Example of a Tradeoff Study
Alternatives
Criteria
Weight of
Importance
Umpires
Assistant
Seeing
Eye Dog
Accuracy 0.75 0.67 0.33
Silence of
Signaling
0.25 0.83 0.17
Sum of weight
times score

0.71
The
winner
0.29

1 1 11 2 21 2 1 12 2 22
and Sum Wt S Wt S Sum Wt S Wt S = + = +
4/12/2012
55
Which parameters could change
the recommendations?
Use this performance index



Compute the semirelative-sensitivity functions.

1 1 2
1 11 2 21 1 12 2 22
0.420
PI Sum Sum
Wt S Wt S Wt S Wt S
=
= + =
2009 Bahill
Semirelative-sensitivity functions*
4/12/2012 56 2009 Bahill

( )
( )
1
2
11
21
12
22
11 12 1
21 22 2
1 11
2 21
1 12
2 22
0.26
0.16
0.50
0.21
-0.25
-0.04
F
Wt
F
Wt
F
S
F
S
F
S
F
S
S S S Wt
S S S Wt
S Wt S
S Wt S
S Wt S
S Wt S
= =
= =
= =
= =
= =
= =
A sensitivity matrix
4/12/2012 57 2009 Bahill
Analytic semirelative-sensitivity function values for PI
1,
the
difference of the alternative ratings
Alternatives
Criteria
Weight of
Importance
Umpires
Assistant
Seeing Eye
Dog
Accuracy
1
1
PI
Wt
S = 0.26
1
11
PI
S
S = 0.50
1
12
PI
S
S = -0.25
Silence of
Signaling
1
2
PI
Wt
S = 0.16
1
21
PI
S
S
= 0.21
1
22
PI
S
S = -0.04

A nice way to
display the sensitivities
4/12/2012
58
What about interactions?
The semirelative-sensitivity function of PI
1
for the
interaction of Wt
1
and S
11
is


which is as big as the first-order terms.

1 11 0 0 0 0
2
1 11 1 11
1 11
NOP
0.5025
F
Wt S
F
S Wt S Wt S
Wt S

c
= = =
c c
2009 Bahill
4/12/2012
59
Interactions for PI
1







So interactions are important.
Semirelative Sensitivity Values Showing Interaction Effects
Function
Normal
values
Values
increased
by 10%
New F
values
F A
Total change
in z
1
F
Wt
S
1
Wt =0.75
1
Wt =0.82
0.446 0.026
11
F
S
S
11
S =0.67
11
S =0.74
0.470 0.050
0.076 F A =

1 11
F
Wt S
S


1
Wt =0.75
11
S =0.67
1
Wt =0.82
11
S =0.74
0.501 0.081 0.081

2009 Bahill
4/12/2012
60
A new performance index
- A problem with performance index PI
1
is that if s
11
=s
12
,
then the sensitivity with respect to Wt
1
becomes equal
to zero.
- Mathematically this is correct, but logically it is wrong.
- Another problem is that the sensitivity with respect to
Wt
1
does not depend on scores for the nonwinning
alternatives and we do want the sensitivities to depend
on the other parameters.
- The following performance index solves both of these
problems.

3
1 1
1
n m
i ij
i j
PI Wt S
m
= =
=

2009 Bahill
4/12/2012
61
Semirelative sensitivity functions

3
3
1
NOP
1
i
m
PI
Wt i i ij
j
i
PI
S Wt Wt S
Wt m
=
c
= =
c


3
ij
i ij
PI
S
Wt S
S
m
=
2009 Bahill
Sensitivity matrix for PI
3
Table VI. Analytic semirelative-sensitivity function values
for PI
3
, the sum of all weight times scores
Alternatives
Criteria
Weight of
Importance
Umpires
Assistant
Seeing Eye
Dog
Accuracy of
the call
3
1
0.38
PI
Wt
S =
3
11
0.25
PI
S
S =
3
12
0.12
PI
S
S =
Silence of
Signaling
3
2
0.13
PI
Wt
S =
3
21
0.10
PI
S
S =

3
22
0.02
PI
S
S =

4/12/2012 2009 Bahill 62
4/12/2012
63
It is not difficult
Although these equations may look formidable, they
are easy to compute with a spreadsheet. For example


is merely the sum of the weight times scores in column
k and this is already in the spreadsheet. Furthermore,
because


and the rest of the second order sensitivities are zero,
Table VI is complete: it has all of the sensitivities in it.

1
n
k kj
k
Wt S
=

3 3
ij i ij
PI PI
S Wt S
S S

=
2009 Bahill
4/12/2012
64
What about interactions?



- Yes, We do have to worry about interactions,
because this is bigger than most of the first order
sensitivities.
3
1 11 0 0
2
3 1 11
1 11
1 11
NOP
0.25
PI
Wt S
PI Wt S
S Wt S
Wt S m

c
= = =
c c
2009 Bahill
4/12/2012
65
Interaction Sensitivity Matrix
Table VII. Analytic semirelative-sensitivity function values for
the interactions of PI
3

Alternatives
Criteria
Weight of
Importance
Umpires
Assistant
Seeing Eye
Dog
Accuracy of
the call

3
1 11
0.25
PI
Wt S
S

=
3
1 12
0.12
PI
Wt S
S

=
Silence of
Signaling

3
2 21
0.10
PI
Wt S
S

=

3
2 22
0.02
PI
Wt S
S

=
These cells contain the same numerical values as Table VI.

2009 Bahill
Tradeoff studies are hierarchical
4/12/2012 66 2009 Bahill
The structure of a hierarchical tradeoff study
Criteria
Normalized
Criteria
Weights
Subcriteria
Normalized
Subcriteria
Weights
Scores for
Alternative-1
Scores for
Alternative-2
Performance
(1)
CW

Subcriteria-1
(1)
1
Wt
(1)
11
S
(1)
12
S
Subcriteria-2
(1)
2
Wt
(1)
21
S
(1)
22
S
Subcriteria-3
(1)
3
Wt
(1)
31
S
(1)
32
S
Subcriteria-4
(1)
4
Wt
(1)
41
S
(1)
42
S
Cost
(2)
CW

Subcriteria-1
(2)
1
Wt
( 2)
11
S
( 2)
12
S
Subcriteria-2
(1)
2
Wt
( 2)
21
S
( 2)
22
S
Schedule
(3)
CW

Subcriteria-1
(1)
1
Wt
(3)
11
S
(3)
12
S
Subcriteria-2
(3)
2
Wt
(3)
21
S
(3)
22
S
Risk
(4)
CW

Subcriteria-1
(4)
1
Wt
( 4)
11
S
( 4)
12
S
Subcriteria-2
(4)
2
Wt
( 4)
21
S
( 4)
22
S
Subcriteria-3
(4)
3
Wt
( 4)
31
S
( 4)
32
S
Alternative
Ratings

1
Sum
2
Sum

4/12/2012
67
A new performance index, PI5
Because most of tradeoff studies are hierarchical, in the
Spin Coach and the PopUp Coach I used this
performance index
( )
( ) ( ) ( )
5
1 1 1
1
n l
k m
l l l
i ij
l i j
PI CW Wt S
m
= = =
=

5
( )
( )
( ) ( ) ( )
1 1
1
l
n l
m
PI l l l
i ij
CW
i j
S CW Wt S
m
= =
=

5
( ) ( ) ( )
1
ij
PI l l l
S i ij
S CW Wt S
m
=
2009 Bahill
4/12/2012
68
Single pole with time delay (when)



Transfer function
( )
( )
( ) 1
s
Y s Ke
M s
R s s
u
t

= =
+

Step-response
( )
( 1)
s
sr
Ke
Y s
s s
u
t

=
+

Semirelative-sensitivity functions
0
0
0
( 1)
sr
s
Y
K
K e
S
s s
u
t

=
+

0
0 0
0
( 1)
sr
s
Y
K e
S
s
u
u
u
t

=
+

0
0 0
2
0
( 1)
sr
s
Y
K e
S
s
u
t
t
t

=
+

Which (for
0
t u > ) transform to
0 0
( )
0
( ) (1 )
sr
y t
K
S t K e
u t
=
0 0
( )
0 0
( )
sr
y t
S t K e
u t
u
u

=
0 0
( )
0 0 0
( ) ( )
sr
y t
S t K t e
u t
t
t u

=
2009 Bahill
4/12/2012 69 2009 Bahill
4/12/2012
70
What does this teach us?
- If the model does not match the physical system in
the early part of the step response, then adjust the
time-delay of the model.
- For steady state
- In the middle
2009 Bahill
4/12/2012
71
Types of sensitivity functions
- Analytic
absolute
relative
semirelative
- We have just examined the analytic sensitivity
functions. We are now ready to look at the
empirical sensitivity functions.
- Empirical
direct observation
sinusoidal variation of parameters
design of experiments
Excel, using what-if analysis

2009 Bahill
4/12/2012
72
Empirical sensitivity functions
1
- The method of direct observation
can be preformed on
real-world systems
models of those systems
simulations of those models
2009 Bahill
4/12/2012 73 2009 Bahill
4/12/2012 74 2009 Bahill
Real
System
Model of
Real
System
Computer
Simulation
of Model
Modelers Modelers
Good
Modelers
Good
Modelers
M
a
t
h
e
m
a
t
i
c
i
a
n
s
E
x
p
e
r
i
m
e
n
t
a
l
i
s
t
s
4/12/2012
75
Estimating derivatives



are small, then the second term on the right can be
neglected.
0
If (x-x ) and ( ) f ,
''
2009 Bahill
4/12/2012
76
Tradeoff study example
For a +10% parameter change, the semirelative-
sensitivity function is


This is very easy to compute.









Tradeoff Study Matrix with S
11
Increased by 5%
Criteria
Weight of
Importance
Umpires
Assistant
Seeing
Eye dog
Accuracy 0.75 0.74 0.33
Silence of Signaling 0.25 0.83 0.17
Sum of weight times
score
0.76 0.29


0 0
0
10
0.1
F
F F
S F
|
| |
| |
A A
~ = = A
A
2009 Bahill
Sensitivity matrix
4/12/2012 2009 Bahill 77
Table X. Numerical estimates for semirelative-sensitivity function
for PI
3
, the sum of the alternative ratings squared, for a plus 10%
parameter perturbation
Alternatives
Criteria
Weight of
Importance
Umpires
Assistant
Seeing Eye
Dog
Accuracy of
the call
3
1
0.38
PI
Wt
S =

3
11
0.25
PI
S
S =

3
12
0.12
PI
S
S =

Silence of
Signaling
3
2
0.13
PI
Wt
S =

3
21
0.10
PI
S
S =

3
22
0.02
PI
S
S =


These are the same results
that were obtained in the analytic
semirelative sensitivity section.
4/12/2012
78
But what about the second-order terms?
Namely

When using the sum of weighted scores combining function

the second derivatives are all zero. So our estimations are all
right. This is not true for the product combining function,

most other combining functions (See Daniels, Werner and
Bahill [2001] for explanations of other combining functions.)
or other performance indices.
In particular lets try PI
3
.

2
0
( )
( )
2!
f
x x
,
''

1 1 11 2 21 2 1 12 2 22
and Sum Wt S Wt S Sum Wt S Wt S = + = +

1 2 1 2
1 11 21 2 12 22
and
Wt Wt Wt Wt
F S S F S S = =
2009 Bahill
4/12/2012
79
Derivative of a function of two variables



- Let us examine the second-order terms, those inside
the { }, for two reasons
to see if they are large and must be included in
computing the first derivative
to estimate the effects of interactions on the
sensitivity analysis
{ }
0 0 0 0 0 0 0 0
2 2
0 0 0 0
( , ) ( , ) ( , )( ) ( , )( )
1
( , )( ) 2 ( , )( )( ) ( , )( )
2!
x y
xx xy yy
f x y f x y f x y x x f x y y y
f x x f x x y y f y y , q , q , q
' '
= +
'' '' ''
+ + +
2009 Bahill
4/12/2012
80
Interactions
Previously we derived the analytic semirelative-
sensitivity function for the interaction of Wt
1
and S
11

as,


which is as big as the first-order semirelative-sensitivity
functions.

0 0
3
1 11 0 0
2
1 11
3
1 11
1 11
NOP
0.25
PI
Wt S
Wt S
PI
S Wt S
Wt S m

c
= = =
c c
2009 Bahill
4/12/2012
81
Interactions
For a 10% change in parameter values, a simple-minded
approximation is



using our tradeoff study values we get


This does not match the analytic value.

What went wrong?


( )
( )
2
2
0 0 0 0
0 0
10
0.1 0.1
F
F
F F
S F
o |
o | o |
o | o |

A
A A
~ = = A
A A
( )
3
1 11
2
10 0.424
PI
Wt S
S F

~ A =
2009 Bahill
4/12/2012
82
Maybe the step size is too big
- Lets reduce the perturbation step size to 0.1%?




- This is closer, but it is still too big.
( )
( )
2
2
0 0 0 0
0 0
1000
0.001 0.001
F
F
F F
S F
o |
o | o |
o | o |

A
A A
~ = = A
A A
( )
3
1 11
2
1000 0.393
PI
Wt S
S F

~ A =
2009 Bahill
4/12/2012
83
What went wrong?
In the previous computations, we
changed both parameters at the
same time and then compared the
value of the function to the value of
the function at its normal operating
point. However, this is not the
correct estimation for the second-
partial derivative.
2009 Bahill
4/12/2012
84
Estimating the second partials
1
To estimate the second-partial derivatives we should
start with
2
0 0 0 0 0
( , ) ( , ) ( , ) f f f
o o
o | o | o |
o | |
' '
c
~
c c A
0 0 0 0
2
0 0
( , ) ( , ) ( , ) ( , )
( , )
f f f f
f
o | o | o | o |
o |
o o
o | |

c
A A
~
c c A

2
0 0 0 0 0 0
( , ) ( , ) ( , ) ( , ) ( , ) f f f f f o | o | o | o | o |
o | o |
c +
~
c c A A
0
| | | = + A
2009 Bahill
Estimating the second partials
2
4/12/2012 85 2009 Bahill
Values to be Used in Estimating the Second Derivative
Terms
Parameter values with a 0.1% step size,
that is
1
Wt A =0.00075 and
11
S A =0.00067
Function
values
( , ) f o |
1
Wt =0.75075
11
S =0.67067
0.50063
0
( , ) f o |
11
S =0.67067
0.50025
0
( , ) f o |
1
Wt =0.75075
0.50038
0 0
( , ) f o |
1
Wt =0.75000
11
S =0.67000
0.50000


2
3
1 11
0.50063 0.50038 0.50025 0.50000
0.5
*0.00075*0.00067
PI
Wt S m
c +
~ =
c c
4/12/2012
86
Estimating the sensitivity functions
To get the semirelative-sensitivity function we multiply
the second-partial derivative by the normal values of
Wt
1
and S
11
to get



Now, this is the same result that we derived in the
analytic semirelative sensitivity section.
3
1 11 0 0 0 0
2
3
1 11 1 11
1 11
NOP
0.5 0.25
PI
Wt S
PI
S Wt S Wt S
Wt S

c
= = =
c c
2009 Bahill
4/12/2012
87
Lessons learned
- For a tradeoff study using the sum combining
function and a simple performance index,
anything works.
- Otherwise, the perturbation step size should be
small. Five and 10% perturbations are not
acceptable.
- It is incorrect to estimate the second partial
derivative by changing two parameters at the
same time and then comparing that value of the
function to the value of the function at its
normal operating point. Estimating second
derivatives requires evaluation of four not two
numerator terms.
2009 Bahill
4/12/2012
88
Sensitivity analysis of a risk analysis
Let
be the probability of occurrence, the severity, and the
risk, for the j
th
failure mode. Risk is

Use the performance index* and
calculate the semirelative-sensitivity functions


The largest sensitivities are always those for the largest
risk. This means that we should spend extra time and
effort estimating the probability and severity of the
highest ranked risk, which seems eminently sensible.

, and
j j j
P S R

j j j
R P S =
1
n
j
j
PI R
=
=

0 j
PI
P j j j
S S P R = =

0 j
PI
S j j j
S R S R = =
2009 Bahill
4/12/2012
89
Linearity
Although the model is linear, the
sensitivity functions are not.
2009 Bahill
4/12/2012 90 2009 Bahill
4/12/2012
91
Types of sensitivity functions
- Analytic
absolute
relative
semirelative
- Empirical
direct observation
sinusoidal variation of parameters
design of experiments
Excel, using what-if analysis

2009 Bahill
4/12/2012
92
Empirical sensitivity functions
2
- Sinusoidal variation of parameters,
also called
frequency-domain experiments
response-surface methodology

2009 Bahill
4/12/2012
93
Sinusoidal variation of parameters
- Make two runs of the system
1. all parameters are set at their normal values
2. all parameters are modulated sinusoidally
- Compute the power spectrum of each
- Form the ratio of the two spectra at each frequency
- Spikes will be observed
at the modulation frequencies
at frequencies related to nonlinearities
at frequencies related to product effects
2009 Bahill
4/12/2012
94
A first-order, negative-feedback system
K
______
s + A
H
E
R(s) Y(s) +
-
2009 Bahill
4/12/2012
95
A first-order, negative-feedback, control
system (continued)



- Input: one Hertz, unit-amplitude sinusoid
- Duration: one second
- Sampling: 2048 evenly spaced samples
- Modulation frequencies: 5, 30 and 170 Hz
HK A s
K
s R
s Y
s M
+ +
= =
) (
) (
) (
2009 Bahill
4/12/2012
96
Modulation equations
- A = 0.1 (1 + 0.5 sin (5 2tt))
- H = 50 (1 + 0.5 sin (30 2tt))
- K = 1.0 (1 + 0.5 sin (170 2tt))
2009 Bahill
Step response with modulated parameters
4/12/2012 97 2009 Bahill
4/12/2012
98
Location of spikes
- If a parameter is modulated at e
a
we expect to see
power at e
a

- If there is a parabolic nonlinearity we also expect power
at 2e
a

because 2 sin
2
x = 1 - cos 2x
- If the system is sensitive to the product of two
parameters modulated at e
a
and e
b,
then we expect
power at e
a
e
b
because 2 sin x sin y = cos(x-y) - cos(x+y)
These product terms are called interactions.
2009 Bahill
4/12/2012 99 2009 Bahill
4/12/2012
100
Split peaks are due to the one Hertz input.

This technique probably produces relative-sensitivities.

2009 Bahill
The semirelative-sensitive functions*
4/12/2012 101 2009 Bahill
( )
( ) ( ) 1 . 50
~
2 2
0 0
1 . 0
0 0 0
+ + +
+
=
+
=
s
H K A
s
A K
S
s
s M
K


( ) ( ) 1 . 50
~
2 2
0
50
0 0 0
2
0
+ + +

= =
s
H K A
s
H
K
S
M
H


( ) ( ) 1 . 50
~
2 2
0 0
1 . 0
0 0 0
+ + +


= =
s
H K A
s
A K
S
M
A

4/12/2012
102
An M/M1 queue
- service rate =
= 0.8 + 0.2 sin(46/4096 2tt)
- arrival rate =
= 0.4 + 0.2 sin(4/4096 2tt)
2009 Bahill
4/12/2012
103
An M/M1 queue
- service rate =
= 0.8 + 0.2 sin(46/4096 2tt)
- arrival rate =
= 0.4 + 0.2 sin(4/4096 2tt)
2009 Bahill
4/12/2012 104 2009 Bahill
4/12/2012
105
- An M/M1 queue is a low pass filter
- The interaction peaks are not the same height.
2009 Bahill
Bode diagram of a low-pass filter
4/12/2012 106 2009 Bahill
4/12/2012
107
Some problems with sinusoidal
variation of parameters
- There must be an input signal.
- Shape of the spikes depends on parameters of the input
signal.
- The output cannot be stationary.
- The frequency response of the system (e.g. low-pass,
high-pass, resonance, etc.) must be known.
- The range of linearity of the system must be known.*
- Parameters of the FFT and the windows must be
understood.
2009 Bahill
4/12/2012 108 2009 Bahill
4/12/2012
109
Types of sensitivity functions
- Analytic
absolute
relative
semirelative
- Empirical
direct observation
sinusoidal variation of parameters
design of experiments (DoE)
Excel, using what-if analysis

2009 Bahill
4/12/2012
110
Empirical sensitivity functions
3
- Design of experiments (DoE) should be used when
experiments are expensive.
- When doing a Taguchi 3-level design pick a normal
value, a high value and a low value.
If the high and low are some percentage change,
then you are doing a relative sensitivity analysis.
If the high and low are plus and minus a unit, then
you are doing an absolute sensitivity analysis.
Alternatively, the high and low could be realistic
design options, in which case it does not
correspond to any of our sensitivity functions.

2009 Bahill
4/12/2012 111 2009 Bahill
Low High
Altitude
Temperature
Relative Humidity
Barometric Pressure
P
e
r
c
e
n
t

C
h
a
n
g
e

i
n

A
i
r

D
e
n
s
i
t
y
0 ft
70 F 2600 feet
85 F
50%
760 mm Hg
90%
745 mm Hg
10%
775 mm Hg
100 F
5200 feet
1%
3%
5%
7%
9%
-1%
-3%
-5%
-7%
-9%
Percent change in air density
over the parameter ranges to
be expected for a typical July
afternoon in United States
ballparks.
Medium
4/12/2012
112
Types of sensitivity functions
- Analytic
absolute
relative
semirelative
- Empirical
direct observation
sinusoidal variation of parameters
design of experiments
Excel, using what-if analysis

2009 Bahill
The Pinewood Derby
4/12/2012 113 2009 Bahill
4/12/2012
114
The most important parameters
in the Pinewood Derby
- baseline for Overall Happiness scoring
function
- baseline for Percent Happy Scouts
scoring function
- importance weight for Overall
Happiness evaluation criterion
- baseline for Number of Repeat Races
scoring function
- input value for Percent Happy Scouts
evaluation criterion
- input value for Number of Repeat
Races evaluation criterion

2009 Bahill
4/12/2012
115
Of the 89 parameters only 3 could
change the preferred alternative
- 1. The Tradeoff Function
with 90% for Performance and 10% for
Utilization of Resources the preferred
alternative was a round robin with best
time scoring
with 57% for Performance and 43% for
Resources the preferred alternative
switched to the double elimination
tournament
2009 Bahill
4/12/2012 116 2009 Bahill
Sensitivity Analysis of Pinewood Derby (simulation data)
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Performance Weight
O
v
e
r
a
l
l

S
c
o
r
e
Single elimination
Double elimination
Round robin, mean-time
Round robin, best-time
Round robin, points
4/12/2012 117 2009 Bahill
Sensitivity of Pinewood Derby (prototype data)
0.5
0.55
0.6
0.65
0.7
0.75
0.8
0.85
0.9
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Performance Weight
O
v
e
r
a
l
l

S
c
o
r
e
Double elimination
Round robin, best-time
Round robin, points
4/12/2012
118
Of the 89 parameters only 3 could
change the preferred alternative
2
- 2. The slope of the
Percent Happy Scouts scoring function
- 3. The baseline for the
Percent Happy Scouts scoring function

2009 Bahill
4/12/2012 119 2009 Bahill
4/12/2012
120
Validation
1
- If a system (or its model) is very sensitive to
parameters over which the customer has no control,
then it may be the wrong system for that customer.
- If the sensitivity analysis reveals the most important
parameters and that result is a surprise, then it may be
the wrong system.
- If a system is more sensitive to its parameters than to
its inputs, then it may be the wrong system or the
wrong operating point.
- If the sensitivities of the model are different from the
sensitivities of the physical system, then it may be the
wrong model.

2009 Bahill
4/12/2012
121
Validation
2
- If you delete a requirement, then your completeness
measure (a traceability matrix) should show a vacuity.
- After you make a decision, do a sensitivity analysis and
see if changing a parameter would change your
decision.
- Domain experts should agree with the sensitivity
analysis about which criteria in a tradeoff study are the
most important.
- Domain experts should agree with the sensitivity
analysis about which risks are the most important.
- Do a sensitivity analysis of prioritized lists: see if
changing the most important criteria would change the
prioritization.

2009 Bahill
4/12/2012
122
Verification
- Unplanned excessive sensitivity to any parameter is a
verification mistake.
- Sensitivity to interactions should be flagged and
studied: such interactions may be unexpected and
undesirable.
2009 Bahill
4/12/2012
123
Rsum
- After you
build a model, or
write a set of requirements, or
do a tradeoff study, or
design a system,
you should study that thing to see if it
makes sense.
- One of the best ways to study a thing is
with a sensitivity analysis.
2009 Bahill
Describe this talk to your Vice President
- Professor Bahill modeled our potion production
process and did a sensitivity analysis of it. So we now
have a better understanding of our process. His
sensitivity analysis accounts for nonlinearities and
parameter interactions. His equations for estimating
parameters are correct (because Szidarovszky
derived them). This analysis shows which parameters
are the most important for making our potent.
- I recommend that we name our potion Love Potion
Number Nine. We should buy the copyright for that
song and play it in the background of our TV
commercials.

- Play an audio clip.


4/12/2012 2009 Bahill 124
4/12/2012 125 2009 Bahill