Nonlinear Optimization: Introduction

Linear vs. nonlinear objective functions

Linear Nonlinear

When objective function is linear Optimum always attained at constraint boundaries A local optimum is also a global optimum When objective function is nonlinear Optima may be in the interior as well as at boundaries of constraints A local optimum is not necessarily a global optimum
September 12, 2007 (1 : 6)

Nonlinear Optimization: Introduction

Convexity and optimality

f(x)

A convex function f on a convex set S : f is below or on the linear interpolant between any two points in the set x1 , x2 S
)+ (x 1 f
x1

( )f 1

x 2)

x2

f x1 + (1 )x2 f (x1 ) + (1 )f (x2 ) [0, 1] Examples of convex functions on R n (a) f (x ) = c T x (also concave!) T (b) f (x ) = + c T x + 1 2 x Qx (Q positive semidenite matrix)
September 12, 2007 (2 : 6)

Nonlinear Optimization: Introduction

S Global minimizer: x S such that f (x ) f (x ) x S Local minimizer: x S such that, for some > 0, f (x ) f (x ) x S B (x ; ) where B (x ; ): ball of radius

Rn

f(x)
Local Local

Global

centered at x

Theorem. For convex functions on convex sets holds that each local minimizer is a global minimizer
Example: min f (x ) such that
x

Ax b , where f (x ) = c T x (linear program) or T f (x ) = + c T x + 1 2 x Qx , Q positive semidenite (quadratic program)

September 12, 2007 (3 : 6)

Nonlinear Optimization: Introduction

Proof. Let x be a local minimizer. Then > 0 s. t. f (x ) f (x ) x S B (x ; ). Let x S arbitrarily. Will show f (x ) f (x ). S convex x + (1 )x = x S x local min f (x ) f (x ) = f x + (1 )x [f convex] f (x ) + (1 )f (x ), that is, f (x ) f (x ) f (x ) f (x )

[0, 1]

Choose > 0 such that x S B (x ; )

>0

September 12, 2007

(4 : 6)

Nonlinear Optimization: Introduction

Unconstrained optimization

Will start to consider unconstrained optimization min f (x ) n or, equivalently, Find x R n such that f (x ) f (x ) x R n

x R

Function f is nonlinear in x . Unconstrained optimization meaningless for linear f , since linear f on R n are unbounded or constant Most common application for unconstrained optimization: inverse problems, parameter estimation

September 12, 2007

(5 : 6)

Nonlinear Optimization: Introduction

Unconstrained optimization: example application

At time intervals t1 , t2 , . . . , tm a physical process generates a time sequence of m observations (measurements) b1 , b2 , . . . , bm A model of the process says that bk b (tk ) where b (t ) = x1 + x2 e x3 t + x4 e x5 t Model not exact: measuring errors (noise), modeling errors We want to nd the coecients x1 , . . . , x5 that best matches the observations Dene the residual rk = bk (x1 + x2 e x3 tk + x4 e x5 tk ) and solve the unconstrained optimization problem
m x1 ,...,x5

min

2 rk k =1

September 12, 2007

(6 : 6)