Está en la página 1de 36

Tutorial on Convex Optimization

for Engineers
Part I
M.Sc. Jens Steinwandt
Communications Research Laboratory
Ilmenau University of Technology
PO Box 100565
D-98684 Ilmenau, Germany
jens.steinwandt@tu-ilmenau.de

January 2014

Course mechanics
strongly based on the advanced course Convex Optimization I by

Prof. Stephen Boyd at Stanford University, CA


info, slides, video lectures, textbook, software on web page:

www.stanford.edu/class/ee364a/
mandatory assignment (30 % of the final grade)

Prerequisites
working knowledge of linear algebra

Course objectives
to provide you with
overview of convex optimization
working knowledge on convex optimization

in particular, to provide you with skills about how to


recognize convex problems
model nonconvex problems as convex
get a feel for easy and difficult problems
efficiently solve the problems using optimization tools

Outline

1. Introduction
2. Convex sets
3. Convex functions
4. Convex optimization problems
5. Lagrangian duality theory
6. Disciplined convex programming and CVX

1. Introduction

Mathematical optimization
(mathematical) optimization problem
minimize f0(x)
subject to fi(x) bi,

i = 1, . . . , m

x = (x1, . . . , xn): optimization variables


f0 : Rn R: objective function
fi : Rn R, i = 1, . . . , m: constraint functions

optimal solution x has smallest value of f0 among all vectors that


satisfy the constraints

Introduction

12

Solving optimization problems


general optimization problem
very difficult to solve
methods involve some compromise, e.g., very long computation time, or
not always finding the solution

exceptions: certain problem classes can be solved efficiently and reliably


least-squares problems
linear programming problems
convex optimization problems

Introduction

14

Least-squares
minimize kAx bk22
solving least-squares problems
analytical solution: x = (AT A)1AT b
reliable and efficient algorithms and software
computation time proportional to n2k (A Rkn); less if structured
a mature technology
using least-squares
least-squares problems are easy to recognize
a few standard techniques increase flexibility (e.g., including weights,
adding regularization terms)
Introduction

15

Linear programming
minimize cT x
subject to aTi x bi,

i = 1, . . . , m

solving linear programs


no analytical formula for solution
reliable and efficient algorithms and software
computation time proportional to n2m if m n; less with structure
a mature technology
using linear programming
not as easy to recognize as least-squares problems
a few standard tricks used to convert problems into linear programs
(e.g., problems involving 1- or -norms, piecewise-linear functions)
Introduction

16

Convex optimization problem


minimize f0(x)
subject to fi(x) bi,

i = 1, . . . , m

objective and constraint functions are convex:


fi(x + y) fi(x) + fi(y)
if + = 1, 0, 0
includes least-squares problems and linear programs as special cases

Introduction

17

Why is convex optimization so essential?


convex optimization formulation
always achieves global minimum, no local traps
certificate for infeasibility
can be solved by polynomial time complexity algorithms (e.g., interior point

methods)
highly efficient software available (e.g., SeDuMi, CVX)
the dividing line between easy and difficult problems (compare with

solving linear equations)


= whenever possible, always go for convex formulation

Introduction

Brief history of convex optimization


theory (convex analysis): ca19001970
algorithms

1947: simplex algorithm for linear programming (Dantzig)


1960s: early interior-point methods (Fiacco & McCormick, Dikin, . . . )
1970s: ellipsoid method and other subgradient methods
1980s: polynomial-time interior-point methods for linear programming
(Karmarkar 1984)
late 1980snow: polynomial-time interior-point methods for nonlinear
convex optimization (Nesterov & Nemirovski 1994)
applications
before 1990: mostly in operations research; few in engineering
since 1990: many new applications in engineering (control, signal
processing, communications, circuit design, . . . ); new problem classes
(semidefinite and second-order cone programming, robust optimization)
Introduction

115

2. Convex sets

Subspaces
a space S Rn is a subspace if
x1 , x2 S,

1 , 2 R

1 x1 + 2 x2 S

geometrically: plane through x1 , x2 S and the origin


representation:
range(A) = {Aw | w Rq }

(A = [a1 , . . . aq ])

= {w1 a1 + wq aq | wi R}
= span {a1 , a2 , . . . , aq }
null(A) = {x | Bx = 0}

(B = [b1 , . . . bp ]T )

= x | b1T x = 0, . . . , bpT x = 0

Convex sets

Affine set
line through x1, x2: all points
x = x1 + (1 )x2

( R)

x1
= 1.2
=1
= 0.6
x2
=0
= 0.2

affine set: contains the line through any two distinct points in the set
example: solution set of linear equations {x | Ax = b}
(conversely, every affine set can be expressed as solution set of system of
linear equations)
Convex sets

22

Convex set
line segment between x1 and x2: all points
x = x1 + (1 )x2
with 0 1
convex set: contains line segment between any two points in the set
x1, x2 C,

01

x1 + (1 )x2 C

examples (one convex, two nonconvex sets)

Convex sets

23

Convex combination and convex hull


convex combination of x1,. . . , xk : any point x of the form
x = 1 x1 + 2 x2 + + k xk
with 1 + + k = 1, i 0
convex hull conv S: set of all convex combinations of points in S

Convex sets

24

Convex cone
conic (nonnegative) combination of x1 and x2: any point of the form
x = 1 x1 + 2 x2
with 1 0, 2 0

x1
x2
0

convex cone: set that contains all conic combinations of points in the set

Convex sets

25

Hyperplanes and halfspaces


hyperplane: set of the form {x | aT x = b} (a 6= 0)
a
x0
x
aT x = b

halfspace: set of the form {x | aT x b} (a 6= 0)


a
x0

aT x b

aT x b

a is the normal vector


hyperplanes are affine and convex; halfspaces are convex
Convex sets

26

Euclidean balls and ellipsoids


(Euclidean) ball with center xc and radius r:
B(xc, r) = {x | kx xck2 r} = {xc + ru | kuk2 1}
ellipsoid: set of the form
{x | (x xc)T P 1(x xc) 1}
with P Sn++ (i.e., P symmetric positive definite)

xc

other representation: {xc + Au | kuk2 1} with A square and nonsingular


Convex sets

27

Norm balls and norm cones


norm: a function k k that satisfies
kxk 0; kxk = 0 if and only if x = 0
ktxk = |t| kxk for t R
kx + yk kxk + kyk
notation: k k is general (unspecified) norm; k ksymb is particular norm
norm ball with center xc and radius r: {x | kx xck r}
1

0.5

norm cone: {(x, t) | kxk t}


Euclidean norm cone is called secondorder cone

0
1
1
0

x2 1 1

x1

norm balls and cones are convex


Convex sets

28

Polyhedra
solution set of finitely many linear inequalities and equalities
Ax  b,

Cx = d

(A Rmn, C Rpn,  is componentwise inequality)


a1

a2

P
a5
a3
a4

polyhedron is intersection of finite number of halfspaces and hyperplanes


Convex sets

29

Positive semidefinite cone


notation:
Sn is set of symmetric n n matrices
Sn+ = {X Sn | X  0}: positive semidefinite n n matrices
X Sn+

z T Xz 0 for all z

Sn+ is a convex cone


Sn++ = {X Sn | X 0}: positive definite n n matrices

x y
y z

S2+

0.5

example:

0
1
1
0

0.5

y
Convex sets

1 0

x
210

Operations that preserve convexity


practical methods for establishing convexity of a set C
1. apply definition
x1, x2 C,

01

x1 + (1 )x2 C

2. show that C is obtained from simple convex sets (hyperplanes,


halfspaces, norm balls, . . . ) by operations that preserve convexity

intersection
affine functions
perspective function
linear-fractional functions

Convex sets

211

Examples of convex sets


linear subspace: S = {x | Ax = 0} is a convex cone
affine subspace: S = {x | Ax = b} is a convex set
polyhedral set: S = {x | Ax  b} is a convex set
PSD matrix cone: S = {A | A is symmetric, A  0} is convex
second order cone: S = {(t, x) | t kxk } is convex

intersection

linear subspace
linear subspaces
affine subspace
affine subspaces
is also a
intersection of
convex cone
convex cones
convex set
convex sets
example: a polyhedron is intersection of a finite number of halfspaces
Convex sets

3. Convex functions

Definition
f : Rn R is convex if dom f is a convex set and
f (x + (1 )y) f (x) + (1 )f (y)
for all x, y dom f , 0 1
(y, f (y))
(x, f (x))

f is concave if f is convex
f is strictly convex if dom f is convex and
f (x + (1 )y) < f (x) + (1 )f (y)
for x, y dom f , x 6= y, 0 < < 1
Convex functions

32

Examples on R
convex:
affine: ax + b on R, for any a, b R
exponential: eax, for any a R
powers: x on R++, for 1 or 0
powers of absolute value: |x|p on R, for p 1
negative entropy: x log x on R++
concave:
affine: ax + b on R, for any a, b R
powers: x on R++, for 0 1
logarithm: log x on R++

Convex functions

33

Examples on Rn and Rmn


affine functions are convex and concave; all norms are convex
examples on Rn
affine function f (x) = aT x + b
Pn
norms: kxkp = ( i=1 |xi|p)1/p for p 1; kxk = maxk |xk |

examples on Rmn (m n matrices)


affine function

f (X) = tr(AT X) + b =

m X
n
X

Aij Xij + b

i=1 j=1

spectral (maximum singular value) norm


f (X) = kXk2 = max(X) = (max(X T X))1/2

Convex functions

34

First-order condition
f is differentiable if dom f is open and the gradient
f (x) =

f (x) f (x)
f (x)
,
,...,
x1
x2
xn

exists at each x dom f


1st-order condition: differentiable f with convex domain is convex iff
f (y) f (x) + f (x)T (y x)

for all x, y dom f

f (y)
f (x) + f (x)T (y x)
(x, f (x))

first-order approximation of f is global underestimator


Convex functions

37

Second-order conditions
f is twice differentiable if dom f is open and the Hessian 2f (x) Sn,
2f (x)
f (x)ij =
,
xixj
2

i, j = 1, . . . , n,

exists at each x dom f


2nd-order conditions: for twice differentiable f with convex domain
f is convex if and only if
2f (x)  0

for all x dom f

if 2f (x) 0 for all x dom f , then f is strictly convex

Convex functions

38

Examples
quadratic function: f (x) = (1/2)xT P x + q T x + r (with P Sn)
2f (x) = P

f (x) = P x + q,
convex if P  0

least-squares objective: f (x) = kAx bk22


f (x) = 2AT (Ax b),

2f (x) = 2AT A

convex (for any A)


quadratic-over-linear: f (x, y) = x2/y
y
x



y
x

T

0

f (x, y)

2
2f (x, y) = 3
y

2
1
0
2

2
0

convex for y > 0


Convex functions

0 2

x
39

Epigraph and sublevel set


-sublevel set of f : Rn R:
C = {x dom f | f (x) }
sublevel sets of convex functions are convex (converse is false)
epigraph of f : Rn R:
epi f = {(x, t) Rn+1 | x dom f, f (x) t}
epi f
f

f is convex if and only if epi f is a convex set


Convex functions

311

Properties of convex functions


convexity over all lines:

f (x) is convex

f (x0 + th) is convex in t for all x0 , h

positive multiple:

f (x) is convex

f (x) is convex for all 0

sum of convex functions:

f1 (x), f2 (x) convex

f1 (x) + f2 (x) is convex

pointwise maximum:

f1 (x), f2 (x) convex

max{f1 (x), f2 (x)} is convex

affine transformation of domain:

f (x) is convex
Convex functions

f (Ax + b) is convex

Operations that preserve convexity


practical methods for establishing convexity of a function
1. verify definition (often simplified by restricting to a line)
2. for twice differentiable functions, show 2f (x)  0
3. show that f is obtained from simple convex functions by operations
that preserve convexity

nonnegative weighted sum


composition with affine function
pointwise maximum and supremum
composition
minimization
perspective

Convex functions

313

Quasiconvex functions
f : Rn R is quasiconvex if dom f is convex and the sublevel sets
S = {x dom f | f (x) }
are convex for all

f is quasiconcave if f is quasiconvex
f is quasilinear if it is quasiconvex and quasiconcave
Convex functions

323