N ORMALIZED C UT AND I MAGE S EGMENTATION

Jianbo Shi and Jitendra Malik

Presented by Iftekhar Naim

1 / 34

Normalized Cut

O UTLINE
1

I NTRODUCTION

AND OVERVIEW Overview of the Paper Problem Formulation USING

G RAPH PARTITIONING

N ORMALIZED C UT

Graph Representation Minimum Cut for Partitioning Normalized Cut

R ESULTS
Segmentation Results Comparison with other Graph-partition Methods

C ONCLUSION

2 / 34

Normalized Cut

O UTLINE
1

I NTRODUCTION

AND OVERVIEW Overview of the Paper Problem Formulation

2 Graph Partitioning using Normalized Cut Graph Representation Minimum Cut for Partitioning Normalized Cut 3 Results Segmentation Results Comparison with other Graph-partition Methods 4 Conclusion

3 / 34

Normalized Cut

OVERVIEW OF THE PAPER

Introduces a general framework for perceptual grouping (Shi and Malik [2000])

Specic focus on image segmentation

4 / 34

Normalized Cut

OVERVIEW OF THE PAPER

Introduces a general framework for perceptual grouping (Shi and Malik [2000])

Specic focus on image segmentation

An image can be partitioned in different ways

Which one is the right one? May not be a single correct answer

4 / 34

Normalized Cut

OVERVIEW OF THE PAPER

Introduces a general framework for perceptual grouping (Shi and Malik [2000])

Specic focus on image segmentation

An image can be partitioned in different ways

Which one is the right one? May not be a single correct answer

A Bayesian view might be appropriate

However, difcult to integrate low and high level prior knowledge

4 / 34

Normalized Cut

OVERVIEW OF THE PAPER

Introduces a general framework for perceptual grouping (Shi and Malik [2000])

Specic focus on image segmentation

An image can be partitioned in different ways

Which one is the right one? May not be a single correct answer

A Bayesian view might be appropriate

However, difcult to integrate low and high level prior knowledge

Partitioning task is inherently hierarchical

Hierarchical partitions are more appropriate

4 / 34

Normalized Cut

P ROBLEM F ORMULATION

Image Segmentation: Two basic questions:

What is the criterion that should be optimized? Is there an efcient algorithm to optimize the criterion?

5 / 34

Normalized Cut

P ROBLEM F ORMULATION

Image Segmentation: Two basic questions:

What is the criterion that should be optimized? Is there an efcient algorithm to optimize the criterion?

This paper proposes:

A graph theoretic criterion for image segmentation An efcient algorithm to optimize the criterion

5 / 34

Normalized Cut

P ROBLEM F ORMULATION

Image Segmentation: Two basic questions:

What is the criterion that should be optimized? Is there an efcient algorithm to optimize the criterion?

This paper proposes:

A graph theoretic criterion for image segmentation An efcient algorithm to optimize the criterion

Weighted undirected graph: G = (V, E)

V = set of vertices, each vertex represents a pixel vi . E = set of edges between a pair of vertices.

5 / 34

Normalized Cut

P ROBLEM F ORMULATION

Image Segmentation: Two basic questions:

What is the criterion that should be optimized? Is there an efcient algorithm to optimize the criterion?

This paper proposes:

A graph theoretic criterion for image segmentation An efcient algorithm to optimize the criterion

Weighted undirected graph: G = (V, E)

V = set of vertices, each vertex represents a pixel vi . E = set of edges between a pair of vertices.

Partition V into disjoint subsets V1 , V2 , . . . , Vm such that:

High similarity within any group Vi Low similarity across different groups Vi and Vj
5 / 34 Normalized Cut

O UTLINE
1 Introduction and Overview Overview of the Paper Problem Formulation 2

G RAPH PARTITIONING

USING

N ORMALIZED C UT

Graph Representation Minimum Cut for Partitioning Normalized Cut

3 Results Segmentation Results Comparison with other Graph-partition Methods 4 Conclusion

6 / 34

Normalized Cut

G RAPH R EPRESENTATION

Partition G = (V, E) into two disjoint sets A and B AB = V AB = 0 /

vj

wij vi

w (i , j ) = Similarity between vi and vj

7 / 34

Normalized Cut

M INIMUM C UT PARTITION

Graph cut:

cut (A, B ) =

u A,v B

w (u , v )

8 / 34

Normalized Cut

M INIMUM C UT PARTITION

Graph cut:

cut (A, B ) =

u A,v B

w (u , v )

Minimum Cut Partition: Find (A, B ) that minimizes cut (A, B )

Efcient Algorithm Exists

8 / 34

Normalized Cut

M INIMUM C UT PARTITION

Graph cut:

cut (A, B ) =

u A,v B

w (u , v )

Minimum Cut Partition: Find (A, B ) that minimizes cut (A, B )

Efcient Algorithm Exists

Problem: Small sets with isolated nodes (Wu and Leahy [1993])

cut (A, B ) increases with the number of edges across A and B

8 / 34

Normalized Cut

M INIMUM C UT PARTITION

F IGURE : Mincut generates isolated small groups (Shi and Malik [2000])

9 / 34

Normalized Cut

N ORMALIZED C UT

Normalized Cut: Ncut (A, B ) as a measure of disassociation: Ncut (A, B ) = where assoc (A, V ) = cut (A, B ) assoc (A, V )

cut (A, B ) assoc (B , V )

u A,t V

w (u , t )

10 / 34

Normalized Cut

N ORMALIZED C UT

Normalized Cut: Ncut (A, B ) as a measure of disassociation: Ncut (A, B ) = where assoc (A, V ) = cut (A, B ) assoc (A, V )

cut (A, B ) assoc (B , V )

u A,t V

w (u , t )

Measure of association: Nassoc (A, B ) = assoc (A, A) assoc (A, V )

assoc (B , B ) assoc (B , V )

10 / 34

Normalized Cut

N ORMALIZED C UT

Normalized Cut: Ncut (A, B ) as a measure of disassociation: Ncut (A, B ) = where assoc (A, V ) = cut (A, B ) assoc (A, V )

cut (A, B ) assoc (B , V )

u A,t V

w (u , t )

Measure of association: Nassoc (A, B ) = assoc (A, A) assoc (A, V )

assoc (B , B ) assoc (B , V )

It can be shown: Ncut (A, B ) = 2 Nassoc (A, B )

10 / 34 Normalized Cut

N ORMALIZED C UT

Minimum Normalized Cut simultaneously:

Minimizes disassociation between groups Maximizes association within groups

Let x be a |V | dimensional indicator vector xi = 1 if xi A if xi B

Find partition that minimizes Ncut min Ncut (x)

x

Exact minimization of Ncut is NP-Complete (Papadimitrou [1997]) Approximate solution by relaxation

11 / 34

Normalized Cut

N ORMALIZED C UT
Ncut (A, B ) can be written as: cut (A, B ) cut (A, B )

Ncut (A, B )

= =

assoc (A, V ) assoc (B , V ) xi >0,xj <0 wij xi xj xi >0,xj <0 wij xi xj + xi >0 di xi <0 di

where d (i ) = j w (i , j )

Optimization Problem:

x = min Ncut (x)

x

(1)

Subject to xi {1, 1}

12 / 34

Normalized Cut

R ELAXATION OF N ORMALIZED C UT

The objective function (1) can be written as: min Ncut (x) = min
x y

yT (D W )y yT Dy

(2)

Subject to yi {1, b} and y T D = 0

Still an NP-Complete problem Relax by yi After relaxation, (2) becomes the well-known Rayleigh quotient

13 / 34

Normalized Cut

R ELAXATION OF N ORMALIZED C UT

Rayleigh quotient can be minimized by solving the generalized eigenvalue system: (D W )y = Dy (3) We can also rewrite as standard eigenvalue problem: D 2 (D W )D 2 z = z where z = D 2 y
1 1 1

14 / 34

Normalized Cut

R ELAXATION OF N ORMALIZED C UT

Rayleigh quotient can be minimized by solving the generalized eigenvalue system: (D W )y = Dy (3) We can also rewrite as standard eigenvalue problem: D 2 (D W )D 2 z = z where z = D 2 y
1 1 1

Solution: The second smallest eigenvector of the normalized Laplacian matrix Lsym = D 2 (D W )D 2
1 1

(4)

14 / 34

Normalized Cut

G ROUPING A LGORITHM

Algorithm (Shi and Malik [2000])

1 2

Given the image, construct a weighted graph G = (V , E ). Solve the eigenproblem:

(D W )x = Dx
3

Use the eigenvector with the second smallest eigenvalue to bipartition the graph Decide if the current partition should be subdivided. If necessary, recursively repartition the segmented parts.

15 / 34

Normalized Cut

E XAMPLE : I MAGE S EGMENTATION

F IGURE : Segmentation of a gray scale image of a baseball game (Shi and Malik [2000])

16 / 34

Normalized Cut

1. C ONSTRUCTION OF W EIGHTED G RAPH

Construct a weighted graph G = (V , E )

Every pixel is a vertex An edge between every pairs of pixels

Dene Edge weights wij to be the similarity between nodes i and j

wij = e

F(i ) F (j ) 2 I2

e 0

X(i ) X (j ) 2 2 X

if X(i ) X(j ) otherwise

<r

F(i ) = Feature vector of i-th pixel (brightness, texture, motion etc.) X(i ) = Spatial location of i-th pixel

17 / 34

Normalized Cut

2. S OLVE GENERALIZED EIGEN - SYSTEM

Solve for eigenvectors with smallest eigenvalues

(D W )y = Dy
Transform to standard eigenvalue problem: D 2 (D W )D 2 z = z

1 1

(5)

Complexity: O (n3 ) For sparse graphs, signicant speedup by Lanczos method (O (mn))

n = number of pixels m = number of iterations for Lanczos method

18 / 34

Normalized Cut

3. G RAPH PARTITIONING USING E IGENVECTORS

Eigenvectors can take continuous values (due to relaxation) Find a splitting point that bipartitions the Graph Two approaches are suggested:

Use a threshold value of 0 or median value Search for the splitting point that minimizes Ncut (A, B )

The authors preferred the second approach

19 / 34

Normalized Cut

4. R ECURSIVE PARTITIONING

Use next eigenvectors to recursively repartition each of the parts Decide whether to repartition:

Stop recursion if the Ncut value exceeds a threshold Ignore eigenvectors that are unstable (smoothly varying)

20 / 34

Normalized Cut

E XAMPLE : I MAGE S EGMENTATION

F IGURE : Segmentation of a gray scale image of a baseball game (Shi and Malik [2000])

21 / 34

Normalized Cut

E XAMPLE : I MAGE S EGMENTATION

F IGURE : Eigenvectors (Shi and Malik [2000])

22 / 34 Normalized Cut

S EGMENTATION R ESULT

F IGURE : Resulting partitions with Ncut < 0.04. Parameter setting:I = 0.1, X = 4.0, r = 5 (Shi and Malik [2000])

23 / 34

Normalized Cut

O UTLINE
1 Introduction and Overview Overview of the Paper Problem Formulation 2 Graph Partitioning using Normalized Cut Graph Representation Minimum Cut for Partitioning Normalized Cut 3

R ESULTS
Segmentation Results Comparison with other Graph-partition Methods

4 Conclusion

24 / 34

Normalized Cut

S EGMENTATION USING B RIGHTNESS F EATURES

Ref: Shi and Malik [2000])

25 / 34 Normalized Cut

S EGMENTATION USING T EXTURE F EATURES

Ref: Shi and Malik [2000])

26 / 34

Normalized Cut

C OMPARISON WITH OTHER METHODS

F IGURE : Other graph partition methods (Shi and Malik [2000])

27 / 34

Normalized Cut

S YNTHETIC 1D DATA (S HI AND M ALIK [2000]

28 / 34

Normalized Cut

C OMPARISON WITH OTHER METHODS

F IGURE : Average association failed (Shi and Malik [2000])

29 / 34 Normalized Cut

C OMPARISON WITH OTHER METHODS

F IGURE : Average cut failed (Shi and Malik [2000])

30 / 34 Normalized Cut

C OMPARISON WITH OTHER METHODS

F IGURE : Comparable results for all 3 methods (Shi and Malik [2000])
31 / 34 Normalized Cut

O UTLINE
1 Introduction and Overview Overview of the Paper Problem Formulation 2 Graph Partitioning using Normalized Cut Graph Representation Minimum Cut for Partitioning Normalized Cut 3 Results Segmentation Results Comparison with other Graph-partition Methods 4

C ONCLUSION

32 / 34

Normalized Cut

C ONCLUSION

Perceptual grouping should:

Extract global big picture of a scene Provide hierarchical description of a scene

Normalized Cut does satisfy the two requirements The paper proposes an algorithm for approximate solution Can be expensive for large graphs (O (N 3 )) There are ways to improve scalability (speed/memory)

Lanczos method for sparse graph Power Iteration Clustering (Lin and Cohen [2010]) KASP (Yan et al. [2009])

33 / 34

Normalized Cut

R EFERENCES

F. Lin and W.W. Cohen. Power iteration clustering. ICML, 2010. J. Shi and J. Malik. Normalized cuts and image segmentation. IEEE Transactions on pattern analysis and machine intelligence, 22(8):888905, 2000. Z. Wu and R. Leahy. An optimal graph theoretic approach to data clustering: Theory and its application to image segmentation. IEEE transactions on pattern analysis and machine intelligence, pages 11011113, 1993. ISSN 0162-8828. D. Yan, L. Huang, and M.I. Jordan. Fast approximate spectral clustering. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 907916. ACM, 2009.

34 / 34

Normalized Cut