Fourth Annual Progress Seminar

Tri-Stereo Image Analysis

Raghavendra Bhalerao
10431002

PhD Supervisor
Prof. Shirish S. Gedam

Centre of Studies in Resources Engineering

IIT Bombay
1 December 2014
3
 4

Stereo Image Matching

Global
Limitation of optimization
Local
Feature Based ( Limited due to Sparse matching )
Area Based
 5

Issues in Stereo Matching

Noise
Occlusion
Repetitive Patterns
Moving Objects
Poorly Textured Regions
Disappearing Objects
Disappearing Shadows
Distinct object discontinues
Radiometric artefacts as specular reflection
 6

Constraints for stereo Matching

Ordering Constraint ( street light )
Uniqueness Constraint (boundary)
Similarity Constraint
Continuity Constraint
Epipolar Constraint
Disparity Range
Fronto Planar
Multi Images Matching
Multi Resolution Analysis
 7

Research Gaps in ABM

LSM based techniques are used for satellite
images matching but it requires a good
initialization
Assumption of similar disparity inside the
window
Using WTA method the global minimum is the
solution but due to a very large search range we
may end up in error
 8

Motivation
The ill-posed stereo matching problem, that is
limitation of WTA to get a correct match at
global maximum
Limited use of epipolar based using DSI
Need of a method to integrate the redundant
information from triplet of image to get depth
information
 9

Research Objectives
Development of technique for estimating
epipolar geometric transformation parameters of
all the three images using Scale Invariant
Feature Transform (SIFT) based approach.
Development of the algorithm to generate 3D
surface with maximum possible confidence
using modified NCC and SSD based approach for
matching.
 10

Research Objectives cont

Development of the algorithm using triplets to
define a measure to use Global minimum and
Local minima of Disparity Space Image to get
correct disparity.
Development of technique to densify the blank
areas.(densification of the surface with
additional matches which were filtered due to
low confidence measure )
 11

Proposed
Work Flow

Work Flow 1. Work Flow for

Tri Stereo Image Analysis
 12

Proposed Work Flow

Step 1 Epipolar Resampling
Step 2 Dense Matching
Step 3 Removal of Incorrect Matches
Step 4 Refinement
Step 5 Accuracy Assessment
 13

Tri Stereo Sensors

Table 1. Tri Stereo Satellite Platforms

 14

Dataset Used
Table 2. Details of TMC Dataset

Table 3. Details of LOLA Dataset

 15

TMC Triplets

Fig 2. TMC dataset (Source- ISSDC)

 16

LOLA DATASET

Fig 3. LOLA dataset (Source- (http://ode.rsl.wustl.edu/moon/)

 17

Current work
Epipolar Resampling
Dense Matching
Removing Incorrect Matches
Refinement for hole filling
 18

Quasi Epipolar Resampling

Need of Epipolar resampling
Converts 2D pixel search to 1D search
Epipolar Geometry
Sensor Geometry and TMC conguration
 19

Epipolar Gemetry
epipole el Ol Or
Ol Or

epipole

image Plane il image Plane ir er epipolar line elr

epipolar line ell

pr epipolar line elr pl pr

pl
epipolar line
ell Epipolar Geometry
epipolar
plane e
P P

(A) (B)

Ol Or
epipolar image Plane ir
epipolar line elplane
l e
image epipolar line
pl pr
Plane il elr

P (C) Fig4. Epipolar Geometry

 20

TMC configuration
Aft and Fore Angles

F
N
A
25
25

100 Km
Sensor
Height

Aft
Nadir
Fore
(B)
(A)

Fig. 5 TMC Configuration

26.54374

100 Km
111 Km
11
1.7
7
Km

Swath - 20 Km
100 Km

5m Pixel Nadir

5m

5.5m 5.5165m Pixel Fore / Aft Swath - 22.066 Km

5m

6.1 m
(B)
(A) Fig.6 Details of footprint
 21

Quasi Epipolar Resampling

Extract SIFT Features
Match features using BBF algorithm
Plot graphs
plot between y location of extracted features of I 1
and I2
plot between y parallax and extracted features
Determine the compression ratio
 22

Initial Matching

Aft Nadir Fore

Fig.6. Goldschmidt B Crater 70.6 N 6.7 W

Diameter 10 Km (= 2000 pixels )
 23

Various Plots

Fig.7. Plot of extracted SIFT Features of AN NF view

 24

Matches
after
resampling

Fig.8 Matching points of

AN NF pair
 25

Anaglyphic view

Fig. 9 Anaglyph view for AN NF view

 26

Observations
Table 4 Observations for the scaling for resampling different dataset

0.8864* 4000 =3545 0.8964* 4000 =3585

0.8877* 4000= 3550 0.8956*4000=3582
 27

Possible reason for variation

The basic camera geometry is defined for flat
terrain but in real the surface is not flat hence a
small variation in along single strip and larger
variation across different strips exists.
 28

Triplet matching Methodology

Rectified triplet Images are matched using SSD
A cost function is defined for matching
A decision rules is defined for discarding
incorrect matches
Discarded points are refined for correct matches
 29

Cost and Disparity

 30

Aft Image (left) Epipolar resampled Nadir Image (right) Epipolar resampled

A'1

cost
Aft Zoomed

B1 Search range

Nadir Zoomed
31
 32

Disparity Space Image

 33

Disparity Space Image

 34

Left Right Consistency

 35

Left Right Consistency

 36

LRC in DSI
 37

LC CR Consistency
 38

Local Minima Consideration

 39

Work Flow -1
Image Il Image Ic Image Ir

Cl Cr

DSIl DSIr

D cl D rl D cr Drr M cl Mrl M cr Mrr

N N Dcr- Drr< T1 N Mcl- Mrl < T2 N Mcr- Mrr< T2

Dcl- Drl < T1

Y Y Y Y
No value CPl No value CPr No value PMl No value PMr

^
CP ^
PM
r r

^
CP = CPl CP ^
PM = PMl PM
r r
 40

Work Flow -2a START

j=0

N
CP(j) is present Select all PM in row j

Y Select CP 3 from left and 3

from right

Fir curve using 6 points

Calculate distance between

curve and PMjk as ak

N
ak<T3 Df(j)=No value

Df((j)=CP(j) Df(j)=PMj,k
Y

j=j+1

j< n

Exit
 41

Steps for threshold determination

T1 - Initially the value of T1 is set to 0 and using this CPl and CPr are

determined from equation 4.8 and 4.9. Initially the values of k and t are

determined empirically, and later chosen for which the 4m is lowest. Now

the value of T1 is incremented and selected iff 4m does not increase, else its

value is used as 0. The value of 4m lies in range of 0-10 pixels.

T2 and T3 - The error of 4m will always be present when disparity is

determined from three views (observations from two DSI images) and

hence it is set to the the permissible error. The same value is set to the

thresholds for T2 and T3 .

 42

Work Flow -2b

A weightage of is assigned to a disparity if it is

local minima(or global minimum) in row /
column in left/right DSI
Disparity Continuity in any of 8 directions a
weightage of 1/8 is assigned
Disparity with total weight 2 is best disparity
 43

Work Flow cont

Iteration 1 Iteration 2

Disparity 1 Disparity 2 Disparity 1 Disparity 2

Ratio1 4/5(0.80) 5/9 1/5(0.20) 4/9

Ratio2 5/6 (0.83) 5/9 1/6(0.16) 4/9

Ratio1 Total number of disparity with score 2 in a particular sub - surface

Total number of disparity with score 2

Ratio 2 Total number of connected disparity in a particular sub - surface

Total number of connected disparity
 44

Results
 45

Publications
Presented papers in International Conference
1. Raghavendra H. Bhalerao, Shirish S. Gedam, and Jyoti Joglekar,2014,"Scan line optimization for tri-stereo image matching ",
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XL-3, 2014
ISPRS Technical Commission III Symposium, 5 7 September 2014, Zurich, Switzerland.
2. Raghavendra H. Bhalerao, Shirish S.Gedam and Andrs Almansa, 2013,Fast Epipolar Resampling of Trinocular Linear
Scanners Images Using Chandrayaan-1 TMC Dataset , 2013 IEEE Second International Conference on Image Information
Processing, Shimla India, 11-13 Dec 2013, pp 1-5.
3. Raghavendra H. Bhalerao, Shirish S.Gedam and Andrs Almansa, 2013,Lunar Impact Crater Modeling Using Trinocular
Stereoscopic Depth Inpainting, 2013, IEEE Second International Conference on Image Information Processing, Shimla India,
11-13 Dec 2013, pp 12-17. (BEST PAPER)
4. Raghavendra H. Bhalerao, Shirish S.Gedam and Jyoti Joglekar, 2012 Fast method for disparity map generation using Surface
Geometry in Remote Sensing and Photogrammetry Societys Annual Conference, London, (12-14 Sept 2012)
5. Raghavendra H. Bhalerao and Shirish S.Gedam 2012 Occlusion and Break line Detection using Trinocular Stereo Matching
Geomatrix12 3rd International Conference on Geospatial Techniques and Applications, I ITB Mumbai, CD Proceedings.
6. Raghavendra H. Bhalerao and Shirish S.Gedam, 2011 Disparity computation of Corner points by enhancing Feature Extractors
for Sparse Narrow-Baseline Stereo matching proceedings of IGARSS 2011, Vancouver Canada, 24-29 July 2011, pp 562-566.
Manuscript under review in Journals
7. Raghavendra H. Bhalerao, Shirish S.Gedam and Andrs Almansa Resampling linear scanners trinocualr images for stereo
matching Submitted to Defence Science Journal on 28 Oct 2014).(Previously submitted to Journal of Spatial Science and they
asked to resubmit with ma jor revision by Dec 18, 2014)
8. Raghavendra H. Bhalerao, and Shirish S.Gedam Modified Dual Winner Takes All approach for Tri stereo matching using
Disparity Space Images (Submitted to The Photogrammetric Record 14th Nov 2014 )
Presented paper in National Conference

9. Raghavendra H. Bhalerao and Shirish S.Gedam 2011 Classification of Stable SIFT Features using Parallax for Sparse-Stereo
Image Correspondence Geomatrix11 2nd national Conference on Geospatial Techniques and Applications, IITB Mumbai,
CD Proceedings.
10. Raghavendra H. Bhalerao and Shirish S.Gedam 2011 DEM generation using tri stereo planetary images Geomatrix14 3rd
 46

Reviewers Comments
Associate Editor's Comments:
This paper concerns an interesting and important technical problem. The paper is much improved
compared to an earlier one by the authors. However the reviewers are divided on its contribution. One
believes that it lacks sufficient novelty. In addition the paper still suffers from a number of grammatical
errors in the use of English - these are not major, but there are many of them. Some of the technical writing
also needs to be more precise, such as the statement in section V that the "method gives considerable good
results". Overall, although the paper has some technical merit, I believe it is still not suitable for publication.
Reviewer: 1
Comments to the Author
You have presented a tri-stereo matching method using some of the less explored properties of the Disparity
Space Image. Thanks for making a significant contribution to the published knowledge in this field.
 47

THANKS