8

TABLE OF CONTENTS
CHAPTER NO.

TITLE

PAGE NO.

ABSTRACT
LIST OF TABLES
LIST OF FIGURES
LIST OF ABBREVIATIONS
1
INTRODUCTION
1.1 GENERAL
1.2 PATTERN CLASSIFICATION
1.3 CLASSIFIER ENSEMBLE
1.4 DATASET
1.5 FEATURE SELECTION
1.6 OPTIMIZATION AND METAHEURISTICS
1.7 PROBLEM IDENTIFICATION
1.8 OBJECTIVE OF THE RESEARCH
1.9 RESEARCH APPROACH
1.10
ORGANIZATION OF THE THESIS

iv
xiii
xv
xvi
1
1
3
3
4
5
6
7
7
8

2 REVIEW OF LITERATRUE
11
2.1 GENERAL
11
2.2 OVER PRODUCE AND CHOOSE STRATEGY
11
2.3 HETEROGENEOUS MODEL OF ENSEMBLE
12
2.4 FEATURE SELECTION USING ANT COLONY SYSTEM 13
2.4.1 Hybrid ACO for FS
2.4.2 FS for Classification using an Ant Colony System
2.5 ABC BASED FEATURE SELECTION
15
2.5.1 ABC Hybrid with Rough Set Theory for FS
2.6 OPTIMIZATION OF STACKING
16
2.6.1 GA-Stacking
2.6.2 ACO-Stacking
2.7 OPTIMIZING THE SELECTION OF ATTRIBUTES
OF AN ENSEMBLE
2.7.1 GA-Ensemble
2.7.2 Maximum Entropy Classifier Ensemble using GA
2.8 SUMMARY
21
3 THEORETICAL BACKGROUND
3.1 GENERAL
3.2 AN OVERVIEW OF CLASSIFIER ENSEMBLES
3.2.1 Classifier Ensembles
3.2.2 Types of Classifier Ensembles

14
14
15
17
19
20
20
21

23
23
23
23
24

3.2.3

Diversity

29

3.3 OVERVIEW OF FEATURE SELECTION

30
3.4 NATURE-INSPIRED METAHEURISTIC ALGORITHMS
AN OVERVIEW
3.4.1 Metaheuristics
3.4.2 Nature-Inspired Metaheuristic Algorithms
3.4.3 Ant Colony Optimization
3.4.4 Artificial Bee Colony
3.5
3.6
3.7
3.8

32
32
33
34
35

EXECUTION ENVIRONMENT
DATASETS USED IN THIS STUDY
PERFORMANCE EVALUATION
SUMMARY

4 DESIGN OF NEW ENSEMBLE MODELS

4.1
GENERAL
4.2
NEED AND MOTIVATION
INTEGRATION OF CLASSIFIER ENSEMBLES
4.3.1 Classifier Selection
4.3.2 Feature Selection
4.3.3 Diversity
4.3.4 Ensemble Formulation
4.3.5 CE-Integration (or) Ensemble of Ensembles
4.3.6 Experimental Results and Discussions
4.4
BOOSTING WITH MIXED LEARNER MODELS
4.4.1 Experimental Results and Discussion
4.5
SUMMARY
5 ACO BASED FEATURE SELECTION
5.1
GENERAL
5.2
ACO FOR FS
5.3
CE DESIGN USING ACO-FS
5.4
EFFECT OF ACO PARAMETERS ON FS AND

37
38
40
42

43
43
43

4.3

5.5

CLASSIFICATION
5.4.1 ACO Parameters
5.4.2 ACO Algorithm
5.4.3 Experiments and Discussions
SUMMARY

6 ARTIFICIAL BEE COLONY FOR OPTIMIZATION

OF FEATURE SELECTION
6.1
GENERAL
6.2
MOTIVATIONS TO USE ABC
6.3
FS OPTIMIZATION USING ABC

44
44
46
46
47
49
50
60
61
65
66
66
66
67
70
70
71
71
79
80
80
80
81

10

6.4

6.5

6.3.1 The ABC-FS Algorithm

6.3.2 Experiments and Discussions

82
85

HYBRID OF ACO AND ABC FOR FS

6.4.1 Highlights of the AC-ABC Hybrid Algorithm

88
93

6.4.2 Experiments
6.4.3 Discussion of Results
SUMMARY

93
95
104

7 OPTIMIZATION OF STACKING ENSEMBLE

CONFIGURATIONS THROUGH ABC ALGORITHM
7.1
GENERAL
7.2
ISSUES WITH STACKING
7.3
THE ABC-STACKING APPROACH
7.3.1 ABC-Stacking1 or Base-Level Stacking
7.3.2 ABC-Stacking2 or Meta-Level Stacking
7.4
EXPERIMENTAL SETUP
7.4.1 Learning Algorithms
7.4.2 Parameter Settings
7.4.3 Implementation of ABC-Stacking
7.5
RESULTS AND DISCUSSIONS
7.5.1 Performance Comparison with
Constituent Classifiers
7.5.2 Performance Comparison with Other Ensembles
7.5.3 Performance Comparison with Other
Optimization Based Stacking Methods
7.5.4 Comparison of ABC-Stacking1 and ABC-Stacking2
7.5.5 RMSE Measures
7.6

SUMMARY

122
125
128

130
130
130
130
133

SUMMARY

9 CONCLUSIONS AND FUTURE WORK

9.1
GENERAL
9.2
SALENT CONCLUSIONS
9.3
SCOPE FOR FUTURE WORK

118
120

129

8 DESIGN OF OPTIMAL CLASSIFIER

ENSEMBLE MODEL
8.1
GENERAL
8.2
NEED FOR OPTIMAL ENSEMBLE MODEL
8.3 ABC APPROACH FOR OPTIMAL DESIGN OF
CLASSIFIER ENSEMBLE (ABC-CE)
8.3.1 Experiments and Discussions
8.4

105
105
105
106
106
109
110
110
111
111
113

138
139
139
139
141

11

REFERENCES
LIST OF PUBLICATIONS
VITAE
ANNEXURE I

142
152
154
155

LIST OF TABLES

TABLE NO.

TITLE

PAGE NO.

3.1

Datasets Description

39

3.2
4.1

Confusion Matrix
Results of Class Predictions for Six UCI Datasets
by Classification Algorithms

40

4.2

Diversity Statistics: Pair wise Disagreement Measure between

Classifiers for Six UCI Datasets

4.3

52

Diversity Statistics: Feature Selection Induced Pair wise

Disagreement Measure between Classifiers for Six UCI Datasets

4.4

51

Comparison of Recognition Rates of Different Ensembles

Based on Posterior Probability Estimates with and without

53

12

Feature Selection

55

4.5

Comparison of Recognition Rates of Different Ensembles

Based on Advanced Combination with and without
Feature Selection
4.6Recognition Rates of Integration of Classifier Ensembles
MV, NB, BKS and DT
4.7
Classification Results obtained through BMLM
5.1
Results of ACO Based FS and Classification Accuracy
for Heart-C Dataset
5.2
Results of ACO Based FS and Classification Accuracy
For Hepatitis Dataset
5.3
Results of ACO Based FS and Classification Accuracy
For Lung Cancer Dataset
5.4
Results of ACO Based FS and Classification Accuracy
For Dermatology Dataset
5.5
Results of ACO Based FS and Classification Accuracy
for Pima Dataset
5.6
Results of ACO Based FS and Classification Accuracy
for Lymphography Dataset

TABLE NO.
5.7
5.8
5.9
5.10

TITLE

56
57
63
72
73
73
74
74
74

PAGE NO.

Results of ACO Based FS and Classification Accuracy

forLiver Dataset
Results of ACO Based FS and Classification Accuracy
forWisconsin Dataset
Results of ACO Based FS and Classification Accuracy
for Heart-Stalog Dataset
Results of ACO Based FS and Classification Accuracy
for Diabetes Dataset

6.1

Results of FS and Classification by ABC-FS

6.2

Comparison of Features Selected by ACO and ABC

6.3

Comparison of Prediction Accuracies of 10 Datasets through

10-Fold Cross Validation
Parameter Settings of AC-ABC Hybrid Algorithm
Results of AC-ABC Hybrid for 13 Different UCI Datasets
Performance Comparison of the AC-ABC Hybrid with
other Meta-Heuristic Approaches for FSand Classification
in the Literature
Comparison of Execution Time of AC-ABC Hybrid with
ACO based FS

6.4
6.5
6.6
6.7

75
75
75
76
86
87
88
94
96
99
102

6.8

Measure of Power Level of Classification (F-Test MANOVA)

103

7.1

ABC Parameters

111

7.2

Base-Classifier Configuration of ABC-Stacking1

13

7.3

7.4
7.5

TABLE NO.
7.6

(with J48 as Meta-Classifier)

Base-Classifier Configuration and Selected Meta-Classifier
of ABC-Stacking2
Comparison of Predictive Accuracies of ABC-Stacking with
Base-Classifiers
Comparison of Predictive Accuracies of ABC-Stacking with
Other Ensembles

TITLE

115
117

119
121

PAGE NO.

7.7

Classification Accuracies of the Optimization based

Stacking methods
RMSE Measures for ABC-Stacking

123
128

8.1
8.2

Performance of ABC-CE1
Performance of ABC-CE2

134
135

LIST OF FIGURES

14

FIGURE NO.
3.1
3.2
3.3
4.1
4.2
4.3
4.4
5.1

5.2

TITLE

PAGE NO.

Architecture of Classifier Ensemble

Feature Selection Process
Steps of the ABC algorithm
Architecture of Classifier Ensembles Integration
Prediction Results from Feature Selected Ensembles and
Integration Methods
Boosting with Mixed Learner Models
Performance Comparison of BMLM Ensembles with normal
Boosting
Performance Comparisons for the Four UCI Datasets
by the Constituent Classifiers, Traditional Ensembles and
Ensembles Constructed using ACO-FS
Effect of Pheromone Evaporation Rate on
Classification Accuracy

6.1
6.2
6.3
7.1

ABC Based Optimization of FS

The ABC-FSAlgorithm
Main Steps in the AC-ABC Hybrid Algorithm
Schematic diagrams of (a) Stacking, (b) ABC-Stacking1 and
(c) ABC-Stacking2
7.2
The ABC-Stacking1 algorithm
7.3
The ABC-Stacking2 Algorithm
7.4
Optimal Selection of Base-Level Classifiers by ABC-Stacking1
7.5 (a) Evaluation of ABC-Stacking1 by means of
Accuracy vs Iterations
7.5 (b) Evaluation of ABC-Stacking2 by means of
Accuracy vs Iterations
8.1
Schematic Diagram of Optimal CE Model using ABC
8.2
Performance of optimal CE model on original feature set
and on reduced feature subset

24
31
37
49
59
61
64
69

79
82
83
90
107
109
110
112
127
127
132

137

LIST OF ABBREVIATIONS
ABC

Artificial Bee Colony

ABC-CE

Artificial Bee Colony Approach for Optimal Design of

Classifier
Ensemble

ABC-FS

Artificial Bee Colony based Feature Selection

AC-ABC

Ant Colony and Artificial Bee Colony

ACO

Ant Colony Optimization

ACO-FS

Ant Colony Optimization for Feature Selection

15

AVG

Average

BCO

Bee Colony Optimization

BKS

Behavioral Knowledge Space

BMLM

Boosting with Mixed Learner Models

CE

Classifier Ensemble

CV

Cross-Validation

DT

Decision Template

FS

Feature Selection

GA

Genetic Algorithm

HACO

Hybrid ACO

KNN

K- Nearest Neighbor

LPU

Local Pheromone Updating factor

LR

Logistic Regression

MAX

Maximum

MCS

Multiple Classifier Systems

MIN

Minimum

MV

Majority Voting

NB

Nave Bayes

NIMA

Nature Inspired Metaheuristic Algorithms

NN

Neural Networks

PA

Predictive Accuracy

PER

Pheromone Evaporation Rate

PRO

Product

PSO

Particle Swarm Optimization

RMSE

Root Mean Square Error

RST

Rough Set Theory

SD

Standard Deviation

16

SF

Selected Features

SVM

Support Vector Machine

UCI

University of California, Irvine

WEKA
WER

Waikato Environment for Knowledge Analysis

Werneckes