Binary Classification Performance Measures/Metrics:
A Comprehensive Visualized Roadmap to Gain New Insights
Grol Canbek
METU and HAVELSAN, Ankara, Turkey
gcanbek@havelsan.com.tr
Tugba Taskaya Temizel
METU, Ankara, Turkey
ttemizel@metu.edu.tr
AbstractBinary classification is one of the most frequent
studies in applied machine learning problems in various domains,
from medicine to biology to meteorology to malware analysis.
Many researchers use some performance metrics in their
classification studies to report their success. However, the
literature has shown a widespread confusion about the
terminology and ignorance of the fundamental aspects behind
metrics. This paper clarifies the confusing terminology, suggests
formal rules to distinguish between measures and metrics for the
first time, and proposes a new comprehensive visualized roadmap
in a leveled structure for 22 measures and 22 metrics for exploring
binary classification performance. Additionally, we introduced
novel concepts such as canonical notation, duality, and
complementation for measures/metrics, and suggested two new
canonical base measures simplifying equations. It is expected that
the study will guide other studies to have standardized approach
to performance metrics for machine learning based solutions.
Keywordsbinary classification; classification performance;
metrics; measures; machine learning; visualization; ontology
I.
INTRODUCTION
Machine learning classification performance that is an
important subject in several domains is related to state how well
a classifier that implements a specific machine learning
algorithm or model makes a correct distinction between classes.
The most basic and studied classification type is binary
classification or two-class classification that separates a given
input into two opposite classes such as 'presence' vs. 'absence' of
a disease or a condition, respond vs. no respond for a
treatment [1], 'spam' vs. 'non-spam' for an e-mail, and 'malign'
vs. 'benign' for software.
Stating or comparing a classification performance with only
4 base measures is not suitable and understandable. Therefore,
several metrics have been proposed for evaluating classification
performances. Area Under (ROC) Curve (AUC) has its origins
in signal detection theory in the 1970s is considered as a best
metric to state the performance [2], but there are other combined
metrics that are useful for indicating the successful and
978-1-5386-0930-9/17/$31.00 2017 IEEE
Seref Sagiroglu
Gazi University, Ankara, Turkey
ss@gazi.edu.tr
Nazife Baykal
METU, Ankara, Turkey
baykal@metu.edu.tr
unsuccessful aspects of a classification model. Some represent
performance from the specific point of view while ignoring the
others. Many researchers who design a classification model give
narrow metrics causing misperceptions.
In this study, our approach to performance metrics is from
holistic perspective covering the wide range of the subject. This
is important for some emerging domains such as malware
classification or other new machine learning classification
applications that focus on implementation details and acquainted
with only a few misleading metrics such as Accuracy (ACC),
True Positive Rate (TPR) or F-measures to claim their success.
The researchers who want to improve their machine learning
algorithms on different domain problems and compare their test
results with others have difficulties to understand performance
metrics and select the most proper ones from the wide set of
possibilities. For this reason, an originally developed visually
enhanced performance metrics roadmap is designed as a chart
based on the confusion matrix to help these researchers.
The proposed comprehensive roadmap shows the complete
set of primary metrics not only the common ones such as ACC,
TPR, True Negative Rate (TNR), False Positive Rate, (FPR),
Positive Predictive Value (PPV), False Negative Rate (FNR), F1
score but also the others such as Prevalence, (Label) Bias,
INFORM (informedness), MARK (markedness), MCC
(Matthews Correlation Coefficient), BACC (balanced accuracy,
also known as strength), Gm (G-mean), Cohens Kappa (CK),
and Matthews Correlation Coefficient (MCC).
The roadmap is domain independent and useful in all the data
mining, machine learning and statistics studies. We aim that this
study is also a reference study for covering all the primary
measures and metrics with their equations specifically arranged
in binary classification context. We also reviewed metrics'
naming used interchangeably in academic and online resources
and included here in order to suit different naming conventions.
The corresponding terminology in other domains such as
meteorology, medicine, or statistics is provided to see the
synonyms of the measures and metrics.

8%0.QG,QWHUQDWLRQDO&RQIHUHQFHRQ&RPSXWHU6FLHQFHDQG(QJLQHHULQJ 

 The aim of our study is to review, summarize and clarify the
large number of binary classification primary performance
measures and metrics from a structural point of view so that the
individual measures/metrics and their dependencies and
relations become visible and could be understood easily in a
natural way. The study makes the following contributions:
Reviewed and clarified performance measure-metrics
terminology,
Put the difference between measures and metrics with
other confused terms such as scores, indicators, criteria,
factors or indices,
Introduced canonical forms for performance equations,
Proposed formal rules to decide whether an equation is
a measure or metric,
Provided the systematic approach on which the
performance measures and metrics are defined with a
leveled structure showing dependency and relations,
Discovered duality and complementation in measures
and metrics and presented notations for expressing
duality and complementation,
Suggested diagonal and off-diagonal totals as new
additional measures: True Classification (TC) and False
Classification (FC),
Provided equations to calculate measures and metrics in
binary classification notation as a reference study,
Transformed the related equations into a new simplified
canonical form for easy interpretation, and
Designed a roadmap visualizing 22 measures and 22
metrics in all-in-one style so that the designated new
geometry and leveling approach provide a better
understanding of them with relations and dependencies.
The rest of the paper is organized as follows. Section II
surveys literature on performance metrics. This section is short
due to the lack of space nevertheless it provides many resources
examining different aspects of performance metrics. Section III
provides the clear distinction among confused terminologies
metrics, measures, and indicators. Section IV presents and
describes our comprehensive visualized roadmap for binary
classification performance metrics and describes the base and 1st
level measures. It also describes the new two measures proposed
by us. Section V introduces our new canonical form approach in
measures and metrics equations and our new suggestion of
certain formal rules to define a given equation as a measure or
metric. Section VI reintroduces duality and complementation in
measures and metrics with proposed notation to state them.
Section VII describes the 2nd level measures, Section VIII
describes base and conventional 1st level metrics with
column/row geometry and examines Accuracy as the most used
performance metric in studies and other base metrics. It
addressed the complicated usage of Accuracy metric. Section IX
briefs the 1st and 2nd level metrics that are rarely used in binary
classification studies in machine learning. The final section
summarizes the goals and outlines the contributions.
II. LITERATURE REVIEW
Performance measures and metrics are historically based on
binary similarity, distance or association measures and metrics
and several studies on several domains examine them [3][6].
There are hundreds of measures and metrics defined in the
literature [7]. However, many of them are duplicate and
including the ones we provided in this study [7], [8]. Relatively
new works have been concentrated on performance measures
and metrics and their properties related to classification
(skewness effect [9], accuracy paradox [10], multi-class [11],
chance correction [12], usage cost [13], constraints [14],
chronology [7], equation patterns [3]), requirements and
recommendations [15]). Lavesson and Davidsson examine the
conventional performance metrics from quality attributes and
metrics in software engineering domain (robustness, complexity
along with performance that includes accuracy, space, and time
sub-attributes) [16]. Similar to performance metrics, the
combination of binary association measures is practically
limitless. Paradowski proposed generalized measures/metrics
based on coefficients [6] whereas Koyejo et al. introduced
generalized performance metric [17].
III. THE CLARIFICATION OF TERMINOLOGY
The classification performance is calculated, presented and
compared by some 'performance metrics' that are also called as
'performance measures', 'evaluation measures', and 'prediction
scores' in supervised learning where a training set is provided
with known inputs along with the correct labels (outcomes, i.e.
the corresponding class for given inputs). Classification
performance is named differently in other domains such as
diagnostic (test) accuracy in medicine [18] or skill score or
forecast skill in meteorology (forecast vs. observation) [19].
Categorization is commonly used in philosophy and statistics
instead of classification [2].
Although metrics, scores, measures, indicators,
criteria, factors or indices seem the same and are used
interchangeably to state classification performance, there are
slight and important differences. Surprisingly, the confusion
over such terms is widespread in the literature. Even, the studies
related to classification performance use incorrect terminologies
and to the best of our knowledge, this study is the first one that
clarifies the terminology semantically in this section and
formally as described in explained in Section V.
As specifically examined from the general perspective by
Texel [20], measures at the bottom of pyramid hierarchy of
concept are numerical values with little or no context, metrics
that are above measures possess a collection of measures in
context, and indicators at the top are the comparison of metrics
to a baseline. Score has a similar meaning to measure. Therefore,
correct usage in classification is performance metrics.
Another terminological clarification that we address is about
the four types of direct results of binary classification namely
(number of) True Positives (TP), False Positives (FP), False
Negatives (FN), and True Negatives (TN) that are explained in
Section IV.A. These values, which are displayed in 2 rows by 2
columns of confusion matrix or contingency table for binary
classification, should be stated as classification measures not
metrics, etc. Performance metrics are calculated based on these
base measures and some other measures are derived from those
base measures. These three terminology clarifications (i.e.
measure vs. metric) in mind, you see that many past and even
current classification studies in the literature use incorrect
terminology and intermingle with one another.

8%0.QG,QWHUQDWLRQDO&RQIHUHQFHRQ&RPSXWHU6FLHQFHDQG(QJLQHHULQJ 

 IV. THE PROPOSED COMPHREHENSIVE VISUALIZED
ROADMAP FOR BINARY CLASSIFICATION PERFORMANCE
We have designed Figure 1 originally to provide a visualized
roadmap for binary classification performance measures and
metrics in all-in-one style. Both 22 measures and 22 metrics that
are described in next sections are shown in one chart like the
periodic table of elements. Gray colored cells correspond to
measures and orange colored ones correspond to metrics and
they are grouped in a leveled structure and positioned according
to similarities and row/column geometries. The depended
metrics and measures in their equations that are also shown in
the roadmap. Measures and metrics are built upon total 4 and 3
levels, respectively. The roadmap provides a neat distinction
between measure and metrics and describes measures, metrics,
their similarities, dependencies, and types via geometrical and
visualization techniques.
The roadmap also presents useful information as explained
in the legend. The names of measures and metrics that have no
upper limit are written in bold. The numbering for measures are
written in italic. The measures and metrics in the left of the
confusion matrix square are row type (depended solely upon
base measures, Sample Size (Sn) and OP or ON whereas the
ones in above the confusion matrix square are column type (the
same as row type but OP or ON are replaced by number of
condition Positive (P) and Negative (N). Row types are related
to testing or prediction whereas column types are related to
reality as described more in below. The roadmap with these
details is worthy of all-in-one attribution. Note that AUC
metric is out of scope in this study.
In this section and next sections, we provide all the
performance measures and metrics equations in binary
classification specific terms and give equivalent or derived
canonical equations to assist interpreting them easily. The
measures that represent all combinations of a classification are
stated by two combinations of two letter groups as T: True or
matches vs. F: False or non-matches for classification
correctness with P: Positive vs. N: Negative for
representation of two-classes. They are presented in 2x2
confusion matrix or contingency table.
A.
Confusion Matrix, The Four Base Performance Measures
As stated in Section III, we call the four direct outputs of
classification performance (TP, FP, FN, TN) as base measures
in a binary classification with supervised learning approach.
True classification results or prediction/reality matches (TP
and TN) are located on diagonal whereas False classification
results, non-matches or errors (FP and FN) are off-diagonal in
confusion matrix as seen in Figure 1.
In critical engineering and medicine practices, type II errors,
False Negatives, could be more serious or worse than type I
errors, False Positives. But the proper approach depends on the
domain and its specific application. For example, in malware
analysis, it could be better to mistakenly label a benign software
as malign than miss a malign software by incorrectly labeling
it as benign (labeled malware are prioritized and an expert could
go through further manual malware analysis to eliminate false
positives then [21]). Although in law or social perspective the
opposite is likely to be valid to ensure presumption of
innocence. An anti-malware product should be designed or
configured to decrease the False Positives (or False Alarms) to
avoid annoying interruptions due to excessive malware
warnings. Whereas precautionary logic focuses on more
underestimates (False Negatives) than overestimates (False
Positives) in criminal justice [22].
B.
The Proposed Performance Measures (TC, FC) and The
1st Level Measures (P, N, OP, ON, Sn)
The 1st level performance measures are one level above the
base performance measures. Condition Positive (P) and
condition Negative (N) measures that are column totals (also
known as marginal totals in probability theory) of confusion
matrix represent the real or actual values of the two classes (i.e.
the real labels). These measures correspond to the reality,
observed or ground truth. OP and ON measures that are row
totals (also known as marginal totals in probability theory) of
confusion matrix represent the predicted (test or classification
result) of the two classes (i.e. the given labels). These measures
correspond to the prediction or estimated (classification output).
When we examined many measures and metrics we had seen
that some of them have specific elements that are the total value
of diagonal base measures (TP and TN) and off-diagonal ones
(FP and FN). For the first time, we have introduced and named
those totals as True Classification (TC) and False Classification
(FC), respectively. Substituting those totals have significantly
simplified the metrics equation and their interpretation. Sn
measure that is the sum of four base performance measures
could be stated by P plus N measures, by OP plus ON, and by
TC plus FC.
Before going into details of performance measures and
metrics based on the measures explained above, two attributes
of them, namely duality and complement concepts, are
introduced and described here. The literature has not addressed
metrics duality and complements sufficiently. We also
introduced the canonical transformation of the equations and
proposed formal rules to define classification performance
measures and metrics. In the abundance of metrics, these
attributes that we presented in this study make measures and
metrics more comprehensible.
V.
A NEW FORMAL VIEW: CANONICAL FORMS AND
DEFINITIONS OF MEASURES AND METRICS
In this study, we suggest canonical forms of measure and
metric equations that are defined as follows:
Definition: Canonical Forms
The forms where equations are stated by 11 base measures
and 1st level measures (i.e. TP, FP, FN, TN, P, N, OP, ON, TC,
FC, and Sn).
We have clarified the difference between metrics and
measures semantically in Section III. Determining a given
equation is a performance measure or metric is also not studied
before. Because a binary classification performance is related to
TP, FP, FN, TN base measures, metrics should depend on at
least one of them. The following proposed rules are valid for
measures. Otherwise, given canonical form is a metric.

8%0.QG,QWHUQDWLRQDO&RQIHUHQFHRQ&RPSXWHU6FLHQFHDQG(QJLQHHULQJ 

Fig. 1. Visualized roadmap for binary classification performance measures/metrics (22 measures and 22 metrics providing names,
abbreviations, other namings, levels, dependencies, row/column geometries, equations with new canonical forms, and special notes such as
range apart from (0,1), duals, complements, etc.). See the legend for details. Visit http://bitly.com/metricsroadmap for feature updates.

Knowing this transformation, we could become aware of duality

Definition: Measure
The canonical form includes only P, N, OP, ON, or Sn
Otherwise, the possible values have no lower (-
)
and/or upper limit (+
)
VI. THE DUALITY AND COMPLEMENTATION REVEALED IN
CLASSIFICATION PERFORMANCE MEASURES/METRICS
Duality is basically related to the transformation of one
concept into another concept in a bilateral approach. We
introduce the duality concept in classification performance
metrics in this study. The proper transformation for performance
metrics duality lies behind the column-row approach on
confusion matrix that we described above. The row versus
column transformation corresponds to prediction versus reality.
formally in every metric or measure.
We propose to adapt duality notation used in vector space
that states a duality of a vector space V to its dual vector space
with asterisk superscript (*) as V*. In this notation, we can
express the duality among performance measures and metrics.
For example, the followings are the dualities for the 1st level
measures defined in previous subheadings:
  & 

  & 
  & 

  &  *+
The symmetry or involution is always valid for the duality of
performance measures and metrics. The duality is especially
important to transform a mapping known in one concept to the
same mapping in dual concept. For example, a mapping exists
in one definition of a metric could be transformed or seek in
corresponding dual metric.

8%0.QG,QWHUQDWLRQDO&RQIHUHQFHRQ&RPSXWHU6FLHQFHDQG(QJLQHHULQJ 

 Because all the performance measures and metrics are for
binary classes and normalized in range (0, 1) mostly and (-1, 1)
rarely, the ratios could be complemented. Therefore, all the
measures and metrics have complements. The complements
could be written by this adapted notation:
classification model and always labeled a given instance with
Negative. Any classification should perform better than this
limit. NIR that is an improved version of NER specifies the
minimum performance by counting the larger condition from
Positive and Negative conditions. Eventually, the following
conditions should be achieved at least for a better classification.
"  #
     Note that a classification may have a close performance to NER
# & ! "
 % "   ( " ( 
& $ &
%"  % ( " ( 
In contrast with duality, having both a measure/metric and
its complement does not contribute any extra information. A
complement could be used for simplification of equations or
switching the primary point of view to another one such as
switching from positive condition based view (e.g. TPR or PPV)
to negative one (e.g. FNR or FDR) or focusing on errors (i.e.
Misclassification Rate, MCR) instead of correctness (i.e. ACC)
as illustrated in Figure 1.
VII. THE 2ND LEVEL MEASURES
Prevalence is the ratio of Positive condition size (P) to the
total number of conditions (P + N) or sample size (Sn).
Prevalence (PREV), which is called as Pretest Probability of
Admission in medicine, is related to the reality. (Label) Bias
(BIAS) is the ratio of Outcome Positive size (OP) to the total
number of test size (OP + ON) or sample size (Sn). Bias is
related to the prediction (classification output). It is also known
as Detection Prevalence or Warning Rate. Prevalence and
bias are dual measures. (Class) skew that is the ratio of majority
class to minority class (usually N to P) is an important measure
along with Prevalence in classification [9].
VIII.
BASE AND CONVENTIONAL 1ST LEVEL METRICS
The base metrics are calculated within measures in confusion
matrix columns (vertically) or rows (horizontally) as seen in
Figure 1. The column type base metrics (TPR, TNR, FPR, FNR)
are the rate of each base measure to corresponding condition
(Positive or Negative). The row type base metrics are the rates
of each base measure to corresponding test output. The first level
metrics are the most preferred and known metrics in expressing
binary classification performance as a single value.
A.
Accuracy (ACC)
Accuracy is the ratio of total number of correct
classifications to sample size. It is a diagonal metric covering
neither rows nor columns of confusion matrix. It was defined as
a similarity measure called matching coefficient between two
individuals characterized by a number of binary attributes by
Sokal and Michener in 1958 [23]. Accuracy is the most provided
and perhaps the most abused metric in binary classification
performance reports as described below.
B.
Minimum Expected Performance and Other Measures
Although the ultimate goal of a classification is achieving the
highest accuracy as possible another performance criterion may
be disregarded: what is the minimum performance measure that
should be expected that a binary classification? The two
measures are the answers to this question namely Null Error
Rate (NER) and No Information Rate (NIR). NER specifies the
minimum successful classification rate if we do not have a
and NIR measures, this case is called as Accuracy Paradox
[10], [24]. Therefore, providing Accuracy as a single
performance metric is not sufficient.
 '  ) 
CKc is Kappa Chance coefficient that is also called as
Random Agreement consists of column and row totals. Go
further in geometry as seen in Figure 1, likelihood ratios (LRP,
LRN) are row based measures that do not have an upper limit
(e.g. not in range (0, 1) as in other metrics). Odds Ratio (OR) is
a 3rd level measure. It is the ratio of Positive Likelihood Ratio to
Negative Likelihood Ratio 2nd level measures. Positives have
OR times the odds of being positive compared to Negatives [25].
Discriminant Power (DP) and OR that are different from all the
metrics and measures are in (-
, +
) range.
IX. BEYOND THE CONTROVERSIAL ACCURACY METRIC
A.
Informedness, Markedness
Informedness and Markedness are dual metrics, the former
is about reality whereas the latter is about prediction. It
represents consolidated true classification capability per
condition so that the performance is about reality. Informedness
is equal to Peirce Skill Score (e.g. in weather forecasting [19]).
Markedness, also known as Difference in Proportions
(NPV
FOR) represents consolidated true classification
capability per test outcome so that the performance is about
prediction. Markedness is equal to Clayton Skill Score [19]
B.
More TPR-TNR Combinations
BACC and Gm are the alternatives to ACC. They are actually
means of correct classification rates; the former is arithmetic and
the latter is geometric mean of TPR and TNR. BACC, also known
as Strength. Gm is also known as FowlkesMallows index.
C.
F-measures (F1, F0.5, F2, F), Cohens Kappa (CK),
Matthews Correlation Coefficient (MCC)
F-measures are the metrics covering two base metrics one
from column (TPR) and one from row (PPV). F-measures that
are harmonic means of those two metrics are actually parametric
metrics, which define the weights on TPR and PPV. They are
insensitive to TN [26]. F1 is harmonic mean of precision and
recall with equal weights whereas F0.5 puts more emphasize on
TPR whereas F2 puts more emphasize on PPV. CK that is a
bidirectional metric in (-1, 1) range is related to Accuracy [9] (it
is equal to Heidke Skill Score [19]). MCC is bidirectional in (-1,
1). F-measures, CK, and MCC are the unconventional metrics
that we suggest the researchers take into account additional to
Accuracy and other metrics.
X.
DISCUSSION AND CONCLUSION
Performance metrics are critical instruments for assessing
and expressing the success of a binary classification study.
Although many metrics are available in the literature only a few

8%0.QG,QWHUQDWLRQDO&RQIHUHQFHRQ&RPSXWHU6FLHQFHDQG(QJLQHHULQJ 

permission metrics are reported in the conducted studies. As
explained in Accuracy Paradox, these accustomed metrics
have some complications. Using the proper and sufficient
number of metrics while comparing different binary
classification approaches leads to a more objective assessment.
We have seen that there are many studies in the literature on
several domains (botanical, meteorology, chemistry, biology,
medicine, economics, malware analysis, etc.) in which different
metrics and measures are proposed. Because of domain
independence of performance measures, those studies may
provide researchers new alternative resources in other domains
to explore so that knowledge transfer would be possible.
We also have seen that there are many different notation
adaptations in the related studies and established different
notation conventions for the same metrics on different domains
(e.g. recall vs. sensitivity for TPR). Nevertheless, we suggested
our specific naming and notation that still reflects the majority
of common conventions in machine learning classification
literature. We preferred abbreviations of metrics that are more
explicit for binary classification.
In this study, we have uncovered the semantic and formal
distinction between performance measure and metric. Although
classification performance measures/metrics are indispensable
instruments of classification experiments, the confusing
terminology that is widespread in even academic studies has not
been clarified before. We have suggested formal rules to
determine a given equation as a measure or metric as well as
establishing the terminology. We have provided 44 measures
and metrics with their equations and introduced new concepts
on handling them such as canonical forms to use in definitions,
relating different measures/metrics by duality/complementation.
In addition, we have designed a comprehensive, visualized,
row/column structured, leveled, and all-in-one style roadmap
shown in Figure 1 that could be considered as the periodic table
of elements in binary classification performance in machine
learning. Our new basing and leveling approach in the layout
contributes to ordering measures/metrics logically and providing
a more comprehensive style. It is expected that the domain-
independent roadmap gives researchers a more standard,
complete and easier way to understand the performance
measures/metrics, their relations and dependencies.
ACKNOWLEDGMENT
G.C. thanks HAVELSAN for supporting this study.
REFERENCES
[1] A. Shaar, T. Abdessalem, and O. Segard, Pessimistic Uplift Modeling,
in 22nd SIGKDD Conference on Knowledge Discovery and Data Mining
(ACM SIGKDD), 2016.
[2] C. Sammut and G. I.Webb, Eds., Encyclopedia of Machine Learning.
New York: Springer, 2011.
[3] M. J. Warrens, Similarity Coefficients for Binary Data, Leiden
University, 2008.
[4] M. R. Anderberg, Measures of Association among Variables, in
Cluster Analysis for Applications: Probability and Mathematical
Statistics: A Series of Monographs and Textbooks, Academic Press,
1973, pp. 8292.
[5] M. M. Deza and E. Deza, Encyclopedia of Distances. Springer, 2009.
[6] M. Paradowski, On the Order Equivalence Relation of Binary
Association Measures, Int. J. Appl. Math. Comput. Sci., vol. 25, no. 3,
pp. 645657, 2015.
[7] C. Seung-Seok, C. Sung-Hyuk, and C. C. Tappert, A Survey of Binary
Similarity and Distance Measures., J. Syst. Cybern. Informatics, vol. 8,
no. 1, pp. 4348, 2010.
[8] Z. Hublek, Coefficients of Association and Similarity, Based on
Binary (Presence-Absence) Data: an Evaluation, Biol. Rev., vol. 57, no.
4, pp. 669689, 1982.
[9] S. Straube and M. M. Krell, How to evaluate an agents behavior to
infrequent events? Reliable performance estimation insensitive to class
distribution, Frontiers in Computational Neuroscience, vol. 8, no.
April, pp. 16, 2014.
[10] F. J. Valverde-Albacete and C. Pelez-Moreno, 100% classification
accuracy considered harmful: The normalized information transfer factor
explains the accuracy paradox, PLoS One, vol. 9, no. 1, 2014.
[11] M. Sokolova and G. Lapalme, A systematic analysis of performance
measures for classification tasks, Inf. Process. Manag., vol. 45, no. 4,
pp. 427437, 2009.
[12] V. Labatut and H. Cherifi, Evaluation of Performance Measures for
Classifiers Comparison, Ubiquitous Comput. Commun. J., vol. 6, pp.
2134, 2011.
[13] B.-G. Hu and W.-M. Dong, A study on cost behaviors of binary
classification measures in class-imbalanced problems, Comput. Res.
Repos., vol. abs/1403.7, 2014.
[14] A. Forbes, Classification-algorithm evaluation: five performance
measures based on confusion matrices, J. Clin. Monit. Comput., vol. 11,
no. 3, pp. 189206, 1995.
[15] R. E. Tulloss, Assessment of Similarity Indices for Undesirable
Properties and a new Tripartite Similarity Index Based on Cost
Functions, in Mycology in Sustainable Development: Expanding
Concepts, Vanishing Borders., 1997, pp. 122143.
[16] N. Lavesson and P. Davidsson, Analysis of Multi-Criteria Metrics for
Classifier and Algorithm Evaluation, in Proceedings of the 24th
AnnualWorkshop of the Swedish Artificial Intelligence Society, 2007, pp.
1122.
[17] O. O. Koyejo, N. Natarajan, P. K. Ravikumar, and I. S. Dhillon,
Consistent Binary Classification with Generalized Performance
Metrics, Adv. Neural Inf. Process. Syst. 27 Annu. Conf. Neural Inf.
Process. Syst. 2014, December 8-13 2014, Montr. Quebec, Canada, pp.
27442752, 2014.
[18] K. J. van Stralen, V. S. Stel, J. B. Reitsma, F. W. Dekker, C. Zoccali, and
K. J. Jager, Diagnostic methods I: sensitivity, specificity, and other
measures of accuracy, Kidney Int., vol. 75, no. 12, pp. 12571263, 2009.
[19] D. S. Wilks, Statistical methods in the atmospheric sciences, 2nd ed.,
vol. 59. Elsevier, 2006.
[20] P. P. Texel, Measure, metric, and indicator: An object-oriented
approach for consistent terminology, in Proceedings of IEEE
Southeastcon, 2013.
[21] G. McWilliams, S. Sezer, and S. Y. Yerima, Analysis of Bayesian
classification-based approaches for Android malware detection, IET
Inf. Secur., vol. 8, no. 1, pp. 2536, 2014.
[22] H. M. Lomel, Punishing the uncommitted crime: Prevention, pre-
emption, precaution and the transformation of criminal law, in Justice
and Security in the 21st Century: Risks, Rights and the Rule of Law, 1st
ed., B. Hudson and S. Ugelvik, Eds. Abingdon, Oxon, United Kingdom:
Routledge, 2012.
[23] D. W. Goodall, The distribution of the matching coefficient,
Biometrics, vol. 23, no. 4, pp. 647656, 1967.
[24] T. Bruckhaus, The business impact of predictive analytics, in
Knowledge Discovery and Data Mining: Challenges and Realities, X.
Zhu and I. Davidson, Eds. Information Science Reference, 2007, pp.
114138.
[25] M. Szumilas, Explaining odds ratios, J. Can. Acad. Child Adolesc.
Psychiatry, vol. 19, no. 3, pp. 227229, 2010.
[26] D. M. W. Powers, What the F-measure doesnt measure: Features,
Flaws, Fallacies and Fixes, KIT-14-001, 2015.

8%0.QG,QWHUQDWLRQDO&RQIHUHQFHRQ&RPSXWHU6FLHQFHDQG(QJLQHHULQJ