730 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 12, NO.
3, MARCH 2017
Detecting Anomalous Behavior in VoIP Systems:
A Discrete Event System Modeling
Diksha Golait and Neminath Hubballi
Abstract Session initiation protocol (SIP) is an application
layer protocol used for signaling purposes to manage voice over
IP connections. SIP being a text-based protocol is vulnerable to
a range of denial of service (DoS) attacks. These DoS attacks can
render the SIP servers/SIP proxy servers unusable by depleting
memory and CPU time. In this paper, we consider two types of
DoS attacks, namely, flooding attacks and coordinated attacks
for detection. Flooding attacks affect both stateless and stateful
SIP servers while coordinated attacks affect stateful SIP servers.
We model the SIP operation as discrete event system (DES)
and design a new state transition machine, which we name as
probabilistic counting deterministic timed automata (PCDTA) to
describe the behavior of SIP operations. We also identify different
types of anomalies that can occur in a DES model, which appear
in the form of illegal transitions, violating timing constraints, and
appear in number which is otherwise not seen. Subsequently, we
map various DoS attacks in SIP to a type of anomaly in DES.
PCDTA can learn probabilities of various transitions and timings
delay from a set of nonmalicious training sequences. A trained
PCDTA can detect anomalies, and hence various DoS attacks in
SIP. We perform a thorough experiment with computer simulated
SIP traffic and report the detection performance of PCDTA on
various attacks generated through custom scripts.
Index Terms Communication system security, Computer
security, Network security.
I. I NTRODUCTION
V OICE over IP (VoIP) is an economical alternative for
telephone communication compared to traditional Public
Switched Telephone Network (PSTN) communication. In VoIP
communication the voice conversation data is sent using IP
packets over Internet. A typical voice call communication
involves two phases as signaling and data transmission. Sig-
naling is used to establish and maintain the end to end VoIP
call; the actual data transmission usually happens in a different
session. VoIP can use a range of protocols (H.323, SIP) for
signaling purposes. Session Initiation Protocol (SIP) is an
application layer signaling protocol for VoIP communication.
It is used to establish, modify and terminate multimedia
sessions between two VoIP clients also called user agents.
It is also used to request and deliver clients presence; send
and receive instant messages between clients. SIP server(s)
and/or SIP proxy servers mediate the session initiation and
Manuscript received February 1, 2016; revised June 27, 2016, September 8,
2016, and November 14, 2016; accepted November 14, 2016. Date of
publication November 23, 2016; date of current version January 18, 2017.
The associate editor coordinating the review of this manuscript and approving
it for publication was Prof. Wanlei Zhou.
D. Golait was with IIT Indore, Indore 453552, India. She is now with
R&D, Microsoft India, City - Hyderabad (Telangana) 500032, India (e-mail:
digola@microsoft.com).
N. Hubballi is with IIT Indore, Indore 453552, India (e-mail:
neminath@iiti.ac.in).
Color versions of one or more of the figures in this paper are available
online at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TIFS.2016.2632071
1556-6013 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
termination between two clients. SIP being a text-based pro-
tocol and designed to work with both UDP and TCP is
vulnerable to a range of security attacks [1] particularly Denial
of Service (DoS) attacks. DoS attacks include various SIP
flooding attacks like BYE flooding, INVITE flooding, multi-
attribute flooding, etc. Recently it has been discovered that a
number of coordinated attacks [2] can also be mounted on SIP
(proxy) servers and user agents, creating DoS. These flooding
and coordinated attacks can completely cripple the communi-
cation between VoIP servers rendering them unusable. Hence
detecting these attacks is important.
There are mitigation and detection techniques [3] proposed
in the literature for protecting SIP operation from various types
of DoS attacks. Many of these methods propose cryptographic
extensions to secure SIP or use rule based engine to detect
malformed SIP messages (which sometimes cause DoS) or
propose a machine learning algorithm to predict different DoS
attacks. Most of the prior works focused on detecting specific
types of DoS attack. In this paper we propose a generic
formal framework which is custom designed to describe SIPs
operational behavior and use it to detect different types of DoS
attacks. In particular we make following specific contributions
in this paper.
1) We consider the SIP operation sequence as a Discrete
Event System (DES). Subsequently we develop a proba-
bilistic timed transition model (PCDTA) to characterize
SIP event sequences and their timings.
2) We also propose to learn transition and delay prob-
abilities of various events of state transition diagram
from a set of known non malicious SIP event sequences
thus making learning automatic which is otherwise done
manually.
3) We identify a range of anomalies that can occur in any
timed DES model and map these anomalies to various
DoS attacks in SIP.
4) We use the timed transition model as an anomaly
detection system to detect anomalies, arising as a conse-
quence of occurrence of illegal transitions, timings and
in number for a particular message type, which help
detect different SIP attacks.
II. SIP OVERVIEW
SIP has a distributed architecture. It includes the following
entities.
User agent: These are VoIP phones with a valid URI
(user name used by a user). Multimedia sessions are setup
and terminated between user agents.
Registrar server: User agents register with a registrar
server when they connect to network and also update
them periodically.
GOLAIT AND HUBBALLI: DETECTING ANOMALOUS BEHAVIOR IN VoIP SYSTEMS: A DES MODELING
Fig. 1. SIP call setup.
Location server: Stores different locations of user agents
which are identified by their IP addresses. There can be
more than one location associated with a user.
Proxy server: Forwards the connection requests to the
intended recipients on behalf of user agents.
Redirect server: If a user has more than one location,
redirect server helps to fork a connection request to
different addresses.
Figure 1 shows a typical call setup using SIP. A typical SIP
session begins with an INVITE from the caller. SIP proxy
server receives the INVITE and immediately sends a 100
(TRYING) message to caller. When the request reaches the
callee, it starts ringing and it indicates the RINGING status in
a 180 response. If the callee accepts the call, callees phone
sends a 200 (OK) response, otherwise an error response would
be sent. Callers phone acknowledges it by sending an ACK.
This completes the three-way handshake INVITE/200/ACK
used to establish SIP sessions. After this, media session start.
At the end of call, either of the two user agents can send a
BYE request to end the call. The other user agent responds to
it with a 200 (OK) response.
III. SIP T HREAT V ECTORS
In this section we describe different types of DoS attacks
to which SIP is vulnerable. These attacks can be grouped
as flooding and coordinated attacks; details of these two
are given in next two subsections. It is worth differentiating
between stateful and stateless SIP proxy servers at this point
as few types of attacks affect only particular SIP proxy
server. A stateful proxy server maintains the state information
about various incomplete and ongoing transactions, whereas a
stateless proxy server just forwards the SIP messages without
maintaining any state information. A stateful proxy server can
be of type transaction stateful or call stateful.
Transaction stateful: A proxy is transaction stateful if it
maintains the client and server transaction state machines
as defined in the specification of RFC 3261 [4].
Call stateful: A proxy is call stateful if it retains state for
a dialog from the initiating INVITE to the terminating
BYE request. A call stateful proxy is always transaction
stateful.
A. SIP Flooding Attacks
In flooding attacks large number of messages of a particular
type are sent to overwhelm the SIP server or SIP proxy server.
731
These attacks affect registrar servers; and both stateful and
stateless proxy servers. There are mainly 4 types of flooding
attacks possible in SIP, as detailed below.
1) Invite Flooding: In this attack, large number of INVITE
messages are sent to a SIP server by a/many malicious client(s)
which never completes the handshake with other user agents
leaving these transactions incomplete. Each such request has a
bearing on resources available at server. Further if the interme-
diate proxy servers are stateful, they also have the risk of run-
ning out of resources if the rate of INVITE messages is high.
2) BYE Flooding: In this attack a large number of BYE
messages are sent to a SIP server. There are two variations of
BYE flooding cases.
1) With random call-ID: Here attacker sends large number
of crafted BYE message to SIP server with a randomly
generated call-ID. This can terminate arbitrary ongoing
connections if the call-ID of such messages match
with call-ID of any ongoing call for which server has
maintained a state.
2) With a known call-ID: This is a targeted attack where
the attacker is able to sniff the traffic exchanged between
an SIP user agent and SIP server and get hold of
call-ID. Now the attacker can generate a BYE message
using that call-ID which will terminate the connection.
3) REGISTER Flooding: Here a large number of user agent
REGISTER requests are sent to the SIP registrar. The server
spends lot of computing and memory resources in addressing
these requests.
SIP mandates every user agent to first REGISTER with
a registrar server. This registration process can be secure
with added authentication or can be unsecured without any
security. SIP uses a digest authentication algorithm to negotiate
credentials with the user agents. In this, the user initially
sends a request to the server that requires authentication but
does not provide credentials; the server responds with the 401
Unauthorized response code, providing the authentication
realm and a randomly generated value called nonce; the
client then re-sends the same REGISTER request with an
authentication header that includes the response code; if the
credentials are correct SIP server answers this request with
a 200 OK message. There are two variations of REGISTER
flooding attacks that can be generated here depending on
whether it uses security or not.
1) With Random URI: In this case attacker can use a
random user agent name and try to REGISTER with a
SIP server. Depending on whether the user is having an
account or not SIP server will respond back with either
OK or 404 user not found message.
2) With a valid URI: In this case known user agent names
are used to REGISTER with random IP addresses. If the
authentication is enabled SIP server sends a validate
authentication message, the attacker just ignores the
challenge and starts a new request with another user
agent name. However the server keeps the allocated
memory for a certain period of time, in addition to the
wasted CPU resources for calculating the nonce. This
can cause DoS if number of such requests is very high.
732 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 12, NO. 3, MARCH 2017
4) Multi-Attribute Flooding: In this attack a large number
of messages using all four SIP message types (INVITE,
BYE, RINGING, OK) are sent to overwhelm the SIP server.
An attacker can cleverly craft the attack to balance the ratio
of each type of messages rendering the anomaly detection
methods like [5] and [6] ineffective which use the imbal-
ance between different message types to detect DoS flooding
attacks.
B. Coordinated SIP Attacks
Coordinated SIP attacks are carried out by colluding users
who have registered in the VoIP service. Following are the
two prominent attacks.
1) Ringing Attacks: These attacks attempt to create incom-
plete transactions at the server with host cooperation [2]. The
attacker sends invite requests to known peers which reply
with provisional (1xx) messages like RINGING messages but
dont accept the call. Instead callee just repeatedly sends these
provisional RINGING messages. This can prolong the lifetime
of a transaction to several minutes. Large number of such
incomplete transactions will cause DoS. The attacker needs to
send fewer requests (compared to flooding cases) to deplete
the memory of the SIP server. Ringing attacks affect both types
of stateful proxy servers.
2) Prolonged Calls: In these attacks, the attackers exchange
with the SIP server the same initiation messages as in a normal
SIP session. But once the call is established, the attackers stay
in the call for as much indefinite time as they can or till the
server interrupt occurs. Prolonged call attacks affect only call
stateful proxy servers.
IV. D ISCRETE E VENT S YSTEMS
A discrete event system is characterized by following three
properties.
1) Discrete States: The system can be in any one of the
finite number of states. The state in which the system
is currently in indicates the status of system. The most
basic status of system used to detect anomalies indicates
either normal or anomalous state.
2) Dynamic: The new state in which the system stays is
dependent on the current state.
3) Event-driven: The change in the system state is com-
pletely driven by events occurring at certain times.
V. T YPES OF A NOMALIES
In this section we adopt and describe few types of anomalies
seen in any DES [7] with few of our own additions. We also
establish mapping between these anomaly types to VoIP based
SIP threat vectors described previously.
1) Anomalous Event: An anomalous event is an event
which causes the system to move to a state from its
current state which is not a regular next state of current
state.
For example, BYE message must appear only after
a successful INVITE initiated dialogue. However any
event with a BYE message without corresponding
INVITE, RINGING and OK messages, with an arbitrary
call-ID is an anomalous event. These type of messages
appear in case of BYE flooding attack.
2) Anomalous Path: It may be the case that an individual
event may be normal, however a sequence of events
taken together may signify anomalous behavior.
For example, the SIP system can be in the state
RINGING received. Further it accepts an OK message.
However occasionally messages do get lost in network,
or due to errors the end hosts retransmit such messages.
Thus it is possible that being in RINGING received
state to receive another RINGING message with same
call-ID. However in case of coordinated RINGING
attack, the system may repeatedly receive RINGING
messages when the system is in RINGING received state
to unnecessarily prolong the call setup process.
3) Non Occurrence of Events/Stalled Progress: As the
system changes its state or makes progress only after
receiving events, it is anomalous not to receive messages
to further the progress. Although this can happen due to
various reasons like when connectivity goes down for a
node responsible for sending that message, etc. Never-
theless frequency of such cases may raise suspicion and
need to be detected.
For example, in VoIP system a peer may deliberately
stop sending the next expected OK message after receiv-
ing an INVITE message. If such events occur in large
number, the proxy server or SIP server may experience
resource crunch and become victim of DoS.
4) Anomalous Timing of Events: Normally the next
expected event should occur within a time period after
the current events timing. If the next event takes unusu-
ally longer than expected timing, it is considered an
anomalous timing event.
For example, in RINGING attacks, after receiving initial
INVITE message the peer entity deliberately delays
sending RINGING message.
5) Anomalous Sample Timing Path: Timing of a path may
be anomalous even if every events timing is normal, if
the aggregate timing of the sequence of events is not
within a prior identified range.
For example, in coordinated SIP attack, calls are pro-
longed unnecessarily by delaying messages and repeat-
edly sending some of the messages.
In section VIII we show that all the SIP DoS attacks
described previously can be detected if these anomalies are
identified in a DES model of SIP.
VI. P ROPOSED DES M ODEL
In this section we formally define a DES model which we
use to describe the behavior of VoIP based SIP communica-
tion. We treat SIP events as timed sequences indicating every
event occurs in the system at discrete time. Figure 2 and
Figure 3 show the timed events appearing in INVITE dialogue
and REGISTER operation of user agent and as observed at SIP
server and Registrar server respectively.
In order to characterize these timed events we propose a
state transition machine as a DES model. One of the motive for
this novel state transition model is, it should detect all types
GOLAIT AND HUBBALLI: DETECTING ANOMALOUS BEHAVIOR IN VoIP SYSTEMS: A DES MODELING 733

qi Current state
qj Next state
c Counter value at the state qi
Input symbol
(t) Boolean conjunction of constraints on a subset
of clock variables t T to be satisfied
Fig. 2. Timing diagram showing basic path of a SIP call sequence.
Reset (t) Is a subset of clock variables t T to be reset
on this transition
I nc(c) Is a function which maps the current state of
counter to a new value

PCDTA is similar to a Probabilistic Timed Automata (PTA)

Fig. 3. Timing diagram showing basic path of a SIP REGISTER sequence.
of anomalies described previously. As different DoS attacks
of VoIP system are mapped to one of these anomaly type this
model in effect should be able to detect all DoS attacks.
and has constraints on timings of various events similar to
Timed Automata [8] with added counting ability similar to [9]
with a subtle difference in the way probability values are
assigned to each transition. In PTA the following probability
constraint holds
s

Our proposed DES model is a probabilistic timed transition pi j (a) = 1 a  (1)

state machine with the ability to count the number of times a
particular state has been visited. Counting the number of times
a state has been visited is required for identifying repeated
transmissions of same message type. Similarly timing con-
straints help in identifying unusually delayed transitions. To be
precise, we describe a Probabilistic Counting Deterministic
Timed Automaton (PCDTA). The state machine PCDTA is
denoted as
M = (Q, , C, T , , q0 , , , , , F) where
Q Is a finite set of states
q0 Q Is an initial state
 Is a finite set of input symbols
 Is an input symbol
C Is a finite set of Counters with each ci C
taking values in N
T Is a finite set of Clock Variables where each
ti T take non negative values in R
Is a set of transitions defined as
: Q C 2|T |  (T ) Q 2|T |
(C C)
 Is a set of transition probability distributions
 Is a set of timing delay probability distributions
: i  is a transition probability function,
which assigns a probability distribution  transitions defined. Thus any string S  is generated with a
to a collection of transitions i where i
represents the transitions to next state q j from
current state qi . Every  is defined as
:  [0, 1] with  N a set of counter
values of state qi .
:  is a transition time probability
j =1
In Equation 1, pi j (a) denotes the probability of automaton
to move from state qi to state q j on input symbol a. The value
s is the number of such next states of qi . This ensures that on
input a the sum of probabilities of all outgoing edges from qi
add to 1. In PCDTA the probability of transitions satisfy the
constraint of Equation 2.

s
pi j (/c) p(c) = 1  (2)
cN j =1
Equation 2 enforces that the sum of probabilities of all
transitions from state qi to q j over all possible values of a
counter value c at qi and for all possible input symbols 
add to 1.
The PCDTA is a machine which generates a set of strings
starting from start state to an accepting state. The set of
strings generated by the PCDTA is the language generated
by it and is denoted as L  . The string generation can
be defined as a recursive process. For example the string with
S = 1 , 2 , , n can be generated as (q0 , 1 , 2 , , n )
where denote the extended transition function which is
defined recursively as ( (q0 , 1 , 2 , , n1 ), n ). While
generating a string if qi is the current state and k is the input
symbol, the next state is chosen according to the probability of
probability given by Equation 3. Later we use this probability
of a string to detect anomalous paths. Similarly we also define
the timing probability of a path as a product of their respective
event timing delays.

n

function, which assigns a probability distribution P(S) = p(qi , i+1 ) (3)

 to every transition. Every  is
defined as :  [0, 1] with  R a set of
time values.
F A subset of states (F Q) which are final states
Each transition i is a seven tuple qi , q j , c,
, (t), Reset (t), I nc(c) with the elements representing
i=0
VII. L EARNING THE M ODEL
In this section we describe how to learn the PCDTA model
from a set of observations of normal VoIP SIP call sequence
behavior. Normally in the discrete event systems the transitions
are identified using a set of observation sequences. There
734 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 12, NO. 3, MARCH 2017
Fig. 4. Simplified PCDTA for SIP INVITE dialogue.
Fig. 5. Simplified PCDTA for SIP REGISTER operation.
are algorithms like ALERGIA [10] and Acyclic Probabilistic
Finite Automata Learning [11] to learn both transitions and
probabilities from input sequences. A good coverage of dif-
ferent types of learning methods for transitions and the proba-
bilities is in [12]. However in our case the progress of system
(transition) behavior is very well defined and is unambiguously
described (as shown in Figures 2 and 3) in SIP protocol
specification, hence we create the transition diagram of the
machine from this specification [4]. However some of the
SIP messages may be retransmitted due to errors and losses.
These retransmissions need to be handled to avoid them being
treated as anomalous events. These messages carry the same
call-ID as the original message. These events are handled by
having suitable additional transitions in the machine. Further
the delay between successive message sequence are governed
by timing constraints. Thus we wish to learn the probabilities
of transitions and timings from a sequence of observations.
Generating event sequence diagram from SIP specification,
learning the probabilities of transitions and probabilities of
timings of transitions in the model are described in next three
subsections.
A. Generating Event Sequence Diagram
We model two SIP use case scenarios as shown in Figure 4
and Figure 5 using Probabilistic Counting Deterministic Timed
Automata (PCDTA). The first is the INVITE initiated dialog
which is terminated with a BYE (and its corresponding OK
response). The second is the REGISTER operation. Both the
operations are modelled as observed on a proxy server and
registrar server respectively.
Assumptions: We make the following two assumptions
while modeling the SIP event sequences.
i. If there are any redirection messages, the User Agent
Client (UAC) or the caller reuses the same To, From,
and call-ID used in the original redirected request so
that it follows the same automata instance.
ii. Any 4xx except 401(Unauthorized) or 407(Proxy
Authentication Required), 5xx and 6xx response is con-
sidered to lead to an error state that terminates the
existing dialog.
Separate machine for each operation: The machine for each
session is identified by its call-ID. Hence there will be many
machines running in parallel when our system is put into use
for validating SIP traffic. If the machine successfully follows
all the transitions and reaches the final state from start state
with the path probability less than threshold probability (to
be discussed), the SIP operation is considered to be normal.
If the machine terminates by reaching the error state, if it
reaches a deadlock (stalls somewhere) or if the machine gets
involved in a live lock, the SIP operation is considered to
be anomalous. The deadlock condition is handled by the timer
constraint associated with the concerned transition. In order
to measure the time difference between events, system time
is captured and associated with timing of each event. If the
timing delay at any instance is greater than the threshold, the
machine reaches the stalled state. The live lock condition is
checked by the path transition probability.
1) Complexity: It is worth noting that maintaining state
information of transition diagram involves some amount of
CPU and memory overhead. As the transition diagrams are
created for every new VoIP call and are destroyed either
because of error, timeout (in case of DoS attack) or successful
completion and graceful termination of call, at any time
there are only handful of active transition diagrams. Moreover
many of the operations involved in training and testing phase
are not computationally (asymptotically) expensive. Assuming
each transition has fixed memory requirements, the overall
complexity is linear in terms of average number of VoIP calls.
a) INVITE initiated dialog: We define a PCDTA for
modeling INVITE initiated SIP dialogue as follows
Q = {q0 , , q6 , E, S}
q0 is start state
F = {q6 } is the final state
 = {I N V I T E, O K , AC K , 2x x, 3x x, 4x x, 5x x, 6x x, BY E}
{}
T = {1 }
C = {c0 , c6 , c E , c S }
transition set and defined as shown in Table I.1
A simplified representation of state transition corresponding
to single INVITE initiated dialogue is shown in Figure 4. The
machine is initially in state q0 waiting for an INVITE event to
appear with a different call-ID than all the existing dialogues.
When a User Agent Client (UAC) or the caller user agent sends
a new INVITE request (new call-ID), a new instance of the
machine is created and the machine moves to state q1 , labeled
1 Trasitions with indicate either there is no constraint on timing or
resetting clock or updating counter value is not required
GOLAIT AND HUBBALLI: DETECTING ANOMALOUS BEHAVIOR IN VoIP SYSTEMS: A DES MODELING
TABLE I
T RANSITION TABLE FOR INVITE D IALOGUE PCDTA
TABLE II
T RANSITION TABLE FOR REGISTER E VENT PCDTA
as INVITE received state. After receiving the first INVITE the
proxy server may send some redirect response. However for
the sake of brevity we have ignored these cases from modeling
as it does not affect the ultimate detection ability of machine.
Further if authentication is enabled then 401, 407 responses
may be sent to the UAC. Being in INVITE received state
(q1 ) if the proxy server receives any INVITE retransmission
or 401 or 407 responses the machine remains in the same
state.2 If any error message appears (other 4xx, 5xx or 6xx),
the machine goes to Error (E) state from state q1 terminating
the existing dialog.3 If the proxy server successfully accepts
the INVITE request, it forwards the INVITE request to the
corresponding User Agent Server (UAS) or the callee user
agent and sends a provisional TRYING message to the UAC.
When UAS completely understands the INVITE message
it sends a RINGING message to UAC. When the proxy
receives a RINGING message from UAS, the machine makes
a transition to the state q2 which denotes RINGING message
received for that dialogue. If UAS wishes to reject the request
it sends an error response to the proxy server (one of 4xx, 5xx,
6xx) and the state machine moves to the Error (E) state. If UAS
wishes to accept the request it sends a 200 OK response to the
proxy, and machine reaches state q3 . In either case, the proxy
2 Similar retransmission provisions are there at all the states except start and
final state
3 Similar exit option is available from every state other than start and final
state
735
forwards the response to the UAC. If the UAC receives a 200
OK response, it sends an ACK to the proxy and the machine
proceeds to state q4 .
After the previous message sequence the media session
begins. Media session can last for arbitrary time, moreover
it happens in a separate session which we do not model here.
Hence, the proxy waits for a BYE event to occur from either
of the UAC or UAS which indicates end of media session.
When it receives a BYE message, the machine goes to the
state q5 . The other party then sends a corresponding 200 OK
to the proxy, and the machine reaches the final state q6 . The
INVITE initiated call event hence is terminated.
b) REGISTER operation: Similar to the previous case we
define a PCDTA for REGISTER operation also, as below
Q = {q7 , q8 , q9 , E, S}
q7 is the start state
F = {q9 } is the final state
 = {R E G I ST E R, O K , 2x x, 3x x, 4x x, 5x x, 6x x} {}
T = {1 }
C = {c7 , c8 , c9 , c E , c S }
transition set and defined as shown in Table II.
The machine at the registrar is initially at state q7 . As the
registrar receives a REGISTER request from a UAC, the
machine goes to state q8 . If any retransmissions, authentication
requests or redirect requests occur, the machine stays in the
same state. If the request is accepted the registrar responds
with a 200 OK response, and the machine goes to the final
736 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 12, NO. 3, MARCH 2017
state q9 . If any error occurs, the machine goes to the Error (E)
state.
In both Figure 4 and Figure 5 there is a transition from
all the states to a state with label S except start and final
state. This transition is an epsilon () transition triggered
when the delay between the previous event and next event
timing exceeds a threshold set on each of these transitions.
How to learn and set these delay constraints is discussed
subsequently. It should be noted that even  transition happens
deterministically hence does not violate definition. Also for
brevity we do not show transitions from each state for each
input symbol as all such missing transitions should be treated
as illegal transitions.
B. Learning Transition Probability
As defined in previous section each transition probability
indicates the likelihood of system changing the state from qi
to q j upon occurrence of a symbol . This probability
has two components as p(/c) prior probability on input
on present counter value c, and second is probability of
counter taking certain vaue at a state and is denoted with p(c).
We learn these two probabilities by treating these two as
independent. The counter indicates the number of times the
state qi has been visited by the repeated occurrence of events
in the system. Thus the probability of a transition is the product
of p( ) and p(c). Counter here indicates how anomalous it is
to revisit a state with repeated occurrence of an event. Usually,
higher the counter value, lesser is its probability indicating rare
event.
We derive the value of p(c) (the probability value of
counter) by treating it as a discrete random variable and
assigning a probability distribution to it. In particular we use
Poisson distribution [13] calculated using the mean value of
number of times a state has been visited. The probability mass
function of a Poisson Random Variable c taking value n N
is given by Equation 4. In Equation 4, represents mean value
of counter c and e is a Eulers number (e = 2.71828).
e n
p(c = n) =
n!
To learn the transition probability p( ) we use a sequence
of strings of language L  . Let S1 , S2 , , Sm
(with m 1) be a set of sequences from training set (strings of
L) with each string of form Si = 1 , t1 , 2 , t2 , , n , tn 
be a sequence of timed events (indicating input symbols with
their timing of occurrence). To learn the probability of event ,
we use only event sequences (ignoring timing) of transitions.
Each such sequence of events represent a path from start state
to a final state where each edge label is a symbol name. Thus
each internal state of PCDTA represents a prefix of string. For
each edge qi to q j , count the number of such transitions in
all strings. Then the probability of a transition is the ratio of
number of edges going to state q j to the total number of edges
from qi to any other state qk on any symbol . This is
given by the Equation 5.
M
t =1 (qi , q j , )
pi j ( ) =  M  ||
t =1 y=1 (qi , qk , y )
C. Learning Transition Timing Probability
After the previous step of learning, we have a transition
diagram with various events and their respective probability.
In the second step we learn the timing probability on tran-
sitions. In order to do this we use timing of various events
of sequences. For a particular event  of a sequence
sk we deduce the time difference between the previous event
occurrence and current event timings. An example sequence
is shown here Sk = 1 , N A, 2 , et2 , , n , etn . In this
converted sequence the first input symbol does not have a
time delay constraint as it can occur at any time. From
the second symbol onward etl indicates the observed time
delay of symbol r from previous occurrence of event r1 .
After converting all the sequences into this format we find
the mean time delay of a particular event among all the M
sequences of training set. Using this mean value of time dif-
ference we subsequently fit a probability distribution function
(Poisson distribution) which best describes these sequences
of time differences. The probability distribution function will
have its maximum probability around the mean and decrease as
the delay increases which indicates it is more anomalous. The
choice of Poisson distribution is justified by the factors affect-
ing the timing of events. Timings of different events depend on
round trip delay between two machines. This includes many
parameters like network congestion, number of hops traversed
and processing delay by node. These parameters will show
randomness with not a very significant variation. Hence the
time differences will also be random with not much variation
and are best described with Poisson distribution.
VIII. SIP ATTACK D ETECTION
In this section we describe how the PCDTA model is
used to detect various SIP DoS attacks described earlier.
As mentioned in Section III we mainly deal with two types of
SIP attacks, namely flooding and coordinated attacks. The next
two subsections outline how these two attacks can be detected.
(4) A. Detecting SIP Flooding Attacks
In case of flooding DoS attack, a SIP server or proxy
server is targeted with many messages of a particular type. For
example, in case of INVITE flooding large number of INVITE
messages are sent either from a single source or from multiple
sources. Each such message creates an instance of machine,
however these transactions fail to make progress after initial
transition as source is not interested in completing INVITE
dialogue; hence all of these transitions will time out (with a
transition to state S). Thus in order to detect this flooding
attack it is sufficient to count the number of such transactions
terminated from INVITE received state to the time out state
and can be compared with a threshold value (How to select this
threshold is described in Section IX-C). Similar observation
can be made for REGISTER transaction (with valid URI and
authentication enabled), where also large number of timeouts
are seen as malicious user is not interested in completing the
(5) transaction. Algorithm 1 describes the method for detecting
INVITE and REGISTER flooding (with random URI) attacks.
GOLAIT AND HUBBALLI: DETECTING ANOMALOUS BEHAVIOR IN VoIP SYSTEMS: A DES MODELING
Algorithm 1 Detecting INVITE/REGISTER Flooding Attacks
Input: PC DT A - DES model for INVITE and REGISTER
Input: T -Interval Period
Input: Event_T ype- Type of Event to be monitored
Output: Flooding Attack Detection of Event_T ype
1: while Not interrupted do
2: Event_T i meOut = 0
3: for t=tstt est art to t + T do
4: Event New timeout Event of type Event_T ype
Detected from PC DT A
5: Event_T i meOut = Event_T i meOut + 1
6: end for
7: if Event_T i meOut T h then
8: Flooding Attack of Type Event Detected in interval
art to t + T
tstt est
9: end if
10: art t + T
tstt est
11: end while
This algorithm counts the number of such timeout events in a
particular time window T .
In case of BYE flooding, the PCDTA observes illegal transi-
tions from start state to BYE received state or from any other
state to BYE received state (if the call-ID of BYE matches
with any other ongoing call). This attack can be detected by
counting the corresponding number of such illegal transitions
in T interval. In case of REGISTER with random URI
there will be large number of 404 error messages generated as
server does not allow users who do not have accounts at the
registrar to REGISTER. It is easy to notice that, Algorithm 1
can be modified to count such illegal transitions and error
transitions. Similarly in case of multi-attribute flooding all
types of messages appear (with their ratio balanced), hence few
messages will timeout and few others will have illegal state
changes and few may have transitions with error. To detect
these attacks Algorithm 1 can be modified to count respective
type of events (transitions) in T time period.
B. Detecting Coordinated Attacks
During RINGING attacks, the server continues to receive
provisional responses (180 RINGING) but no final response
(200 OK). The repetition of occurrence of any event is taken
care by the counter value associated with the state where the
livelock is made. This attack can be detected by computing
the probability of path taken by event sequence. If RINGING
events are retransmitted then the counter at state q2 has a
high value which results in a lower probability for p(c). Since
probability of event is the product of counter and transition,
this decreases the overall probability rapidly. It might also
happen that rather than repeating the provisional responses, the
attacker might wait for long times (or stops responding) after
sending the RINGING response before sending OK message.
Hence this can also be detected by calculating the path timing
probability.
During prolonged call attacks, the system stalls (almost)
after it has successfully established the session. The call
737
Algorithm 2 Detecting Coordinated Attacks
Input: Trained Model PC DT A
Output: Coordinated Attack Detection for each Observation
Sequence
1: while Not interrupted do
2: if New call-ID then
3: Si Create a new Sequence
4: end if
5: for Every Observation Sequence Si do
6: Ptrans get SequenceTr ansi ti on Pr obabili t y(Si )
7: Pt ime get SequenceT i mi ng Pr obabili t y(Si )
8: if Ptrans T h trans or Pt ime T h t ime then
9: Coordinated Attack Detected for Si
10: end if
11: end for
12: end while
lengths are usually long and a call is ended mostly by the
server timeouts. We try to capture this anomaly in anomalous
path timing delay probability. Thus in order to detect these
attacks, we monitor the path transition probability and timing
delay probability. If any of these is lesser than the threshold
set, we identify the sequence as a coordinated attack attempt.
Algorithm 2 describe these steps.
IX. E XPERIMENTAL E VALUATION
In this section we describe our experiments to validate the
proposed detection techniques. In the next seven subsections
we elaborate on the testbed setup, data collection for training,
how to set different thresholds, detection performance of
PCDTA on various attacks, sensitivity analysis of PCDTA with
different threshold values, comparison with recent work on
SIP flooding detection and comparison with other coordinated
attack detection method respectively.
A. Testbed Setup
In order to study the detection performance of PCDTA, we
designed and created a testbed similar to the one described
in [6] and shown in Figure 6. This testbed simulates two
enterprise networks named A and B. We simulated the two
enterprise networks with two PCs and used another machine
as a server. All the three machines run Ubuntu 12.04 64-bit
operating system with hardware configuration of Intel Core
i5-4590 CPU running at 3.30 GHz having 8 GB RAM.
We installed Asterisk 11.18.0 [14] on the server machine
which acts as a VoIP PBX.4 We also installed a network
emulator software netem [15] in the proxy server to emulate
the Internet behavior. We set up a simulation of 15 user agents
on both networks A and B. These user agents can make calls to
user agents in other network and to agents in the same network.
We used a modified version of VoIP Bot [16] to represent user
agents. It uses Jain-SIP api for handling SIP messages and
4 It stands for VoIP Private Branch Exchange an equivalent of telephone
exchange.
738 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 12, NO. 3, MARCH 2017

TABLE III
M EAN C OUNTER VALUES . (a) C ALL PCDTA.
(b) R EGISTRATION PCDTA

Fig. 6. Testbed setup.

Java Media Framework (JMF) stack for generating RTP media

flows. The original version though uses the communication
agent based on the Internet Relay Chat (IRC) for exchange TABLE IV
of commands between the bots and the bot manager, but our P ROBABILITY FOR T RANSITIONS . (a) T RANSITION P ROBABILITY FOR
version allows the bots to make calls without the help of the INVITE I NITIATED PCDTA. (b) T RANSITION P ROBABILITY FOR REG-
manager. All user agents are registered at the Asterisk server. ISTER I NITIATED PCDTA
Each agent runs as a separate process on a different UDP port.
The Asterisk server runs on port 5060, which is the default
port for a SIP server.

B. Traffic Generation for Training

As described earlier PCDTA transition model is created
with SIP specification and both transition probability, counter
probability and timing probability need to be learned from a
set of successful SIP transactions. To facilitate this learning we
collected successful5 SIP transaction sequences by scheduling
these agents to make calls. While generating this traffic, we set
transmission delay to 50ms and packet loss rate to 0.42 per-
cent through emulator. This emulator randomly drops packets
maintaining this loss ratio and emulates internet behavior. The
user agents were scheduled to make calls at random intervals
for random choices of bots, but closely mimicking the real
behavior. Each user agent made calls with a minimum and
maximum span of 40 and 120 minutes respectively between
successive calls. The network traffic was collected using an
open source traffic collector tcpdump [17]. We wrote a java
program using Jnetpacp [18] library to extract the details of
SIP messages. In total there were 2055 number of SIP INVITE
initiated transactions and 1800 number of REGISTER initiated
of same transaction since they carry same call-ID. From each
dialogues in 5 days of simulation data. We analyzed these SIP
sequence and for each transaction we count number of such
sequences and collected statistics for various parameters to
retransmissions and derive the average of these values over a
learn various probabilities.
period of these 5 days. The mean number of retransmissions
1) Learning Probability of Counter at Various States:
for every state are shown in Table III. As we can see from the
Each of the 2055 number of INVITE transactions are of the
table, number of retransmissions are not more than three for all
form s1 , s2 , s3 , , sn with occasional retransmissions. Each
the states indicating retransmissions are rare. The state q1 has
of these events further the machine to progress to the next state
a slightly higher rate of retransmission, as with authentication
of PCDTA if they are distinct, or otherwise result in a traversal
enabled it is visited twice, once without credentials and second
of self loop at respective states. An example sequence with
time with credentials.
retransmissions looks like s1 , s1 , s2 , s3 , , sn where the first
In order to derive the probability of revisiting a particular
event occur two times. We treat these retransmissions as part
state we fit a Poisson distribution for every state. Figure 7
5 Successful SIP calls are those which successfully establish media session shows the distribution graphs for various counter values at
and gracefully terminate. various states.
GOLAIT AND HUBBALLI: DETECTING ANOMALOUS BEHAVIOR IN VoIP SYSTEMS: A DES MODELING
Fig. 7. Poisson distribution graphs for counters at states from q1
to q6 of INVITE dialogue. (a) Distribution C1. (b) Distribution C2.
(c) Distribution C3. (d) Distribution C4. (e) Distribution C5.
(f) Distribution C6.
2) Learning Transition Probability: As mentioned in
Section VI, transition probability is the ratio of fraction of
transitions to a particular next state to all possible next states
of a particular state. Using this method we derive the transition
probability of all the transitions for both INVITE dialogue
and REGISTER transaction. Table IV and Table IV show the
transition probabilities for various transitions for two PCDTAs
learnt from the transaction sequence. All illegal transitions (not
shown here) have 0 probability.
3) Learning Transition Delay Probability: In order to
learn the timing delay probabilities for various transitions,
we calculated the mean delay between successive events of
the sequences. Using these mean values we again fit a Pois-
son distribution graph for every transition delay probability.
Table V and Table V show the average delay observed between
successive events in the sequences of training set. For the
sake of brevity we do not show the corresponding probability
distribution graphs here (can be drawn by plugging mean delay
value into Poisson distribution equation).
4) Setting Threshold on Timing Constraints for Transitions:
In PCDTA, transitions are constrained by timing of occurrence
and machine will wait for the next event in a particular state
only until this timer expires, upon which an epsilon transition
is activated which will terminate the dialogue with a time
expired message. Setting an appropriate threshold for the time
guard is also learned from the event training sequences. It is
739
TABLE V
M EAN T IMING D ELAY VALUES . (a) T RANSITION D ELAY FOR INVITE
PCDTA. (b) T RANSITION D ELAY FOR REGISTER PCDTA
worth noting that there is no timing constraint from the state q4
(which represents the media transmission is on and waiting for
BYE event), as media transmission can last arbitrary amount
of time and it would be wrong to put a limit on how long
the users can stay in a call. These thresholds are calculated
from the probability distributions of transition timing delays
using Equation 6. In this equation, j is the standard deviation,
t mean is the mean of distribution j , and is an experimental
j
value. We chose to be 100 for our experiment. This value
is derived experimentally so as to minimize false stalled state
detection. We do a sensitivity analysis for different values of
in subsection IX-E.
f (i ) = t mean
j + j (6)
We get these threshold values from the mean values calcu-
lated in the table V. We declare a state qi stalled when the
maximum timing delay threshold amongst all the transitions
from qi is crossed.
C. Setting Threshold for Attack Detection
We generated and used one more days SIP transaction
sequences to derive and set various thresholds for attack
detection. The rational behind this is to use an independently
generated dataset to set appropriate thresholds. As explained
in Section VIII, INVITE and REGISTER flooding attacks
are detected by counting the number of timeout transitions
in a T interval of time, however in training dataset there
were no timeout and illegal transitions. In order to derive
threshold for these cases we generated traffic by randomly
deleting random number of relevant transitions so that timeout
740 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 12, NO. 3, MARCH 2017
TABLE VI
F LOODING T HRESHOLDS
TABLE VII
PATH P ROBABILITY T HRESHOLD VALUES
events are generated at some states. Similarly we also injected
random illegal transitions into this dataset. Using the dataset
so generated we derived mean number of timeout and mean
number of illegal transition thresholds (T h) for detecting
attacks. We used time period T to be 10 minutes. For multi-
attribute flooding detection we need a threshold on both types
of transitions combined. In each case we set these thresholds
to 2 times the average number of such transitions appearing in
this 10 minutes interval approximated to next highest integer
value. Table VI shows the values of these thresholds. In
subsection IX-E we show the attack detection and rate of false
alarm generated with different threshold values.
In order to detect the coordinated attacks we need a thresh-
old on transition sequence probability and transition timing
probability. These two are set at the 2 times of average of all
probabilities of transition and timing probabilities in this days
sequences. Since the probability of a sequence is calculated as
multiplication of probabilities of individual events/transitions,
there is a chance of underflow issue from implementation
perspective hence we converted these calculations into log
values. Table VII shows the thresholds on both transition and
timing thresholds in log scale.
D. Traffic Generation for Testing
We generated two sets of testing data to test the detection
performance of PCDTA. First set of data is normal SIP
operation data and was recorded using the same setup as in
training case. In total there were 150 normal intervals (with
call rate as in training case). Second set of dataset is created
with different attacks generated in separate intervals. The rate
of flooding attacks were increased in step size of 10 samples
per interval starting from 10 to 100 totaling 10 intervals. For
example the INVITE flooding attack was generated with the
first interval having 10 INVITE calls and second interval with
20 INVITE calls upto 100 INVITES in a period of 10 minutes
which add up to 550 INVITE requests. This helps in assessing
the detection capability of PCDTA to different attack rates.
Similar experiments were repeated for all other cases too.
We generated these flooding attacks using custom scripts as
detailed below.
1) Flooding Attacks: The flooding attacks were gener-
ated through Python scripts which used custom text of
INVITE, REGISTER and BYE SIP messages and invoked
Scapy 2.3.1 [19] to send the messages to SIP server using
UDP as transport protocol.
INVITE flooding: The script sends any required number
of INVITE messages to valid/invalid users, but ignores
any responses received from the server or the other user.
REGISTER flooding: We generated these attacks taking
into consideration whether authentication is enabled at
the registrar or disabled.
a. When authentication is enabled, the script sends
REGISTER requests as invalid users, which results in
403 (Forbidden) /404 (Not Found) error responses at the
server.
b. When authentication is disabled, the script sends
REGISTER requests to the registrar, but ignores any
challenge (called nonce) issued by the server to verify
his identity. This results in stalled sequences.
BYE flooding: These attacks were generated in two ways
as follows.
a. The script sends BYE messages with random call-IDs
to the server, which result in 481 (Transaction Does Not
Exist) error responses.
b. The script sends BYE messages for the ongoing
communications, hence terminating legitimate calls. The
ongoing call information was written to a text file by a
packet sniffer cum analyzer written using jnetpacp library.
As the user agent doesnt send any BYE request, the
reception of a corresponding OK is detected as an illegal
transition sequence at the proxy server of other user agent
side (UAS or callees proxy server).
Using the testing dataset sequences we again calculated
the number of timeout, illegal transitions by setting time
window to T to 10 minutes to detect flooding attacks.
The details of detection performance of PCDTA is shown in
Table VIII. We use Recall and Accuracy of detection system
the two commonly used metrics to assess the performance of
PCDTA. Recall is the effectiveness of an algorithm to identify
positive labels. Accuracy is the overall effectiveness of an
algorithm. Recall and Accuracy are given by the Equation 7
and Equation 8. In these equations the terms carry following
meaning t p = number of true flooding intervals detected,
tn = number of normal intervals, f p = number of false
flooding attack intervals detected, f n = number of attack
intervals labeled as normal.
tp
Recall = 100 (7)
tp + f n
t p + tn
Accur acy = 100 (8)
t p + tn + f n + f p
2) Coordinated Attacks: The VoIP Bot [16] program itself
was modified to generate these attacks.
RINGING attacks: The program now allows bots to
send repetitive RINGING responses (minimum number
being 3 and maximum being 10); and to send delayed OK
responses (with a delay of 4 to 5 minutes) after sending
RINGING responses.
GOLAIT AND HUBBALLI: DETECTING ANOMALOUS BEHAVIOR IN VoIP SYSTEMS: A DES MODELING
TABLE VIII
F LOODING ATTACK D ETECTION R ESULTS
TABLE IX
C OORDINATED ATTACK D ETECTION R ESULTS
Prolonged call attacks: The bots could stay in call now
for unusually long times. The call times allowed were
between 20 to 70 minutes, which were exceptionally
long for normal users. The average call time for training
set was 10 minutes in our experiments (generated by
transmitting recorded media files of user conversations).
For coordinated attack detection we selected 500 nor-
mal sequences (from the testing dataset) and mixed with
sequences of coordinated attacks. Table IX shows the number
of sequences, Recall and Accuracy of PCDTA. Unlike flooding
cases, here the true positives, true negatives, false positives and
false negatives are counted for number of sequences rather
than number of intervals as coordinated attack is detected for
every sequence.
E. Threshold Parameter Sensitivity Analysis
As mentioned previously we identify timeout transitions by
multiplying to the standard deviation of transition timings
and adding it to the mean of timing values of events. Hence the
value of affects the detected number of stalled states. Thus
it is very essential to choose its value such that we minimize
the number of falsely identified stalled states. We performed
sensitivity analysis on different values of starting from
10 to 200 using the average number of stalled state counts
in a window period from training sequences. We used 3 types
of dataset to study the sensitivity of false detection to values.
First is 950 normal INVITE sequences (newly generated),
entire dataset used for flooding attack case (Table VIII which
has both normal and different flooding scenarios) and coor-
dinated attack dataset (Table IX). The results of percentage
of falsely identified stalled states6 when tested with these
3 types of datasets are shown in Table X. We can notice that
the number of falsely identified stalled states lessen as we
reach 100. After 100, it showed no improvement hence we
used the value of as 100.
6 These are not false alarms. False alarms are generated when an interval
has threshold number of timeouts
741
As mentioned previously in order to detect flooding attacks
we count the number of timeout transitions or illegal transi-
tions in a window period and to detect coordinated attacks
we use path transition probability (per transition sequence).
It is worth noting that, the detection performance of PCDTA
is governed by the threshold on these two values.
1) For flooding detection, a threshold value which is too
conservative may detect all flooding attack instances
however it may generate too many false detection cases.
On the other hand a threshold which is too large may
miss many genuine flooding intervals. We experimented
with different threshold multipliers using the flooding
dataset (entire dataset used for Table VIII) and chosen a
threshold value of 2 balancing the recall and false alarm
rate as shown in Table XI.
2) For detecting coordinated attacks the threshold mul-
tiplier is set for the transition sequence probability
directly, which keeps on decreasing as more number of
repeated transitions are observed. Thus any sequence
probability which is lower than the threshold proba-
bility is detected as attack (since values less than 1
get multiplied). In order to set an appropriate value
which minimizes the chances of false alarms we experi-
mented with different values for the threshold multipliers
ranging from 1 to 3 in step size of 0.5. Table XII
shows the detection performance and false alarm rate
generated by PCDTA with different threshold values (it
is a multiplier of average number of such transitions
in normal intervals) when tested with 1000 instances of
normal sequences and 425 instances of each coordinated
attack type (as in Table IX). We can notice that for
a threshold multiplier of 2 we get 100% recall and
acceptable false alarm rate (4 + 9)/(1000)100 = 1.3%.
F. Comparison of PCDTA With Other
Flooding Detection Method
In this subsection we report the comparison of PCDTA with
one of recent work on flooding attack detection method in
742 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 12, NO. 3, MARCH 2017
TABLE X
S ENSITIVITY OF FALSE S TALLED S TATE D ETECTION TO VALUES
TABLE XI
P ERFORMANCE OF PCDTA W ITH D IFFERENT T HRESHOLDS FOR
T IMEOUT /I LLEGAL T RANSITION C OUNT
TABLE XII
P ERFORMANCE OF PCDTA W ITH D IFFERENT T HRESHOLDS FOR PATH
P ROBABILITY
TABLE XIII
F LOODING ATTACK D ETECTION C OMPARISON W ITH [20]
prior art. For comparison purpose we implemented a method
proposed in [20] and evaluated its performance on our dataset.
The method described in [20] generates statistics of various
SIP messages grouped on the basis of call-ID, ToUri and IP
address of source. Statistics so generated in a block period
will be compared with two thresholds. Any interval traffic
statistics not lying within this range is declared a flooding
attempt. Thresholds are derived from the normal sequences
data. The maximum and minimum values of the number of
packets in a group in a particular interval become the two
thresholds. These thresholds get updated after every normal
traffic interval is observed. We set the interval here to be
10 minutes, same as the interval we chose for our case. For
most of the cases, the two thresholds came out to be 2 and 5.
Two thresholds are set for this case because, if an attacker
attempts flooding, the number of packets sent from her side
with a unique combination of parameters will be less than
that sent by a normal caller provided she uses different SIP
parameters for different attack instances (attack strategy 1),
in a second method attacker can use the same fields for the
SIP packet to generate the attacks (attack strategy 2), like
repeating the same IP, call-ID, and ToUri, this time, number
of the packets in same group can go beyond than that for
the normal user. Table XIII shows a comparison of recall and
TABLE XIV
D ETECTION F EATURE C OMPARISON
Fig. 8. Random early termination.
false alarm rate for our method and that of [20]. False attacks
were detected using this method [20] mainly in two cases, first
when the legitimate call could not be completed because the
other user was busy and other, when the number of packets
in the normal sequences exceeded the maximum threshold,
which may occur in case of retransmissions.
In literature there are other works based on Machine
learning and anomaly detection techniques to detect flooding
attacks. An advantage of PCDTA over other techniques is, it
is generic and can detect both flooding and coordinated types
of DoS attacks, whereas the methods proposed in literature are
for specific attack types. Machine learning technique require
large number of features to be extracted and training with
each attack type which is often difficult. Hellinger distance
based technique [6] use the correlation between different SIP
messages for detecting attack, while this may be evaded by
balancing the number of these messages cleverly. PCDTA
doesnt need training with any attack type and can detect
all cases of DoS. A comparison on detection capability of
different methods is shown in Table XIV.
G. Comparison of PCDTA With Random
Early Termination Method
We also compare our method to a technique called Random
Early Termination proposed in [2]. Random Early Termination
is a technique to terminate connections which are in the
RINGING stage to avoid resource crunch at the proxy server.
It selects active sessions probabilistically for termination based
on the age of session in RINGING phase.
All the active sessions or transactions which are in RING-
ING stage are sorted based on their starting time and two
thresholds T1 and T2 are used for identifying calls which
are to be terminated. Most recent T1 number of calls will
not be considered for termination (white area in Figure 8).
This is done to avoid terminating any call which has just
been initiated. All calls which are older than T1 but less than
T2 will be dropped with probability pr ob which is given by
Equation 9 (grey area in Figure 8). If the number of active
RINGING transactions are greater than T2 all transactions
whose sequence number is older than T2 (black area in
Figure 8) will be definitely terminated.
ageM RT T
pr ob = 1 e M RT T (9)
GOLAIT AND HUBBALLI: DETECTING ANOMALOUS BEHAVIOR IN VoIP SYSTEMS: A DES MODELING
In equation 9 MRTT is the Minimum Ringing Time Thresh-
old and age is the duration of RINGING time. MRTT is the
threshold time on the age of RINGING call below which it is
not considered for termination (and is done regardless of num-
ber of active transactions). We conducted experiments similar
to the one described in [2]. We generated 900 normal calls and
also Ringing attacks in a span of 25 minutes (which is a higher
rate of calls compared to our previous experiments). Normal
and Ringing attacks are generated using bots as described
previously. Table XV shows the performance comparison of
PCDTA with RET method. In case of RET the number of
calls terminated by it are considered to calculate Recall and
Accuracy. For these experiments we set threshold values
T1= 18 and T2= 25 calls and also MRTT to 18 seconds in
case of RET and same path probability threshold (15.2737 in
log scale) used for PCDTA in our previous experiment. These
thresholds are computed after observing the performance of
RET method for different values. The value of MRTT is set
to 18 seconds as normal calls in the dataset we generated had
Ringing duration ranging between 5 seconds to 18 plus few
fraction of seconds. We can notice that PCDTA outperforms
RET in all cases. The lowered Recall and Accuracy of RET is
because it does not terminate any call even if its age is greater
than MRTT if the number of concurrent transactions does not
cross T1 and it may also terminate some normal calls if any
normal calls ringing time is greater than MRTT.
In order to access the performance of RET against different
threshold values and MRTT values we performed sensitivity
analysis using different values. Table XVI shows the detection
performance of RET with MRTT set at 18 seconds and for
different threshold values of T1 and T2. We can see that as
the threshold values are decreased more number of ringing
sequences are detected. Similarly in the second experiment we
set the threshold values T1 and T2 at 18 and 25 respectively
(as these thresholds detected maximum number of ringing
sequences in previous case) and varied the MRTT value
between 12 to 18 seconds in step size of 3 seconds. Table XVII
shows the performance of RET with these values. We can
again observe that lower MRTT value can detect many attacks
as in this case more number of transactions qualify to be
counted for the thresholds T1 and T2. However as many
normal sequences also have ringing time greater than MRTT
and are likely to be terminated by probabilistic selection which
increases the false alarm rate.
X. P RIOR W ORK
In this section we describe prior work related to SIP flooding
and anomaly detection. A very brief discussion about most
closely related work in discrete event systems used to detect
anomalies in other domains is also given here.
A. Voice Over IP Denial of Service
To protect against SIP flooding attacks, there are prior works
which can be classified as preventive methods (which try
to secure SIP itself) and detection methods which identify
flooding cases by monitoring network traffic.
743
1) Preventive Techniques: SIP specification does not sug-
gest any particular security mechanism; instead it allows the
use of other security mechanisms like TLS and SMIME for
securing SIP messages [26]. Preventive methods are mainly in
the form of cryptographic extensions to secure the SIP [27].
These techniques authenticate messages exchanged between
user agents and SIP servers preventing malicious users from
generating spoofed requests.
2) Detection Techniques: Detection methods broadly fall
under four cases as, signature based, statistical methods,
machine learning based and formal models.
a) Signature based detection: Signature based
approaches have encoded patterns for different types of
attacks, and the detection system systematically scan the
incoming traffic for these patterns. Works in [28][30] propose
signatures for detecting DoS caused due to maliciously formed
SIP messages. These patterns are based on SIP grammar as
defined in RFC 3261 [4]. However, these approaches can
only detect the previously encoded attacks. A framework
to describe known vulnerability in SIP and preventing such
vulnerabilities is described in [31].
b) Statistical methods: Statistical deviation detection
approaches attempt to detect abnormalities in the incoming
messages by observing significant deviation from the normal
behavior. Many researchers [6], [24], [25] have proposed to
use hellinger distance (HD) to measure aberration between
normal and attack traffic distribution. Since these approaches
identify the difference in the occurrence count of events of
various types they may sometimes generate false alarms or
fail to detect some attacks. For example an attacker can craft
a flooding attack balancing the different types of messages
(INVITE, REGISTER, BYE, OK, ACK) such that there are
no differences in distributions (Multi-Attribute flooding) com-
pared to normal scenario, rendering few detection methods
ineffective. Reynolds and Ghoshal [5] proposed to measure
the difference between the number of attempted connection
establishments and the number of completed connections. This
is motivated by the fact that, in flooding cases there would be
many call setup requests which are not completed. A signal
processing technique which observes the change in energy
level of a wavelet to detect slow rate SIP floods is described
in [32].
c) Machine learning methods: There are several attempts
to use machine learning methods as anomaly detector for
VoIP flooding. Nassar et al. [23] proposed a machine learning
approach to classify and detect SIP traffic and also different
flooding attacks. They used 38 features extracted as statistics
of various message types and intervals to train a SVM. Akbar
and Farooq [21] evaluated two machine learning algorithms
(Naive Bayes and Decision Tree) using a set of features
extracted from a window period to detect flooding attacks.
These features are extracted as statistics directly from first
line of SIP packets. Tsiatsikas et al. [22] proposed to generate
a hash value from first line of SIP packet and count number of
unique hash values in a time window as feature. Similar to [21]
it used different machine learning algorithms (SMO, Naive
Bayes, Neural Network, Decision Tree and Random Forest
classifiers) to detect attacks of different rates. Mehta et al. [33]
744 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 12, NO. 3, MARCH 2017

TABLE XV
C OMPARISON OF PCDTA W ITH R ANDOM E ARLY T ERMINATION M ETHOD

TABLE XVI
S ENSITIVITY A NALYSIS OF R ANDOM E ARLY T ERMINATION M ETHOD W ITH T HRESHOLDS

TABLE XVII
S ENSITIVITY A NALYSIS OF R ANDOM E ARLY T ERMINATION M ETHOD W ITH MRTT

have pointed various inefficiencies of machine learning tech-
niques in detecting attacks and high degree of false alarms
particularly of those methods using Euclidean distance.
d) Formal models: Many prior works have used formal
methods to study and analyze the behavior of SIP protocol.
Works described in [34] and [35] model the SIP INVITE
transactions using Coloured Petri Nets (CPNs) over reliable
and unreliable transport medium. They examine the effect of
various losses in events and corresponding state space models
to find defects in SIPs implementation. Another work [36]
utilizes CPN state space analysis to identify the states in
SIP INVITE transaction which are vulnerable to Denial of
Service attacks. The standard INVITE server transition model
described in RFC 3261 is annotated with state constraints
in [37] to detect flooding attacks. Work described in [38]
proposed an extended CPN model known as the timed Hier-
archical CPN (HCPN) as an intrusion detection system for
detecting SIP post-session and pre-session flooding attacks.
One major limitation of these techniques is they either verify
the SIP protocol behavior or mainly focus on detecting only
the INVITE flooding attacks.
Some recent surveys on various SIP vulnerabilities and their
detection methods can be found in [3] and [39].
B. Discrete Event Systems
Discrete Event System modeling has been extensively used
to formally describe various types of failures in control sys-
tems [40]. Klerx et al. [7] described a probabilistic automata
to identify anomalies in discrete event systems. In particular
they consider ATM machine operation as a discrete event
system and describe range of anomalies found in discrete
event systems. A probabilistic learning automata is used to
characterize the behavior of system and detect these anomalies.
A discrete event system based model is proposed for detecting
faults in powerline networks [41]. X10 powerline network
system is used as a case study and based on commands
exchanged an automaton is generated which represents the
normal behavior profile and subsequently used to detect
anomalies.
XI. C ONCLUSION
Session Initiation Protocol has become the defacto standard
for session management in Voice over IP implementations. SIP
is vulnerable to a range of Denial of Service attacks including
flooding and coordinated attacks. In this paper we modeled
different SIP dialogues and transactions as discrete event sys-
tems and proposed a probabilistic state transition machine to
describe these dialogues and transactions. Further we identified
a range of anomalies generated in a DES system. We described
algorithms to detect various DoS attacks using the proposed
state transition model. We designed and experimented with a
range of DoS attacks generated through custom programs and
report that proposed DES model can detect these attacks with
high accuracy and detection rate.
R EFERENCES
[1] G. Ormazabal, S. Nagpal, E. Yardeni, and H. Schulzrinne, Secure SIP:
A scalable prevention mechanism for DoS attacks on SIP based VoIP
systems, in Proc. Principles, Syst. Appl. IP Telecommun. Serv. Secur.
Next Generat. Netw., 2008, pp. 107132.
[2] W. Conner and K. Nahrstedt, Protecting SIP proxy servers from
ringing-based denial-of-service attacks, in Proc. 10th IEEE Int. Symp.
Multimedia (ISM), Dec. 2008, pp. 340347.
[3] A. D. Keromytis, A comprehensive survey of voice over IP security
research, IEEE Commun. Surveys Tut., vol. 14, no. 2, pp. 514537,
2nd Quart. 2012.
[4] J. Rosenberg et al., SIP: Session Initiation Protocol,
document RFC 3261, 2002.
GOLAIT AND HUBBALLI: DETECTING ANOMALOUS BEHAVIOR IN VoIP SYSTEMS: A DES MODELING
[5] B. Reynolds and D. Ghoshal, Secure IP telephony using multi-
layered protection, in Proc. 10th Annu. Netw. Distrib. Syst. Secur.
Symp. (NDSS), 2003, pp. 113.
[6] H. Sengar, H. Wang, D. Wijesekera, and S. Jajodia, Detecting VoIP
floods using the Hellinger distance, IEEE Trans. Parallel Distrib. Syst.,
vol. 19, no. 6, pp. 794805, Jun. 2008.
[7] T. Klerx, M. Anderka, H. K. Bning, and S. Priesterjahn, Model-based
anomaly detection for discrete event systems, in Proc. IEEE 26th Int.
Conf. Tools Artif. Intell. (ICTAI), Nov. 2014, pp. 665672.
[8] R. Alur and D. L. Dill, A theory of timed automata, Theory Comput.
Scince, vol. 126, no. 2, pp. 183235, 1994.
[9] C. Meiners, E. Norige, A. X. Liu, and E. Torng, FlowSifter: A counting
automata approach to layer 7 field extraction for deep flow inspection,
in Proc. IEEE INFOCOM, Mar. 2012, pp. 17461754.
[10] R. C. Carrasco and J. Oncina, Learning stochastic regular grammars by [34] L. Ding and L. Liu, Modelling and analysis of the INVITE transaction
means of a state merging method, in Proc. Int. Colloq. Grammatical
Inference Appl., 1994, pp. 139152.
[11] D. D. Ron, Y. Singer, and N. Tishby, On the learnability and usage of [35] L. Liu, Verification of the SIP transaction using coloured Petri Nets,
acyclic probabilistic finite automata, in Proc. 8th Annu. Conf. Comput.
Learn. Theory, 1995, pp. 3140.
[12] C. De La Higuera, Grammatical Inference: Learning Automata and
Grammars. New York, NY, USA: Cambridge Univ. Press, 2010.
[13] S. M. Ross, A First Course in Probability, 8th ed. New York, NY, USA: [37] D. Seo, H. Lee, and E. Nuwere, SIPAD: SIP-VoIP anomaly detec-
Prentice-Hall, 2010.
[14] [Online]. Available: http://www.asterisk.org/
[15] [Online]. Available: https://wiki.linuxfoundation.org/networking/netemNetem[38] Y. Ding and G. Su, Intrusion detection system for signal based SIP
[16] M. Nassar, R. State, and O. Festor, Labeled VoIP data-set for intrusion
detection evaluation, in Proc. 16th EUNICE/IFIP Conf. Netw. Serv.
Appl. Eng. Control Manage., 2010, pp. 97106.
[17] [Online]. Available: http://www.tcpdump.org/Tcpdump
[18] [Online]. Available: http://jnetpcap.com/jNetPcap
[19] [Online]. Available: http://www.secdev.org/projects/scapy/Scapy
[20] J. Lee, K. Cho, C. Lee, and S. Kim, VoIP-aware network attack
detection based on statistics and behavior of SIP traffic, Peer-to-Peer
Netw. Appl., vol. 8, no. 5, pp. 872880, 2015.
[21] M. A. Akbar and M. Farooq, Securing SIP-based VoIP infrastructure
against flooding attacks and spam over IP telephony, J. Knowl. Inf.
Syst., vol. 38, no. 2, pp. 491510, 2014.
[22] Z. Tsiatsikas, A. Fakis, D. Papamartzivanos, D. Geneiatakis,
G. Kambourakis, and C. Kolias, Battling against DDoS in SIP. is
machine learning-based detection an effective weapon? in Proc. 12th
Int. Conf. Secur. Cryptogr. (SECRYPT), 2015, pp. 301308.
[23] M. Nassar, R. State, and O. Festor, Monitoring SIP traffic using
support vector machines, in Proc. 11th Int. Symp. Recent Adv. Intrusion
Detection (RAID), 2008, pp. 311330.
[24] J. Tang, Y. Cheng, Y. Hao, and W. Song, SIP flooding attack detection
with a multi-dimensional sketch design, IEEE Trans. Depend. Sec.
Comput., vol. 11, no. 6, pp. 582595, Nov/Dec. 2014.
[25] J. Tang, Y. Cheng, and C. Zhou, Sketch-based SIP flooding detection
using Hellinger distance, in Proc. 28th IEEE Conf. Global Telecommun.
(GLOBECOM), Nov. 2009, pp. 16.
[26] D. Geneiatakis, G. Kambourakis, T. Dagiuklas, C. Lambrinoudakis, and
S. Gritzalis, SIP security mechanisms: A state-of-the-art review, in
Proc. 5th Int. Netw. Conf. (INC), 2005, pp. 147155.
[27] R. Farley and X. Wang, VoIP Shield: A transparent protection of
deployed VoIP systems from SIP-based exploits, in Proc. IEEE Netw.
Oper. Manage. Symp. (NOMS), Apr. 2012, pp. 486489.
[28] D. Geneiatakis, G. Kambourakis, C. Lambrinoudakis, T. Dagiuklas,
and S. Gritzalis, A framework for protecting a sip-based infrastructure
against malformed message attacks, Comput. Netw., vol. 51, no. 10,
pp. 25802593, 2006.
[29] S. Ehlert et al., Two layer denial of service prevention on SIP VoIP
infrastructures, Comput. Commun., vol. 31, no. 10, pp. 24432456,
2008.
745
[30] A. Lahmadi and O. Festor, VeTo: An exploit prevention language from
known vulnerabilities in SIP services, in Proc. Netw. Oper. Manage.
Symp. (NOMS), 2010, pp. 216223.
[31] A. Lahmadi and O. Festor, A framework for automated exploit preven-
tion from known vulnerabilities in voice over IP services, IEEE Trans.
Netw. Service Manage., vol. 9, no. 2, pp. 114127, Jun. 2012.
[32] J. Tang and Y. Cheng, Quick detection of stealthy SIP flooding attacks
in VoIP networks, in Proc. IEEE Int. Conf. Commun. (ICC), Jun. 2011,
pp. 15.
[33] A. Mehta, N. Hantehzadeh, V. K. Gurbani, T. K. Ho, J. Koshiko,
and R. Viswanathan, On the inefficacy of Euclidean classifiers for
detecting self-similar session initiation protocol (SIP) messages, in
Proc. 12th IFIP/IEEE Int. Symp. Integr. Netw. Manage. (IM), May 2011,
pp. 329336.
of the session initiation protocol using coloured Petri Nets, in Proc.
29th Int. Conf. Appl. Theory Petri Nets, 2008, pp. 132151.
in Proc. 32nd Austral. Comput. Sci. Conf., 2009, pp. 6372.
[36] L. Liu, Uncovering SIP vulnerabilities to DoS attacks using coloured
Petri Nets, in Proc. 10th IEEE Int. Conf. Trust, Secur. Privacy Comput.
Commun., Nov. 2011, pp. 2936.
tion using a stateful rule tree, Comput. Commun., vol. 36, no. 3,
pp. 562574, Mar. 2013.
attacks through timed HCPN, in Proc. 2nd Int. Conf. Availability, Rel.
Secur. (ARES), 2007, pp. 190197.
[39] S. Ehlert, D. Geneiatakis, and T. Magedanz, Survey of network security
systems to counter SIP-based denial-of-service attacks, Comput. Secur.,
vol. 29, no. 1, pp. 225243, 2010.
[40] C. G. Cassandras and S. Lafortune, Introduction to Discrete Event
Systems. New York, NY, USA: Springer, 2008.
[41] A. Arora, R. Jagannathan, and Y.-M. Wang, Model-based fault detection
in powerline networking, in Proc. 16th Int. Parallel Distrib. Process.
Symp. (IPDPS), 2002, pp. 18.
Diksha Golait was born in Bhopal, India. She
received the B. Tech. degree in computer science
engineering from IIT Indore, India, in 2016. In
July 2016, she joined Microsoft India (Research
and Development), where she currently pursues a
career in software development. Her research inter-
ests include network and system security.
Neminath Hubballi received the Ph.D. degree from
the Department of Computer Science and Engineer-
ing, IIT Guwahati, India. He was with corporate
Research and Development Centers of Samsung,
Infosys Lab and Hewlett-Packard. He is currently
an Assistant Professor in computer science with IIT
Indore, India. He has authored or coauthored in the
area of security. He is also a regular reviewer in
many security journals and conferences and also
served as a TPC member of several conferences.
His areas of interest include network and system
security.