Está en la página 1de 15

NEAR EAST UNIVERSITY

GRADUATE SCHOOL OF APPLIED SCIENCE

ALGORITHM DESIGN AND ANALYSIS

Assignment I: Big O notation, algorithm efficiency and its examples

Fitehalew Ashagrie Demilew…………………20176813

Habtamu Girma Debebe...…………………….20176801

Ass. Prof Dr. BORAN ŞEKEROĞLU

Nicosia, Cyprus

Wednesday, July 25, 2018

1|Page
Contents
List of figures ................................................................................................................................................. 2
Big O notation ............................................................................................................................................... 3
Efficiency of Algorithm .............................................................................................................................. 4
Sorting Algorithms and Big-O Analysis...................................................................................................... 6
Bubble Sort............................................................................................................................................ 9
Selection Sort ...................................................................................................................................... 10
Merge Sort .......................................................................................................................................... 11
References .................................................................................................................................................. 15

List of figures
Figure 1: f(n)=O(g(n)) graph .......................................................................................................................... 4
Figure 2: Big-O Complexity Chart .................................................................................................................. 9
Figure 3: Recursively dividing the problem................................................................................................. 11
Figure 4: Common Data Structure Operations ........................................................................................... 13
Figure 5: Array Sorting Algorithms.............................................................................................................. 14

2|Page
Big O notation
It is a way to determine the efficiency of a program, or more specific, an algorithm. In most cases, it is
about how efficient the process time is. But an algorithm analysis is a fundamental computer science
topic, algorithm analysis does not only involve time efficiency, and it involves space (memory) efficiency
as well. But Big O is all about the time efficiency in most cases (it can represent the space complexity). So
in short Big O notation is used to describe or calculate time complexity (worst-case performance) of an
algorithm

Many “operations” in real life can help us with finding what the order is. When analyzing
algorithms/operations, we often consider the worst-case scenario. What’s the worst that can happen to
our algorithm and when does our algorithm need the most instructions to complete the execution? Big O
notation will always assume the upper limit where the algorithm will perform the maximum number of
iterations.

Example. Let’s assume I am standing in the front of a class of students and one of them has my bag. Here
are few scenarios and ways in which I can find my bag and their corresponding order of notation.

O(n) — Linear Time:

Scenario: Only one student in my class who hid my bag knows about it.
Approach: I will have to ask each student individually in the class if they have my bag. If they don’t, I move
on to ask the next student.
Worst-Case Scenario: In the worst case scenario, I will have to ask n questions.

A theoretical measure of the execution of an algorithm, usually the time or memory needed, given the
problem size n, which is usually the number of items. Informally, saying some equation f(n) = O(g(n))
means it is less than some constant multiple of g(n). The notation is read, "f of n is big oh of g of n".

Formal Definition: f(n) = O(g(n)) means there are positive constants c and k, such that 0 ≤ f(n) ≤ cg(n) for
all n ≥ k. The values of c and k must be fixed for the function f and must not depend on n.

Also known as O, asymptotic upper bound.

3|Page
Figure 1: f(n)=O(g(n)) graph

The importance of this measure can be seen in trying to decide whether an algorithm is adequate, but
may just need a better implementation, or the algorithm will always be too slow on a big enough input.
For instance, quicksort, which is O(n log n) on average, running on a small desktop computer can beat
bubble sort, which is O(n²), running on a supercomputer if there are a lot of numbers to sort. To sort
1,000,000 numbers, the quicksort takes 20,000,000 steps on average, while the bubble sort takes
1,000,000,000,000 steps!

Any measure of execution must implicitly or explicitly refer to some computation model. Usually this is
some notion of the limiting factor. For one problem or machine, the number of floating point
multiplications may be the limiting factor, while for another, it may be the number of messages passed
across a network. Other measures that may be important are compares, item moves, disk accesses,
memory used, or elapsed ("wall clock") time.

Efficiency of Algorithm

When solving a problem there may be several suitable algorithms are available. Obviously choose the
best. How to decide? If we had one or two instances (may be small too), simply choose the easiest to
program or use the one for which a program exists. Otherwise, we have to choose very carefully.

The empirical (or a posteriori) approach to choosing an algorithm consists of programming the computing
techniques and trying them on different instances with the help of a computer. The theoretical (or a priori)
approach, which we favor consists of determining mathematically the quantity of resources needed by
each algorithm as function of the size of the instances considered. The resources of most interest are:
computing time (most critical) and storage space. Throughout the class when we speak of the efficiency

4|Page
of an algorithm, we shall mean how fast it runs. Occasionally, we'll also be interested in an algorithm's
storage requirements or any other resources (like number of processors required by a parallel algorithm).

The size of an instance corresponds formally to the number of bits needed to represent the instance on a
computer. In our analysis, we'll be less formal, the use of "size" to mean any integer that in some way
measures the number of components in an instance.

Examples:

 In sorting size is the number of elements to be sorted.


 In graphs size is the number of nodes or edges or both.

The advantage of the theoretical approach is that it doesn't depend on the computer used, nor the
programming language, nor the skill of the programmer. There is the hybrid approach where efficiency is
determined theoretically, then any required numerical parameters are determined empirically for a
particular program and a machine.

Bits are a natural way to measure storage. We want to be able to measure efficiency in terms of time it
takes to arrive at an answer. There is no such obvious choice. An answer is given by the principle of
invariance, which states that: "Two different implementations of the same algorithm will not differ in
efficiency by more than a multiplicative constant." Different implementations mean different machines
or programming language.
If two implementations of the same algorithm take t1 (n) and t2 (n) seconds
respectively, to solve an instance of size n, then there exists always
positive constants c and d such that:
t1(n)≤ct2(n)and t2(n) ≤dt1(n) whenever n is sufficiently large.

In other words the running time of either implementation is bounded by a constant multiple of the
running time of the other. Thus a change of machine may allow us to solve a problem 10 times faster or
100 times faster, giving an increase in speed by a constant factor. A change of algorithm on the other hand
may give us an improvement that gets more and more marked as the size of the instances increases.

The principle allows us to decide that there will be no such unit. Instead, we only express the time taken
by an algorithm to within a multiplicative constant. We say: "An algorithm for some problem takes a time
in the order of t(n for a given function t, if there exists a positive constant c, and an implementation of the
algorithm capable of solving every instance of size n in not more than ct(n) (seconds) arbitrary time units.

5|Page
Units are irrelevant ct(n) seconds, bt(n) years …
Certain orders are so frequent that we give them names:
 cn: Algorithm in the order of n and called linear algorithm
 cn2: Algorithm in the order of n2 and called quadratic
 nk: Algorithm in the order of nk and called polynomial
 cn : Algorithm in the order of cn and called exponential

Sorting Algorithms and Big-O Analysis

A sorting algorithm is one that takes an unordered list and returns it ordered. Various orderings can be
used but for this assignment we will focus on numeric ordering, i.e. 5,3,4,1,2 => 1,2,3,4,5. We are going
to present 3 different sorting algorithms, walk through how they work and compare their efficiency using
Big-O notation — which we will briefly explain. All our examples will be in Python.

All of these algorithms fundamentally do the same thing and the practical difference between them is in
their performance. We will describe their performance using Big-O notation. Big-O, describe the
performance of an algorithm by estimating the number of operations required as the size of the input
approaches infinity.

The notation format is O(g(n)) for Big O. g(n) represents the complexity of algorithm f(n) and indicates to
us how an algorithm’s complexity will scale as the input increases. An example of Big-O notation might
look like O(n²). This tells us that the algorithm’s complexity scales in quadratic time, or in other words that
the number of operations is at most n*n.

The exact computational complexity of an algorithm will always vary with the subtleties of its input—for
example feeding a sorted list into an efficient sorting algorithm should take a lot less work to process then
feeding in an unsorted list of the same size. With Big-whatever notation we make generalized statements
about an algorithms performance like ‘this algorithm is at least this complex or this algorithm is at most
that complex.’ To represent these different claims we use Big-O, Big-Omega, or Big-Theta. Big-O looks at
the maximum number of operations that will be required, Big-Omega looks at the minimum, and Big-
Theta looks at both boundaries—or in more technical terms Big-Theta shows how tightly bound the
algorithm is, but in this paper we will only discus about Big O.

With Big-O (and Omega/Theta) we are making approximations. We disregard all operations in the
algorithm but the one that scales the least efficiently. We are able to do this because as the input increases

6|Page
and approaches infinitely large, the computationally complex parts of the algorithm will drastically
outpace the less complex.

For example, imagine we had a function that loops through an array, checking if each element in the array
matches a value and then returning the number of elements matching that value after the loop. The
number of operations required to complete the for loop will increase linearly with the size of the input
array (O(n)), and printing the sum will only needs to happen once (O(1)). Assuming an array with a size n,
the loop is size n operations whereas the sum is always 1 operation. If we were assuming an input of n=1
than the sum operation would be very substantial. However, in Big-O we are assuming extremely large
inputs approaching infinity. In that situation the sum is inconsequential in comparison to the n-sized loop
and can be safely ignored.

Similarly, if there were a portion of the algorithm that required n² operations — such as a double for loop
where you iterate through the array and compare each element to all other elements — and then a final
single O(n) size loop through the array, we would be able to safely disregard the O(n) portion of the
algorithm for our analysis and call the algorithm O(n²). Even if the input was something small like n=1000,
the O(n²) portion would 10,000 times more operations then the O(n) portion.

We are going to focus on Big-O and ignore Omega and Theta we’ve given a couple examples of Big-O
notation already but here are a few more common ones with simple examples:

O(1)
def func(arr):
return arr[0]

This function returns the first element of the input array and will always take a single operation to
complete, regardless of the size of the input.

O(n)
def (arr, val):
for el in arr:
if el == val:
print el
return

The for loop in this function will always take one operation per element in the input array. O(n) functions
like this have a linear increase in complexity.

7|Page
O(log(n))
def nlog(n):
if n == 1:
return
else:
nlog(n/2)

This function, while entirely useless, is a really simple demonstration of O(log(n)). This recursive function
calls itself repeatedly on the input value divided by two until n = 1. This repeated halving of the input with
each iteration is indicative of O(log(n)) functions. Initially their difficult increases at a fast pace but then
quickly levels off as the input gets larger.

O(n²)
def containsDuplicate(arr):
for i in range(0, len(arr)):
for j in range(0, len(arr)):
if i != j:
if arr[i] == arr[j]:
return true
return false

The outer for loop has one operation per element in the input array, but every time the outer loop iterates,
the inner one will complete a full loop the size of the input array. O(n²) algorithms like this have a
complexity equal to the square of the input and are said to be in quadratic time.

O(2ⁿ)
def countFibonacci(num):
if num <= 1:
return num
else:
return countFibonacci(num-2) + countFibonacci(num-1)

This last example is a recursive function that returns the nth Fibonacci number. Without getting into the
details, this is a recursive algorithm where the number of additions increases at an exponential rate which
results in massive numbers of operations as n increases.

8|Page
Figure 2: Big-O Complexity Chart

Bubble Sort

This sorting algorithm uses a double for loop to iterate through the array while repeatedly comparing and
swapping out of order adjacent elements. The inner for loop does all the swapping and the outer for loop
ensures there is at least one pass through the inner loop per array element. When an out of place element
is found it will get dragged along by the constant swapping until it reaches its correct location. Here is a
walk-through of the algorithm:
Pass One:
[ 5, 3, 4, 1, 2 ] 5 > 3 Swap
[ 3, 5, 4, 1, 2 ] 5 > 4 Swap
[ 3, 4, 5, 1, 2 ] 5 > 1 Swap
[ 3, 4, 1, 5, 2 ] 5 > 2 Swap
[ 3, 4, 1, 2, 5 ]
Pass Two:
[ 3, 4, 1, 2, 5 ] 3 < 4 No Swap
[ 3, 4, 1, 2, 5 ] 4 > 1 Swap
[ 3, 1, 4, 2, 5 ] 4 > 2 Swap
[ 3, 1, 2, 4, 5 ] 4 < 5 No Swap
Pass Three:
[ 3, 1, 2, 4, 5 ] 3 > 1 Swap
[ 1, 3, 2, 4, 5 ] 3 > 2 Swap
[ 1, 2, 3, 4, 5 ] 3 < 4 No Swap

9|Page
In this example we only needed 3 passes to complete the sorting. However, in a worst case scenario where
the list was fully reversed we would need one pass per element. For each pass we then iterate through
the full array once per pass to do the comparisons and swaps putting us in O(n²) time. Here is the sample
code written in python:
def bubbleSort(arr):
# Loop through list once per element
for _ in range ( 0, len(arr)):
for i in range(0, len(arr)-1):
# compare and swap adjacent elements
if arr[i] > arr[i+1]:
arr[i], arr[i+1] = arr[i+1], arr[i]
return arr
Selection Sort

This algorithm is an improvement over Bubble Sort where it makes only one swap per inversion. Like
Bubble Sort, Selection Sort is O(n²). However, they only appear to be computationally equivalent due to
Big-O ignoring lower complexity code in the algorithm. In Bubble Sort, elements are dragged along from
index to index through repeated swaps. In Selection Sort, we iterate through the list and find the largest
out of place element and then make a single swap to place it in the correct location.

This is achieved by an outer loop counting down from last (largest) element of the array and an inner loop
counting up with a conditional checking for inversions and swapping if j > i, where j is the inner iterator
and i is the outer iterator. Here is a walkthrough of Selection Sort. For this one I’ll keep track of the outer
iterator i.

Pass One:
i
[ 5, 3, 4, 1, 2 ] 5 is the largest element in range 0-i so swap them
i
[ 2, 3, 4, 1, 5 ] 4 is the largest element in range 0-i so swap them
i
[ 2, 3, 1, 4, 5 ] 3 is the largest element in range 0-i so swap them
i
[ 2, 1, 3, 4, 5 ] 2 is the largest element in range 0-i so swap them
i
[ 1, 2, 3, 4, 5 ] done!

Here is the Selection Sort code written in python:

def selSort(arr):
for i in range(len(arr)-1,0,-1):
# Temporary storage for index of highest element
maxIndex = 0
# Iterate from index 0 to index i inclusive
for j in range(0, i+1):

10 | P a g e
# Find highest value index
if arr[j] > arr[maxIndex]:
maxIndex = j
# Store element to be swapped out
temp = arr[i]
# Swap the two elements
arr[i], arr[maxIndex] = arr[maxIndex], temp
return arr
Merge Sort

This last algorithm is by far the most efficient of the three clocking in at O(n log(n)). It is a recursive
algorithm which falls under the classification of Divide and Conquer Algorithms (D&C). It also has some
interesting side effects which give it a number of uses outside of sorting numbers. D&C is a basic design
pattern for algorithms which follows three steps:

1. Divide the input into smaller sub-problems that are smaller instances of the same type of problem.
2. Conquer the sub-problems recursively
3. Cleanup and recombine the results

In D&C we recursively break down a large problem into smaller and smaller sections until it becomes easy
to solve and then build back up into the larger problem. In the case of Merge Sort we use two recursive
calls to split the problem in half over and over until we get to the base case of one single element array.

Figure 3: Recursively dividing the problem

11 | P a g e
In the diagram above we have made two recursive calls on the two halves of the array. This creates a tree
structure with two branches at every level. Once we reach the base case and start returning actual values
back up the tree we can start sorting. 5 and 3 are returned to the left most recursive call on the 3rd level
where they are compared and swapped. Then [5, 3] from the left recursive call and [ 4 ] from right
recursive call are returned to 2nd level. On the 2nd level we check for out of order elements between the
two returned arrays, sort them, and then return to the next level up. This continues until we have sorted
the entire problem.

The trickiest part of implementing Merge Sort is handling the sorting between the two arrays returned by
the pair of recursive calls. Comparing two one element arrays is simple. But how do we handle larger array
comparisons? It turns out its not complicated at all. Both arrays are already sorted and all we have to do
is walk through both arrays at once and compare the lowest values yet to be added to the output arrays.
Each array gets its own iterator which only increases when one of its elements is added to the output. If
one array completes its iteration first then we know all the remaining elements in the other array have to
be bigger and the entire other array is added to the output array.

Let’s look at our example focusing on the last sorting step with arrays [3, 4, 5] and [1, 2]. i and j represent
the iterators for the two arrays:

i j
[ 3, 4, 5 ] + [ 1, 2 ] = [ 1 ] j was smaller so it gets added
i j
[ 3, 4, 5 ] + [ 1, 2 ] = [ 1, 2 ] j was smaller so it gets added
i j
[ 3, 4, 5 ] + [ 1, 2 ] = [ 1, 2, 3, 4, 5] j completed its iteration so
all of i's array are added to the output

Looking at the tree diagram of all the recursive calls you can see the characteristic halving of the input you
see in O(log(n)) functions. However, in addition we are iterating through the arrays at each recursion level.
n elements are iterated log(n) times or in other words O(n*log(n))—log linear on the graph included
earlier. This shows Merge Sort to be substantially faster than the Bubble and Selection Sort.

12 | P a g e
Here is an example implementation of Merge Sort written in python:

def mergeSort(arr):
if len(arr) == 1:
return arr
else:
a = arr[:len(arr)/2]
b = arr[len(arr)/2:]
a = mergeSort(a)
b = mergeSort(b)
c = []
i = 0
j = 0
while i < len(a) and j < len(b):
if a[i] < b[j]:
c.append(a[i])
i = i + 1
else:
c.append(b[j])
j = j + 1
c += a[i:]
c += b[j:]
return c

The following picture shows the most common data structures and their worst case scenario or Big O
complexity.

Figure 4: Common Data Structure Operations

13 | P a g e
The following picture shows the most common sorting algorithms and their worst case scenario or Big O
complexity.

Figure 5: Array Sorting Algorithms

14 | P a g e
References
1. Steven S Skiena, April 27, 2011, The Algorithm Design Manual.
2. big-O%20notation_files/webidblue_1linecentr.htm
3. Udi Manber, January 11, 1989, Introduction to algorithm: A creative approach
4. https://xlinux.nist.gov/dads/HTML/algorithm.html
5. Jon Kleinberg, March 26, 2005,Algorithm design
6. http://en.wikipedia.org/wiki/Big_O_notation
7. http://bigocheatsheet.com

15 | P a g e

También podría gustarte