Chapter 1 Sample Space and Probability

Probability
Tuesday, February 4, 14
What is Probability?
Probability is a mathematical discipline akin to geometry.
Formal Logic Content. Intuitive Background. Applications.
What is Probability?
The actual science of logic is conversant at present only with things either certain, impossible, or entirely doubtful, none of which (fortunately) we have to reason on. Therefore the true logic for this world is the calculus of Probabilities, which takes account of the magnitude of the probability which is , or ought to be, in a reasonable man's mind James Clerck Maxwell (1850)
Why to study Probability?

CS Theory
Alg
M Le ach ar ine nin g
ori
thm
Probability
ion t a y m or eor f In Th
N w et ks or
Probabilistic Model
The sample space , which is the set of all possible outcomes of an experiment. The probability law, which assigns to a set A of possible outcomes (also called an event) a nonnegative number P(A) (called probability of A) that encodes our knowledge or belief about the collective likelihood of the elements A. The probability law must satisfy certain properties to be introduced shortly.
Probabilistic Model
Probability law B
P( B )
A
Experiment Sample Space
P ( A)
(Set of possible outcomes)
Events
Set Theory

Finite, Countable and Uncountable Sets. Set Operations. The Algebra of Sets.
Sets
A set is a collection of objects, which are the elements of the set. xS x /S
Specifying Sets
As a list of elements: S = {x1 , x2 , . . . , xn } With words: the set of even natural numbers Specify a rule or algorithm: S = {r Q : r < 2}
2
Subsets
If every element of a set S is also an element of a set T , we say that S is a subset of T , and we write S T or T S If S T and T S the two sets are equal , and we write S=T The universal set, denoted by , contains all the objects of interest in a particular context.
Set Operations
The complement of a set S , with respect to the universe is the set of elements in that do not belong to S : S = {x : x / S}
c
Set Operations
The union of two sets S and T is the set of all elements that belong to S or T (or both): S T = {x : x S orx T } The intersection of two sets S and T is the set of all elements that belong to both S and T : S T = {x : x S andx T }
Algebra of Sets
De Morgans laws
Functions
Domain
Range
Cardinality
{Thorin,Balin, Bifur, Bofur, Bombur, Dori, Fili, Gloin, Kili} {Gruon, Mocoso, Tmido, Mudito, Dormiln, Felz, Sabio}
Cardinality
Cardinality
Cardinality
Cardinality
Cardinality
Cardinality
Power Sets and Cantors Theorem
Probabilistic Model
The sample space , which is the set of all possible outcomes of an experiment. The probability law, which assigns to a set A of possible outcomes (also called an event) a nonnegative number P(A) (called probability of A) that encodes our knowledge or belief about the collective likelihood of the elements A. The probability law must satisfy certain properties to be introduced shortly.
Probabilistic Model
Probability law B
P( B )
A
Experiment Sample Space
P ( A)
(Set of possible outcomes)
Events
Sample Space

Different elements of the sample space should be distinct and mutually exclusive. The sample space must be collectively exhaustive.
Sample Space (Examples)

For a single toss coin the sample space consist of two points: = {H, T } (We exclude possibilities like the coin stands on edge, the coin disappears, etc.)

For n tosses of a coin the sample space is = {w : w = (a1 , . . . , an ), ai = H or T } and the general number N () of outcomes is 2n .

For n tosses of a coin the sample space is = {w : w = (a1 , . . . , an ), ai = H or T } and the general number N () of outcomes is 2n .

ue Tre nc eia ba ld s es ed cr ip tio n
(1,1) 1 (1,2) (1,3) (1,4)
Sample space for pair of 4-sided rolls 4

2nd roll
se q
3 root 2 1 1 2 3 1st roll 4
Probability Laws
1. (Non-negativity) P(A) 0, for every event A.
Probability Laws
2. (Additivity) If A and B are two disjoint events, then the probability of their union satises P ( A B ) = P ( A) + P ( B ) More generally, if the sample space has an innite number of elements and A1 , A2 , . . . is a sequence of disjoint events then the probability of their union satises P ( A1 A2 ) = P ( A1 ) + P ( A2 ) +
Probability Laws
3. (Normalization) The probability of the entire sample space is equal to 1, that is P() = 1
Properties
Let A, B and C be events. 1. P() = 0. 2. If A B , then P(A) P(B ) 3. P(A B ) = P(A) + P(B ) P(A B ) 4. P(A B ) P(A) + P(B )
Discrete Probability Laws

If the sample space consists of a nite number of possible outcomes, then the probability law is specied by the probabilities of the events that consists of a single element. In particular, the probability of any event A = {s1 , s2 , . . . , sn } is the sum of the probabilities of its elements: P ( A) = P ( s 1 ) + P ( s 2 ) + + P ( s n )
Discrete Uniform Probability Laws

If the sample space consists of n possible outcomes which are equally likely (i.e. all single-element events have the same probability), then the probability of any event A is given by N ( A) P ( A) = n
Discrete Uniform Probability Laws

Sample space for pair of 4-sided rolls 4
2nd roll
3 2 1 1 2 3 1st roll 4
{at least one roll is a 4}
{the rst roll is equal to the second}

Counting
Cardinality
Product Rule
Theorem (Pairs). With n elements a1 , a2 , . . . , an and m elements b1 , b2 , . . . , bm , it is possible to form nm pairs (ai , bj ) containing one element for each group.
Generalized Product Rule

Theorem. Given n1 elements a1 , a2 , . . . , an1 and n2 elements b1 , b2 , . . . , bn2 , etc., up to nr elements x1 , x2 , . . . , xnr ; it is possible to form n1 n2 nr ordered r-tuples (ai1 , ai2 , . . . , air ) containing one element for each group.
Generalized Product Rule

n1 n2 n3 n4
choices Stage 1
choices Stage 2
choices Stage 3
choices Stage 4
Examples
Throwing a dice r times. No 1 in r throws? Display of ags. Suppose r ags of dierent colors are to be displayed on n poles in a row. In how many ways can this be done? Loops, Recursions, ...
Ordered Samples
Consider a set or population of n elements a1 , . . . , an . Any ordered arrangement (ai1 , . . . , aik ) of k symbols is called an ordered sample of size k drawn from our population. Sampling with replacement (repetitions are allowed). Sampling without replacement (repetitions are not allowed).
Ordered Samples
We now consider examples involving the selection of k balls from an urn containing n distinguishable balls.
Ordered Sample with Replacement
Ordered Samples
Ordered Samples
Ordered Samples
Ordered Samples
N () = nk
Ordered Samples
N () = nk
Ordered Samples
Ordered Sample without Replacement
Ordered Samples
Ordered Samples
Ordered Samples
Ordered Samples
N () = (n)k
Ordered Samples
N () = (n)k
Ordered Samples
Theorem. For a population of n elements and a prek scribed sample size k , there exist n dierent samples with replacement and (n)k samples without replacement.
Ordered Samples
When k = n that sample is called a permutation of the elements of the population and (n)n = n! Whenever we speak of random samples of xed size k , the adjective random is to imply that all samples have the same probability, namely, nk in sampling with replacement and 1/(n)k in sampling without replacement. If n is large and k is relatively small, the ratio (n)k /nk is near unity i.e. the two ways of sampling are practically equivalent.
Examples (Balls and Bins)

Probability an element is not included in the sample? If n balls are randomly placed into n cells, what is the probability every cell is occupied?
Birthday Paradox
Throw a dice 6 times, what is the probability all six faces appear? Elevator (10 oors, 7 people)
The Sum Rule

Theorem. If A1 , A2 , . . . , An are disjoint sets, then: | A1 A2 An | = | A1 | + | A2 | + + | An |
Counting Passwords
On a certain computer system, a valid password is a sequence between six and eight symbols. The rst symbol must be a letter which can be upper case or lower case and the remaining symbols must be either letter or numbers.
Subpopulations
Two populations are considered dierent only if one contains an element not contained in the other.
Subpopulations
Unordered Sample without Replacement
Subpopulations
Subpopulations
Subpopulations
Subpopulations
n N () = k
Subpopulations
n N () = k
Subpopulations
Theorem. A population of n elements possesses ferent sub-populations of size r n. n n = r nr n = 1, 0! = 1 0
n r
dif-
The Division Rule

Theorem. If f : A B is k -to-1, then: | A| = k | B | Unlike as it may seem, many counting problems are made much easier by initially counting every item multiple times and the correcting the answer using the division rule.
Examples
Hands of poker. What is the probability that a hand of poker contains contains ve dierent face values? Occupancy problem. Probability that a specied cell contains exactly k balls?
Examples (Balls and bins)
Partitions
Theorem. Let k1 , . . . , kr be integers such that k1 + k2 + + kr = n, ki 0,
The number of ways in which a population of n elements can be divided into k ordered parts (partitioned into k subpopulations) of which the rst contains k1 elements, the second k2 elements, etc., is n! k1 ! k2 ! kr !
Example: Sequences with repetitions

How many sequences can be formed by permuting the letters in the 10-letter word BOOKKEEPER?
Example: Balls and Bins
Example: Binomial Theorem

Theorem. For all n N and a, b R: ( a + b) =
n
k=0
n X
n n k k a b k
Example: Multinomial Theorem

Theorem. For all n N and zi R: X ( z1 + z2 + + zm ) n k1 k2 km z1 z2 zm k 1 , . . . km
n
k1 , . . . , k m N k1 + + km = n
Combinatorial Proofs
Symmetry n n = k nk Pascals Identity n n1 n1 = + k k1 k 3n n = n X
r =0
n r
2n nr
Conditional Probability
Conditional Probability provides us with a way to reason about the outcome of an experiment, based on partial information. We seek to construct a new probability law that takes into account the new information: a probability law that form ant even A, species the conditional probability of A given B denoted by P ( A| B )
P(A|B ) must constitute a legitimate probability law. The conditional probabilities must be consistent with our intuition in important special cases P( A B ) P ( A| B ) = P( B ) where we assume that P(B ) > 0.
1. (Non-negativity) P(A|B ) 0, for every event A.
2. (Additivity) If A1 and A2 are two disjoint events, then the probability of their union satises P ( A1 A2 | B ) = P ( A1 | B ) + P ( A2 | B ) More generally, if the sample space has an innite number of elements and A1 , A2 , . . . is a sequence of disjoint events then the probability of their union satises P ( A 1 A2 | B ) = P ( A1 | B ) + P ( A2 | B ) +
3. (Normalization) The probability of the entire sample space is equal to 1, that is P( | B ) = 1
Examples
We toss a coin three successive times. Compute the conditional probability P(A|B ) when A and B are the events: A = { more heads than tails come up} B = {1st toss is a head}
Conditional Probability for Modeling

When building models for experiments with sequential character, it is natural to rst specify conditional probabilities and the use them to determine unconditional probabilities: P ( A B ) = P ( B ) P ( A| B )
Examples
If an aircraft is present in a certain area, a radar detects it and generates an alarm signal with probability 0.99. If an aircraft is not present, the radar generates a (false) alarm with probability 0.10. We assume that an aircraft is present with probability 0.005. What is the probability of no aircraft presence and a false alarm? What is the probability of aircraft presence and no detection?
Examples
Aircraft Present
P(
P(
P
5 0 . 0
) A | (B
.99 0 =
= ) A
P( B c | A)
=0
.01
Missed Detection
Ac )=
0. 9
P
5
A (B |
c)
0 1 . 0
False Alarm
P( B c |A c )=
0.90
Multiplication Rule
Assuming that all of the conditioning events have positive probability, we have:
n \ n 1 \ i=1
P(
i=1
Ai ) = P ( A1 ) P ( A2 | A1 ) P ( A3 | A1 A2 ) P ( An |
Ai )
Example
Three cards are drawn from an ordinary 52-card deck without replacement. What is the probability that none of the three cards is a heart, assuming that all triplets are equally likely?
The Monty Hall Problem

1 2 3

1 2 3

1 2 3

1 2 3

D 1 1 2 3 O 2 3 3 2 P( D )
1 3 1 3 1 3 1 3
P( O | D ) p (1 p) 1 1
P ( D ) P( O | D ) 1 3p 1 3 (1 p)
1 3 1 3
Keep Win Win Lose Lose
Switch Lose Lose Win Win
Minimum Global Cut

Global min cut. Given a connected, undirected graph G = (V, E ), nd a cut (A, B ) of minimum cardinality. Applications. Identify clusters, network reliability, TSP solvers.
Contraction algorithm
Minimum Global Cut

- replace u and v by single new super-node w - keep parallel edges, but delete self-loops - preserve edges, updating endpoints of u and v to w
Contraction algorithm. [Karger 1995]
Pick an edge e = (u, v) uniformly at random. Contract edge e.
Repeat until graph has just two nodes v1 and v1. Return the cut (all nodes that were contracted to form v1).
b d e
u
f
contract u-v f
10
p parallel edges, but delete self-loops

1
at until graph has just two nodes v1 and v1.
Minimum Global Cut n the cut (all nodes that were contracted to form v ).
Reference: Thore Husfeldt
Minimum Global Cut
Total Probability Theorem

Let A1 , A2 , . . . , An be events that form a partition of the sample space , that is Ai Aj = for i 6= j and
n [
Ai =
i=1
and assume that P(Ai ) > 0, for all i. Then, for any event B , we have P ( B ) = P ( A1 ) P ( B | A1 ) + P ( An ) P ( B | An )
Total Probability Theorem

A1 B A3
A4 A2
Example
You enter a chess tournament where your probability of winning is 0.3 against half the players (call then type I), 0.4 against a quarter of the players (call the type II), and 0.5 against the remainder quarter of the players (call them type III). You play a game against a randomly chosen opponent. What is the probability of winning?
Inference and Bayes Theorem

Let A1 , A2 , . . . , An be disjoint events that form a partition of the sample space, and assume that P(Ai ) > 0, for all i, then, for any event B such that P(B ) > 0, we have P ( Ai | B ) = = P ( Ai ) P ( B | Ai ) P( B ) P ( Ai ) P ( B | Ai ) P ( A1 ) P ( B | A1 ) + + P ( An ) P ( B | An )
Inference and Bayes Theorem

Prior
P ( Ai ) P ( B | Ai ) P ( Ai | B ) = P( B )
Posterior
Example
Probability that an aircraft is present? Suppose that you won. What is the probability you had an opponent of type I? The False-Positive Puzzle.
Examples
If an aircraft is present in a certain area, a radar detects it and generates an alarm signal with probability 0.99. If an aircraft is not present, the radar generates a (false) alarm with probability 0.10. We assume that an aircraft is present with probability 0.005. What is the probability of aircraft presence given alarm went o ?
Example
You enter a chess tournament where your probability of winning is 0.3 against half the players (call then type I), 0.4 against a quarter of the players (call the type II), and 0.5 against the remainder quarter of the players (call them type III). You play a game against a randomly chosen opponent. What is the probability of having an opponent of type I given you won?
Independence
Independent Events
When the occurrence of B does not alter the probability of A: P ( A| B ) = P ( A) we say that A is independent of B . Equivalently, P ( A B ) = P ( A) P ( B )
Example
Consider an experiment involving two successive rolls of a 4-sided die in which all 16 possible outcomes are equally likely. Ai = {1st roll results in i}, A = {1st roll is a 1}, Bj = {2nd roll results in j } B = {sum is a 5}
A = {minimum is 2},
B = {maximum is 2}
Conditional Independence
Given an even C , the events A and B are called conditionally independent if P ( A B | C ) = P ( A| C ) P ( B | C )
Example
Independence of a Collection of Events

We say that the events A1 , A2 , . . . , An are independent if Y \ P(Ai ), for every subset S of {1, 2, . . . , n} P( Ai ) =
i S i S
Pairwise independence does not imply independence
Example
Minimum Global Cut
Network Connectivity
0.8 0.9
E
0. 9
0.9
A
0. 7 5
0.85
0.95
Network Connectivity
1
2 3 Parallel Connection
Series Connection
Independent Trials and Binomial Probabilities

If an experiment involves a sequence of independent but identical stages, we say we have a sequence of independent trials. In the special case where there are only two possible results at each stage, we say we have a sequence of independent Bernoulli trials.
Contention Resolution in a Contention resolution in a distributed system Distributed System

Contention resolution. Given n processes P1, , Pn, each competing for access to a shared database. If two or more processes access the database simultaneously, all processes are locked out. Devise protocol to ensure all processes get through on a regular basis. Restriction. Processes can't communicate. Challenge. Need symmetry-breaking paradigm.
P1
P2 . . . Pn
4

Chapter 1 Sample Space and Probability

Cargado por

Información del documento

Descripción original:

Derechos de autor

Formatos disponibles

Compartir este documento

Compartir o incrustar documentos

Opciones para compartir

¿Le pareció útil este documento?

¿Este contenido es inapropiado?

Copyright:

Formatos disponibles

Chapter 1 Sample Space and Probability

Cargado por

Copyright:

Formatos disponibles

Probability

Probability is a mathematical discipline akin to geometry.

Formal Logic Content. Intuitive Background. Applications.

Why to study Probability?

M Le ach ar ine nin g

Experiment Sample Space

(Set of possible outcomes)

Power Sets and Cantors Theorem

Experiment Sample Space

(Set of possible outcomes)

Sample Space (Examples)

Sample Space (Examples)

Sample Space (Examples)

Sample Space (Examples)

(1,1) 1 (1,2) (1,3) (1,4)

Sample space for pair of 4-sided rolls 4

3 root 2 1 1 2 3 1st roll 4

Discrete Probability Laws

Discrete Uniform Probability Laws

Discrete Uniform Probability Laws

{at least one roll is a 4}

{the rst roll is equal to the second}

Generalized Product Rule

Generalized Product Rule

Ordered Sample with Replacement

Ordered Sample with Replacement

Ordered Sample with Replacement

Ordered Sample with Replacement

Ordered Sample with Replacement

Ordered Sample with Replacement

Ordered Sample without Replacement

Ordered Sample without Replacement

Ordered Sample without Replacement

Ordered Sample without Replacement

Ordered Sample without Replacement

Ordered Sample without Replacement

Examples (Balls and Bins)

The Sum Rule

Unordered Sample without Replacement

Unordered Sample without Replacement

Unordered Sample without Replacement

Unordered Sample without Replacement

Unordered Sample without Replacement

Unordered Sample without Replacement

The Division Rule

Examples (Balls and bins)

Example: Sequences with repetitions

Example: Balls and Bins

Example: Binomial Theorem

Example: Multinomial Theorem

Conditional Probability for Modeling

The Monty Hall Problem

The Monty Hall Problem

The Monty Hall Problem

The Monty Hall Problem

The Monty Hall Problem

Keep Win Win Lose Lose

Switch Lose Lose Win Win

Minimum Global Cut

Minimum Global Cut

Contraction algorithm. [Karger 1995]

Pick an edge e = (u, v) uniformly at random. Contract edge e.

p parallel edges, but delete self-loops