corneretageres.com

Understanding Bernoulli Distribution: A Comprehensive Guide

Written on

In this tutorial series, we will thoroughly explore probability distributions. Our focus will extend beyond merely presenting formulas; we will delve into the foundational definitions behind these formulas, enhancing comprehension through practical Python examples.

Authors: Pratik Shukla, Roberto Iriondo Last updated: January 7, 2021

Table of Contents:

  1. What is a Random Variable?
  2. Discrete Random Variable
  3. Continuous Random Variable
  4. Probability Distributions
  5. Bernoulli Distribution
  6. Probability Mass Function (PMF)
  7. Mean of Bernoulli Distribution
  8. Variance of Bernoulli Distribution
  9. Standard Deviation of Bernoulli Distribution
  10. Mean Deviation of Bernoulli Distribution
  11. Moment Generating Function for Bernoulli Distribution
  12. Cumulative Density Function (CDF) for Bernoulli Distribution
  13. Python Implementation
  14. Summary of the Bernoulli Distribution
  15. Resources
  16. References

Before we delve into probability distributions, it's crucial to grasp some key terminology related to random variables.

What is a Random Variable?

A random variable is defined as one whose value cannot be precisely determined. In simpler terms, if we cannot ascertain the value of a variable through any function, it qualifies as a random variable. It represents potential numerical results from a random event.

Properties of a Random Variable:

  • Random variables are typically denoted by capital letters.
  • They can be classified as either discrete or continuous.

Examples:

  1. Flipping a Fair Coin:

    As depicted, the result of flipping a coin is independent of other variables, resulting in a random outcome.

  2. Rolling a Fair Die:

    Similar to the coin, the outcome of a die roll cannot be predicted beforehand, affirming its randomness.

Next, we will examine non-random variables briefly.

In the first example, we can determine the value of x by simple arithmetic, thus making it a fixed value rather than random. In the second example, y's value hinges on x, showing that y can be reproduced by inputting the same x value. Therefore, y is not random. When dealing with probability distributions, we focus on random variables.

Discrete Random Variable:

A random variable is categorized as discrete if its values can be counted. Discrete variables can be counted a finite number of times, and they may not necessarily be integers; finite floating-point values can also be discrete.

Examples:

  1. The number of students on a school bus.
  2. The count of cookies on a plate.
  3. The number of heads when flipping a coin.
  4. The number of planets orbiting a star.
  5. The total income of family members.

Continuous Random Variable:

Conversely, a random variable is termed continuous if its values are derived from measurements. Unlike discrete variables, continuous variables cannot be counted in a finite timeframe.

Examples:

  1. The precise weight of a random animal.
  2. The exact height of a randomly selected student.
  3. The distance traveled within an hour.
  4. The total food consumed the previous day.
  5. The winning time of an athlete.

It's essential to emphasize the term "Exact," indicating that all measurements are taken with utmost precision.

For instance, when measuring an athlete's race completion time, we can specify that they finished in 9.5 seconds, and further refine it to 9.5238 seconds, demonstrating the infinite precision possible for continuous variables.

Key Differences Between Discrete and Continuous Variables:

Example: What is your current age?

Consider whether this represents a continuous or discrete variable. Your age is classified as continuous. We can express age with increasing precision, making it impossible to count it exactly in a finite amount of time. Thus, it is continuous.

However, if we ask for your age in years, then it becomes discrete, as it represents a specific number of years.

Next, we will explore Probability Distributions, which are categorized based on data types into Discrete or Continuous.

Probability Distribution:

A probability distribution is a mathematical construct that indicates the probabilities associated with various possible outcomes of an experiment.

Bernoulli Distribution:

Conditions for the Bernoulli Distribution:

  1. Only one trial must be conducted.
  2. The trial can yield only two outcomes: success or failure.
  3. P(Success) = p
  4. P(Failure) = 1 - p = q
  5. Conventionally, we assign the value of 1 to the event with probability p and 0 to the event with probability 1 - p.
  6. Typically, we assume p > 1 - p, meaning P(Success) > P(Failure).
  7. We need the probability of at least one of the events or historical data indicating the experimental probability.

If the data meets these criteria, then:

A discrete random variable X adheres to a Bernoulli distribution with a success probability of p.

Visual representation of the Bernoulli distribution:

Examples:

For example, in a two-candidate election, Patrick and Gary, where a vote can be cast for either:

  • P(Success) = P(1) = Vote for Patrick = 0.7
  • P(Failure) = P(0) = Vote for Gary = 0.3

Here, we have one trial and two potential outcomes, indicating that the data follows a Bernoulli distribution.

Probability Mass Function (PMF):

The probability mass function (PMF) for a discrete random variable X assigns probabilities to each possible value of X. The PMF allows us to determine the probabilities for each random variable.

Let X be a discrete random variable with possible values x1, x2, x3, …, xn. The PMF must satisfy the following conditions:

Properties of PMF:

  1. The total of all probabilities in a PMF must equal 1.
  2. All probability values must be non-negative.

Probability Mass Function (PMF) for Bernoulli Distribution:

Let’s visualize the function:

Mean for Bernoulli Distribution:

The mean of a discrete random variable X is calculated as a weighted average, where the probabilities weight each value of X. In Bernoulli Distribution, X can take only two values: 0 and 1, and we can compute the weight using the PMF.

The expected value E[X] indicates the likelihood of the favored outcome.

The expected value or mean of the Bernoulli Distribution is expressed as:

Mean of Bernoulli Distribution:

Variance for Bernoulli Distribution:

Variance (?²) quantifies how far each number in a set of random numbers deviates from the mean. The standard deviation is the square root of the variance.

Definition of Variance:

The variance of a discrete probability distribution is defined as:

In our case, variable x can only take the values of 0 and 1.

Variance of Bernoulli Distribution:

A more common representation for calculating variance in statistics is:

Deriving Variance:

Variance represents the expected value of the squared deviation between each value and the mean of the distribution.

From the definition of variance, we can derive:

Calculating variance using this formula:

As shown, the variance of the Bernoulli distribution remains consistent, regardless of the formula utilized.

Standard Deviation for Bernoulli Distribution:

Standard deviation measures the dispersion of a group’s measurements from the average (mean or expected value).

A low standard deviation indicates that most values are near the average, while a high standard deviation suggests greater spread.

Mean Deviation for Bernoulli Distribution:

Mean deviation signifies the average of absolute deviations of a dataset relative to its mean.

Definition:

For Discrete probability Distribution:

Calculating the mean deviation for the Bernoulli distribution:

Moment Generating Function For Bernoulli Distribution:

For the following derivations, we will utilize formulas from our previous tutorial. We recommend reviewing our tutorial on Moment Generating Functions.

Moment Generating Function:

Finding Raw Moments:

#### 1. First Moment:

a. First Raw Moment:

#### 2. Second Moment:

a. Second Raw Moment:

b. Second Central Moment (Variance):

#### 3. Third Moment:

a. Third Raw Moment:

b. Third Central Moment:

c. Third Standardized Moment (Skewness):

#### 4. Fourth Moment:

a. Fourth Raw Moment:

b. Fourth Centered Moment:

c. Fourth Standardized Moment (Kurtosis):

Cumulative Distribution Function (CDF):

Using the PMF, we can articulate the Cumulative Distribution Function (CDF) for the Bernoulli distribution as follows:

Now, let's move on to the practical application in Python.

Python Implementation:

  1. Import necessary libraries:
  2. Determine the moments:
  3. Calculate the mean value:
  4. Calculate the median value:
  5. Calculate the variance:
  6. Calculate the standard deviation:
  7. Probability Mass Function (PMF):
  8. Plotting the PMF:
  9. Cumulative Density Function (CDF):
  10. Plotting the CDF:
  11. Creating a bar graph for PMF:
  12. Creating a bar graph for CDF:
  13. Outputs from various experiments:

Summary of the Bernoulli Distribution:

This concludes our tutorial on the Bernoulli distribution. We hope you found it informative and learned something new. We will strive to cover additional probability distributions in detail in future tutorials. Your suggestions and feedback are invaluable for our improvement, so please share your thoughts in the comments.

DISCLAIMER: The opinions expressed in this article belong to the author(s) and do not represent those of Carnegie Mellon University or any associated entities. These writings are intended as reflections of current ideas and serve as a basis for discussion and enhancement.

Published via Towards AI

Resources:

  • Google Colab implementation.
  • GitHub repository.

References:

[1] Probability Distribution, Wikipedia, https://en.wikipedia.org/wiki/Probability_distribution [2] Bernoulli Distribution, Statlect, https://www.statlect.com/probability-distributions/Bernoulli-distribution [3] Variance, Wikipedia, https://en.wikipedia.org/wiki/Variance [4] Bernoulli Distribution, Wikipedia, https://en.wikipedia.org/wiki/Bernoulli_distribution [5] Bernoulli Distribution, SciPy.org, https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.stats.bernoulli.html [6] Probability Mass Function, Wikipedia, https://en.wikipedia.org/wiki/Probability_mass_function [7] Mean and Variance of Probability Distributions, Probabilistic World, https://www.probabilisticworld.com/mean-variance-probability-distributions/ [8] Standard Deviation, Wikipedia, https://en.wikipedia.org/wiki/Standard_deviation