Naive Bayes Explained

Multinomial Naïve Bayes

Let us see how naïve bayes works by performing a movie review sentimental analysis task.

Let’s say we have a small movie review dataset which looks like this:

Now I want to use Naïve Bayes algorithm to predict the sentiment of the movie review “Amazing movie”

1) Find Prior Probability

First we look at the small movie review dataset and count the frequency of good and bad movies

Probability of good reviews

P(Good) = Number of good reviews / Total reviews = 0.6

Probability of bad reviews

P(Bad) = Number of bad reviews / Total reviews = 0.4

2) Conditional Probability

Now we count the frequency of each words in the movie review dataset and put them into a table.

Good Reviews

Bad Reviews

We then compute the conditional probability for each words

For example:

P(Amazing | Good) = Frequency of the word “Amazing” in good reviews / total number of words in good reviews

We do the same for bad reviews as well

3) Now we can make predict the sentiment of the review

“Amazing Movie”

P(Good | “Amazing Movie”) = P(Good) X P(Amazing | Good) X P(Movie | Good) = 0.6 X 0.5333 X 0.2 = 0.064

P(Bad | “Amazing Movie”) = P(Bad) X P(Amazing | Bad) X P(Movie | Bad) = 0.4 X 0.2 X 0.2 = 0.016

We can clearly see that P(Good | “Amazing Movie”) > P(Bad | “Amazing Movie”), so the Naïve Bayes algorithm will predict the review “Amazing Movie” as positive sentiment

Laplace Smoothing

What about the review “Horrible Romance Movie”, lets analyze the sentiment using Naïve Bayes algorithm

P(Good | “Horrible Romance Movie”)

= P(Good) X P(Horrible | Good) X P(Romance | Good) X P(Movie | Good)

= 0.6 X 0.0667 X 0.2 X 0.2

= 0.0016

P(Bad | “Horrible Romance Movie”)

= P(Bad) X P(Horrible | Bad) X P(Romance | Bad) X P(Movie | Bad)

= 0.4 X 0.6 X 0 X 0.2

= 0

So P(Good | “Horrible Romance Movie”) > P(Bad | “Horrible Romance Movie”) hence the review “Horrible Romance movie” is a positive sentiment ?!

Hmm.. something is not quite right.

The problem is that there are no “Romance” words in bad reviews so P(Romance | Bad) = 0 hence the result will always be 0.

We can address this problem by using a technique called Laplace smoothing

We increase every word frequency by 1 to get rid of the 0, then we calculate the new conditional probability.

Positive reviews

Negative Reviews

Let’s find the sentiment of “Horrible Romance Movie” again to see if we get a different result after Laplace Smoothing

P(Good | “Horrible Romance Movie”)

= P(Good) X P(Horrible | Good) X P(Romance | Good) X P(Movie | Good)

= 0.6 X 0.1053 X 0.2105 X 0.2105

= 0.0028

P(Bad | “Horrible Romance Movie”)

= P(Bad) X P(Horrible | Bad) X P(Romance | Bad) X P(Movie | Bad)

= 0.4 X 0.5 X 0.0714 X 0.2143

= 0.0031

Since P(Bad | “Horrible Romance Movie”) > P(Good | “Horrible Romance Movie”), hence the review “Horrible Romance Movie” will be classified as negative review which is correct.

Gaussian Naïve Bayes

We use Gaussian Naïve Bayes to find the conditional probability of continuous features (e.g: weight, height).

We assume that these features are normally distributed (aka Gaussian distribution – hence the name)

Let’s use Gaussian Naïve Bayes to classify whether a person is male or female based on their height and weight. Suppose we are given a dataset of 6 person.

Now we want to predict whether a person with height 173cm and weight 80kg is a male or female

1) Find prior probability
P(Male) = Number of male / total number of person

= 3/6 = 0.5

P(Female) = Number of female / total number of person

= 3/6 = 0.5

2) Find the mean and variance of each feature

Mean of height (male) = 171 + 175 + 180 / 3 =  175.33

Standard Deviation of height (male) = 4.5093

Mean of height (female) = 166.33

Standard Deviation of height (female) = 4.1633

Mean of weight

(male) = 74.33

(female) = 52.33

Std Dev of weight

(male) = 9.0185

(female) = 2.5166

3) P(Male | height = 173cm, weight = 80kg)

= P(Male) X P(height=173 | Male) X P(weight=80 | Male)

=