You're buying a product online,
and you see 3 different sellers. They're all offering that same product,
at essentially the same price. One of them has a 100% positive rating,
but with only 10 reviews. Another has a 96 positive rating with 50 total reviews. And yet another has a 93% positive rating,
but with 200 total reviews. Which one should you buy from? [music] I think we all have this instinct that
the more data we see, it gives us more confidence in a given rating. We're a little suspicious of seeing 100% ratings since, more often than not,
they come from a tiny number of reviews. Which makes it feel more plausible that things could
have gone another way, and given a lower rating.
But how do you make that intuition quantitative? What's the rational way to reason
about the trade-off here, between the confidence gained from more data,
versus the lower absolute percentage? This particular example, is a slight modification from
one than John Cook gave in his excellent blog post "A Bayesian view of Amazon Resellers". For you and me, it's a wonderful excuse to dig into
a few core topics in probability and statistics. And to really cover these topics properly, it takes time, so what I'm gonna do is break this down into three videos. Where, in this first one,
we'll set up a model for the situation and start by talking about the binomial distribution. The second is gonna bring in ideas of
Bayesian updating and how to work with probabilities
over continuous values. And in the third, we'll look at something known as a
Beta distribution and pull up a little Python to analyze the data we have, and come to various different answers
depending on what it is you want to optimize.
Now, to throw you a bone
before we dive into all the math, let me just show you
what one of the answers turns out to be, since it's delightfully simple. When you see an online rating, something like this 10/10, you pretend that there were two more reviews. One that was positive and one that's negative. In this case, that means you pretend that it's 11/12, which would give 91.7%. This number is your probability of
having a good experience with that seller. So in the case of 50 reviews,
where you have 48 positive and 2 negative, you pretend that it's really 49 positive and 3 negative, which would give you 49/52, or 94.2%. That's your probability of having a good experience with the second seller.
Playing the same game with
our third seller who had 200 reviews, you get 187/202 or 92.6%. So, according this rule, it would mean your best bet is
to go with seller number two. This is something known as Laplace's rule of succession
– it dates back to the 18th century – and to understand what assumptions
are underlying this, and how changing either those assumptions or
your underlying goal can change the choice you make, we really do need to go through all the math. So without further ado, let's dive in! [Music] Step 1: How exactly are we modeling the situation
and what exactly is it that you want to optimize? One option is to think of each seller
as producing random experiences, that are either positive or negative, and that each seller has some kind of constant
underlying probability of giving a good experience, what we're gonna call the "success rate," or "s" for short. The whole challenge is that we don't know "s." When we see that first rating of 100% with 10 reviews, that doesn't mean the underlying success rate is 100% It could very well be something like 95%. And just to illustrate, let me run a little simulation where I choose a random number between 0 and 1, and if it's less than 0.95, I'll record it as a positive review otherwise, I'll record it as a negative review.
Now do this ten times in a row… And then make ten more… And ten more… And ten more, and so on to get a sense of, what sequences of 10 reviews, for a seller with this success rate, 0.95,
would tend to look like. Quite a few of those – around 60% actually – give 10/10. So the data we see seems perfectly plausible if the seller's true success rate was 95%. Or… maybe it's really 90%, or 99%. The whole challenge is that we just don't know. As to the goal, let's say you simply want to maximize your probability of having a positive experience, despite not being sure of the success rate.
So, think about this here. We've got many different possible success rates
for each seller, any number from zero up to one, and we need to say something about how likely
each one of these success rates is, a kind of probability of probabilities. Unlike the more gamified examples
like coin flips and die tosses and the sort of things you see in an intro probability class, where you go in assuming a long-run frequency,
like 1/2 or 1/6, what we have here is uncertainty about
the long-run frequency itself, which is a much more potent kind of doubt.
I should also emphasize that this kind of setup is relevant to many, many situations in the real world, where you need to make a judgement
about a random process from limited data. For example, let's say that you're
setting up a factory producing cars, and in an initial test of 100 cars,
2 of them come out with some kind of problem. If you plan to spin things up to produce a million cars, what are you willing to *confidently* say about how many total cars
will have problems that need addressing? It's not like the test *guarantees*
that the true error rate is 2%, but what exactly *does* it say? As your first challenge, let me ask you this: if you did magically know
the true success rate for a given seller, say it was 95%, how would you compute
the probability of seeing, say 10 positive reviews and 0 negative reviews? Or 48 and 2? Or 186 and 14? In other words, what's the probability of seeing the data given an assumed success rate? A moment ago, I showed you something like this
with a simulation, generating 10 random reviews.
And, with a little programming, you could just
do that many times, building up a histogram to get some sense of
what this distribution looks like. [music] Likewise, you could simulate sets of 50 reviews
and get some sense for how probable it would be to see 48 positive and 2 negative. You see, this is the nice thing about probability; a little programming can almost always let you "cheat"
a little and see what the answer is ahead of time by simulating it For example, after a few hundred thousand samples
of 50 reviews, assuming the success rate is 95%, it looks like about 26.1% of them
would give us this 48/50 review. Luckily, in this case, an exact formula is not bad at all. The probability of seeing exactly 48/50 looks like this. This first term is pronounced "50 choose 48", and it represents the total number of ways
that you could take 50 slots and fill out 48 of them.
For example, maybe you start with 48 good reviews
and end with 2 bad reviews. Or maybe you start with 47 good reviews,
and then it goes bad-good-bad and so on. In principle, if you were to enumerate every single
possible way of filling 48/50 slots like this, the total number of these patterns is "50 choose 48", which, in this case, works out to be 1225. What do we multiply by this count?
Well it's the probability of any one of these patterns. Which is the probability of a single positive review,
raised to the 48th [power], times the probability of a single negative review squared.
Crucial is that we assume each review
is independant of the last, so we can multiply all the probabilities together like this, and with the numbers we have, when you evaluate it,
it works out to be 0.261, which matches what we just saw empirically
with the simulation. You could also replace this 48
with some other value and compute the probability of seeing
any other number of positive reviews. Again, assuming a given success rate. What you're looking at right now,
by the way, is known in the business as a binomial distribution, one of THE most fundamental distributions
in probability. It comes up whenever you have something like
a coin flip, a random event that can go one of two ways,
and you repeat it some number of times.
And what you want to know is the probability
of getting various different totals. For our purposes, this formula gives us the probability of
seeing the data, given an assumed success rate, Which, ultimately, we want to somehow use
to make judgments about the opposite: the probability of a success rate
given the fixed data that we see. These are related, but definitely distinct. To get more in that direction,
let's play around with this value of "s" and see what happens as we change it
to different numbers between 0 and 1.
The binomial distribution that it produces kind of looks like this pile
that's centered around whatever "s" times 50 is. The value we care about,
the probability of seeing 48 out of 50 reviews, is represented by this highlighted 48th bar. So let's draw a second plot on the bottom
representing how that value depends on "s". When "s" is equal to 0.96,
that value is as high as it's ever gonna get. And this should kind of makes sense. Because when you look at that review of 96%, it should be most likely
if the true underlying success rate was 96%. As "s" increases, it kind of peters out;
going to 0 as "s" approaches 1. Since someone with a perfect success rate
would never have those 2 negative reviews. Also, as you move to the left,
it approaches 0 pretty quickly. By the time you get to "s" = 0.8, getting 48 out of 50 reviews by chance
is exceedingly rare. It would happen 1 in a 1000 times.
This plot we have on the bottom is a great start
to getting a more quantitative description, for which *values* of "s" feel more or less plausible. Written down as a formula, what I want you to remember
is that, as a function of the success rate s, the curve looks like some constant times "s" to the number of positive reviews times 1 minus "s" to the number of negative reviews. In principle, if we had more data,
like 480 positive reviews and 20 negative reviews, the resulting plot would still be centered around .96,
but it would be smaller and more concentrated. A good exercise right now would be to see
if you could explain *why* that's the case. There is a lingering question, though,
of what to actually *do* with these curves.
I mean, our goal is to compute the probability
that you have a good experience with this seller. So what do you do? Naively, you might think that probability is 96%
since that's where the peak of the graph is, which, in a sense, is the most likely success rate. But think of the example with 10 out of 10 positives. In that case, the whole binomial formula simplifies to be "s" to the power 10. The probability of seeing 10 consecutive good reviews is the probability of seeing *one* of them
raised to the 10th. The closer the true success rate is to 1,
the higher the probability of seeing a 10 out of 10. Our plot on the bottom only ever increases
as "s" approaches 1. But even if "s = 1" is the value that maximizes this probability, surely you wouldn't feel comfortable saying that, you personally, have a 100% probability
of a good experience with this seller. Maybe you think that, instead, we should look for
some kind of center of mass of this graph and that would absolutely be
on the right track.
First though, we need to explain
how to take *this* expression for the probability of the data we're seeing,
given a value of "s", and get the probability *for* a value of "s"
(the thing we actually don't know) given the data (the thing we actually know). And that requires us to talk about Bayes rule
and also probability density functions. For those, I'll see you in part two. [music].
