Bayesian Machine Learning – Data Science Central

How many clusters will be formed?

To answer this question let us take a sample data set visual :

Have a close look to the image above, and you will feel that there are 3 clusters in it. So lets define our 3 clusters probabilistically.

Now, let us try to write an equation which can define the probability of a point in the data to be a part of every cluster, but before that we need a mechanism or a probability function that can tell to which cluster a point can be from with what probability. Suppose for now let us say that a point can be a part of any cluster with equal probability of 1/3. Then the probabilistic distribution of a point would look like:

But in the above case it is an assumption that every point can belong from any cluster with equal probability. But is not true right!!So somebody has to tell this, and for it we use a Latent Variable which knows the distribution of every point belonging to every cluster. And the Bayesian model looks like:

The above model states that Latent Variable t knows, to which cluster the point x belongs and with what probability. So if we re-write the above equation:

Now let us call all our parameters as

the above equation states that the probability by which the Latent variable takes value c, given all the parameters is π_c

The above equation states that probability of a point sampled from cluster c, given that it has come from cluster c is       Ɲ (X|µ_c, ∑_c²).

Now we know that P(X, t=c | θ) = P(X| t=c, θ) * P(t=c | θ) from our Bayesian model. We can marginalize t to compute   

The above equation is exactly same as our original one.

Fact Checks:

• In the equation P (t = c | θ) = π_{c, this has no condition over the data points x, and thus it can be considered as Prior distribution and as a hard classifier.}
• _{Marginalization was done on t, to have the exact expression on LHS as was chosen originally. Therefore, t remains latent and unobserved.}
• _{Iif we want the probability of a point belonging to a specific cluster, given the parameters} θ and the point x, then the expression looks like:

Where LHS is the posterior distribution of t w.r.t X and θ. Z is the normalization constant and can be calculated as :

The Chicken-Egg Problem in Probabilistic Clustering

Now it is obvious that the Bayesian model we made between t and x, when we say the latent variable knows the cluster number of every point, we say it probabilistically. That means the formulae applied is the posterior distribution as we discussed in Fact – check section. Now the problem is, to compute θ, we should know t and to compute t, we should know θ. We will see this in the following equations:

And the posterior t is computed using

Now, from the above expression it is clear that means and standard deviations are computed if we know the source t and vice-versa.

So, in the next post we will see how to solve the above problem using Expectation – Maximization algorithm.

Thanks for reading!!!