In many applications, you need to compare two or more data sets with each other to see how much they are similar or different. For instance, you have measured the height of men and women in Japan and Netherlands and now you like to know how much they are different.

Two commonly used method for measuring distances are the Kullback-Liebler divergence and the Bhattacharyya distance.

KL divergence (Kullback-Liebler divergence) measures the difference between two probability distributions p and q.

\begin{equation} \label{Kullback_Lieblerdivergence}
D{KL}(p||q)=\int_{-\infty}^\infty p(x)\log\frac{p(x)}{q(x)} \,\mathrm{d}x
\end{equation}

But it only works if your data is made of a single Gaussian and it is not applicable If your data is made of a mixture of Gaussians.

Sfikas et al [1]  have extended the Kullback Liebler divergence for GMM and proposed a distance metric using the values (\mu ,\Sigma,\pi ) for each one of the two distributions in the following form:

\begin{equation} \label{analytical_Kullback_Lieblerdivergence}
C2(p||q)=-\log \large[ \frac{2\sum{i,j}\pi{i}\pi{j}^{\prime} \sqrt{ \frac{|V{ij}|}{e^{k{ij}}|\sum{i}| |\sum{j}^{\prime}|} } }
{
\sum{i,j}\pi{i}\pi{j} \sqrt{ \frac{|V{ij}|}{e^{k{ij}}|\sum{i}| |\sum{j}|} }+
\sum{i,j}\pi{i}^{\prime}\pi{j}^{\prime} \sqrt{ \frac{|V{ij}|}{e^{k{ij}}|\sum{i}^{\prime}| |\sum{j}^{\prime}|} }
}
\large]
\end{equation}
Where:
\begin{equation}\label{Kullback_Liebler_divergenceDetails1}
V{ij}=(\Sigma{i}^{-1} +\Sigma{j}^{-1})^{-1}
\end{equation}
and
\begin{equation}\label{Kullback_Liebler_divergenceDetails2}
K{ij}=\mu{i}^{T}\Sigma{i}^{-1}(\mu{i}-\mu{j}^{\prime})+\mu{j}^{\prime T}\Sigma{j}^{\prime -1}(\mu{j}^{\prime}-\mu{i})
\end{equation}

Code in matlab:

Update: Here is a very nice interactive vizualiztaion of Kullback-Liebler divergence.