Which clustering algorithm to use

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4. For details, see the Google Developers Site Policies.

Clustering in Machine Learning. Clustering Workflow. Create a Similarity Measure. Let's quickly look at types of clustering algorithms and when you should choose each type.

Types of Clustering Several approaches to clustering exist. Centroid-based Clustering Centroid-based clustering organizes the data into non-hierarchical clusters, in contrast to hierarchical clustering defined below.

Figure 1: Example of centroid-based clustering. Density-based Clustering Density-based clustering connects areas of high example density into clusters. Figure 2: Example of density-based clustering.

Distribution-based Clustering This clustering approach assumes data is composed of distributions, such as Gaussian distributions. There are several single Gaussian models that act as hidden layers in this hybrid model.

So the model calculates the probability that a data point belongs to a specific Gaussian distribution and that's the cluster it will fall under. It breaks the data into little summaries that are clustered instead of the original data points.

The summaries hold as much distribution information about the data points as possible. This algorithm is commonly used with other clustering algorithm because the other clustering techniques can be used on the summaries generated by BIRCH.

You can't use this for categorical values unless you do some data transformations. This clustering algorithm is completely different from the others in the way that it clusters data.

Each data point communicates with all of the other data points to let each other know how similar they are and that starts to reveal the clusters in the data. You don't have to tell this algorithm how many clusters to expect in the initialization parameters. As messages are sent between data points, sets of data called exemplars are found and they represent the clusters. An exemplar is found after the data points have passed messages to each other and form a consensus on what data point best represents a cluster.

When you aren't sure how many clusters to expect, like in a computer vision problem, this is a great algorithm to start with. This is another algorithm that is particularly useful for handling images and computer vision processing. Mean-shift is similar to the BIRCH algorithm because it also finds clusters without an initial number of clusters being set. This is a hierarchical clustering algorithm, but the downside is that it doesn't scale well when working with large data sets.

It works by iterating over all of the data points and shifts them towards the mode. The mode in this context is the high density area of data points in a region. That's why you might hear this algorithm referred to as the mode-seeking algorithm. It will go through this iterative process with each data point and move them closer to where other data points are until all data points have been assigned to a cluster.

It's a density-based algorithm similar to DBSCAN, but it's better because it can find meaningful clusters in data that varies in density. It does this by ordering the data points so that the closest points are neighbors in the ordering. This makes it easier to detect different density clusters. There's also a special distance stored for each data point that indicates a point belongs to a specific cluster.

This is the most common type of hierarchical clustering algorithm. It's used to group objects in clusters based on how similar they are to each other. This is a form of bottom-up clustering, where each data point is assigned to its own cluster. Then those clusters get joined together. At each iteration, similar clusters are merged until all of the data points are part of one big root cluster.

Agglomerative clustering is best at finding small clusters. The end result looks like a dendrogram so that you can easily visualize the clusters when the algorithm finishes. We've covered eight of the top clustering algorithms, but there are plenty more than that available. There are some very specifically tuned clustering algorithms that quickly and precisely handle your data.

Here are a few of the others that might be of interest to you. There's another hierarchical algorithm that's the opposite of the agglomerative approach. It starts with a top-down clustering strategy.

So it will start with one large root cluster and break out the individual clusters from there. This is known as the Divisive Hierarchical clustering algorithm. There's research that shows this is creates more accurate hierarchies than agglomerative clustering, but it's way more complex.

Mini-Batch K-means is similar to K-means, except that it uses small random chunks of data of a fixed size so they can be stored in memory. This helps it run faster than K-means so it converges to a solution in less time. The last algorithm we'll briefly cover is Spectral Clustering. This algorithm is completely different from the others we've looked at.

It works by taking advantage of graph theory. This algorithm doesn't make any initial guesses about the clusters that are in the data set. It treats data points like nodes in a graph and clusters are found based on communities of nodes that have connecting edges. Mini-Batch K-means is an unsupervised ML method meaning that the labels assigned by the algorithm refer to the cluster each array was assigned to, not the actual target integer. With the functions returned above, we can determine the accuracy of our algorithm.

Some metrics can be applied to the clusters directly, regardless of associated labels. The metrics used are:. Previously, we made assumptions while choosing a particular value for K, but it might not always be the case. The function to calculate metrics for a model is defined below. The centroid is the point that is representative in every cluster.

If we were dealing with A, B points, the centroid would simply be a point on the graph. As the number of clusters and the number of data points increase, the relative saving in computational time increases. The saving in time is more noticeable only when the number of clusters is enormous. The effect of batch size in computational time is also more noticeable only when the number of clusters is larger.

An increase in the count of clusters decreases the similarity of the Mini-Batch K-Means solution to the K-Means solution. As the number of clusters increases, the agreement between partitions decreases. This means the final partitions are different but closer in quality.

There are certain evaluation metrics to check how good the clusters obtained by your clustering algorithm are. Homogeneity metric : Clustering results satisfy homogeneity if all its clusters contain only data points that are members of a single class. This metric is independent of the absolute value of labels. The homogeneity score is bounded between 0 and 1.

A low value indicates low homogeneity, and 1 stands for perfectly homogeneous labeling. Perfect labelings are homogeneous. Non-perfect labelings that further split classes into more clusters can be homogeneous. Clustering results satisfy completeness only if the data points of a given class are part of the same cluster. Perfect labelings are complete. Non-perfect labelings that assign all class members to the same clusters are still complete.

V-measure cluster labeling gives a ground truth. The V measure is the harmonic mean between homogeneity and completeness. This metric is symmetric. When the real ground truth is unknown, this metric can be helpful to calculate the acceptance of the two independent label assignment techniques on the same dataset.

The similarity between two clusters can be calculated using the Rand Index RI by counting all pairs of samples and counting pairs assigned in different or same clusters in the true and predicted clusters. Please refer to the link for a detailed user guide. For a user guide, please refer to the link.

The GitHub repo has the data and all notebooks for this article. This blog covered the most critical aspects of clustering, image compression, digit classification, customer segmentation, implementation of different clustering algorithms, and evaluation metrics.

Hope you guys learned something new here. Machine Learning Engineer at OptiSol Data Labs Data science professional with experience in predictive modeling, data processing, chatbots and data mining algorithms to solve challenging business problems. On a high level, Machine Learning is the union of statistics and computation.

The crux of machine learning revolves around the concept of algorithms or models which are in fact statistical estimations on steroids. However, any given model has several limitations depending on the data distribution. None of them can be entirely accurate since they are just estimations even if on steroids.

These limitations are popularly known by the name of bias and variance. A model with high bias will oversimplify by not paying much attention to the training points e. The issue arises when the limitations are subtle, like when we have to choose between a random forest algorithm and a gradient boosting algorithm or between two variations of the same decision tree algorithm. Both will tend to have high variance and low bias. GDPR compliant.

Privacy policy. Contents: What are clustering algorithms? Types of clustering algorithms and how to select one for your use case. Applications of clustering in different fields. Issues with the unsupervised modeling approach.

Factors to consider when choosing clustering algorithms. Different practical use cases of clustering in Python. Clustering metrics. Connectivity models — like hierarchical clustering, which builds models based on distance connectivity. Centroid models — like K-Means clustering, which represents each cluster with a single mean vector. Distribution models — here, clusters are modeled using statistical distributions.

They only offer grouping information. Graph-based models — a subset of nodes in the graph such that an edge connects every two nodes in the subset can be considered as a prototypical form of cluster. For example, consider customer segmentation with four groups. Each customer can belong to either one of four groups. Soft clustering — a probability score is assigned to data points to be in those clusters. A cluster can be defined by the max distance needed to connect to the parts of the cluster.

These algorithms provide a hierarchy of clusters that at certain distances are merged. In the dendrogram, the y-axis marks the distance at which clusters merge. Agglomerative — it starts with an individual element and then groups them into single clusters.

Divisive — it starts with a complete dataset and divides it into partitions. Each data point is treated as a single cluster. We have K clusters in the beginning. At the start, the number of data points will also be K. Now we need to form a big cluster by joining 2 closest data points in this step. This will lead to total K-1 clusters.

Two closest clusters need to be joined now to form more clusters. This will result in K-2 clusters in total. Repeat the above three steps until K becomes 0 to form one big cluster. No more data points are left to join. After forming one big cluster at last, we can use dendrograms to split the clusters into multiple clusters depending on the use case.

AHC is easy to implement, it can also provide object ordering, which can be informative for the display. In the AHC approach smaller clusters will be created, which may uncover similarities in data.

Whenever outliers are found, they will end up as a new cluster, or sometimes result in merging with other clusters. The number of samples. A total number of features.

The number of informative features. The number of redundant features. The number of duplicate features drawn randomly from redundant and informative features. The number of clusters per class. Clusters obtained by Hierarchical Cluster Algorithm. Initially, a K number of centroids is chosen. There are different methods for selecting the right value for K. Shuffle the data and initialize centroids—randomly select K data points for centroids without replacement.

Create new centroids by calculating the mean value of all the samples assigned to each previous centroid. K-Means clustering uses the Euclidean Distance to find out the distance between points. It adapts to new examples very frequently.

K-Means clustering is good at capturing the structure of the data if the clusters have a spherical-like shape. It always tries to construct a nice spherical shape around the centroid.