Explore Courses Blog Tutorials Interview Questions
0 votes
in Machine Learning by (19k points)

I run a clustering algorithm and want to evaluate the result by using silhouette score in scikit-learn. But in the scikit-learn, it needs to calculate the distance matrix: distances = pairwise_distances(X, metric=metric, **kwds)

Due to the fact that my data is order of 300K, and my memory is 2GB, and the result is out of memory. And I can not evaluate the clustering result.

Does anyone know how to overcome this problem?

1 Answer

0 votes
by (33.1k points)

In this problem, you should set the sample_size parameter in the call to silhouette_score to some value smaller than 300K. 

Using this parameter, you will sample data points from X and calculate the silhouette_score on those instead of the entire array.

For more details, study K-means Clustering

Hope this answer helps.

Browse Categories