How to sort an RDD in Scala Spark?

Question

Reading Spark method sortByKey :

sortByKey([ascending], [numTasks])   When called on a dataset of (K, V) pairs where K implements Ordered, returns a dataset of (K, V) pairs sorted by keys in ascending or descending order, as specified in the boolean ascending argument.

Is it possible to return just "N" amount of results. So instead of returning all results, just return the top 10. I could convert the sorted collection to an Array and use take method but since this is an O(N) operation is there a more efficient method ?

Answer 1

If you only need the top 10, use rdd.top(10). It is faster, as it avoids sorting.

rdd.top creates one parallel pass through the data, collecting the top N in each partition in a heap, and then it merges the heaps. It is an O(rdd.count) operation. Sorting would be O(rdd.count log rdd.count), and incur a lot of data transfer — it does a shuffle, so all of the data would be transmitted over the network.

If you want to know more about Spark, then do check out this awesome video tutorial: