MLlib is a unattached collection of high-level algorithms that runs on Spark. This is what Mahout used to be the only Mahout of old was on Hadoop MapReduce. In 2014 Mahout announced it would no longer accept Hadoop Mapreduce code and completely switched new development to Spark (with other engines possibly in the offing, like H2O).
The most important thing to come out of this is a Scala-based generalized distributed optimized linear algebra engine and conditions including an interactive Scala shell. Perhaps the most relevant word is "generalized". Since it runs on Spark anything possible in MLlib can be applied with the linear algebra engine of Mahout-Spark.
If you need a common engine that will do a lot of what tools like R do but on really big data, look at Mahout. If you need a particular algorithm, look at each to see what they have. For instance, Kmeans runs in MLlib but if you need to cluster A'A (a co-occurrence matrix used in recommenders) you'll need them both because MLlib doesn't have a matrix transpose or A'A.
If you want more knowledge regarding Spark, refer the following video: