Since union doesn't move any data around, it is considered as an efficient method. If rdd1 has 10 partitions and rdd2 has 20 partitions then rdd1.union(rdd2) will have 30 partitions: the partitions of the two RDDs put after each other. This is simply a bookkeeping change, here no shuffling is involved.

But necessarily it discards the partitioner. A partitioner is always constructed for a given number of partitions. Partitions of the resulting RDD is different from both rdd1 and rdd2.

After applying the union, you can run repartition to shuffle the data and organize it by key.

There is one exception to the above. If rdd1 and rdd2 have the same partitioner (with the same number of partitions), union behaves differently. In this case, union will join the partitions of the two RDDs pairwise, giving it the same number of partitions as each of the inputs had. This may involve moving data around (if the partitions were not co-located) but will not involve a shuffle. In this case the partitioner is retained. (The code for this is in __PartitionerAwareUnionRDD.scala.__)