I would like to calculate group quantiles on a Spark dataframe (using PySpark). Either an approximate or exact result would be fine. I prefer a solution that I can use within the context of groupBy / agg, so that I can mix it with other PySpark aggregate functions. If this is not possible for some reason, a different approach would be fine as well.
I also have access to the percentile_approx Hive UDF but I don't know how to use it as an aggregate function.
For the sake of specificity, suppose I have the following dataframe:
from pyspark import SparkContext
import pyspark.sql.functions as f
sc = SparkContext()
df = sc.parallelize([
df_grp = df.groupBy('grp').agg(f.magic_percentile('val', 0.5).alias('med_val'))