0 votes
1 view
in Big Data Hadoop & Spark by (11.5k points)

I've successfully create a row_number() partitionBy by in Spark using Window, but would like to sort this by descending, instead of the default ascending. Here is my working code:

from pyspark import HiveContext
from pyspark.sql.types import *
from pyspark.sql import Row, functions as F
from pyspark.sql.window import Window

data_cooccur.select("driver", "also_item", "unit_count",

That gives me this result:

 |   s10|      s11|         1|     1|
 |   s10|      s13|         1|     2|
 |   s10|      s17|         1|     3|

And here I add the desc() to order descending:

data_cooccur.select("driver", "also_item", "unit_count", F.rowNumber().over(Window.partitionBy("driver").orderBy("unit_count").desc()).alias("rowNum")).show()

And get this error:

AttributeError: 'WindowSpec' object has no attribute 'desc'

1 Answer

0 votes
by (32.5k points)

desc should be applied on a column, not on a window definition. You can use either apply this method on a column:

from pyspark.sql.functions import col  


Or on a standalone function:

from pyspark.sql.functions import desc


You can learn in-depth about SQL statements, queries and become proficient in SQL queries by enrolling in an industry-recognized SQL online course.

Welcome to Intellipaat Community. Get your technical queries answered by top developers !