0 votes
1 view
in Big Data Hadoop & Spark by (11.5k points)

I would like to access to the min and max of a specific column from my dataframe but I don't have the header of the column, just its number, so I should I do using scala?

maybe something like this :

val q = nextInt(ncol) //we pick a random value for a column number
col = df(q)

val minimum = col.min()

Sorry if this sounds like a silly question but I couldn't find any info on SO about this question

1 Answer

0 votes
by (31.4k points)

I would suggest you to get the column name from the metadata. Do something like this:

val selectedColumnName = df.columns(q) //pull the (q + 1)th column from the columns array

df.agg(min(selectedColumnName), max(selectedColumnName))

Welcome to Intellipaat Community. Get your technical queries answered by top developers !