View RDD contents in Python Spark?

Question

Running a simple app in pyspark.

f = sc.textFile("README.md")
wc = f.flatMap(lambda x: x.split(' ')).map(lambda x: (x, 1)).reduceByKey(add)

I want to view RDD contents using foreach action:

wc.foreach(print)

This throws a syntax error:

SyntaxError: invalid syntax

What am I missing?

Answer 1

You are encountering this error because in Python 2.6 print isn't a function.

You can either use the __future__ library to treat print as a function:

>>> from __future__ import print_function

>>> wc.foreach(print)

or

define a helper UDF that performs the print:

>>> from operator import add

>>> f = sc.textFile("README.md")

>>> def g(x):

...     print x

...

>>> wc.foreach(g)

Note: for each executes on the worker nodes and the outputs may not necessarily appear in your driver/shell (it probably will in local mode, but not when running on a cluster).

Therefore, it would be better to use collect() to bring the RDD contents back to the driver.

>>> for x in wc.collect():

...     print x

If you want to know more about Spark, then do check out this awesome video tutorial: