How do you parallelize apply() on Pandas Dataframes making use of all cores on one machine?

Question

As of August 2017, Pandas DataFame.apply() is unfortunately still limited to working with a single core, meaning that a multi-core machine will waste the majority of its compute-time when you run

df.apply(myfunc, axis=1).

How can you use all your cores to run apply on a dataframe in parallel? 

Answer 1

Using this below code will apply function f in a parallel fashion to column col of dataframe df:

import multiprocessing as mp

pool = mp.Pool(mp.cpu_count())

df['newcol'] = pool.map(f, df['col'])

pool.terminate()

pool.join()

If you want to make your career in Artificial Intelligence then go through this video: