Scikit-Learn's Pipeline: A sparse matrix was passed, but dense data is required

Question

I'm finding it difficult to understand how to fix a Pipeline I created (read: largely pasted from a tutorial). It's a python 3.4.2:

df = pd.DataFrame

df = DataFrame.from_records(train)

test = [blah1, blah2, blah3]

pipeline = Pipeline([('vectorizer', CountVectorizer()), ('classifier', RandomForestClassifier())])

pipeline.fit(numpy.asarray(df[0]), numpy.asarray(df[1]))

predicted = pipeline.predict(test)

When I run it, I get:

TypeError: A sparse matrix was passed, but dense data is required. Use X.toarray() to convert to a dense numpy array.

This is for the line pipeline.fit(numpy.asarray(df[0]), numpy.asarray(df[1])).

I've experimented a lot with solutions through numpy, scipy, and so forth, but I still don't know how to fix it. And yes, similar questions have come up before, but not inside a pipeline. Where is it that I have to apply toarray or todense?

Answer 1

A CountVectorizer provides a sparse matrix and the RandomForestClassifier requires a dense matrix. It is possible to convert using X.todense(). Doing this will essentially increase your memory footprint.