Efficiently checking if arbitrary object is NaN in Python / numpy / pandas?

Question

My numpy arrays use np.nan to designate missing values. As I iterate over the data set, I need to detect such missing values and handle them in special ways.

Naively I used numpy.isnan(val), which works well unless val isn't among the subset of types supported by numpy.isnan(). For example, missing data can occur in string fields, in which case I get:

>>> np.isnan('some_string')

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

TypeError: Not implemented for this type

Other than writing an expensive wrapper that catches the exception and returns False, is there a way to handle this elegantly and efficiently?

Answer 1

pandas.isnull() is used for checking the missing values in numeric and string/object arrays.

Example:

import pandas as pd

import numpy as np

s = pd.Series(['hello’', np.nan, 'world'])

pd.isnull(s)

Output: 

0    False

1     True

2    False

dtype: bool

Here, numpy.nan is used for representing the missing values.