Pandas DataFrame – Check for None

nannumpypandaspython

I would like to find where None is found in the dataframe.

pd.DataFrame([None,np.nan]).isnull()
OUT: 
      0
0  True
1  True

isnull() finds both numpy Nan and None values.

I only want the None values and not numpy Nan. Is there an easier way to do that without looping through the dataframe?

Edit:
After reading the comments, I realized that in my dataframe in my work also include strings, so the None were not coerced to numpy Nan. So the answer given by Pisdom works.

Best Answer

If you want to get True/False for each line, you can use the following code. Here is an example as a result for the following DataFrame:

df = pd.DataFrame([[None, 3], ["", np.nan]])

df
#      0      1
#0  None    3.0
#1          NaN

How to check None

Available: .isnull()

>>> df[0].isnull()
0     True
1    False
Name: 0, dtype: bool

Available: .apply == or is None

>>> df[0].apply(lambda x: x == None)
0     True
1    False
Name: 0, dtype: bool

>>> df[0].apply(lambda x: x is None)
0     True
1    False
Name: 0, dtype: bool

Available: .values == None

>>> df[0].values == None
array([ True, False])

Unavailable: is or ==

>>> df[0] is None
False

>>> df[0] == None
0    False
1    False
Name: 0, dtype: bool

Unavailable: .values is None

>>> df[0].values is None
False

How to check np.nan

Available: .isnull()

>>> df[1].isnull()
0    False
1     True
Name: 1, dtype: bool

Available: np.isnan

>>> np.isnan(df[1])
0    False
1     True
Name: 1, dtype: bool

>>> np.isnan(df[1].values)
array([False,  True])

>>> df[1].apply(lambda x: np.isnan(x))
0    False
1     True
Name: 1, dtype: bool

Unavailable: is or == np.nan

>>> df[1] is np.nan
False

>>> df[1] == np.nan
0    False
1    False
Name: 1, dtype: bool

>>> df[1].values is np.nan
False

>>> df[1].values == np.nan
array([False, False])

>>> df[1].apply(lambda x: x is np.nan)
0    False
1    False
Name: 1, dtype: bool

>>> df[1].apply(lambda x: x == np.nan)
0    False
1    False
Name: 1, dtype: bool
Related Question