BUG: Don't cast categorical nan to int

[dsaxton]

• closes Converting from categorical to int ignores NaNs #28406
• passes black pandas
• tests added / passed
• whatsnew entry

This raises an error when attempting to cast a Categorical or CategoricalIndex containing nans to an integer dtype. Also had to remove the casting within get_indexer_non_unique since this won't always be possible.

[dsaxton]

FWIW the issue with test_get_indexer_non_unique is here https://github.com/pandas-dev/pandas/blob/master/pandas/core/indexes/base.py#L4717. If target is Categorical then the values can't always be cast to the dtype of the categories.

[jbrockmendel]

should the use_inf_as_na option matter here?

[dsaxton]

should the use_inf_as_na option matter here?

Wasn't familiar with that option, but it shouldn't affect numpy behavior I think?

[TomAugspurger]

I think this PR should focus on just NaN handling.

I'd rather see Float64Index properly handle astyping inf values to int (raise), and then do self.categories.take(self.codes) with a check if any of the codes is negative, so that we raise.

[dsaxton]

I think this PR should focus on just NaN handling.

I'd rather see Float64Index properly handle astyping inf values to int (raise), and then do self.categories.take(self.codes) with a check if any of the codes is negative, so that we raise.

Ok, so for the purposes of this PR remove the references to np.inf in the code and tests?

[TomAugspurger]

That would be my preference. Did my earlier comment make sense?

The NaN issue can be solved by checking Categorical.codes

In [7]: c = pd.Categorical([1, None, 2])

In [8]: c.codes < 0
Out[8]: array([False,  True, False])

The inf issue is deeper, and is present in Float64Index, so I think it should be split to its own PR.

In [9]: c = pd.Categorical([1, np.inf])

In [10]: c.categories.astype(int)
Out[10]: Int64Index([1, -9223372036854775808], dtype='int64')

[dsaxton]

That would be my preference. Did my earlier comment make sense?

The NaN issue can be solved by checking Categorical.codes

In [7]: c = pd.Categorical([1, None, 2])

In [8]: c.codes < 0
Out[8]: array([False,  True, False])

The inf issue is deeper, and is present in Float64Index, so I think it should be split to its own PR.

In [9]: c = pd.Categorical([1, np.inf])

In [10]: c.categories.astype(int)
Out[10]: Int64Index([1, -9223372036854775808], dtype='int64')

I believe so, you want to fix the inf casting behavior in one place (Float64Index) and then use the fixed astype method here? Regarding the nan issue, is there a reason to prefer self.codes < 0 over self.isna()?

[jreback]

ValueError: cannot convert float NaN to integer is what we do now on
pd.Series([1,2,np.nan],dtype='Int64').astype('int') so would replicate this message

[TomAugspurger]

Kinda pedantic, but we can have other NA values here.

In [18]: cat = pd.Categorical([pd.Timestamp('2000'), pd.NaT])

In this case, it's not a float that we're refusing to cast. So perhaps Cannot convert NA to integer.

[dsaxton]

Agree this seems more correct; do we want to think about consistency with Series as @jreback pointed out above? Incidentally I just noticed that Series seems to be misbehaving as well in this special case, so probably worth a separate issue or PR:

[ins] In [5]: pd.Series([pd.Timestamp("2000"), pd.NaT]).astype(int)             
Out[5]: 
0     946684800000000000
1   -9223372036854775808
dtype: int64

[jreback]

thanks @dsaxton