BUG: Series.setitem losing precision when enlarging #47342
Conversation
phofl
added
Dtype Conversions
pandas/core/indexing.py
Outdated
| # this preserves dtype of the value | ||
| new_values = Series([value])._values | ||
| # this preserves dtype of the value and of the object | ||
| if isna(value == value): |
There was a problem hiding this comment.
value == value to see if it's pd.NA? Not exactly clear why value is being compare to itself
There was a problem hiding this comment.
Yep, is there a better way to achieve this?
There was a problem hiding this comment.
I think isna(value) should just work right?
In [1]: pd.isna(pd.NA)
Out[1]: True
In [2]: pd.isna(np.nan)
Out[2]: True
There was a problem hiding this comment.
I don’t want to run in there when I get nan, because nan does not fit into int64 for example
There was a problem hiding this comment.
I see. Well If you only want to check for pd.NA I think value is pd.NA is clearer IMO
There was a problem hiding this comment.
The current code only checks for pd.NA, so my example above of pd.NaT isn't actually applicable right now. But in light of:
we should strive to have the result consistent regardless of enlargement or not.
We should maybe handle np.nan as well?
We currently treat np.nan as "missing value" when setting into a nullable series without enlargement. So then we should also treat it as "missing" in case of of enlargement? (and so preserve the nullable Int64 dtype, instead of converting to float64)
There was a problem hiding this comment.
I like your idea of checking nans more broadly. So currently, if we are setting nan into Int64 it gets converted to pd.NA?
There was a problem hiding this comment.
Yes, for example:
In [7]: s = pd.Series([1, 2, 3], dtype="Int64")
In [8]: s[0] = np.nan
In [9]: s
Out[9]:
0 <NA>
1 2
2 3
dtype: Int64
There was a problem hiding this comment.
Wasn’t aware of this. Will adjust accordingly. You are correct, this should be consistent
There was a problem hiding this comment.
Could you have another look @jorisvandenbossche? I tried to add relevant cases for enlargement and non-enlargement to ensure consistency. Let me know if there is something missing.
There is one open case:
ser = pd.Series([1, 2], dtype="Int64")
ser[1] = "a"
This raises while expansion casts to object
ser = pd.Series([1, 2], dtype="Int64")
ser[2] = "a"
This is true for rhs="a" and rhs=pd.NaT. With non-ea dtypes we are casting to object. Do we want to be consistent here or is the difference intended?
|
thanks @phofl |
|
Thx, I'll open a follow up for the ea case with non matching dtypes |
doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.We have to preserve the dtype here. This only fixes this case for Series, not for DataFames