Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: Inconsistent dtype with GroupBy for StrDtype and all missing values #60810

Open
3 tasks done
WillAyd opened this issue Jan 28, 2025 · 1 comment · May be fixed by #60985
Open
3 tasks done

BUG: Inconsistent dtype with GroupBy for StrDtype and all missing values #60810

WillAyd opened this issue Jan 28, 2025 · 1 comment · May be fixed by #60985
Assignees
Labels
Bug Groupby Strings String extension data type and string data
Milestone

Comments

@WillAyd
Copy link
Member

WillAyd commented Jan 28, 2025

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

>>> df = pd.DataFrame({"a": ["a"] * 3, "b": pd.Series([None] * 3, dtype=pd.StringDtype(na_value=np.nan))})
>>> df
   a    b
0  a  NaN
1  a  NaN
2  a  NaN
>>> df.groupby("a").sum()
   b
a   
a  0
>>> df.groupby("a").sum().dtypes
b    str
dtype: object
>>> df.groupby("a").min()
    b
a    
a NaN
>>> df.groupby("a").min().dtypes
b    float64
dtype: object

Issue Description

The sum reduction return type is partially discussed in #60229 but I didn't see anything for min

Note that this discrepancy is the root cause of the test failure shown at

@pytest.mark.xfail(using_string_dtype(), reason="TODO(infer_string)")

@rhshadrach

Expected Behavior

I think in all cases here we should still be returning a str type.

Installed Versions

'3.0.0.dev0+1824.g8d6d29cac3.dirty'

@WillAyd WillAyd added Bug Groupby Strings String extension data type and string data labels Jan 28, 2025
@rhshadrach
Copy link
Member

rhshadrach commented Feb 22, 2025

Thanks @WillAyd - taking this up now. I also checked out all other reductions, it appears only min and max have incorrect behavior.

One I'll note is that idxmin and idxmax return float NaN, but I think this is expected. This is because these are returning index values which are integers. When one changes the index to string, we do indeed return string dtype as well.

pd.set_option("infer_string", True)
df = pd.DataFrame({"a": "a", "b": pd.array(3*[None], dtype="str")}, index=["x", "y", "z"])
print(df.groupby("a").idxmin())
#      b
# a     
# a  NaN

print(df.groupby("a").idxmin().dtypes)
# b    str
# dtype: object

@rhshadrach rhshadrach self-assigned this Feb 22, 2025
@rhshadrach rhshadrach added this to the 2.3 milestone Feb 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Groupby Strings String extension data type and string data
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants