REGR: 1.3 behavior change with groupby and asindex=False

[aberres]

Code Sample, a copy-pastable example

pd.DataFrame(columns=["date", "crew_member_id"])
    .groupby(["date", "crew_member_id"], as_index=False)
    .sum()

Problem description

With 1.3rc1 this fails with ValueError: Length of values (0) does not match length of index (1) while 1.2 returns and empty frame.

Expected Output

If this is the expected behavior fine. Just wanted to ensure that it is not an accidental regression.

[simonjayhawkins]

Thanks @aberres for the report

With 1.3rc1 this fails with ValueError: Length of values (0) does not match length of index (1) while 1.2 returns and empty frame.

first bad commit: [48a5853] REF: preserve Index dtype in BlockManager._combine (#41354)

before this commit the result was

2021-06-14T12:25:54.6429723Z 1.3.0.dev0+1555.g3cce96f515
2021-06-14T12:25:54.6430581Z Empty DataFrame
2021-06-14T12:25:54.6430967Z Columns: [date, crew_member_id]
2021-06-14T12:25:54.6431332Z Index: []
2021-06-14T12:25:54.7091204Z Bisecting: 1 revision left to test after this (roughly 1 step)

which is consistent with the result from a non-empty DataFrame, unlike the result on 1.2.4 which was an empty DataFrame with no columns

cc @jbrockmendel

[phofl]

Yep my ci picked that up as well. Should we label this as a regression @simonjayhawkins ?

[simonjayhawkins]

sure. I left it in the needs triage state for another set of eyes as I only bisected the change of behavior. If we use a populated DataFrame in the code sample instead of an empty one. the result does not make much sense to me.

[phofl]

This one fails too:

df = pd.DataFrame(columns=["a", "b"]).groupby(["a"], as_index=False).sum()

The initial code snippet seems a bit artificial to me, mostly because I don't know what to actually expect. But this one returned

Empty DataFrame
Columns: [a, b]
Index: []

on 1.2.4 which looks sensible to me.
Labelling as a regression

[jbrockmendel]

ive got it on my todo list (i.e. an open browser tab) to look at this, but nothing obvious off the top of my head

[rhshadrach]

The method _wrap_agged_manager uses mgr.shape[1] for the length of the index, it seems to me it should be using

mgr.shape[1] if not mgr.shape[0] == 0 else 0

This fixes the issue in the OP and gives the output

Empty DataFrame
Columns: [date, crew_member_id]
Index: []

which seems to me in-line with the result of

(
    pd.DataFrame([[1, 2], [3, 4]], columns=["date", "crew_member_id"])
    .groupby(["date", "crew_member_id"], as_index=False)
    .sum()
)

which gives

   date  crew_member_id
0     1               2
1     3               4

However, for the issue in #41998 (comment), it gives

Empty DataFrame
Columns: [a]
Index: []

which to me looks wrong - as @phofl said we should also be getting the column 'b'. I don't this the column information in the mgr passed to _wrap_agged_manager, so maybe the fix needs to happen more upstream.

[rhshadrach]

Ah - this path is dropping column 'b' because it is object dtype. Doing

print(pd.DataFrame(columns=["a", "b"], dtype=int).groupby(["a"], as_index=False).sum())

gives

Empty DataFrame
Columns: [a, b]
Index: []

which is better. Running tests now, PR to follow.