ENH: raise limit for completion number of columns and warn beyond

[bthyreau]

Hi,

Currently the user-completion (ipython "<tab>") on a dataframe with more than 100 columns will silently ignore some columns, letting an unaware user confused on whether data disapeared.

This is actually documented, and due to an arbitrary limit set to workaround a performance issue ( See #18587 )

Dataframe with more than 100 columns are quite common, so this can potentially affect and suprise many users. Therefore, I suggest to increase that limit to, say, 1000. In any case, it would probably be good to warn the user hitting that limit.

The attached quickfix raises the limit to 1000 and adds a warning beyond.
(Note that I didn't experience any completion latency increasing with the axis size, so I'm not sure whether this limit is still relevant in the first place).

[simonjayhawkins]

Thanks @bthyreau for the PR.

The attached quickfix raises the limit to 1000 and adds a warning beyond.

This is still an arbitrary limit. adding a warning is maybe a good idea. Maybe this could be a pandas option to allow users to change this if required.

It maybe better to raise a feature request for more discussion on this topic.

[arw2019]

I opened #37996 to house the discussion about this.

Thanks @bthyreau for bringing this up! We of course welcome other PRs on this (once there's a consensus on the way forward) or any other issues!