ENH: raise column number limit for user-completion or add warning

[arw2019]

xref PR #35207

Currently the user-completion (ipython "") on a dataframe with more than 100 columns will silently ignore some columns, letting an unaware user confused on whether data disapeared.

This is actually documented, and due to an arbitrary limit set to workaround a performance issue ( See #18587 )

Dataframe with more than 100 columns are quite common, so this can potentially affect and suprise many users. Therefore, I suggest to increase that limit to, say, 1000. In any case, it would probably be good to warn the user hitting that limit.

The attached quickfix raises the limit to 1000 and adds a warning beyond.

xref #35207 (comment)

This is still an arbitrary limit. adding a warning is maybe a good idea. Maybe this could be a pandas option to allow users to change this if required.

[TheDerivator]

I think I don't have the performance issue but regardless, I would propose to make it optional, and let the value (currently 100) be taken from the options so that

pd.set_option('display.max_dir_items', None)

would remove the cap.

[mzeitlin11]

Would be +1 on adding an option to give some flexibility here