Improve performance of equality comparison between a simple Index and a MultiIndex

[tlaytongoogle]

Previously, Index.equals(MultiIndex) and MultiIndex.equals(Index) both involved converting the MultiIndex into an ndarray. Since this conversion requires resolving MultiIndex's by-reference structure (i.e. its codes + levels) into ndarray's by-value structure, it can be substantially computationally expensive.

However, it is only possible for a simple Index to equal a MultiIndex in two cases:

• the MultiIndex has only 1 level
• the MultiIndex has d levels, and the Index is an object index of size-d sequences (e.g. d-tuples)

Thus, if the Index is not object-typed, and its nlevels differs from that of the MultiIndex, then the two are determined to be unequal without ndarray conversion.

MWE:

import pandas as pd

long_cheap_index = pd.RangeIndex(1000000)
short_expensive_index = pd.IntervalIndex(
    [pd.Interval(pd.Timestamp(2018, 10, 1), pd.Timestamp(2018, 10, 2))])

large_expensive_multiindex = pd.MultiIndex.from_product(
    [long_cheap_index, short_expensive_index])
trivial_simple_index = pd.Int64Index([])

# These operations no longer convert large_expensive_multiindex into an ndarray
# Previously, each took ~10 s; now, ~0.02 ms
large_expensive_multiindex.equals(trivial_simple_index)
trivial_simple_index.equals(large_expensive_multiindex)

• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff

[jreback]

can you add an asv for this and a whatnew note

[jreback]

can you add a note to the whatsnew section (performance)

[jreback]

@tlaytongoogle can you update, this looks good.

[pep8speaks]

Hello @tlaytongoogle! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2019-12-05 16:56:31 UTC

[jreback]

lgtm. comment on the benchmark. can you update, ping on green.

[jreback]

thanks @tlaytongoogle very nice!