Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: Fix #57608: queries on categorical string columns in HDFStore.select() return unexpected results. #61225

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

BUG: Fix #57608: queries on categorical string columns in HDFStore.select() return unexpected results. #61225

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
55dc32d
BUG: Fix #57608: queries on categorical string columns in
SofiaSM45 Apr 3, 2025
386b566
BUG: Follow-up for #57608: check if metadata is sorted before search
SofiaSM45 Apr 4, 2025
6243e26
BUG: Follow-up for #57608: use direct match via np.flatnonzero
SofiaSM45 Apr 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v3.0.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -736,6 +736,7 @@ I/O
- Bug in :meth:`DataFrame.to_stata` when writing more than 32,000 value labels. (:issue:`60107`)
- Bug in :meth:`DataFrame.to_string` that raised ``StopIteration`` with nested DataFrames. (:issue:`16098`)
- Bug in :meth:`HDFStore.get` was failing to save data of dtype datetime64[s] correctly (:issue:`59004`)
- Bug in :meth:`HDFStore.select` causing queries on categorical string columns to return unexpected results (:issue:`57608`)
- Bug in :meth:`read_csv` causing segmentation fault when ``encoding_errors`` is not a string. (:issue:`59059`)
- Bug in :meth:`read_csv` raising ``TypeError`` when ``index_col`` is specified and ``na_values`` is a dict containing the key ``None``. (:issue:`57547`)
- Bug in :meth:`read_csv` raising ``TypeError`` when ``nrows`` and ``iterator`` are specified without specifying a ``chunksize``. (:issue:`59079`)
Expand Down
8 changes: 7 additions & 1 deletion pandas/core/computation/pytables.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -239,7 +239,13 @@ def stringify(value):
if conv_val not in metadata:
result = -1
else:
result = metadata.searchsorted(conv_val, side="left")
# Check if metadata is sorted
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well we probably won't want to do this check here because this also incurs some performance penalty. I was just staying if there's something in the preprocessing code above that already checked this for us.

If not, just using np.flatnonzero here directly is fine

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn’t look like there’s a check to ensure the metadata is ordered — this part seems to work with just the array of the category values, so it may not be able to confirm whether it was ordered

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK thanks. You can revert this to how you had it before

if np.all(metadata[:-1] <= metadata[1:]):
# If it is, use searchsorted for efficient lookup
result = metadata.searchsorted(conv_val, side="left")
else:
# Find the index of the first match of conv_val in metadata
result = np.flatnonzero(metadata == conv_val)[0]
return TermValue(result, result, "integer")
elif kind == "integer":
try:
Expand Down
23 changes: 23 additions & 0 deletions pandas/tests/io/pytables/test_store.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,9 @@
timedelta_range,
)
import pandas._testing as tm
from pandas.api.types import (
CategoricalDtype,
)
from pandas.conftest import has_pyarrow
from pandas.tests.io.pytables.common import (
_maybe_remove,
Expand Down Expand Up @@ -1106,3 +1109,23 @@ def test_store_bool_index(tmp_path, setup_path):
df.to_hdf(path, key="a")
result = read_hdf(path, "a")
tm.assert_frame_equal(expected, result)


@pytest.mark.parametrize("model", ["name", "longname", "verylongname"])
def test_select_categorical_string_columns(tmp_path, model):
# Corresponding to BUG: 57608

path = tmp_path / "test.h5"

models = CategoricalDtype(categories=["name", "longname", "verylongname"])
df = DataFrame(
{"modelId": ["name", "longname", "longname"], "value": [1, 2, 3]}
).astype({"modelId": models, "value": int})

with HDFStore(path, "w") as store:
store.append("df", df, data_columns=["modelId"])

with HDFStore(path, "r") as store:
result = store.select("df", "modelId == model")
expected = df[df["modelId"] == model]
tm.assert_frame_equal(result, expected)
Loading