BUG: Negation of .str.isnumeric() changes dtype when pd.NA is present

[noahblakesmith]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

s = pd.Series(["", "0", "123", " 123", pd.NA])
print(s.str.isnumeric())
print(~s.str.isnumeric())

t = pd.Series(["", "0", "123", " 123"])
print(t.str.isnumeric())
print(~t.str.isnumeric())

Issue Description

When pd.NA is present in a Series object, negating the .str.isnumeric() method changes bool values to int values.

Expected Behavior

Negation should adhere to the Kleene logic implemented elsewhere in pandas.

Installed Versions

INSTALLED VERSIONS

commit : 0691c5c
python : 3.10.16
python-bits : 64
OS : Linux
OS-release : 6.8.0-1021-azure
Version : #25-Ubuntu SMP Wed Jan 15 20:45:09 UTC 2025
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : C.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.2.3
numpy : 2.2.2
pytz : 2025.1
dateutil : 2.9.0.post0
pip : 25.0
Cython : None
sphinx : None
IPython : 8.34.0
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.13.3
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : 2025.3.0
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : 3.1.6
lxml.etree : 5.3.1
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : 3.1.5
pandas_gbq : None
psycopg2 : None
pymysql : None
pyarrow : 19.0.1
pyreadstat : None
pytest : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : 2.0.39
tables : None
tabulate : None
xarray : None
xlrd : 2.0.1
xlsxwriter : None
zstandard : None
tzdata : 2025.1
qtpy : None
pyqt5 : None

[rhshadrach]

Thanks for the report! Your input to isnumeric is object dtype, and so you get object dtype back. Thus the results are Python's True and False. The behavior pandas displays here is then consistent with the operations on the Python objects:

print(~True)
# -2

If you would like Kleene-logic, then you should likely specify dtype=pd.StringDtype() or dtype="string".

Edit: The remainder of this comment is misleading, see below.

When doing this, I'm seeing isnumeric come out at False for pd.NA. It's not clear to me whether or not that is the proper result.

pd.set_option("infer_string", True)
s = pd.Series(["", pd.NA])
print(s.str.isnumeric())
# 0    False
# 1    False
# dtype: bool

cc @jorisvandenbossche @WillAyd @mroeschke

[noahblakesmith]

Thanks for the response, @rhshadrach. What you said makes sense.

However, using pd.set_option("infer_string", True) raises an error upon negation when pd.NA is present:

pd.set_option("infer_string", True)

s = pd.Series(["", pd.NA])
print(~s.str.isnumeric())
# TypeError: bad operand type for unary ~: 'float'

t = pd.Series([""])
print(~t.str.isnumeric())
# 0    False
# dtype: bool
# 0    True
# dtype: bool

[simonjayhawkins]

Negation should adhere to the Kleene logic implemented elsewhere in pandas.

Not sure that should be the case here with the legacy object array. pd.NA, although a python object that can be held by an object array, since it can hold any Python object is not the internal representation of a missing value.

I would suspect that s.str.isnumeric() should probably return False for pd.NA in the legacy object array since the Kleene logic is appropriate to nullable arrays only. The return value would then be a bool array which give the expected result for the logical negation.

If we wanted to propagate the pd.NA value for the legacy object array we would need to return a pandas nullable boolean array which would mean the return type of s.str.isnumeric() is value dependent which is something we try to avoid.

So it appears that the return type being a object array when pd.NA is present is a bug. Thanks for the report.

[WillAyd]

I think there are a few bugs wrapped up in this discussion. To clarify, the behavior you are looking for is achievable when you use the pd.StringDtype(), which is naturally backed by pd.NA and follows Kleene logic:

>>> ser = pd.Series(["", pd.NA], dtype=pd.StringDtype())
>>> ~ser.str.isnumeric()
0    True
1    <NA>
dtype: boolean

The "infer_string" option uses that same pd.StringDtype but with np.nan as the missing value indicator (i.e. dtype=pd.StringDtype(na_value=np.nan)), which does not follow Kleene logic.

[rhshadrach]

Thanks @WillAyd - I've edited my comment above. So I think we're good here; my example above should have been:

pd.set_option("infer_string", True)
s = pd.Series(["", np.nan])
print(s.str.isnumeric())
# 0    False
# 1    False
# dtype: bool

With NaN, I think we want this to come out as False as it does today instead of propagating the nan value. Does that sound right?

[WillAyd]

Hmm that's tricky. I think if you were to just evaluate the result of an inversion with np.nan, it would be strange to get False back, especially since that is a lossy inversion. However, if the idea is that the inversion is strictly going to be used as an indexer, then it would be helpful to do that in line with work like #59616

Generally there isn't a universal solution to a problem like this using np.nan as a missing value indicator, so if the OP is looking for Kleene logic I would advise staying away from np.nan altogether

[simonjayhawkins]

not sure why this is closed.

the docs https://pandas.pydata.org/docs/reference/api/pandas.Series.str.isnumeric.html state that the return type is "Series or Index of boolean values with the same length as the original Series/Index.". All the examples show dtype: bool

here we have dtype: object Series returned when pd.NA is present in the Series of object dtype.

s = pd.Series(["", "0", "123", " 123", pd.NA])
print(s)
print(s.str.isnumeric())
# 0        
# 1       0
# 2     123
# 3     123
# 4    <NA>
# dtype: object
# 0    False
# 1     True
# 2     True
# 3    False
# 4     <NA>
# dtype: object

Surely this is a bug?

The expected result of s.str.isnumeric() for an object dtype would be...

# 0    False
# 1     True
# 2     True
# 3    False
# 4    False
# dtype: bool

so that the result could be used as an indexer and that logical negation would work as expected?

I would not expect <NA> to be propagated in an object dtype otherwise the return type would have to be a nullable pandas Boolean dtype (and not the object type) and this would result in non default pandas dtypes being presented to the user?

[rhshadrach]

Thanks @simonjayhawkins - I missed your previous comment. Reopening.

However it's not clear to me that the propagation of the NA value in object dtype is a bug. I'd guess it's likely that the documentation of isnumeric was written without specifically thinking of NA, but could be wrong here. Has NA behavior in methods like these been discussed in the past?

No strong opinion on my side.

I would not expect <NA> to be propagated in an object dtype otherwise the return type would have to be a nullable pandas Boolean dtype (and not the object type)

Why not object?

[simonjayhawkins]

Why not object?

So it appears that object dtype is returned using the string accessor on object series containing non-string values...

s = pd.Series(["", "0", "123", " 123", 123])
print(s)
print(s.str.isnumeric())
# 0        
# 1       0
# 2     123
# 3     123
# 4     123
# dtype: object
# 0    False
# 1     True
# 2     True
# 3    False
# 4      NaN
# dtype: object

In this case I would have expected the numeric 123 value to also be False and a boolean array returned since the str accessor should only be operating on the string values IMO.

So this does not look like an inconsistency arising specifically from having the pd.NA object as a value in a object dtype Series.

No strong opinion on my side.

Agree. If returning an object array when using the str accessor on object arrays that have some non-string values is long standing behavior then I'm inclined to not consider the pd.NA being persevered a bug at this time.