ENH: Remove ArrowStringArray and StringDtype("pyarrow")

[gsheni]

Feature Type

• Adding new functionality to pandas
• Changing existing functionality in pandas
• Removing existing functionality in pandas

Problem Description

I wish I could use pandas to create pyarrow backend Series for strings.
I wish there was a single data type and single extension array for strings (rather than 2).

Currently, we have 2 pyarrow data types & arrays for strings

• StringDtype("pyarrow") backend by arrays.ArrowStringArray
• ArrowDtype(pa.string()) backend by arrays.ArrowExtensionArray

I propose we use ArrowDtype(pa.string()) and ArrowExtensionArray.

Feature Description

import pyarrow as pa
import pandas as pd 

series_str_arry = pd.Series(['red', 'blue', None], dtype="string[pyarrow]")

string_ext_arry = pd.ArrowDtype(pa.string())
series_ext_arry = pd.Series(['red', 'blue', None], dtype=string_ext_arry)

assert series_str_arry.dtype == series_ext_arry.dtype
assert series_str_arry.dtype.construct_array_type() == series_ext_arry.dtype.construct_array_type()

Alternative Solutions

• Keep both data types and arrays

Additional Context

• Docs: https://pandas.pydata.org/docs/dev/reference/arrays.html#pyarrow
• Related PR: TST: Fix unsigned pyarrow types in SIGNED_INT_PYARROW_DTYPES #48457 (comment)

[mroeschke]

Noting that there was discussion to not immediately deprecate StringDtype("pyarrow") when ArrowDtype was introduced.

Also ArrowExtensionArray would need to adopt string manipulation methods for them to be equivalent.

Nonetheless, agreed that ArrowDtype should be the primary way to use pyarrow string types in the future

[jorisvandenbossche]

As I mentioned in #47818 (review), I am personally -1 on moving away from StringDtype(storage="arrow").
I think we should rather consider moving ArrowDtype towards the same model as StringDtype (and I am meaning the user-facing API, not necessarily implementation wise).

[jbrockmendel]

As I mentioned in #47818 (review), I am personally -1 on moving away from StringDtype(storage="arrow").
I think we should rather consider moving ArrowDtype towards the same model as StringDtype (and I am meaning the user-facing API, not necessarily implementation wise).

Is the idea here to introduce storage parameter/option/... for other dtypes so that "int64" could alias to one of, say, "int64[pyarrow]", "int64[masked]", "int64[numpy]"?

[jbrockmendel]

What if we used ArrowDtype(pa.string()) under the hood but patched StringDtype.__instancecheck__ to return True for ArrowDtype(pa.string())?

[thesword53]

The real issue is string[pyarrow] is pointing to StringDtype("pyarrow") instead of ArrowDtype(pa.string()). However str[pyarrow] is pointing to ArrowDtype(pa.string()), and these two types are behaving differently:

>>> s = pd.Series(["a,b,c", "c,d"], dtype="str[pyarrow]")
>>> s.str.split(",")
0    ['a' 'b' 'c']
1        ['c' 'd']
dtype: list<item: string>[pyarrow]

>>> s = pd.Series(["a,b,c", "c,d"], dtype="string[pyarrow]")
>>> s.str.split(",")
0    [a, b, c]
1       [c, d]
dtype: object

str[pyarrow] and string[pyarrow] should be the same type and the current string[pyarrow] (StringDtype("pyarrow")) should be renamed in something like string_pyarrow.
Every dtypes ending with [pyarrow] should use ArrowDtype.

[simonjayhawkins]

I'm going to close this issue since we are not going to "Remove ArrowStringArray and StringDtype("pyarrow")" anytime soon. Conversely, it will effectively become the default, albeit a variant using nan semantics, in pandas 3.0.

There maybe a few points raised in this discussion that participants may feel have not been addressed, but I think they can be opened as separate issues if needed.