BUG: Setting a pyarrow scalar value with the same type that expands the series changes the dtype into object

[char101]

Pandas version checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of pandas.
• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

s = pd.Series(datetime.now(), dtype='date32[pyarrow]')

s[1] = pa.scalar(datetime.now(), type='date32')

s
Out[60]: 
0    2023-03-27
1    2023-03-27
dtype: object

s = pd.Series([1,2,3], dtype='int32[pyarrow]')

s[3] = pa.scalar(4, type='int32')

s
Out[72]: 
0    1
1    2
2    3
3    4
dtype: object

Issue Description

Adding new pyarrow scalar value of the same type to a series changes its dtype into object. This make it impossible(?) to add to a date32 series without changing its dtype.

Expected Behavior

The series dtype should be kept.

Installed Versions

INSTALLED VERSIONS

commit : 5c15588
python : 3.11.1.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19043
machine : AMD64
processor : Intel64 Family 6 Model 158 Stepping 9, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_Indonesia.1252

pandas : 2.1.0.dev0+284.g5c155883fd
numpy : 1.24.2
pytz : 2022.6
dateutil : 2.8.2
setuptools : 65.6.3
pip : 23.0.1
Cython : 0.29.32
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.9.2
html5lib : 1.1
pymysql : None
psycopg2 : 2.9.5
jinja2 : 3.1.2
IPython : 8.6.0
pandas_datareader: None
bs4 : 4.11.1
bottleneck : 1.3.7
brotli : 1.0.9
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.7.1
numba : None
numexpr : 2.8.4
odfpy : None
openpyxl : 3.1.0
pandas_gbq : None
pyarrow : 11.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.10.1
snappy : None
sqlalchemy : 1.4.45
tables : None
tabulate : None
xarray : None
xlrd : 2.0.1
zstandard : None
tzdata : 2022.6
qtpy : 2.3.0
pyqt5 : None

[mroeschke]

Thanks for the report. This is similar to #52056 and would be greatly aided if/when pyarrow is a required dependency in pandas