pandas slow replace with int64 columns in dataframe

[apiszcz]

Reproduced issue in part without n2 and n3 and with n2 n3 see prior issue:
#12257

The replace runs 10x slower when encountering int64 columns

WITHOUT n2,n3, 1.73 seconds

a1=np.zeros((40000000,1))
a1[:,:]=np.inf
df=pd.DataFrame(a1)
df['a1']=''
df['n1']=0.0
# df['n2']=0.0
# df['n2']=df['n2'].astype(np.int64)
# df['n3']=0.0
# df['n3']=df['n3'].astype(np.int64)
df['n1'].astype('datetime64[ns]')
df['a1']=df['a1'].astype('category')
%time df.replace([np.inf, -np.inf], np.nan)

WITH n2, n3 21.4 seconds

a1=np.zeros((40000000,1))
a1[:,:]=np.inf
df=pd.DataFrame(a1)
df['a1']=''
df['n1']=0.0
df['n2']=0.0
df['n2']=df['n2'].astype(np.int64)
df['n3']=0.0
df['n3']=df['n3'].astype(np.int64)
df['n1'].astype('datetime64[ns]')
df['a1']=df['a1'].astype('category')
%time df.replace([np.inf, -np.inf], np.nan)

[jbrockmendel]

Looks like in core.internals.blocks.replace there is a path form handling isinstance(to_replace, list) that could be optimized significantly