TYP: Update type hints for ExtensionArray and ExtensionDtype #39501

Dr-Irv · 2021-01-31T14:54:12Z

Modifications to reduce error reports from pyright --verifytypes related to ExtensionArray and ExtensionDtype.

Some notes:

Had to use Any in a number of places, because an ExtensionArray can contain any type
Because numpy doesn't have a released version with types, pyright complains whenever we work with numpy types, so for type checking purposes, I redefine np.dtype and np.ndarray to a dummy type. When numpy 1.20 is released, we can most likely remove that. Alternatively, I can find out if there is a way to see if pyright has something like TYPE_CHECKING to bracket those changes.

I anticipate these changes might be controversial, so constructive criticism is welcome!

MarcoGorelli · 2021-01-31T15:27:45Z

-        method: Optional[Literal["backfill", "bfill", "ffill", "pad"]] = None,
+        method: Optional[Literal[backfill, bfill, ffill, pad]] = None,

Looks like a bug in no-string-hints, sorry about that, will test + update!

Have updated and pushed a new tag, the version in pandas' pre-commit-config will get updated tomorrow morning as part of the scheduled weekly update

simonjayhawkins · 2021-02-01T12:16:16Z

2. When numpy 1.20 is released

@Dr-Irv can you update

Dr-Irv · 2021-02-01T12:53:12Z

When numpy 1.20 is released

@Dr-Irv can you update

Ha! It was just released in pypl on 1/30/21, which is when I pushed this! Waiting for it to be on conda and then will update.

Dr-Irv · 2021-02-01T18:32:36Z

When numpy 1.20 is released

@Dr-Irv can you update

I did try this, and now mypy pandas reports more errors because numpy 1.20 has typing. #39513

I'm asking the pyright people to see if there is a different workaround. Ideally, I'd like to tell pyright "process pandas, but ignore numpy"

Dr-Irv · 2021-02-02T16:31:08Z

I'm asking the pyright people to see if there is a different workaround. Ideally, I'd like to tell pyright "process pandas, but ignore numpy"

You can now use pyright to find pandas symbols that are missing types by using pyright --verifytypes pandas! (The ! means don't look at imported libraries). That means it won't look at the numpy imports. Based on that (and including this PR), we have 75.7% type completeness for pandas.

Dr-Irv · 2021-02-03T03:22:00Z

Looks like a bug in no-string-hints, sorry about that, will test + update!

Have updated and pushed a new tag, the version in pandas' pre-commit-config will get updated tomorrow morning as part of the scheduled weekly update

@MarcoGorelli Has this been done yet? I'm still getting the pre-commit failure

MarcoGorelli · 2021-02-03T09:02:28Z

Looks like a bug in no-string-hints, sorry about that, will test + update!
Have updated and pushed a new tag, the version in pandas' pre-commit-config will get updated tomorrow morning as part of the scheduled weekly update

@MarcoGorelli Has this been done yet? I'm still getting the pre-commit failure

if you approve/merge #39524 and then rebase/merge, then it'll pass

pandas/core/arrays/base.py

github-actions · 2021-03-06T00:13:26Z

This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.

Dr-Irv · 2021-03-12T20:48:40Z

@simonjayhawkins Updated the PR based on your latest stuff with typing, so now pandas/core/arrays/base.py is fully typed and that allowed me to remove a bunch of # type; ignore things. Main changes from last review is an overload for astype and fixing isin. Looking forward to your review.

simonjayhawkins · 2021-03-13T14:35:08Z

Thanks @Dr-Irv for working on this.

I would imagine that this PR will be easier to check if we limit the changes to pandas/core/arrays/base.py and the other changes are done in a separate PR(s).

There is a conflict here and I think some other overlaps with other PRs removing the ignores.

some of the changes to remove ignores need more discussion than others. If ignores are being removed that's great, if ignores or changed or others added we need to review in more depth.

pandas/core/algorithms.py

pandas/core/arrays/datetimelike.py

simonjayhawkins · 2021-03-13T16:01:41Z

I'll push a commit merging master.

some new mypy errors and now also another conflict from a subsequent merge to master related to #39501 (comment)

Dr-Irv · 2021-03-13T16:11:38Z

I would imagine that this PR will be easier to check if we limit the changes to pandas/core/arrays/base.py and the other changes are done in a separate PR(s).

I'm not sure that it can be done alone, because pandas/core/arrays/base.py has the base ExtensionArray class, so you have to fix things upstream when fixing things there.

There is a conflict here and I think some other overlaps with other PRs removing the ignores.

some of the changes to remove ignores need more discussion than others. If ignores are being removed that's great, if ignores or changed or others added we need to review in more depth.

I removed ignores only based on when mypy said they were not needed any more. I don't think I added any ignores.

Let me get the current version in a state that goes green.

simonjayhawkins · 2021-03-13T16:24:47Z

I'm not sure that it can be done alone, because pandas/core/arrays/base.py has the base ExtensionArray class, so you have to fix things upstream when fixing things there.

I think that while numpy types resolved to Any this PR adding types to pandas/core/arrays/base.py was moreless ready.

all the other changes are related to numpy types now having some meaning. I'm sure these could be done separately (and in smaller chunks)

changes to 36 files takes some review, especially as changes are made as other PRs are merged.

typing PRs need to be atomic to be easily reviewed.

Main changes from last review is an overload for astype and fixing isin. Looking forward to your review.

so maybe a PR for the astype fixes and a PR for the isin fixes?

Dr-Irv · 2021-03-13T16:31:10Z

all the other changes are related to numpy types now having some meaning. I'm sure these could be done separately (and in smaller chunks)

If I limit things to pandas/core/arrays/base.py, I'm pretty sure I can't get it to be green. Is that OK?

Dr-Irv · 2021-03-13T16:33:18Z

@simonjayhawkins Latest commit 5bb24d4 still has a variety of changes. It just takes what you merged in and should make it mypy happy.

Dr-Irv · 2021-03-13T16:47:48Z

so maybe a PR for the astype fixes and a PR for the isin fixes?

I'm going to create a separate PR that is effectively this one, MINUS the astype and isin.

Dr-Irv · 2021-03-14T14:59:14Z

jbrockmendel · 2021-03-28T16:00:57Z

Things that can be done independent of approval of #40421

Looks like you've made some progress on that checklist. Is this PR still active?

Dr-Irv · 2021-03-28T16:19:08Z

Things that can be done independent of approval of #40421

Looks like you've made some progress on that checklist. Is this PR still active?

I will change it to draft status, as I'm using it to track the changes I would do after #40421 is approved. Then when I finish the checklist, I'll close this one.

simonjayhawkins · 2021-05-22T15:48:35Z

I will change it to draft status, as I'm using it to track the changes I would do after #40421 is approved. Then when I finish the checklist, I'll close this one.

Is this still the master PR? if so could you merge master so we can monitor progress. and more easily see the outstanding review comments.

Dr-Irv · 2021-05-23T18:21:43Z

I will change it to draft status, as I'm using it to track the changes I would do after #40421 is approved. Then when I finish the checklist, I'll close this one.

Is this still the master PR? if so could you merge master so we can monitor progress. and more easily see the outstanding review comments.

Yes. But I'd prefer to do the merge once #41203, #41251, #41258 and #41513 are merged in so we can see where we are with respect to the comments.

simonjayhawkins · 2021-05-24T19:48:46Z

ok. will close for now. reopen when ready.

Dr-Irv added 3 commits January 23, 2021 11:33

small typing fixes

f2c52a4

fix ExtensionArray and EXtensionDtype

d7ff8d3

merge with master

49fa06e

Dr-Irv added 2 commits January 31, 2021 12:23

fixes for delete, isin, unique

03b2c4a

fix import of Literal

3e19958

Dr-Irv changed the title ~~Update type hints for ExtensionArray and ExtensionDtype~~ TYP: Update type hints for ExtensionArray and ExtensionDtype Jan 31, 2021

remove quotes on ExtensionDType.construct_from_string

6861901

MarcoGorelli mentioned this pull request Feb 1, 2021

STYLE Upgrade no-string-hints #39524

Merged

move numpy workaround to _typing.py

9be6486

jreback added ExtensionArray Extending pandas with custom dtypes or arrays. Typing type annotations, mypy/pyright type checking labels Feb 2, 2021

Dr-Irv added 2 commits February 2, 2021 11:26

remove numpy dummy

260b367

remove extra line in _typing

6276725

Dr-Irv added 2 commits February 3, 2021 08:06

Merge remote-tracking branch 'upstream/master' into extensiontyping

4dafaca

import Literal

8b2cee2

MarcoGorelli reviewed Feb 3, 2021

View reviewed changes

pandas/core/arrays/base.py Outdated Show resolved Hide resolved

Merge remote-tracking branch 'upstream/master' into extensiontyping

3a7d839

github-actions bot added the Stale label Mar 6, 2021

merge with master

a21bb60

Dr-Irv removed the Stale label Mar 8, 2021

Dr-Irv added 2 commits March 8, 2021 10:58

isort precommit fix

8cd6b76

fix interval.repeat() typing

e0e0131

Dr-Irv added 5 commits March 12, 2021 15:55

fix Literal import

db8ed9b

fix logic in ensure_int_or_float

f8191f8

fix conflict with master

575645f

fix typing in groupby to_numpy call

6f8fcb5

fix groupby again. Allow kwargs for extension to_numpy

3ea2420

Merge remote-tracking branch 'upstream/master' into extensiontyping

c83a628

simonjayhawkins reviewed Mar 13, 2021

View reviewed changes

pandas/core/algorithms.py Outdated Show resolved Hide resolved

simonjayhawkins reviewed Mar 13, 2021

View reviewed changes

pandas/core/arrays/datetimelike.py Outdated Show resolved Hide resolved

fixes for merge with master

5bb24d4

Dr-Irv mentioned this pull request Mar 13, 2021

Fix typing for extension arrays and extension dtypes without isin and astype #40421

Closed

This was referenced Mar 20, 2021

TYP: make EA registry private #40538

Merged

TYP: make typing DatetimeLikeScalar a Union instead of TypeVar #40540

Merged

Dr-Irv marked this pull request as draft March 28, 2021 16:19

simonjayhawkins mentioned this pull request Apr 24, 2021

TYP: Fix typehints for ExtensionDtype #41097

Merged

simonjayhawkins closed this May 24, 2021

Dr-Irv deleted the extensiontyping branch February 13, 2023 20:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TYP: Update type hints for ExtensionArray and ExtensionDtype #39501

TYP: Update type hints for ExtensionArray and ExtensionDtype #39501

Dr-Irv commented Jan 31, 2021

MarcoGorelli commented Jan 31, 2021 •

edited

Loading

simonjayhawkins commented Feb 1, 2021

Dr-Irv commented Feb 1, 2021

Dr-Irv commented Feb 1, 2021

Dr-Irv commented Feb 2, 2021 •

edited

Loading

Dr-Irv commented Feb 3, 2021

MarcoGorelli commented Feb 3, 2021 •

edited

Loading

github-actions bot commented Mar 6, 2021

Dr-Irv commented Mar 12, 2021

simonjayhawkins commented Mar 13, 2021

simonjayhawkins commented Mar 13, 2021

Dr-Irv commented Mar 13, 2021

simonjayhawkins commented Mar 13, 2021

Dr-Irv commented Mar 13, 2021

Dr-Irv commented Mar 13, 2021

Dr-Irv commented Mar 13, 2021

Dr-Irv commented Mar 14, 2021 •

edited

Loading

jbrockmendel commented Mar 28, 2021

Dr-Irv commented Mar 28, 2021

simonjayhawkins commented May 22, 2021

Dr-Irv commented May 23, 2021

simonjayhawkins commented May 24, 2021

TYP: Update type hints for ExtensionArray and ExtensionDtype #39501

TYP: Update type hints for ExtensionArray and ExtensionDtype #39501

Conversation

Dr-Irv commented Jan 31, 2021

MarcoGorelli commented Jan 31, 2021 • edited Loading

simonjayhawkins commented Feb 1, 2021

Dr-Irv commented Feb 1, 2021

Dr-Irv commented Feb 1, 2021

Dr-Irv commented Feb 2, 2021 • edited Loading

Dr-Irv commented Feb 3, 2021

MarcoGorelli commented Feb 3, 2021 • edited Loading

github-actions bot commented Mar 6, 2021

Dr-Irv commented Mar 12, 2021

simonjayhawkins commented Mar 13, 2021

simonjayhawkins commented Mar 13, 2021

Dr-Irv commented Mar 13, 2021

simonjayhawkins commented Mar 13, 2021

Dr-Irv commented Mar 13, 2021

Dr-Irv commented Mar 13, 2021

Dr-Irv commented Mar 13, 2021

Dr-Irv commented Mar 14, 2021 • edited Loading

jbrockmendel commented Mar 28, 2021

Dr-Irv commented Mar 28, 2021

simonjayhawkins commented May 22, 2021

Dr-Irv commented May 23, 2021

simonjayhawkins commented May 24, 2021

MarcoGorelli commented Jan 31, 2021 •

edited

Loading

Dr-Irv commented Feb 2, 2021 •

edited

Loading

MarcoGorelli commented Feb 3, 2021 •

edited

Loading

Dr-Irv commented Mar 14, 2021 •

edited

Loading