Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: un-caught date_range overflows #24123

Closed
jbrockmendel opened this issue Dec 6, 2018 · 0 comments · Fixed by #24255
Closed

BUG: un-caught date_range overflows #24123

jbrockmendel opened this issue Dec 6, 2018 · 0 comments · Fixed by #24255
Labels
Bug Datetime Datetime data dtype
Milestone

Comments

@jbrockmendel
Copy link
Member

jbrockmendel commented Dec 6, 2018

TL;DR: can we assume that np.float128 exists (and is actually 128 bits)? If so then this has an easy solution. Otherwise, it's easy-but-tedious. update Turns out the np.float128 solution doesn't as easily as I expected.

start = pd.Timestamp('1700-01-01')
end = pd.Timestamp('2250-12-31')

dti = pd.date_range(start, end, freq='D')

# alternative construction that _should_ work

dti2 = pd.date_range(start, periods=len(dti), freq='D')   # raises OutOfBoundsDatetime
dti3 = pd.date_range(end=end, periods=len(dti), freq='D')   # raises OutOfBoundsDatetime

It's actually worse than that, because it is raising the wrong OutOfBoundsDatetime. What is happening here is that in arrays.datetimes._generate_range_overflow_safe we are doing:

    try:
        other_end = checked_add_with_arr(np.int64(endpoint),
                                         np.int64(periods) * stride)
    except OverflowError:
        raise tslib.OutOfBoundsDatetime('Cannot generate range with '

In these cases, np.int64(periods) * stride is wrapping around.

The good news is that this is fairly easy to fix by casting to np.uint64 instead of np.int64. The bad news is that when we call checked_add_with_arr, if the arguments have mismatched dtypes, then it gives back a float64, and we can get an incorrect result when casting back to int64.

Obvious next step: OK, so let's pass args with matching dtypes. if endpoint is positive we can cast it to uint64 and we're done. Otherwise [break down into 5 cases..., straightforward but tedious]

The alternative is to just cast to np.float128, do both the multiplication and addition there, then check for overflows before casting back to int64. But IIRC some (windows) platforms don't have a "real" float128. Is that a sufficiently ancient case that it can be ignored?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Datetime Datetime data dtype
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants