ENH: add origin to to_datetime #15828

jreback · 2017-03-28T20:45:10Z

closes #11276
closes #11745

superseded #11470

jreback · 2017-03-28T20:49:24Z

@jorisvandenbossche @chris-b1

ok, rebase and updated original PR. pls have a look.

IIRC there were some more test cases needed, but dont' really remember

chris-b1 · 2017-03-28T21:24:21Z

doc/source/timeseries.rst

+
+   pd.to_datetime([1, 2, 3], unit='D', origin=pd.Timestamp('1960-01-01'))
+
+The default is set at ``origin='epoch'``, which defaults to ``1970-01-01 00:00:00``.


'epoch' seems non-descriptive - in a sense, aren't all origins epochs? Maybe default should be 'unix' like @shoyer had suggested?

Unix time (also known as POSIX time or epoch time) ?

https://en.wikipedia.org/wiki/Unix_time

Certainly the unix epoch date is implied by "epoch time" but it's also a general term, that could refer to epoch timekeeping in general.

https://en.wikipedia.org/wiki/Epoch_(reference_date)#Computing

chris-b1 · 2017-03-28T21:29:25Z

pandas/tseries/tools.py

+        - If 'epoch', origin is set to 1970-01-01.
+        - If 'julian', unit must be 'D', and origin is set to beginning of
+          Julian Calendar. Julian day number 0 is assigned to the day starting
+          at noon on January 1, 4713 BC.


Not critical, but could expand to other semi-common origins @bashtage mentions here.
#11470 (comment)

hmm, maybe, can always add if people really want this.

codecov · 2017-03-28T22:15:31Z

Codecov Report

Merging #15828 into master will decrease coverage by 0.03%.
The diff coverage is 91.11%.

@@            Coverage Diff             @@
##           master   #15828      +/-   ##
==========================================
- Coverage   90.98%   90.95%   -0.04%     
==========================================
  Files         143      143              
  Lines       49449    49464      +15     
==========================================
- Hits        44993    44991       -2     
- Misses       4456     4473      +17

Flag	Coverage Δ
#multiple	`?`
#single	`?`

Impacted Files	Coverage Δ
pandas/tseries/tdi.py	`90.37% <100%> (+0.03%)`	⬆️
pandas/tseries/tools.py	`85.2% <90.24%> (+0.26%)`	⬆️
pandas/io/gbq.py	`25% <0%> (-58.34%)`	⬇️
pandas/io/packers.py	`87.57% <0%> (-0.94%)`	⬇️
pandas/core/common.py	`90.96% <0%> (-0.34%)`	⬇️
pandas/io/html.py	`84.48% <0%> (-0.33%)`	⬇️
pandas/core/series.py	`94.86% <0%> (-0.11%)`	⬇️
pandas/core/frame.py	`97.56% <0%> (-0.1%)`	⬇️
pandas/compat/numpy/function.py	`94.19% <0%> (-0.04%)`	⬇️
pandas/core/indexing.py	`94.08% <0%> (-0.03%)`	⬇️
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 67cc021...ebb4acd. Read the comment docs.

jreback · 2017-03-28T22:30:00Z

ok changed epoch -> unix and updated docs.

jorisvandenbossche · 2017-03-29T08:16:23Z

pandas/tseries/tools.py

    infer_datetime_format : boolean, default False
        If True and no `format` is given, attempt to infer the format of the
        datetime strings, and if it can be inferred, switch to a faster
        method of parsing them. In some cases this can increase the parsing
        speed by ~5-10x.
+    origin : scalar convertible to Timestamp / string ('julian', 'unix'),
+        default 'unix'.


The problem is that this does not work ... type explanation should be on one line (for good html docs) (ref numpy/numpydoc#87)

so pretty much make this shorter then?

jorisvandenbossche · 2017-03-29T08:17:10Z

pandas/tseries/tools.py

@@ -297,8 +312,13 @@ def to_datetime(arg, errors='raise', dayfirst=False, yearfirst=False,
    >>> %timeit pd.to_datetime(s,infer_datetime_format=False)
    1 loop, best of 3: 471 ms per loop

-    """
+    Using non-epoch origins to parse date
+    >>> pd.to_datetime([1,2,3], unit='D', origin=pd.Timestamp('1960-01-01'))


blank line between the two lines above

jorisvandenbossche · 2017-03-29T08:32:49Z

pandas/tseries/tools.py

+                             "to a Timestamp".format(origin))
+
+        if offset is not None:
+            result = result + offset


As I commented in the original PR (#11470 (comment)), I think it can solve some corner cases to do the origin handling before the actual parsing (as you do for the 'julian' case).

A bit an artificial example, but to show one of the problems:

In [11]: pd.to_datetime(200*365, unit='D') Out[11]: Timestamp('2169-11-13 00:00:00') In [12]: pd.to_datetime(200*365, unit='D', origin='1870-01-01') Out[12]: Timestamp('2069-11-13 00:00:00') In [13]: pd.to_datetime(300*365, unit='D', origin='1870-01-01') ... OutOfBoundsDatetime: cannot convert input with unit 'D'

So the last one is actually not an OutOfBounds datetime, but just because we first parse the number as epoch, it raises this error. If we first subtract the correct value from the numeric argument, and then parse as epoch, the above will work fine.

fixed! this was tricky.

jorisvandenbossche · 2017-03-29T08:34:49Z

pandas/tseries/tools.py

+            offset = tslib.Timestamp(origin) - tslib.Timestamp(0)
+        except tslib.OutOfBoundsDatetime:
+            raise ValueError(
+                "origin {} is Out of Bounds".format(origin))


We should add a todo item to solve this

what part, OutOfBounds dates generally?

No, I mean for origin specifically (by circumventing the Timestamp creation), to have something like pd.to_datetime(.., origin='0001-01-01') working if the end timestamp is not out of bounds.

Eg by using dateutil.parser.parse(origin), but then we still need to convert that to the correct offset

jorisvandenbossche · 2017-03-29T08:51:52Z

Some cases where something goes wrong:

This should raise an error (if the origin is first processed and the argument adapted for it, then this will raise an error):
```
In [2]: pd.to_datetime("2005-01-01", origin="1960-01-01")
Out[2]: Timestamp('1995-01-01 00:00:00')
```
Not sure how this can return 12:00 as the hour? (EDIT -> this is probably just the definition? "starting at noon")
```
In [4]: pd.to_datetime(2456658, origin='julian', unit='D')
Out[4]: Timestamp('2013-12-31 12:00:00')
```
The error message in case of out of bounds here:
```
In [3]: pd.to_datetime(1, origin="julian", unit='D')
....
OutOfBoundsDatetime: cannot convert input -2440586.5 with the unit 'D'
```
is a bit unfortunate (the "input -2440586.5"). But that is of course the consequence of processing the origin first and adapting the arg based on that, before passing to the actual parsing methods ... (so that's a clear drawback of this approach I suggested also for the other origins). Not sure whether this could easily be solved ..

jorisvandenbossche · 2017-03-29T08:55:55Z

doc/source/whatsnew/v0.20.0.txt

+to_datetime has gained an origin parameter
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+``pd.to_datetime`` has gained a new parameter, ``origin``, to define an offset


let's use 'reference date' instead of 'offset' here as well (as you did in the docstring)

jreback · 2017-03-29T15:28:29Z

Here are the results with new push.

In [1]: pd.to_datetime(200*365, unit='D')
Out[1]: Timestamp('2169-11-13 00:00:00')

In [2]: pd.to_datetime(200*365, unit='D', origin='1870-01-01')
Out[2]: Timestamp('2069-11-13 00:00:00')

In [3]: pd.to_datetime(300*365, unit='D', origin='1870-01-01')
Out[3]: Timestamp('2169-10-20 00:00:00')

In [4]: pd.to_datetime("2005-01-01", origin="1960-01-01")
ValueError: '2005-01-01' is not compatible with origin='1960-01-01'; it must be numeric with a unit specified 

In [5]: pd.to_datetime(2456658, origin='julian', unit='D')
Out[5]: Timestamp('2013-12-31 12:00:00')

In [6]: pd.to_datetime(1, origin="julian", unit='D')
OutOfBoundsDatetime: 1 is Out of Bounds for origin='julian'

note [1], [2] (were correct before)
[5] seems fine as well.

jorisvandenbossche · 2017-04-02T20:58:31Z

pandas/tseries/tools.py

-    origin : scalar convertible to Timestamp / string ('julian', 'unix'),
-        default 'unix'.
-        Define reference date. The numeric values would be parsed as number
-        of units (defined by `unit`) since this reference date.


I would leave this sentence in, it is a good explanation of what an origin is.

jorisvandenbossche

New changes look good!

jorisvandenbossche · 2017-04-02T21:04:47Z

pandas/tseries/tools.py

+        if np.any(arg > j_max) or np.any(arg < j_min):
+            raise tslib.OutOfBoundsDatetime(
+                "{original} is Out of Bounds for "
+                "origin='julian'".format(original=original))


You can have the same problem with a custom defined origin (not unix of julian)

yes, but that is handled below (this is only for the julian section). I do raise the same error (w/o the julian reference)

elif origin not in ['unix', 'julian']: # arg must be a numeric original = arg if not ((is_scalar(arg) and (is_integer(arg) or is_float(arg))) or is_numeric_dtype(np.asarray(arg))): raise ValueError( "'{arg}' is not compatible with origin='{origin}'; " "it must be numeric with a unit specified ".format( arg=arg, origin=origin)) # we are going to offset back to unix / epoch time try: offset = tslib.Timestamp(origin) - tslib.Timestamp(0) except tslib.OutOfBoundsDatetime: raise tslib.OutOfBoundsDatetime( "origin {} is Out of Bounds".format(origin)) except ValueError: raise ValueError("origin {} cannot be converted " "to a Timestamp".format(origin))

jorisvandenbossche · 2017-04-02T21:07:43Z

pandas/tseries/tools.py

+        # this should be lossless in terms of precision
+        offset = offset // tslib.Timedelta(1, unit=unit)
+
+        arg = np.asarray(arg)


Wouldn't it be simpler to check if it is a list and then convert to array? (or check that is is not a scalar/series/index)

Then the arg = arg + offset does just what you want, and the conversion back as below is not needed

closes pandas-dev#11275 closes pandas-dev#11745

provide out-of-bounds for julian dates

jreback · 2017-04-02T22:55:42Z

@jorisvandenbossche make those changes and merged. thanks for the review.

jorisvandenbossche · 2017-04-03T10:11:01Z

And thanks for picking up the PR!

jreback added Enhancement Datetime Datetime data dtype labels Mar 28, 2017

jreback added this to the 0.20.0 milestone Mar 28, 2017

jreback mentioned this pull request Mar 28, 2017

ENH: Adding origin parameter in pd.to_datetime #11470

Closed

jreback force-pushed the datetime-unit branch from 8265416 to afafe3b Compare March 28, 2017 20:48

jreback mentioned this pull request Mar 28, 2017

Add origin parameter to Timestamp/to_datetime epoch support. #11745

Closed

chris-b1 reviewed Mar 28, 2017

View reviewed changes

jreback force-pushed the datetime-unit branch from afafe3b to 3dce7ea Compare March 28, 2017 22:14

jreback force-pushed the datetime-unit branch from 90347d2 to 109adef Compare March 29, 2017 00:10

jorisvandenbossche reviewed Mar 29, 2017

View reviewed changes

jreback mentioned this pull request Mar 29, 2017

BUG: out-of-bounds datetime parsing #15829

Closed

chris-b1 mentioned this pull request Mar 29, 2017

API: to_datetime with ordinals and no unit #15836

Open

jreback force-pushed the datetime-unit branch from 109adef to f5361ec Compare March 29, 2017 15:25

jorisvandenbossche reviewed Apr 2, 2017

View reviewed changes

sumitbinnani and others added 6 commits April 2, 2017 18:30

BUG: Series creation with datetime64 with non-ns unit as object dtype

ad7356e

closes pandas-dev#11275 closes pandas-dev#11745

update docs / tests

6a8a779

rename epoch -> unix

a24e88c

add Timedelta floordiv ops

56663a5

provide out-of-bounds for julian dates

bug fix

209591a

doc fixes & cleanup

ebb4acd

jreback force-pushed the datetime-unit branch from 6ecee75 to ebb4acd Compare April 2, 2017 22:40

jreback closed this in cd24fa9 Apr 2, 2017

Winand mentioned this pull request Apr 3, 2017

Support more sas7bdat date/datetime formats #15871

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: add origin to to_datetime #15828

ENH: add origin to to_datetime #15828

jreback commented Mar 28, 2017

jreback commented Mar 28, 2017

chris-b1 Mar 28, 2017

jreback Mar 28, 2017

chris-b1 Mar 28, 2017

chris-b1 Mar 28, 2017

jreback Mar 28, 2017

codecov bot commented Mar 28, 2017 •

edited

Loading

jreback commented Mar 28, 2017

jorisvandenbossche Mar 29, 2017

jreback Mar 29, 2017

jorisvandenbossche Mar 29, 2017

jorisvandenbossche Mar 29, 2017

jreback Mar 29, 2017

jorisvandenbossche Mar 29, 2017

jreback Mar 29, 2017

jorisvandenbossche Mar 29, 2017

jorisvandenbossche commented Mar 29, 2017 •

edited

Loading

jorisvandenbossche Mar 29, 2017

jreback commented Mar 29, 2017

jorisvandenbossche Apr 2, 2017

jorisvandenbossche left a comment

jorisvandenbossche Apr 2, 2017

jreback Apr 2, 2017

jreback Apr 2, 2017

jorisvandenbossche Apr 2, 2017

jreback commented Apr 2, 2017

jorisvandenbossche commented Apr 3, 2017


		pd.to_datetime([1, 2, 3], unit='D', origin=pd.Timestamp('1960-01-01'))

		The default is set at ``origin='epoch'``, which defaults to ``1970-01-01 00:00:00``.

ENH: add origin to to_datetime #15828

ENH: add origin to to_datetime #15828

Conversation

jreback commented Mar 28, 2017

jreback commented Mar 28, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Mar 28, 2017 • edited Loading

Codecov Report

jreback commented Mar 28, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorisvandenbossche commented Mar 29, 2017 • edited Loading

Choose a reason for hiding this comment

jreback commented Mar 29, 2017

Choose a reason for hiding this comment

jorisvandenbossche left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Apr 2, 2017

jorisvandenbossche commented Apr 3, 2017

codecov bot commented Mar 28, 2017 •

edited

Loading

jorisvandenbossche commented Mar 29, 2017 •

edited

Loading