ENH: Adding origin parameter in pd.to_datetime #11470

sumitbinnani · 2015-10-29T11:20:00Z

Fixes #11276
Fixes #11745

jreback · 2015-10-29T11:20:51Z

tests!

sumitbinnani · 2015-10-29T11:22:28Z

On it. Any boundary conditions that I must take into consideration?

jreback · 2015-10-29T11:23:35Z

what do you mean?

jreback · 2015-10-29T11:25:04Z

also this would need to be added in Timestamp as that accepts a unit as well

sumitbinnani · 2015-11-02T02:48:36Z

Apologies for the late reply. I am not able to understand which file to edit for Timestamp. Kindly guide, @jreback?

jreback · 2015-11-02T11:46:09Z

https://github.com/pydata/pandas/blob/master/pandas/tslib.pyx#L222

sumitbinnani · 2015-11-03T09:51:58Z

Thanks @jreback. Will go through the file asap :-)

sumitbinnani · 2015-11-10T13:18:17Z

Hi, I am bit confused regarding the implementation. As far as I could understand, following might help. Kindly correct me if I am wrong.

## Constructor in Timestamp Class.
def __new__(cls, object ts_input, object offset=None, tz=None, unit=None):
        cdef _TSObject ts
        cdef _Timestamp ts_base

        if unit == 'julian':
            unit = 'D'
            ts_input = ts_input - Timestamp(0).to_julian_date()

        ts = convert_to_tsobject(ts_input, tz, unit)

        if ts.value == NPY_NAT:
            return NaT

        if util.is_string_object(offset):
            from pandas.tseries.frequencies import to_offset
            offset = to_offset(offset)

        # make datetime happy
        ts_base = _Timestamp.__new__(cls, ts.dts.year, ts.dts.month,
                                     ts.dts.day, ts.dts.hour, ts.dts.min,
                                     ts.dts.sec, ts.dts.us, ts.tzinfo)

        # fill out rest of data
        ts_base.value = ts.value
        ts_base.offset = offset
        ts_base.nanosecond = ts.dts.ps / 1000

        return ts_base

Does it seem okay, @jreback?

jreback · 2015-11-13T13:57:24Z

@sumitbinnani this should be implemented in cast_from_unit

jreback · 2015-11-25T15:41:25Z

this needs tests

jreback · 2015-12-02T23:35:58Z

see #11745; let's call this origin (and in this case I think it would be a string, e.g. 'julian', rather than a specific date)

sumitbinnani · 2015-12-03T05:30:10Z

Have been a bit occupied for last a few weeks owing to applications and submission deadlines. Will make the necessary changes for #11745 asap. Facing a bit difficulty writing test cases (going through documentations and will mostly be done by this weekend).

sumitbinnani · 2015-12-08T09:09:57Z

Have changed the definition of to_datetime to include a new parameter origin, and have added following lines in the code:

    ...
    if origin is not None:
        import datetime
        from pandas.core.api import Timestamp
        if origin == 'julian':
            arg = arg - Timestamp(0).to_julian_date()
        elif isinstance(origin, datetime.date):
            arg = arg + Timestamp(origin).to_julian_date() - Timestamp(0).to_julian_date()

   return ...

Should I remove the changes made to the class Timestamp as unit is no longer being used for the required functionality? @jreback?

jreback · 2015-12-09T15:19:40Z

@sumitbinnani yes, remove the original unit changes and go with origin as discussed

sumitbinnani · 2015-12-10T04:55:48Z

Hi @jreback,

Have made the changes as followed:

Added origin as a new parameter with default value of None to to_datetime
Updated documentation for to_datetime
Added test case for the patch (wrote test case for the first time and might require a bit of changes)

Let me know if any additional changes are required.

jreback · 2015-12-10T12:28:30Z

pandas/tseries/tests/test_timeseries.py

@@ -743,6 +743,22 @@ def test_to_datetime_unit(self):
        result = to_datetime(s,unit='s')
        expected = Series([ Timestamp('2013-06-09 02:42:28') + timedelta(seconds=t) for t in range(20) ] + [NaT])
        assert_series_equal(result,expected)
+
+    def test_to_datetime_origin(self):
+        # for origin as julian


add the issue number as a comment

jreback · 2015-12-10T12:36:44Z

pls add whatsnew note in Enhancements for 0.18.0

jorisvandenbossche · 2016-08-18T19:46:47Z

pandas/tseries/tools.py

+    0    1960-01-01
+    1    1960-01-02
+    ...
+    98   1960-04-08


The output here of the example does not yet match the example code

Opps. My Bad. Updated the documentation with proper output.

Rectified mistake in documentation.

sumitbinnani · 2016-08-19T09:26:06Z

Have rectified the mistake in example. Unsure about how to proceed for the aforementioned invalid-origin :-(

jreback · 2016-08-19T10:35:00Z

In [13]: pd.to_datetime([2000*365], unit='D', origin='0001-01-01')
...
ValueError: Invalid 'origin' or 'origin' Out of Bound

I think this is fine for now. We could make this happen, but the code then gets convoluted, but it is so rare as not to be important.

I would just assert that this example raises.

jreback · 2016-08-19T10:36:03Z

doc/source/whatsnew/v0.19.0.txt

@@ -24,6 +24,16 @@ New features

 .. _whatsnew_0190.dev_api:

+to_datetime can be used with Offset


to_datetime has gained an origin kwarg.

don't call this Offset which is a very specific meaning

in the doc-string this is a 'reference date', so I would use that here.

jreback · 2016-09-09T22:34:51Z

can you rebase / update?

jorisvandenbossche · 2016-10-04T14:17:39Z

@sumitbinnani Can you move the whatsnew notice to the v0.20.0.txt file? As 0.19.0 has been releases in the meantime (also, all the other changes in the 0.19.0.txt file should not be included in here)

sumitbinnani · 2016-10-05T07:51:58Z

@jorisvandenbossche The v0.19.0 had been reverted to the master. Changes added to v0.20.0.

jorisvandenbossche · 2016-10-05T08:18:36Z

Some examples that now go wrong:

In [1]: pd.to_datetime("2005-01-01", origin="1960-01-01")
Out[1]: Timestamp('1995-01-01 00:00:00')

In [2]: pd.to_datetime(1, origin="julian", unit='D')
...
OutOfBoundsDatetime: cannot convert input with unit 'D'

In [3]: pd.to_datetime(2456658, origin='julian', unit='D')
Out[3]: Timestamp('2013-12-31 00:00:00')

In [4]: pd.to_datetime(2456658, origin='0001-01-01', unit='D')
...
OutOfBoundsDatetime: cannot convert input with unit 'D'

So in In [1], this should raise, as the origin should only be used when having numeric input. In [2] is correct I think, but we should add a test for this that it indeed raises an OutOfBoundsDatetime error. In [4] should work (In [3] is just to show this is a number that is not out of bounds).

EDIT: regarding In [4], we decided above that it is OK that this does not work for now (as it is difficult to parse the out of bounds origin), but, it should have a better error message, and you should add a test for it.

jorisvandenbossche · 2016-10-05T08:58:12Z

@sumitbinnani what I think you could do to simplify the code (having all handling of origin in one place) is the following: start with processing the origin keyword (and only do this for numerical input), convert it to an offset (according the the unit), and adapt the numerical input with this offset before further parsing it.

So something like (pseudocode)

if origin != 'epoch':
    if origin == 'julian':
        ... (what you already have in this case)
        arg = arg - tslib.Timestamp(0).to_julian_date()
    else:
        ... (parse origin and convert to correct number according to unit)
        arg = arg - offset

... (original code to parse `arg` (what you now put in the 'intermediate_result' function)

A way to deal with the out of bounds origin would be to use np.datetime64(dateutil.parser.parse(origin)), but it's OK to leave that for now.

jreback · 2016-11-25T14:39:09Z

can you rebase / update according to comments

jreback · 2016-12-26T21:49:15Z

@sumitbinnani I know this has been open a long time. can you rebase and we'll see if can get it in?

pls change release note to 0.20.0

sumitbinnani · 2016-12-27T10:42:28Z

@jreback Sure. Would go through the code this weekend and rebase :-)

sumitbinnani · 2017-01-19T08:29:26Z

@jreback There are quite a many changes. I would start clean and update asap. Sorry for the delay.

bashtage · 2017-01-25T09:49:16Z

@sumitbinnani Useful feature so would be good to get this in

bashtage · 2017-01-25T09:54:27Z

pandas/tseries/tools.py

-        return _convert_listlike(arg, box, format, name=arg.name)
-    elif is_list_like(arg):
-        return _convert_listlike(arg, box, format)
+    def intermediate_result(arg):


Is there some reason this is a function? It looks like it should just be an if, else if, else

bashtage · 2017-01-25T09:55:58Z

pandas/tseries/tools.py

+        Define reference date. The numeric values would be parsed as number
+        of units (defined by `unit`) since this reference date.
+
+        - If 'epoch', origin is set to 1970-01-01.


You could consider other standard offsets: excel, Jan 0 1900, stata Jan 0, 1960, matlab, Jan 0, 0000? Just an idea

jreback · 2017-03-28T20:45:42Z

superseded by #15828

@sumitbinnani I rebase and updated your PR. thanks!

closes #11276 closes #11745 superseded #11470 Author: Jeff Reback <jeff@reback.net> Author: Sumit Binnani <sumit.binnani@gmail.com> Closes #15828 from jreback/datetime-unit and squashes the following commits: ebb4acd [Jeff Reback] doc fixes & cleanup 209591a [Jeff Reback] bug fix 56663a5 [Jeff Reback] add Timedelta floordiv ops a24e88c [Jeff Reback] rename epoch -> unix 6a8a779 [Jeff Reback] update docs / tests ad7356e [Sumit Binnani] BUG: Series creation with datetime64 with non-ns unit as object dtype

sumitbinnani changed the title ~~Added unit julian to to_datetime~~ ENH: Added unit julian to to_datetime Oct 29, 2015

jreback added Enhancement Datetime Datetime data dtype labels Oct 29, 2015

jreback changed the title ~~ENH: Added unit julian to to_datetime~~ ENH: Added julian unit in pd.to_datetime Oct 29, 2015

jreback mentioned this pull request Dec 2, 2015

Add origin parameter to Timestamp/to_datetime epoch support. #11745

Closed

sumitbinnani force-pushed the ENH-Adding-unit-``julian``-to-``to_datetime`` branch from 86681d2 to 45037a8 Compare December 9, 2015 19:11

sumitbinnani changed the title ~~ENH: Added julian unit in pd.to_datetime~~ ENH: Adding origin parameter in pd.to_datetime Dec 9, 2015

sumitbinnani force-pushed the ENH-Adding-unit-``julian``-to-``to_datetime`` branch 2 times, most recently from 1deddb0 to 86681d2 Compare December 9, 2015 20:53

jreback reviewed Dec 10, 2015
View reviewed changes

jorisvandenbossche reviewed Aug 18, 2016
View reviewed changes

DOC: Updated Documentation for origin parameter

978f0d2

Rectified mistake in documentation.

jreback reviewed Aug 19, 2016
View reviewed changes

sumitbinnani force-pushed the ENH-Adding-unit-``julian``-to-``to_datetime`` branch from 763a620 to ffd9b28 Compare October 4, 2016 04:55

CLN: Worked on Code Reveiew

4396d81

sumitbinnani force-pushed the ENH-Adding-unit-``julian``-to-``to_datetime`` branch from a99fdb3 to 4396d81 Compare October 4, 2016 05:57

DOC: Adding doc for new kwarg origin

4b9a3f4

bashtage reviewed Jan 25, 2017

View reviewed changes

jreback mentioned this pull request Mar 28, 2017

ENH: add origin to to_datetime #15828

Closed

jreback closed this Mar 28, 2017

jreback added this to the 0.20.0 milestone Mar 28, 2017

		@@ -24,6 +24,16 @@ New features

		.. _whatsnew_0190.dev_api:

		to_datetime can be used with Offset

ENH: Adding origin parameter in pd.to_datetime #11470

ENH: Adding origin parameter in pd.to_datetime #11470

Conversation

sumitbinnani commented Oct 29, 2015

jreback commented Oct 29, 2015

sumitbinnani commented Oct 29, 2015

jreback commented Oct 29, 2015

jreback commented Oct 29, 2015

sumitbinnani commented Nov 2, 2015

jreback commented Nov 2, 2015

sumitbinnani commented Nov 3, 2015

sumitbinnani commented Nov 10, 2015

jreback commented Nov 13, 2015

jreback commented Nov 25, 2015

jreback commented Dec 2, 2015

sumitbinnani commented Dec 3, 2015

sumitbinnani commented Dec 8, 2015

jreback commented Dec 9, 2015

sumitbinnani commented Dec 10, 2015

Choose a reason for hiding this comment

jreback commented Dec 10, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sumitbinnani commented Aug 19, 2016

jreback commented Aug 19, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Sep 9, 2016

jorisvandenbossche commented Oct 4, 2016

sumitbinnani commented Oct 5, 2016

jorisvandenbossche commented Oct 5, 2016 • edited Loading

jorisvandenbossche commented Oct 5, 2016

jreback commented Nov 25, 2016

jreback commented Dec 26, 2016

sumitbinnani commented Dec 27, 2016

sumitbinnani commented Jan 19, 2017

bashtage commented Jan 25, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Mar 28, 2017

jorisvandenbossche commented Oct 5, 2016 •

edited

Loading