BUG: Series.repr crashing with tzlocal() #24310

h-vetinari · 2018-12-16T19:44:39Z

Encountered while writing tests for PR to #23833, where I wanted to use tz_aware_fixture for testing promotions of datetimes with tz. The following is an equivalent minimal example, but constructed directly.

>>> import pandas as pd
>>> from dateutil.tz import tzlocal
>>> from pandas.core.dtypes.dtypes import DatetimeTZDtype
>>> dtype = DatetimeTZDtype(tz = tzlocal())
>>> s = pd.Series([10 ** 9], dtype=dtype)

The constructor passes, but worryingly, already the repr of s fails with a huge stacktrace:

>>> s
Traceback (most recent call last):
[~130 lines of stacktrace]
OSError: [Errno 22] Invalid argument

Same goes for s[0].

The underlying values are still there though:

>>> s.values
array(['1970-01-01T00:00:01.000000000'], dtype='datetime64[ns]')
>>> s._data
SingleBlockManager
Items: RangeIndex(start=0, stop=1, step=1)
DatetimeTZBlock: 1 dtype: datetime64[ns, tzlocal()]

This seems like a pretty serious regression to me. @mroeschke @jbrockmendel @jreback @TomAugspurger @jorisvandenbossche

The text was updated successfully, but these errors were encountered:

TomAugspurger · 2018-12-16T19:48:15Z

I can't reproduce on master. Can you post the full traceback in a <details> tag?

h-vetinari · 2018-12-16T19:57:25Z

@TomAugspurger

Here's the stacktrace:

>>> s
Traceback (most recent call last):
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\dateutil\tz\_common.py", line 140, in fromutc
    return f(self, dt)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\dateutil\tz\_common.py", line 254, in fromutc
    dt_wall = self._fromutc(dt)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\dateutil\tz\_common.py", line 218, in _fromutc
    dtoff = dt.utcoffset()
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\dateutil\tz\tz.py", line 210, in utcoffset
    if self._isdst(dt):
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\dateutil\tz\tz.py", line 279, in _isdst
    dstval = self._naive_is_dst(dt)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\dateutil\tz\tz.py", line 248, in _naive_is_dst
    return time.localtime(timestamp + time.timezone).tm_isdst
OSError: [Errno 22] Invalid argument
Exception ignored in: 'pandas._libs.tslibs.conversion._tz_convert_tzlocal_utc'
Traceback (most recent call last):
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\dateutil\tz\_common.py", line 140, in fromutc
    return f(self, dt)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\dateutil\tz\_common.py", line 254, in fromutc
    dt_wall = self._fromutc(dt)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\dateutil\tz\_common.py", line 218, in _fromutc
    dtoff = dt.utcoffset()
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\dateutil\tz\tz.py", line 210, in utcoffset
    if self._isdst(dt):
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\dateutil\tz\tz.py", line 279, in _isdst
    dstval = self._naive_is_dst(dt)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\dateutil\tz\tz.py", line 248, in _naive_is_dst
    return time.localtime(timestamp + time.timezone).tm_isdst
OSError: [Errno 22] Invalid argument
Traceback (most recent call last):
  File "C:\Users\[...]\pddev\pandas\core\arrays\datetimes.py", line 1597, in objects_to_datetime64ns
    require_iso8601=require_iso8601
  File "pandas\_libs\tslib.pyx", line 464, in pandas._libs.tslib.array_to_datetime
    cpdef array_to_datetime(ndarray[object] values, str errors='raise',
  File "pandas\_libs\tslib.pyx", line 541, in pandas._libs.tslib.array_to_datetime
    raise ValueError('Tz-aware datetime.datetime '
ValueError: Tz-aware datetime.datetime cannot be converted to datetime64 unless utc=True

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "pandas\_libs\tslibs\timezones.pyx", line 156, in pandas._libs.tslibs.timezones.get_utcoffset
    return tzinfo._utcoffset
AttributeError: 'tzlocal' object has no attribute '_utcoffset'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "pandas\_libs\tslibs\timezones.pyx", line 156, in pandas._libs.tslibs.timezones.get_utcoffset
    return tzinfo._utcoffset
AttributeError: 'tzlocal' object has no attribute '_utcoffset'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "pandas\_libs\tslibs\timezones.pyx", line 158, in pandas._libs.tslibs.timezones.get_utcoffset
    return tzinfo.utcoffset(obj)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\dateutil\tz\tz.py", line 210, in utcoffset
    if self._isdst(dt):
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\dateutil\tz\tz.py", line 279, in _isdst
    dstval = self._naive_is_dst(dt)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\dateutil\tz\tz.py", line 248, in _naive_is_dst
    return time.localtime(timestamp + time.timezone).tm_isdst
OSError: [Errno 22] Invalid argument
Exception ignored in: 'pandas._libs.tslibs.conversion._tz_convert_tzlocal_utc'
Traceback (most recent call last):
  File "pandas\_libs\tslibs\timezones.pyx", line 158, in pandas._libs.tslibs.timezones.get_utcoffset
    return tzinfo.utcoffset(obj)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\dateutil\tz\tz.py", line 210, in utcoffset
    if self._isdst(dt):
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\dateutil\tz\tz.py", line 279, in _isdst
    dstval = self._naive_is_dst(dt)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\dateutil\tz\tz.py", line 248, in _naive_is_dst
    return time.localtime(timestamp + time.timezone).tm_isdst
OSError: [Errno 22] Invalid argument
Traceback (most recent call last):
  File "C:\Users\[...]\pddev\pandas\core\arrays\datetimes.py", line 1597, in objects_to_datetime64ns
    require_iso8601=require_iso8601
  File "pandas\_libs\tslib.pyx", line 464, in pandas._libs.tslib.array_to_datetime
    cpdef array_to_datetime(ndarray[object] values, str errors='raise',
  File "pandas\_libs\tslib.pyx", line 541, in pandas._libs.tslib.array_to_datetime
    raise ValueError('Tz-aware datetime.datetime '
ValueError: Tz-aware datetime.datetime cannot be converted to datetime64 unless utc=True

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "pandas\_libs\tslibs\timezones.pyx", line 156, in pandas._libs.tslibs.timezones.get_utcoffset
    return tzinfo._utcoffset
AttributeError: 'tzlocal' object has no attribute '_utcoffset'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\[...]\pddev\pandas\core\base.py", line 77, in __repr__
    return str(self)
  File "C:\Users\[...]\pddev\pandas\core\base.py", line 56, in __str__
    return self.__unicode__()
  File "C:\Users\[...]\pddev\pandas\core\series.py", line 1275, in __unicode__
    max_rows=max_rows, length=show_dimensions)
  File "C:\Users\[...]\pddev\pandas\core\series.py", line 1319, in to_string
    result = formatter.to_string()
  File "C:\Users\[...]\pddev\pandas\io\formats\format.py", line 266, in to_string
    fmt_values = self._get_formatted_values()
  File "C:\Users\[...]\pddev\pandas\io\formats\format.py", line 256, in _get_formatted_values
    float_format=self.float_format, na_rep=self.na_rep)
  File "C:\Users\[...]\pddev\pandas\io\formats\format.py", line 903, in format_array
    return fmt_obj.get_result()
  File "C:\Users\[...]\pddev\pandas\io\formats\format.py", line 924, in get_result
    fmt_values = self._format_strings()
  File "C:\Users\[...]\pddev\pandas\io\formats\format.py", line 1310, in _format_strings
    is_dates_only = _is_dates_only(values)
  File "C:\Users\[...]\pddev\pandas\io\formats\format.py", line 1247, in _is_dates_only
    values = DatetimeIndex(values)
  File "C:\Users\[...]\pddev\pandas\core\indexes\datetimes.py", line 254, in __new__
    dayfirst=dayfirst, yearfirst=yearfirst, ambiguous=ambiguous)
  File "C:\Users\[...]\pddev\pandas\core\arrays\datetimes.py", line 240, in _from_sequence
    dayfirst=dayfirst, yearfirst=yearfirst, ambiguous=ambiguous)
  File "C:\Users\[...]\pddev\pandas\core\arrays\datetimes.py", line 1512, in sequence_to_dt64ns
    data, dayfirst=dayfirst, yearfirst=yearfirst)
  File "C:\Users\[...]\pddev\pandas\core\arrays\datetimes.py", line 1601, in objects_to_datetime64ns
    values, tz_parsed = conversion.datetime_to_datetime64(data)
  File "pandas\_libs\tslibs\conversion.pyx", line 186, in pandas._libs.tslibs.conversion.datetime_to_datetime64
    _ts = convert_datetime_to_tsobject(val, None)
  File "pandas\_libs\tslibs\conversion.pyx", line 375, in pandas._libs.tslibs.conversion.convert_datetime_to_tsobject
    offset = get_utcoffset(obj.tzinfo, ts)
  File "pandas\_libs\tslibs\timezones.pyx", line 158, in pandas._libs.tslibs.timezones.get_utcoffset
    return tzinfo.utcoffset(obj)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\dateutil\tz\tz.py", line 210, in utcoffset
    if self._isdst(dt):
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\dateutil\tz\tz.py", line 279, in _isdst
    dstval = self._naive_is_dst(dt)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\dateutil\tz\tz.py", line 248, in _naive_is_dst
    return time.localtime(timestamp + time.timezone).tm_isdst
OSError: [Errno 22] Invalid argument

and here's the output of pd.show_versions():

>>> pd.show_versions()
No module named 'dask'

INSTALLED VERSIONS
------------------
commit: 6c767c5c0d8049a8544b38189adfef450d1b9e6e
python: 3.6.6.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 78 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.24.0.dev0+1299.g6c767c5c0
pytest: 4.0.0
pip: 18.1
setuptools: 40.6.2
Cython: 0.29
numpy: 1.15.4
scipy: 1.1.0
pyarrow: 0.11.1
xarray: 0.11.0
IPython: 7.1.1
sphinx: 1.8.2
patsy: 0.5.1
dateutil: 2.7.5
pytz: 2018.7
blosc: None
bottleneck: 1.2.1
tables: 3.4.4
numexpr: 2.6.8
feather: 0.4.0
matplotlib: 3.0.2
openpyxl: 2.5.9
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.1.2
lxml.etree: 4.2.5
bs4: 4.6.3
html5lib: 1.0.1
sqlalchemy: 1.2.14
pymysql: 0.8.1
psycopg2: None
jinja2: 2.10
s3fs: 0.1.6
fastparquet: 0.1.6
pandas_gbq: None
pandas_datareader: None
gcsfs: 0.1.2

TomAugspurger · 2018-12-16T19:59:36Z

Just to verify, can you verify that your C extensions have been (re)built recently?

jbrockmendel · 2018-12-16T20:01:24Z

what timezone are you in? your tzlocal() may not be the same as others

h-vetinari · 2018-12-16T20:01:29Z

Just to verify, can you verify that your C extensions have been (re)built recently?

Of course, that's the first thing I double-check in such cases. (I tend to rebuild the extensions on every branch change)

h-vetinari · 2018-12-16T20:05:41Z

@jbrockmendel

what timezone are you in? your tzlocal() may not be the same as others

What's tzlocal() supposed to be? Was surprised this didn't have a useful repr.

>>> from dateutil.tz import tzlocal
>>> tzlocal()
tzlocal()

jbrockmendel · 2018-12-16T20:11:03Z

What's tzlocal() supposed to be?

tzlocal represents the local timezone in system's locale. i.e. for a user in California tzlocal() corresponds to US/Pacific (though it is not the same object you'd get with dateutil.tz.gettz('US/Pacific'))

h-vetinari · 2018-12-16T20:13:59Z

@jbrockmendel

tzlocal represents the local timezone in system's locale.

I get that, I meant more specifically whether tzlocal() should already resolve to a string like US/Eastern (was surprised there was no info in the repr)

(EDIT: resolved some of my confusion below)

h-vetinari · 2018-12-16T20:15:06Z

It seems to (wrongly) believe I'm in python2-land...?

>>> tzlocal().tzname
<bound method tzname_in_python2.<locals>.adjust_encoding of tzlocal()>

h-vetinari · 2018-12-16T20:16:48Z

@jbrockmendel
To answer your previous question:

what timezone are you in?

I'm in Europe/Amsterdam.

h-vetinari · 2018-12-16T21:10:53Z

OK, it should be clear from my attempts above that I never worked with tzlocal(), haha. Anyway, I figured out a little more (it seemed very weird to me that tzlocal().tzname needs an actual datetime object to tell me which timezone it is, until I remembered DST changes):

>>> from datetime import datetime, timedelta
>>> dt = datetime.now()
>>> tzlocal().tzname(dt)
'W. Europe Standard Time'
>>> dt = datetime.now() - timedelta(days=30 * 6)
>>> tzlocal().tzname(dt)
'W. Europe Daylight Time'

Back to the issue at hand, dir(tzlocal()) (as well as dir(tzlocal)) shows that it has an attribute utcoffset, whereas in the stacktrace above, there's an error looking for _utcoffset (which is not in the dir):

AttributeError: 'tzlocal' object has no attribute '_utcoffset'

mroeschke · 2018-12-17T04:03:33Z

This dateutil issue has a mildly similar traceback with tzlocal and Windows: dateutil/dateutil#197

h-vetinari · 2018-12-17T17:36:02Z

@mroeschke
Thanks for the link, that issue was indeed pertinent. There's a two-layered problem here:

A CPython bug, namely that time.localtime() cannot deal with negative values on Windows (whereas it works on Linux)
A dateutil peculiarity (at least at first glance), that there exists tzlocal (crashes) and tzwinlocal (working). I've made a comment in the issue about it.

I'll be skipping the tests in my PR for now.

mroeschke added Output-Formatting __repr__ of pandas objects, to_string Timezones Timezone data dtype labels Dec 18, 2018

mroeschke added Window rolling, ewma, expanding Windows Windows OS and removed Window rolling, ewma, expanding labels Oct 20, 2019

mroeschke added Error Reporting Incorrect or improved errors from pandas Bug labels Mar 29, 2020

vfilimonov mentioned this issue Apr 25, 2020

BUG: DataFrame with index having tzlocal() timezone could not be saved to parquet #33786

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: Series.repr crashing with tzlocal() #24310

BUG: Series.repr crashing with tzlocal() #24310

h-vetinari commented Dec 16, 2018

TomAugspurger commented Dec 16, 2018

h-vetinari commented Dec 16, 2018 •

edited

Loading

TomAugspurger commented Dec 16, 2018

jbrockmendel commented Dec 16, 2018

h-vetinari commented Dec 16, 2018 •

edited

Loading

h-vetinari commented Dec 16, 2018

jbrockmendel commented Dec 16, 2018

h-vetinari commented Dec 16, 2018 •

edited

Loading

h-vetinari commented Dec 16, 2018

h-vetinari commented Dec 16, 2018

h-vetinari commented Dec 16, 2018

mroeschke commented Dec 17, 2018

h-vetinari commented Dec 17, 2018

BUG: Series.__repr__ crashing with tzlocal() #24310

BUG: Series.__repr__ crashing with tzlocal() #24310

Comments

h-vetinari commented Dec 16, 2018

TomAugspurger commented Dec 16, 2018

h-vetinari commented Dec 16, 2018 • edited Loading

TomAugspurger commented Dec 16, 2018

jbrockmendel commented Dec 16, 2018

h-vetinari commented Dec 16, 2018 • edited Loading

h-vetinari commented Dec 16, 2018

jbrockmendel commented Dec 16, 2018

h-vetinari commented Dec 16, 2018 • edited Loading

h-vetinari commented Dec 16, 2018

h-vetinari commented Dec 16, 2018

h-vetinari commented Dec 16, 2018

mroeschke commented Dec 17, 2018

h-vetinari commented Dec 17, 2018

BUG: Series.repr crashing with tzlocal() #24310

BUG: Series.repr crashing with tzlocal() #24310

h-vetinari commented Dec 16, 2018 •

edited

Loading

h-vetinari commented Dec 16, 2018 •

edited

Loading

h-vetinari commented Dec 16, 2018 •

edited

Loading