Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PERF: Improve perf initalizing DataFrame with a range #30171

Merged
merged 2 commits into from
Dec 10, 2019
Merged

PERF: Improve perf initalizing DataFrame with a range #30171

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
50f9100
Improve perf initalizing DataFrame with a range
topper-123 Dec 9, 2019
b0a3e94
Add ASV
topper-123 Dec 10, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions asv_bench/benchmarks/frame_ctor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,4 +105,16 @@ def time_frame_from_lists(self):
self.df = DataFrame(self.data)


class FromRange:

goal_time = 0.2

def setup(self):
N = 1_000_000
self.data = range(N)

def time_frame_from_range(self):
self.df = DataFrame(self.data)


from .pandas_vb_common import setup # noqa: F401 isort:skip
1 change: 1 addition & 0 deletions doc/source/whatsnew/v1.0.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -634,6 +634,7 @@ Performance improvements
- Performance improvement in indexing with a non-unique :class:`IntervalIndex` (:issue:`27489`)
- Performance improvement in `MultiIndex.is_monotonic` (:issue:`27495`)
- Performance improvement in :func:`cut` when ``bins`` is an :class:`IntervalIndex` (:issue:`27668`)
- Performance improvement when initializing a :class:`DataFrame` using a ``range`` (:issue:`30171`)
- Performance improvement in :meth:`DataFrame.corr` when ``method`` is ``"spearman"`` (:issue:`28139`)
- Performance improvement in :meth:`DataFrame.replace` when provided a list of values to replace (:issue:`28099`)
- Performance improvement in :meth:`DataFrame.select_dtypes` by using vectorization instead of iterating over a loop (:issue:`28317`)
Expand Down
5 changes: 4 additions & 1 deletion pandas/core/internals/construction.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -246,10 +246,13 @@ def init_dict(data, index, columns, dtype=None):
# ---------------------------------------------------------------------


def prep_ndarray(values, copy=True):
def prep_ndarray(values, copy=True) -> np.ndarray:
if not isinstance(values, (np.ndarray, ABCSeries, Index)):
if len(values) == 0:
return np.empty((0, 0), dtype=object)
elif isinstance(values, range):
arr = np.arange(values.start, values.stop, values.step, dtype="int64")
Copy link
Member

@jschendel jschendel Dec 10, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like some care is needed here in respect to dtypes. Specifically if the range contains values only supported by uint64, or values only supported by Python integers.

For example, the following works on master:

In [2]: pd.DataFrame(range(2**63, 2**63 + 4))
Out[2]: 
                     0
0  9223372036854775808
1  9223372036854775809
2  9223372036854775810
3  9223372036854775811

In [3]: _.dtypes
Out[3]: 
0    uint64
dtype: object

In [4]: pd.DataFrame(range(2**73, 2**73 + 4))
Out[4]: 
                        0
0  9444732965739290427392
1  9444732965739290427393
2  9444732965739290427394
3  9444732965739290427395

In [5]: _.dtypes
Out[5]: 
0    object
dtype: object

But both fail with the changes in this PR:

In [2]: pd.DataFrame(range(2**63, 2**63 + 4))
---------------------------------------------------------------------------
OverflowError: Python int too large to convert to C long

In [3]: pd.DataFrame(range(2**73, 2**73 + 4))
---------------------------------------------------------------------------
OverflowError: Python int too large to convert to C long

Admittedly, this is a bit of a corner case. It also looks like the issue isn't limited to the PR, as the Series equivalent of the above fails on master.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

xref #30173 for the failing Series example on master; I'd expect that both issues could be solved in a similar way.

return arr[..., np.newaxis]

def convert(v):
return maybe_convert_platform(v)
Expand Down