CI: remove failing line #24092

TomAugspurger · 2018-12-04T15:33:59Z

No description provided.

pep8speaks · 2018-12-04T15:34:02Z

Hello @TomAugspurger! Thanks for submitting the PR.

There are no PEP8 issues in the file asv_bench/benchmarks/frame_methods.py !

jreback · 2018-12-04T16:12:48Z

https://travis-ci.org/pandas-dev/pandas/jobs/463403386

maybe picking up a new moto?

jreback · 2018-12-04T16:14:20Z

https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=4681

datapythonista · 2018-12-04T17:29:38Z

I don't know about the error in travis. We started to check the benchmarks yesterday, and they just run when there are changes to it. So, I guess they were already broken, may be for a while. Happy to ignore the errors in the benchmarks here, I can take a look tonight and fix them in a separate PR, if nobody does it before.

TomAugspurger · 2018-12-04T17:41:09Z

I'm cleaning them up now. A couple are just broken / weren't updated with removal of deprecations.

The HTML repr stuff may be related to how we're running stuff in azure. I think I have a workaround.

h-vetinari · 2018-12-04T17:47:55Z

@jreback @datapythonista
The error in https://travis-ci.org/pandas-dev/pandas/jobs/463403386 is not from the asv tests, but a new (flaky?) moto/boto error. Searching for the error leads to https://stackoverflow.com/a/52417342/2965879, which recommends pip install google-compute-engine as a solution.

The ASVs are only properly tested since a few CI runs (after the linebreak thing?), and I guess there are still errors hiding in the code. e.g.:

[  0.57%] ··· algorithms.Match.time_match_string                          failed
AttributeError: module 'pandas' has no attribute 'match'
[  9.80%] ··· frame_methods.Reindex.time_reindex_both_axes_ix 
KeyError: "None of [Int64Index([4000, 4001, 4002, 4003, 4004, 4005, 4006, 4007, 4008, 4009,\n
[ 10.37%] ··· frame_methods.Repr.time_html_repr_trunc_mi 
ValueError: I/O operation on closed file
[ 10.44%] ··· frame_methods.Repr.time_html_repr_trunc_si
ValueError: I/O operation on closed file
[ 36.58%] ··· sparse.ToCoo.time_sparse_series_to_coo
ValueError: The name None occurs multiple times, use a level number
[ 47.30%] ··· algorithms.Hashing.time_series_timedeltas  
TypeError: setup_cache() takes 0 positional arguments but 1 was given

@TomAugspurger: didn't see you already posted something on that while I was writing...

codecov · 2018-12-04T18:40:04Z

Codecov Report

Merging #24092 into master will increase coverage by 49.68%.
The diff coverage is 100%.

@@             Coverage Diff             @@
##           master   #24092       +/-   ##
===========================================
+ Coverage   42.52%    92.2%   +49.68%     
===========================================
  Files         161      161               
  Lines       51697    51684       -13     
===========================================
+ Hits        21982    47657    +25675     
+ Misses      29715     4027    -25688

Flag	Coverage Δ
#multiple	`90.61% <100%> (?)`
#single	`43% <0%> (+0.48%)`	⬆️

Impacted Files	Coverage Δ
pandas/io/formats/html.py	`98.61% <100%> (+98.61%)`	⬆️
pandas/core/sparse/scipy_sparse.py	`100% <100%> (+89.7%)`	⬆️
pandas/core/computation/pytables.py	`92.37% <0%> (+0.3%)`	⬆️
pandas/io/pytables.py	`92.3% <0%> (+0.92%)`	⬆️
pandas/util/_test_decorators.py	`93.24% <0%> (+4.05%)`	⬆️
pandas/compat/__init__.py	`58.36% <0%> (+8.17%)`	⬆️
pandas/core/config_init.py	`99.24% <0%> (+9.84%)`	⬆️
pandas/core/reshape/util.py	`100% <0%> (+11.53%)`	⬆️
pandas/compat/numpy/__init__.py	`92.85% <0%> (+14.28%)`	⬆️
pandas/core/computation/common.py	`85.71% <0%> (+14.28%)`	⬆️
... and 120 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1d3ed91...ce0e25c. Read the comment docs.

codecov · 2018-12-04T18:40:04Z

Codecov Report

Merging #24092 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #24092   +/-   ##
=======================================
  Coverage   42.52%   42.52%           
=======================================
  Files         161      161           
  Lines       51697    51697           
=======================================
  Hits        21982    21982           
  Misses      29715    29715

Flag	Coverage Δ
#single	`42.52% <ø> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1d3ed91...e79d52e. Read the comment docs.

TomAugspurger · 2018-12-04T18:40:42Z

ASV stuff fixed in
1f15d10

Just debugging some moto stuff in
e79d52e. Not a permenant fix.

TomAugspurger · 2018-12-04T18:42:45Z

pandas/core/sparse/scipy_sparse.py

-                    return index.get_level_values(i)
-
-            ilabels = list(zip(*[robust_get_level_values(i) for i in subset]))
+            ilabels = list(zip(*[index._get_level_values(i) for i in subset]))


I think this is OK to do... _sparse_series_to_coo is the only caller, and it uses _get_level_number(x) to go from maybe labels to numbers.

TomAugspurger · 2018-12-04T18:43:21Z

pandas/io/formats/html.py

@@ -161,15 +161,7 @@ def write_result(self, buf):
            _classes.extend(self.classes)

        if self.notebook:
-            div_style = ''
-            try:
-                import IPython


The import IPython was causing a failure. I figure we're OK to drop IPython 2.x compat code :)

TomAugspurger · 2018-12-04T19:29:26Z

Not sure what's up with the SSL failures, but FWIW, pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_url is randomly hanging / skipping / passing for me locally.

TomAugspurger · 2018-12-04T19:45:02Z

Well I'm not sure what to do here...

It's possible that the network tests are hitting some bug in pytest or something, I'm not sure. We really shouldn't be hitting the network in most unit tests.

Do we want to mock these network requests?

In a hacky version, we just mock out HTTPResponse's .read, since that's all we use... In a more sophisticated version we mock the actual server (which we should do at a minimum). But this will be made obsolete by #21504.

h-vetinari

Based on the comment, I think the one upstream change that might have caused this recently is pytest 4.0.1. Let's try pinning pytest<=4.0.0?

h-vetinari · 2018-12-04T19:55:15Z

pandas/tests/io/conftest.py

@@ -37,6 +37,8 @@ def s3_resource(tips_file, jsonl_file):
    """
    pytest.importorskip('s3fs')
    boto3 = pytest.importorskip('boto3')
+    # GH-24092. See if boto.plugin skips the test or fails.
+    pytest.importorskip("boto.plugin")


For some reason, this line fails instead of skipping. https://travis-ci.org/pandas-dev/pandas/jobs/463496968

That job is using the newest pytest 4.0.1, but maybe there's a bug in importorskip for submodules?

Going to try to avoid pinning pytest for now.

TomAugspurger · 2018-12-04T21:02:22Z

Merging on green (I think everything will pass this time).

h-vetinari · 2018-12-04T21:24:15Z

pandas/tests/io/conftest.py

+    try:
+        pytest.importorskip("boto.plugin")
+    except AttributeError:
+        raise pytest.skip("moto/moto error")


wow, that is indeed nasty, haha.

h-vetinari · 2018-12-04T21:27:23Z

@TomAugspurger
Unfortunately, the travis-36 job just failed. Seems the issues with the moto-import were masking the other issue with the crash in pytest:

INTERNALERROR> KeyError: <class 'ssl.SSLSocket'>
INTERNALERROR> 
INTERNALERROR> During handling of the above exception, another exception occurred:
[...]
INTERNALERROR> execnet.gateway_base.DumpError: can't serialize <class 'ssl.SSLSocket'>

TomAugspurger · 2018-12-04T21:30:29Z

Such a mess :) I'm marking the network tests as single, under the theory that they're doing something with sockets that can't be serialized.

h-vetinari · 2018-12-04T21:31:48Z

@TomAugspurger

I was just gonna suggest that approach. I found another xdist issue that seems to point to exactly this:
pytest-dev/pytest-xdist#384

TomAugspurger · 2018-12-04T22:05:29Z

All green. Merging.

CI: remove failing line

a0a37cc

TomAugspurger added this to the 0.24.0 milestone Dec 4, 2018

TomAugspurger added the CI Continuous Integration label Dec 4, 2018

topper-123 mentioned this pull request Dec 4, 2018

CLN: remove unused import of warnings module #24095

Closed

TomAugspurger mentioned this pull request Dec 4, 2018

CI: Some Azure Pipelines cleanups #23111

Merged

TomAugspurger added 2 commits December 4, 2018 12:36

CI: ASV fixups

1f15d10

debug boto

e79d52e

TomAugspurger commented Dec 4, 2018

View reviewed changes

TomAugspurger added 2 commits December 4, 2018 12:59

No isorting benchmarks

1d0a775

correct dtype

d3df86f

TomAugspurger mentioned this pull request Dec 4, 2018

CI for boto: fix errors; add coverage; add skip for uncatchable ResourceWarning #23731

Merged

2 tasks

TomAugspurger mentioned this pull request Dec 4, 2018

BUG: Fix concat series loss of timezone #24027

Merged

h-vetinari reviewed Dec 4, 2018

View reviewed changes

TomAugspurger added 3 commits December 4, 2018 14:07

more hacky workarounds

378515a

Remove unused import

092320a

fix skip

e20850e

h-vetinari reviewed Dec 4, 2018

View reviewed changes

try marking single...

ce0e25c

TomAugspurger merged commit 4b5f4d1 into pandas-dev:master Dec 4, 2018

TomAugspurger deleted the azure-fixup branch December 4, 2018 22:05

Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

CI: Various fixups (pandas-dev#24092)

25a9807

Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

CI: Various fixups (pandas-dev#24092)

1f1b384

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: remove failing line #24092

CI: remove failing line #24092

TomAugspurger commented Dec 4, 2018

pep8speaks commented Dec 4, 2018

jreback commented Dec 4, 2018

jreback commented Dec 4, 2018

datapythonista commented Dec 4, 2018

TomAugspurger commented Dec 4, 2018

h-vetinari commented Dec 4, 2018 •

edited

Loading

codecov bot commented Dec 4, 2018 •

edited

Loading

codecov bot commented Dec 4, 2018

TomAugspurger commented Dec 4, 2018

TomAugspurger Dec 4, 2018

TomAugspurger Dec 4, 2018 •

edited

Loading

TomAugspurger commented Dec 4, 2018

TomAugspurger commented Dec 4, 2018

h-vetinari left a comment

h-vetinari Dec 4, 2018

TomAugspurger Dec 4, 2018

TomAugspurger commented Dec 4, 2018

h-vetinari Dec 4, 2018

h-vetinari commented Dec 4, 2018

TomAugspurger commented Dec 4, 2018

h-vetinari commented Dec 4, 2018

TomAugspurger commented Dec 4, 2018

CI: remove failing line #24092

CI: remove failing line #24092

Conversation

TomAugspurger commented Dec 4, 2018

pep8speaks commented Dec 4, 2018

jreback commented Dec 4, 2018

jreback commented Dec 4, 2018

datapythonista commented Dec 4, 2018

TomAugspurger commented Dec 4, 2018

h-vetinari commented Dec 4, 2018 • edited Loading

codecov bot commented Dec 4, 2018 • edited Loading

Codecov Report

codecov bot commented Dec 4, 2018

Codecov Report

TomAugspurger commented Dec 4, 2018

TomAugspurger Dec 4, 2018

Choose a reason for hiding this comment

TomAugspurger Dec 4, 2018 • edited Loading

Choose a reason for hiding this comment

TomAugspurger commented Dec 4, 2018

TomAugspurger commented Dec 4, 2018

h-vetinari left a comment

Choose a reason for hiding this comment

h-vetinari Dec 4, 2018

Choose a reason for hiding this comment

TomAugspurger Dec 4, 2018

Choose a reason for hiding this comment

TomAugspurger commented Dec 4, 2018

h-vetinari Dec 4, 2018

Choose a reason for hiding this comment

h-vetinari commented Dec 4, 2018

TomAugspurger commented Dec 4, 2018

h-vetinari commented Dec 4, 2018

TomAugspurger commented Dec 4, 2018

h-vetinari commented Dec 4, 2018 •

edited

Loading

codecov bot commented Dec 4, 2018 •

edited

Loading

TomAugspurger Dec 4, 2018 •

edited

Loading