CI: Check ASV for failed benchmarks #19236

mroeschke · 2018-01-14T07:25:32Z

closes #19518

xref #19104

Not sure if this should belong in ci/lint.sh, but testing the possibility of running the asv benchmarks on ci for failed benchmarks.

jreback · 2018-01-14T14:27:27Z

ci/lint.sh

@@ -125,6 +125,16 @@ if [ "$LINT" ]; then
        RET=1
    fi
    echo "Check for deprecated messages without sphinx directive DONE"
+


put this in a separete script (ci/asv.sh)

and we can enable it in (.travis.yml) via an ENV variable (e.g. like we do for coverage).

Then its easy to put it on only 1 build.

also need to install asv (on that build)

codecov · 2018-01-14T18:55:33Z

Codecov Report

Merging #19236 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #19236   +/-   ##
=======================================
  Coverage   91.62%   91.62%           
=======================================
  Files         150      150           
  Lines       48680    48680           
=======================================
  Hits        44603    44603           
  Misses       4077     4077

Flag	Coverage Δ
#multiple	`89.99% <ø> (ø)`	⬆️
#single	`41.73% <ø> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e8620ab...54727be. Read the comment docs.

mroeschke · 2018-01-14T19:04:09Z

Created asv.sh. I'm not too familiar where to enable the script in travis.yml.

jreback · 2018-01-14T21:19:04Z

I pushed a commit to add this to travis. let's see what it does.

mroeschke · 2018-01-15T03:58:45Z

Hmm so the builds are returning /home/travis/.travis/job_stages: line 57: ci/asv.sh: Permission denied

Do I need to check in the file with execution permissions? https://stackoverflow.com/a/42187039

jreback · 2018-01-15T04:58:04Z

it should be chmod 755

mroeschke · 2018-01-15T20:39:56Z

Looks like travis is all green.

jreback · 2018-01-15T23:50:00Z

I am not convinced this actually did anything. sure it installed but it doesn't say it even ran.

23.33s$ ci/asv.sh
inside ci/asv.sh
Check for failed asv benchmarks
Collecting git+https://github.com/spacetelescope/asv
  Cloning https://github.com/spacetelescope/asv to /tmp/pip-Dzkkcb-build
Requirement already satisfied: six>=1.4 in /home/travis/miniconda3/envs/pandas/lib/python2.7/site-packages (from asv==0.3.dev1207+7edf1903)
Installing collected packages: asv
  Running setup.py install for asv ... done
Successfully installed asv-0.3.dev1207+7edf1903
real	0m0.410s
user	0m0.361s
sys	0m0.048s
Check for failed asv benchmarks DONE

jreback · 2018-01-15T23:49:23Z

ci/asv.sh

+    fi
+    echo "Check for failed asv benchmarks DONE"
+fi
+


can you add an else like we do in lint. so that we get a message if this is NOT run.

jreback · 2018-01-15T23:51:22Z

ci/asv.sh

+    fi
+    echo "Check for failed asv benchmarks DONE"
+fi
+


can you add an else like we do in lint. so that we get a message if this is NOT run.

jreback · 2018-01-15T23:52:48Z

ci/asv.sh

+
+    cd asv_bench
+
+    time asv run --quick | grep "failed"


once we are convinced these run, then can add it back

jreback · 2018-01-16T00:15:32Z

here was a run on my local system

real    21m8.666s
user    18m20.882s
sys     2m47.438s

so for sure this has to be a separate build.

mroeschke · 2018-01-16T04:43:07Z

Added the else statement. I also think this may not be working because asv wants to know the machine configurations upon install, so I added an asv machine --yes statement.

mroeschke · 2018-01-16T17:57:08Z

Here's the latest output from Travis:

$ ci/asv.sh
inside ci/asv.sh
Check for failed asv benchmarks
Collecting git+https://github.com/spacetelescope/asv
  Cloning https://github.com/spacetelescope/asv to /tmp/pip-eJJWMU-build
Requirement already satisfied: six>=1.4 in /home/travis/miniconda3/envs/pandas/lib/python2.7/site-packages (from asv==0.3.dev1207+7edf1903)
Installing collected packages: asv
  Running setup.py install for asv ... done
Successfully installed asv-0.3.dev1207+7edf1903
· No information stored about machine 'travis-job-pandas-dev-pandas-329321549.travisci.net'. I know about nothing.
            
I will now ask you some questions about this machine to identify it in the benchmarks.
1. machine:  A unique name to identify this machine in the results.
   May be anything, as long as it is unique across all the machines
   used to benchmark this project.  NOTE: If changed from the default,
   it will no longer match the hostname of this machine, and you may
   need to explicitly use the --machine argument to asv.
machine [travis-job-pandas-dev-pandas-329321549.travisci.net]: 2. os:  The OS type and version of this machine.  For example,
   'Macintosh OS-X 10.8'.
os [Linux 4.14.12-041412-generic]: 3. arch:  The generic CPU architecture of this machine.  For example,
   'i386' or 'x86_64'.
arch [x86_64]: 4. cpu:  A specific description of the CPU of this machine, including
   its speed and class.  For example, 'Intel(R) Core(TM) i5-2520M CPU
   @ 2.50GHz (4 cores)'.
cpu [Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz]: 5. ram:  The amount of physical RAM on this machine.  For example,
   '4GB'.
ram [61815180]: 
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
The build has been terminated

So it appears travis is running the build, but it thinks that there's a build issue because the ASV benchmarks take a while to run.

jreback · 2018-01-17T00:39:08Z

ok I pushed to a separate build. if this works, then we would simply capture the output (as well as output to the screen). then grep it. you have to keep showing output otherwise things will timeout.

mroeschke · 2018-01-17T05:42:34Z

Thanks @jreback

It looks like the alternative is to prefix the asv script execution with - travis_wait 30 ci/asv.sh to tell travis to wait for 30 minutes, or more if needed.

https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received

mroeschke · 2018-01-17T20:21:07Z

Darn, looks like the build is taking to long. It ran for 49 min 57 sec and output:

The job exceeded the maximum time limit for jobs, and has been terminated.

Looks like the max time for travis-ci.org jobs is 50 minutes.

https://docs.travis-ci.com/user/customizing-the-build#Build-Timeouts

jreback · 2018-01-18T00:21:33Z

i restarted that job again, i think this should complete

mroeschke · 2018-01-18T17:57:57Z

Looks like the job hit the time limit again.

It looks like ci/install_travis.sh itself took ~11 minutes. I'm not sure if there anything in this script that can be skipped to allow more time for the asv script

jorisvandenbossche · 2018-01-19T09:13:44Z

ci/asv.sh

    cd asv_bench

    asv machine --yes

-    time asv run --quick | grep "failed"
+    time asv run --quick


Not directly related to this change (rather to the previous PR that introduced this), but shouldn't this be asv dev ? Because I thought asv run would create a new env, while everything already has been installed on travis?

The link below mentions about asv dev

This runs a benchmark suite in a mode that is useful during development. It is equivalent to ``asv run --quick --show-stderr --python=same``

So the command should be essentially the same asv dev, but I can still change it to asv dev nonetheless.

Ref: https://asv.readthedocs.io/en/latest/commands.html#asv-dev

There is an essential difference between what you quote (asv run --quick --show-stderr --python=same) and what is above, namely the --python=same.

And I also think --show-stderr is useful for on travis, to be able to easier see what is failing.

jorisvandenbossche · 2018-01-19T09:15:04Z

.travis.yml

+    # In allow_failures
+    - dist: trusty
+      env:
+        - JOB="3.6_ASV" ASV=true


Could you put this just before the docs build? (I think I will more refer to the doc build, and then it is easier to say the last build :-))

jreback · 2018-01-23T11:34:33Z

I would not grep for the output yet, this obscure the progress; let's see how far it gets.

mroeschke · 2018-01-25T17:33:40Z

@jreback Looks like the build ran the asv within the time limit:

real	36m57.758s
user	32m53.399s
sys	3m42.420s
Check for failed asv benchmarks DONE

I will move forward with greping the output if all looks good.

jreback · 2018-01-25T17:41:54Z

what changed to make this work?

mroeschke · 2018-01-25T18:54:54Z

I think the change from asv run to asv dev made this work.

My next commit will be to grep the output and see if it still runs within the time limit of the build.

mroeschke · 2018-01-26T00:35:27Z

After adding a grep, received

No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received

Will now be adding a travis wait

mroeschke · 2018-01-26T17:49:34Z

So I think it works now? I guess I could also intentionally make one of the benchmarks wrong to see if this "exited with 1". Would appreciate a second look @jreback

From the build:

$ travis_wait 40 ci/asv.sh
Still running (33 of 40): ci/asv.sh
The command ci/asv.sh exited with 0.
Log:
inside ci/asv.sh
Check for failed asv benchmarks
· No information stored about machine 'travis-job-pandas-dev-pandas-333531431.travisci.net'. I know about nothing.
            
I will now ask you some questions about this machine to identify it in the benchmarks.
1. machine:  A unique name to identify this machine in the results.
   May be anything, as long as it is unique across all the machines
   used to benchmark this project.  NOTE: If changed from the default,
   it will no longer match the hostname of this machine, and you may
   need to explicitly use the --machine argument to asv.
machine [travis-job-pandas-dev-pandas-333531431.travisci.net]: 2. os:  The OS type and version of this machine.  For example,
   'Macintosh OS-X 10.8'.
os [Linux 4.14.12-041412-generic]: 3. arch:  The generic CPU architecture of this machine.  For example,
   'i386' or 'x86_64'.
arch [x86_64]: 4. cpu:  A specific description of the CPU of this machine, including
   its speed and class.  For example, 'Intel(R) Core(TM) i5-2520M CPU
   @ 2.50GHz (4 cores)'.
cpu [Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz]: 5. ram:  The amount of physical RAM on this machine.  For example,
   '4GB'.
ram [61815180]: 
real	32m40.209s
user	29m11.933s
sys	3m24.802s
Check for failed asv benchmarks DONE
/home/travis/.travis/job_stages: line 169:  4585 Terminated              travis_jigger $! $timeout $cmd
The command "travis_wait 40 ci/asv.sh" exited with 0.

jreback · 2018-01-27T17:47:18Z

.travis.yml

@@ -73,6 +73,10 @@ matrix:
      env:
        - JOB="3.6_NUMPY_DEV" TEST_ARGS="--skip-slow --skip-network" PANDAS_TESTING_MODE="deprecate"
    # In allow_failures
+    - dist: trusty


move this after the doc build

@jorisvandenbossche prefered this before the doc build

#19236 (comment)

jreback · 2018-01-27T17:47:29Z

.travis.yml

@@ -93,6 +97,9 @@ matrix:
      - dist: trusty
        env:
          - JOB="3.6_NUMPY_DEV" TEST_ARGS="--skip-slow --skip-network" PANDAS_TESTING_MODE="deprecate"
+      - dist: trusty


@jorisvandenbossche prefered this before the doc build

#19236 (comment)

jreback · 2018-01-27T17:48:10Z

ci/asv.sh

+    asv machine --yes
+
+	RET=0
+


i would tee here; i like the output to the terminal

mroeschke · 2018-01-30T18:00:30Z

@jreback Okay I think this works now. The build failure for #35124.4 is unrelated, and the asv build "failed" because some of the rolling benchmarks for cov and corr took more than 60 seconds.

jreback · 2018-01-31T11:14:51Z

looking good. a couple of things.

would catch future warnings for most things (you can use catch_warnings so its generic) as these are distracting
would tee to a file of the failed to a tmp file, then cat it at the end (and put some echos around this to make it prominent). So that we can see what exactly failed.

mroeschke · 2018-02-02T22:08:42Z

Warnings have been caught and now checking failed benchmarks through a file, failed_asv.txt

...
[100.00%] ··· Running ...a.TimedeltaProperties.time_timedelta_seconds     4.20μs
real	33m10.909s
user	29m41.485s
sys	3m37.056s
The following asvs benchmarks (if any) failed.
DONE displaying failed asvs benchmarks.
Check for failed asv benchmarks DONE

jorisvandenbossche · 2018-02-03T10:02:49Z

@mroeschke If it is not too difficult, can you do the clean-up of the benchmarks for rolling in a separate PR?

jorisvandenbossche · 2018-02-03T10:04:01Z

asv_bench/benchmarks/gil.py

@@ -1,8 +1,6 @@
 import numpy as np
 import pandas.util.testing as tm
-from pandas import (DataFrame, Series, rolling_median, rolling_mean,
-                    rolling_min, rolling_max, rolling_var, rolling_skew,
-                    rolling_kurt, rolling_std, read_csv, factorize, date_range)


I might be 'overhead' to keep, but we can also simply move those imports to withing the else statement, and then this keeps working for old releases.

create asv.sh add to travis Changed file permission Add else statement and asv machine config move to separate travis build CI: Run ASV on Travis for failed benchmarks create asv.sh add to travis move to separate travis build Change build order and change to asv dev Put asv before doc Fix some failing benchmarks Grep for failed benchmarks Add travis wait Don't grep output yet Remove travis_wait for now fix some benchmarks Now grep output add travis wait 40 to ci/asv.sh Add tee Remove wait Start catching warnings catch more warnings Fix ci strategy Remove rolling_* from gil.py catch more warnings Add back rolling methods for older releases simplify rolling setup

mroeschke · 2018-02-03T19:01:56Z

@jorisvandenbossche Added back the rolling methods. Sorry I saw your other comment too late and squashed my commits where I made edits for the rolling methods. Is it still possible to cherry pick from a squashed commit?

mroeschke · 2018-02-06T07:46:52Z

Ping @jorisvandenbossche any additional feedback?

jreback · 2018-02-08T01:18:16Z

thanks @mroeschke

let's see how this works. its the longest build, but ok for now.

jreback reviewed Jan 14, 2018

View reviewed changes

jreback added the Benchmark Performance (ASV) benchmarks label Jan 14, 2018

mroeschke force-pushed the asv_travis branch from b8f31c2 to 2801e02 Compare January 15, 2018 06:42

jreback requested changes Jan 15, 2018

View reviewed changes

jreback reviewed Jan 15, 2018

View reviewed changes

ci/asv.sh Outdated

cd asv_bench

time asv run --quick | grep "failed"

Copy link

Contributor

jreback Jan 15, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

once we are convinced these run, then can add it back

jorisvandenbossche reviewed Jan 19, 2018

View reviewed changes

mroeschke force-pushed the asv_travis branch from f0e4b34 to dcfffd9 Compare January 20, 2018 19:16

mroeschke force-pushed the asv_travis branch 2 times, most recently from 2b29b7a to 55da335 Compare January 25, 2018 03:22

jreback requested changes Jan 27, 2018

View reviewed changes

mroeschke force-pushed the asv_travis branch 3 times, most recently from a9b995f to c71ee2d Compare January 30, 2018 05:46

mroeschke force-pushed the asv_travis branch from c71ee2d to 640ace9 Compare January 31, 2018 07:20

mroeschke force-pushed the asv_travis branch 3 times, most recently from 5a94403 to a1bf2c8 Compare February 2, 2018 20:17

mroeschke mentioned this pull request Feb 2, 2018

asv ImportError #19518

Closed

jorisvandenbossche reviewed Feb 3, 2018

View reviewed changes

mroeschke force-pushed the asv_travis branch from a1bf2c8 to b8be7fd Compare February 3, 2018 18:54

mroeschke force-pushed the asv_travis branch from b8be7fd to 54727be Compare February 3, 2018 18:57

mroeschke mentioned this pull request Feb 7, 2018

ASV Benchmarks Broken #19586

Closed

WillAyd mentioned this pull request Feb 7, 2018

PERF: Cythonize Groupby Rank #19481

Merged

4 tasks

jreback added this to the 0.23.0 milestone Feb 8, 2018

jreback approved these changes Feb 8, 2018

View reviewed changes

jreback merged commit affb5d9 into pandas-dev:master Feb 8, 2018

mroeschke deleted the asv_travis branch February 8, 2018 07:36

mroeschke mentioned this pull request Feb 9, 2018

Numerous FutureWarning with asv_bench #17732

Closed

harisbal pushed a commit to harisbal/pandas that referenced this pull request Feb 28, 2018

CI: Run ASV on Travis for failed benchmarks (pandas-dev#19236)

f9db3b5

CI: Check ASV for failed benchmarks #19236

CI: Check ASV for failed benchmarks #19236

Conversation

mroeschke commented Jan 14, 2018 • edited Loading

Choose a reason for hiding this comment

codecov bot commented Jan 14, 2018 • edited Loading

Codecov Report

mroeschke commented Jan 14, 2018

jreback commented Jan 14, 2018

mroeschke commented Jan 15, 2018

jreback commented Jan 15, 2018

mroeschke commented Jan 15, 2018

jreback commented Jan 15, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Jan 16, 2018

mroeschke commented Jan 16, 2018

mroeschke commented Jan 16, 2018 • edited Loading

jreback commented Jan 17, 2018

mroeschke commented Jan 17, 2018

mroeschke commented Jan 17, 2018

jreback commented Jan 18, 2018

mroeschke commented Jan 18, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Jan 23, 2018

mroeschke commented Jan 25, 2018

jreback commented Jan 25, 2018

mroeschke commented Jan 25, 2018 • edited Loading

mroeschke commented Jan 26, 2018

mroeschke commented Jan 26, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mroeschke commented Jan 30, 2018 • edited Loading

jreback commented Jan 31, 2018

mroeschke commented Feb 2, 2018

jorisvandenbossche commented Feb 3, 2018

Choose a reason for hiding this comment

mroeschke commented Feb 3, 2018

mroeschke commented Feb 6, 2018

jreback commented Feb 8, 2018

mroeschke commented Jan 14, 2018 •

edited

Loading

codecov bot commented Jan 14, 2018 •

edited

Loading

mroeschke commented Jan 16, 2018 •

edited

Loading

mroeschke commented Jan 25, 2018 •

edited

Loading

mroeschke commented Jan 26, 2018 •

edited

Loading

mroeschke commented Jan 30, 2018 •

edited

Loading