CI: Some Azure Pipelines cleanups #23111

vtbassmatt · 2018-10-12T15:46:52Z

Hey all, I'm a PM on Azure Pipelines. I noticed you had some comments in your azure-pipelines.yml regarding conditional steps and possibly adding a Linux flavor. I've taken the liberty of addressing those issues.

One thing I didn't completely finish: getting Linux to publish test results. The problem is that there are directories the publish task can't read already sitting in /tmp. I think the right solution is to use mktemp -d and then use that directory in the test scripts. But that felt like a pretty invasive change that you'd want to discuss before I just powered through and did it :)

Also, the contents of azure-Linux-35.yaml are identical to macOS right now, because I wasn't sure what needed to differ between those environments.

codecov · 2018-10-12T17:08:01Z

Codecov Report

Merging #23111 into master will increase coverage by 49.43%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           master   #23111       +/-   ##
===========================================
+ Coverage   42.52%   91.95%   +49.43%     
===========================================
  Files         161      161               
  Lines       51697    51831      +134     
===========================================
+ Hits        21982    47660    +25678     
+ Misses      29715     4171    -25544

Flag	Coverage Δ
#multiple	`90.61% <ø> (?)`
#single	`42.88% <ø> (+0.36%)`	⬆️

Impacted Files	Coverage Δ
pandas/core/computation/pytables.py	`92.37% <0%> (+0.3%)`	⬆️
pandas/io/pytables.py	`92.3% <0%> (+0.92%)`	⬆️
pandas/util/_test_decorators.py	`93.24% <0%> (+4.05%)`	⬆️
pandas/compat/__init__.py	`58.36% <0%> (+8.17%)`	⬆️
pandas/core/config_init.py	`99.24% <0%> (+9.84%)`	⬆️
pandas/core/reshape/util.py	`100% <0%> (+11.53%)`	⬆️
pandas/compat/numpy/__init__.py	`92.85% <0%> (+14.28%)`	⬆️
pandas/core/computation/common.py	`85.71% <0%> (+14.28%)`	⬆️
pandas/core/api.py	`100% <0%> (+14.81%)`	⬆️
pandas/core/indexes/api.py	`99% <0%> (+14.85%)`	⬆️
... and 118 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d7e96d8...87fb3f9. Read the comment docs.

datapythonista · 2018-10-12T19:45:06Z

Thanks for the clean up @vtbassmatt. I don't know much about the Windows part, but lgtm. @TomAugspurger ?

@vtbassmatt unrelated to this PR, I'm working on moving the linting from travis to azure-pipelines in #22854. If you have feedback about it, that would be great.

alimcmaster1 · 2018-10-13T09:44:23Z

ci/azure-Linux-35.yaml

@@ -0,0 +1,28 @@
+name: pandas


Minor issue but should "Linux" be all lower case in the file name to be consistent with the windows yaml files?

Thanks very much for taking a look at this @vtbassmatt

You're welcome! I was cheating a bit by using the parameters.name as the ENV_FILE file name in the Linux/macOS case. Since it's cased as a display name, I matched that. Maybe I should capitalize Windows to match? Or I can think of another solution. (We don't have string.ToLower in our expression language.)

vtbassmatt · 2018-10-13T11:13:53Z

@datapythonista I will definitely take a look. Probably Monday at this point.

vtbassmatt · 2018-10-16T11:24:27Z

Boo, build is failing on Linux and I don't quite understand why.

TomAugspurger · 2018-10-16T11:30:57Z

I haven't looked too deeply, but the logs in https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=1216&view=logs reference numpy 1.10. We recently bumped our minimum to numpy 1.12.

datapythonista · 2018-11-04T18:36:45Z

@vtbassmatt can you rebase?

vtbassmatt · 2018-11-05T14:11:04Z

Rebased. Hopefully I didn't break anything -- I still have a shadow pipeline definition set up, so I'll know shortly.

vtbassmatt · 2018-11-05T15:02:02Z

Mine looks clean.

TomAugspurger

I think there are some issues here. I believe we lost the following builds (which were added after you started this PR @vtbassmatt)

azure-27-compat
azure-36-locale_slow
azure-37-locale

I don't see a linux build in https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=2458&view=logs

So, could we add an ENV_FILE to the parameters passed to the macos-linux file, and then use those parameters in azure-pipelines.yml to have the following builds

macOS, ci/azure-macos-35.yaml
linux, azure-27-compat
linux, azure-36-locale_slow
linux, azure-37-locale

TomAugspurger · 2018-11-05T22:02:56Z

azure-pipelines.yml

-# Windows Python 2.7 needs VC 9.0 installed, and not sure
-# how to make that a conditional task, so for now these are
-# separate templates as well
+# - template: ci/azure/macos-linux.yml


Is this commented out right now, or is being substituted?

Whoops, I really screwed this rebase up. Will fix it tomorrow.

TomAugspurger · 2018-11-05T22:28:53Z

No worries, I think it's our fault for letting this sit unmerged for too long :) I believe the new linux builds were added after you opened the PR.

…

On Mon, Nov 5, 2018 at 4:14 PM Matt Cooper ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In azure-pipelines.yml <#23111 (comment)>: > -# Windows Python 2.7 needs VC 9.0 installed, and not sure -# how to make that a conditional task, so for now these are -# separate templates as well +# - template: ci/azure/macos-linux.yml Whoops, I really screwed this rebase up. Will fix it tomorrow. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#23111 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIi8ai5vgiGr7_eXdPO-q_306YSyPks5usLg_gaJpZM4XZuip> .

vtbassmatt · 2018-11-09T20:55:05Z

Let's see if that rebase was a little better...

vtbassmatt · 2018-11-14T12:17:23Z

I'm sorry, I wasn't watching this closely enough and missed the build failure.

vtbassmatt · 2018-11-14T13:13:26Z

It looks like the Mac agent freaked out on that last run. It was working in the prior run and I didn't change anything, plus it worked on my private copy of the pipeline. I think this is back in working order.

jreback · 2018-12-03T02:01:58Z

is this PR still relevant? @datapythonista

datapythonista · 2018-12-03T02:19:39Z

yes, that's something we'd like to have, as among other things it avoids repetitive code in the CI config.

@vtbassmatt can you merge master and fix the conflicts? And as I think you're addressing different things here, do you think it makes sense to split this PR, so we can merge the parts that are ready?

vtbassmatt · 2018-12-03T20:47:13Z

I'll take a look tomorrow. I had already resolved a similar rebase before, and it took a bit of finesse to make sure I didn't break anything.

vtbassmatt · 2018-12-04T13:56:46Z

@datapythonista pending the outcome of the Azure build, I think this is ready to merge. Can we please merge it soon so I don't have to rebase again? 😋

datapythonista

Thanks @vtbassmatt, nice job, great to have this simplified.

lgtm, but I'm a bit lost reviewing the Windows part, @TomAugspurger can you take a look please?

datapythonista · 2018-12-04T17:20:13Z

Btw @vtbassmatt I think the problem we've got with Azure not reporting the end of the build to GitHub happens in all the PRs now, including this one. Not sure if it happens to other projects too, but probably worth reporting it again.

TomAugspurger · 2018-12-04T17:22:36Z

I'm working on the benchmarks failures (on azure) and the travis failures right now in #24092.

vtbassmatt · 2018-12-05T15:40:24Z

We think we've spotted the bug - a missing null check when the build errors without a message. We're preparing a hotfix now.

datapythonista · 2018-12-05T15:47:57Z

That's great, thanks @vtbassmatt, and thanks for keeping us updated.

@TomAugspurger, now that #24092 has been merged, if you can take a look at this PR, I think it should be ready to be merged too.

datapythonista · 2018-12-11T00:16:20Z

@TomAugspurger if you can take a look, and merge if you're happy

TomAugspurger · 2018-12-11T12:37:46Z

I probably won't have time to look today, so I'd say go ahead and merge if you're comfortable with the changes.

…

On Mon, Dec 10, 2018 at 6:16 PM Marc Garcia ***@***.***> wrote: @TomAugspurger <https://github.com/TomAugspurger> if you can take a look, and merge if you're happy — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#23111 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIubAK06_mdVogi8L_o53Kr4l_MVHks5u3vlZgaJpZM4XZuip> .

datapythonista · 2018-12-11T15:08:15Z

Double checked the changes, and the build, and everything looks all right. Merging.

Thanks @vtbassmatt

* upstream/master: pct change bug issue 21200 (pandas-dev#21235) DOC: Fix summaries not ending with a period (pandas-dev#24190) DOC: Add back currentmodule to api.rst (pandas-dev#24232) DOC: Fixed implicit imports for whatsnew (v >= version 20.0) (pandas-dev#24199) remove enum import for PY2 compat, xref pandas-dev#22802 (pandas-dev#24170) CI: Simplify Azure Pipelines configuration files (pandas-dev#23111) REF/TST: Add more pytest idiom to indexing/multiindex/test_getitem.py (pandas-dev#24053)

datapythonista added Testing pandas testing functions or related to the test suite CI Continuous Integration labels Oct 12, 2018

alimcmaster1 reviewed Oct 13, 2018

View reviewed changes

vtbassmatt mentioned this pull request Oct 15, 2018

CI: Linting with azure instead of travis #22854

Merged

datapythonista changed the title ~~Some Azure Pipelines cleanups~~ CI: Some Azure Pipelines cleanups Nov 4, 2018

alimcmaster1 mentioned this pull request Nov 5, 2018

Fail Azure CI for failing tests #23454

Merged

2 tasks

TomAugspurger reviewed Nov 5, 2018

View reviewed changes

vtbassmatt added 5 commits December 4, 2018 08:47

unify macOS and Linux

77203a0

remove old macOS and Linux defs

bbb3379

unify Windows flavors

e2fb16f

remove old Windows def

e9c66fd

fix format function

87fb3f9

datapythonista approved these changes Dec 4, 2018

View reviewed changes

jreback added this to the 0.24.0 milestone Dec 5, 2018

datapythonista self-assigned this Dec 11, 2018

datapythonista merged commit 9ea8176 into pandas-dev:master Dec 11, 2018

Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

CI: Simplify Azure Pipelines configuration files (pandas-dev#23111)

2cf4290

Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

CI: Simplify Azure Pipelines configuration files (pandas-dev#23111)

4b7dc81

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: Some Azure Pipelines cleanups #23111

CI: Some Azure Pipelines cleanups #23111

vtbassmatt commented Oct 12, 2018

codecov bot commented Oct 12, 2018 •

edited

Loading

datapythonista commented Oct 12, 2018

alimcmaster1 Oct 13, 2018

alimcmaster1 Oct 13, 2018

vtbassmatt Oct 13, 2018

vtbassmatt commented Oct 13, 2018

vtbassmatt commented Oct 16, 2018 •

edited

Loading

TomAugspurger commented Oct 16, 2018

datapythonista commented Nov 4, 2018

vtbassmatt commented Nov 5, 2018

vtbassmatt commented Nov 5, 2018

TomAugspurger left a comment

TomAugspurger Nov 5, 2018

vtbassmatt Nov 5, 2018

TomAugspurger commented Nov 5, 2018 via email

vtbassmatt commented Nov 9, 2018

vtbassmatt commented Nov 14, 2018

vtbassmatt commented Nov 14, 2018

jreback commented Dec 3, 2018

datapythonista commented Dec 3, 2018

vtbassmatt commented Dec 3, 2018

vtbassmatt commented Dec 4, 2018

datapythonista left a comment

datapythonista commented Dec 4, 2018

TomAugspurger commented Dec 4, 2018 •

edited

Loading

vtbassmatt commented Dec 5, 2018 •

edited

Loading

datapythonista commented Dec 5, 2018

datapythonista commented Dec 11, 2018

TomAugspurger commented Dec 11, 2018 via email

datapythonista commented Dec 11, 2018

CI: Some Azure Pipelines cleanups #23111

CI: Some Azure Pipelines cleanups #23111

Conversation

vtbassmatt commented Oct 12, 2018

codecov bot commented Oct 12, 2018 • edited Loading

Codecov Report

datapythonista commented Oct 12, 2018

alimcmaster1 Oct 13, 2018

Choose a reason for hiding this comment

alimcmaster1 Oct 13, 2018

Choose a reason for hiding this comment

vtbassmatt Oct 13, 2018

Choose a reason for hiding this comment

vtbassmatt commented Oct 13, 2018

vtbassmatt commented Oct 16, 2018 • edited Loading

TomAugspurger commented Oct 16, 2018

datapythonista commented Nov 4, 2018

vtbassmatt commented Nov 5, 2018

vtbassmatt commented Nov 5, 2018

TomAugspurger left a comment

Choose a reason for hiding this comment

TomAugspurger Nov 5, 2018

Choose a reason for hiding this comment

vtbassmatt Nov 5, 2018

Choose a reason for hiding this comment

TomAugspurger commented Nov 5, 2018 via email

vtbassmatt commented Nov 9, 2018

vtbassmatt commented Nov 14, 2018

vtbassmatt commented Nov 14, 2018

jreback commented Dec 3, 2018

datapythonista commented Dec 3, 2018

vtbassmatt commented Dec 3, 2018

vtbassmatt commented Dec 4, 2018

datapythonista left a comment

Choose a reason for hiding this comment

datapythonista commented Dec 4, 2018

TomAugspurger commented Dec 4, 2018 • edited Loading

vtbassmatt commented Dec 5, 2018 • edited Loading

datapythonista commented Dec 5, 2018

datapythonista commented Dec 11, 2018

TomAugspurger commented Dec 11, 2018 via email

datapythonista commented Dec 11, 2018

codecov bot commented Oct 12, 2018 •

edited

Loading

vtbassmatt commented Oct 16, 2018 •

edited

Loading

TomAugspurger commented Dec 4, 2018 •

edited

Loading

vtbassmatt commented Dec 5, 2018 •

edited

Loading