conda.recipe #281

jreback · 2016-11-03T16:30:41Z

closes #210

builds a clr conda package
builds for 2.7/3.5 on windows (will simply fail on other platforms), should be easy to extend though
fixes the non-standard dev tag (-dev is not valid, should be .dev, but no big deal)
release tags should work, but not actually tests (e.g. should generate clr-v2.2.0-py35
the actual installation should really be into a site-packages\clr\ sub-dir in the target, but this just mimics how the wheels work.
agreement to transition to MIT license
fix old msbuild without C# 6.0 support for python 3.3 and 3.4 or add C# 6.0 compiler as nuget dependency
pip 9.0 issues
keep both python and conda builds by either having two appveyor.yml and two server configuration or merge them into one.

den-run-ai · 2016-11-03T17:03:26Z

@jreback can you add a note how to build/install this? I did not find this in conda docs.

jreback · 2016-11-03T17:22:04Z

sure where would you like the note? further this required conda-build for actually executing this (do you have a dev requirements list somewhere?)

e.g.

conda build conda.recipe --python 3.5

den-run-ai · 2016-11-03T17:45:17Z

@jreback you can add the note in wiki - it is open to everyone:

https://github.com/pythonnet/pythonnet/wiki/Installation

We do not have requirements files yet, due to dependency on .NET/Mono tools and nuget. But I opened an issue for this, since it is a good idea anyway:

#282

I tried your instruction and it is failing badly on me with latest conda and conda-build.

C:\Users\denis.akhiyarov\Downloads\pythonnet>cd pythonnet-clr-recipe\

C:\Users\denis.akhiyarov\Downloads\pythonnet\pythonnet-clr-recipe>conda build co
nda.recipe --python 2.7
Cloning into bare repository 'C:\Python\Python27_32b\conda-bld\git_cache\C_\User
s\denis.akhiyarov\Downloads\pythonnet\pythonnet-clr-recipe'...
fatal: 'C:\Users\denis.akhiyarov\Downloads\pythonnet\pythonnet-clr-recipe' does
not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Cloning into bare repository 'C:\Python\Python27_32b\conda-bld\git_cache\C_\User
s\denis.akhiyarov\Downloads\pythonnet\pythonnet-clr-recipe'...
fatal: 'C:\Users\denis.akhiyarov\Downloads\pythonnet\pythonnet-clr-recipe' does
not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Warning: failed to download source.  If building, will try again after downloadi
ng recipe dependencies.
Error was:
Command '['C:\\Users\\denis.akhiyarov\\AppData\\Local\\Atlassian\\SourceTree\\gi
t_local\\bin\\git.exe', 'clone', '--mirror', 'C:\\Users\\denis.akhiyarov\\Downlo
ads\\pythonnet\\pythonnet-clr-recipe', 'C:\\Python\\Python27_32b\\conda-bld\\git
_cache\\C_\\Users\\denis.akhiyarov\\Downloads\\pythonnet\\pythonnet-clr-recipe']
' returned non-zero exit status 128
BUILD START: clr--py27_0
    (actual version deferred until further download or env creation)
updating index in: C:\Python\Python27_32b\conda-bld\win-32
updating index in: C:\Python\Python27_32b\conda-bld\noarch

The following NEW packages will be INSTALLED:

    pip:            8.1.2-py27_0
    python:         2.7.12-0
    setuptools:     27.2.0-py27_1
    vs2008_runtime: 9.00.30729.1-2
    wheel:          0.29.0-py27_0

Cloning into bare repository 'C:\Python\Python27_32b\conda-bld\git_cache\C_\User
s\denis.akhiyarov\Downloads\pythonnet\pythonnet-clr-recipe'...
fatal: 'C:\Users\denis.akhiyarov\Downloads\pythonnet\pythonnet-clr-recipe' does
not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Cloning into bare repository 'C:\Python\Python27_32b\conda-bld\git_cache\C_\User
s\denis.akhiyarov\Downloads\pythonnet\pythonnet-clr-recipe'...
fatal: 'C:\Users\denis.akhiyarov\Downloads\pythonnet\pythonnet-clr-recipe' does
not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Warning: failed to download source.  If building, will try again after downloadi
ng recipe dependencies.
Error was:
Command '['C:\\Users\\denis.akhiyarov\\AppData\\Local\\Atlassian\\SourceTree\\gi
t_local\\bin\\git.exe', 'clone', '--mirror', 'C:\\Users\\denis.akhiyarov\\Downlo
ads\\pythonnet\\pythonnet-clr-recipe', 'C:\\Python\\Python27_32b\\conda-bld\\git
_cache\\C_\\Users\\denis.akhiyarov\\Downloads\\pythonnet\\pythonnet-clr-recipe']
' returned non-zero exit status 128
Traceback (most recent call last):
  File "C:\Python\Python27_32b\Scripts\conda-build-script.py", line 5, in <modul
e>
    sys.exit(conda_build.cli.main_build.main())
  File "C:\Python\Python27_32b\lib\site-packages\conda_build\cli\main_build.py",
 line 242, in main
    execute(sys.argv[1:])
  File "C:\Python\Python27_32b\lib\site-packages\conda_build\cli\main_build.py",
 line 234, in execute
    already_built=None, config=config)
  File "C:\Python\Python27_32b\lib\site-packages\conda_build\api.py", line 77, i
n build
    need_source_download=need_source_download, config=config)
  File "C:\Python\Python27_32b\lib\site-packages\conda_build\build.py", line 109
9, in build_tree
    config=recipe_config)
  File "C:\Python\Python27_32b\lib\site-packages\conda_build\build.py", line 667
, in build
    assert not need_source_download, "Source download failed.  Please investigat
e."
AssertionError: Source download failed.  Please investigate.

jreback · 2016-11-03T17:52:02Z

@denfromufa you need to be one level up and make sure this is in fact a git repository. This is not a source build as the version numbers need git.

den-run-ai · 2016-11-03T17:59:13Z

I downloaded your git branch from github and extracted it. Then I tried all
sorts of combinations and all fail:

C:\Users\denis.akhiyarov\Downloads\pythonnet>conda build
pythonnet-clr-recipe\conda.recipe

C:\Users\denis.akhiyarov\Downloads\pythonnet>conda build
pythonnet-clr-recipe --python 2.7

On Thu, Nov 3, 2016 at 12:52 PM, Jeff Reback notifications@github.com
wrote:

@denfromufa https://github.com/denfromufa you need to be one level up
and make sure this is in fact a git repository.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#281 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHgZ5XnUhjpwMrfcsiYV0exnvqbdM9xsks5q6h9EgaJpZM4KonMM
.

jreback · 2016-11-03T18:03:52Z

@denfromufa I doubt this actually works on 32bit. 64-bit only.

jreback · 2016-11-03T18:49:02Z

@denfromufa ok give a try again, my recipe was totally messed up, was using some older stuff

den-run-ai · 2016-11-03T19:36:11Z

Does this conda recipe work on Appveyor? There are MiniConda
environments available there. We cannot just include recipes without CI.
I will try to test this on Python 2.7/3.5, 64-bit, but why is 32-bit not
supported? This seems very weird.
Can conda build directly work from git, like pip (pip install git+
https://github.com/pythonnet/pythonnet)?

On Thu, Nov 3, 2016, 1:49 PM Jeff Reback notifications@github.com wrote:

@denfromufa https://github.com/denfromufa ok give a try again, my
recipe was totally messed up, was using some older stuff

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
#281 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHgZ5baUoMfu0Gpk-gTs9f1lROJEN87lks5q6iyfgaJpZM4KonMM
.

den-run-ai · 2016-11-06T03:11:52Z

do you agree to MIT license transition (ZPL 2.0 currently), can you keep pip-based builds (otherwise we need to keep 2 appveyor files), why is it only working on python 2.7?

jreback · 2016-11-06T03:30:12Z

my change actually is working fine
but pip 9.0 broke all of the builds (which is used elsewhere in appveyor)

den-run-ai · 2016-11-06T04:46:41Z

@jreback oh I see one instance like this with pip 9.0 warning on py35_64bit:

https://ci.appveyor.com/project/pythonnet/pythonnet-0kq5d/build/1.0.71/job/kkeja6tft2bhs697#L524

This is most likely issue with Appveyor CI, which incorrectly tags a pip upgrade warning as an exception.

What I see in other instances is that build is failing on py34, py33 due to something else:

https://ci.appveyor.com/project/pythonnet/pythonnet-0kq5d/build/1.0.71/job/8au8ysiax64gjouy#L398
https://ci.appveyor.com/project/pythonnet/pythonnet-0kq5d/build/1.0.71/job/acm5axfij3v3ved4#L395

This is due to using pre- C# 6.0 compiler like in this issue:

#247

I'm not sure how this happened to your branch! Especially since it compiles fine on py27 builds. Let me look closer.

den-run-ai · 2016-11-06T05:51:41Z

@jreback the pip 9.0 issue on Python35-x64 is caused by these global settings in your appveyor.yml:

PYTHON_BLD: "C:\Python35-x64"
PYTHON_BLD_VERSION: "3.5"
PYTHON_BLD_ARCH: "x64"

It looks like pip is not updated to version 9.0 in this directory %PYTHON_BLD%, but used during pythonnet installation at the end.

See successful and failing logs are here:

py35_32bit:
https://ci.appveyor.com/project/vmuriart/pythonnet-olooj/build/1.0.90/job/cxi6138w6xogu2qb

py35_64_bit:
https://ci.appveyor.com/project/vmuriart/pythonnet-olooj/build/1.0.90/job/nvowcn5argpobapy

The quick solution is probably to add this line in install section of appveyor.yml:

"%PYTHON_BLD%\python.exe -m pip install --upgrade pip"

den-run-ai · 2016-11-09T07:15:15Z

@jreback I added what needs to be done before this pull request can be merged. But more importantly if you agree to MIT license transition, then someone can continue this effort. Can you please confirm?

BTW, what is the benefit of using conda to build pip wheels and then install it with pip?

jreback · 2016-11-09T11:36:11Z

i am away

ok on license change

conda packages are much easier to distribute esp when using a custom environment / much more standard in the community

just have both
ideally this recipe should be on conda-forge (free service); which can auto build on releases

jreback · 2016-11-09T11:37:48Z

these are not pip wheels at all
they are conda packages (you build one or the other)

further to above users can then have a dependency on this package (can't do this with wheels)

filmor · 2016-11-09T13:01:59Z

Just as a heads up, I'm in favour of merging this (using conda myself) as soon as the remaining issues have been sorted out.

jreback · 2016-11-15T19:03:21Z

ok, this is finally passing. turns out that you have to use a modern MSBuild even though you are using an older compiler (only matters on 3.3/3.4).

den-run-ai · 2016-11-16T03:24:33Z

@jreback this looks great! I see now that both a conda package and a wheel are built.

Regarding adding an MIT license clause, can you please change this to ZPL 2.0 for now? There is still one contributor @tiran blocking transition to MIT license.

Can you configure this to upload to anaconda.org once pythonnet account is "unlocked"?

https://ci.appveyor.com/project/filmor/pythonnet/build/1.0.119/job/yalrdor5g871vujn#L533

I'm still having an issue with login to pythonnet account on anaconda.org:

anaconda/anaconda-client#375

den-run-ai · 2016-11-16T03:26:35Z

@jreback one more item - do you know how to expose the conda package as an artifact on appveyor?

Right now only wheels are detected as downloadable artifacts by appveyor:

https://ci.appveyor.com/project/filmor/pythonnet/build/1.0.119/job/yalrdor5g871vujn/artifacts

den-run-ai · 2016-11-16T05:24:21Z

@jreback to expose conda package as artifact, please add a post-build script to copy from %PYTHON_BLD%\conda-bld\**\*.tar.bz2 to %APPVEYOR_BUILD_FOLDER%\dist\

jreback · 2016-11-16T13:03:23Z

@denfromufa
generally people use https://github.com/conda-forge/staged-recipes to actually make the conda packages. This is nice because they are generated automatically (via a pull-request to the repo), generally for the released versions, and are populated to the conda-forge channel.

This recipe is designed for development builds, which you could also host on anaconda.

closes pythonnet#210

jreback · 2016-11-16T16:21:18Z

@denfromufa ok this will now put the conda package in the dist folder (and now actually builds correct win-32/win-64). Since you normally upload the artifacts to anaconda where they are put in the correct win-32 or win-64 folder (as they are named the same), I am not sure that they artifacts are actually handled correctly by appveyor.

den-run-ai · 2016-11-16T17:17:43Z

@filmor @vmuriart @tonyroberts please review!

vmuriart · 2016-11-17T04:08:19Z

I'm traveling so only got a quick glance at the code but... did we remove the original builds and only kept the conda recipe?

vmuriart · 2016-11-17T04:09:16Z

And i just read the first post having an item check for this, so i guess that answered my question. nvm! 🤕

den-run-ai · 2016-11-17T21:05:42Z

let's merge this for next release, so please wait

filmor · 2016-11-23T10:44:02Z

@denfromufa Am I understanding you correctly in that we should hold off merging this for now? If yes, why?

den-run-ai · 2016-11-24T20:22:06Z

@filmor I'm merging this for v2.2.0

vmuriart · 2017-01-12T02:35:33Z

@jreback I just noticed that Appveyor enviroments have miniconda installed on them. Can we use those instead of downloading them for the tests?

jreback force-pushed the clr-recipe branch 3 times, most recently from 77d06f5 to 0325757 Compare November 3, 2016 20:15

den-run-ai closed this Nov 6, 2016

den-run-ai reopened this Nov 6, 2016

jreback force-pushed the clr-recipe branch 5 times, most recently from b629848 to a1b001c Compare November 15, 2016 18:37

den-run-ai mentioned this pull request Nov 16, 2016

how to expose conda packages as artifacts? appveyor/ci#1174

Closed

den-run-ai mentioned this pull request Nov 16, 2016

pip upgrade warning breaks builds appveyor/ci#1175

Open

jreback force-pushed the clr-recipe branch 8 times, most recently from 2b2978f to 4ceebf4 Compare November 16, 2016 15:04

conda.recipe

8112c23

closes pythonnet#210

jreback force-pushed the clr-recipe branch from 4ceebf4 to 8112c23 Compare November 16, 2016 16:19

den-run-ai approved these changes Nov 16, 2016

View reviewed changes

IlyaFinkelshteyn mentioned this pull request Nov 16, 2016

Upgrade PIP to latest version appveyor/ci#1180

Closed

filmor approved these changes Nov 17, 2016

View reviewed changes

Merge branch 'master' into clr-recipe

b3d7c89

den-run-ai merged commit e1b6384 into pythonnet:master Nov 24, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

conda.recipe #281

conda.recipe #281

jreback commented Nov 3, 2016 •

edited

Loading

den-run-ai commented Nov 3, 2016

jreback commented Nov 3, 2016

den-run-ai commented Nov 3, 2016

jreback commented Nov 3, 2016 •

edited

Loading

den-run-ai commented Nov 3, 2016

jreback commented Nov 3, 2016

jreback commented Nov 3, 2016

den-run-ai commented Nov 3, 2016

den-run-ai commented Nov 6, 2016

jreback commented Nov 6, 2016

den-run-ai commented Nov 6, 2016

den-run-ai commented Nov 6, 2016 •

edited

Loading

den-run-ai commented Nov 9, 2016 •

edited

Loading

jreback commented Nov 9, 2016

jreback commented Nov 9, 2016

filmor commented Nov 9, 2016

jreback commented Nov 15, 2016

den-run-ai commented Nov 16, 2016

den-run-ai commented Nov 16, 2016

den-run-ai commented Nov 16, 2016

jreback commented Nov 16, 2016

jreback commented Nov 16, 2016

den-run-ai commented Nov 16, 2016

vmuriart commented Nov 17, 2016

vmuriart commented Nov 17, 2016

den-run-ai commented Nov 17, 2016

filmor commented Nov 23, 2016

den-run-ai commented Nov 24, 2016

vmuriart commented Jan 12, 2017

conda.recipe #281

conda.recipe #281

Conversation

jreback commented Nov 3, 2016 • edited Loading

den-run-ai commented Nov 3, 2016

jreback commented Nov 3, 2016

den-run-ai commented Nov 3, 2016

jreback commented Nov 3, 2016 • edited Loading

den-run-ai commented Nov 3, 2016

jreback commented Nov 3, 2016

jreback commented Nov 3, 2016

den-run-ai commented Nov 3, 2016

den-run-ai commented Nov 6, 2016

jreback commented Nov 6, 2016

den-run-ai commented Nov 6, 2016

den-run-ai commented Nov 6, 2016 • edited Loading

den-run-ai commented Nov 9, 2016 • edited Loading

jreback commented Nov 9, 2016

jreback commented Nov 9, 2016

filmor commented Nov 9, 2016

jreback commented Nov 15, 2016

den-run-ai commented Nov 16, 2016

den-run-ai commented Nov 16, 2016

den-run-ai commented Nov 16, 2016

jreback commented Nov 16, 2016

jreback commented Nov 16, 2016

den-run-ai commented Nov 16, 2016

vmuriart commented Nov 17, 2016

vmuriart commented Nov 17, 2016

den-run-ai commented Nov 17, 2016

filmor commented Nov 23, 2016

den-run-ai commented Nov 24, 2016

vmuriart commented Jan 12, 2017

jreback commented Nov 3, 2016 •

edited

Loading

jreback commented Nov 3, 2016 •

edited

Loading

den-run-ai commented Nov 6, 2016 •

edited

Loading

den-run-ai commented Nov 9, 2016 •

edited

Loading