Skip to content

fix FreeType build on Azure #13077

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Aug 27, 2019
Merged

Conversation

anntzer
Copy link
Contributor

@anntzer anntzer commented Jan 2, 2019

PR Summary

followup to #12982.

Not sure if the Directory.Build.Props setting is problematic on older Windows versions.

PR Checklist

  • Has Pytest style unit tests
  • Code is Flake 8 compliant
  • New features are documented, with examples if plot related
  • Documentation is sphinx and numpydoc compliant
  • Added an entry to doc/users/next_whats_new/ if major new feature (follow instructions in README.rst there)
  • Documented in doc/api/api_changes.rst if API changed in a backward-incompatible way

@anntzer
Copy link
Contributor Author

anntzer commented Jan 2, 2019

attn @tacaswell (@zooba too perhaps)
Looks like FreeType 2.6.1 is building fine.
The main remaining question is whether the Directory.Build.Props setting is problematic on older Windows versions (and more generally whether we should keep the old build code, or if the new one or some version thereof will work on older Windows); I can't really test this easily. (But note that AppVeyor seems to build FreeType fine too.)

(The Azure build is not failing over not finding libpng, probably just needs to be installed.)

Feel free to push to this PR, or adapt the patch back to #12982.

@tacaswell tacaswell added this to the v3.1 milestone Jan 2, 2019
@anntzer anntzer force-pushed the azure-windows branch 2 times, most recently from 25c6733 to 9fff6f6 Compare January 2, 2019 19:38
@anntzer
Copy link
Contributor Author

anntzer commented Jan 2, 2019

Cherry-picked #13084 into this to help with libpng.

@anntzer anntzer force-pushed the azure-windows branch 4 times, most recently from be09e55 to adbc758 Compare January 2, 2019 20:41
@anntzer
Copy link
Contributor Author

anntzer commented Jan 2, 2019

Looks like things are working on Azure now.

I had to explicitly not link zlib in the png build as the libpng package provided by nuget does not include zlib.lib, and anyways it looks like that's not a problem. But I think that means we're effectively using a dynamic link build so we'd need to ship libpng16.dll and zlib.dll as well (again not a problem, there's code for that in setupext).

Would have been nice to check that appveyor still works after that change but it's running into conda/conda#8051 (I think).

The change to test_backend_nbagg is basically due to https://bugs.python.org/issue33617.

@tacaswell
Copy link
Member

@tacaswell
Copy link
Member

or the issues with trying to use the name of NamedTemporaryFile on windows a-la https://stackoverflow.com/questions/23212435/permission-denied-to-write-to-my-temporary-file

@tacaswell
Copy link
Member

and we are not seeing this failure on appveyor because we are not using the requirements files and likely do not have nbconvert installed so the test is skipped.

@tacaswell
Copy link
Member

🎉 everything is all green!

I am a bit concerned about down-stream packaging issues from conda/conda-forge/Christoph, but I am optomistic

@tacaswell
Copy link
Member

For conda, this will let them drop a at least part of a patch (https://github.com/conda-forge/matplotlib-feedstock/blob/master/recipe/setupext.patch) so I am no longer worried about that.

@cgohlke will this change cause you problems?

@cgohlke
Copy link
Contributor

cgohlke commented Jan 3, 2019

AFAICT this will build on my system but the wheels will be unusable because DLLs are not packaged. It's probably better to link to static libraries by default.

@anntzer
Copy link
Contributor Author

anntzer commented Jan 3, 2019

But you can just put the dlls next to the pyds I think? There's even code for that in setupext to include them in the wheel (it's the dlls=True option in setup.cfg.template).

@cgohlke
Copy link
Contributor

cgohlke commented Jan 3, 2019

But you can just put the dlls next to the pyds I think? There's even code for that in setupext to include them in the wheel (it's the dlls=True option in setup.cfg.template).

Yes, this is used to ship msvcp runtime DLLs in the official mpl wheels.

Still, it looks to me that by default this PR will produce broken wheels and IMO dynamic linking in wheels for PyPI should be avoided if feasible. How about using setup.cfg entries or environment variables to make static linking possible without renaming/copying library files or patching mpl source code?

@anntzer
Copy link
Contributor Author

anntzer commented Jan 3, 2019

That's basically #9693; can probably be adapted for here.
However, I wonder whether there is really a downside to dynamic linking on Windows (i.e. shipping dlls next to the pyds)? (I really don't know, but would prefer having as few build variants as possible.)
For this PR we could additionally call ctypes.util.find_library("libpng16") and copy the result to matplotlib/ automatically (and likewise for zlib).

@jklymak
Copy link
Member

jklymak commented Feb 22, 2019

Is this still active?

@anntzer
Copy link
Contributor Author

anntzer commented Feb 22, 2019

Yes, but this is basically waiting for @tacaswell for policy choices I guess (on the technical side I think it's done).

@tacaswell
Copy link
Member

What policy choice?

I don't think this should be merged if @cgohlke is concerned about making broken wheels.

@tacaswell
Copy link
Member

rebased.

If I understand the issue that @cgohlke is raising correctly it is if we use this as a build system to produce the wheels that we put on pypi we should not use dynamic linking to avoid the problemswhere different wheels expect different versions of the dependencies and then things go badly.

However, this applies to the c-extensions that Matplotlib owns (not freetype which looks like it is producing *.libs which is correct?) and how we build the wheels correct?

If my understanding is correct I am inclined to merge this so we can start using azure for all 3 platforms and then fix the wheel building in a later PR.

@cgohlke Is the process you use to build the wheels for pypi documented someplace?

@tacaswell
Copy link
Member

I particularly want to get this in as the python 3.8 release is coming up and this gets us access to Steve's previews on windows.

@tacaswell
Copy link
Member

ok, tried to cherry-pick @Kojoley 's work from #9693 . I don't currently have a windows machine I can test this on so hopefully CI does it's thing...

@tacaswell
Copy link
Member

ok, so now it works on appveyor, but there seems to not be a static version of zlib on azure?

Copy link
Contributor

@dopplershift dopplershift left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general looks good to me.

zooba and others added 4 commits August 26, 2019 19:59
Adds macOS and Windows builds to Azure Pipelines

Correct VM image name

Fix job names and remove extension reference

Add dependency installs for macOS and Ubuntu

Attempt to use libpng from nuget for Windows build

Ensures nuget is available on PATH
This should make sure we have consistent dependencies installed
across all of the CI services.
@tacaswell
Copy link
Member

force-pushed hopefully for the last time.

@cgohlke If you have a chance, could you confirm that we are actually using the static libraries? I think it should now default to using the static libpng and zlib versions on windows.

@cgohlke
Copy link
Contributor

cgohlke commented Aug 27, 2019

I think it should now default to using the static libpng and zlib versions on windows. I think it should now default to using the static libpng and zlib versions on windows.

I built the anntzer:azure-windows branch on my system. It linked to the static libpng and zlib libraries.

@tacaswell
Copy link
Member

On the positive report from Christoph I'm going to go ahead and merge this !

@tacaswell tacaswell merged commit bea4e1d into matplotlib:master Aug 27, 2019
@tacaswell
Copy link
Member

Thanks @zooba @Kojoley and @anntzer ! It took a while, but we got this landed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants