-
-
Notifications
You must be signed in to change notification settings - Fork 10.9k
DOC: Windows and F2PY #20311
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DOC: Windows and F2PY #20311
Changes from all commits
abb199b
03aafd8
a4c51cd
c52faf5
50cafae
7c88bd2
366c64b
d40ff6f
f792764
17c19a7
ee113d4
dfc4ea4
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,34 @@ | ||
.. _f2py-win-conda: | ||
|
||
========================= | ||
F2PY and Conda on Windows | ||
========================= | ||
|
||
As a convienience measure, we will additionally assume the | ||
existence of ``scoop``, which can be used to install tools without | ||
administrative access. | ||
|
||
.. code-block:: powershell | ||
|
||
Invoke-Expression (New-Object System.Net.WebClient).DownloadString('https://get.scoop.sh') | ||
|
||
Now we will setup a ``conda`` environment. | ||
|
||
.. code-block:: powershell | ||
|
||
scoop install miniconda3 | ||
# For conda activate / deactivate in powershell | ||
conda install -n root -c pscondaenvs pscondaenvs | ||
Powershell -c Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser | ||
conda init powershell | ||
# Open a new shell for the rest | ||
|
||
``conda`` pulls packages from ``msys2``, however, the UX is sufficiently different enough to warrant a separate discussion. | ||
|
||
.. warning:: | ||
|
||
As of 30-01-2022, the `MSYS2 binaries`_ shipped with ``conda`` are **outdated** and this approach is **not preferred**. | ||
|
||
|
||
|
||
.. _MSYS2 binaries: https://github.com/conda-forge/conda-forge.github.io/issues/1044 |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,208 @@ | ||
.. _f2py-windows: | ||
|
||
================= | ||
F2PY and Windows | ||
================= | ||
|
||
.. warning:: | ||
|
||
F2PY support for Windows is not at par with Linux support, and | ||
OS specific flags can be seen via ``python -m numpy.f2py`` | ||
|
||
Broadly speaking, there are two issues working with F2PY on Windows: | ||
|
||
- the lack of actively developed FOSS Fortran compilers, and, | ||
- the linking issues related to the C runtime library for building Python-C extensions. | ||
|
||
The focus of this section is to establish a guideline for developing and | ||
extending Fortran modules for Python natively, via F2PY on Windows. | ||
|
||
Overview | ||
======== | ||
From a user perspective, the most UNIX compatible Windows | ||
development environment is through emulation, either via the Windows Subsystem | ||
on Linux, or facilitated by Docker. In a similar vein, traditional | ||
virtualization methods like VirtualBox are also reasonable methods to develop | ||
UNIX tools on Windows. | ||
|
||
Native Windows support is typically stunted beyond the usage of commercial compilers. | ||
However, as of 2022, most commercial compilers have free plans which are sufficient for | ||
general use. Additionally, the Fortran language features supported by ``f2py`` | ||
(partial coverage of Fortran 2003), means that newer toolchains are often not | ||
required. Briefly, then, for an end user, in order of use: | ||
|
||
Classic Intel Compilers (commercial) | ||
These are maintained actively, though licensing restrictions may apply as | ||
further detailed in :ref:`f2py-win-intel`. | ||
|
||
Suitable for general use for those building native Windows programs by | ||
building off of MSVC. | ||
|
||
MSYS2 (FOSS) | ||
In conjunction with the ``mingw-w64`` project, ``gfortran`` and ``gcc`` | ||
toolchains can be used to natively build Windows programs. | ||
|
||
Windows Subsystem for Linux | ||
Assuming the usage of ``gfortran``, this can be used for cross-compiling | ||
Windows applications, but is significantly more complicated. | ||
|
||
Conda | ||
Windows support for compilers in ``conda`` is facilitated by pulling MSYS2 | ||
binaries, however these `are outdated`_, and therefore not recommended (as of 30-01-2022). | ||
|
||
PGI Compilers (commercial) | ||
Unmaintained but sufficient if an existing license is present. Works | ||
natively, but has been superseded by the Nvidia HPC SDK, with no `native | ||
Windows support`_. | ||
|
||
Cygwin (FOSS) | ||
Can also be used for ``gfortran``. Howeve, the POSIX API compatibility layer provided by | ||
Cygwin is meant to compile UNIX software on Windows, instead of building | ||
native Windows programs. This means cross compilation is required. | ||
|
||
The compilation suites described so far are compatible with the `now | ||
deprecated`_ ``np.distutils`` build backend which is exposed by the F2PY CLI. | ||
Additional build system usage (``meson``, ``cmake``) as described in | ||
:ref:`f2py-bldsys` allows for a more flexible set of compiler | ||
backends including: | ||
|
||
Intel oneAPI | ||
The newer Intel compilers (``ifx``, ``icx``) are based on LLVM and can be | ||
used for native compilation. Licensing requirements can be onerous. | ||
|
||
Classic Flang (FOSS) | ||
The backbone of the PGI compilers were cannibalized to form the "classic" or | ||
`legacy version of Flang`_. This may be compiled from source and used | ||
natively. `LLVM Flang`_ does not support Windows yet (30-01-2022). | ||
|
||
LFortran (FOSS) | ||
One of two LLVM based compilers. Not all of F2PY supported Fortran can be | ||
compiled yet (30-01-2022) but uses MSVC for native linking. | ||
|
||
|
||
Baseline | ||
======== | ||
|
||
For this document we will asume the following basic tools: | ||
|
||
- The IDE being considered is the community supported `Microsoft Visual Studio Code`_ | ||
- The terminal being used is the `Windows Terminal`_ | ||
- The shell environment is assumed to be `Powershell 7.x`_ | ||
- Python 3.10 from `the Microsoft Store`_ and this can be tested with | ||
``Get-Command python.exe`` resolving to | ||
``C:\Users\$USERNAME\AppData\Local\Microsoft\WindowsApps\python.exe`` | ||
- The Microsoft Visual C++ (MSVC) toolset | ||
|
||
With this baseline configuration, we will further consider a configuration | ||
matrix as follows: | ||
|
||
.. _table-f2py-winsup-mat: | ||
|
||
.. table:: Support matrix, exe implies a Windows installer | ||
|
||
+----------------------+--------------------+-------------------+ | ||
| **Fortran Compiler** | **C/C++ Compiler** | **Source** | | ||
+======================+====================+===================+ | ||
| Intel Fortran | MSVC / ICC | exe | | ||
+----------------------+--------------------+-------------------+ | ||
| GFortran | MSVC | MSYS2/exe | | ||
+----------------------+--------------------+-------------------+ | ||
| GFortran | GCC | WSL | | ||
+----------------------+--------------------+-------------------+ | ||
| Classic Flang | MSVC | Source / Conda | | ||
+----------------------+--------------------+-------------------+ | ||
| Anaconda GFortran | Anaconda GCC | exe | | ||
+----------------------+--------------------+-------------------+ | ||
|
||
For an understanding of the key issues motivating the need for such a matrix | ||
`Pauli Virtanen's in-depth post on wheels with Fortran for Windows`_ is an | ||
excellent resource. An entertaining explanation of an application binary | ||
interface (ABI) can be found in this post by `JeanHeyd Meneide`_. | ||
|
||
Powershell and MSVC | ||
==================== | ||
|
||
MSVC is installed either via the Visual Studio Bundle or the lighter (preferred) | ||
`Build Tools for Visual Studio`_ with the ``Desktop development with C++`` | ||
setting. | ||
HaoZeke marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
.. note:: | ||
|
||
This can take a significant amount of time as it includes a download of around | ||
2GB and requires a restart. | ||
|
||
It is possible to use the resulting environment from a `standard command | ||
prompt`_. However, it is more pleasant to use a `developer powershell`_, | ||
with a `profile in Windows Terminal`_. This can be achieved by adding the | ||
following block to the ``profiles->list`` section of the JSON file used to | ||
configure Windows Terminal (see ``Settings->Open JSON file``): | ||
Comment on lines
+135
to
+138
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not sure if this is because I'm using VS 2022, but I get slightly different outcomes. First, the developer powershell seems to be readily available from the start menu, so I didn't really need to open this JSON file. Second, here's the output from the example code:
Note that I don't see anything printed to the terminal when executing the program. |
||
|
||
.. code-block:: json | ||
|
||
{ | ||
"name": "Developer PowerShell for VS 2019", | ||
"commandline": "powershell.exe -noe -c \"$vsPath = (Join-Path ${env:ProgramFiles(x86)} -ChildPath 'Microsoft Visual Studio\\2019\\BuildTools'); Import-Module (Join-Path $vsPath 'Common7\\Tools\\Microsoft.VisualStudio.DevShell.dll'); Enter-VsDevShell -VsInstallPath $vsPath -SkipAutomaticLocation\"", | ||
"icon": "ms-appx:///ProfileIcons/{61c54bbd-c2c6-5271-96e7-009a87ff44bf}.png" | ||
} | ||
|
||
Now, testing the compiler toolchain could look like: | ||
|
||
.. code-block:: powershell | ||
|
||
# New Windows Developer Powershell instance / tab | ||
# or | ||
$vsPath = (Join-Path ${env:ProgramFiles(x86)} -ChildPath 'Microsoft Visual Studio\\2019\\BuildTools'); | ||
Import-Module (Join-Path $vsPath 'Common7\\Tools\\Microsoft.VisualStudio.DevShell.dll'); | ||
Enter-VsDevShell -VsInstallPath $vsPath -SkipAutomaticLocation | ||
********************************************************************** | ||
** Visual Studio 2019 Developer PowerShell v16.11.9 | ||
** Copyright (c) 2021 Microsoft Corporation | ||
********************************************************************** | ||
cd $HOME | ||
echo "#include<stdio.h>" > blah.cpp; echo 'int main(){printf("Hi");return 1;}' >> blah.cpp | ||
cl blah.cpp | ||
.\blah.exe | ||
# Hi | ||
rm blah.cpp | ||
|
||
It is also possible to check that the environment has been updated correctly | ||
with ``$ENV:PATH``. | ||
|
||
|
||
Windows Store Python Paths | ||
========================== | ||
|
||
The MS Windows version of Python discussed here installs to a non-deterministic | ||
path using a hash. This needs to be added to the ``PATH`` variable. | ||
|
||
.. code-block:: powershell | ||
|
||
$Env:Path += ";$env:LOCALAPPDATA\packages\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\localcache\local-packages\python310\scripts" | ||
|
||
.. toctree:: | ||
:maxdepth: 2 | ||
|
||
intel | ||
msys2 | ||
conda | ||
pgi | ||
|
||
|
||
.. _the Microsoft Store: https://www.microsoft.com/en-us/p/python-310/9pjpw5ldxlz5 | ||
.. _Microsoft Visual Studio Code: https://code.visualstudio.com/Download | ||
.. _more complete POSIX environment: https://www.cygwin.com/ | ||
.. _This MSYS2 document: https://www.msys2.org/wiki/How-does-MSYS2-differ-from-Cygwin/ | ||
.. _Build Tools for Visual Studio: https://visualstudio.microsoft.com/downloads/#build-tools-for-visual-studio-2019 | ||
.. _Windows Terminal: https://www.microsoft.com/en-us/p/windows-terminal/9n0dx20hk701?activetab=pivot:overviewtab | ||
.. _Powershell 7.x: https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-windows?view=powershell-7.1 | ||
.. _standard command prompt: https://docs.microsoft.com/en-us/cpp/build/building-on-the-command-line?view=msvc-160#developer_command_file_locations | ||
.. _developer powershell: https://docs.microsoft.com/en-us/visualstudio/ide/reference/command-prompt-powershell?view=vs-2019 | ||
.. _profile in Windows Terminal: https://techcommunity.microsoft.com/t5/microsoft-365-pnp-blog/add-developer-powershell-and-developer-command-prompt-for-visual/ba-p/2243078 | ||
.. _Pauli Virtanen's in-depth post on wheels with Fortran for Windows: https://pav.iki.fi/blog/2017-10-08/pywingfortran.html#building-python-wheels-with-fortran-for-windows | ||
.. _Nvidia HPC SDK: https://www.pgroup.com/index.html | ||
.. _JeanHeyd Meneide: https://thephd.dev/binary-banshees-digital-demons-abi-c-c++-help-me-god-please | ||
.. _legacy version of Flang: https://github.com/flang-compiler/flang | ||
.. _native Windows support: https://developer.nvidia.com/nvidia-hpc-sdk-downloads#collapseFour | ||
.. _are outdated: https://github.com/conda-forge/conda-forge.github.io/issues/1044 | ||
.. _now deprecated: https://github.com/numpy/numpy/pull/20875 | ||
.. _LLVM Flang: https://releases.llvm.org/11.0.0/tools/flang/docs/ReleaseNotes.html |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,57 @@ | ||
.. _f2py-win-intel: | ||
|
||
============================== | ||
F2PY and Windows Intel Fortran | ||
============================== | ||
|
||
As of NumPy 1.23, only the classic Intel compilers (``ifort``) are supported. | ||
|
||
.. note:: | ||
|
||
The licensing restrictions for beta software `have been relaxed`_ during | ||
the transition to the LLVM backed ``ifx/icc`` family of compilers. | ||
However this document does not endorse the usage of Intel in downstream | ||
projects due to the issues pertaining to `disassembly of components and | ||
liability`_. | ||
|
||
Neither the Python Intel installation nor the `Classic Intel C/C++ | ||
Compiler` are required. | ||
|
||
- The `Intel Fortran Compilers`_ come in a combined installer providing both | ||
Classic and Beta versions; these also take around a gigabyte and a half or so. | ||
|
||
We will consider the classic example of the generation of Fibonnaci numbers, | ||
``fib1.f``, given by: | ||
|
||
.. literalinclude:: ../code/fib1.f | ||
:language: fortran | ||
|
||
For ``cmd.exe`` fans, using the Intel oneAPI command prompt is the easiest approach, as | ||
it loads the required environment for both ``ifort`` and ``msvc``. Helper batch | ||
scripts are also provided. | ||
|
||
.. code-block:: bat | ||
|
||
# cmd.exe | ||
"C:\Program Files (x86)\Intel\oneAPI\setvars.bat" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can you clarify what this line means? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is similar to |
||
python -m numpy.f2py -c fib1.f -m fib1 | ||
python -c "import fib1; import numpy as np; a=np.zeros(8); fib1.fib(a); print(a)" | ||
|
||
Powershell usage is a little less pleasant, and this configuration now works with MSVC as: | ||
|
||
.. code-block:: powershell | ||
|
||
# Powershell | ||
python -m numpy.f2py -c fib1.f -m fib1 --f77exec='C:\Program Files (x86)\Intel\oneAPI\compiler\latest\windows\bin\intel64\ifort.exe' --f90exec='C:\Program Files (x86)\Intel\oneAPI\compiler\latest\windows\bin\intel64\ifort.exe' -L'C:\Program Files (x86)\Intel\oneAPI\compiler\latest\windows\compiler\lib\ia32' | ||
HaoZeke marked this conversation as resolved.
Show resolved
Hide resolved
|
||
python -c "import fib1; import numpy as np; a=np.zeros(8); fib1.fib(a); print(a)" | ||
# Alternatively, set environment and reload Powershell in one line | ||
cmd.exe /k '"C:\Program Files (x86)\Intel\oneAPI\setvars.bat" && powershell' | ||
python -m numpy.f2py -c fib1.f -m fib1 | ||
python -c "import fib1; import numpy as np; a=np.zeros(8); fib1.fib(a); print(a)" | ||
|
||
Note that the actual path to your local installation of `ifort` may vary, and the command above will need to be updated accordingly. | ||
|
||
.. _have been relaxed: https://www.intel.com/content/www/us/en/developer/articles/release-notes/oneapi-fortran-compiler-release-notes.html | ||
.. _disassembly of components and liability: https://software.sintel.com/content/www/us/en/develop/articles/end-user-license-agreement.html | ||
.. _Intel Fortran Compilers: https://www.intel.com/content/www/us/en/developer/articles/tool/oneapi-standalone-components.html#inpage-nav-6-1 | ||
.. _Classic Intel C/C++ Compiler: https://www.intel.com/content/www/us/en/developer/articles/tool/oneapi-standalone-components.html#inpage-nav-6-undefined |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
.. _f2py-win-msys2: | ||
|
||
=========================== | ||
F2PY and Windows with MSYS2 | ||
=========================== | ||
|
||
Follow the standard `installation instructions`_. Then, to grab the requisite Fortran compiler with ``MVSC``: | ||
|
||
.. code-block:: bash | ||
|
||
# Assuming a fresh install | ||
pacman -Syu # Restart the terminal | ||
pacman -Su # Update packages | ||
# Get the toolchains | ||
pacman -S --needed base-devel gcc-fortran | ||
pacman -S mingw-w64-x86_64-toolchain | ||
|
||
|
||
.. _`installation instructions`: https://www.msys2.org/ |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
.. _f2py-win-pgi: | ||
|
||
=============================== | ||
F2PY and PGI Fortran on Windows | ||
=============================== | ||
|
||
A variant of these are part of the so called "classic" Flang, however, | ||
as classic Flang requires a custom LLVM and compilation from sources. | ||
|
||
.. warning:: | ||
|
||
Since the proprietary compilers are no longer available for | ||
usage they are not recommended and will not be ported to the | ||
new ``f2py`` CLI. | ||
|
||
|
||
|
||
.. note:: | ||
|
||
**As of November 2021** | ||
|
||
As of 29-01-2022, `PGI compiler toolchains`_ have been superceeded by the Nvidia | ||
HPC SDK, with no `native Windows support`_. | ||
|
||
However, | ||
|
||
.. _PGI compiler toolchains: https://www.pgroup.com/index.html | ||
.. _native Windows support: https://developer.nvidia.com/nvidia-hpc-sdk-downloads#collapseFour |
Uh oh!
There was an error while loading. Please reload this page.