Add iOS platform usage documentation.

python · ned-deily · Mar 28, 2024 · Mar 20, 2024 · Mar 20, 2024 · Mar 20, 2024
commit 09e9302a94d62f1968f3be92bc8dc9c23f13fc1f
diff --git a/Doc/library/intro.rst b/Doc/library/intro.rst
@@ -129,15 +129,23 @@ iOS is, in most respects, a POSIX operating system. File I/O, socket handling,
 and threading all behave as they would on any POSIX operating system. However,
 there are several major differences between iOS and other POSIX systems.
 
+* iOS can only use Python in "embedded" mode. There is no Python REPL, and no
+  ability to execute binaries that are part of the normal Python developer
+  experience, such as ``pip``. To add Python code to your iOS app, you must use
+  the :ref:`Python embedding API <embedding>` to add a Python interpreter to an
+  iOS app created with Xcode. See the :ref:`iOS usage guide <using-ios>` for
+  more details.
+
 * An iOS app cannot use any form of subprocessing, background processing, or
   inter-process communication. If an iOS app attempts to create a subprocess,
   the process creating the subprocess will either lock up, or crash. An iOS app
   has no visibility of other applications that are running, nor any ability to
   communicate with other running applications, outside of the iOS-specific APIs
   that exist for this purpose.
 
-* iOS apps have limited access to modify system resources. These resources will
-  often be *readable*, but attempts to modify those resources will either fail.
+* iOS apps have limited access to modify system resources (such as the system
+  clock). These resources will often be *readable*, but attempts to modify
+  those resources will usually fail.
 
 * iOS apps have a limited concept of console input and output. ``stdout`` and
   ``stderr`` *exist*, and content written to ``stdout`` and ``stderr`` will be
@@ -151,4 +159,4 @@ there are several major differences between iOS and other POSIX systems.
   ``stdin``.
 
   As a result, Python library that involve console manipulation (such as
-  ``curses`` and ``readline``) are not available on iOS.
+  :mod:`curses` and :mod:`readline`) are not available on iOS.
@@ -916,6 +916,20 @@ See ``Mac/README.rst``.
    Specify the name for the python framework on macOS only valid when
    :option:`--enable-framework` is set (default: ``Python``).
 
+iOS Options
+-----------
+
+See ``iOS/README.rst``.
+
+.. option:: --enable-framework=INSTALLDIR
+
+   Create a Python.framework. Unlike macOS, the *INSTALLDIR* argument
+   specifying the installation path is mandatory.
+
+.. option:: --with-framework-name=FRAMEWORK
+
+   Specify the name for the framework (default: ``Python``).
+
 
 Cross Compiling Options
 -----------------------

diff --git a/Doc/using/iOS.rst b/Doc/using/iOS.rst
@@ -0,0 +1,314 @@
+.. _using-ios:
+
+===================
+Using Python on iOS
+===================
+
+:Authors:
+    Russell Keith-Magee (2024-03)
+
+Python on iOS is unlike Python on desktop platforms. On a desktop platform,
+Python is generally installed as a system resource that can be used by any user
+of that computer. Users then interact with Python by running a ``python``
+executable and entering commands at an interactive prompt, or by running a
+Python script.
+
+On iOS, there is no concept of installing as a system resource. The only unit
+of software distribution is an "app". There is also no console where you could
+run a ``python`` executable, or interact with a Python REPL.
+
+As a result, the only way you can use Python on iOS is in embedded mode - that
+is, by writing a native iOS application, and embedding a Python interpreter
+using ``libPython``, and invoking Python code using the :ref:`Python embedding
+API <embedding>`. The full Python interpreter, the standard library, and all
+your Python code is then packaged as a standalone bundle that can be
+distributed via the iOS App Store.
+
+If you're looking to experiment for the first time with writing an iOS app in
+Python, projects such as `BeeWare <https://beeware.org>`__ and `Kivy
+<https://kivy.org>`__ will provide a much more approachable user experience.
+These projects manage the complexities associated with getting an iOS project
+running, so you only need to deal with the Python code itself.
+
+Python at runtime on iOS
+========================
+
+Platform identification
+-----------------------
+
+When executing on iOS, ``sys.platform`` will report as ``ios``. This value will
+be returned on an iPhone or iPad, regardless of whether the app is running on
+the simulator or a physical device.
+
+Information about the specific runtime environment, including the iOS version,
+device model, and whether the device is a simulator, can be obtained using
+:func:`platform.ios_ver()`. :func:`platform.system()` will report ``iOS`` or
+``iPadOS``, depending on the device.
+
+:func:`os.uname()` reports kernel-level details; it will report a name of
+``Darwin``.
+
+Standard library availability
+-----------------------------
+
+The Python standard library has some notable ommissions and restrictions on
+iOS. See the :ref:`API availability guide for iOS <iOS-availability>` for
+details.
+
+Binary extension modules
+------------------------
+
+One notable difference about iOS as a platform is that App Store distribution
+imposes hard requirements on the packaging of an application. One of these
+requirements governs how binary extension modules are distributed.
+
+The iOS App Store requires that *all* binary modules in an iOS app must be
+dynamic libraries, contained in a framework with appropriate metadata, stored
+in the ``Frameworks`` folder of the packaged app. There can be only a single
+binary per framework, and there can be no executable binary material outside
+the Frameworks folder.
+
+This conflicts with the usual Python approach for distributing binaries, which
+allows a binary extension module to be loaded from any location on
+``sys.path``. To ensure compliance with App Store policies, an iOS project must
+post-process any Python packages, converting ``.so`` binary modules into
+individual standalone frameworks with appropriate metadata and signing. For
+details on how to perform this post-processing, see the guide for :ref:`adding
+iOS to your project <adding-ios>`.
+
+To help Python discover binaries in their new location, the original ``.so``
+file on ``sys.path`` is replaced with a ``.fwork`` file. This file is a text
+file containing the location of the framework binary, relative to the app
+bundle. To allow the framework to resolve back to the original location, the
+framework must contain a ``.origin`` file that contains the location of the
+``.fwork`` file, relative to the app bundle.
+
+For example, consider the case of an import ``from foo.bar import _whiz``,
+where ``_whiz`` is implemented with the binary module
+``sources/foo/bar/_whiz.abi3.so``, with ``sources`` being the location
+registered on ``sys.path``, relative to the application bundle. This module
+*must* be distributed as ``Frameworks/foo.bar._whiz.framework/foo.bar._whiz``
+(creating the framework name from the full import path of the module), with an
+``Info.plist`` file in the ``.framework`` directory identifying the binary as a
+framework. The ``foo.bar._whiz`` module would be represented in the original
+location with a ``sources/foo/bar/_whiz.abi3.fwork`` marker file, containing
+the path ``Frameworks/foo.bar._whiz/foo.bar._whiz``. The framework would also
+contain ``Frameworks/foo.bar._whiz.framework/foo.bar._whiz.origin``, containing
+the path to the ``.fwork`` file.
+
+When running on iOS, the Python interpreter will install a :class:`loader
+<importlib.AppleFrameworkLoader>` that is able to read and import ``.fwork``
+files. Once imported, the ``__file__`` attribute of the binary module will
+report as the location of the ``.fwork`` file. However, the
+:class:`~importlib.machinery.ModuleSpec` for the loaded module will report the
+``origin`` as the location of the binary in the framework folder.
+
+Compiler stub binaries
+----------------------
+
+Xcode doesn't expose explicit compilers for iOS; instead, it uses an ``xcrun``
+script that resolves to a full compiler path (e.g., ``xcrun --sdk iphoneos
+clang`` to get the ``clang`` for an iPhone device). However, using this script
+poses 2 problems:
+
+* The output of ``xcrun`` includes paths that are machine specific, resulting
+  in a sysconfig module that cannot be shared between users; and
+
+* It results in ``CC``/``CPP``/``LD``/``AR`` definitions that include spaces.
+  There is a lot of C ecosystem tooling that assumes that you can split a
+  command line at the first space to get the path to the compiler executable;
+  this isn't the case when using ``xcrun``.
+
+To avoid these problems, Python provided stubs for these tools. These stubs are
+shell script wrappers around the underingly ``xcrun`` tools, distributed in a
+``bin`` folder distributed alongside the compiled iOS framework. These scripts
+are relocatable, and will always resolve to the appropriate local system paths.
+By including these scripts in the bin folder that accompanies a framework, the
+contents of the ``sysconfig`` module becomes useful for end-users to compile
+their own modules. When compiling third-party Python modules for iOS, you
+should ensure these stub binaries are on your path.
+
+Installing Python on iOS
+========================
+
+Tools for building iOS apps
+---------------------------
+
+Building for iOS requires the use of Apple's Xcode tooling. It is strongly
+recommended that you use the most recent stable release of Xcode. This will
+require the use of the most (or second-most) recently released macOS version,
+as Apple does not maintain Xcode for older macOS versions. The Xcode Command
+Line Tools are not sufficient for iOS development; you need a *full* Xcode
+install.
+
+If you want to run your code on the iOS simulator, you'll also need to install
+an iOS Simulator Platform. You should be prompted to select an iOS Simulator
+Platform when you first run Xcode. Alternatively, you can add an iOS Simulator
+Platform by selecting an open the Platforms tab of the Xcode Settings panel.
+
+.. _adding-ios:
+
+Adding iOS to an XCode project
+------------------------------
+
+Python can be added to any iOS project, using either Swift or Objective C. The
+following examples will use Objective C; if you are using Swift, you may find a
+library like `PythonKit <https://github.com/pvieito/PythonKit>`__ to be
+helpful.
+
+To add Python to an iOS Xcode project:
+
+1. Build or obtain a Python ``XCFramework``. See the instructions in
+   ``iOS/README.rst`` (in the CPython source distribution) for details on how
+   to build a Python ``XCFramework``. At a minimum, you will need a build that
+   supports `arm64-apple-ios`, plus one of either `arm64-apple-ios-simulator`
+   or `x86_64-apple-ios-simulator`.
+
+2. Drag the ``XCframework`` into your iOS project. In the following
+   instructions, we'll assume you've dropped the ``XCframework`` into the root
+   of your project; however, you can use any other location that you want by
+   adjusting paths as needed.
+
+3. Drag the ``iOS/Resources/dylib-Info-template.plist`` file into your project,
+   and ensure it is associated with the app target.
+
+4. Add your application code as a folder in your Xcode project. In the
+   following instructions, we'll assume that your user code is in a folder
+   named ``app`` in the root of your project; you can use any other location by
+   adjusting paths as needed. Ensure that this folder is associated with your
+   app target.
+
+5. Select the app target by selecting the root node of your Xcode project, then
+   the target name in the sidebar that appears.
+
+6. In the "General" settings, under "Frameworks, Libraries and Embedded
+   Content", add ``Python.xcframework``, with "Embed & Sign" selected.
+
+7. In the "Build Settings" tab, modify the following:
+
+   - Build Options
+
+     * User script sandboxing: No
+     * Enable Testability: Yes
+
+   - Search Paths
+
+     * Framework Search Paths: ``$(PROJECT_DIR)``
+     * Header Search Paths: ``"$(BUILT_PRODUCTS_DIR)/Python.framework/Headers"``
+
+   - Apple Clang - Warnings - All languages
+
+     * Quoted Include in Framework Header: No
+
+8. Add a build step that copies the Python standard library into your app. In
+   the "Build Phases" tab, add a new "Run Script" build step *before* the
+   "Embed Frameworks" step, but *after* the "Copy Bundle Resources" step. Name
+   the step "Install Target Specific Python Standard Library", disable the
+   "Based on dependency analysis" checkbox, and set the script content to:
+
+   .. code-block:: bash
+
+       set -e
+
+       mkdir -p "$CODESIGNING_FOLDER_PATH/python/lib"
+       if [ "$EFFECTIVE_PLATFORM_NAME" = "-iphonesimulator" ]; then
+           echo "Installing Python modules for iOS Simulator"
+           rsync -au --delete "$PROJECT_DIR/Python.xcframework/ios-arm64_x86_64-simulator/lib/" "$CODESIGNING_FOLDER_PATH/python/lib/"
+       else
+           echo "Installing Python modules for iOS Device"
+           rsync -au --delete "$PROJECT_DIR/Python.xcframework/ios-arm64/lib/" "$CODESIGNING_FOLDER_PATH/python/lib/"
+       fi
+
+   Note that the name of the simulator "slice" in the XCframework may be
+   different, depending the CPU architectures your ``XCFramework`` supports.
+
+9. Add a second build step that processes the binary extension modules in the
+   standard library into "Framework" format. Add a "Run Script" build step
+   *directly after* the one you added in step 8, named "Prepare Python Binary
+   Modules". It should also have "Based on dependency analysis" unchecked, with
+   the following script content:
+
+   .. code-block:: bash
+
+       set -e
+
+       install_dylib () {
+           INSTALL_BASE=$1
+           FULL_EXT=$2
+
+           # The name of the extension file
+           EXT=$(basename "$FULL_EXT")
+           # The location of the extension file, relative to the bundle
+           RELATIVE_EXT=${FULL_EXT#$CODESIGNING_FOLDER_PATH/}
+           # The path to the extension file, relative to the install base
+           PYTHON_EXT=${RELATIVE_EXT/$INSTALL_BASE/}
+           # The full dotted name of the extension module, constructed from the file path.
+           FULL_MODULE_NAME=$(echo $PYTHON_EXT | cut -d "." -f 1 | tr "/" ".");
+           # A bundle identifier; not actually used, but required by Xcode framework packaging
+           FRAMEWORK_BUNDLE_ID=$(echo $PRODUCT_BUNDLE_IDENTIFIER.$FULL_MODULE_NAME | tr "_" "-")
+           # The name of the framework folder.
+           FRAMEWORK_FOLDER="Frameworks/$FULL_MODULE_NAME.framework"
+
+           # If the framework folder doesn't exist, create it.
+           if [ ! -d "$CODESIGNING_FOLDER_PATH/$FRAMEWORK_FOLDER" ]; then
+               echo "Creating framework for $RELATIVE_EXT"
+               mkdir -p "$CODESIGNING_FOLDER_PATH/$FRAMEWORK_FOLDER"
+               cp "$CODESIGNING_FOLDER_PATH/dylib-Info-template.plist" "$CODESIGNING_FOLDER_PATH/$FRAMEWORK_FOLDER/Info.plist"
+               plutil -replace CFBundleExecutable -string "$FULL_MODULE_NAME" "$CODESIGNING_FOLDER_PATH/$FRAMEWORK_FOLDER/Info.plist"
+               plutil -replace CFBundleIdentifier -string "$FRAMEWORK_BUNDLE_ID" "$CODESIGNING_FOLDER_PATH/$FRAMEWORK_FOLDER/Info.plist"
+           fi
+
+           echo "Installing binary for $FRAMEWORK_FOLDER/$FULL_MODULE_NAME"
+           mv "$FULL_EXT" "$CODESIGNING_FOLDER_PATH/$FRAMEWORK_FOLDER/$FULL_MODULE_NAME"
+           # Create a placeholder .fwork file where the .so was
+           echo "$FRAMEWORK_FOLDER/$FULL_MODULE_NAME" > ${FULL_EXT%.so}.fwork
+           # Create a back reference to the .so file location in the framework
+           echo "${RELATIVE_EXT%.so}.fwork" > "$CODESIGNING_FOLDER_PATH/$FRAMEWORK_FOLDER/$FULL_MODULE_NAME.origin"
+        }
+
+        PYTHON_VER=$(ls -1 "$CODESIGNING_FOLDER_PATH/python/lib")
+        echo "Install Python $PYTHON_VER standard library extension modules..."
+        find "$CODESIGNING_FOLDER_PATH/python/lib/$PYTHON_VER/lib-dynload" -name "*.so" | while read FULL_EXT; do
+           install_dylib python/lib/$PYTHON_VER/lib-dynload/ "$FULL_EXT"
+        done
+
+        # Clean up dylib template
+        rm -f "$CODESIGNING_FOLDER_PATH/dylib-Info-template.plist"
+
+        echo "Signing frameworks as $EXPANDED_CODE_SIGN_IDENTITY_NAME ($EXPANDED_CODE_SIGN_IDENTITY)..."
+        find "$CODESIGNING_FOLDER_PATH/Frameworks" -name "*.framework" -exec /usr/bin/codesign --force --sign "$EXPANDED_CODE_SIGN_IDENTITY" ${OTHER_CODE_SIGN_FLAGS:-} -o runtime --timestamp=none --preserve-metadata=identifier,entitlements,flags --generate-entitlement-der "{}" \;
+
+10. Add Objective C code to initialize and use a Python interpreter in embedded
+    mode. You should ensure that:
+
+   * :c:member:`UTF-8 mode <PyPreConfig.utf8_mode>` is *enabled*;
+   * :c:member:`Buffered stdio <PyConfig.buffered_stdio>` is *disabled*;
+   * :c:member:`Writing bytecode <PyConfig.write_bytecode>` is *disabled*;
+   * :c:member:`Signal handlers <PyConfig.install_signal_handlers>` are *enabled*;
+   * ``PYTHONHOME`` for the interpreter is configured to point at the
+     ``python`` subfolder of your app's bundle; and
+   * The ``PYTHONPATH`` for the interpreter includes:
+
+     - the ``python/lib/python3.X`` subfolder of your app's bundle,
+     - the ``python/lib/python3.X/lib-dynload`` subfolder of your app's bundle, and
+     - the ``app`` subfolder of your app's bundle
+
+   Your app's bundle location can be determined using ``[[NSBundle mainBundle]
+   resourcePath]``.
+
+Steps 8, 9 and 10 of these instructions assume that you have a single folder of
+pure Python application code, named ``app``. If you have third-party binary
+modules in your app, some additional steps will be required:
+
+* You need to ensure that any folders containing third-party binaries are
+  either associated with the app target, or copied in as part of step 8. Step 8
+  should also purge any binaries that are not appropriate for the platform a
+  specific build is targetting (i.e., delete any device binaries if you're
+  building app app targeting the simulator).
+
+* Any folders that contain third-party binaries must be processed into
+  framework form by step 9. The invocation of ``install_dylib`` that processes
+  the ``lib-dynload`` folder can be copied and adapted for this purpose.
+
+* If you're using a separate folder for third-party packages, ensure that folder
+  is included as part of the ``PYTHONPATH`` configuration in step 10.
diff --git a/Doc/using/index.rst b/Doc/using/index.rst
@@ -18,4 +18,5 @@ interpreter and things that make working with Python easier.
    configure.rst
    windows.rst
    mac.rst
+   iOS.rst
    editors.rst
@@ -182,7 +182,10 @@ This can be done by defining the ``LIBLZMA_CFLAGS``, ``LIBLZMA_LIBS``,
 ``BZIP2_CFLAGS``, ``BZIP2_LIBS``, ``LIBFFI_CFLAGS``, and ``LIBFFI_LIBS``
 environment variables, and the ``--with-openssl`` configure option. Versions of
 these libraries pre-compiled for iOS can be found in `this repository
-<https://github.com/beeware/cpython-apple-source-deps/releases>`__.
+<https://github.com/beeware/cpython-apple-source-deps/releases>`__. LibFFI is
+especially important, as many parts of the standard library (including the
+``platform``, ``sysconfig`` and ``webbrowser`` modules) require the use of the
+``ctypes`` module at runtime.
 
 By default, Python will be compiled with an iOS deployment target (i.e., the
 minimum supported iOS version) of 12.0. To specify a different deployment
@@ -252,12 +255,7 @@ the XCframework::
     cp path/to/iphonesimulator/lib Python.xcframework/ios-arm64_x86-64-simulator
 
 Note that the name of the architecture-specific slice for the simulator will
-depend on the CPU architecture that you build.
-
-Then, add symbolic links to "common" platform names for each slice::
-
-    ln -si ios-arm64 Python.xcframework/iphoneos
-    ln -si ios-arm64_x86-64-simulator Python.xcframework/iphonesimulator
+depend on the CPU architecture(s) that you build.
 
 You now have a Python.xcframework that can be used in a project.