Skip to content

[feature] Port python-ldap to PEP 630 (Isolating Extension Modules) #540

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
5 tasks
tiran opened this issue Oct 30, 2023 · 0 comments
Closed
5 tasks

[feature] Port python-ldap to PEP 630 (Isolating Extension Modules) #540

tiran opened this issue Oct 30, 2023 · 0 comments

Comments

@tiran
Copy link
Member

tiran commented Oct 30, 2023

I propose to port python-ldap to PEP 630. Actually, I have already done the work and have a working prototype. PEP 630 is an informational PEP that covers three PEPs

  • PEP 384 – Defining a Stable ABI
  • PEP 489 – Multi-phase extension module initialization
  • PEP 573 – Module State Access from C Extension Methods

Isolated C extension module will give us two major benefits:

  • Stable ABI / limited API will allow us / our users to create abi3 binary wheels. My prototype implements cp36-abi3-linux_x86_64.whl support. The wheels is compatible with all Python versions from 3.6 to 3.12, as well as future versions of Python 3 with ABI3. For example @cgohlke won't need to create Windows binaries for each Python version.
  • It makes python-ldap compatible with PEP 684 Per-Interpreter GIL and safe to use with multiple subinterpreters.

Required changes

  • Port PyTypeObject LDAP_Type to heap type.
    • Replace static PyTypeObject LDAP_Type with PyType_Slot and PyType_Spec
    • Allocate object with PyObject_GC_New and free it with PyObject_GC_UnTrack / PyObject_GC_Del
    • Add GC protocol (traverse, clear) to track the heap reference to the type
    • Initialize type with PyType_FromSpec
  • Create _ldap C extension module with multi-phase initialization
    • Export module-level functions from functions.c and ldapcontrol.c. This also gets rid of LDAPadd_methods hack, LDAPinit_functions, and LDAPinit_control
    • Define PyModuleDef_Slot to init type, constants, and pkginfo
    • Make PyModuleDef global static
    • Initialize module with PyModuleDef_Init
  • Remove static globals LDAP_Type, LDAPexception_class, and errobjects in favor of module state struct.
    • Define LDAPModState and add it to PyModuleDef
    • Add state pointer to LDAPObject. PyType_GetModuleState() is not available in Python < 3.9 and became part of the stable ABI in 3.10.
    • Change all functions to get reference to type or exceptions from module state. This change is the largest diff and requires changes to several helper functions, especially error functions like LDAPerror.
  • Replace some macros with limited API functions, e.g. PySequence_Fast_GET_ITEM() with PySequence_GetItem(). We use unsafe macros in a few places. In theory the macros are a tiny bit faster. In practice it doesn't make a difference except for lots of calls in a hot path. I don't except a performance impact.
  • Build _ldap extension module with limited API and stable ABI parameters.
tiran added a commit to tiran/python-ldap that referenced this issue Oct 30, 2023
@tiran
refactor: Merge all header files
b7c15cb 
Merge all header files except `constants_generated.h` into a single
header file `pythonldap.h`. A single header file makes it far easier to
port python-ldap to heap types and module state for Per-Interpreter GIL.

`pythonldap.h` uses new macros `PYLDAP_FUNC` and `PYLDAP_DATA` to declare
functions and data, which are used across C files.

Remove unused macro `streq`.

See: python-ldap#540
Signed-off-by: Christian Heimes <cheimes@redhat.com>
tiran added a commit to tiran/python-ldap that referenced this issue Oct 30, 2023
@tiran
refactor: Merge all header files
166dc50 
Merge all header files except `constants_generated.h` into a single
header file `pythonldap.h`. A single header file makes it far easier to
port python-ldap to heap types and module state for Per-Interpreter GIL.

`pythonldap.h` uses new macros `PYLDAP_FUNC` and `PYLDAP_DATA` to declare
functions and data, which are used across C files.

Remove unused macro `streq`.

See: python-ldap#540
Signed-off-by: Christian Heimes <cheimes@redhat.com>
tiran added a commit to tiran/python-ldap that referenced this issue Oct 30, 2023
@tiran
refactor: Merge all header files
b9901b9 
Merge all header files except `constants_generated.h` into a single
header file `pythonldap.h`. A single header file makes it far easier to
port python-ldap to heap types and module state for Per-Interpreter GIL.

`pythonldap.h` uses new macros `PYLDAP_FUNC` and `PYLDAP_DATA` to declare
functions and data, which are used across C files.

Remove unused macro `streq`.

See: python-ldap#540
Signed-off-by: Christian Heimes <cheimes@redhat.com>
@tiran tiran mentioned this issue Oct 30, 2023
Merged
tiran added a commit to tiran/python-ldap that referenced this issue Oct 30, 2023
@tiran
refactor: Use functions from limited API
bad8aef 
Replace unsafe macros and direct struct access with functions from
the subset of limited API functions.

* `PySequence_Fast_GET_ITEM` -> `PySequence_GetItem`
* `PyTuple_SET_ITEM` -> `PyTuple_SetItem`
* `PyList_SET_ITEM` -> `PyList_SetItem`
* `PyBytes_AsStringAndSize` -> `PyUnicode_AsUTF8String` +
  `PyBytes_AsStringAndSize`. The function `PyUnicode_AsUTF8AndSize` is
  not in limited API before Python 3.10.
* `const char *tp_name` -> `Py_TYPE()` string representation

See: python-ldap#540
Signed-off-by: Christian Heimes <cheimes@redhat.com>
@tiran tiran mentioned this issue Oct 30, 2023
Closed
tiran added a commit to tiran/python-ldap that referenced this issue Nov 1, 2023
@tiran
refactor: Use functions from limited API
b45e0c3 
Replace unsafe macros and direct struct access with functions from
the subset of limited API functions.

* `PySequence_Fast_GET_ITEM` -> `PySequence_GetItem`
* `PyTuple_SET_ITEM` -> `PyTuple_SetItem`
* `PyList_SET_ITEM` -> `PyList_SetItem`
* `PyUnicode_AsUTF8AndSize` -> `PyUnicode_AsUTF8String` +
  `PyBytes_AsStringAndSize`. The function `PyUnicode_AsUTF8AndSize` is
  not in limited API before Python 3.10.
* `const char *tp_name` -> `Py_TYPE()` string representation

See: python-ldap#540
Signed-off-by: Christian Heimes <cheimes@redhat.com>
tiran added a commit that referenced this issue Nov 3, 2023
@tiran
refactor: Merge all header files
75a765f 
Merge all header files except `constants_generated.h` into a single
header file `pythonldap.h`. A single header file makes it far easier to
port python-ldap to heap types and module state for Per-Interpreter GIL.

`pythonldap.h` uses new macros `PYLDAP_FUNC` and `PYLDAP_DATA` to declare
functions and data, which are used across C files.

Remove unused macro `streq`.

See: #540
Signed-off-by: Christian Heimes <cheimes@redhat.com>
tiran added a commit to tiran/python-ldap that referenced this issue Nov 3, 2023
@tiran
refactor: Port to multi-phase module init
e765d78 
The `_ldap` module now uses modern multi-phase module initialization.

Replace `LDAPadd_methods` hack with proper PyMethodDef for
module-level functions. The old approache is incompatible with
multi-phase init. Module-level functions are now prefixed with
`LDAPMod_` and exported.

Use `PyModuleDef_Slot` to initialize the `_ldap` C extension.

See: python-ldap#540
Signed-off-by: Christian Heimes <cheimes@redhat.com>
@tiran tiran mentioned this issue Nov 3, 2023
Closed
tiran added a commit to tiran/python-ldap that referenced this issue Nov 3, 2023
@tiran
refactor: Port LDAP_Type to heap type
c54a8e9 
The `LDAP` type has been converted from a static type to a heap type.
The limited API does not support static types.

Heap types behave more closely like Python classes. They are allocated
on the heap and reference counted. Instances have a strong reference to
their type and must use GC protocol to track this reference.

The LDAP type can no longer be instantiated by Python code. This was
never supported and resulted in an invalid LDAP connection. Code like
`type(_ldap.initialize(""))()" now fails with a `TypeError`.

See: python-ldap#540
Signed-off-by: Christian Heimes <cheimes@redhat.com>
tiran added a commit to tiran/python-ldap that referenced this issue Nov 3, 2023
@tiran
refactor: Port LDAP_Type to heap type
539cab7 
The `LDAP` type has been converted from a static type to a heap type.
The limited API does not support static types.

Heap types behave more closely like Python classes. They are allocated
on the heap and reference counted. Instances have a strong reference to
their type and must use GC protocol to track this reference.

The LDAP type can no longer be instantiated by Python code. This was
never supported and resulted in an invalid LDAP connection. Code like
`type(_ldap.initialize(""))()" now fails with a `TypeError`.

See: python-ldap#540
Signed-off-by: Christian Heimes <cheimes@redhat.com>
@tiran tiran mentioned this issue Nov 3, 2023
Closed
tiran added a commit to tiran/python-ldap that referenced this issue Nov 3, 2023
@tiran
refactor: Use functions from limited API
c37d196 
Replace unsafe macros and direct struct access with functions from
the subset of limited API functions.

* `PySequence_Fast_GET_ITEM` -> `PySequence_GetItem`
* `PyTuple_SET_ITEM` -> `PyTuple_SetItem`
* `PyList_SET_ITEM` -> `PyList_SetItem`
* `PyUnicode_AsUTF8AndSize` -> `PyUnicode_AsUTF8String` +
  `PyBytes_AsStringAndSize`. The function `PyUnicode_AsUTF8AndSize` is
  not in limited API before Python 3.10.
* `const char *tp_name` -> `Py_TYPE()` string representation

See: python-ldap#540
Signed-off-by: Christian Heimes <cheimes@redhat.com>
tiran added a commit to tiran/python-ldap that referenced this issue Nov 3, 2023
@tiran
refactor: Use functions from limited API
5d49f27 
Replace unsafe macros and direct struct access with functions from
the subset of limited API functions.

* `PySequence_Fast_GET_ITEM` -> `PySequence_GetItem`
* `PyTuple_SET_ITEM` -> `PyTuple_SetItem`
* `PyList_SET_ITEM` -> `PyList_SetItem`
* `PyUnicode_AsUTF8AndSize` -> `PyUnicode_AsUTF8String` +
  `PyBytes_AsStringAndSize`. The function `PyUnicode_AsUTF8AndSize` is
  not in limited API before Python 3.10.
* `const char *tp_name` -> `Py_TYPE()` string representation

See: python-ldap#540
Signed-off-by: Christian Heimes <cheimes@redhat.com>
tiran added a commit to tiran/python-ldap that referenced this issue Nov 6, 2023
@tiran
refactor: Port LDAP_Type to heap type
b69c742 
The `LDAP` type has been converted from a static type to a heap type.
The limited API does not support static types.

Heap types behave more closely like Python classes. They are allocated
on the heap and reference counted. Instances have a strong reference to
their type and must use GC protocol to track this reference.

The LDAP type can no longer be instantiated by Python code. This was
never supported and resulted in an invalid LDAP connection. Code like
`type(_ldap.initialize(""))()" now fails with a `TypeError`.

See: python-ldap#540
Signed-off-by: Christian Heimes <cheimes@redhat.com>
tiran added a commit to tiran/python-ldap that referenced this issue Nov 7, 2023
@tiran
refactor: Port LDAP_Type to heap type
70303d7 
The `LDAP` type has been converted from a static type to a heap type.
The limited API does not support static types.

Heap types behave more closely like Python classes. They are allocated
on the heap and reference counted. Instances have a strong reference to
their type and must use GC protocol to track this reference.

The LDAP type can no longer be instantiated by Python code. This was
never supported and resulted in an invalid LDAP connection. Code like
`type(_ldap.initialize(""))()" now fails with a `TypeError`.

See: python-ldap#540
Signed-off-by: Christian Heimes <cheimes@redhat.com>
@tiran tiran closed this as not planned Won't fix, can't repro, duplicate, stale Feb 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@tiran