`test_dllist` fails on NetBSD: dl_iterate_phdr doesn't report shared libraries #131565

furkanonder · 2025-03-21T18:53:18Z

Bug report

Bug description:

The test_dllist tests in Lib/test/test_ctypes/test_dllist.py are failing on NetBSD. The issue appears to be that the dl_iterate_phdr function on NetBSD does not report the same information about loaded shared libraries as it does on Linux and other platforms.

Specifically, on NetBSD, dl_iterate_phdr only returns the main Python executable and doesn't include any of the shared libraries that are actually loaded.

./python -m test test_ctypes -m test_dllist -v

Output:

FAIL: test_lists_system (test.test_ctypes.test_dllist.ListSharedLibraries.test_lists_system)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/blue/cpython/Lib/test/test_ctypes/test_dllist.py", line 33, in test_lists_system
    self.assertTrue(
        any(lib in dll for dll in dlls for lib in KNOWN_LIBRARIES), f"loaded={dlls}"
    )
AssertionError: False is not true : loaded=['/home/blue/cpython/./python']

FAIL: test_lists_updates (test.test_ctypes.test_dllist.ListSharedLibraries.test_lists_updates)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/blue/cpython/Lib/test/test_ctypes/test_dllist.py", line 54, in test_lists_updates
    self.assertGreater(dlls2, dlls1, f"newly loaded libraries: {dlls2 - dlls1}")
AssertionError: {'/home/blue/cpython/./python'} not greater than {'/home/blue/cpython/./python'} : newly loaded libraries: set()

Running ldd on the Python executable confirms that these libraries are actually loaded:

./python:
   -lintl.1 => /usr/lib/libintl.so.1
   -lc.12 => /usr/lib/libc.so.12
   -lpthread.1 => /usr/lib/libpthread.so.1
   -lutil.7 => /usr/lib/libutil.so.7
   -lm.0 => /usr/lib/libm.so.0
   -lgcc_s.1 => /usr/lib/libgcc_s.so.1

OS: NetBSD 10.0
Architecture: amd64/x86_64
Python version: 3.14.0a6+

CPython versions tested on:

CPython main branch, 3.14

Operating systems tested on:

Other

The text was updated successfully, but these errors were encountered:

furkanonder · 2025-03-21T18:55:33Z

CC: @WardBrian

WardBrian · 2025-03-21T20:55:04Z

Interesting! I don't have access to a NetBSD machine to test on, but the man page for dl_iterate_phdr on netbsd.org is almost identical to Linux's or FreeBSD's, so the behavior you're describing is surprising, to say the least

The most immediate "fix" is probably to update the availability to only FreeBSD (instead of BSD libc) and disable the test on this platform?

furkanonder · 2025-03-23T19:33:58Z

Interesting! I don't have access to a NetBSD machine to test on, but the man page for dl_iterate_phdr on netbsd.org is almost identical to Linux's or FreeBSD's, so the behavior you're describing is surprising, to say the least

Yes, dl_iterate_phdr on NetBSD is almost identical to Linux. I wonder why the tests failed, maybe NetBSD's dynamic linker uses a different mechanism than what ctypes is looking for in the process's address space. I am open to ideas to analyze it.

furkanonder · 2025-03-23T19:34:12Z

The most immediate "fix" is probably to update the availability to only FreeBSD (instead of BSD libc) and disable the test on this platform?

Yes, this is a feasible solution. However, I think it is more important to find the underlying problem.

WardBrian · 2025-03-24T13:56:45Z

I suppose the right place to start is probably checking the basic operation of the c function. If I compile and run this locally:

#define _GNU_SOURCE // presumably not necessary on netbsd, but harmless?
#include <link.h>
#include <stdio.h>

int callback(struct dl_phdr_info *info, size_t size, void *data) {
    printf("name=%s, base=%p, size=%d\n", info->dlpi_name, (void *)info->dlpi_addr, info->dlpi_phnum);
    return 0;
}

int main(void) {
    dl_iterate_phdr(callback, NULL);

    return 0;
}

I get

name=, base=0x64c94a98b000, size=13
name=linux-vdso.so.1, base=0x7ffce8be4000, size=4
name=/lib/x86_64-linux-gnu/libc.so.6, base=0x747b8d400000, size=14
name=/lib64/ld-linux-x86-64.so.2, base=0x747b8d74f000, size=11

which matches ldd:

$ ldd main 
        linux-vdso.so.1 (0x00007ffdb569c000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007e3048400000)
        /lib64/ld-linux-x86-64.so.2 (0x00007e3048740000)

Could you run a similar experiment?

furkanonder · 2025-03-24T21:04:23Z

I suppose the right place to start is probably checking the basic operation of the c function. If I compile and run this locally:

#define _GNU_SOURCE // presumably not necessary on netbsd, but harmless?
#include <link.h>
#include <stdio.h>

int callback(struct dl_phdr_info *info, size_t size, void *data) {
printf("name=%s, base=%p, size=%d\n", info->dlpi_name, (void *)info->dlpi_addr, info->dlpi_phnum);
return 0;
}

int main(void) {
dl_iterate_phdr(callback, NULL);
return 0;
}
I get
name=, base=0x64c94a98b000, size=13
name=linux-vdso.so.1, base=0x7ffce8be4000, size=4
name=/lib/x86_64-linux-gnu/libc.so.6, base=0x747b8d400000, size=14
name=/lib64/ld-linux-x86-64.so.2, base=0x747b8d74f000, size=11
which matches ldd:
$ ldd main 
        linux-vdso.so.1 (0x00007ffdb569c000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007e3048400000)
        /lib64/ld-linux-x86-64.so.2 (0x00007e3048740000)
Could you run a similar experiment?

╰─$ cat main.c
#define _GNU_SOURCE // presumably not necessary on netbsd, but harmless?
#include <link.h>
#include <stdio.h>

int callback(struct dl_phdr_info *info, size_t size, void *data) {
    printf("name=%s, base=%p, size=%d\n", info->dlpi_name, (void *)info->dlpi_addr, info->dlpi_phnum);
    return 0;
}

int main(void) {
    dl_iterate_phdr(callback, NULL);

    return 0;
}

╰─$ gcc main.c -o main
╰─$ ./main
name=./main, base=0x0, size=7
name=/usr/lib/libc.so.12, base=0x7ebdaac00000, size=8
name=/usr/libexec/ld.elf_so, base=0x7f7fa6e00000, size=7
╰─$ ldd ./main
./main:
	-lc.12 => /usr/lib/libc.so.12
╭─blue@home ~
╰─$ uname -a
NetBSD home.localhost 10.0 NetBSD 10.0 (GENERIC) #0: Thu Mar 28 08:33:33 UTC 2024  mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/amd64/compile/GENERIC amd64

WardBrian · 2025-03-26T14:12:57Z

Ok, interesting! Mind trying the next level up?

file main.c:

#define _GNU_SOURCE
#include <link.h>
#include <stdio.h>

int callback(struct dl_phdr_info *info, size_t size, void *data) {
  printf("name=%s, base=%p, size=%d\n", info->dlpi_name,
         (void *)info->dlpi_addr, info->dlpi_phnum);
  return 0;
}

__attribute__((visibility("default"))) void call_me(void) {
  dl_iterate_phdr(callback, NULL);
}

gcc -shared -fpic -o libdllist_test.so main.c
python -c "import ctypes; ctypes.CDLL('./libdllist_test.so').call_me()"

In particular, the output should contain something like

name=./libdllist_test.so, base=0x7e88733fd000, size=11

as well as the system libraries.

furkanonder · 2025-03-27T13:47:56Z

Ok, interesting! Mind trying the next level up?

file main.c:

#define _GNU_SOURCE
#include <link.h>
#include <stdio.h>

int callback(struct dl_phdr_info *info, size_t size, void *data) {
printf("name=%s, base=%p, size=%d\n", info->dlpi_name,
(void *)info->dlpi_addr, info->dlpi_phnum);
return 0;
}

attribute((visibility("default"))) void call_me(void) {
dl_iterate_phdr(callback, NULL);
}
gcc -shared -fpic -o libdllist_test.so main.c
python -c "import ctypes; ctypes.CDLL('./libdllist_test.so').call_me()"
In particular, the output should contain something like
name=./libdllist_test.so, base=0x7e88733fd000, size=11
as well as the system libraries.

╰─$ cat main.c

#define _GNU_SOURCE
#include <link.h>
#include <stdio.h>

int callback(struct dl_phdr_info *info, size_t size, void *data) {
  printf("name=%s, base=%p, size=%d\n", info->dlpi_name,
         (void *)info->dlpi_addr, info->dlpi_phnum);
  return 0;
}

__attribute__((visibility("default"))) void call_me(void) {
  dl_iterate_phdr(callback, NULL);
}

╰─$ gcc -shared -fpic -o libdllist_test.so main.c

╰─$ python3.12 -c "import ctypes; ctypes.CDLL('./libdllist_test.so').call_me()"
name=python3.12, base=0x1c7e00000, size=8
name=/usr/pkg/lib/libpython3.12.so.1.0, base=0x79c970600000, size=8
name=/usr/lib/libintl.so.1, base=0x79c970200000, size=7
name=/usr/lib/libpthread.so.1, base=0x79c96fe00000, size=7
name=/usr/lib/libcrypt.so.1, base=0x79c96fa00000, size=7
name=/usr/lib/libutil.so.7, base=0x79c96f600000, size=7
name=/usr/lib/libm.so.0, base=0x79c96f200000, size=7
name=/usr/lib/libc.so.12, base=0x79c96ec00000, size=8
name=/usr/lib/libgcc_s.so.1, base=0x79c96e800000, size=7
name=/usr/lib/i18n/libUTF8.so.5.0, base=0x79c96de00000, size=7
name=/usr/pkg/lib/python3.12/lib-dynload/_ctypes.so, base=0x79c96da00000, size=7
name=/usr/pkg/lib/libffi.so.8, base=0x79c96d600000, size=7
name=/usr/pkg/lib/python3.12/lib-dynload/_struct.so, base=0x79c96d200000, size=7
name=./libdllist_test.so, base=0x79c96ce00000, size=6
name=/usr/libexec/ld.elf_so, base=0x7f7fa7400000, size=7
╭─blue@home ~
╰─$ ./cpython/python -c "import ctypes; ctypes.CDLL('./libdllist_test.so').call_me()"
name=./cpython/python, base=0x0, size=8
name=/usr/lib/libintl.so.1, base=0x72cc62400000, size=7
name=/usr/lib/libpthread.so.1, base=0x72cc62000000, size=7
name=/usr/lib/libutil.so.7, base=0x72cc61c00000, size=7
name=/usr/lib/libm.so.0, base=0x72cc61800000, size=7
name=/usr/lib/libgcc_s.so.1, base=0x72cc61400000, size=7
name=/usr/lib/libc.so.12, base=0x72cc60e00000, size=8
name=/usr/lib/i18n/libUTF8.so.5.0, base=0x72cc60400000, size=7
name=/home/blue/cpython/build/lib.netbsd-10.0-amd64-3.14/_ctypes.cpython-314d.so, base=0x72cc60000000, size=6
name=/usr/pkg/lib/libffi.so.8, base=0x72cc5fc00000, size=7
name=/home/blue/cpython/build/lib.netbsd-10.0-amd64-3.14/_struct.cpython-314d.so, base=0x72cc5f800000, size=6
name=./libdllist_test.so, base=0x72cc5f400000, size=6
name=/usr/libexec/ld.elf_so, base=0x7f7f48a00000, size=7
╭─blue@home ~
╰─$

WardBrian · 2025-03-27T14:39:47Z

Okay, fascinating

To recap:

In a C program, dl_iterate_phdr works as intended
In a shared library, called by ctypes, dl_iterate_phdr works as intended
Loaded directly from libc, using ctypes.CDLL(None)["dl_iterate_phdr"], it behaves differently.

It may be beyond my debugging-over-email abilities to determine the cause here. If someone more familiar with either the internals of how ctypes calls the system loader or with NetBSD, that would be appreciated!

Another guess is it could be related to the fact that loading it with CDLL(None), which I believe is equivalent to calling dlopen with NULL. E.g., this test program:

#define _GNU_SOURCE
#include <dlfcn.h>
#include <stdio.h>

struct dl_phdr_info {
  void *dlpi_addr;
  const char *dlpi_name;
  const void *dlpi_phdr;
  unsigned short dlpi_phnum;
};

int callback(struct dl_phdr_info *info, size_t size, void *data) {
  printf("name=%s, base=%p, size=%d\n", info->dlpi_name, info->dlpi_addr,
        info->dlpi_phnum);
  return 0;
}

typedef int (*dl_iterate_phdr_t)(int (*callback)(struct dl_phdr_info *info,
                                            size_t size, void *data),
                          void *data);

int main(void) {

  void* handle = dlopen(NULL, RTLD_NOW);
  if (!handle) {
    fprintf(stderr, "dlopen failed: %s\n", dlerror());
    return 1;
  }

  dl_iterate_phdr_t dl_iterate_phdr = dlsym(handle, "dl_iterate_phdr");
  if (!dl_iterate_phdr) {
    fprintf(stderr, "dlsym failed: %s\n", dlerror());
    dlclose(handle);
    return 1;

  }

  dl_iterate_phdr(callback, NULL);
}

This hypothesis could also be tested by replacing the None with a hardcoded path to libc in the dllist python code

furkanonder · 2025-03-29T11:04:53Z

@WardBrian

Inspired by your examples, I followed an approach using a library and got more meaningful results.

gcc -shared -fpic -o libdllist_helper.so dllist_helper.c

╰─cat dllist_helper.c                                                                         
#define _GNU_SOURCE
#include <link.h>
#include <stdlib.h>
#include <string.h>

// Function to count how many libraries are loaded
int count_libraries(struct dl_phdr_info *info, size_t size, void *data) {
    int *count = (int*)data;
    (*count)++;
    return 0;
}

// Function to fill the library names
int get_library_names(struct dl_phdr_info *info, size_t size, void *data) {
    char **names = (char**)data;
    static int index = 0;

    // Copy the library name
    if (info->dlpi_name && info->dlpi_name[0] != '\0') {
        names[index] = strdup(info->dlpi_name);
    } 
    else {
        // For the main executable with empty name
        names[index] = strdup("main_executable");
    }

    index++;
    return 0;
}

// Function to get the library count
__attribute__((visibility("default")))
int get_library_count(void) {
    int count = 0;
    dl_iterate_phdr(count_libraries, &count);
    return count;
}

// Function to get the library names
__attribute__((visibility("default")))
char** get_library_names_array(void) {
    int count = get_library_count();

    // Allocate memory for the array of strings
    char **names = (char**)malloc(count * sizeof(char*));
    if (!names) 
        return NULL;

    // Fill the array with library names
    dl_iterate_phdr(get_library_names, names);

    return names;
}

// Function to free the memory
__attribute__((visibility("default")))
void free_library_names(char **names, int count) {
    if (!names) 
        return;

    for (int i = 0; i < count; i++) {
        if (names[i]) free(names[i]);
    }

    free(names);
}

╰─$ cat test_libdllist_helper.py

import os
import sys
import ctypes

def dllist():
    helper_path = os.path.join(os.path.dirname(__file__), "libdllist_helper.so")
    helper = ctypes.CDLL(helper_path)

    get_count = helper.get_library_count
    get_count.restype = ctypes.c_int
    count = get_count()

    # Get the array of library names
    get_names = helper.get_library_names_array
    get_names.restype = ctypes.POINTER(ctypes.c_char_p)
    names_array = get_names()

    libraries = []
    for i in range(count):
        name_bytes = names_array[i]
        if name_bytes:
            name = os.fsdecode(name_bytes)
            if name == "main_executable":
                name = sys.executable
            libraries.append(name)

    # Free the memory allocated in C
    free_names = helper.free_library_names
    free_names.argtypes = [ctypes.POINTER(ctypes.c_char_p), ctypes.c_int]
    free_names(names_array, count)

    return libraries

print(dllist())

Output:

╰─$ ./Desktop/cpython/python test_libdllist_helper.py
['./Desktop/cpython/python', '/usr/lib/libintl.so.1', '/usr/lib/libpthread.so.1', '/usr/lib/libutil.so.7', '/usr/lib/libm.so.0', '/usr/lib/libgcc_s.so.1', '/usr/lib/libc.so.12', '/usr/lib/i18n/libUTF8.so.5.0', '/home/blue/Desktop/cpython/build/lib.netbsd-10.0-amd64-3.14/_ctypes.cpython-314d.so', '/usr/pkg/lib/libffi.so.8', '/home/blue/Desktop/cpython/build/lib.netbsd-10.0-amd64-3.14/_struct.cpython-314d.so', '/home/blue/libdllist_helper.so', '/usr/libexec/ld.elf_so']
╰─$ python3.12 test_libdllist_helper.py
['python3.12', '/usr/pkg/lib/libpython3.12.so.1.0', '/usr/lib/libintl.so.1', '/usr/lib/libpthread.so.1', '/usr/lib/libcrypt.so.1', '/usr/lib/libutil.so.7', '/usr/lib/libm.so.0', '/usr/lib/libc.so.12', '/usr/lib/libgcc_s.so.1', '/usr/lib/i18n/libUTF8.so.5.0', '/usr/pkg/lib/python3.12/lib-dynload/_ctypes.so', '/usr/pkg/lib/libffi.so.8', '/usr/pkg/lib/python3.12/lib-dynload/_struct.so', '/home/blue/libdllist_helper.so', '/usr/libexec/ld.elf_so']

furkanonder · 2025-03-29T11:07:31Z

Another guess is it could be related to the fact that loading it with CDLL(None), which I believe is equivalent to calling dlopen with NULL. E.g., this test program:

#define _GNU_SOURCE
#include <dlfcn.h>
#include <stdio.h>

struct dl_phdr_info {
void *dlpi_addr;
const char *dlpi_name;
const void *dlpi_phdr;
unsigned short dlpi_phnum;
};

int callback(struct dl_phdr_info *info, size_t size, void *data) {
printf("name=%s, base=%p, size=%d\n", info->dlpi_name, info->dlpi_addr,
info->dlpi_phnum);
return 0;
}

typedef int (*dl_iterate_phdr_t)(int (*callback)(struct dl_phdr_info *info,
size_t size, void *data),
void *data);

int main(void) {

void* handle = dlopen(NULL, RTLD_NOW);
if (!handle) {
fprintf(stderr, "dlopen failed: %s\n", dlerror());
return 1;
}

dl_iterate_phdr_t dl_iterate_phdr = dlsym(handle, "dl_iterate_phdr");
if (!dl_iterate_phdr) {
fprintf(stderr, "dlsym failed: %s\n", dlerror());
dlclose(handle);
return 1;

}

dl_iterate_phdr(callback, NULL);
}

This hypothesis could also be tested by replacing the None with a hardcoded path to libc in the dllist python code

I have tried the hardcoded libc path, but it doesn't seem to work.

╰─$ cat hard_coded_path.py

import os
import sys
import ctypes
from ctypes.util import find_library

if sys.platform.startswith('netbsd'):
    libc_path = "/usr/lib/libc.so.12"  # NetBSD's libc path
    _libc = ctypes.CDLL(libc_path)

    # Define structure for NetBSD's dl_phdr_info
    class _dl_phdr_info(ctypes.Structure):
        _fields_ = [
            ("dlpi_addr", ctypes.c_void_p),      # Elf_Addr
            ("dlpi_name", ctypes.c_char_p),      # const char *
            ("dlpi_phdr", ctypes.c_void_p),      # const Elf_Phdr *
            ("dlpi_phnum", ctypes.c_ushort),     # Elf_Half
            ("dlpi_adds", ctypes.c_ulonglong),   # unsigned long long int
            ("dlpi_subs", ctypes.c_ulonglong),   # unsigned long long int
            ("dlpi_tls_modid", ctypes.c_size_t), # size_t
            ("dlpi_tls_data", ctypes.c_void_p)   # void *
        ]

    _dl_phdr_callback = ctypes.CFUNCTYPE(
        ctypes.c_int,
        ctypes.POINTER(_dl_phdr_info),
        ctypes.c_size_t,
        ctypes.c_void_p,
    )

    @_dl_phdr_callback
    def _info_callback(info, size, data):
        libraries = ctypes.cast(data, ctypes.POINTER(ctypes.py_object)).contents.value
        if info.contents.dlpi_name:
            name = os.fsdecode(info.contents.dlpi_name)
            if name:
                libraries.append(name)
        return 0

    _dl_iterate_phdr = _libc.dl_iterate_phdr
    _dl_iterate_phdr.argtypes = [
        _dl_phdr_callback,
        ctypes.c_void_p,
    ]
    _dl_iterate_phdr.restype = ctypes.c_int

    def dllist():
        libraries = []
        data = ctypes.py_object(libraries)
        _dl_iterate_phdr(_info_callback, ctypes.addressof(data))

        # Add the Python executable if it's not already in the list
        executable = sys.executable
        if not any(executable in lib for lib in libraries):
            libraries.insert(0, executable)

        return libraries


print(dllist())

Output:

╭─blue@home ~/Desktop/cpython ‹main●›
╰─$ ./python hard_coded_path.py
['/home/blue/Desktop/cpython/python', '/home/blue/Desktop/cpython/./python']

furkanonder added 3.14 new features, bugs and security fixes tests Tests in the Lib/test dir topic-ctypes type-bug An unexpected behavior, bug, or error labels Mar 21, 2025

furkanonder removed the type-bug An unexpected behavior, bug, or error label Mar 22, 2025

picnixz added the type-bug An unexpected behavior, bug, or error label Mar 23, 2025

vstinner added the OS-netbsd label Apr 14, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`test_dllist` fails on NetBSD: dl_iterate_phdr doesn't report shared libraries #131565

`test_dllist` fails on NetBSD: dl_iterate_phdr doesn't report shared libraries #131565

furkanonder commented Mar 21, 2025 •

edited by github-actions bot

Loading

furkanonder commented Mar 21, 2025

WardBrian commented Mar 21, 2025

furkanonder commented Mar 23, 2025

furkanonder commented Mar 23, 2025

WardBrian commented Mar 24, 2025

furkanonder commented Mar 24, 2025

WardBrian commented Mar 26, 2025

furkanonder commented Mar 27, 2025

WardBrian commented Mar 27, 2025

furkanonder commented Mar 29, 2025

furkanonder commented Mar 29, 2025

test_dllist fails on NetBSD: dl_iterate_phdr doesn't report shared libraries #131565

test_dllist fails on NetBSD: dl_iterate_phdr doesn't report shared libraries #131565

Comments

furkanonder commented Mar 21, 2025 • edited by github-actions bot Loading

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

furkanonder commented Mar 21, 2025

WardBrian commented Mar 21, 2025

furkanonder commented Mar 23, 2025

furkanonder commented Mar 23, 2025

WardBrian commented Mar 24, 2025

furkanonder commented Mar 24, 2025

WardBrian commented Mar 26, 2025

furkanonder commented Mar 27, 2025

WardBrian commented Mar 27, 2025

furkanonder commented Mar 29, 2025

furkanonder commented Mar 29, 2025

`test_dllist` fails on NetBSD: dl_iterate_phdr doesn't report shared libraries #131565

`test_dllist` fails on NetBSD: dl_iterate_phdr doesn't report shared libraries #131565

furkanonder commented Mar 21, 2025 •

edited by github-actions bot

Loading