test_dllist fails on NetBSD: dl_iterate_phdr doesn't report shared libraries

[furkanonder]

Bug report

Bug description:

The test_dllist tests in Lib/test/test_ctypes/test_dllist.py are failing on NetBSD. The issue appears to be that the dl_iterate_phdr function on NetBSD does not report the same information about loaded shared libraries as it does on Linux and other platforms.

Specifically, on NetBSD, dl_iterate_phdr only returns the main Python executable and doesn't include any of the shared libraries that are actually loaded.

---

./python -m test test_ctypes -m test_dllist -v

Output:

FAIL: test_lists_system (test.test_ctypes.test_dllist.ListSharedLibraries.test_lists_system)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/blue/cpython/Lib/test/test_ctypes/test_dllist.py", line 33, in test_lists_system
    self.assertTrue(
        any(lib in dll for dll in dlls for lib in KNOWN_LIBRARIES), f"loaded={dlls}"
    )
AssertionError: False is not true : loaded=['/home/blue/cpython/./python']

FAIL: test_lists_updates (test.test_ctypes.test_dllist.ListSharedLibraries.test_lists_updates)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/blue/cpython/Lib/test/test_ctypes/test_dllist.py", line 54, in test_lists_updates
    self.assertGreater(dlls2, dlls1, f"newly loaded libraries: {dlls2 - dlls1}")
AssertionError: {'/home/blue/cpython/./python'} not greater than {'/home/blue/cpython/./python'} : newly loaded libraries: set()

Running ldd on the Python executable confirms that these libraries are actually loaded:

./python:
   -lintl.1 => /usr/lib/libintl.so.1
   -lc.12 => /usr/lib/libc.so.12
   -lpthread.1 => /usr/lib/libpthread.so.1
   -lutil.7 => /usr/lib/libutil.so.7
   -lm.0 => /usr/lib/libm.so.0
   -lgcc_s.1 => /usr/lib/libgcc_s.so.1

• OS: NetBSD 10.0
• Architecture: amd64/x86_64
• Python version: 3.14.0a6+

CPython versions tested on:

CPython main branch, 3.14

Operating systems tested on:

Other

[furkanonder]

CC: @WardBrian

[WardBrian]

Interesting! I don't have access to a NetBSD machine to test on, but the man page for dl_iterate_phdr on netbsd.org is almost identical to Linux's or FreeBSD's, so the behavior you're describing is surprising, to say the least

The most immediate "fix" is probably to update the availability to only FreeBSD (instead of BSD libc) and disable the test on this platform?

[furkanonder]

Interesting! I don't have access to a NetBSD machine to test on, but the man page for dl_iterate_phdr on netbsd.org is almost identical to Linux's or FreeBSD's, so the behavior you're describing is surprising, to say the least

Yes, dl_iterate_phdr on NetBSD is almost identical to Linux. I wonder why the tests failed, maybe NetBSD's dynamic linker uses a different mechanism than what ctypes is looking for in the process's address space. I am open to ideas to analyze it.

[furkanonder]

The most immediate "fix" is probably to update the availability to only FreeBSD (instead of BSD libc) and disable the test on this platform?

Yes, this is a feasible solution. However, I think it is more important to find the underlying problem.

[WardBrian]

I suppose the right place to start is probably checking the basic operation of the c function. If I compile and run this locally:

#define _GNU_SOURCE // presumably not necessary on netbsd, but harmless?
#include <link.h>
#include <stdio.h>

int callback(struct dl_phdr_info *info, size_t size, void *data) {
    printf("name=%s, base=%p, size=%d\n", info->dlpi_name, (void *)info->dlpi_addr, info->dlpi_phnum);
    return 0;
}

int main(void) {
    dl_iterate_phdr(callback, NULL);

    return 0;
}

I get

name=, base=0x64c94a98b000, size=13
name=linux-vdso.so.1, base=0x7ffce8be4000, size=4
name=/lib/x86_64-linux-gnu/libc.so.6, base=0x747b8d400000, size=14
name=/lib64/ld-linux-x86-64.so.2, base=0x747b8d74f000, size=11

which matches ldd:

$ ldd main 
        linux-vdso.so.1 (0x00007ffdb569c000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007e3048400000)
        /lib64/ld-linux-x86-64.so.2 (0x00007e3048740000)

Could you run a similar experiment?

[furkanonder]

I suppose the right place to start is probably checking the basic operation of the c function. If I compile and run this locally:

#define _GNU_SOURCE // presumably not necessary on netbsd, but harmless?
#include <link.h>
#include <stdio.h>

int callback(struct dl_phdr_info *info, size_t size, void *data) {
printf("name=%s, base=%p, size=%d\n", info->dlpi_name, (void *)info->dlpi_addr, info->dlpi_phnum);
return 0;
}

int main(void) {
dl_iterate_phdr(callback, NULL);

return 0;

}
I get

name=, base=0x64c94a98b000, size=13
name=linux-vdso.so.1, base=0x7ffce8be4000, size=4
name=/lib/x86_64-linux-gnu/libc.so.6, base=0x747b8d400000, size=14
name=/lib64/ld-linux-x86-64.so.2, base=0x747b8d74f000, size=11

which matches ldd:

$ ldd main 
        linux-vdso.so.1 (0x00007ffdb569c000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007e3048400000)
        /lib64/ld-linux-x86-64.so.2 (0x00007e3048740000)

Could you run a similar experiment?

╰─$ cat main.c
#define _GNU_SOURCE // presumably not necessary on netbsd, but harmless?
#include <link.h>
#include <stdio.h>

int callback(struct dl_phdr_info *info, size_t size, void *data) {
    printf("name=%s, base=%p, size=%d\n", info->dlpi_name, (void *)info->dlpi_addr, info->dlpi_phnum);
    return 0;
}

int main(void) {
    dl_iterate_phdr(callback, NULL);

    return 0;
}

╰─$ gcc main.c -o main
╰─$ ./main
name=./main, base=0x0, size=7
name=/usr/lib/libc.so.12, base=0x7ebdaac00000, size=8
name=/usr/libexec/ld.elf_so, base=0x7f7fa6e00000, size=7
╰─$ ldd ./main
./main:
	-lc.12 => /usr/lib/libc.so.12
╭─blue@home ~
╰─$ uname -a
NetBSD home.localhost 10.0 NetBSD 10.0 (GENERIC) #0: Thu Mar 28 08:33:33 UTC 2024  mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/amd64/compile/GENERIC amd64

[WardBrian]

Ok, interesting! Mind trying the next level up?

file main.c:

#define _GNU_SOURCE
#include <link.h>
#include <stdio.h>

int callback(struct dl_phdr_info *info, size_t size, void *data) {
  printf("name=%s, base=%p, size=%d\n", info->dlpi_name,
         (void *)info->dlpi_addr, info->dlpi_phnum);
  return 0;
}

__attribute__((visibility("default"))) void call_me(void) {
  dl_iterate_phdr(callback, NULL);
}

gcc -shared -fpic -o libdllist_test.so main.c
python -c "import ctypes; ctypes.CDLL('./libdllist_test.so').call_me()"

In particular, the output should contain something like

name=./libdllist_test.so, base=0x7e88733fd000, size=11

as well as the system libraries.