I've been battling the Linux linker for a while, specifically the dlopen() call. Today I came to understand exactly why it doesn't match my expectations. Hopefully what I've learned will help others avoid the trap I've fallen into a couple of times. Maybe it will even spark a change in glibc.

The linker searches for libraries in at least four places:

• The paths specified by the environment variable LD_LIBRARY_PATH.
• The paths compiled into executables and shared libraries. Most libraries and executables don't seem to use this, but it's available. See the "-rpath" option in the "ld" man page.
• The paths specified in /etc/ld.so.cache, which /sbin/ldconfig generates by searching for libraries in the paths listed in /etc/ld.so.conf. This is why you have to re-run ldconfig after installing or removing shared libraries.
• The default library paths, /lib and /usr/lib. There may be architecture-specific default paths as well.

There is no provision for modifying the library search path once your program has started. You can pass absolute paths to dlopen(), but if the library you're loading requires private libraries and it loads the libraries without using absolute paths, you're stuck. Your program can not dynamically add paths to the library search path.

Today I was fiddling with JPype, a Python extension that loads a Java virtual machine. To use the extension on Linux, you have to set the LD_LIBRARY_PATH environment variable, providing two paths so that the Java VM can find its own libraries. Even if you try to modify LD_LIBRARY_PATH at runtime, the linker ignores your changes. Therefore the cleanest way to load the JPype extension is to create a wrapper script that sets up LD_LIBRARY_PATH before starting Python.

I spent a lot of time trying to improve this situation, but I think the missing linker feature may actually be intentional--maybe someone is concerned that changing the library search path at runtime will lead to security holes. So the only reasonable options are to either modify the static search path using LD_LIBRARY_PATH or /etc/ld.so.conf, or to use -rpath when compiling the library you want your library to load. Getting everyone (particularly Sun) to use -rpath seems like a tall order, so LD_LIBRARY_PATH will just have to do. It's ugly, but not horrible.