Written 23 Jan 2015

Windows DLL search path

On windows, we want applications to be able to start without the need for a path specification, yet we need our DLLs to be found. In the past, we relied on PATH for this, but this is going away, for the following reasons:

  1. The PATH variable on windows is extremely limited (260 chars) and it runs out fast. on User’s machines, it can lead to problems.
  2. If two or more versions of an application are installed, one of the two will end up using the libraries of the other, with horrible consequences, depending on which come first in the PATH specification.
  3. Using the path doesn’t solve the problem that some users encounter, that is, if they have a library in the System32 directory that happens to have the same name, it will override ours and generate strange errors.

Windows has a weird lookup strategy for dlls, and it depends on flags and other circumstances. What [http://msdn.microsoft.com/en-us/library/7d83bc18.aspx MSDN] says is the following order, in the most simplified approach:

An important thing to note is that “executable module” is not necessarily the running exe. If exe depends on a library dll1 which in turn depends on another library dll2, the “executable module” will be dll1. To add to the confusion, we need to deal with both regular dependencies (e.g. linkage) and with dynamic opening (“dlopen” like) which is extensively used by python to load “pyd” files (basically, compiled python modules).

We want to make the last entry irrelevant, that is, we don’t want to depend on the path for library lookup. Excluding the system directories and magic registry tricks, we are left only with the first option. Which means that we need to put all stuff in the right place. We also don’t want system libraries to take over or influence our installation.

A temporary solution has been devised, but requires care. Also, the solution may involve two different strategies for developers and users (deployed version)

Developers

For developers, keep relying on %PATH% for lookup, and point this to the external-libs/build/lib. You may still have weird behaviors if there’s another application installed, another python, or another set of libraries somewhere on the system. This is an acceptable compromise.

The alternative would be littering external-libs/build/ with copies of dll files in the right places, near the relevant modules.

Users

As said above, the right place depends on what entity we are talking about. There are many possible situations that may occur

Four possible places where dlls may need to go

I present here all cases, together with the proper placement.

NOTE: If you have both a direct dependency for an executable, and a direct or dynamic dependency on the same dll somewhere else, you might have to have two copies of the dll in two different places.

Open Problems

Other attempts

Here is a list of alternative strategies we examined to tackle the problem, unsuccessful for some reason or another.