Skip to content

Enable support for editable dependencies #2593

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
iFreilicht opened this issue May 16, 2025 · 1 comment
Open

Enable support for editable dependencies #2593

iFreilicht opened this issue May 16, 2025 · 1 comment

Comments

@iFreilicht
Copy link

iFreilicht commented May 16, 2025

When installing editable dependencies with pip or other python package managers into a virtualenv, they end up in site-packages as .pth files which contain the actual absolute path of the package.

Usually, python is able to follow these as if they were symlinks and import packages that were installed that way. However, this doesn't seem to work with Python.NET. I just get an error in Py.Import:

An exception of type 'Python.Runtime.PythonException' occurred in Python.Runtime but was not handled in user code: 'No module named 'mymodule''

I do have a workaround for this, which is just to read all the .pth files and add those paths to PYTHONPATH manually, but it would be much nicer if this could just work out of the box.

    string virtualEnvPath = $@".\myDir\.venv";
    string venvSitePackagesPath = $@"{sVirtualEnvPath}\Lib\site-packages";

    // Resolve editable dependencies manually because Python.NET doesn't do so
    string[] editableDependencyReferences = Directory.GetFiles(venvSitePackagesPath, "*.pth");
    string[] editableDependencyPaths = editableDependencyReferences
      .Select(filePath => File.ReadAllText(filePath).Trim())
      .ToArray();

    string pythonPath = $"{venvSitePackagesPath};" +
                         @"C:\Program Files\Python312\Lib;" +
                         @"C:\Program Files\Python312\DLLs;" +
                         string.Join(";", editableDependencyPaths); // Add them to the path

    Environment.SetEnvironmentVariable("PATH", 
                                       Environment.GetEnvironmentVariable("PATH").TrimEnd(';') + ";" +
                                       virtualEnvPath, EnvironmentVariableTarget.Process);
    Environment.SetEnvironmentVariable("PYTHONHOME", virtualEnvPath, EnvironmentVariableTarget.Process);
    Environment.SetEnvironmentVariable("PYTHONPATH", pythonPath, EnvironmentVariableTarget.Process);

    PythonEngine.PythonHome = virtualEnvPath;
    PythonEngine.PythonPath = pythonPath;
    PythonEngine.Initialize();

    using (Py.GIL())
    {
      dynamic mymodule = Py.Import("mymodule"); // Succeeds now!
    }
@filmor
Copy link
Member

filmor commented May 17, 2025

This is how our import method is implemented:

public static PyObject Import(string name)
{
if (name is null) throw new ArgumentNullException(nameof(name));
NewReference op = Runtime.PyImport_ImportModule(name);
return IsModule(op.BorrowOrThrow()) ? new PyModule(op.Steal()) : op.MoveToPyObject();
}

This is the underlying function in the C API:

Citing the docs of PyImport_Import:

This is a higher-level interface that calls the current “import hook function” (with an explicit level of 0, meaning absolute import). It invokes the __import__() function from the __builtins__ of the current globals. This means that the import is done using whatever import hooks are installed in the current environment.

We do exactly what the built in import does.

If you can identify an actual issue with our setup (e.g. with regard to the initial configuration), feel free to reopen. But "parsing" pth files (which your code example is doing in a too simplistic fashion, pth files are allowed to contain Python code) can't be the answer and I don't consider this as of now a bug in pythonnet itself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants