Troubleshooting Python.NET initialization problems (.NET Core) with virtual environments

pythondotnetinit.cs

        public static void Init(string pathVirtualEnv)
        {
            string pathVeScripts;
            if (Environment.OSVersion.Platform == PlatformID.Win32NT)
                pathVeScripts = PathVirtualEnv + @"\Scripts";
            else
                pathVeScripts = PathVirtualEnv + @"/bin";
            Environment.SetEnvironmentVariable("PATH", pathVeScripts, EnvironmentVariableTarget.Process);

            var pythonPath = string.Empty;

            var proc = new Process();
            proc.StartInfo.FileName = pathVeScripts + Path.DirectorySeparatorChar + "python";            
            proc.StartInfo.Arguments = $"-c \"import sys; print('{Path.PathSeparator}'.join(sys.path))\"";
            proc.StartInfo.RedirectStandardOutput = true;
            if (!proc.Start())
                throw new Exception("Couldn't initialize Python in virtual environment");
            proc.WaitForExit();

            pythonPath = proc.StandardOutput.ReadToEnd();
            pythonPath = pythonPath.Replace(Environment.NewLine, "");
            if (string.IsNullOrEmpty(pythonPath))
                throw new Exception("Couldn't initialize Python.NET");

            Environment.SetEnvironmentVariable("PYTHONPATH", pythonPath, EnvironmentVariableTarget.Process);
            PythonEngine.PythonPath = pythonPath;
        }

I'm using Python.NET to develop a .NET library that relies on a well known Python project.

The code has to run on a Linux server, so I've chosen .NET Core.

Sadly, I couldn't get the runtime to work using a virtual environment, as shown in the project tutorial

My setup

• Windows 10
• There's no system wide Python intepreter (No Python found in %PATH%)
• Several Python 3 versions installed in %LOCALAPPDATA%\Programs
• Using Python Launcher 3.8.5 (py.exe, found in %PATH%) to create virtual environments using a specific Python version for each project.
• .NET Core 3.1, .NET Standard 2.0
• Using Python.NET as the unofficial .NET Core compatible NuGet package LostTech.Python.Runtime
• Python.NET package version 3.0.3
• A virtual environment created with Python 3.7 (As Python.NET is compiled against this version)

1) The runtime is unable to find Python DLL

First, the required DLLs are not found in the virtual environment folders

To create a venv using Python 3.6 with this command py -3.6 -m venv venv36 copies all DLLs to venv36\Scripts folder

If we try the same with Python 3.7 py -3.7 -m venv venv37 it does not copy the dynamic libraries.

I didn't go much deeper with this. It might be bug 3942 or not, but I decided to copy the binaries manually.

Then, I had to set the process %PATH% to the location of the dynamic library, for the runtime to find the DLL.

2) The runtime is unable to find the standard library and/or the venv modules

I went for a quick workaround rather than properly debugging Python.NET.

The normal activation of the virtualenv works sucesfully, I guess that means sys.path is properly build. So I decided to run a process to cache its CPython intepreter initialization and then manually recreating the variable it when launching the .NET runtime.

This is done by printing sys.path to stdout and redirecting the standard output to a .NET string. Then I just initialize %PYTHONPATH% environment variable and let the runtime correctly initializes the list.

This seems to work for Windows (with and without a system-wide Python) and Ubuntu Linux with a system Python3.

The initialization code is as shown in pythondotnetinit.cs