Access a Jupyter Kernel inside WSL from a JupyterLab installation running on windows.

TLDR: It is possible to run only the kernel component of a JupyterLab installation inside Windows Subsystem for Linux, while the JupyterLab server runs on Windows.

Conda made installing Python on Windows easy. All fight-like experiences from the past are gone and by using different environments the whole dependency troubles are also solved. One issue still exists, that a package is only available for Linux (and perhaps OS X), often due to compatibility of the project with a linux-like system only. In WIndows 10, Microsoft provided WSL or Windows Subsystem for Linux as a solution for that. A small wrapper layer that translates Linux kernel calls into Windows kernel commands.

Most of my programming work now-a-day is explorative using a Jupyter Server and one of my tools (Meep) is only available in Linux or inside WSL, but a lot of my environments work fine in Windows and I run my JupyterLab server in a Conda environment on Windows. So I was wondering if I can create a conda environment inside WSL, activate a JupyterLab Kernel there and connect from my normal JupyterLab installation to it. It works. I can now have the Linux conda environments and the Windows conda environments next to each other. I documented my process below, and hope somebody else can use this as-well. If you want to run the whole Jupyter stack on WSL, a lot of guides exist.

We start with a working WSL environment, which is beyond this guide. We installed a linux image, e.g. the last Ubuntu image from the Microsoft Store and then the next lines install Miniconda in the the <desired_prefix> path.

wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
bash miniconda.sh -b -p <desired_prefix>
export PATH=<desired_prefix>/bin:$PATH

I usually install Miniconda inside /opt/conda, but feel free to choose something else. Then the base conda environment can be permanently activated in your WSL bash by running conda init bash.

Most of my conda environments run in the Windows Anaconda installation, and I want to keep this. At the same time I want to run Meep simulations and explore them inside of Jupyter. Unfortunately Meep does not run on Windows, but that is what WSL was developed for. The WSL system uses the same network stack, which makes it possible to easily interact on the computer using the loopback adress 127.0.0.1 and we can tell our Jupyter server running in Windows to consume the kernel running in the WSL.

We create a new environment in WSL, that we want to use in the Jupyter installation.

conda create -n py3-wsl-meep
conda activate py3-wsl-meep
conda config --add channels conda-forge
conda config --set channel_priority strict

We add the conda-forge channel and use it as the main resource. The conda-forge is a project to make automatic, open builds in conda. The last line sets the priority to resolving packages to use what ever possible from the given repository. Now we can install meep.

conda install pymeep

Jupyter is build on a front-end server that is programming language agnostic and connects to a backend kernel, that evaluates the code. Kernels for a lot of different programming languages are available. We choose the Python Kernel (ipykernel) and install this kernel in our environment py3-wsl-meep.

conda install ipykernel

Now the environment inside of WSL is prepared, but we need to tell the Jupyter server that this kernel exists. If this would just be a kernel in a new environment on windows, a simple command would be enough. In our case we have to adapt the standard use-case to work with the WSL.

The kernel tells the Jupyter server in the kernelspecs how to interact with it. Most of the information is contained in the kernel.jsonfile. When the Jupyter server wants to execute a new kernel it reads the specs files, runs some kind of command and tells the kernel how to connect to the server. Then it is back to the kernel to connect to the server. We have to modify this specification file manually to make it possible for this interaction to work.

As a first step in the WSL conda environment (py3-wsl-meep), we create the specification files for the kernel to a temporary location e.g. /tmp.

python -m ipykernel install --name 'py3-wsl-meep' --prefix /tmp --display-name "Python 3 (WSL - Meep)"

The kernelspecs are created in /tmp/share/jupyter/kernels/py3-wsl-meep and contain the kernel.json file that we need to adapt.

{
 "argv": [
  "/opt/conda/envs/py3-wsl-meep/bin/python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python 3 (WSL - Meep)",
 "language": "python"
}

We first copy all the files from the subsystem to a temporary location, e.g. on your Desktop. This can be done by running explorer.exe . in the directory /tmp/share/jupyter/kernels/ and copy the folder py3-wsl-meep. Then we open the kernel.json file from this folder in an editor.

There are two problems we have to fix. First, the jupyter server can not just launch the python command, because it is inside the WSL. We have to translate that command to be executed in the WSL system. Second, the location for the connection file ({connection_file}) that is generated by the Jupyter server is located on the Windows file system that is mounted below /mnt/c on the WSL.

The command execution we fix by executing the command through the wsl.exe binary provided by Windows, that executes the command inside the subsystem using the login shell, in the common case bash.

{
 "argv": [
  "C:\\windows\\system32\\wsl.exe",
  "-d",
  "ubuntu",
  "/opt/conda/envs/py3-wsl-meep/bin/python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python 3 (WSL - Meep)",
 "language": "python"
 }

The argument -d ubuntu allows me to choose the correct installed WSL image, in my case the latest Ubuntu Image. You can find the available options on your system by running wsl --list in a normal command prompt.

Now the second item to fix is the variable {connection_file} that is replaced by the Jupyter server by a file that explains how to connect to the server. This is a Windows path, that can not be directly accessed by the WSL system. Microsoft provides wslpath to remedy this.

wslpath -a '{connection_file}'converts the path from a Windows file path to a path that can be is available inside the WSL. We adapt the specs file one more time.

{
 "argv": [
  "C:\\windows\\system32\\wsl.exe",
  "-d",
  "ubuntu",
  "/opt/conda/envs/py3-wsl-meep/bin/python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "\"$(wslpath -a '{connection_file}')\""
 ],
 "display_name": "Python 3 (WSL - Meep)",
 "language": "python"
}

The specs file is now ready to be used by the jupyter server. Open a command prompt in the environment where the jupyter server is installed, and run the command python -m jupyter kernelspec list. This should give you a list of all the installed kernels and there should be at least one in there.

I assume the folder with the specification of the kernel was copied to C:\Users\Samuel\Desktop\py3-wsl-meep. Now you can install the specs by either copying the spec folder into one of the locations shown by the list command or by running the following command.

python -m jupyter kernelspec install "C:\Users\Samuel\Desktop\py3-wsl-meep"

On the next start of the Jupyter server the kernel is available to choose. All the things, such as writing files with relative paths, work just as well as with a “local” kernel. If you use absolute paths into the windows file system, you have to prepend the path with /mnt/c/.

Let me know in the comments, if you have improvements or edge cases where you found a solution for. This prooves again how powerful the Jupyter ecosystem is.