For the last couple of years, I have always turned to anaconda/miniconda for creating virtual environments with python, both under windows and linux. Although it is possible to work with just pip and things like virtualenv, especially in the beginning I often had issues trying to install certain packages under windows that required compilation steps. In the meantime, that might have improved, but just stuck with using conda for the environments.
So while using conda as the main environment/package manager, it happens sometimes that some package I need is not available using conda, but only via pip. Fortunately, using pip within a conda environment this should not be a problem, it would use the pip installed within the environment and install the pip package into the current anaconda environment.
They keyword in that assumption is should.....
On my windows machine, over time I saw that my base conda environment got polluted with all kinds of pip packages, and packages that were installed with pip were sometimes not available in my active environment, even though I just had installed them WHILE the prompt indicated I was working in this project environment.
As it turns out, the conda environment in the windows command prompt appears to be a bit of an illusion:
When working, it happens every now and then that I type a command that I did not want to type, I might have forgotten to add some arguments. For example, I originally typed:
conda search dvc
while I actually wanted to search on conda-forge, so I quickly press
Ctrl-C, modify my command to
conda search -c conda-forge dvc
and get to see what is the latest version of DVC on conda-forge.
Unfortunately, it is the pressing of
Ctrl-C to abort the running conda search command that seems to mess up the internal state of conda, at least on windows machines
I double-checked, and indeed on my windows system (not on my linux system) I can reproduce it as follows:
# Display the location of pip (my_awesome_project) C:\Users\guido> python -m pip Usage: C:\Users\guido\Miniconda3\envs\my_awesome_project\python.exe -m pip <command> [options] Commands: install Install packages. download Download packages. ...
As you can see, the output of pip shows that it is running using the python.exe from the selected environment folder.
If I now run a
conda search command in this terminal that I terminate with a
Ctrl-c as follows:
conda search dvc # Press Ctrl-C while the conda command is running (my_awesome_project) C:\Users\guido> conda search dvc Loading channels: / CondaError: KeyboardInterrupt
After the prompt returns, I still see the (my_awesome_project) in the prompt, letting me believe I am still in the same active prompt.
However, if I know run the exact same
python -m pip command, I see a different output:
(my_awesome_project) C:\Users\guido> python -m pip Usage: C:\Users\guido\Miniconda3\python.exe -m pip <command> [options] Commands: install Install packages. download Download packages. ...
As you can see, right now the conda environment used by pip is NOT the project enevironment, but it is the conda base environment. Any package you now install with pip will not be installed in the project environment, but in your base environment....
I tried the exact same steps on my linux computer and there it appears I cannot reproduce this exact same problem.
A very simple workaround for this problem: do not use
Ctrl-c on any running conda process....
I am not sure if this is a problem for conda in general, or whether this is a specific conda on windows problem.