Python Virtual Environments tutorial using Virtualenv and Poetry

A mini-guided tutorial showing how to use virtual environment and why it's matters on virtualenv and poetry illustrated examples.

Intro

What is virtual environment

Why to use virtual environment

How to use virtual environment

Virtualenv
Poetry

Intro

What this blog post is about

This blog post is mostly aimed at people who didn't work with it. Here you will find that this is not a complete Python virtual environment reference, it is rather a mini-guided tutorial about:

what Python virtual environment is

why bother using it

how to use it

... with example process of two popular modules: virtualenv and poetry, and software such as IntelliJ IDE via Python plugin, PyCharm Community Edition, VSCode, Windows system, and Git Bash terminal.

If you're using JetBrains products you also need to index installed site-packages from the virtual environment.

Indexing is a core JetBrains features: code completion, inspections, finding usages, navigation, syntax highlighting, refactoring, and more.

What is virtual environment?

A thing that isolates other things

Python virtual environment is basically a separate folder that creates an independent set of installed packages, Python binaries in its own directory, that isolates any other installation of Python on your computer.

A thing that prevents conflicts

Python virtual environment is used to prevent interfering with the behavior of other applications. Therefore it will prevent packages or Python version conflicts when working with different projects that are running on the same system.

Why to use virtual environment?

To avoid Python version conflicts

Python virtual environment allows multiple versions of Python to coexist with each other. It will let you work with the old version of Python after installing a newer version all on the same system.

If you try to do it without separated virtual environment things will break pretty quickly:

To avoid package version conflicts

Say you're on two projects, two of them are using serpapi library which is installed globally (system-wide) with a 1.15 version.

Project_1 depends on the 1.05 version and Project_2 depends on the 1.08 version. What will happen if each project tries to import a serpapi library...

Example without the virtual environment

Since Python doesn’t distinguish between different versions of the same library in the /site-packages directory, this leads to the problem when you have two projects that require different versions of the same library and globally installed library have a completely different version.

Example with the virtual environment

When using a Python virtual environment you can use different versions of the same library or different versions of the Python separated by different virtual environments - folders.

📌Note: You can install globally different versions of site-packages and use them but as stated before it would become a mess pretty quickly and could break system tools or other projects.

How to use virtual environment?

Let's look at examples of how to use Python virtual environment from the initial install, creating and activating environment, adding dependencies using virtualenv and poetry modules, and deactivating virtual environment when done.

An example process of using Virtualenv

Install virtualenv:

$ pip install virtualenv

Create environment folder inside the current package/project directory:

$ virtualenv env

env is a folder created by virtualenv module.

Activate environment:

# On Windows
source env/Scripts/activate

(env)

# On Linux
$ source env/bin/activate

(env) $

(env) indicates that you're in the virtual environment.

Add site-packages (third-party libraries) to the activated environment.

Add the latest version:

(env) $ pip install google-search-results

Add specific version using equals == sign:

(env) $ pip install 'google-search-results==1.3.0'

📌Note: if you're on Windows and using Command Line Prompt, use double quotes " when specifying versions:

pip install 'google-search-results==1.3.0'
ERROR: Invalid requirement: "'google-search-results==1.3.0'"

Add specific version without overwriting lower version(s):

(env) $ pip install -I 'google-search-results==1.3.0'

-I argument will ignore already installed packages.

A quick look at how you can install site-package (virtualenv) and create a virtual environment for a specific Python version:

# install package for specific Python version (https://bit.ly/3pXtHng)

# For Windows:
$ py -3.6 -m pip install virtualenv

# For Linux
$ python3.6 -m pip install virtualenv

# create venv for specific Python version (https://bit.ly/3EjH0ni)
$ virtualenv -p python3.6 test_env

Use and index added site-packages inside IDE

Refer to activate and index installed packages section with the illustrated process using poetry examples for PyCharm, IntelliJ, and VSCode.

Everything is almost the same except you don't need to find a poetry cache folder via command line to find a path to python.exe file because the env folder is already in your project directory that was created earlier above.

Deactivate virtual environment when done:

(env) $ deactivate
$

An example process of using Poetry

Install poetry:

$ pip install poetry

A quick look at how you can install site-package (poetry) for a specific Python version:

# For Windows:
$ py -3.6 -m pip install poetry

# For Linux:
$ python3.6 -m pip install poetry

Create (initialize) poetry inside current package/project directory:

$ poetry init

The init command will ‘initialize’ an existing directory and create a pyproject.toml which will manage your project and its dependencies:

# pyproject.toml file

[tool.poetry]
name = "virtual environments"
version = "0.1.0"
description = ""
authors = ["Dimitry Zub <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.9"
google-search-results = "^2.4.0"
# other site-packages will appear here..

[tool.poetry.dev-dependencies]
# development dependencies will appear here..

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

What the heck is pyproject.toml?

In short, pyproject.toml is the new unified Python project settings file that contains build system requirements and information, which are used by pip to build the package/project, and it is almost a replacement for setup.py.

Before pyproject.toml creation, $ poetry init will interactively ask you to fill the fields about your package/project:

--name: Name of the package/package.
--description: Description of the package.
--author: Author of the package.
--python Compatible Python versions.
--dependency: Package to require with a version constraint. Should be in format package:1.0.0(version).
--dev-dependency: Development requirements, see --require.

Add dependencies to your package/project:

$ poetry add google-search-results

...
Creating virtualenv PROJECT-9SrbZw5z-py3.9 in C:\Users\USER\AppData\Local\pypoetry\Cache\virtualenvs

Using version ^2.4.0 for google-search-results

Updating dependencies
Resolving dependencies...

Writing lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing google-search-results (2.26.0)

The add command adds dependencies to pyproject.toml and poetry.lock, and installs them.
Creating virtualenv will create a virtual environment with the showed path. Environment creation will be done once.
Writing lock file will write dependencies to poetry.lock file.

poetry.lock prevents from automatically getting the latest versions of your dependencies.

You can explicitly write lock command to lock dependencies listed in the pyproject.toml

Add specific version:

# multiple ways
# double quotes ("foo") for Windows CMD 

$ poetry add google-search-results@^2.1.0

$ poetry add 'google-search-results>=1.8.5'

$ poetry add 'google-search-results==1.8.5'

$ poetry add google-search-results@latest

If you specify a constraint (@ or >=), the dependency will be updated by using the specified constraint.

Otherwise, if you try to add a package that is already present, you will get an error.

(optional) Install from existing project/package dependencies.

If you're using an already created project that has either poetry.lock or pyproject.toml files, you can install those dependencies to the virtual environment:

$ poetry install

The install command read pyproject.toml or poetry.lock file and installs all listed dependencies.

If there's a poetry.lock file:

Poetry uses the exact versions listed in poetry.lock.

If there is no poetry.lock file:

Poetry will resolves all dependencies from the pyproject.toml file and downloads the latest version of their files.

(optional) To not install development dependencies, use --no-dev argument:

$ poetry install --no-dev

Use added site packages inside IDE

If using poetry, find a location of the initialized environment first via config --list command. Look for virtualenvs.path in the output:

$ poetry config --list

cache-dir = "C:\\Users\\USER\\AppData\\Local\\pypoetry\\Cache"
experimental.new-installer = true
installer.parallel = true
virtualenvs.create = true
virtualenvs.in-project = null
👉virtualenvs.path = "{cache-dir}\\virtualenvs"  👉👉👉# C:\Users\USER\AppData\Local\pypoetry\Cache\virtualenvs

A few more steps to do:

Go to the virtualenvs.path folder and open created environment folder (in my case its: PROJECT-9SrbZw5z-py3.9).

Go to Scripts (Windows) or bin (Linux) folder, copy the full path and add python.exe at the end of the path:

C:\Users\USER\AppData\Local\pypoetry\Cache\virtualenvs\PROJECT-9SrbZw5z-py3.9\Scripts\python.exe

virtualenvs.path is needed to find a path to python.exe inside created virtual environment which will let JetBrains or VSCode to index installed site-packages.

If using virtualenv, go to env\Scripts\python.exe folder in your project and copy the full path to the python.exe file and enter it as a System Interpreter inside IDE.

Activate and index installed packages

Currently, if you run the script inside IDE, it will look at the globally installed package (serpapi, for example) and will throw an error because globally there's no such library installed (it won't throw an error if it's installed):

Traceback (most recent call last):
  File "C:\Users\USER\PyCharmProjects\PROJECT\environment.py", line 1, in <module>
    from serpapi import GoogleSearch
ModuleNotFoundError: No module named 'serpapi'

VSCode version

To fix this in VSCode we need to select a virtual environment Python Interpreter and set it as a System Interpreter.

Open command palette CTRL+SHIFT+P and type: Python: System Interpreter (Python extension should be installed).

Both for virtualenv and poetry, VSCode should automatically detect a proper python.exe file from the virtual environment.

If you don't see a proper path to python.exe from your virtual environment then you need to locate and enter it.

Now you can run your Python scripts from the virtual environment either by the command line or using VSCode Code Runner extension.

PyCharm version

To fix this in PyCharm we need to add the path to python.exe from the virtualenv folder and set it as a PyCharm System Interpreter which will index all site-packages from the virtual environment:

IntelliJ IDEA version

To fix this in IntelliJ IDEA we need to add the path to python.exe from the virtualenv folder as well and set it as a PyCharm System Interpreter with a few additional tweaks which will index all site-packages from the virtual environment:

Deactivate virtual environment when done

To deactivate virtual environment in order to use system Python both in PyCharm, IntelliJ IDEA and VSCode you need to set Python System Interpreter back to the default one without virtualenv prefix for example: "Python 3.9 virtualenv.." ==> "Python 3.9", a reverse process of what's being shown above.

Conclusion

Different project - different environment

In short, it is better to use a virtual environment if you need to work with several projects at the same time which:

use different library versions.

use different Python versions.

Installing globally will become a mess

Installing globally different versions of the same library for different projects will quickly turn into a mess, there will be no order, or if there will be a need to install different versions of Python it will turn into a mess of all messes:

Links

Virtualenv

Poetry

SerpApi libraries

Black&White illustration by xkcd.com

Acknowledgments

A big thanks to these guys for helping out with the feedback about illustrations:

Outro

If you have anything to share, any questions or suggestions to this blog post, feel free to drop a comment below or reach out via Twitter at @dimitryzub, or @serp_api.

Yours,
Dimitry, and the rest of SerpApi Team.

Join us on Reddit | Twitter | YouTube

Add a Feature Request💫 or a Bug🐞