Software Engineering

Setting Up Phyton Local Environment with Poetry, Pyenv and Pipx

  • October 21st 2024
  • 12 min read

⚠️ Don’t just copy and paste — understand each step!

Table of content

Context and motivation

During recent years, I’ve encountered more and more projects implemented with different, incompatible versions of Python, such as Python 3.12 and 3.6, within the same company and teams. In some cases, these versions were even deployed on the same servers, sharing environments.

With this post, I aim to establish clear guidelines for configuring development environments that offer the flexibility to seamlessly switch between Python versions as needed and manage different projects in isolation.

Environment isolation and flexibility

To ensure a consistent development experience across various operating systems, it’s important to keep our local development environment as agnostic as possible to the OS being used (Linux, Mac or Windows).

Additionally, because Python is widely used in systems and applications, we need to ensure that our development environment is isolated from the Python version managed by the operating system. This prevents potential conflicts, such as system dependencies or issues with other Python projects or applications using Python.

It’s not hard to imagine the following scenario (believe me): a system running an outdated Python 3.6 version. You need to upgrade to Python 3.12 to develop new applications. However, upgrading Python system-wide could break existing tools that depend on the older version, like the Cloudera CDP Command Line Interface.

This is just one example, but you can easily imagine other cases, such as shared Jenkins nodes for CI/CD pipelines, where conflicting Python versions could create significant problems for multiple teams.

Each project should have its own Python version, set of dependencies, and tooling. This isolation ensures that no project shares resources or interferes with system-level Python installations or other projects you’re working on.

There are different ways to achieve this. In this guide, we will use a combination of tools to try to maintain a robust and flexible environment.

Specifically, we will leverage Poetry for Python packaging and dependency management, ensuring that each project remains isolated and easy to manage. The installation of Poetry will also involve pipx, which in turn will lead us to pyenv. Ultimately, our environment will rely on a combination of these three tools: Poetry, Pyenv, and Pipx

Why Poetry? Poetry is both easy to use and powerful. It comes with good documentation and is widely adopted within the Python community, making it a reliable choice for managing dependencies and packaging Python projects.

Prerequisites

Before setting up the environment, you need to have at least Python 3.12 installed. The installation process is outside the scope of this tutorial, as it varies depending on the operating system (and it’s quite straightforward).

For the purposes of this document, we will assume that you are using Ubuntu 24.10. However, after installing Python and PyEnv, the process should be nearly the same across all operating systems. If you’re using Windows, my recommendation is to use WSL2 with Ubuntu for a more seamless development experience.

We also assume that you have other essential tools, like Git, installed. These tools are often required by scripts used in the installation process, so make sure they’re available on your system.

Additionally, you should create a directory that will contain your project’s source code at the location $ROOT_PROJECT.

🗒️ $ROOT_PROJECT is used in this documentation as a placeholder. Replace it with the full path to your project directory, or use an environment variable that holds the full path to the project.

Pyenv

Two key requirements for isolation are:

  • We need to isolate the Python interpreter used for development from the one used by the OS.
  • We need to be able to switch between Python versions as the project requires.

We can achieve this using tools like pyenv and virtualenv:

  • While virtualenv is a lightweight and faster option, it’s tied to the Python version installed on your system. This is sufficient in many cases, but when working on projects that require different versions (whether newer or legacy) virtualenv alone falls short.
  • On the other hand, pyenv allows us to easily manage multiple Python installations, providing the flexibility to switch between different Python versions for different projects.

Due to the flexibility Pyenv offers in managing Python versions, we will proceed with using Pyenv for our environment setup.

Installation steps

To install Pyenv, follow the steps outlined in the official documentation:

curl https://pyenv.run | bash

After executing the Pyenv installation script:

  • Don’t forget to add Pyenv to the PATH, as mentioned in the terminal instructions.
  • Optionally, if you plan to install Python from the source code, ensure you install the necessary build tools (e.g., build-essential on Ubuntu or the equivalent for your operating system).

Usage

As an example, let’s install Python 3.12 and Python 3.13 and then start one shell with each version of Python installed. From the $ROOT_PROJECT folder:

cd $ROOT_PROJECT

python3 --version
# Output:
# Python 3.12.3

pyenv versions
# Output:
# * system (set by /home/angelcc/.pyenv/version)
 
pyenv install 3.12.7
# Output:
# Downloading Python-3.12.7.tar.xz...
# -> https://www.python.org/ftp/python/3.12.7/Python-3.12.7.tar.xz
# Installing Python-3.12.7...
# Installed Python-3.12.7 to /home/angelcc/.pyenv/versions/3.12.7

pyenv versions
# Output:
# * system (set by /home/angelcc/.pyenv/version)
#   3.12.7

pyenv install 3.13
# Output:
# Downloading Python-3.13.0.tar.xz...
# -> https://www.python.org/ftp/python/3.13.0/Python-3.13.0.tar.xz
# Installing Python-3.13.0...
# Installed Python-3.13.0 to /home/angelcc/.pyenv/versions/3.13.0

pyenv versions
# Output:
# * system (set by /home/angelcc/.pyenv/version)
#   3.12.7
#   3.13.0 

pyenv shell 3.13.0
python --version
# Output:
# Python 3.13.0
 
pyenv shell 3.12.7 
python --version  
# Output:
# Python 3.12.7

🗒️ Our projects should have a .python-version file with information about the Python version that Pyenv will use, so pyenv local will be enough to enable it.

In the next section, we will learn how to use it.

Pipx

Pipx is another layer of abstraction. Where Pyenv# is used to manage and isolate Python installations, Pipx provides a way to run Python applications in isolated environments. Is in this environment where we’re going to install our project tools. One of these tools is going to be Poetry.

Why use Pipx? We’re using Pipx to install Poetry because it is the recommended method in Poetry’s official documentation.

To install it, let’s select the Python installation to use. It’s possible to use pyenv shell 3.12.7 to do it, but we want to persist it our project. So let’s move to our project and set version 3.12.7 as the default one.

cd $ROOT_PROJECT
pyenv local 3.12.7
ll -a
# Output:
# total 12K
# drwxrwxr-x 2 angelcc angelcc 4.0K Oct 19 15:06 .
# drwxrwxr-x 3 angelcc angelcc 4.0K Oct 19 14:21 ..
# -rw-rw-r-- 1 angelcc angelcc    7 Oct 19 15:06 .python-version

cat .python-version
# Output: 
# 3.12.7

python --version
# Output:
# Python 3.12.7

As you can see, a file called .python-version has been created in the project folder. This will be the version used if you run python from this folder. You don’t need to enable it again.

Now, let’s install pipx.

pip install --user pipx
#  Output:
#  Collecting pipx
#    Downloading pipx-1.7.1-py3-none-any.whl.metadata (18 kB)
#  Collecting argcomplete>=1.9.4 (from pipx)
#    Downloading argcomplete-3.5.1-py3-none-any.whl.metadata (16 kB)
#  Collecting packaging>=20 (from pipx)
#    Using cached packaging-24.1-py3-none-any.whl.metadata (3.2 kB)
#  Collecting platformdirs>=2.1 (from pipx)
#    Using cached platformdirs-4.3.6-py3-none-any.whl.metadata (11 kB)
#  Collecting userpath!=1.9,>=1.6 (from pipx)
#    Downloading userpath-1.9.2-py3-none-any.whl.metadata (3.0 kB)
#  Collecting click (from userpath!=1.9,>=1.6->pipx)
#    Downloading click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
#  Downloading pipx-1.7.1-py3-none-any.whl (78 kB)
#  Downloading argcomplete-3.5.1-py3-none-any.whl (43 kB)
#  Using cached packaging-24.1-py3-none-any.whl (53 kB)
#  Using cached platformdirs-4.3.6-py3-none-any.whl (18 kB)
#  Downloading userpath-1.9.2-py3-none-any.whl (9.1 kB)
#  Downloading click-8.1.7-py3-none-any.whl (97 kB)
#  Installing collected packages: platformdirs, packaging, click, argcomplete, userpath, pipx
#  Successfully installed argcomplete-3.5.1 click-8.1.7 packaging-24.1 pipx-1.7.1 platformdirs-4.3.6 userpath-1.9.2

pipx ensurepath

At this point, we have Pipx installed.

Important note about pipx installation

Although pipx is designed to install and run Python applications in isolated environments, the installation process of pipx itself is not isolated. Once installed, Pipx is tied to the Python environment it was initially installed in, typically located at $HOME/.local/bin/pipx (or your system’s equivalent).

☢️ From this point onward, regardless of the Python version you enable or switch to using tools like pyenv, you will always be using the same instance of pipx.

Unfortunately, there is no way to install a separate pipx for each Python version, which somewhat contradicts the isolation it offers for other Python applications.

One side effect of this design is that if you use sudo apt install pipx, it will likely install an outdated version of pipx. Why? It seems that upgrading system-installed versions requires ensuring that none of the packages depending on it (like those used by Ubuntu) break in the process. A similar issue occurred with Python 3.6 in Ubuntu, where the version wasn’t updated for a long time due to compatibility concerns.

🤷 If you’re going to work with Python, be prepared to encounter more inconsistencies like this. Welcome to the Python ecosystem.

As a workaround, you can specify the Python version you want to use when installing packages by doing it in a shell. As an example:

pyenv shell 3.12.7
pipx install <package-name>

Installing Poetry.

For packaging and dependency management, we will use Poetry. While we considered other options like UV, it’s still in its early stages and lacks strong integration with widely-used Python developer tools and IDEs, such as PyCharm.

To install Poetry, run:

pyenv shell 3.12.7
pipx install poetry

Remember to enable tab completion for your shell for an improved experience.

Usage

The detailed usage of Poetry is beyond the scope of this document. However, you can find all the necessary documentation on the official Poetry site.

Below is a list of common commands you will use when working with this environment:

Poetry commands

  • poetry init: Initialize a new project with a pyproject.toml file.
  • poetry add <package>: Add a dependency to your project.
  • poetry install: Install all dependencies listed in the pyproject.toml file.
  • poetry update: Update all dependencies to their latest versions.
  • poetry run <command>: Run a command inside the project’s virtual environment.
  • poetry shell: Start a new shell within the virtual environment.
  • poetry build: Build the project for distribution.
  • poetry publish: Publish the project to PyPI.
  • poetry remove <package>: Remove a dependency from your project.
  • poetry env info: List information about the environment enable.

Pyenv commands

  • pyenv install <version>: Install a specific Python version.
  • pyenv versions: List all installed Python versions.
  • pyenv shell <version>: Set the Python version for the current shell session.
  • pyenv local <version>: Set the Python version for the current directory (creates a .python-version file).
  • pyenv global <version>: Set the global Python version across your system.
  • pyenv uninstall <version>: Uninstall a specific Python version.

Pipx commands

  • pipx install <package>: Install a Python package in an isolated environment.
  • pipx list: List all installed packages managed by pipx.
  • pipx run <package>: Run a package directly without installing it globally.
  • pipx upgrade <package>: Upgrade a specific installed package.
  • pipx uninstall <package>: Uninstall a specific package.
  • pipx reinstall-all: Reinstall all pipx-managed packages (useful when switching Python versions but needs a complex file to list the packages, again, tied to the Python version).

Ex. Creating a new project

# Create th folder that will contains the project.
mkdir projectX
cd projectX

# Check that the desired Python version is installed.
pyenv install 3.12.7

# Set this version as the one to use in the project.
# This will create a file .python-version
pyenv local 3.12.7

# Install poetry if it's not there.
# If it's there, it will use the previous one maybe installed in another Python installation.
pipx install poetry

# Create the project using and a virtual environment inside the project.
poetry init
poetry config virtualenvs.in-project true
poetry install
poetry shell

# Check where the virtual environment has been created.
poetry env info

# Add packages. Ex.
poetry add --group=dev flake8 
poetry add requests

Drawbacks of using Poetry

While Poetry provides a streamlined approach for managing dependencies and packaging Python projects, there are some limitations to be aware of:

  • Partial Isolation: Although Poetry simplifies environment management, it’s not entirely isolated from the Python version it was installed with. This means that while it manages dependencies well within projects, switching between different Python versions still requires additional tools like pyenv.

  • Compatibility with Certain Libraries: Despite Poetry’s widespread adoption, not all libraries or frameworks are fully compatible or have well-documented integration steps. For instance, popular libraries like PyTorch or TensorFlow may require extra configuration or workarounds when using Poetry, which can sometimes complicate the setup process.

  • Learning Curve and Integration: For developers familiar with more traditional tools like pip and virtualenv, there may be a learning curve when adapting to Poetry’s workflow and command structure. Additionally, integrating Poetry into established projects or CI/CD pipelines may require extra effort if the project wasn’t initially built with Poetry in mind.

Conclusion

While using Poetry in combination with Pipx and Pyenv provides a workable solution for managing Python environments and dependencies, it can sometimes feel more like a workaround than a fully comprehensive approach.

The challenge of efficiently managing multiple Python versions and environments across different projects remains, and the isolation provided by these tools isn’t always as seamless or complete as desired.

In any case, Poetry provides a standardized way to manage dependencies and run other common tasks that are often implemented differently in each project through custom scripting.

In a future post, we will explore how to use UV as a potential replacement for this complex mix of tooling.

Attributions