When building Python projects, we inevitably need to work with third-party packages. Most developers start with the familiar requirements.txt approach, but this method has its pitfalls. It’s all too common to install a new package and forget to update the requirements file. Even worse, when you look at your requirements.txt, it’s hard to distinguish between packages you actually need and those that were installed as dependencies of other packages. This makes it tricky to clean up your project when you want to remove unnecessary packages.

To address these issues, I will show you how to use poetry for more efficient project dependency management.

You can see the files in my repository.

Initializing Poetry

You can initialize a new Poetry project in a folder with a virtual environment by executing the following command. During this process, you’ll be prompted to provide some project information. Alternatively, if you wish to skip the interactive process and use default values, you can use poetry init -n. Let’s proceed with the interactive setup

mkdir new_python_project && cd new_python_project
python -m venv .venv # Create a new virtual environment named .venv
. ./.venv/bin/activate # For Linux and MacOS Users to activate the virtual environment 
pip install poetry # Install Poetry in the virtual environment
poetry init

Package name:  poetry-example
Version: 0.1.0
Description: An example project for poetry
Author :  <Your Name>
License: MIT
Compatible Python versions: <Use the default value>

Would you like to define your main dependencies interactively? no
Would you like to define your development dependencies interactively? no
Do you confirm generation? yes

You will see a file pyproject.toml in the project folder. This file is the core file containing the project metadata and dependencies.

Installing Dependencies

You can start to install dependencies via poetry. For example, you can install pandas with the following command.

poetry add pandas

In the main dependencies section of the pyproject.toml file, you’ll see the following lines. These lines indicate that pandas is a required package, and there’s a version constraint specified. In this example, the version should be equal to or higher than 2.1.1. You will also see a poetry.lock file in the project folder. This file contains the exact versions of the packages installed.

[tool.poetry.dependencies]
...
pandas = "^2.1.1"

You can also install a package with a specific version. For instance, you can install scikit-learn version 1.2.0 using the following command. The result of this installation will be reflected in the pyproject.toml file.

poetry add [email protected]

[tool.poetry.dependencies]
...
pandas = "^2.1.1"
scikit-learn = "1.2.0"

To update a package to its latest version, you can use poetry update <Package Name>.

poetry update scikit-learn

However, if you run the command as shown in this example, you won’t observe any changes because we’ve specified the version as 1.2.0. To enable the package to receive updates, you should modify the version constraint to "^1.2.0". After making this adjustment, you can rerun the update command, and the package should then be updated

...
scikit-learn = "^1.2.0"

Eventually, you can use poetry remove <Package Name> to remove a package.

poetry remove scikit-learn

Organizing Dependencies in Groups

Poetry provides the flexibility to categorize dependencies into groups. By defining these groups, you can install packages for various purposes. For instance, you can add pytest to the test dependency group and ipykernel to the dev group using the following command. The dependencies will then appear in their respective groups within the pyproject.toml file.

poetry add -G test pytest
poetry add -G dev ipykernel

[tool.poetry.group.test.dependencies]
pytest = "^7.4.2"

[tool.poetry.group.dev.dependencies]
ipykernel = "^6.25.2"

Installing Dependencies from pyproject.toml and poetry.lock

Imagine you have a new machine and need to install the project dependencies. In such a scenario, you can utilize the following command to install the dependencies based on the information in the pyproject.toml and poetry.lock files.

# Before running this command, you can remove the old virtual environment 
# and create a new one to see how it works
rm -rf .venv
python -m venv .venv
. .venv/bin/activate # For Linux and MacOS Users
pip install poetry

poetry install

By default, all non-optional groups will be installed by the command poetry install. You can designate a group as optional in the pyproject.toml as the following example.

[tool.poetry.group.test]
optional = true

[tool.poetry.group.test.dependencies]
pytest = "^7.4.2"

It’s important to note that if the poetry.lock file is absent, Poetry will resolve and install all dependencies using the latest versions specified in the pyproject.toml file. When the poetry.lock file is present, Poetry will install the exact versions as defined in that file.

Furthermore, you have the option to exclude one or more groups using the –without flag. In this example, only the main dependencies will be installed. This feature is particularly valuable when building a Docker image to install only the necessary dependencies, thereby optimizing build times.

poetry install --without test,dev

Learn More

• Managing dependencies. A comprehensive guide to managing group dependencies with Poetry.