Skip to content

Boring Python: dependency management

Originally published: May 13, 2022    Updated: September 1, 2026    Categories: Django, Python

This post has been significantly updated since it was originally written. The most recent update was in September 2026; a summary of what changed is at the end.

This is the first in hopefully a series of posts I intend to write about how to build/manage/deploy/etc. Python applications in as boring a way as possible.

So before I go any further, I want to be absolutely clear on what I mean by “boring”: I don’t mean “reliable” or “bug-free” or “no incidents”. While there is some overlap, and some of the things I’ll be recommending can help to reduce bugs, I also want to be clear: there will be bugs. There will be incidents where a feature or maybe an entire service is down. “Boring”, to me, is about the sources of those incidents. It’s difficult enough to manage your own code and the bugs and other problems that will inevitably pop up in it from time to time; you don’t want to compound that by having bugs or other surprises coming from the tools and processes you use to build, manage, and deploy it. So when I call something “boring”, I mean it’s unlikely to add another source of bugs and nasty surprises that cause a pager to go off at 2AM; the pager will, of course, eventually go off at 2AM, but when it does you’ll be able to feel reasonably confident (if that’s the right word for such a situation) that the source of it was something in your own code that you can diagnose and fix.

And so I’m planning several posts exploring different aspects of making Python development boring (in this sense). But for this first installment, I’ll be talking about one of the internet’s favorite topics: managing dependencies.

If you’re just interested in the end result, the process I recommend is pretty simple and can be found in the section titled “Putting it all together”. The rest of this post exists to explain, in as much detail as I can manage, all the things going on behind those simple-looking recommendations, and why I make those specific recommendations.

Python packaging: a brief overview

I’ve written about this before, but to make sure it’s clear I want to do a quick refresher.

As I see it, there are three basic aspects to “packaging”, and one of the difficult things about “Python packaging” discussions is lack of clarity around which are being discussed. They are:

  1. Given some working code, define and produce from it a distributable artifact.
  2. Given that someone has produced a distributable artifact, use it to cause the corresponding code to appear, in working form, somewhere else.
  3. Given the existence of potentially many separate projects or versions of projects each with their own independent and potentially-conflicting sets of dependencies, make it possible to work on and run more than one at a time, on the same system.

Most of the actual remaining complexity in “Python packaging” today is concentrated in item (1), and even then only really kicks in if you’re packaging extensions written in non-Python languages.

A lot of the perceived complexity of “Python packaging” is due to its long history, evolving expectations over the course of that history, and the resulting pile of strata of different things people have come up with to try to suit the expectations of different eras.

Much of the public grumbling and complaining about “Python packaging” these days is due to the lack of a single, standard first-party, all-in-one high-level tool to handle all aspects of packaging, and the proliferation of third-party tools to try to fill that niche.

Finally, most of the meaningful work in Python packaging, for quite a while now, has been the unglamorous, underappreciated slog of people on mailing lists and discussion forums patiently working out what Python packaging really should look like and thoroughly specifying/standardizing it, but with the added thrill of all that having to occur in a way that doesn’t break things for the vast number of people relying on Python and its huge package ecosystem for their day-to-day work.

To the extent that we now have a bunch of shiny fancy flashy third-party all-in-one packaging tools, it’s because of that heroic unsung work that’s been quietly going on for years. The fancy new tools get to be so fancy and flashy in large part because modern Python packaging has such strong foundations (and also because, unlike the pre-existing tools, the new ones don’t have to maintain compatibility with literal decades of historical features and configuration options, all of which have been relied on, at some point, by someone).

If I went into detail on everything I’d wind up with several posts’ worth of material, so let me just pick out a few things that I think are the most crucial underpinnings of modern Python packaging:

Boring Python packaging: a philosophy

When I wrote the first version of this post, four years ago, I focused on defining a workflow that relied as much as possible on standard first-party (as in, included with Python itself) tooling. I recommended exactly one third-party tool, and only as an optional convenience; the functionality it provided could also be obtained from chaining together the right invocations of first-party tooling.

That focus on first-party tooling was deliberate. As I wrote at the time:

[The] core default tools I’ve mentioned have all been around and stable for a long time, in software terms, at least — they’re all over a decade old, and their end-user interfaces evolve incredibly slowly, when they evolve at all. They’re reliable. They’re well-understood. That’s exactly what I want and exactly what I recommend.

This is in line with what I said at the start of this post about wanting things to be boring. And my stance has not changed, but advances in the foundational standards of Python packaging do let me be a bit more flexible. So although I’m still going to be focused on first-party tools I’ll now point out a couple of places where I think it’s safe to use some of the hip and trendy new third-party tools if you want the added conveniences they offer.

Also, I should note here that my use case for Python is deploying networked (primarily web) services, on servers. If you primarily use Python for, say, machine learning or data science, you are likely to already be a happy user of a completely different world of tools optimized for those use cases (in particular, for the much greater amount of non-Python code wrapped by/accessed through Python in those fields and for their different patterns of code reuse and sharing). Please continue using them! They suit your use case really well, and replacing them with my preferred tools and workflow would probably be a regression for you.

Now, let’s get started.

Lock it up

The standard Python package manager is pip. And my eventual goal is to invoke pip install to, well, do what it says: install a bunch of Python packages. But how to tell pip which packages to install? There are a couple of options:

Historically, the second option was referred to as a “requirements file”, and by convention was generally named requirements.txt. Now that PEP 751 has defined a standard Python lock file format, and now that pip supports producing and consuming that standard lock file format, that’s the format I recommend.

This prompts a question, though: since dependencies could already be specified as part of the metadata in pyproject.toml, why did we need another file format to record them? By way of an answer, consider a web application written using the Django web framework. You might think it’s simple enough to just declare Django as a dependency in your pyproject.toml file, like so:

[project]
dependencies = ["Django"]

But if I create a new virtual environment and run pip install Django inside it, three packages get installed:

$ pip install Django
Collecting Django
  Using cached django-6.1-py3-none-any.whl.metadata (3.9 kB)
Collecting asgiref>=3.9.1 (from Django)
  Using cached asgiref-3.12.1-py3-none-any.whl.metadata (9.4 kB)
Collecting sqlparse>=0.5.0 (from Django)
  Using cached sqlparse-0.6.0-py3-none-any.whl.metadata (6.0 kB)
Using cached django-6.1-py3-none-any.whl (8.4 MB)
Using cached asgiref-3.12.1-py3-none-any.whl (25 kB)
Using cached sqlparse-0.6.0-py3-none-any.whl (50 kB)
Installing collected packages: sqlparse, asgiref, Django
Successfully installed Django-6.1 asgiref-3.12.1 sqlparse-0.6.0

And which versions of those packages I get will depend on when I run pip install. Right now I get Django 6.1, but Django does frequent bug fix releases, so if I re-run that pip install in the future I’ll get Django 6.1.1, and further in the future I’ll get Django 6.1.2, and so on. If I test my application locally with the set of packages I installed today, and then deploy it to a server later using pip install to fetch dependencies, the deployment will very likely get a different set of packages than I used during development, which is a potential source of problems.

So the key difference is that the pyproject.toml file typically specifies a list of direct dependencies, often with broad version constraints or none at all, while a lock file specifies an entire environment, with all direct and transitive dependencies resolved and pinned to exact versions along with the expected checksums of the packages, to be reproduced as exactly as possible.

There are two main approaches to generating a lock file:

Managing your local environment and producing your lock file is one of the areas where I think it’s generally OK to use a third-party tool. If something goes wrong with it, it might temporarily mess with your ability to get work done on your own computer, but that won’t break your actual deployed application.

The hip and trendy third-party all-in-one tool these days is uv, which has its own tool-specific lock file format (uv.lock), but also has a uv export command which can export to the standard pylock.toml format.

My own personal preference, and the tool I use on my own projects, is PDM, which has a similar feature set to uv and can export to pylock.toml from its own lock format, but also can just manage a project directly in a pylock.toml file.

As long as the tool you choose is one you and your team are all happy with, I think you can pick anything that can produce the standard lock file format and be OK. Also, you should probably automate the process of exporting the lock file(s), to ensure developers don’t have to remember how to manually do it. I personally like to include a Makefile in most projects, with a target specifically to refresh the lock file(s) (make lock, for example), but the idea is portable to a lot of task-automation tools, so it’s easy to make it work with whatever your team likes to use.

I also want to point out here that if you’re deploying services written in Python, you can build them as distributable packages and then install those packages to automatically pull their dependencies, rather than using a lock file. I just don’t recommend this, for a few reasons:

Getting testy

In addition to direct dependencies to run an application, it’s fairly common to have additional dependencies needed to run a test suite, linters, or other development and quality-control tasks. But there’s no need for a production deployment to have all those extra dependencies installed, and in general the less stuff you put in your production environment the fewer things you have that can go wrong, so I generally like to install those only when the test suite (or other task requiring extra dependencies) is actually going to be run.

The ideal solution for this is dependency groups, which were first defined for library-style package dependencies but are also part of the lock file specification. Unfortunately, as far as I can tell the current release of pip (26.2.1 as I’m writing this) does not support selecting dependency groups when installing from a lock file.

So for the moment (at least until pip gains support for selecting dependency groups from a lock file), my recommendation for handling different sets of dependencies is just to have multiple lock files. The exact syntax for this will vary depending on what packaging frontend tool you decide to use. But suppose you have a FastAPI application and want to use pytest to test it. Here’s an example of how that would look with uv:

uv add fastapi       # Regular dependency
uv add --dev pytest  # Used only for tests
uv export --no-dev --format pylock.toml -o pylock.deploy.toml
uv export --format pylock.toml -o pylock.tests.toml

Or with PDM:

pdm add fastapi
pdm add --dev pytest
pdm export --prod --format pylock -o pylock.deploy.toml
pdm export --dev --format pylock -o pylock.tests.toml

In either case, the pylock.deploy.toml file will contain only the dependencies for FastAPI, and is what you’d use for deployment. Meanwhile, pylock.tests.toml would also contain pytest, and is what you’d use for a development or test/CI environment. You can split this up further if you want—I’ve done finer-grained splits of dependencies before—but at the very least I think you should be keeping test-only dependencies separate from the rest.

Using the right invocations

Before putting all this together, it’s worth covering one more detail: how to invoke the tools. This may seem a bit silly, since they all provide executable entry points. Just run pip install, right?

But there’s a potential issue here: in a moment I’m going to recommend creating a Python virtual environment, which opens up the possibility of multiple Python environments coexisting on the same machine. This is certainly a useful feature, but it comes with a new concern, which is how to ensure you’re using and running things in the environment you expect to be using.

For example, one way to run into trouble with multiple Python environments is to have one environment’s package directory be first on your $PYTHONPATH (which controls Python import locations) while a different one’s bin/ directory is first on your general $PATH (which is where executable scripts will be found). If you just run pip install you’ll get the second one’s instance of pip, which may not be at all what you want.

This is why the official Python packaging guides, and official documentation for tools like pip, always use a different approach: they tell you to run python -m pip instead of pip, and python -m venv instead of a standalone script like virtualenv. The -m flag allows a Python module to be run directly like a script (as long as it’s been written to provide an entry point for this, which pip and venv both have), and can prevent a lot of potential hard-to-debug issues that can accidentally result from things like manually hacking around with paths, by ensuring you’re getting the version of pip or venv that actually goes with the Python environment you invoked.

And to guarantee that you get the Python environment you actually want, you can specify the full path. For example, many base Python container images will put the Python interpreter in /usr/local/bin, so invoking /usr/local/bin/python instead of just python ensures you get that interpreter.

There’s also one more thing I like to do that isn’t (currently) in some of the popular guides, and that’s invoking Python with the -I flag. This runs Python in “isolated” mode, which removes some of the automatic implicit directories from the import path and also ignores environment variables like PYTHONPATH. Once again, this reduces the number of things that can go wrong (for example, with -I the current working directory won’t be implicitly added to the import path, so you can’t accidentally depend on it being importable). So whenever you can invoke Python in isolated mode, I generally recommend that you do.

Putting it all together

That was a lot of explanation for what ends up being, ultimately, a pretty simple process to actually use. So now let’s finally take a look at it.

First things first: always work in, and always deploy in, a virtual environment. Even if you think you don’t need one. In fact, especially if you don’t think you need one. This may seem like strange advice if you’re already using a container or other virtual machine, since you’re probably thinking that provides all the isolation you’ll need. But virtual environments don’t cost you anything to create, and if you ever do end up with multiple Python interpreters—which is easy to accidentally do, if you use a base system with a purpose-built Python and then install a system package that turns out to depend on the distro’s own Python, for example—using one from the start will help to save you from potentially having a pager go off one night when suddenly the wrong Python is being invoked.

Virtual environments also provide a useful “Python environment” artifact that can be copied between stages of a container build, and many tools automatically recognize and can work with them. And ever since PEP 668 started to be adopted by operating-system vendors, many “system” Python installations will require you to use a virtual environment in order to install packages with pip (and only allow you to interact with the “system” Python environment through the system’s own package manager). So use a virtual environment, even if you’re working in a container or other VM.

When you’re working in a local directory on your own computer, you can invoke the correct Python version with -m venv to create a virtual environment and install things into it, but again this is an area where third-party package tools can be useful, because they’ll manage this for you automatically. If you do create your local virtual environment manually, you should almost certainly put it inside your project in a subdirectory named .venv, because that’s already the common unofficial convention supported by a lot of IDEs and other tools, and likely to become the official standard convention once PEP 832 finalizes.

In a container, I think the choice that’s most in line with Linux filesystem hierarchy standards is to put a Python virtual environment somewhere under /opt. Generally I like to create an /opt/venvs in a base “build” stage, which can populate multiple virtual environments with different package sets to be copied into later stages for tasks like CI or deployment.

So, taking as an example current Debian stable (as I write this, Debian 13 “trixie”) and the most recent Python (as I write this, Python 3.14), let’s see what this actually looks like. This example assumes you’re producing standard Python lock files as described above, and naming them according to their purposes as I did (otherwise, adjust the names before using this snippet):

# syntax=docker/dockerfile:1

ARG DISTRO="slim-trixie"
ARG PYTHON_VERSION="3.14"

FROM python:${PYTHON_VERSION}-${DISTRO} AS dependencies

RUN <<END
mkdir -p /opt/venvs/deploy
mkdir -p /opt/venvs/tests
# This is a cache directory for package downloads, to speed up
# rebuilds when the package set doesn't change (or doesn't
# change very much).
mkdir -p /var/cache/pip
/usr/local/bin/python -Im venv /opt/venvs/deploy
/usr/local/bin/python -Im venv /opt/venvs/tests
END

COPY pylock.deploy.toml  /opt/venvs/deploy/
COPY pylock.tests.toml /opt/venvs/tests/

The next step is to ensure pip is present at the latest version in each virtual environment, and invoke it to install packages.

At this point you might be wondering: if I was willing to recommend using a third-party packaging frontend earlier for local development use and producing lock files, why am I insisting on pip here? And the answer is that this is one of the places where I think using boring standard default tools really matters. As I said above, if something goes wrong with your fancy third-party package tool in a local development setup, it just causes a problem for that local development setup. If it goes wrong in your production build/deploy process, it breaks all your builds and deployments until you resolve the issue. I want to avoid that, so I stick to pip here, because even if it’s not as fancy as some of the newer third-party tools, in my experience it’s much less likely to be a source of unexpected issues than the newer third-party alternatives.

Also, I’m not going to go into a full explanation of an ideal Python application Dockerfile here, but the example below does at least use a Docker cache mount to store the downloaded packages, so that subsequent rebuilds only have to download new or changed packages, and can pull everything else from cache.

The pip invocation adds a few flags, mostly to ensure maximum safety and reproducibility:

If you’re using GitHub Actions as your CI/CD, Python core developer Brett Cannon has written a reusable action which automatically invokes pip with these flags.

But continuing with the example of a Dockerfile containing a “dependencies” stage, here’s what it looks like:

# Turn off pip upgrade reminders, since we're about to upgrade
# it anyway, and also specify the directory pip should use as
# its package cache location.
ENV PIP_DISABLE_VERSION_CHECK=1 \
    XDG_CACHE_HOME=/var/cache/pip

# Install test dependencies first since they're likely to be a
# superset of the deployment dependencies and will populate the
# package cache.
RUN --mount=type=cache,sharing=locked,target=/var/cache/pip,id=pip <<END
/opt/venvs/tests/bin/python -Im pip install --upgrade pip
/opt/venvs/tests/bin/python -Im pip install \
    --no-deps \
    --only-binary :all: \
    --require-hashes \
    -r /opt/venvs/tests/pylock.tests.toml
END

RUN --mount=type=cache,sharing=locked,target=/var/cache/pip,id=pip <<END
/opt/venvs/deploy/bin/python -Im pip install --upgrade pip
/opt/venvs/deploy/bin/python -Im pip install \
    --no-deps \
    --only-binary :all: \
    --require-hashes \
    -r /opt/venvs/deploy/pylock.deploy.toml
END

Now you can pull the installed packages into later stages of your build by copying the virtual environment. For example, in a deployment stage:

FROM python:${PYTHON_VERSION}-${DISTRO} AS deploy

COPY --from=dependencies /opt/venvs/deploy /opt/venvs/deploy

# Do the rest of your deployment stage setup here: copying in
# your application source code, setting the entry point, etc.

Once again, I’m not going to go into a ton of detail here on other things that make a good Python Dockerfile. If you want to learn more about that, I’d recommend reading Itamar Turner-Trauring’s articles on Python and Docker or Hynek Schlawack’s guide to Python and Docker. Both are regularly updated, and they’ll both give you a solid education in how to containerize Python.

Be up-to-date, but stay cool

The only thing still missing here is how to handle updates as new versions of your dependencies are released. For security updates this is crucial, but it’s also important as a general practice. If you make dependency updates a regular, routine part of your development process that’s easy for developers to do, then it’ll also be routine and easy to apply critical updates when they appear. And by applying updates as they come, a couple dependencies at a time, you avoid building up a huge backlog of deferred updates that will make a critical issue even more difficult to address when one inevitably occurs.

And luckily, all of the popular third-party package management tools have straightforward commands you can run to identify and update outdated packages. But there is one wrinkle: you probably don’t want to always eagerly accept most routine dependency updates as soon as they’re released, since that can expose you to potential security issues if a dependency has been compromised but nobody’s noticed it yet. The solution to this is dependency cooldowns: when upgrading, only consider packages which have had at least a bit of time for security researchers and scanners to take a look. The typical recommended window is three days; anything newer than that should be excluded from your package updates unless it’s a critical security patch.

At this point, most popular tools in the Python ecosystem support dependency cooldowns. For example:

You also can configure third-party dependency update bots to apply updates for you as they appear. GitHub’s Dependabot automatically applies a three-day cooldown for everything except critical security updates, and so does the Renovate update bot.

Whether to have a dependency update be a periodic manual task for a developer or something to automate via a tool like Dependabot is up to you (though I personally prefer automation). The important thing is you do it and commit to it, on a cadence that lets you take in updates only one or a few at a time, rather than falling behind and needing to update dozens of packages all in one go.

And that’s a wrap

As promised, that was a lot of words for what’s really a pretty simple set of recommendations. Unfortunately there’s a lot of complexity—necessary complexity, of a sort that pops up in any software packaging ecosystem—lurking in this topic, and explaining all that is what drives up the word count.

But hopefully you now understand how to do “boring” Python dependency management, relying primarily on standard first-party tooling. Even if you don’t want to adopt my recommendations, I’d like to think that learning what they are and why I make them is helpful to you. For me, these recommendations are the result of over a decade of work, across multiple employers, to develop a dependency management workflow that keeps things up-to-date with minimal risk of causing pagers to go off.

Meanwhile I’ve got some ideas for further “boring Python” articles, but those will have to wait for another day.

Changelog

The version of this post you’re looking at right now was written mostly during late August 2026, and published in early September 2026. It’s a major revision of the original, which was written and published in May 2022.

The key changes between the two versions are: