10. Updating Project Requirements

Our pip workflow is based on the approach proposed in this article. Note that requirements/* lists unpinned requirements that we added to the project while requirements.txt and requirements_dev.txt list pinned dependencies and sub-dependencies. Pinning requirements are necessary to be able to generate reproducible environments (rather than installing by mistake a new package that might fail to install due to a backward-incompatible change). requirements/* helps us track which packages we added to the project intentionally as opposed to ones installed as sub-dependencies. It also allows us to constrain certain packages to specific versions like limiting django to LTS versions or pinning a dependency due to a bug or pinning report generation packages since each update requires visual inspection of generated reports.

10.1. Steps for updating project dependencies

We will try to have a monthly check (approximately every other sprint) during which project dependencies are updated.

At a high level, the flow is: use pip-outdated to identify packages that need to be updated and then use pip-compile to update requirements.txt and requirements_dev.txt. The detailed steps for doing that are the following:

  • run pip-outdated requirements/production.txt. That should return the list of packages in production.txt that have new versions. The idea is that we want the “Installed” column to be the same as the “Wanted” column. Also, we should run pip-outdated with requirements/base.txt and requirements/local.txt.

  • If there are packages to update, update them with pip-compile --output-file=requirements.txt --upgrade-package '<package>==<version>' requirements/production.txt pip-compile --output-file=requirements_dev.txt --upgrade-package '<package>==<version>' requirements/local.txt

  • Do the last step for all packages that need to be updated and make sure to test the corresponding functionality of packages that were upgraded.

  • Be sure to check the changelog of the packages to understand the changes introduced in the new version (especially breaking changes).

  • choose one package and add a test for it. This helps in reducing the burden of manual testing in future upgrades.

10.2. Steps for adding a new package

  • The package should be added to either base.txt, production.txt or local.txt. It should be unpinned unless there is a very specific reason on why it is pinned and in that case, add the reason as a comment in the corresponding requirements file.

  • Then, run to update the auto-generated requirements for prod and dev:

    • pip-compile --output-file=requirements.txt requirements/production.txt

    • pip-compile --output-file=requirements_dev.txt requirements/local.txt

10.3. Steps for removing a package

This is similar to adding a package. First, remove it from base.txt, production.txt or local.txt. Then, run these commands to update the auto-generated requirements for prod and dev:

  • pip-compile --output-file=requirements.txt requirements/production.txt

  • pip-compile --output-file=requirements_dev.txt requirements/local.txt

10.4. Few notes around package upgrades

  • You can check the docs for pip-tools to know more about how it works.

  • This process updates unpinned packages in base.txt, production.txt and local.txt. For the ones that are intentionally pinned, please check the reason they are pinned and see if it’s possible to upgrade them to the latest.

  • For Django, we will update the patch version every other release to get security updates, but major updates are done by moving from one LTS to the next every 2 years and is handled in its own ticket.

  • For report packages, they should always be pinned since any updates to them requires visual inspection of the generated reports.