PIP requirements.txt and setup.py

PIP is Python’s next generation package installer following in the footsteps of easy_install. It is pretty handy in conjunction with virtualenv, you probably should be using if you develop Python modules. Virtualenv creates isolated Python environments you can install packages to without the need to use your system’s (or distribution’s) packaging service (e.g. “aptitude”) or more importantly for keeping around different versions of the same software.

A neat feature about PIP is the feature to describe all dependencies in a simple text file called “requirement.txt“. With PIP you can install from almost everywhere: the Python Package Index (PyPI), from a arbitrary download site, from svn, git and other versioning systems. This file comes in handy when setting up a virtual environment with virtualenv. A simple “pip install -r requirements.txt” will download and install all dependencies described there.

Complementary to that, “setup.py” which is used by distutils and setuptools to package python modules already comes with dependencies handled with keywords to the setup call named  “install_requires” (and “dependency_links” and “extras_require”).

For the project pdfserver I wanted to use both worlds for a good reason: If somebody wants to install the package via “pip install pdfserver” all dependencies should be fulfilled during the install in setup.py. But as the software is to be deployed on a server many users will probably download and extract the source package and then created a virtualenv around it using “pip install -r requirements.txt”.

As I found no simple solution to answer my needs, I extended pdfserver’s setup.py to parse the dependencies given in a requirements.txt file. The file is parsed twice, first to extract all dependency names and then again for all URLs for packages not found on PyPI. What it doesn’t do so far is parse the versioning information. 

Here’s the basic code:

def parse_requirements(file_name):    requirements = []    for line in open(file_name, 'r').read().split('n'):        if re.match(r'(s*#)|(s*$)', line):            continue        if re.match(r's*-es+', line):            requirements.append(re.sub(r's*-es+.*#egg=(.*)$', r'1', line))        elif re.match(r's*-fs+', line):            pass        else:            requirements.append(line)    return requirementsdef parse_dependency_links(file_name):    dependency_links = []    for line in open(file_name, 'r').read().split('n'):        if re.match(r's*-[ef]s+', line):            dependency_links.append(re.sub(r's*-[ef]s+', '', line))    return dependency_linkssetup(    install_requires = parse_requirements('requirements.txt'),    dependency_links = parse_dependency_links('requirements.txt'),    ...)

You can find the full setup.py here.

With this setup all dependencies go to requirements.txt and setup.py will automatically pick them up.

Distutils2 will make setup.py obsolete so I read. Let’s see how to handle dependencies then. For now this works for me.

Update: Distribute in Version 0.6.14 cannot install from distributed version control systems (DVCS). So urls like git+git://… won’t work with “python setup.py install”. Pip though can still install packages built with “python setup.py sdist” (even thought with a bug: http://bitbucket.org/ianb/pip/issue/194/dvcs-support-for-uppercase-package-na…


One comment

  1. Hi,Great idea! We liked it so much we’ve developed it further. We’ve published SetupFixer on PyPI and will release an update later this week and which will coincide with a GitHub code release.http://pypi.python.org/pypi/SetupFixer/SetupFixer honours versions and can also build `setup_requires`, `tests_require` and `extras_require`. 0.1.1 will include better documentation.Thought I should let you know seeing as though it was your idea. I also thought you might actually like to use SetupFixer or at least test it out :-)Regards, Martin.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s