Python is a wonderful programming language and much more. One of its weakest points is packaging. This is a well-known fact in the community. Installing, importing, using and creating packages has improved over the years, but it's still not on par with newer languages like Go and Rust that could learn a lot from the struggles of Python and other more mature languages.
In this tutorial, you'll learn everything you need to know to build and share your own packages. For general background on Python packages.
Packaging a Project
Packaging a project is the process by which you take a hopefully coherent set of Python modules and possibly other files and put them in a structure that can be used easily. There are various things you have to consider, such as dependencies on other packages, internal structure (sub-packages), versioning, target audience, and form of package (source and/or binary).
Example
Let's start with a quick example. The conman package is a package for managing configuration. It supports several file formats as well as distributed configuration using etcd.
A package's contents are typically stored in a single directory (although it is common to split sub-packages in multiple directories) and sometimes, as in this case, in its own git repository.
The root directory contains various configuration files (setup.py is mandatory and the most important one), and the package code itself is usually in a subdirectory whose name is the name of the package and ideally a tests directory. Here is what it looks like for "conman":
- > tree
- ├── LICENSE
- ├── MANIFEST.in
- ├── README.md
- ├── conman
- │ ├── __init__.py
- │ ├── __pycache__
- │ ├── conman_base.py
- │ ├── conman_etcd.py
- │ └── conman_file.py
- ├── requirements.txt
- ├── setup.cfg
- ├── setup.py
- ├── test-requirements.txt
- ├── tests
- │ ├── __pycache__
- │ ├── conman_etcd_test.py
- │ ├── conman_file_test.py
- │ └── etcd_test_util.py
- └── tox.ini
- from setuptools import setup, find_packages
- setup(name='conman',
- version='0.3',
- url='https://github.com/the-gigi/conman',
- license='MIT',
- author='Gigi Sayfan',
- author_email='the.gigi@gmail.com',
- description='Manage configuration files',
- packages=find_packages(exclude=['tests']),
- long_description=open('README.md').read(),
- zip_safe=False,
- setup_requires=['nose>=1.0'],
- test_suite='nose.collector')
The Configuration Files
In addition to setup.py, there are a few other optional configuration files that can show up here and serve various purposes.
Setup.py
The setup() function takes a large number of named arguments to control many aspects of package installation as well as running various commands. Many arguments specify metadata used for searching and filtering when uploading your package to a repository.
name: the name of your package (and how it will be listed on PYPI)
version: this is critical for maintaining proper dependency management
url: the URL of your package, typically GitHub or maybe the readthedocs URL
packages: list of sub-packages that need to be included; find_packages() helps here
setup_requires: here you specify dependencies
test_suite: which tool to run at test time
The long_description is set here to the contents of the README.md file, which is a best practice to have a single source of truth.
Setup.cfg
The setup.py file also serves a command-line interface to run various commands. For example, to run the unit tests, you can type: python setup.py test
- running test
- running egg_info
- writing conman.egg-info/PKG-INFO
- writing top-level names to conman.egg-info/top_level.txt
- writing dependency_links to conman.egg-info/dependency_links.txt
- reading manifest file 'conman.egg-info/SOURCES.txt'
- reading manifest template 'MANIFEST.in'
- writing manifest file 'conman.egg-info/SOURCES.txt'
- running build_ext
- test_add_bad_key (conman_etcd_test.ConManEtcdTest) ... ok
- test_add_good_key (conman_etcd_test.ConManEtcdTest) ... ok
- test_dictionary_access (conman_etcd_test.ConManEtcdTest) ... ok
- test_initialization (conman_etcd_test.ConManEtcdTest) ... ok
- test_refresh (conman_etcd_test.ConManEtcdTest) ... ok
- test_add_config_file_from_env_var (conman_file_test.ConmanFileTest) ... ok
- test_add_config_file_simple_guess_file_type (conman_file_test.ConmanFileTest) ... ok
- test_add_config_file_simple_unknown_wrong_file_type (conman_file_test.ConmanFileTest) ... ok
- test_add_config_file_simple_with_file_type (conman_file_test.ConmanFileTest) ... ok
- test_add_config_file_simple_wrong_file_type (conman_file_test.ConmanFileTest) ... ok
- test_add_config_file_with_base_dir (conman_file_test.ConmanFileTest) ... ok
- test_dictionary_access (conman_file_test.ConmanFileTest) ... ok
- test_guess_file_type (conman_file_test.ConmanFileTest) ... ok
- test_init_no_files (conman_file_test.ConmanFileTest) ... ok
- test_init_some_bad_files (conman_file_test.ConmanFileTest) ... ok
- test_init_some_good_files (conman_file_test.ConmanFileTest) ... ok
- ----------------------------------------------------------------------
- Ran 16 tests in 0.160s
- OK
The setup.cfg is an ini format file that may contain option defaults for commands you pass to setup.py. Here, setup.cfg contains some options for nosetests (our test runner):
- [nosetests]
- verbose=1
- nocapture=1
MANIFEST.in
This file contains files that are not part of the internal package directory, but you still want to include. Those are typically the readme file, the license file and similar. An important file is the requirements.txt . This file is used by pip to install other required packages.
Here is conman's MANIFEST.in file:
- include LICENSE
- include README.md
- include requirements.txt
Dependencies
You can specify dependencies both in the install_requires section of setup.py and in a requirements.txt file. Pip will install automatically dependencies from install_requires, but not from the requirements.txt file. To install those requirements, you'll have to specify it explicitly when running pip: pip install -r requirements.txt.
Theinstall_requires option is designed to specify minimal and more abstract requirements at the major version level. The requirements.txt file is for more concrete requirements often with pinned down minor versions.
Here is the requirements file of conman. You can see that all the versions are pinned, which means it can be negatively impacted if one of these packages upgrades and introduces a change that breaks conman.
- PyYAML==3.11
- python-etcd==0.4.3
- urllib3==1.7
- pyOpenSSL==0.15.1
- psutil==4.0.0
- six==1.7.3
Pinning gives you predictability and peace of mind. This is especially important if many people install your package at different times. Without pinning, each person will get a different mix of dependency versions based on when they installed it. The downside of pinning is that if you don't keep up with your dependencies development, you may get stuck on an old, poorly performing and even vulnerable version of some dependency.
I originally wrote conman in 2014 and didn't pay much attention to it. Now, for this tutorial I upgraded everything and there were some major improvements across the board for almost every dependency.
Written by: Gigi Sayfan
If you found this post interesting, follow and support us.
Suggest for you:
No comments:
Post a Comment