Building and Deploying Python Packages: A Comprehensive Tutorial for Experts

Introduction

Developing and distributing Python packages is a crucial aspect of software development. This tutorial will guide you through the process of creating a Python package, configuring it with setuptools, and deploying it using pip. We’ll cover topics such as project structure, setup.py and setup.cfg configurations, versioning, dependencies, and more.
In the example below, I will only example names, packages and versions. Feel free to change the fields according to your needs.

Prerequisites

Make sure you have the following installed on your system:

• Python (>=3.7)
• pip
• setuptools

1. Project Structure

A well-organized project structure makes it easier for users to understand and use your package. Here is a sample structure:

my_package/
│
├── my_package/
│   ├── __init__.py
│   ├── module1.py
│   ├── module2.py
│   └── ...
│
├── tests/
│   ├── __init__.py
│   ├── test_module1.py
│   ├── test_module2.py
│   └── ...
│
├── setup.py
├── README.md
├── LICENSE
└── requirements.txt

• my_package/: Top-level package directory.
  • __init__.py: Initializes the package.
  • module1.py, module2.py, …: Package modules.
• tests/: Directory for unit tests.
• setup.py: Configuration file for setuptools.
• README.md: Documentation for your package.
• LICENSE: License file.
• requirements.txt: List of dependencies.

2. setup.py Configuration

Create a setup.py file in the project root:

from setuptools import setup, find_packages

setup(
    name='my_package',
    version='0.1.0',
    packages=find_packages(),
    install_requires=[
        'dependency1',
        'dependency2',
    ],
    entry_points={
        'console_scripts': [
            'my_script = my_package.module:main_function',
        ],
    },
    author='Your Name',
    author_email='your.email@example.com',
    description='Description of your package',
    long_description=open('README.md').read(),
    long_description_content_type='text/markdown',
    url='https://github.com/your_username/my_package',
    license='MIT',
    classifiers=[
        'Development Status :: 3 - Alpha',
        'Intended Audience :: Developers',
        'License :: OSI Approved :: MIT License',
        'Programming Language :: Python :: 3',
        'Programming Language :: Python :: 3.7',
        'Programming Language :: Python :: 3.8',
    ],
)

• name: Package name.
• version: Package version.
• packages: Automatically discover and include all packages.
• install_requires: List of dependencies.
• entry_points: Define console scripts.
• author, author_email: Author information.
• description: Short description of your package.
• long_description: Detailed description loaded from README.md.
• url: URL for more information.
• license: Package license.
• classifiers: Trove classifiers.

3. setup.cfg Configuration

A setup.cfg file can be used for a cleaner configuration:

[metadata]
name = my_package
version = attr: my_package.__version__
author = Your Name
author_email = your.email@example.com
description = Description of your package
long_description = file: README.md
long_description_content_type = text/markdown
url = https://github.com/your_username/my_package
license = MIT
classifiers =
    Development Status :: 3 - Alpha
    Intended Audience :: Developers
    License :: OSI Approved :: MIT License
    Programming Language :: Python :: 3
    Programming Language :: Python :: 3.7
    Programming Language :: Python :: 3.8

[options]
package_dir =
    = my_package
packages = find:
python_requires = >=3.7
install_requires =
    dependency1
    dependency2

[options.packages.find]
where = my_package

This configuration achieves the same result as setup.py. The name and version are extracted from the package, and other metadata are specified in the [metadata] section.

4. Versioning

Maintain a version number in your package to track changes. Use a tool like bump2version for seamless versioning:

pip install bump2version

Run the following commands to bump the version:

bump2version patch  # or 'minor' or 'major'

This will update the version in your __version__ attribute and create a new git tag.

The command bump2version is a tool commonly used for managing version numbers in software projects. It simplifies the process of incrementing version numbers based on the Semantic Versioning (SemVer) convention.

• bump2version: This is the command to run the bump2version tool.

• patch (or ‘minor’ or ‘major’): This argument specifies the level of version increment. It indicates whether you want to increment the patch, minor, or major version number.

• patch: Increment the last part of the version number (e.g., from 1.0.0 to 1.0.1). Use this when you make backward-compatible bug fixes.

• minor: Increment the middle part of the version number (e.g., from 1.0.0 to 1.1.0). Use this when you add backward-compatible features.

• major: Increment the first part of the version number (e.g., from 1.0.0 to 2.0.0). Use this when you make incompatible API changes.

By running one of these commands, bump2version automatically updates the version number in your project files according to the specified increment level. This tool is especially useful in maintaining a consistent versioning scheme across your project and simplifying the process of releasing new versions.

Here’s a quick example:

# Increment the patch version
bump2version patch

# Increment the minor version
bump2version minor

# Increment the major version
bump2version major

After running one of these commands, bump2version will update the version number in your project files, create a new commit with the updated version, and optionally create a new git tag corresponding to the version. This is a convenient way to manage versioning during development and before releases.

5. Building and Distributing

To create distribution packages, run:

python setup.py sdist bdist_wheel

This generates a source distribution and a wheel distribution in the dist/ directory.

6. Installing and Testing Locally

To install the package locally for testing, use:

pip install .

Or for development with auto-reloading:

pip install -e .

This installs the package in “editable” mode, reflecting changes without reinstalling.

6.1 Installing a Package in Editable Mode:

The pip install -e . command is used to install a Python package in “editable” or “development” mode. This means that instead of copying the package files into the site-packages directory like a regular installation, a link to the project’s source code is created. This enables developers to make changes to the code, and those changes will immediately take effect without the need for reinstalling the package.

1. Navigate to the Project Directory: Open a terminal and navigate to the directory containing your Python project, where the setup.py file is located.

   cd /path/to/your/project
   

2. Run pip install -e .: Use the following command to install the package in editable mode:

   pip install -e .
   

   The dot (`.`) represents the current directory.

6.1.1 How Editable Mode Works:

• Link to Source Code: Instead of copying the package files, pip creates a link to the source code in the site-packages directory.

• Changes Reflect Instantly: Any changes you make to the code within the project directory will instantly affect the installed package without requiring a reinstallation.

• Useful for Development: Editable mode is particularly useful during development when you are actively working on the package code and want to see the impact of changes without constantly reinstalling.

6.1.2 examples:

Example 1: Basic Usage

Consider a simple project structure:

my_package/
│
├── my_package/
│   ├── __init__.py
│   └── module.py
│
└── setup.py

After navigating to the project directory, run:

pip install -e .

This will create a link to your my_package project, and you can immediately use the package in your Python environment.

Example 2: Development with Dependencies

If your package has external dependencies, they will be installed in the usual way, but your package will still be installed in editable mode. For instance:

my_package/
│
├── my_package/
│   ├── __init__.py
│   └── module.py
│
├── setup.py
└── requirements.txt

Contents of requirements.txt:

numpy>=1.18.0

Navigate to the project directory and run:

pip install -e . -r requirements.txt

This installs your package in editable mode along with its dependencies specified in requirements.txt.

Example 3: Development with Extras

You can also use “extras” for additional dependencies. Update your setup.py like this:

from setuptools import setup

setup(
    name='my_package',
    version='0.1.0',
    install_requires=[
        'numpy>=1.18.0',
    ],
    extras_require={
        'dev': [
            'pytest>=6.0.0',
        ],
    },
)

Install the package with extras:

pip install -e .[dev]

This installs your package in editable mode with both the main dependencies and the extra development dependencies.

7. Deploying to PyPI

To deploy your package to PyPI, install twine:

pip install twine

Upload the distributions to PyPI:

twine upload dist/*

Ensure you have the correct PyPI credentials configured.

Conclusion

Creating and deploying Python packages involves proper project organization, versioning, and distribution. By following this tutorial, you’ve gained insights into creating a Python package, configuring it with setuptools, versioning, and deploying it using pip and PyPI. Continue improving and expanding your package to meet the needs of your users.