Django vs Flask vs Pyramid: Choosing a Python Web Framework_part2 (end)

5.2 Flask

Flask uses the Django-inspired Jinja2 templating language by default but can be configured to use another language. A programmer in a hurry couldn't be blamed for mixing up Django and Jinja templates. In fact, both the Django examples above work in Jinja2. Instead of going over the same examples, let's look at the places that Jinja2 is more expressive than Django templating.
Both Jinja and Django templates provide a feature called filtering, where a list can be passed through a function before being displayed. A blog that features post categories might make use of filters to display a post's categories in a comma-separated list.

1. <!-- Django -->
2. <div class="categories">Categories: {{ post.categories|join:", " }}</div>
3. <!-- now in Jinja -->
4. <div class="categories">Categories: {{ post.categories|join(", ") }}</div>

In Jinja's templating language it's possible to pass any number of arguments to a filter because Jinja treats it like a call to a Python function, with parenthesis surrounding the arguments. Django uses a colon as a separator between the filter name and the filter argument, which limits the number of arguments to just one.
Jinja and Django for loops are also similar. Let's see where they differ. In Jinja2, the for-else-endfor construct lets you iterate over a list, but also handle the case where there are no items.

1. {% for item in inventory %}
2. <div class="display-item">{{ item.render() }}</div>
3. {% else %}
4. <div class="display-warn">
5. <h3>No items found</h3>
6. <p>Try another search, maybe?</p>
7. </div>
8. {% endfor %}

The Django version of this functionality is identical, but uses for-empty-endfor instead of for-else-endfor.

1. {% for item in inventory %}
2. <div class="display-item">{{ item.render }}</div>
3. {% empty %}
4. <div class="display-warn">
5. <h3>No items found</h3>
6. <p>Try another search, maybe?</p>
7. </div>
8. {% endfor %}

Other than the syntactic differences above, Jinja2 provides more control over its execution environment and advanced features. For example, it's possible to disable potentially dangerous features to safely execute untrusted templates, or to compile templates ahead of time to ensure their validity.

5.3 Pyramid

Like Flask, Pyramid supports many templating languages (including Jinja2 and Mako) but ships with one by default. Pyramid uses Chameleon, an implementation of ZPT (the Zope Page Template) templating language. Let's look back at our first example, adding a user's name to the top bar of our site. The Python code looks much the same except that we don't need to explicitly call a render_template function.

1. @view_config(renderer='templates/home.pt')
2. def my_view(request):
3.     # do stuff...
4.     return {'user': user}

But our template looks pretty different. ZPT is an XML-based templating standard, so we use XSLT-like statements to manipulate data.

1. <div class="top-bar row">
2.   <div class="col-md-10">
3.   <!-- more top bar things go here -->
4.   </div>
5.   <div tal:condition="user"
6.        tal:content="string:You are logged in as ${user.fullname}"
7.        class="col-md-2 whoami">
8.   </div>
9. </div>

Chameleon actually has three different namespaces for template actions. TAL (template attribute language) provides basics like conditionals, basic string formatting, and filling in tag contents. The above example only made use of TAL to complete its work. For more advanced tasks, TALES and METAL are required. TALES (Template Attribute Language Expression Syntax) provides expressions like advanced string formatting, evaluation of Python expressions, and importing expressions and templates.
METAL (Macro Expansion Template Attribute Language) is the most powerful (and complex) part of Chameleon templating. Macros are extensible, and can be defined as having slots that are filled when the macro is invoked.

6 Frameworks in Action

For each framework let's take a look at making an app called wut4lunch, a social network to tell the whole internet what you ate for lunch. Free startup idea right there, totally a gamechanger. The application will be a simple interface that allows users to post what they had for lunch and to see a list of what other users ate. The home page will look like this when we're done.

6.1 Demo App with Flask

The shortest implementation clocks in at 34 lines of Python and a single 22 line Jinja template. First we have some housekeeping tasks to do, like initializing our app and pulling in our ORM.
from flask import Flask

1. # For this example we'll use SQLAlchemy, a popular ORM that supports a
2. # variety of backends including SQLite, MySQL, and PostgreSQL
3. from flask.ext.sqlalchemy import SQLAlchemy
4. app = Flask(__name__)
5. # We'll just use SQLite here so we don't need an external database
6. app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///test.db'
7. db = SQLAlchemy(app)

Now let's take a look at our model, which will remain almost the same for our other two examples as well.

1. class Lunch(db.Model):
2.     """A single lunch"""
3.     id = db.Column(db.Integer, primary_key=True)
4.     submitter = db.Column(db.String(63))
5.     food = db.Column(db.String(255))

Wow, that's pretty easy. The hardest part was finding the right SQLAlchemy data types and picking a length for our String  fields in the database. Using our models is also extremely simple, thanks to the SQLAlchemy query syntax we'll see later.

Building our submission form is just as easy. After importing Flask-WTForms and the correct field types, you can see the form looks quite a bit like our model. The main difference is the new submit button and prompts for the food and submitter name fields.

The SECRET_KEY field in the app config is used by WTForms to create CSRF tokens. It is also used by itsdangerous (included in Flask) to sign cookies and other data.
from flask.ext.wtf import Form

1. from wtforms.fields import StringField, SubmitField
2. app.config['SECRET_KEY'] = 'please, tell nobody'
3. class LunchForm(Form):
4.     submitter = StringField(u'Hi, my name is')
5.     food = StringField(u'and I ate')
6.     # submit button will read "share my lunch!"
7.     submit = SubmitField(u'share my lunch!')

Making the form show up in the browser means the template has to have it. We'll pass that in below.

1. from flask import render_template
2. @app.route("/")
3. def root():
4.     lunches = Lunch.query.all()
5.     form = LunchForm()
6.     return render_template('index.html', form=form, lunches=lunches)

Alright, what just happened? We got a list of all the lunches that have already been posted with Lunch.query.all(), and instantiated a form to let the user post their own gastronomic adventure. For simplicity, the variables are passed into the template with the same name, but this isn't required.

1. <html>
2. <title>Wut 4 Lunch</title>
3. <b>What are people eating?</b>
4. <p>Wut4Lunch is the latest social network where you can tell all your friends
5. about your noontime repast!</p>

Here's the real meat of the template, where we loop through all the lunches that have been eaten and display them in a <ul>. This almost identical to the looping example we saw earlier.

1. <ul>
2. {% for lunch in lunches %}
3. <li><strong>{{ lunch.submitter|safe }}</strong> just ate <strong>{{ lunch.food|safe }}</strong>
4. {% else %}
5. <li><em>Nobody has eaten lunch, you must all be starving!</em></li>
6. {% endfor %}
7. </ul>
8. <b>What are YOU eating?</b>
9. <form method="POST" action="/new">
10.     {{ form.hidden_tag() }}
11.     {{ form.submitter.label }} {{ form.submitter(size=40) }}
12.     <br/>
13.     {{ form.food.label }} {{ form.food(size=50) }}
14.     <br/>
15.     {{ form.submit }}
16. </form>
17. </html>

The <form> section of the template just renders the form labels and inputs from the WTForm object we passed into the template in the root() view. When the form is submitted, it'll send a POST request to the /new endpoint which will be processed by the function below.
from flask import url_for, redirect

1. @app.route(u'/new', methods=[u'POST'])
2. def newlunch():
3.     form = LunchForm()
4.     if form.validate_on_submit():
5.         lunch = Lunch()
6.         form.populate_obj(lunch)
7.         db.session.add(lunch)
8.         db.session.commit()
9.     return redirect(url_for('root'))

After validating the form data, we put the contents into one of our Model objects and commit it to the database. Once we've stored the lunch in the database it'll show up in the list of lunches people have eaten.

1. if __name__ == "__main__":
2.     db.create_all()  # make our sqlalchemy tables
3.     app.run()

Finally, we have to do a (very) little bit of work to actually run our app. Using SQLAlchemy we create the table we use to store lunches, then start running the route handlers we wrote.

6.2 Demo App with Django

The Django version of wut4lunch is similar to the Flask version, but is spread across several files in the Django project. First, let's look at the most similar portion: the database model. The only difference between this and the SQLAlchemy version is the slightly different syntax for declaring a database field that holds text.

1. # from wut4lunch/models.py
2. from django.db import models
3. class Lunch(models.Model):
4.     submitter = models.CharField(max_length=63)
5.     food = models.CharField(max_length=255)

On to the form system. Unlike Flask, Django has a built-in form system that we can use. It looks much like the WTForms module we used in Flask with different syntax.

1. from django import forms
2. from django.http import HttpResponse
3. from django.shortcuts import render, redirect
4. from .models import Lunch
5. # Create your views here.
6. class LunchForm(forms.Form):
7.     """Form object. Looks a lot like the WTForms Flask example"""
8.     submitter = forms.CharField(label='Your name')
9.     food = forms.CharField(label='What did you eat?')

Now we just need to make an instance of LunchForm to pass in to our template.

1. lunch_form = LunchForm(auto_id=False)
2. def index(request):
3.     lunches = Lunch.objects.all()
4.     return render(
5.         request,
6.         'wut4lunch/index.html',
7.         {
8.             'lunches': lunches,
9.             'form': lunch_form,
10.         }
11.     )

The render function is a Django shortcut that takes the request, the template path, and a context dict. Similar to Flask's render_template, but it also takes the incoming request.

1. def newlunch(request):
2.     l = Lunch()
3.     l.submitter = request.POST['submitter']
4.     l.food = request.POST['food']
5.     l.save()
6.     return redirect('home')

Saving the form response to the database is different, instead of using a global database session Django lets us call the model's ..save() method and handles session management transparently. Neat!

Django provides some nice features for us to manage the lunches that users have submitted, so we can delete lunches that aren't appropriate for our site. Flask and Pyramid don't provide this automatically, and not having to write Yet Another Admin Page when making a Django app is certainly a feature. Developer time isn't free! All we had to do to tell Django-admin about our models is add two lines to wut4lunch/admin.py.

1. from wut4lunch.models import Lunch
2. admin.site.register(Lunch)

Bam. And now we can add and delete entries without doing any extra work.

Lastly, let's take a look at the differences in the homepage template.

1. <ul>
2. {% for lunch in lunches %}
3. <li><strong>{{ lunch.submitter }}</strong> just ate <strong>{{ lunch.food }}</strong></li>
4. {% empty %}
5. <em>Nobody has eaten lunch, you must all be starving!</em>
6. {% endfor %}
7. </ul>

Django has a handy shortcut for referencing other views in your pages. The url tag makes it possible for you to restructure the URLs your application serves without breaking your views. This works because the url tag looks up the URL of the view mentioned on the fly.

1. <form action="{% url 'newlunch' %}" method="post">
2.   {% csrf_token %}
3.   {{ form.as_ul }}
4.   <input type="submit" value="I ate this!" />
5. </form>

The form is rendered with different syntax, and we need to include a CSRF token manually in the form body, but these differences are mostly cosmetic.

6.3 Demo App with Pyramid

Finally, let's take a look at the same program in Pyramid. The biggest difference from Django and Flask here is the templating. Changing the Jinja2 template very slightly was enough to solve our problem in Django. Not so this time, Pyramid's Chameleon template syntax is more reminiscent of XSLT than anything else.

1. <!-- pyramid_wut4lunch/templates/index.pt -->
2. <div tal:condition="lunches">
3.   <ul>
4.     <div tal:repeat="lunch lunches" tal:omit-tag="">
5.       <li tal:content="string:${lunch.submitter} just ate ${lunch.food}"/>
6.     </div>
7.   </ul>
8. </div>
9. <div tal:condition="not:lunches">
10.   <em>Nobody has eaten lunch, you must all be starving!</em>
11. </div>

Like in Django templates, a lack of the for-else-endfor construct makes the logic slightly more verbose. In this case, we end up with if-for and if-not-for blocks to provide the same functionality. Templates that use XHTML tags may seem foreign after using Django- and AngularJS-style templates that use {{ or {% for control structures and conditionals.

One of the big upsides to the Chameleon templating style is that your editor of choice will highlight the syntax correctly, since the templates are valid XHTML. For Django and Flask templates your editor needs to have support for those templating languages to highlight correctly.

1. <b>What are YOU eating?</b>
2. <form method="POST" action="/newlunch">
3.   Name: ${form.text("submitter", size=40)}
4.   <br/>
5.   What did you eat? ${form.text("food", size=40)}
6.   <br/>
7.   <input type="submit" value="I ate this!" />
8. </form>
9. </html>

The form rendering is slightly more verbose in Pyramid because the pyramid_simpleform doesn't have an equivalent to Django forms' form.as_ul function, which renders all the form fields automatically.

Now let's see what backs the application. First, we'll define the form we need and render our homepage.

1. # pyramid_wut4lunch/views.py
2. class LunchSchema(Schema):
3.     submitter = validators.UnicodeString()
4.     food = validators.UnicodeString()
5. @view_config(route_name='home',
6.              renderer='templates/index.pt')
7. def home(request):
8.     lunches = DBSession.query(Lunch).all()
9.     form = Form(request, schema=LunchSchema())
10.     return {'lunches': lunches, 'form': FormRenderer(form)}

The query syntax to retrieve all the lunches is familiar from Flask because both demo applications use the popular SQLAlchemy ORM to provide persistent storage. In Pyramid lets you return your template's context dictionary directly instead of needing to call a special render function. The @view_config decorator automatically passes the returned context to the template to be rendered. Being able to skip calling the render method makes functions written for Pyramid views easier to test, since the data they return isn't obscured in a template renderer object.

1. @view_config(route_name='newlunch',
2.              renderer='templates/index.pt',
3.              request_method='POST')
4. def newlunch(request):
5.     l = Lunch(
6.         submitter=request.POST.get('submitter', 'nobody'),
7.         food=request.POST.get('food', 'nothing'),
8.     )
9.     with transaction.manager:
10.         DBSession.add(l)
11.     raise exc.HTTPSeeOther('/')

Form data is easy to retrieve from Pyramid's request object, which automatically parsed the form POST data into a dict that we can access. To prevent multiple concurrent requests from all accessing the database at the same time, the ZopeTransactions module provides context managers for grouping database writes into logical transactions and prevent threads of your application from stomping on each others' changes, which can be a problem if your views share a global session and your app receives a lot of traffic.

7 Summary

Pyramid is the most flexible of the three. It can be used for small apps as we've seen here, but it also powers big-name sites like Dropbox. Open Source communities like Fedora choose it for applications like their community badges system, which receives information about events from many of the project's tools to award achievement-style badges to users. One of the most common complaints about Pyramid is that it presents so many options it can be intimidating to start a new project.
By far the most popular framework is Django, and the list of sites that use it is impressive. Bitbucket, Pinterest, Instagram, and The Onion use Django for all or part of their sites. For sites that have common requirements, Django chooses very sane defaults and because of this it has become a popular choice for mid- to large-sized web applications.

Flask is great for developers working on small projects that need a fast way to make a simple, Python-powered web site. It powers loads of small one-off tools, or simple web interfaces built over existing APIs. Backend projects that need a simple web interface that is fast to develop and will require little configuration often benefit from Flask on the frontend, like jitviewer which provides a web interface for inspecting PyPy just-in-time compiler logs.

All three frameworks came up with a solution to our small list of requirements, and we've been able to see where they differ. Those differences aren't just cosmetic, and they will change how you design your product and how fast you ship new features and fixes. Since our example was small, we've seen where Flask shines and how Django can feel clunky on a small scale. Pyramid's flexibility didn't become a factor because our requirements stayed the same, but in the real world new requirements are thrown in constantly.
Written by Ryan Brown

If you found this post interesting, follow and support us.
Suggest for you:

Zero to Hero with Python Professional Python Programmer Bundle

The Python Mega Course: Build 10 Python Applications

Complete Python Bootcamp (Hot)

Learning Python for Data Analysis and Visualization

Learn Python for Beginners!

Django vs Flask vs Pyramid: Choosing a Python Web Framework_part1

TL;DR: Pyramid, Django, and Flask are all excellent frameworks, and choosing just one for a project is hard. We'll see working apps with identical functionality in all three frameworks to make comparing the three easier. Skip to Frameworks in Action[1]

1 Introduction

The world of Python web frameworks is full of choices. Django, Flask, Pyramid, Tornado, Bottle, Diesel, Pecan, Falcon, and many more are competing for developer mindshare. As a developer you want to cut the legions of options down to the one that will help you finish your project and get on to the Next Big Thing (tm). We'll focus on Flask, Pyramid, and Django. Their ideal cases span from micro-project to enterprise-size web service.
To help make the choice between the three easier (or at least more informed), we'll build the same application in each framework and compare the code, highlighting the strengths and weaknesses of each approach. If you just want the code, skip straight to Frameworks in Action or view the code on Github.
Flask is a "microframework" primarily aimed at small applications with simpler requirements. Pyramid and Django are both aimed at larger applications, but take different approaches to extensibility and flexibility. Pyramid targets flexibility and lets the developer use the right tools for their project. This means the developer can choose the database, URL structure, templating style, and more. Django aims to include all the batteries a web application will need so developers need only open the box and start working, pulling in Django's many modules as they go.
Django includes an ORM out of the box, while Pyramid and Flask leave it to the developer to choose how (or if) they want their data stored. The most popular ORM for non-Django web applications is SQLAlchemy by far, but there are plenty of other options from DynamoDB and MongoDB to simple local persistence like LevelDB or plain SQLite. Pyramid is designed to use any persistence layer, even yet-to-be-invented ones.

2 About the Frameworks

Django's "batteries included" approach makes it easy for developers who know Python already to dive in to web applications quickly without needing to make a lot of decisions about their application's infrastructure ahead of time. Django has for templating, forms, routing, authentication, basic database administration, and more built in. In contrast, Pyramid includes routing and authentication, but templating and database administration require external libraries.
The extra work up front to choose components for Flask and Pyramid apps yields more flexibility for developers whose use case doesn't fit a standard ORM, or who need to interoperate with different workflows or templating systems.
Flask, the youngest of the three frameworks, started in mid-2010. The Pyramid framework began life in the Pylons project and got the name Pyramid in late 2010, though the first release was in 2005. Django had its first release in 2006, shortly after the Pylons (eventually Pyramid) project began. Pyramid and Django are extremely mature frameworks, and have accumulated plugins and extensions to meet an incredibly large range of needs.
Though Flask has a shorter history, it has been able to learn from frameworks that have come before and has set its sights firmly on small projects. It is clearly used most often in smaller projects with just one or two functions. One such project is httpbin, a simple (but extremely powerful) helper for debugging and testing HTTP libraries.

3 Community

The prize for most active community goes to Django with 80,000 StackOverflow questions and a healthy set of blogs from developers and power users. The Flask and Pyramid communities aren't as large, but their communities are quite active on their mailing lists and on IRC. With only 5,000 StackOverflow questions tagged, Flask is 15x smaller than Django. On Github, they have a nearly identical number of stars with 11,300 for Django, and 10,900 for Flask.
All three frameworks are available under BSD-derived permissive licenses. Both Flask's and Django's licenses are 3-clause BSD, while Pyramid's Repoze Public License RPL is a derivative of the 4-clause BSD license.

4 Bootstrapping

Django and Pyramid both come with bootstrapping tools built in. Flask includes nothing of the sort because Flask's target audience isn't trying to build large MVC applications.

4.1 Flask

Flask's Hello World app has to be the simplest out there, clocking in at a puny 7 lines of code in a single Python file.

1. # from http://flask.pocoo.org/ tutorial
2. from flask import Flask
3. app = Flask(__name__)
4. @app.route("/") # take note of this decorator syntax, it's a common pattern
5. def hello():
6.     return "Hello World!"
7. if __name__ == "__main__":
8.     app.run()

This is why there aren't bootstrapping tools for Flask: there isn't a demand for them. From the above Hello World featured on Flask's homepage, a developer with no experience building Python web applications can get hacking immediately.

For projects that need more separation between components, Flask has blueprints. For example, you could structure your Flask app with all user-related functions in users.py and your sales-related functions in ecommerce.py, then import them and add them to your app in site.py. We won't go over this functionality, as it's beyond the needs of our demo app.

4.2 Pyramid

Pyramid's bootstrapping tool is called pcreate which is part of Pyramid. Previously the Paste suite of tools provided bootstrapping for but has since been replaced with a Pyramid-specific toolchain.

1. $ pcreate -s starter hello_pyramid # Just make a Pyramid project

Pyramid is intended for bigger and more complex applications than Flask. Because of this, its bootstrapping tool creates a bigger skeleton project. It also throws in basic configuration files, an example template, and the files to package your application for uploading to the Python Package Index.

1. hello_pyramid
2. ├── CHANGES.txt
3. ├── development.ini
4. ├── MANIFEST.in
5. ├── production.ini
6. ├── hello_pyramid
7. │   ├── __init__.py
8. │   ├── static
9. │   │   ├── pyramid-16x16.png
10. │   │   ├── pyramid.png
11. │   │   ├── theme.css
12. │   │   └── theme.min.css
13. │   ├── templates
14. │   │   └── mytemplate.pt
15. │   ├── tests.py
16. │   └── views.py
17. ├── README.txt
18. └── setup.py

As in the rest of the framework, Pyramid's bootstrapper is incredibly flexible. It's not limited to one default application; pcreate can use any number of project templates. Included in pcreate there is the "starter" template we used above, along with SQLAlchemy- and ZODB-backed scaffold projects. On PyPi it's possible to find ready-made scaffolds for Google App Engine, jQuery Mobile, Jinja2 templating, modern frontend frameworks, and many more.

4.3 Django

Django also has its own bootstrapping tool built in as a part of django-admin.

1. django-admin startproject hello_django
2. django-admin startapp howdy # make an application within our project

We can already see one of the ways Django differs from Pyramid. Django separates a project into individual applications, where Pyramid and Flask expect a project to be a single "application" with several views or models. It's possible to replicate the project/app distinction in Flask and Pyramid, but the notion does not exist by default.

1. hello_django
2. ├── hello_django
3. │   ├── __init__.py
4. │   ├── settings.py
5. │   ├── urls.py
6. │   └── wsgi.py
7. ├── howdy
8. │   ├── admin.py
9. │   ├── __init__.py
10. │   ├── migrations
11. │   │   └── __init__.py
12. │   ├── models.py
13. │   ├── tests.py
14. │   └── views.py
15. └── manage.py

By default Django only includes empty model and template files, so a new user sees a bit less example code to start out. It also (unfortunately) leaves the choice of how to distribute their application to the developer.
The downside of the bootstrap tool not guiding users to package their apps is that novice users won't. If a developer hasn't packaged an app before, they'll find themselves rudely surprised upon their first deploy. Projects with a large community like django-oscar are packaged and available on PyPi, but smaller projects on Github often to lack uniform packaging.

5 Templating

Just having a Python application that can respond to HTTP requests is a great start, but it's a good bet that most of your users won't be interested in using curl to interact with your web app. Fortunately, all three contenders provide an easy way to fill in HTML with custom info, and let folks enjoy your swanky Bootstrap frontend.
Templating lets you inject dynamic information directly into your page without using making AJAX requests. This is nice from a user experience perspective since you only need to make one round-trip to get the full page and all its dynamic data. This is especially important on mobile sites where round trips can take multiple seconds.
All the templating options we'll see rely on a "context" that provides the dynamic information for the template to render into HTML. The simplest use case for a template would be to populate a logged-in user's name to greet them properly. It would be possible to use AJAX to get this sort of dynamic information, but requiring a whole call just to fill in a user's name would be a bit excessive when templates are this easy.

5.1 Django

Our example use case is about as easy as it gets, assuming that we have a user object that has a fullname property containing a user's name. In Python we'd pass the current user to the template like so:

1. def a_view(request):
2.     # get the logged in user
3.     # ... do more things
4.     return render_to_response(
5.         "view.html",
6.         {"user": cur_user}
7.     )

Populating the template context is as simple as passing a dictionary of the Python objects and data structures the template should use. Now we need to render their name to the page, just in case they forget who they are.

1. <!-- view.html -->
2. <div class="top-bar row">
3.   <div class="col-md-10">
4.   <!-- more top bar things go here -->
5.   </div>
6.   {% if user %}
7.   <div class="col-md-2 whoami">
8.     You are logged in as {{ user.fullname }}
9.   </div>
10.   {% endif %}
11. </div>

First, you'll notice the {% if user %} construct. In Django templates {% is used for control statements like loops and conditionals. The if user statement is there to guard against cases where there is not a user. Anonymous users shouldn't see "you are logged in as" in the site header.
Inside the if block, you can see that including the name is as simple as wrapping the property we want to insert in {{ }}. The {{ is used to insert actual values into the template, such as {{ user.fullname }}.
Another common use for templates is displaying groups of things, like the inventory page for an ecommerce site.

1. def browse_shop(request):
2.     # get items
3.     return render_to_response(
4.         "browse.html",
5.         {"inventory": all_items}
6.     )

In the template we can use the same {% to loop over all the items in the inventory, and to fill in the URL to their individual page.

1. {% for widget in inventory %}
2.     <li><a href="/widget/{{ widget.slug }}/">{{ widget.displayname }}</a></li>
3. {% endfor %}

To do most common templating tasks, Django can accomplish the goal with just a few constructs, making it easy to get started.
Written by Ryan Brown (continue)

If you found this post interesting, follow and support us.

Suggest for you:

Zero to Hero with Python Professional Python Programmer Bundle

The Python Mega Course: Build 10 Python Applications

Complete Python Bootcamp (Hot)

Learn Python for Beginners!

Data Science: Deep Learning in Python

How to Write Your Own Python Packages_part1

Overview

Python is a wonderful programming language and much more. One of its weakest points is packaging. This is a well-known fact in the community. Installing, importing, using and creating packages has improved over the years, but it's still not on par with newer languages like Go and Rust that could learn a lot from the struggles of Python and other more mature languages.

In this tutorial, you'll learn everything you need to know to build and share your own packages. For general background on Python packages.

Packaging a Project

Packaging a project is the process by which you take a hopefully coherent set of Python modules and possibly other files and put them in a structure that can be used easily. There are various things you have to consider, such as dependencies on other packages, internal structure (sub-packages), versioning, target audience, and form of package (source and/or binary).

Example
Let's start with a quick example. The conman package is a package for managing configuration. It supports several file formats as well as distributed configuration using etcd.

A package's contents are typically stored in a single directory (although it is common to split sub-packages in multiple directories) and sometimes, as in this case, in its own git repository.

The root directory contains various configuration files (setup.py is mandatory and the most important one), and the package code itself is usually in a subdirectory whose name is the name of the package and ideally a tests directory. Here is what it looks like for "conman":

1. > tree 
2. ├── LICENSE 
3. ├── MANIFEST.in 
4. ├── README.md 
5. ├── conman 
6. │   ├── __init__.py 
7. │   ├── __pycache__ 
8. │   ├── conman_base.py 
9. │   ├── conman_etcd.py 
10. │   └── conman_file.py 
11. ├── requirements.txt 
12. ├── setup.cfg 
13. ├── setup.py 
14. ├── test-requirements.txt 
15. ├── tests 
16. │   ├── __pycache__ 
17. │   ├── conman_etcd_test.py
18. │   ├── conman_file_test.py
19. │   └── etcd_test_util.py
20. └── tox.ini

Let's take a quick peek at the setup.py file. It imports two functions from the setuptools package: setup() and find_packages(). Then it calls the setup()function and uses  find_packages()   for one of the parameters.

1. from setuptools import setup, find_packages 
2. setup(name='conman',
3.       version='0.3',
4.       url='https://github.com/the-gigi/conman',
5.       license='MIT',
6.       author='Gigi Sayfan',
7.       author_email='the.gigi@gmail.com', 
8.       description='Manage configuration files', 
9.       packages=find_packages(exclude=['tests']),
10.       long_description=open('README.md').read(),
11.       zip_safe=False,
12.       setup_requires=['nose>=1.0'],
13.       test_suite='nose.collector')

This is pretty normal. While the setup.py file is a regular Python file and you can do whatever you want in it, its primary job it to call the setup() function with the appropriate parameters because it will be invoked by various tools in a standard way when installing your package. I'll go over the details in the next section.

The Configuration Files

In addition to setup.py, there are a few other optional configuration files that can show up here and serve various purposes.

Setup.py
The setup() function takes a large number of named arguments to control many aspects of package installation as well as running various commands. Many arguments specify metadata used for searching and filtering when uploading your package to a repository.

name: the name of your package (and how it will be listed on PYPI)
version: this is critical for maintaining proper dependency management
url: the URL of your package, typically GitHub or maybe the readthedocs URL
packages: list of sub-packages that need to be included; find_packages() helps here
setup_requires: here you specify dependencies
test_suite: which tool to run at test time
The long_description is set here to the contents of the README.md file, which is a best practice to have a single source of truth.

Setup.cfg
The setup.py file also serves a command-line interface to run various commands. For example, to run the unit tests, you can type: python setup.py test

1. running test
2. running egg_info
3. writing conman.egg-info/PKG-INFO
4. writing top-level names to conman.egg-info/top_level.txt
5. writing dependency_links to conman.egg-info/dependency_links.txt
6. reading manifest file 'conman.egg-info/SOURCES.txt' 
7. reading manifest template 'MANIFEST.in'
8. writing manifest file 'conman.egg-info/SOURCES.txt'
9. running build_ext
10. test_add_bad_key (conman_etcd_test.ConManEtcdTest) ... ok
11. test_add_good_key (conman_etcd_test.ConManEtcdTest) ... ok
12. test_dictionary_access (conman_etcd_test.ConManEtcdTest) ... ok
13. test_initialization (conman_etcd_test.ConManEtcdTest) ... ok
14. test_refresh (conman_etcd_test.ConManEtcdTest) ... ok
15. test_add_config_file_from_env_var (conman_file_test.ConmanFileTest) ... ok
16. test_add_config_file_simple_guess_file_type (conman_file_test.ConmanFileTest) ... ok
17. test_add_config_file_simple_unknown_wrong_file_type (conman_file_test.ConmanFileTest) ... ok
18. test_add_config_file_simple_with_file_type (conman_file_test.ConmanFileTest) ... ok
19. test_add_config_file_simple_wrong_file_type (conman_file_test.ConmanFileTest) ... ok
20. test_add_config_file_with_base_dir (conman_file_test.ConmanFileTest) ... ok
21. test_dictionary_access (conman_file_test.ConmanFileTest) ... ok
22. test_guess_file_type (conman_file_test.ConmanFileTest) ... ok
23. test_init_no_files (conman_file_test.ConmanFileTest) ... ok
24. test_init_some_bad_files (conman_file_test.ConmanFileTest) ... ok
25. test_init_some_good_files (conman_file_test.ConmanFileTest) ... ok
26. ----------------------------------------------------------------------
27. Ran 16 tests in 0.160s
28. OK

The setup.cfg is an ini format file that may contain option defaults for commands you pass to setup.py. Here, setup.cfg contains some options for nosetests (our test runner):

1. [nosetests]
2. verbose=1
3. nocapture=1

MANIFEST.in

This file contains files that are not part of the internal package directory, but you still want to include. Those are typically the readme file, the license file and similar. An important file is the requirements.txt . This file is used by pip to install other required packages.

Here is conman's MANIFEST.in file:

1. include LICENSE
2. include README.md
3. include requirements.txt

Dependencies

You can specify dependencies both in the install_requires section of setup.py and in a requirements.txt file. Pip will install automatically dependencies from install_requires, but not from the requirements.txt file. To install those requirements, you'll have to specify it explicitly when running pip: pip install -r requirements.txt.

Theinstall_requires option is designed to specify minimal and more abstract requirements at the major version level. The requirements.txt file is for more concrete requirements often with pinned down minor versions.

Here is the requirements file of conman. You can see that all the versions are pinned, which means it can be negatively impacted if one of these packages upgrades and introduces a change that breaks conman.

1. PyYAML==3.11
2. python-etcd==0.4.3
3. urllib3==1.7
4. pyOpenSSL==0.15.1
5. psutil==4.0.0
6. six==1.7.3

Pinning gives you predictability and peace of mind. This is especially important if many people install your package at different times. Without pinning, each person will get a different mix of dependency versions based on when they installed it. The downside of pinning is that if you don't keep up with your dependencies development, you may get stuck on an old, poorly performing and even vulnerable version of some dependency.

I originally wrote conman in 2014 and didn't pay much attention to it. Now, for this tutorial I upgraded everything and there were some major improvements across the board for almost every dependency.

Written by: Gigi Sayfan

If you found this post interesting, follow and support us.

Suggest for you:

Zero to Hero with Python Professional Python Programmer Bundle

The Python Mega Course: Build 10 Python Applications

Complete Python Bootcamp (Hot)

Learning Python for Data Analysis and Visualization

Learn Python for Beginners!

Speeding up Docker build times for Python applications.

I recently wrote a post where I talked about building a better user experience for deploying Python web applications. If one counts page hits as an indicator of interest in a subject then it certainly seems like an area people would like to see improvements.

In that post I talked about a system I was working on which simplified starting up a Python web server for your web application in your local environment, but also then how you can easily move to deploying that Python web application to Docker or OpenShift 3.

In moving to Docker, or OpenShift 3 (which internally also uses Docker), the beauty of the system I described was that you didn’t have to know how to create a Docker image yourself. Instead I used a package called S2I (Source to Image) to construct the Docker image for you.

What S2I does is use a Docker base image which incorporates all the system packages and the language run time environment you need for working in a specific programming language such as Python. That same Docker image also includes a special script which is run to incorporate your web application code into a new Docker image which builds off the base image. A further script within the image starts up an appropriate web server to run your web application. In the typical case, you don’t need to know anything at all about how to configure the web server as everything is done for you.

Docker build times

A problem that can arise any time you use Docker, unless you are careful, is how long it takes to actually perform the build of the Docker image for your web application. If you are making constant changes but need to rebuild the Docker image each time to test it, or redeploy it into a live environment, you could end up waiting quite a long time over the period of your work day. Decreasing the time it takes to build the Docker image can therefore be important.

The general approach usually followed is to very carefully craft your ‘Dockerfile’ so that it uses multiple layers, where the incorporation of parts which change most frequently are done last. By doing this, the fact that Docker will cache layers and start rebuilding only at the first layer changed, means you can avoid rebuilding everything every time.

This approach does break down though in various ways, especially with Python. The use of S2I can also complicate matters because it aims to construct the final image incorporating your application code, as well as all the dependent packages required by your application in a single Docker layer.

One issue with Python is the use of a ‘requirements.txt’ file and ‘pip’ to install packages. If you need to install a lot of packages and you add a single new package to the list, then all of them have to be reinstalled. Further, if those packages are being installed in the same layer as when your application code is being incorporated, as is the case with S2I, then a change to the application code causes all the packages to also be reinstalled.

So although S2I provides a really simple and clean way of constructing Docker images without you yourself needing to know how to create them, long build times are obviously not ideal.

As an example of how long a build time can be, consider the creation of a Docker image for hosting a Wagtail CMS site using Django. The ‘requirements.txt’ file in this case contains only:

1. Django>=1.9,<1.10
2. wagtail==1.3.1
3. psycopg2==2.6.1

Although this isn’t all that is installed. The complete list of packages which gets installed are:

1. beautifulsoup4==4.4.1
2. Django==1.9.2
3. django-appconf==1.0.1
4. django-compressor==2.0
5. django-modelcluster==1.1
6. django-taggit==0.18.0
7. django-treebeard==3.0
8. djangorestframework==3.3.2
9. html5lib==0.9999999
10. Pillow==3.1.1
11. psycopg2==2.6.1
12. pytz==2015.7
13. rcssmin==1.0.6
14. rjsmin==1.0.12
15. six==1.10.0
16. Unidecode==0.4.19
17. wagtail==1.3.1
18. wheel==0.29.0
19. Willow==0.2.2

Using my ‘warpdrive’ script from the previous blog post I referenced, it can take over 5 minutes over my slow Internet connection to bring down all the required Python packages, build them and construct the image.

1. (warpdrive+wagtail-demo-site) $ time warpdrive image wagtail
2. I0301 22:01:01.374459 16060 install.go:236] Using "assemble" installed from "image:///usr/local/s2i/bin/assemble"
3. I0301 22:01:01.374643 16060 install.go:236] Using "run" installed from "image:///usr/local/s2i/bin/run"
4. I0301 22:01:01.374674 16060 install.go:236] Using "save-artifacts" installed from "image:///usr/local/s2i/bin/save-artifacts"
5. ---> Installing application source
6. ---> Building application from source
7. -----> Installing dependencies with pip (requirements.txt)
8. Collecting Django<1.10,>=1.9 (from -r requirements.txt (line 1))
9. Downloading Django-1.9.2-py2.py3-none-any.whl (6.6MB)
10. Collecting wagtail==1.3.1 (from -r requirements.txt (line 2))
11. Downloading wagtail-1.3.1-py2.py3-none-any.whl (9.0MB)
12. Collecting psycopg2==2.6.1 (from -r requirements.txt (line 3))
13. Downloading psycopg2-2.6.1.tar.gz (371kB)
14. ...
15. Installing collected packages: Django, djangorestframework, Unidecode, Pillow, rcssmin, rjsmin, six, django-appconf, django-compressor, Willow, html5lib, django-taggit, pytz, django-modelcluster, beautifulsoup4, django-treebeard, wagtail, psycopg2
16. ...
17. Running setup.py install for psycopg2: finished with status 'done'
18. Successfully installed Django-1.9.2 Pillow-3.1.1 Unidecode-0.4.19 Willow-0.2.2 beautifulsoup4-4.4.1 django-appconf-1.0.1 django-compressor-2.0 django-modelcluster-1.1 django-taggit-0.18.0 django-treebeard-3.0 djangorestframework-3.3.2 html5lib-0.9999999 psycopg2-2.6.1 pytz-2015.7 rcssmin-1.0.6 rjsmin-1.0.12 six-1.10.0 wagtail-1.3.1
19. -----> Collecting static files for Django
20. ...
21. Copying '/opt/warpdrive/demo/static/js/demo.js'
22. ...
23. Copying '/usr/local/python/lib/python2.7/site-packages/django/contrib/admin/static/admin/img/gis/move_vertex_off.svg'
24. 179 static files copied to '/home/warpdrive/django_static_root'.
25. ---> Fix permissions on application source
26. real 5m40.780s
27. user 0m0.850s
28. sys 0m0.115s

If you were running ‘pip’ in your local environment and installing into a Python virtual environment, rerunning ‘pip’ on the ‘requirements.txt’ wouldn't be a big issue. This is because the packages would already be detected as being installed and so wouldn’t need to be downloaded and installed again. Even if you did blow away your Python virtual environment and recreate it, the downloaded packages would be in the cache that ‘pip’ maintains in your home directory. It could therefore just use those.

When creating Docker images however, you don’t get the benefit of those caching mechanisms because everything is done over every time. This means that all the packages have to be downloaded every time.

Using a wheelhouse

A possible solution to this is to create a wheelhouse. That is, you use ‘pip’ to create a directory of the packages you need to install as Python wheels. For pure Python packages these would just be that code, but if a Python package included C extensions, the Python wheel file would include the compiled code as object files. This means that the code doesn’t need to be recompiled every time and can simply be copied into place.

Although this can be done, working this into how you build your Docker images can get a bit messy as shown by Glyph in a blog post he wrote about it. It is therefore an area which is ripe for being simplified and so I have also been working that into what I have been doing with trying to simplify the deployment of web applications. In this post I want to show how that is progressing.


First step now is to create a special Docker image which acts as our Python wheelhouse. This can be done by running the following command.

1. (warpdrive+wagtail-demo-site) $ warpdrive image --build-target wheelhouse wagtail-wheelhouse
2. I0301 23:03:32.687290 17126 install.go:236] Using "assemble" installed from "image:///usr/local/s2i/bin/assemble"
3. I0301 23:03:32.687446 17126 install.go:236] Using "run" installed from "image:///usr/local/s2i/bin/run"
4. I0301 23:03:32.687475 17126 install.go:236] Using "save-artifacts" installed from "image:///usr/local/s2i/bin/save-artifacts"
5. I0301 23:03:32.709215 17126 docker.go:286] Image "wagtail-wheelhouse:latest" not available locally, pulling ...
6. ---> Installing application source
7. ---> Building Python wheels for packages
8. -----> Installing dependencies as wheels with pip (requirements.txt)
9. Collecting Django<1.10,>=1.9 (from -r requirements.txt (line 1))
10. Downloading Django-1.9.2-py2.py3-none-any.whl (6.6MB)
11. Saved ./.warpdrive/wheelhouse/Django-1.9.2-py2.py3-none-any.whl
12. Collecting wagtail==1.3.1 (from -r requirements.txt (line 2))
13. Downloading wagtail-1.3.1-py2.py3-none-any.whl (9.0MB)
14. Saved ./.warpdrive/wheelhouse/wagtail-1.3.1-py2.py3-none-any.whl
15. Collecting psycopg2==2.6.1 (from -r requirements.txt (line 3))
16. Downloading psycopg2-2.6.1.tar.gz (371kB)
17. ...
18. ---> Fix permissions on application source

This command is going to run a bit differently to the command above. Rather than use ‘pip install’ to install the actual packages, it will run ‘pip wheel’ to create the Python wheels we are after. At that point it will stop, as we don’t need it do additional steps such as run ‘collectstatic’ for Django to gather up static file assets. This will still take up to 5 minutes though since the bulk of the time was involved in downloading and building the packages.


Once we have our wheelhouse, when building the Docker image for our application, we can point it at the wheelhouse as a source for the prebuilt Python packages we want to install. We can even tell it to take what it provides as the authority and not consult the Python package index (PyPi) to check whether there aren’t newer versions of the packages when packages haven’t been pinned to a specific package.

1. (warpdrive+wagtail-demo-site) $ time warpdrive image --wheelhouse wagtail-wheelhouse --no-index wagtail
2. warpdrive-image-17312
3. I0301 23:12:54.610882 17329 install.go:236] Using "assemble" installed from "image:///usr/local/s2i/bin/assemble"
4. I0301 23:12:54.611033 17329 install.go:236] Using "run" installed from "image:///usr/local/s2i/bin/run"
5. I0301 23:12:54.611089 17329 install.go:236] Using "save-artifacts" installed from "image:///usr/local/s2i/bin/save-artifacts"
6. ---> Installing application source
7. ---> Building application from source
8. -----> Found Python wheelhouse of packages
9. -----> Installing dependencies with pip (requirements.txt)
10. Collecting Django<1.10,>=1.9 (from -r requirements.txt (line 1))
11. Collecting wagtail==1.3.1 (from -r requirements.txt (line 2))
12. Collecting psycopg2==2.6.1 (from -r requirements.txt (line 3))
13. ...Installing collected packages: Django, Unidecode, pytz, django-modelcluster, djangorestframework, Pillow, django-treebeard, django-taggit, six, Willow, rjsmin, django-appconf, rcssmin, django-compressor, beautifulsoup4, html5lib, wagtail, psycopg2
14. Successfully installed Django-1.9.2 Pillow-3.1.1 Unidecode-0.4.19 Willow-0.2.2 beautifulsoup4-4.4.1 django-appconf-1.0.1 django-compressor-2.0 django-modelcluster-1.1 django-taggit-0.18.0 django-treebeard-3.0 djangorestframework-3.3.2 html5lib-0.9999999 psycopg2-2.6.1 pytz-2015.7 rcssmin-1.0.6 rjsmin-1.0.12 six-1.10.0 wagtail-1.3.1
15. -----> Collecting static files for Django
16. ...
17. Copying '/opt/warpdrive/demo/static/js/demo.js'
18. ...
19. Copying '/usr/local/python/lib/python2.7/site-packages/django/contrib/admin/static/admin/img/gis/move_vertex_off.svg'
20. 179 static files copied to '/home/warpdrive/django_static_root'.
21. ---> Fix permissions on application source
22. real 0m45.859s
23. user 0m3.555s
24. sys 0m2.575s

With our wheelhouse, building of the Docker image for our web application has dropped from over 5 minutes down to less than a minute. This is because when installing the Python packages, it is able to reuse the pre built packages from the wheelhouse. This means a quicker turnaround for creating a new application image. We will only need to rebuild the wheelhouse itself if we change what packages we need to have installed.

Incremental builds

Reuse therefore allows us to speed up the building of Docker images considerably where we have a lot of Python packages that need to be installed. The reuse of previous builds can also be used in another way, which is to reuse the prior wheelhouse itself when updating the wheelhouse after changes to the list of packages we need.

1. (warpdrive+wagtail-demo-site) $ time warpdrive image --build-target wheelhouse wagtail-wheelhouse
2. I0301 23:18:24.150533 17448 install.go:236] Using "assemble" installed from "image:///usr/local/s2i/bin/assemble"
3. I0301 23:18:24.151074 17448 install.go:236] Using "run" installed from "image:///usr/local/s2i/bin/run"
4. I0301 23:18:24.151121 17448 install.go:236] Using "save-artifacts" installed from "image:///usr/local/s2i/bin/save-artifacts"
5. ---> Restoring wheelhouse from prior build
6. ---> Installing application source
7. ---> Building Python wheels for packages
8. -----> Installing dependencies as wheels with pip (requirements.txt)
9. Collecting Django<1.10,>=1.9 (from -r requirements.txt (line 1))
10. File was already downloaded /opt/warpdrive/.warpdrive/wheelhouse/Django-1.9.2-py2.py3-none-any.whl
11. Collecting wagtail==1.3.1 (from -r requirements.txt (line 2))
12. File was already downloaded /opt/warpdrive/.warpdrive/wheelhouse/wagtail-1.3.1-py2.py3-none-any.whl
13. Collecting psycopg2==2.6.1 (from -r requirements.txt (line 3))
14. Using cached psycopg2-2.6.1.tar.gz
15. ...
16. ---> Fix permissions on application source
17. real 1m17.180s
18. user 0m3.653s
19. sys 0m2.316s

Here we have run the exact same command as we ran before to create the wheelhouse in the first place, but instead of taking 5 minutes to build, it has taken just over 1 minute.

This speed up was achieved because we were able to copy across the ‘pip’ cache as well as the directory of Python wheel files from the previous instance of the wheelhouse.

Not a Dockerfile in sight

Now what you didn’t see here at all was a ‘Dockerfile’. For me this is a good thing.

The problem with Docker right now is that the novelty still hasn’t warn off, with it still not being seen for what it is, just another tool we can use. As a result we are still in this phase where developers using Docker like to play with it and so try and do everything themselves from scratch. We need to get beyond that phase and start incorporating best practices into canned scripts and systems and simply get on with using it.

Anyway, this is where I am at least heading with the work I am doing. That is, encapsulate all the best practices for Python web application deployment, including the building of Docker images which you can run directly, or with a PaaS using Docker such as OpenShift. The aim here being to make it so much easier for you, with you knowing that you can trust that the mechanisms have been put together will all the best practices being followed. After all, do you really want to keep reinventing the wheel all the time?

Written by: Graham Dumpleton  

The Python Mega Course: Build 10 Python Applications

Complete Python Bootcamp (Hot)

Learning Python for Data Analysis and Visualization 

Learn Python for Beginners!

Data Science: Deep Learning in Python