Re: Why Django Sucks

I recently came across this presentation on 'Why Django Sucks' and the associated discussion in Hacker News.

One of the key points raised was the perceived insularity of Django's core development community. In this context I have an anecdote to share about the core community or rather how the core community is frequently sloppy with the triage process.

I reported a Django bug and more importantly submitted tests for it. The bug languished for a while and eventually got fixed as a side effect of some changes. I reported this when I found out and a month later someone closed it. The regression tests, however, never made it to the test suite.

I believe that (1) people who take their time to give you a test are doing you a favor (2) one should never miss a chance to add a valid regression test to the suite.

I don't think there was any malice or hubris but the closed nature of the community did have something to do with this. I suspect that how fast and how thoroughly a bug is accepted and processed depends on who submits it. This will prove unhealthy in the long run.

So what can be done to address this? I believe that the bug triage process will benefit from more eyes. These don't have to be those of core developers but of people with the authority to take decisions on bugs. The community should also actively encourage people to submit more tests. Spread the word that this is as important as submitting patches.

Ticket #11627

Logged another Django ticket related to test client. I find that most of my Django pet peeves are related to two areas: the ORM and testing framework.

DRY Logging in Django

I have found custom logging a very useful tool for dealing with production issues. Especially when the users are not easily reachable. I recently configured custom logging messages for a Django project I have been working on. As logging is usually only initiated once per application I chose to do it in my settings.py. The code looked something like this:
import logging import logging.handlers logger = logging.getLogger('project_logger') logger.setLevel(logging.INFO) LOG_FILENAME = '/path/to/log/file/in/development' handler = logging.handlers.TimedRotatingFileHandler(LOG_FILENAME, when = 'midnight') formatter = logging.Formatter(LOG_MSG_FORMAT) handler.setFormatter(formatter) logger.addHandler(handler)
This configuration worked out well in development. However I maintain a separate settings file for production, production.py. Production.py merely imports everything from settings.py and selectively overrides some of the attributes (set DEBUG = False, for instance).

My problem was that I wanted the production logs to be written to a different location and perhaps even use a different handler. I could not think of a good way to do this without calling the logging code fragment again in production.py. I wrote a utility function to do this (configure_logging(filename)). This reduced code repetition but revealed another problem. When production.py imports everything from settings.py it triggers logging to be configured once using the LOG_FILENAME attribute used in development and then *again* using the LOG_FILENAME attribute used in production.

This was ugly. For one there is no need to configure logging twice; for another the initial configuration throws an error if the path described by LOG_FILENAME in development is not present in the production machine.

I wasn't sure how to proceed and I posed this question at Stack Overflow. I got an answer suggesting to switch LOG_FILENAME based on the DEBUG attribute. Something like this:
# Inside settings.py if DEBUG: LOG_FILENAME = '/path/to/log/file/in/development' else: LOG_FILENAME = '/path/to/log/file/in/production' # Configure logging here.
This looked good, but almost immediately revealed a problem. DEBUG is overridden *inside* production.py whereas this code snippet is inside settings.py. Consequently when the above given snippet gets executed DEBUG is still True and LOG_FILENAME will be pointed at the development environment.

I eventually found a solution from an answer to a different question. User Bluebird75 suggested using the 'module singleton' pattern to ensure that logging is only configured once. I extended his(?) suggestion to come up with the following solution:
# Inside settings.py LOG_FILENAME = '/path/to/log/file/in/development' # Inside production.py from settings import * LOG_FILENAME = '/path/to/log/file/in/production' # Singleton module log.py import logging, logging.handlers from django.conf import settings LOGGING_INITIATED = False def init_logging(): logger = logging.getLogger('project_logger') logger.setLevel(logging.INFO) handler = logging.handlers.TimedRotatingFileHandler(settings.LOG_FILENAME, when = 'midnight') formatter = logging.Formatter(LOG_MSG_FORMAT) handler.setFormatter(formatter) logger.addHandler(handler) if not LOGGING_INITIATED: LOGGING_INITIATED = True init_logging()
The singleton module ensures that logging is only configured once. As this module resides within the app directory Django would already have loaded settings before this module is loaded, thereby ensuring the presence of settings.LOG_FILENAME. I can add as many settings files as I want and all I have to do is override the LOG_FILENAME attribute in each file.

Ticket #11475

Logged a Django bug related to test client. Part of my continuing series of adventures in Django.

Quick and dirty serial numbers across pages in Django templates

Update:

Scratch that. I just learned that this can be done in a much simpler way:
{% for object in object_list %} {{ page_obj.start_index|add:forloop.counter0 }} {% endfor %}
I recently had to write a template for a paginated view which displayed a serial number for each object in the object_list. I normally use forloop.counter for general purpose serial numbers. However this did not work with paginated views as the counter gets reset in each page. This caused the serial numbers to go from 1 to #-of-results-in-the-page and then repeat. I wrote a filter to tackle this problem.

The code (add to your filters file under ./templatetags/):
from django import template from django.conf import settings register = template.Library() @register.filter def adjust_for_pagination(value, page): value, page = int(value), int(page) adjusted_value = value + ((page - 1) * settings.RESULTS_PER_PAGE) return adjusted_value
And the template snippet:
{% for object in object_list %} <div class="serial-no"> {% if is_paginated %} {{ forloop.counter|adjust_for_pagination:page }} {% else %} {{ forloop.counter }} {% endif %} </div> ... {% endfor %}
The adjust_for_pagination filter adjusts the value of forloop.counter based on the current page. Page and is_paginated variables are expected to be present in the context. These should respectively denote the current page number (1 based) and if the results are paginated. RESULTS_PER_PAGE is currently taken from the settings file. I couldn't think of a way to pass this value also from the template.

Quick and dirty 'type' for Django's message model

In an earlier post I had mentioned using a 'flash' variable passed to templates as a mechanism for passing messages to users. Following an anonymous tip I started using user.message_set instead to pass messages. This mechanism makes use of Django's message model and is easy to use. However it does have a limitation. Messages cannot be assigned types - say, 'Error' or 'Success' or 'Warning'. This has been discussed at some length at the Django site. A design decision is awaited as the suggested changes will not be backward compatible.

In the meantime I needed message types for a pet application and came up with a quick and dirty way of providing some. Changes were required in three places: the view, where messages are created; the template which displays the messages and the style sheet used for styling the rendered templates.

The solution (did I say it was dirty?) involved prefixing a digit and a separator to all messages. The digit would indicate the type of message. I chose 0 to indicate success, 1 for warnings, 2 for errors etc. Corresponding styles were then created to appropriately style the messages.

Here is a sample view: def save_something(request): if saved: request.user.message_set.create(message = "0|Object was saved successfully") else: request.user.message_set.create(message = "2|Object could not be saved") The template looks something like this: {% if messages %} <div id="messages"> {% for message in messages %} <p class="m{{ message|slice:"0:1"}}">{{ message|slice:"2:" }}</p> {% endfor %} </div> {% endif %} There is a corresponding style sheet as well: #messages p { margin: 0; padding: 0; } #messages p.m0 { background: transparent url(success.gif) 0 100% no-repeat; color: green; } #messages p.m1 { background: transparent url(warning.gif) 0 100% no-repeat; color: yellow; } #messages p.m2 { background: transparent url(failure.gif) 0 100% no-repeat; color: red; } /* etc */

Django Testing Gotchas

I recently started using Django's newly introduced capability to make use of test fixtures. In order to do this all test classes that need fixtures have to be subclasses of Django's custom TestCase class rather than the customary unittest.TestCase. I had written about 17 or so test cases split across two test classes. One of these test classes used Django's fixtures.

Each Django test run (kicked off by running python manage.py test) involves the following steps:

1. Create test database
2. Create the necessary tables
3. Load "initial fixtures" if any. These have to be located in a file named initial_data with suitable extension denoting fixture type (.xml, .json etc.)
4. Load fixtures specified in each test case
5. Execute any one test
6. Remove fixtures loaded in step #4
7. Repeat steps 4 through 6 until all tests have been executed.

Sounds nifty, doesn't it? That is what I thought too, until I ran into reality. All goes well until step 3. After that, the tests take an eternity to execute. OK, it took 159 seconds to execute 17 puny tests. I was surprised. My application was rather simple, involving less than a dozen tables and a few hundreds of rows. And I was only using a subset of the data to test. What was going on? I looked around and found that I was not the only one to face this problem.

After reading up more about the problem I figured that I could probably cut down on the time taken by "refactoring" my tests and moving unnecessary fixtures out to "initial fixtures" so that they would only be loaded once. Luckily this would work in my case as my app only used the data for reference; it did not modify any. I was therefore not constrained to load the data before each test.

I made the necessary changes and re-ran my test cases. The time taken to execute the test cases improved ever so slightly. It now took about ~140 seconds.

I was still unhappy with the results. Since I was not using any fixtures, I decided to substitute Django's TestCase with unittest.TestCase. And voilà! It was as I had engaged hyper drive. 24 tests now took a mere 0.2 seconds to run!

Naturally the question that came to my mind was that in the absence of fixtures, what on earth was taking up so much time? Django is known for its speed vis a vis Rails; I had come to expect the same kind of superior performance while running tests too. I don't have an answer right now. I hope to come up with one after I take a look at Django's innards.

Moving from Rails to Django: Replacing flash[:notice]

One of the most common practices I came across while learning Rails is the flash[:notice] mechanism used to send (error) messages to the template. Consequently it was also one of my first targets to migrate when I started moving a pet application from Rails to Django.

I came up with a simple (and all too obvious) way to send error messages to the template. I am sure that there are better ways to do this. If you know of any, please take a minute to share them.

First, the Rails snippet. def index ... begin army = Army.find(params[:army][:id]) rescue StandardError => se flash[:notice] = "Vare, legiones redde!" return end ... end Next, the equivalent in Django. from django.shortcuts import render_to_response def index(request): ... try: army = Army.objects.get(id = request.POST["army"]) except ( KeyError, Army.DoesNotExist ): return render_to_response( "some_template.html", { "flash" : "Vare, legiones redde!" } ) ... The template in Rails ... ... <% if @flash[:notice] -%> <div id="flash"> <div><%= @flash[:notice] %></div> </div> <% end -%> ... ... and the template in Django. ... {% if flash %} <div id="flash"> <div>{{ flash }}</div> </div> {% endif %} ... While testing in Django, flash can be treated just like any other context variable.

Moving from Rails to Django: Migrating Tests

Context
I am in the process of migrating one of my small "pet" applications developed in Ruby on Rails to Python using Django. One of my first concerns was migrating tests, both unit (testing models) and functional (testing 'controllers' in Rails/'views' in Django) from one framework to the other. After looking around the documentation, asking questions and conducting some experiments I have some answers. Here is a listing of what I have found out so far.

Note
These examples will only work with the latest development version of Django.

Locating your tests
Rails
Usually located under the path [path_to_app]/test/functional/. There is usually one test source per controller, named as [controller_source_without_extension]_test.rb. For example if the source for a controller, say, 'Generator' is written in [path_to_app]/app/controllers/generator_controller.rb, then the corresponding test should be present in a file named generator_controller_test.rb

Django
The easiest way (see the official documentation) is to add all your tests to a file called tests.py and drop it in the application folder, at the same level as your models.py and views.py.

However I prefer to have a separate test source structure if I can help it. This has the added advantage of keeping I decided to adopt the convention used by Rails and came up with the following tree:

• [project]/[application]/test/functional/ for functional tests and
• [project]/[application]/test/unit/ for model tests.

Don't forget to add the __init__.py files to the tree as applicable to make these accessible as python modules. You will still need to retain the tests.py except that you will have to add a line to import your test case classes and nothing else. Test runner will automatically pick up the test classes and run them. # GeneratorTests is a subclass of unittest.TestCase from django_iwg.btd.test.functional.generator_tests import GeneratorTests
Fixtures
Fixtures are readily available in Rails. There are not quite there yet in Django and I am as righteously angry as the next ex-Rails user ;) But do not worry, they are on their way and will be available shortly. I am experimenting with a really dirty solution in the meantime. More on this in a subsequent post.

Writing Tests - setup
Rails
A typical setup() method in Rails looks like this: require File.dirname(__FILE__) + '/../test_helper' require 'generator_controller' class GeneratorControllerTest < Test::Unit::TestCase def setup @controller = GeneratorController.new @request = ActionController::TestRequest.new @response = ActionController::TestResponse.new end end Django
The framework provides a test client to simulate browser calls. The best place to create a test client object would be the setup method. import unittest from django.test.client import Client class GeneratorTests(unittest.TestCase): def setUp(self): self.client = Client()
Writing Tests - GET requests
Rails
My first objective was to test if a GET request to the controller/URL 'generator/index' was being handled properly. "Properly" in this case means three things:

• Ensure that the response came back with an HTTP status of 200 OK;
• Ensure that the appropriate template was used; and
• Ensure that instance variables were present and had expected values.

The resulting test looked like this: def test_GET_index_returns_200_OK get :index assert_template "generator/index" assert_response :success assert_equal assigns(:campaigns).length, 14 end
Django
While the objectives remain the same, the steps vary. The key differences I noticed were:

• There are no built-in wrapper methods for GET/POST. It should be noted that it is easy to enough to code wrappers should one so wish.
• Likewise there are no equivalents for wrappers such as "assert_template" or "assert_response"
• Django uses a different philosophy when it comes to passing variables to templates. Instance variables are not directly passed to templates; rather they have to be explicitly passed using a context object or a call to locals()

The resultant code looks like this: def test_GET_index_returns_200_OK(self): response = self.client.get("/generator/index/") self.failUnlessEqual(response.status_code, 200) self.failUnlessEqual(response.template.name, "generator/index.html") self.failUnlessEqual(len(response.context["campaigns"]), 14)
Writing Tests - POST requests
Rails
A simple "happy path" POST request test looks like this in Rails: def test_POST_index_saves_army_in_session_and_redirects post :index, :army => {:id => 1} assert_equal 1, get_session_variable(:army).id assert_redirected_to(:action => "number_of_units") end Here, in addition to checking the response I am also verifying that a particular session variable was set properly.

Django
Here again we find that there are no wrappers for session and that it is easy enough to add some. The session object itself is readily available and can be accessed using the test client. def test_POST_index_saves_army_in_session_and_redirects(self): response = self.client.post("/generator/index", {"army" : 1}) self.failUnlessEqual(response.status_code, 302) self.failUnlessEqual(response.headers["Location"], "/generator/number_of_units/" ) self.assertEqual(self.client.session["army"].id, 1)
Running Tests
Rails
The easiest way is to execute ./rake from the command line. There are separate tasks available (such as test:functional, for instance) in case you wish to run the functional tests only.

Django
The easiest way is to execute ./manage.py test from the command line. This will run the tests for all applications in the project. If you wish to test only one application, feed its name as an additional parameter.

Django Template Nuances
Django provides simple inheritance mechanism for templates. I personally like it :) What should be remembered is that using inheritance changes the testing process slightly. For instance, consider a view index that renders a template say, index.html. This template inherits from another, named base.html. A GET request test for the view would look like this: def test_GET_index_returns_200_OK(self): response = self.client.get("/generator/index/") ... template_index = [template.name for template in response.template].index("generator/index.html") self.failUnlessEqual(len(response.context[template_index]["campaigns"]), 14) Note the difference from the earlier test. When template inheritance comes into play, response.template will resolve into a list of all the templates involved in the hierarchy, listed parent first. Corresponding to each template, there will be a separate context object at available at the same index in response.context (which will also resolve into a list)