Loading
3 comments

Fixing Custom Tags with Django 1.2

When I upgraded to version 1.4.2 of the AppEngine SDK, I was both intrigued and alarmed to see the warning message that I ought to be using use_library() to specify a version of Django that my application would use. On one hand, I was glad that I could switch to version 1.2, as it provides dramatically-improved if tags over 0.96. On the other hand, I knew that I would be in for a certain amount of pain because things were going to break. The first thing that broke for me was the call to template.register_template_library that my code uses to load a library of custom template tags. It crashed with the extraordinary helpful stack trace:

  webapp.template.register_template_library('MVCTags')
File "C:\google_appengine\google\appengine\ext\webapp\template.py", line 269, in register_template_library
   django.template.add_to_builtins(package_name) 
File "C:\google_appengine\lib\django_1_2\django\template\__init__.py", line 1049, in add_to_builtins
   builtins.append(import_library(module))
File "C:\google_appengine\lib\django_1_2\django\template\__init__.py", line 984, in import_library
   app_path, taglib = taglib_module.rsplit('.',1)
ValueError: need more than 1 value to unpack 

After a great deal of stepping through code in Wing IDEs superb Debugger, I figured out that the register_template_library call had changed. Instead of taking an absolute path to the file to load, it expects a Python Module style path like we use with an import statement.

The fix was simple. I created a directory called ‘tags’ and moved my custom tags library file (MVCTags.py) there. I also added an empty __init__.py file to make it a Module. Finally, I changed the code to template.register_template_library(‘tags.MVCTags’).

With that out of the way, I have moved to fixing the next big problem: a change to the way that template loaders work has broken many of the include statements in my templates. It looks like I am going to be writing a custom template loader, but figuring out how to get it into the collection of template loaders looks daunting.

3 comments

A Better Sharded Counter
My current AppEngine project was crying out for some counters to track site-wide instances of various models, and I recalled watching Brett Slatkin’s video about building highly-scalable web apps. A few seconds of quality Google time later, and I had the ShardCounter classes ready to go. The same code is available in several locations:
http://code.google.com/p/google-app-engine-samples/source/browse/trunk/sharded-counters/generalcounter.py

One thing that quickly caught my attention is that these classes only support incrementing, and while that makes sense for something like a primitive visit counter, it didn’t handle my needs very well at all. My initial attempt to simply copy the increment function and change the critical += to a -= was naive and doomed to failure, but a little tinkering with the way that counts are recorded gave me a nice working solution that completely preserves the desirable performance characteristics of this approach.

Here’s the code that I came up with. Please feel free to use it in your own projects.
from google.appengine.api import memcache
from google.appengine.ext import db
import random

# This code unabashedly stolen from Google
# http://code.google.com/appengine/articles/sharding_counters.html#counter_python

class GeneralCounterShardConfig(db.Model):
    """Tracks the number of shards for each named counter."""
    name = db.StringProperty(required=True)
    num_shards = db.IntegerProperty(required=True, default=20)


class GeneralCounterShard(db.Model):
    """Shards for each named counter"""
    name = db.StringProperty(required=True)
    "The name of the counter."
    
    plus = db.IntegerProperty(required=True, default=0)
    "The number of times that the counter has been incremented."
    
    minus = db.IntegerProperty(required=True, default=0)
    "The number of times that the counter has been decremented."


def get_count(name):
    """Retrieve the value for a given sharded counter.

    Parameters:
      name - The name of the counter
    """
    total = memcache.get(name)
    if total is None:
        total = 0
        for counter in GeneralCounterShard.all().filter('name = ', name):
            total += counter.plus
            total -= counter.minus
        memcache.add(name, str(total), 60)
    return total


def increment(name):
    """Increment the value for a given sharded counter.

    Parameters:
      name - The name of the counter
    """
    config = GeneralCounterShardConfig.get_or_insert(name, name=name)
    def txn():
        index = random.randint(0, config.num_shards - 1)
        shard_name = name + str(index)
        counter = GeneralCounterShard.get_by_key_name(shard_name)
        if counter is None:
            counter = GeneralCounterShard(key_name=shard_name, name=name)
        counter.plus += 1
        counter.put()
    db.run_in_transaction(txn)
    memcache.incr(name)

def decrement(name):
    """Decrement the value for a given sharded counter.

    Parameters:
      name - The name of the counter
    """
    config = GeneralCounterShardConfig.get_or_insert(name, name=name)
    def txn():
        index = random.randint(0, config.num_shards - 1)
        shard_name = name + str(index)
        counter = GeneralCounterShard.get_by_key_name(shard_name)
        if counter is None:
            counter = GeneralCounterShard(key_name=shard_name, name=name)
        counter.minus += 1
        counter.put()
    db.run_in_transaction(txn)
    memcache.decr(name)

def increase_shards(name, num):
    """Increase the number of shards for a given sharded counter.
    Will never decrease the number of shards.

    Parameters:
      name - The name of the counter
      num - How many shards to use

    """
    config = GeneralCounterShardConfig.get_or_insert(name, name=name)
    def txn():
        if config.num_shards < num:
            config.num_shards = num
            config.put()
    db.run_in_transaction(txn)
0 comments

Please, somebody talk me out of this

I'm strongly mulling the idea of building an IDE tailored for creating AppEngine applications.  I'd start with Scintilla.NET for the text-editing component, add as much Intellisense-like help for Python code as I could manage and hook it up to a debugger and the GAE development web server.

Now, there are plenty of IDEs that will help you write Python.  I have been using Komodo Edit, and it does a lot of things very well, but it did not enable me to do source-level debugging, and I just gotta have that.  MUST HAVE.  Developing any other way is painful and unproductive and horribly inefficient.  It's difficult enough to write in a dynamically-typed language, using both a language and a framework that have cringe-worthy documentation (that includes you, Django), but to have to rely on logs to debug you code makes it too painful for serious projects.

However, I'm so impressed with the upside of Google AppEngine , and I'm so sure that it is going to be a meaningful and important web platform, I'm considering taking yet another iron and jamming it into my already-sputtering-and-nearly-out fire.  This thing is absolutely going to be a part of my professional life, so investing some time and energy into making it more reasonable to use seems like it just might be wise.

But please, somebody, talk me out of this.

0 comments

Another release -- this time with all new tags!

I've just uploaded another release of my blog software, and this time it has TAGS!  Tags, of course, are clickable links, and you can use them to search for all posts that also have the given tag.  Cool, I think.

I'm rather proud of the fact that I wrote the tagging-related classes to be completely transportable to anyone else's Google AppEngine code, and integrating this functionality is as easy as making the taggable class a super-class of any Model class.

I'll update this entry once I have release the tagging code as a Google Code open source project.  I might see if I can run it by someone who is a more experienced Pythonista, so I can get a quick sanity check.  I'm just the tiniest bit nervous that while my code might be valid Python, it might not observe common idioms of the language.

0 comments

Still learning, still liking

I'm continuing to work on this blog software, and I am learning a lot about AppEngine, Python and Django.  I used Django's include function to refactor the rendering of posts.  That removed some code duplication.  Got to love that.  I feel like I've already done a full day's work, and it's only 10:30!

My inclination is next to look at extending Python's WSGIHandler further to make it more MVC-like.  I'd never paid much attention to that design pattern before I started using Ruby on Rails, but I'm quite addicted to it now, and doing the extra work -- and admittedly, it's not that much -- to do it the AppEngine way irks me.  I've already taken a step in that direction, subclassing RequestHandler to do all the work of creating a path to the appropriate template file and passing it to the render method, but I know that there's a lot more to do.  I also feel like I could refactor the handling of the dictionary that passes values to the template to reduce the repititious setting of values that happens in each handler.

0 comments

Live at Last!

Learning to create Google App Engine applications via building this simple blog has been hugely fun and not just a little bit frustrating.  Many, many thanks to Google's Marzia for so diligently helping me to figure out the missing piece that kept me from being able to upload this code.

I've come away from the experience feeling very enthusiastic about App Engine.  It's not going to replace Ruby on Rails as my web development platform of choice, but I have to admit that GAE makes it very, very easy to quickly build and deploy an application.  I managed to become a pretty competant Python programmer in just a few days, and I know enough about Django to build a small web application that observes the DRY principle.

I became very frustrated that Django is a whole seperate language unto itself, so I had to learn Python and Django.  I suppose that I have become spolied by Rails, which allows me to write the logic, data and presentation layers in one language.  It's so simple, beautiful and elegant.

That's not neccesarily an argument against GAE, which it seems is going to support multiple language and frameworks as it matures.  The combination of Ruby on Rails's power with Google's essentially infinite scalability would be a mighty and compelling thing.