Tips and Tricks for Django on Google App Engine

Recently I was tasked with creating a dajngo app which runs on Google App Engine (GAE). Now anyone familiar with GAE knows that they don’t fully support relational/SQL storage just yet (Google Cloud SQL is in an experimental phase at this time). Instead they have something called App Engine Datastore which is a schemaless object storage (NoSQL of sorts). So to get django to play nice with this storage, good people at ABP created a fork - django-nonrel which runs seamlessly out of the box on GAE. Easy enough.

Although there were couple of things which I tried to do that were not so straight forward. I’m just going to document those here in hopes that someone will find it useful.

Uploading Images

First up, how about uploading images?

Well in that case you’ll have to install the following libraries:

django-filetransfers - It’s an abstraction layer which allows for uploads into remote datastores, like AppEngine or S3 using django’s standard models.FileField

But what if you wanted to use models.ImageField instead of models.FileField? Since ImageField depends on PIL which is not available on GAE, it won’t work out of the box. You will need to install a mock PIL class library which handles the 3 functions necessary for the ImageField to validate: open(), read(), and verify().

Next up, what if you wanted to upload images through django admin? Well in most likelihood you probably encountered an error like this while trying to do so:

ValueError: The App Engine storage backend only supports BlobstoreFile instances or File instances whose file attribute is a BlobstoreFile.

Luckily I found a fork of the filetransfers library here which adds support for the admin. So instead of using the original package, just use the fork. Then all you have to do is, inherit the admin class from FiletransferAdmin in your admin.py:

from filetransfers.admin import FiletransferAdmin
from myapp.models import MyModel

class MyModelAdmin(FiletransferAdmin):    
    pass

admin.site.register(MyModel, MyModelAdmin)

Volia! That’s all you need to do to get image uploads working correctly using django-nonrel admin on GAE!

Search

Due to the NoSql nature of the GAE storage some of the things you would expect like case-insensitive queries (iexact, istartswith, etc.), JOINs, etc. won’t work as expected. ABP strikes once again with dbindexer which abstracts away most of those differences and lets you use some of that functionality just like how you would in regular django. Also, if you are looking for full-text search support, there’s a great package for that as well - nonreal-search

Also keep in mind that ABP no longer supports or maintains any of these libraries so it’s best to get the latest source from the official Github account - https://github.com/django-nonrel

Hopefully in the future all this wont be necessary as GAE support for django becomes more robust. Although you can still give it a spin while its still experimental: https://developers.google.com/appengine/docs/python/cloud-sql/django

Happy hacking!

My thoughts exactly

This is from Tilo Mitra’s blog post where he featured this comment below which was in response to another blog post. He calls it crazy world of code.

I just have to say I couldn’t agree more:

I agree, I can’t keep up. I just finished learning backbone.js and now I’m found out on that it’s old news, and I should use ember.js, cross that, it has opinions, I should use Meteor, no, AngularJS, no, Tower.js (on node.js), and for html templates I need handlebars, no mustache, wait, DoT.js is better, hang on, why do I need an HTML parser inside the browser? isn’t that what the browser for? so no HTML templates? ok, DOM snippets, fine, Web Components you say? W3C are in the game too? you mean write REGULAR JavaScript like the Google guys? yuck, oh, I just should write it with CofeeScript and it will look ok, not Coffee? Coco? LiveScript? DART? GWT? ok, let me just go back to Ruby on Rails, oh it doesn’t scale? Grails? Groovy? Roo? too “Springy?” ok, what about node.js? doesn’t scale either?? but I can write client side, server side and mongodb side code in the same language? (but does it have to be JavaScript?) ok, what about PHP, you say it’s not really thread safe? they lie?? ok, let me go back to server coding, it’s still Java right? no? Lisp? oh it’s called Clojure? well, it has a Bridge / protocol buffers / thrift implementation so we can be language agnostic, so we can support our Haskell developers. Or just go with Scala/Lift/Play it’s the BEST framework (Foresquare use it, so it has to be good). of course we won’t do SOAP and will use only JSON RESTful services cause it’s only for banks and Walmart, and god forbid to use a SQL database it will never scale

How to obtain JSON/RSS feed for Facebook Pages without OAuth/GraphAPI

Facebook killed their publicly available RSS feed for Facebook Pages couple of years back in favor of their walled garden approach (GraphAPI). For example, if you go to http://graph.facebook.com/Nike/feed it will ask you for a valid access token which you can get obtain via OAuth Authorization which entails quite a few steps. If you just wanted to display/parse any Facebook Page feed without going through those hoops you are out of luck!

I found myself in a similar situation recently and after some Googling I was able to figure out a workaround. Here’s how it goes:

1) Go to https://developers.facebook.com/tools/explorer

2) Put the username/vanity url of the Facebook Page in question in the request bar and hit Submit. For example, if you wanted to get a feed for: https://www.facebook.com/nike you would put “nike” without quotes in the request bar and click on Submit. (Make sure GET request is selected in the drop-down)

3) Grab the id from the results below and paste it in this URL: https://www.facebook.com/feeds/page.php?id=[FACEBOOK-ID-GOES-HERE]&format=[rss20|json] So for example, if we wanted JSON feed for Nike’s Facebook page, we’d go to: https://www.facebook.com/feeds/page.php?id=15087023444&format=json

BONUS:

You can do the same with Twitter (officially supported):

http://api.twitter.com/1/statuses/user_timeline.[rss|json]?screen_name=[USERNAME-GOES-HERE]&include_rts=[1|0]

So for example, if you wanted to get JSON feed for Nike’s Twitter, you’d go to:

http://api.twitter.com/1/statuses/user_timeline.json?screen_name=nike&include_rts=1

Happy Hacking!

Painless HABTM Validation in CakePHP

Recently I was faced with the task of validating a model with a HABTM relationship. I tried looking around for various solutions but all of them seemed quite complicated and time consuming to implement until I came across this nifty solution. Lets say you have two models - posts and tags with a HABTM relationship. You probably already have the relationship defined in your model, if not, it goes something like this:

models/post.php:

var $hasAndBelongsToMany = array(
	'Tag' => array(
		'className' => 'Tag',
		'joinTable' => 'tags_posts',
		'foreignKey' => 'post_id',
		'associationForeignKey' => 'tag_id',
		'with' => 'TagsPost',
		'unique' => true,
		'conditions' => '',
		'fields' => '',
		'order' => '',
		'limit' => '',
		'offset' => '',
		'finderQuery' => '',
		'deleteQuery' => '',
		'insertQuery' => ''
	)
);

Now in addition to that, just go ahead and add multiple validation rule in your post model:

var $validate = array(
        'Tag' => array(
            'multiple' => array(
                'rule' => array('multiple',array('min' => 2)),
                'message' => 'Please select at least 2 tags'),
        ),
    );

And last but not least add a beforeValidate filter …

function beforeValidate() {
	foreach($this->hasAndBelongsToMany as $k=>$v) {
		if(isset($this->data[$k][$k]))
		{
			$this->data[$this->alias][$k] = $this->data[$k][$k];
		}
	}
}

All you have to do now is add the following snippet in your main model’s controller:

controllers/posts_controller.php:

<?php
class PostsController extends Controller {

    function beforeRender()
    {
        $model = Inflector::singularize($this->name);
        foreach($this->{$model}->hasAndBelongsToMany as $k=>$v) {
            if(isset($this->{$model}->validationErrors[$k]))
            {
                $this->{$model}->{$k}->validationErrors[$k] = $this->{$model}->validationErrors[$k];
            }
        }

    }
}
?>

That will check for the errors raised in the main model’s validationErrors and copies them back to the HABTM’s model so that the view can correctly capture and print that validation error.

That’s it!

Parse Twitter & Facebook feeds with jQuery

This post simply demonstrates how to parse Twitter and Facebook feeds using jQuery. I’m utilizing a JavaScript equivalent of C# prototype methods which I had talked about in one of my earlier posts. Also, you’ll note that I’m using two different date/time jQuery plugins to make Twitter and Facebook date parsing a little easier. One of them is timeago and the other one is dateFormat. Although I had to put a condition to check if its IE as the dateFormat function was not playing nice with IE. Rest everything is pretty straight forward.

$(function () {
	$.ajax({
		url: 'http://api.twitter.com/1/statuses/user_timeline.json?screen_name=jesalg&trim_user=1&count=5&include_rts=1&callback=?',
		dataType: 'json',
		success: displayTwitterFeed
	});
	$.ajax({
		url: 'https://graph.facebook.com/calistolabs/feed?limit=5&access_token=XXXXXXXXXXXX|XXXXXXXXXXXXX|XXXXXXXXXXXXXXX&callback=?', //Replace with your own access token
		dataType: 'json',
		success: displayFacebookFeed
	});
});

function displayTwitterFeed(result) {
	var outputTemplate = "<p>{0}<br /> <small><a href=\"{1}\">{2} via {3}</a></small></p>";
	$.each(result, function (i, item) {
		var tweet = item.text.parseURL().parseUsername().parseHashtag();
		var createdAt = $.browser.msie ? item.created_at : $.timeago(dateFormat(item.created_at, "isoUtcDateTime"));
		var source = item.source;
		var tweetURL = "http://twitter.com/CalistoLabs/status/" + item.id_str;
		$("#TwitterFeed").append(outputTemplate.format(tweet, tweetURL, createdAt, source));
	});
}
function displayFacebookFeed(result) {
	var outputTemplate = "<p><strong><a href=\"{0}\">{1}</a></strong> {2}<br /><small>{3}</small></p>";
	$.each(result.data, function (i, item) {
		var username = item.from.name;
		var pageURL = "http://www.facebook.com/" + item.from.name.replace(/ /g, '');
		var date = $.browser.msie ? item.created_time : dateFormat(item.created_time.replace("+0000", ""), "dddd, mmmm dS, yyyy 'at' h:MM TT");
		var body = item.message;
		if (!body) {
			body = "<a href='" + item.link + "'>" + item.name + "</a><br/>" + item.description;
		}
		$("#FacebookFeed").append(outputTemplate.format(pageURL, username, body, date));
	});
}
String.prototype.parseURL = function () {
	return this.replace(/[A-Za-z]+:\/\/[A-Za-z0-9-_]+\.[A-Za-z0-9-_:%&\?\/.=]+/g, function (url) {
		return url.link(url);
	});
};
String.prototype.parseUsername = function () {
	return this.replace(/[@]+[A-Za-z0-9-_]+/g, function (u) {
		var username = u.replace("@", "")
		return u.link("http://twitter.com/" + username);
	});
};
String.prototype.parseHashtag = function () {
	return this.replace(/[#]+[A-Za-z0-9-_]+/g, function (t) {
		var tag = t.replace("#", "%23")
		return t.link("http://search.twitter.com/search?q=" + tag);
	});
};
String.prototype.format = function () {
	var s = this,
        i = arguments.length;
	while (i--) {
		s = s.replace(new RegExp('\\{' + i + '\\}', 'gm'), arguments[i]);
	}
	return s;
};

Edit:

As of Friday June 3rd Facebook Graph API now requires a valid app or user access_token to access feed data: https://developers.facebook.com/blog/post/509/ You can get your access_token from from the Graph API Explorer and append it to your URL as a query string parameter as shown above.

Edit:

Found a workaround which would let us bypass the GraphAPI. See more details in my latest post. Basically instead of using a URL like this:

https://graph.facebook.com/calistolabs/feed?limit=5&access_token=XXXXXXXXXXXX|XXXXXXXXXXXXX|XXXXXXXXXXXXXXX&callback=?

we’ll switch it out for:

https://www.facebook.com/feeds/page.php?id=161821885215&format=json