Initialize Django Tests
I use a lot of doctests for apps that all need to work on a set of initialized data. I was hoping that there would be some kind of hook in Django for this but there is not.
I could switch all of the doctests to unittests and use fixtures but that would be a lot of work and I prefer doctests. I could also go through and paste some sort of init command at the beginning of each test that would ensure the data was loaded or do the loading but that’s just plain bad practice.
I came up with a method of creating a ‘testsetup’ app that is always run before the other apps ensuring that whatever that app configures or loads into the database will be run first before any other apps preform their tests. Here’s how you can do it too.
First create a ‘testsetup’ app and edit its tests.py file
./manage.py startapp testsetup open testsetup/tests.py
__test__ = {"initialize tests": """
>>> your init code here
""" }
In the test.py file you can load the database, prime the cache, or setup whatever else you need initialized. Then add the ‘testapp’ as the first app to your settings.py
INSTALLED_APPS = ( 'testsetup', ... )
Now whenever you run
./manage.py test
It will first run the tests for the ‘testssetup’ script and everything will be primed. If that’s the only kind of test you run then that’s all you’ll need. You’re done with this tutorial.
But if you run app level tests (ie. ./manage.py test someapp anotherapp ) then the above solution is not enough. To ensure the testsuite is run before these apps we’ll make our own TEST_RUNNER. Create a file called ‘testrunner.py’ with the following source.
def run_tests(test_labels, verbosity=1, interactive = True, extra_tests=[]): print "Given these test_labels", test_labels print "With these extra_test", extra_tests from django.test.simple import run_tests as django_run_tests if test_labels: # Make sure 'testsetup' is run first tl = ['testsetup'] tl.extend(list(test_labels)) test_labels = tuple(tl) print "Testing these apps:", test_labels django_run_tests(test_labels, verbosity, interactive, extra_tests)
and then in your settings.py file set the TEST_RUNNER variable
TEST_RUNNER = 'testrunner.run_tests'
This script simply wraps the django test runner to ensure that the testsetup app is tested before any other apps are. It basically makes the django test runner think that you’re running ./manage.py test testsetup someapp when you actually run ./manage.py test someapp.
Proposal
A note to the community: I think it’d be great if the Django settings.py included a TEST_INIT variable which allowed you to point to a function that would be executed immediately before the first test was run. The hook would make the setup process for doctests much easier.
Add comment September 30, 2009
Forwarding Naked Domains for Appengine with Apache
Google Appengine currently does not allow configuration of naked domains. Meaning, if you’re building something on Appengine you’ll have to settle for a URL like http://www.mysite.com or http://whatever.mysite.com and you will not be able to use http://mysite.com.
Its not so bad in most cases, but not having http://mysite.com isn’t going to stop people from trying to go there. So its important to set up some sort of device to forward the naked domain to the www.
Unfortunately you’re going to need a server and for this example it will need to be running Apache2. The redirect is handled by placing the following line in an apache config file (/apache2/httpd.conf).
Redirect permanent / http://www.mysite.com/
Where http://www.mysite.com is the example site being hosted on Appengine.
Or, if your apache server is hosting other apps and domains you’ll need to set up the redirect in a VirtualHost as shown here.
<VirtualHost *:80> ServerName mysite.com Redirect permanent / http://www.mysite.com/ </VirtualHost>
Note that the ‘permanent’ parameter in the Redirect command enforces a 301 or permanent redirect. You can choose other forms of redirect by replacing the ‘permanent’ with either the redirect number (ie 302) or the keywords from the following table I’ve shamelessly copied from here.
| HTTP Code | Status | Description |
|---|---|---|
| 301 | permanent | The resource has permanently moved |
| 302 | temp | The resource has temporarily moved |
| 303 | seeother | The resource has been replaced and refer to new resource |
| 305 | UseProxy | Use proxy to access site |
| 307 | Temp | The resource has temporarily moved |
| 410 | Tegone | The resource has permanently removed |
Finally set your DNS to point the base domain to this apache server and in a few hours it should be permanently re-directing http://mysite.com/ to http://www.mysite.com/.
If you’re not using apache or are looking for more ideas here’s a list of other techniques.
Also, please vote for Google to fix the issue here and here.
Add comment August 26, 2009
Socialbrowse in the StarTribune
Yesterday I was on the cover of the Business section of the StarTribune. Its a nice article that talks a bit about Socialbrowse and my path to what I’m doing right now.
It also talks about Luke Francl and his awesome startup FanChatter which is also from Minnesota and also in YCombinator. Here is a link to the article
http://www.startribune.com/business/52719012.html
The print version comes with pretty pictures of Luke and I which I think is worth the money 
.
Thanks Thomas Lee for the nice coverage.
Add comment August 11, 2009
Log Observer using Python’s subprocess
Today I needed to write a wrapper around a program that would examine the stdio and respond immediately to certain results. The task was handled nicely with a python script using subprocess.
There isn’t a lot of documentation examples on it so to figure it out I wrote this small test app and I thought I’d share it here.
To begin I wrote a simple script that prints out “Program has run for x seconds.” every x seconds to simulate a long running and noisy script that I might be observing. Here is the code.
sample.py
import sys, time
class FlushFile(object):
"""Write-only flushing wrapper for file-type objects."""
def __init__(self, f):
self.f = f
def write(self, x):
self.f.write(x)
self.f.flush()
# Replace stdout with an automatically flushing version
sys.stdout = FlushFile(sys.__stdout__)
for i in xrange(100):
sys.stdout.write("Program has run for %d seconds.\n" % i)
time.sleep(1)
The main part of the program is the last 3 lines where we write “Program has run for %d seconds” 100 times and pause for a second between each.
The FlushFile object is simply a nice hack to overwrite the default stdout object to ensure that the buffer is flushed every time it is written to. Without the hack our log checker would simply be hung up until the task is fully completed instead of being able to read each line of the output as it is written.
The output looks like this
$ python sample.py Program has run for 0 seconds. Program has run for 1 seconds. Program has run for 2 seconds. .... # and on until 100 seconds
Next I wrote the wrapper which executes the above sample.py script as a python subprocess and watches the output. For this example I check whether a ‘4′ exists in the output. If it is I restart the script therefore creating a continual loop counting between 0 and 4 seconds.
observer.py
import subprocess, os, signal
cmnd = "python sample.py" #change this line to run your script
p, line = True, 'start'
while True:
p = subprocess.Popen(cmnd, shell=True, stdout=subprocess.PIPE)
while line:
line = p.stdout.readline()
print "line is:", line
if line.count('4'):
print "restarting the process"
os.kill(p.pid, signal.SIGUSR1)
line = True
break
del p
The output is continually checked in the ‘while line:’ loop of line 8. In this example its printed out for our convenience. Like 11 checks for the character ‘4′ somewhere in the output. If it does exist it kills the process and the stdout reading loop.
That loop is wrapped in another however that simply restarts the process all over again. The resulting output is
$ python observer.py line is: Program has run for 0 seconds. line is: Program has run for 1 seconds. line is: Program has run for 2 seconds. line is: Program has run for 3 seconds. line is: Program has run for 4 seconds. restarting the process line is: Program has run for 0 seconds. line is: Program has run for 1 seconds. line is: Program has run for 2 seconds. line is: Program has run for 3 seconds. line is: Program has run for 4 seconds. restarting the process line is: Program has run for 0 seconds. .....
So now we have a nice template for wrapping and responding to executables!
Add comment August 10, 2009
Starting Out With Comet (Orbited) Part 3 – The Client
Comet implementations require both server and client side components. In Part 2 of this series we installed and configured the server side component and then used the example STOMP Test client to test it out and get a feel for what was going on.
In this part of the series we’ll be covering the client side to make our custom interface to the comet server. To demonstrate we’ll go through the steps in creating an app I call EZChat which is basically a bare bones comet chat client. The interface will allow you to choose a name and submit messages that will be broadcast to everyone viewing the page in real time.
For it to work you will have to have the configuration of Part 2 setup and your Orbited server running. For reference I learned most of the client side code in this example by playing with the source of the example STOMP Test client. I recommend taking a look at that source if you need more advanced options or another example.
Includes
At the top of the <head> you first need to include the libraries for the Orbited (/static/Orbited.js) and STOMP (/static/protocols/stomp/stomp.js) client-side implementations.
<script>document.domain=document.domain</script> <script src="http://localhost:9000/static/Orbited.js"></script> <script> Orbited.settings.port = 9000; TCPSocket = Orbited.TCPSocket; </script> <script src="http://localhost:9000/static/protocols/stomp/stomp.js"></script> <script src="http://www.json.org/json2.js"></script>
In between the scripts we’ve set up a TCPScocket. It must be don between the two script includes because the STOMP library needs the socket setup for its execution. We’ve also specified the Orbited port which is necessary especially if you change the port on which the orbited and stomp javascript files are hosted on (they can both be hosted on port 80 along with your other scripts).
Lastly, we’ve also included a popular JSON library. Typically I use jquery-json but I’ve kept this tutorial free of javascript frameworks so as not to add unneeded complexity. If you’d like you can of course switch to whatever JSON library you’re used to; just replace the JSON.stringify and JSON.parse functions with your equivalents.
STOMP Setup
Because we’re dealing with the setup we’ll skip to the bottom of the page and add the following script just before the end tag of the body (</body>).
<script type="text/javascript">
(function() { // set up stomp client.
stomp = new STOMPClient();
stomp.onconnectedframe = function() { // Run on initial connection to STOMP (comet) server
stomp.ready = true;
// subscribe to channel CHANNEL = "/ezchat/"
var CHANNEL = '/ezchat/'
stomp.subscribe(CHANNEL);
};
stomp.onmessageframe = function(frame) { // Executed when a messge is received
my_receive( JSON.parse(frame.body) );
};
// Everything is setup. Start the connection!
stomp.connect(document.domain, 61613); //, 'guest', 'guest');
})();
</script>
My apologies as always for the crappiness of the wordpress syntax parser. Lets walk through what’s happening.
At the top, we initialize a new STOMPClient object. The StompClient has the following hooks you can override to trigger your own events.
onopen – Called when the Transport is opened
onclose – Called when the Transport has closed
onerror – Called when the Stomp Client has errored
onerrorframe – Called when there is an error in the message received
onconnectedframe – Called when a the client is fully set up for sending/receiving
onmessageframe – Called when a message is received
The STOMP object also has these functions for connecting and resetting the connections.
reset – Resets the STOMP connection
connect – Connects to the STOMP server
send – Sends the object in the first argument to the channel specified by the second argument
In our simple example only the onconnectedframe and onmessage frames need to be overwritten.
The onconnectedframe function is called when the STOMP server has been connected to and everything is setup for sending and receiving messages. Inside this function we simply need to subscribe/listen to the CHANNEL we’ve setup for our chat. For the example I’ve chosen the channel ‘/ezchat/’. Once subscribed our STOMP client will receive any messages sent to that channel in real time. You can subscribe to multiple channels if you’d like, and you can make clients with different channels if you’d like to have different chat rooms. But for this example we’ll just stick with the hard coded ‘/ezchat/’ channel.
The onmesageframe function is called when a message has been received. It is passed a frame object with the following structure
frame
- body: “{“name”:”Dave”,”message”:”awesome this is working”}”
- headers: Object
- content-encoding: “utf-8″
- content-length: “51″
- content-type: “text/plain”
- destination: “/ezchat/”
- message-id: “/ezchat/_3″
- type: “MESSAGE”
where the body holds the information that has been sent. The STOMP server and client add extra “type” and “headers” objects to communicate between each other. The extra information can be very useful for more complicated applications but for our simple example we’re only interested in the frame “body”.
So you see that the onmessageframe is simply parsing the json object in frame.body of every received message and passing it to my_receive, a function we will soon create.
The Content
For EZChat we need a form where users can specify a name and type messages to send. We also need an area to put the messages. Plop this HTML in at the top of the <body> to handle all of that.
<h2>EZChat - Example Comet Client!</h2> Everyone viewing this page will see the messsages you submit instantly. <form id="message_form" action="#"> Name: <input type="text" name="chat_name" id="chat_name"></input> Message: <textarea name="message" id="message" rows="4" cols="40"></textarea> <input type="submit" name="Send" onclick="return my_send(); return false"></input> </form> <div id="messages"><!--- All received messages will get placed here ---></div>
There are a few things to notice here. First, the important elements in the form have ids ‘chat_name’ and ‘message’ and the area that will be containing all the received messages is called ‘messages’. The names don’t matter except that we’ll use them in the functions we create later.
Second, the onclick event of the submit button is overridden with instead calling the my_send function. We’ll make this function in the next step.
The Functions
Lastly we need to make the custom my_send and my_receive functions that get called to send and receive messages. Insert these functions into the head after the includes.
my_send
The my_send function will get the values from the ‘chat_name’ and ‘message’ form elements, combine them in an object, convert the object to json, and then sends it to the ‘/ezchat/’ channel. The sending is handled using the stomp.send command which takes as input the object to send and second, the channel to send it to.
var CHANNEL = '/ezchat/';
function my_send() {
// Get the values to send from the form
var name = document.getElementById('chat_name').value;
var message = document.getElementById('message').value;
var msg = {'name': name, 'message': message};
var json_msg = JSON.stringify(msg);
stomp.send(json_msg, CHANNEL)
return false;
}
my_receive
As discussed earlier the my_receive function gets the JSON parsed version of whatever was sent in ‘frame.body’. In the case of our app its always an object of the format
msg = {‘name’: <some name>, ‘message’: <some message>}
The my_receive function simply takes this object and converts it into a prettier HTML format and appends it to the top of the message list we created in the HTML.
function my_receive( msg ) {
console.log('received message', msg);
// append the <msg> to the top of the list of messages.
var messages_el = document.getElementById('messages');
var new_message = "
<div><strong>" + msg['name'] + ":</strong> " + msg['message'] + "</div>
";
messages_el.innerHTML = new_message + messages_el.innerHTML;
}
That’s it for the code. Scroll to the bottom of the page for the full version of the source.
Results
Ensure that your Orbited server is running as described in Part 2 of the series. Then load up the page we’ve made in two or more separate windows. Choose a different name for each window and start sending each other messages. You’ll notice that both windows will receive the submitted messages almost instantaneously! The beauty of Comet!
If you open your STOMP Test Client and subscribe to the ‘/ezchat/’ channel you’ll see a more raw input on what’s actually being received by the STOMP clients as you chat.
You can see how the STOMP Test client is incredibly handy for debugging.
That’s it for this part of the Tutorial! There are still parts to come including writing a data handler on the server side and hopefully a much requested post on Django integration.
Full Source
For your convenience here’s the full index.html file for this example.
<html>
<head>
<script>document.domain=document.domain</script>
<script src="http://localhost:9000/static/Orbited.js"></script>
<script>
Orbited.settings.port = 9000;
TCPSocket = Orbited.TCPSocket;
</script>
<script src="http://localhost:9000/static/protocols/stomp/stomp.js"></script>
<script src="http://www.json.org/json2.js"></script>
<script type="text/javascript">
// These are our custom functions for sending and receiving STOMP messages.
// They will be sent in the format msg = {'name': somename, 'message': somemessage}
var CHANNEL = '/ezchat/';
function my_receive( msg ) {
console.log('received message', msg);
// append the <msg> to the top of the list of messages.
var messages_el = document.getElementById('messages');
var new_message = "
<div><strong>" + msg['name'] + ":</strong> " + msg['message'] + "</div>
";
messages_el.innerHTML = new_message + messages_el.innerHTML;
}
function my_send() {
// Get the values to send from the form
var name = document.getElementById('chat_name').value;
var message = document.getElementById('message').value;
var msg = {'name': name, 'message': message};
console.log(msg);
var json_msg = JSON.stringify(msg);
console.log(json_msg);
stomp.send(json_msg, CHANNEL)
return false;
}
</script>
</head>
<body>
<h2>EZChat - Example Comet Client!</h2>
<div>Everyone viewing this page will see the messsages you submit instantly.</div>
<form id="message_form" action="#">
Name:
<input type="text" name="chat_name" id="chat_name"></input>
Message:
<textarea name="message" id="message" rows="4" cols="40"></textarea>
<input type="submit" name="Send" onclick="return my_send(); return false"></input>
</form>
<div id="messages"><!--- All received messages will get placed here ---></div>
<script type="text/javascript">
(function() { // set up stomp client.
stomp = new STOMPClient();
stomp.onconnectedframe = function() { // Run on initial connection to STOMP (comet) server
stomp.ready = true;
// subscribe to channel CHANNEL = "/ezchat/"
stomp.subscribe(CHANNEL);
};
stomp.onmessageframe = function(frame) { // Executed when a messge is received
console.log('frame is', frame);
my_receive( JSON.parse(frame.body) );
};
// Everything is setup. Start the connection!
stomp.connect(document.domain, 61613); //, 'guest', 'guest');
})();
</script>
</body>
</html>
As usual, if there are corrections or questions please be sure to leave them in the comments. And remember to subscribe to catch the rest of the series.
3 comments August 3, 2009
Appengine Example: Quick Thoughts
I’ve made a few apps on Google’s Appengine now and am getting to the point where I can pump them out fairly quickly. I really love that they make user authentication (my least favorite part of web applications) incredibly simplified.
I wrote this app in less than an hour as a simple tool for myself and to test out the authentication tools which I hadn’t gotten a chance to use yet. I’m sharing the source here in case its of any use to others.
Description
The app is called Quick Thoughts and its a very simple private micro blog (a private twitter). Basically you log in and can record quick notes to yourself. They’re dated and only you can see them.
Its all on one page. You can get a good idea of what it is from this screen shot. You can also just try it out yourself by logging into: http://quoughts.appspot.com/
There are a lot of other Appengine tutorials including the official tutorial that do a very thorough job of explaining setting up a development environment and deploying. I’m just going to share some more example code with some helpful comments.
I know looking at code is not pretty, but I’ve included a lot of comments for each part. It should be easiest to understand in that format. Ugh: and I’m sorry that the crappy WordPress syntax highlighting has totally f’d the format. That pretty much makes python code useless, but hopefully you’ll be able to sift through.
Template
There are only three files for this app. The first is the HTML template for the single page that is used.
index.html
</pre>
<html>
<head>
<title>Quick Thoughts</title>
<!--- The following CSS could ofcourse be put in a seperate file, but this is simplest for now --->
<style>
form { width: 320px; }
body {
font-family: "Trebuchet MS, Verdana, Arial, Helvetica, sans-serif"
font-size: 16px;
}
.date { font-style: italic; font-size: 12px; }
.thought { padding-top: 30px; }
div#outer {
width: 500;
background-color:#FFFFFF;
margin-top: 50px;
margin-bottom: 50px;
margin-left: auto;
margin-right: auto;
padding: 10px;
//border: thin solid #000000;
}
h2 { margin-bottom: 0px; }
.username {
margin-bottom: 20px;
}
</style>
</head>
<body>
<div id="outer">
<h2>Quick Thoughts</h2>
<!--- Showing the user's Nickname here. User object also has .email, and .user_id
more info here: http://code.google.com/appengine/docs/python/users/userclass.html
--->
<div>By {{ user.nickname }}</div>
<div>
<form action="." method="POST">
<textarea name="thought" rows="6" cols="40"></textarea>
<div align="right"><input type="submit" value="Record" align="right"></input></div>
</form></div>
<!---
This for loop prints out the different Thoughts stored in the database
Appengine uses Django .96's templating system which I personally think is pretty great.
For more information visit http://www.djangoproject.com/documentation/0.96/templates/
--->
{% for thought in thoughts %}
<div>
<div>{{ thought.thought|linebreaksbr }}</div>
<div>{{ thought.date|date:"D. N jS g:i a" }}</div>
</div>
{% endfor %}</div>
</body>
</html>
<pre>
This is all pretty straightforward especially if you come from the Django world. Appengine wisely uses Django’s templating system to render its HTML pages. For more details on the templates view the Django .96 Template Documentation.
Request Handling
The second file is the python WSGI handler. You can ofcourse use Django on Appengine and have the advantage of the nice url parser and the Django views format but here I just stuck with the WSGI RequestHandlers.
thoughts.py
from google.appengine.api.urlfetch import fetch as urlfetch, GET, POST
from google.appengine.ext import db
from google.appengine.ext.webapp import RequestHandler, WSGIApplication
from google.appengine.ext.webapp import template
from google.appengine.api import users
import os
from wsgiref.handlers import CGIHandler
class Thought(db.Model):
"""
This is the Database Model that stores the different Thought objects that the user submits
Each entry in the database stores a thought, date and the user who wrote it.
"""
thought = db.TextProperty()
date = db.DateTimeProperty(auto_now_add=True)
# The auto_now_add setting automatically adds the date that the object was created so you don't have to.
user = db.UserProperty() # Google handles the user for you. Great!
class ThoughtHandler(RequestHandler):
def get(self):
user = users.get_current_user() # Get the user
if not user:
# If they are not logged in, ask google to authenticate them.
self.redirect(users.create_login_url(self.request.uri))
# These are the variables that will be sent to the template
template_values = {
# This is a GQL query for the appengine datastore.
# Here we're finding all Thoughts for the given User and ordering them by Date descending
# More info on GQL: http://code.google.com/appengine/docs/python/datastore/gqlreference.html
'thoughts': Thought.all().filter("user =", user).order('-date'),
'user': user,
}
# Gather the full path to the template
path = os.path.join(os.path.dirname(__file__), 'index.html')
# Render the template with the template_values we collected above
html = template.render(path, template_values)
# Write out the result
self.response.out.write(html)
def post(self):
"""
A Thought has been submitted via POST.
Create a new Thought object and re-direct back to the front page.
"""
user = users.get_current_user() # Get the user
if not user:
# If they are not logged in, ask google to authenticate them.
self.redirect(users.create_login_url(self.request.uri))
# Get the 'thought' POST data from the request
thought = self.request.get('thought')
# Create a new thought object using the POST data and the authed user
t = Thought( thought = thought, user = user )
# Save the object
t.put()
# Now re-direct back to the front page
self.redirect('/')
def main():
"""
This simple function is the URL parser
There's only one URL for this app, so its a pretty bad example for this 
"""
application = WSGIApplication([
('/', ThoughtHandler),
], debug=True)
CGIHandler().run(application)
if __name__ == '__main__':
main()
There are a few major highlights in this code: the Thought Datastore Model, the Query for your Thoughts, and the simple Google Authentication. I LOVE these three lines of code (yes, I know how nerdy that sounds):
user = users.get_current_user() # Get the user if not user: self.redirect(users.create_login_url(self.request.uri))</pre>
In those 3 lines we’ve requested the User object and asked Google to authenticate them and send them back if they’re not logged in! Super simple! No more login/signup/change password/change username crap to deal with here. The authentication is done for you.
Toward the top is the Thought model that is a subclass of db.Model. For those of you who’re familiar with Django this format will look familiar. The Thought model contains the text of the thought, the date it was recorded and the user who recorded it.
In the ‘get’ Request we query for all of the thoughts of the given user and order them by descending date. The objects are fetched using GQL, the query interface for the Datastore. You can handle most queries by playing with the format of the above example, but here is more information on GQL.
Configuration
Finally we need the configuration file for our app. Its called app.yaml and it tells appengine what App we’ve registered as, and how to handle the URLs.
app.yaml
application: YOURAPPNAME version: 1 runtime: python api_version: 1 handlers: - url: /.* script: thoughts.py
Other tutorials explain this file well. You can expand it to include other scripts and serve static files.
Conclusion
Now you have your own private micro-blog on Google’s datastore! That means it’s theoretically infinitely scalable without you ever having to worry about a thing. You can grow to the size of twitter and never blink an eye .
Hope it helped some people. Feel free to use this code in any way you’d like and feel free to leave questions, comments or corrections.
1 comment July 21, 2009
Starting Out With Comet (Orbited) Part 2 – Installation and STOMP
In this part of the tutorial we will install and setup the server side of a Comet installation using the Orbited implementation. We’ll also be using the MorbitQ STOMP server to handle message passing and we’ll play around with STOMP/Comet setup using Orbited’s STOMP Test demo. If you have not read the first blog post in this series I advise you do so. I will be assuming you have an understanding of those terms and concepts of the previous post for this tutorial.
Before we start I’d like to quote my source. I’ve learned most of what I’m sharing from Michael Carter’s Tutorial and many hours of playing around.
Installation
We will need to install the latest version of Orbited and also some Stomp tools. The stomp tools aren’t required for this step but will be for anything else you want to do with comet so I’ve included them in the installation steps here as well. Conveniently Orbited is setup in the Cheeseshop. You need python2.5+ and if you have not installed the python setup-tools do so now.
Installation simply consists of the following commands.
easy_install twisted easy_install orbited easy_install stompservice easy_install simplejson
To test if it works enter your python shell and test importing the libraries. The following should load without any errors
$ python
>>> import twisted
>>> import orbited
>>> import stompservice
>>> import simplejson
If you have any troubles there is more info on the Orbited Installation Guide.
Configuration
Orbited is configured with a ‘.cfg’ file. Lets call ours example.cfg. Make a directory anywhere for your project and paste the following into example.cfg
[global] session.ping_interval = 300 [listen] http://:9000 stomp://:61613 [access] * -> localhost:61613
Lets go through the different parts.
The ping interval is a number of seconds for the backend to wait before it pings the client. We’ve got it set up for 5 minutes. Good comet implementations have some sort of pinging system. This is a necessary step as due to current HTTP protocols the client cannot tell if something has gone wrong on the server end. It simply waits happily all day for some sort of response for the server. But with a pinging system setup we can tell the client to refresh its connection if it hasn’t heard from the server in the last 300 seconds, and the server will make sure to ping the client at least every 300 seconds, letting it know that the connection is still alive.
The listen parameters tell the orbited server which ports to listen to and who to proxy requests to. In our configuration port 9000 will be serving static html files, and port 61613 will be a proxy for our STOMP server.
And finally, the access parameter gives permission to proxy to the stomp server.
Lets Run It
To run enter your project directory and type
orbited – -config example.cfg
It should look something like this:
06/24/09 21:05:24:651 INFO orbited.start proxy protocol active
06/24/09 21:05:24:511 INFO orbited.start Listening http@9000
06/24/09 21:05:24:525 INFO orbited.start Listening stomp@61613
STOMP Test
Orbited comes with a really nice STOMP demo that also serves as a nice tool for debugging your setup later. We’ll use it to play around with Comet and understand the concepts behind STOMP.
While your orbited server is running visit the following URL.
http://localhost:9000/static/demos/stomp/
There are 3 important tools/rows we’ll be using here: Connect, Subscribe and Send.
First click on “Connect” to connect to the orbited and stomp servers. We’re using MorbitQ has the stomp client, which doesn’t deal with authentication, so any name and password will work. Notice that the STOMP test shell will now say.
→ Transport openned
→ Connected as user guest
Second change the “destination” in the “Subscribe” row to be “/channel/1/” and click Subscribe. You have now created and subscribed to a channel called “/channel/1/”.
Next we’ll send something to that channel using the Send tool. Again change the destination to “/channel/1/” and type something into the message box replacing “hello”. In the image above I’ve chosen “comet is working!”. Now hit Send and notice that your message shows up in your STOMP shell!
Try sending to other destinations and notice that only messages sent to “/channel/1/” will show up in the stomp shell. We can change that however by subscribing to additional channels. Try subscribing to “/anotherchannel/” and then send it a message. Notice that this setup can handle being subscribed to many different channels at once.
The Real Power
All of the things we’ve tried so far could have been fairly easily implemented with simple AJAX. The real power of comet is that it can push information to the client without having to submit a request. Also, the real power of STOMP is that it smoothly handles message passing between clients. Lets demonstrate both of these now by opening up multiple browser windows all pointing to our STOMP Test.
Open up 3 windows and “Connect” them each to the server. Now subscribe the second window to “/channel/1/”, the third window to “/channel/2/” and the first window to both.
Once setup, using the first window send a message “Message to Channel 1″ to “/channel/1/”. You’ll notice that it not only showed up instantly in window 1 (where you submitted), but also in window 2 (where you did nothing)! The STOMP server has passed the message all clients listening to “/channel/1/” and the Comet server has pushed it to the client without it having to poll for updates!
Now send a message to “/channel/2/” and notice that it shows up in windows 1 and 3, but not 2.
Play around with this setup more to become familiar. Each window can subscribe to any number of channels, and each can send messages to any channel, whether it is subscribed to it or not.
Whats Next
We’ve now setup and tested a Comet implementation. You can see its benefits and understand how it works with STOMP. In the next example we’ll work on the client side of the Comet implementation and write a python STOMP client to handle processing and sending the data on a comet server.
In the mean time you may want to look at the other demo’s that came with Orbited or Michael’s Demo.
Update: Part 3 – The Client is now available.
7 comments June 25, 2009
Starting Out With Comet (Orbited) Part 1
This is the first article in a series I’m creating to ease developers into using Comet. The documentation is severely lacking on every comet implementation I’ve come across which I think is Comet’s biggest limitation at the moment. Hopefully this will help those interested in Comet to struggle less with their implementations than I did/do.
The series will specifically focus on using the Orbited implementation of Comet, but many of the concepts will apply to other Comet implementations as well.
This first post will explain some terms and concepts you need to become familiar with.
Comet
Comet is a term referring to a set of techniques (hacks) that enable the server to push data to the client whenever it wants. Traditionally the client has to initiate all requests. If the server wants to send something to the client it has to wait until the client chooses to connect again.
Comet is a set of methods for the server to ping the client.
As a definition, Comet is as arbitrary as the word AJAX. Try not to get hung up on the word any more than you’d get hung up on defining what AJAX is.
What and Why Orbited
Orbited is a Python based implementation of Comet using the Twisted framework. There’s not a lot out there discussing the pro’s and cons of the different comet services so its hard to tell which implementation is better than another. There is clearly not yet a dominant implementation.
I basically chose Orbited because it was the first one I could get up and running thanks to this tutorial, because I love python, and because my implementation will be working with Django.
Orbited is built on the Twisted framework and is event based, so in theory it should scale just fine.
Its important to note that Orbited has both a server side component and a JavaScript library for dealing with the client side of Comet connections as well.
Long-Polling
Long-polling is the technique Orbited uses to maintain a consistent real-time connection between the server and the client, enabling the server to push data to the client whenever it wants. Put simply the client makes a request that is kept open (no immediate response is returned). The server can continue to send data to the client through this open connection until it is terminated by the client.
Streaming
Streaming is another Comet technique. Michael Carter, one of Orbited’s main contributors has a nice article on the advantages of the two techniques.
STOMP
Short for Streaming Text Orientated Messaging Protocol, STOMP is is a message passing protocol commonly used in Comet implementations. It enables a publish/subscribe (pub/sub) model that comes in handy in many real-time applications.
STOMP isn’t a required part of a Comet implementation, but we’re going to be using it in this series.
MorbidQ
MorbidQ is a STOMP server that comes with Orbited. Think of it as Apache, but for STOMP instead of web apps. Its great in that its easy to set up and work with, but its not robust enough for a significant load. When deploying you should instead use RabbitMQ.
Pub/Sub
In the following posts we will be using a publish/subscribe model thanks to the help of STOMP. With this model you can create channels and both publish and subscribe to them. This model works great when multiple clients are listening to the same thing, such as a group chat or a game.
I like to think of it as a telegraphy system. A whole bunch of people can hook up to the wire, and if they do they hear everything everyone else is broadcasting, and can broadcast their own messages as well.
So subscribing is like hooking up your headphones to the transmission lines and publishing is like sending out a signal with your morse key. A different channel would correspond to a different wire.
Orbited Server
Comet requires an event based server to deal with the long requests. Typical servers like Apache for instance were designed to handle a request as quickly as possible. They allocate memory to returning a response as soon as a request is received, and maintain that memory until a response is returned.
An event-based server such as Orbited can chop up the different components of the request/response process. It can accept a request and then forget about it while allocating its memory elsewhere until at some later point something needs to be done (a response returned or data transmitted). For this reason it can handle dramatically more open connections than Apache.
Lets use an example. Pretend we have a typical Apache hosted site that attracts 10,000 people every hour. Lets say with that amount of traffic we get on average 50 new page requests every second and it takes about a second to return each page. That means Apache has at any given time about 50 open connections. That’s what it is designed to do.
But if we now put Comet on that site, all 10,000 of those users need to have a long connection open with our server. Similar to the first estimate lets say only 50 of them are really receiving any data at any point. Our Comet server will only assign memory to the tasks of those 50 connections, where Apache would attempt to assign an equal amount of memory to all 10,000 open connections.
That’s why its important in Comet to separate connections from events, or why we use Orbited (or equivalent comet server) instead of Apache.
Continuing…
That’s it for my explanations/intro post. I’m fairly new to the Comet scene myself and am very busy with my startup. I’m sure there are a lot of additions or corrections, please leave them in the comments and I will try to keep up with corrections..
I’ll put more time in this depending on how much interest there is in the comments and subscribriptions. Make sure to check the Comet Category listings for the other posts.
The next post will be an example implementation.
Update: Continue to the next post in the series: Starting Out With Comet (Orbited) Part 2 – Installation and STOMP
7 comments June 9, 2009
New Blog Design By Amanda Scharlemann
A few weeks ago I published a request for a generous designer to create a unique look for this blog. Amanda Scharlemann was the first to respond and was a pleasure to work with!
Amanda is a design student at Bethany Lutheran College and shows great potential. She had never worked with customizing WordPress Templates and the annoying restrictions that come with that, but was eager and quick to learn. We went through several iterations to narrow down exactly what I was looking for and she spent a lot of time examining the example sites I gave her to get a feel for what I would like. At each iteration she had significant improvements, helpful advice, and most importantly was open and attentive to feedback.
I dislike design processes where the designer is stubbornly biased in certain directions and completes 90% of the work before you’re allowed to review. Amanda was not that way at all. I’m not sure if they teach client-design relationships at Bethany or not, but she was a pleasure to work with.
The custom design was launched last week, and so far the results have been great. I’ve seen a doubling in traffic, and several more comments than usual. Perhaps the most impressive statistic so far is that a friend I hadn’t talked to in a while came across my blog while goggling an issue he was having. My article apparently helped him and he was excited to notice (thanks to the personalized sidebar) that the blog belonged to me!
Over time I will probably tweak things here and there as is my nature but I’m over all very pleased with the new look.
Thanks again Amanda!
Add comment June 8, 2009
Find RSS Feed Links With jQuery
This took me a little while to figure out so I thought I’d share. You can use a jQuery selector to find any RSS links on a page very easily.
The following line will return a list of the RSS link elements.
var link_elements = $('link[type="application/rss+xml"]');
The following snippet will create an array of all the urls to the RSS feeds on the page.
var links = [];
$('link[type="application/rss+xml"]').each(function() { links[links.length] = this.href; });
Add comment June 3, 2009
