Starting Out With Comet (Orbited) Part 2 – Installation and STOMP
In this part of the tutorial we will install and setup the server side of a Comet installation using the Orbited implementation. We’ll also be using the MorbitQ STOMP server to handle message passing and we’ll play around with STOMP/Comet setup using Orbited’s STOMP Test demo. If you have not read the first blog post in this series I advise you do so. I will be assuming you have an understanding of those terms and concepts of the previous post for this tutorial.
Before we start I’d like to quote my source. I’ve learned most of what I’m sharing from Michael Carter’s Tutorial and many hours of playing around.
Installation
We will need to install the latest version of Orbited and also some Stomp tools. The stomp tools aren’t required for this step but will be for anything else you want to do with comet so I’ve included them in the installation steps here as well. Conveniently Orbited is setup in the Cheeseshop. You need python2.5+ and if you have not installed the python setup-tools do so now.
Installation simply consists of the following commands.
easy_install twisted easy_install orbited easy_install stompservice
To test if it works enter your python shell and test importing the libraries. The following should load without any errors
$ python
>>> import twisted
>>> import orbited
>>> import stompservice
If you have any troubles there is more info on the Orbited Installation Guide.
Configuration
Orbited is configured with a ‘.cfg’ file. Lets call ours example.cfg. Make a directory anywhere for your project and paste the following into example.cfg
[global] session.ping_interval = 300 [listen] http://:9000 stomp://:61613 [access] * -> localhost:61613
Lets go through the different parts.
The ping interval is a number of seconds for the backend to wait before it pings the client. We’ve got it set up for 5 minutes. Good comet implementations have some sort of pinging system. This is a necessary step as due to current HTTP protocols the client cannot tell if something has gone wrong on the server end. It simply waits happily all day for some sort of response for the server. But with a pinging system setup we can tell the client to refresh its connection if it hasn’t heard from the server in the last 300 seconds, and the server will make sure to ping the client at least every 300 seconds, letting it know that the connection is still alive.
The listen parameters tell the orbited server which ports to listen to and who to proxy requests to. In our configuration port 9000 will be serving static html files, and port 61613 will be a proxy for our STOMP server.
And finally, the access parameter gives permission to proxy to the stomp server.
Lets Run It
To run enter your project directory and type
orbited –config example.cfg
It should look something like this:
06/24/09 21:05:24:651 INFO orbited.start proxy protocol active
06/24/09 21:05:24:511 INFO orbited.start Listening http@9000
06/24/09 21:05:24:525 INFO orbited.start Listening stomp@61613
STOMP Test
Orbited comes with a really nice STOMP demo that also serves as a nice tool for debugging your setup later. We’ll use it to play around with Comet and understand the concepts behind STOMP.
While your orbited server is running visit the following URL.
http://localhost:9000/static/demos/stomp/
There are 3 important tools/rows we’ll be using here: Connect, Subscribe and Send.
First click on “Connect” to connect to the orbited and stomp servers. We’re using MorbitQ has the stomp client, which doesn’t deal with authentication, so any name and password will work. Notice that the STOMP test shell will now say.
→ Transport openned
→ Connected as user guest
Second change the “destination” in the “Subscribe” row to be “/channel/1/” and click Subscribe. You have now created and subscribed to a channel called “/channel/1/”.
Next we’ll send something to that channel using the Send tool. Again change the destination to “/channel/1/” and type something into the message box replacing “hello”. In the image above I’ve chosen “comet is working!”. Now hit Send and notice that your message shows up in your STOMP shell!
Try sending to other destinations and notice that only messages sent to “/channel/1/” will show up in the stomp shell. We can change that however by subscribing to additional channels. Try subscribing to “/anotherchannel/” and then send it a message. Notice that this setup can handle being subscribed to many different channels at once.
The Real Power
All of the things we’ve tried so far could have been fairly easily implemented with simple AJAX. The real power of comet is that it can push information to the client without having to submit a request. Also, the real power of STOMP is that it smoothly handles message passing between clients. Lets demonstrate both of these now by opening up multiple browser windows all pointing to our STOMP Test.
Open up 3 windows and “Connect” them each to the server. Now subscribe the second window to “/channel/1/”, the third window to “/channel/2/” and the first window to both.
Once setup, using the first window send a message “Message to Channel 1″ to “/channel/1/”. You’ll notice that it not only showed up instantly in window 1 (where you submitted), but also in window 2 (where you did nothing)! The STOMP server has passed the message all clients listening to “/channel/1/” and the Comet server has pushed it to the client without it having to poll for updates!
Now send a message to “/channel/2/” and notice that it shows up in windows 1 and 3, but not 2.
Play around with this setup more to become familiar. Each window can subscribe to any number of channels, and each can send messages to any channel, whether it is subscribed to it or not.
Whats Next
We’ve now setup and tested a Comet implementation. You can see its benefits and understand how it works with STOMP. In the next example we’ll work on the client side of the Comet implementation and write a python STOMP client to handle processing and sending the data on a comet server.
In the mean time you may want to look at the other demo’s that came with Orbited or Michael’s Demo.
3 comments June 25, 2009
Starting Out With Comet (Orbited) Part 1
This is the first article in a series I’m creating to ease developers into using Comet. The documentation is severely lacking on every comet implementation I’ve come across which I think is Comet’s biggest limitation at the moment. Hopefully this will help those interested in Comet to struggle less with their implementations than I did/do.
The series will specifically focus on using the Orbited implementation of Comet, but many of the concepts will apply to other Comet implementations as well.
This first post will explain some terms and concepts you need to become familiar with.
Comet
Comet is a term referring to a set of techniques (hacks) that enable the server to push data to the client whenever it wants. Traditionally the client has to initiate all requests. If the server wants to send something to the client it has to wait until the client chooses to connect again.
Comet is a set of methods for the server to ping the client.
As a definition, Comet is as arbitrary as the word AJAX. Try not to get hung up on the word any more than you’d get hung up on defining what AJAX is.
What and Why Orbited
Orbited is a Python based implementation of Comet using the Twisted framework. There’s not a lot out there discussing the pro’s and cons of the different comet services so its hard to tell which implementation is better than another. There is clearly not yet a dominant implementation.
I basically chose Orbited because it was the first one I could get up and running thanks to this tutorial, because I love python, and because my implementation will be working with Django.
Orbited is built on the Twisted framework and is event based, so in theory it should scale just fine.
Its important to note that Orbited has both a server side component and a JavaScript library for dealing with the client side of Comet connections as well.
Long-Polling
Long-polling is the technique Orbited uses to maintain a consistent real-time connection between the server and the client, enabling the server to push data to the client whenever it wants. Put simply the client makes a request that is kept open (no immediate response is returned). The server can continue to send data to the client through this open connection until it is terminated by the client.
Streaming
Streaming is another Comet technique. Michael Carter, one of Orbited’s main contributors has a nice article on the advantages of the two techniques.
STOMP
Short for Streaming Text Orientated Messaging Protocol, STOMP is is a message passing protocol commonly used in Comet implementations. It enables a publish/subscribe (pub/sub) model that comes in handy in many real-time applications.
STOMP isn’t a required part of a Comet implementation, but we’re going to be using it in this series.
MorbidQ
MorbidQ is a STOMP server that comes with Orbited. Think of it as Apache, but for STOMP instead of web apps. Its great in that its easy to set up and work with, but its not robust enough for a significant load. When deploying you should instead use RabbitMQ.
Pub/Sub
In the following posts we will be using a publish/subscribe model thanks to the help of STOMP. With this model you can create channels and both publish and subscribe to them. This model works great when multiple clients are listening to the same thing, such as a group chat or a game.
I like to think of it as a telegraphy system. A whole bunch of people can hook up to the wire, and if they do they hear everything everyone else is broadcasting, and can broadcast their own messages as well.
So subscribing is like hooking up your headphones to the transmission lines and publishing is like sending out a signal with your morse key. A different channel would correspond to a different wire.
Orbited Server
Comet requires an event based server to deal with the long requests. Typical servers like Apache for instance were designed to handle a request as quickly as possible. They allocate memory to returning a response as soon as a request is received, and maintain that memory until a response is returned.
An event-based server such as Orbited can chop up the different components of the request/response process. It can accept a request and then forget about it while allocating its memory elsewhere until at some later point something needs to be done (a response returned or data transmitted). For this reason it can handle dramatically more open connections than Apache.
Lets use an example. Pretend we have a typical Apache hosted site that attracts 10,000 people every hour. Lets say with that amount of traffic we get on average 50 new page requests every second and it takes about a second to return each page. That means Apache has at any given time about 50 open connections. That’s what it is designed to do.
But if we now put Comet on that site, all 10,000 of those users need to have a long connection open with our server. Similar to the first estimate lets say only 50 of them are really receiving any data at any point. Our Comet server will only assign memory to the tasks of those 50 connections, where Apache would attempt to assign an equal amount of memory to all 10,000 open connections.
That’s why its important in Comet to separate connections from events, or why we use Orbited (or equivalent comet server) instead of Apache.
Continuing…
That’s it for my explanations/intro post. I’m fairly new to the Comet scene myself and am very busy with my startup. I’m sure there are a lot of additions or corrections, please leave them in the comments and I will try to keep up with corrections..
I’ll put more time in this depending on how much interest there is in the comments and subscribriptions. Make sure to check the Comet Category listings for the other posts.
The next post will be an example implementation.
Update: Continue to the next post in the series: Starting Out With Comet (Orbited) Part 2 – Installation and STOMP
4 comments June 9, 2009
New Blog Design By Amanda Scharlemann
A few weeks ago I published a request for a generous designer to create a unique look for this blog. Amanda Scharlemann was the first to respond and was a pleasure to work with!
Amanda is a design student at Bethany Lutheran College and shows great potential. She had never worked with customizing WordPress Templates and the annoying restrictions that come with that, but was eager and quick to learn. We went through several iterations to narrow down exactly what I was looking for and she spent a lot of time examining the example sites I gave her to get a feel for what I would like. At each iteration she had significant improvements, helpful advice, and most importantly was open and attentive to feedback.
I dislike design processes where the designer is stubbornly biased in certain directions and completes 90% of the work before you’re allowed to review. Amanda was not that way at all. I’m not sure if they teach client-design relationships at Bethany or not, but she was a pleasure to work with.
The custom design was launched last week, and so far the results have been great. I’ve seen a doubling in traffic, and several more comments than usual. Perhaps the most impressive statistic so far is that a friend I hadn’t talked to in a while came across my blog while goggling an issue he was having. My article apparently helped him and he was excited to notice (thanks to the personalized sidebar) that the blog belonged to me!
Over time I will probably tweak things here and there as is my nature but I’m over all very pleased with the new look.
Thanks again Amanda!
Add comment June 8, 2009
Find RSS Feed Links With jQuery
This took me a little while to figure out so I thought I’d share. You can use a jQuery selector to find any RSS links on a page very easily.
The following line will return a list of the RSS link elements.
var link_elements = $('link[type="application/rss+xml"]');
The following snippet will create an array of all the urls to the RSS feeds on the page.
var links = [];
$('link[type="application/rss+xml"]').each(function() { links[links.length] = this.href; });
Add comment June 3, 2009
Google AJAX Feed API Example with Socialbrowse
Today I learned that Google has a really awesome AJAX API for RSS and Atom feeds. It allows you to access RSS or Atom feeds using JavaScript in both JSON and XML format without having to setup your own proxy or deal with anything on the server side.
They have a lot of great examples, but I thought I’d share mine with the usecase of showing the feed of my Socialbrowse shares. In your Socialbrowse settings you can specify a public share name. Once added 3 feeds will be created for you
| profile: | http://socialbrowse.com/shares/PUBLIC_NAME/ |
| shares rss: | http://socialbrowse.com/rss/uname/PUBLIC_NAME/ |
| feed rss: | http://socialbrowse.com/rss/socialfeed/PUBLIC_NAME/ |
where PUBLIC_NAME is the share name you chose.
For this example we’re going to fetch and display the “shares rss” feed using the Google AJAX Feeds API. The example is simulating the Socialbrowse blog widget Zack created a few months ago.
To use the Feeds API you need to get a key.
Now we have everything we need. Create an HTML file and paste in the following code. Make sure to change YOUR_API_KEY, and the feed_title and feed_url variables to your values.
<html>
<head>
<script>
var feed_title = "Dave's Socialbrowse Feed";
var feed_url = "http://socialbrowse.com/rss/uname/PUBLIC_NAME/";
</script>
<script type="text/javascript" src="http://www.google.com/jsapi?key=ABQIAAAAmcnSI-mFmfJW8bidL13qfRQHElLAWyCZ_TZ_pzrAvWp3ncTV5hRgCPRM76Ub8GIqowNBQZMVWYastg"></script>
<script type="text/javascript">
google.load("feeds", "1");
function initialize() {
var feedControl = new google.feeds.FeedControl();
feedControl.addFeed(feed_url, feed_title);
feedControl.draw(document.getElementById("feedControl"));
}
google.setOnLoadCallback(initialize);
</script>
</head>
<body>
<div id="feedControl">Loading</div>
</body>
</html>
Viewing the page in a browser will result in something that looks like this
Its not very pretty but its easy enough to style it however you like using CSS. This code can be added anywhere you’d like to display your Socialbrowse shares.
Finally, note that this code can be used with any feed, not just Socialbrowse. If you want to display the latest Google News for example simply change the feed variables to
var feed_title = "Google News"; var feed_url = "http://news.google.com/news?pz=1&ned=us&hl=en&topic=h&num=3&output=rss";
Add comment June 3, 2009
TicTacToe in jQuery
As a demo application for a project of mine I wrote TicTacToe in Javascript using the jQuery framework.
I’ve added excessive comments to the code to provide an easy walk-through example on the jQuery/Javascript game. I’m in no way a JavaScript expert, there are a hundred different ways to program TicTacToe, and this code is far from clean but here it is!
There are just two files, the HTML page, and a page holding the javascript.
tictactoe.html
<html>
<head>
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.js"></script>
<script src="tictactoe.js"></script>
<style>
table.board {
border: 1px solid green;
height: 600px;
width: 600px;
}
body {
text-align: center;
align: center;
}
td {
height: 200px;
width: 200px;
text-align: center;
vertical-align: middle;
font-size: 100px;
font-weight: bold;
font-color: green;
font-family: geniva, verdana, helvetica;
border: 1px solid green;
}
#menu {
text-align: center;
position: absolute;
width: 400;
height: 400;
margin-left: 100px;
margin-top: 100px;
border: 5px double red;
display: none;
vertical-align: middle;
background-color: white;
}
#play_again {
font-size: 20px;
color: green;
}
</style>
</head>
<body>
<table border="0px" align="center">
<tr><td>
<div id="menu"></div>
<div id="board"></div>
</td></tr>
</table>
</body>
tictactoe.js
/* Main Game Handling class */
var TicTacToe = {
turn: "O", // Keeps a record of who's turn it is
board: ["", "", "", "", "", "", "", "", "", ""], // Keeps a record of the TicTacToe Board
win: false, // records who won if the game is over
/* Clears and starts a new game with a new board */
restartGame: function() {
// Draw the board
var board_table = '<table class="board" border="0px" cellpadding="0px" cellspacing="0px" align="center"><tr><td id="ttt0"> </td><td id="ttt1"> </td><td id="ttt2"> </td></tr><tr><td id="ttt3"> </td><td id="ttt4"> </td><td id="ttt5"> </td></tr><tr><td id="ttt6"> </td><td id="ttt7"> </td><td id="ttt8"> </td></tr></table>';
$("#board").html(board_table);
$("#menu").hide();
// clear the board
this.board = ["", "", "", "", "", "", "", "", "", ""];
// Add on-click events to each of the boxes of the board
$("#board td").click(function(e) {
TicTacToe.move( e.target.id );
});
},
/* Handles clicks spaces on the board */
move: function(id) {
var space = $("#" + id); // Board space table element
var num = id.replace("ttt", ""); // # representing the space on the board
// If no one's gone there, and the game isn't over, go there!
if (!this.board[num] && !this.win) {
space.html( this.turn );
this.board[num] = this.turn;
this.nextTurn(); // End turn
}
},
/* Iterate turn and check if anyone one yet */
nextTurn: function() {
this.turn = (this.turn == "O") ? "X" : "O";
this.win = this.check4Win();
if (this.win) {
this.endGame();
}
},
/* Display who won and options for new games */
endGame: function() {
if (this.win == "Cat") {
$("#menu").html("Cats Game.");
} else {
$("#menu").html(this.win + " wins!");
}
$("#menu").append("<div id='play_again'>Play Again</div>");
// Button for playing again.
$("#play_again").click(function () { TicTacToe.restartGame(); });
$("#menu").show();
this.win = false;
},
// If any of these patters of board spaces have all X's or all O's somebody won!
wins: [[0,1,2], [3,4,5], [6,7,8], [0,3,6], [1,4,7], [2,5,8], [0,4,8], [6,4,2]],
/* Check for whether someone won the game of it was a Cat's game. */
check4Win: function() {
// Loop through all possible winning combinations
for (k in this.wins){
var pattern = this.wins[k];
var p = this.board[pattern[0]] + this.board[pattern[1]] + this.board[pattern[2]];
if (p == "XXX") {
return "X"; // X Won!
} else if (p == "OOO") {
return "O"; // O Won!
}
}
// Check if all spaces in the board are filled, then its a Cat's game
var cnt = 0;
for (s in this.board) {
if (this.board[s]) { cnt+=1; }
}
if (cnt == 9) {
return "Cat"; // Cat's game!
}
}
};
$(document).ready(function() {
// Start a game!
TicTacToe.restartGame();
});
I’d originally setup the game on AppJet but unfortunately today they announced that they are closing down their framework and free hosting for a while to focus on one of their successful apps EtherPad. For at least the next month however you can play the game here, and play with the source code here. Feel free to use my code in any way.
If you do use the source for something, or have suggestions on improvements make sure to leave a comment.
4 comments June 2, 2009
GNU Screen Basic Tutorial
If you’ve ever SSHed into a server, ran something, logged off and got frustrated because whatever you were running stopped running because you had to change coffee shops or go to class then you’ve experienced the need for GNU Screen.
Put simply it allows you to maintain a persistent session, a terminal that does not change or quit just because you logged out. Installation is easy. On a debian machine its simply
sudo apt-get install screen
Usage is straight forward as well. Type the word “screen”
$ screen
and you’ll be shown a welcome window, press space to get through. Now you’ll simply see a standard shell. You can execute whatever you want here and the session will not die until you tell it to. You can start many different sessions and toggle between them. Here’s a table of the basic controls for screen.
- Ctl-a Ctl-c, create a new window
- Ctl-a Ctl-a, switch between windows
- Ctl-a n, toggle to next window
- Ctl-a p, toggle to previous window
- Ctl-a 5, toggle to 5th window
- Ctl-a “, get a menu listing all of the window
- Ctl-a A, tool for adding a name to your window, helpful for the menu view
- Ctl-a k, Kills the window
- Ctl-a d, Detach from the session
Another thing to note. Once you detach from a screen, you need to re-attache the next time you run screen. To do that simply specify the -r parameter.
$ screen -r
Update 6-3-2009: A few fixes were made thanks to Ben Finney’s comments
3 comments May 26, 2009
Deploying MoinMoin on Ubuntu using Apache mod_wsgi
I just went through a somewhat lengthy setup process to deploy a moinmoin wiki on Ubuntu. There’s a lot of documentation on it which actually makes it take a bit longer than I’m used to for getting something running on Ubuntu. I thought I’d share my streamlined notes for this common deployment scenario.
The first step is configuration. Greater detail on all of this can be found here.
wget http://static.moinmo.in/files/moin-1.8.3.tar.gz tar -xzvf moin-1.8.3.tar.gz cd moin-1.8.3 python setup.py install --prefix='/usr/local' --record=install.log #Setup the variables. export PREFIX=/usr/local export SHARE=$PREFIX/share/moin export WIKILOCATION=/path/to/wikis export INSTANCE=your_wiki_name export GROUP=www-data export USER=www-data # Now it copies the default data cd $WIKILOCATION mkdir -P $INSTANCE # make a directory for this instance cp -R $SHARE/data $INSTANCE # copy template data directory cp -R $SHARE/underlay $INSTANCE # copy underlay data directory cp $SHARE/config/wikiconfig.py $INSTANCE # copy wiki configuration sample file # Set the permissions chown -R $USER.$GROUP $INSTANCE # check that USER and GROUP are correct chmod -R ug+rwX $INSTANCE # USER.GROUP may read and write chmod -R o-rwx $INSTANCE # everybody else is rejected # Copy over the server config files cp/usr/local/share/moin/server/*.wsgi $INSTANCE cp/usr/local/share/moin/server/*.cgi $INSTANCE # If you want everyone (not just admins) to be able to edit it you need to run this command # chmod -R a+rwX $INSTANCE
Now, if you haven’t already install apache and mod_wsgi
sudo apt-get install apache2 libapache2-mod-wsgi
Next edit your apache httpd.conf file ( /etc/apache2/httpd.conf ) and add the following to the end. More help on this step can be found here.
LoadModule wsgi_module modules/mod_wsgi.so # Loads mod_wsgi
<VirtualHost *> ServerAdmin youremail@example.com
ServerName wiki.example.com
Alias /moin_static183/ "/usr/local/share/moin/htdocs/" #ScriptAlias /developers "/path/to/wikis/your_wiki_name/moin.cgi"
WSGIScriptAlias / /path/to/wikis/your_wiki_name/moin.wsgi WSGIDaemonProcess developerwiki user=www-data group=www-data home=/root process=5 threads=10 maximum-requests=1000 umask=0007
WSGIProcessGroup developerwiki
</VirtualHost>
And finally you need to add your wiki directory to the python path.
echo "/path/to/wikis/your_wiki_name/" > /usr/lib/python2.5/site-packages/wikis.pth
Restart Apache and you should be set up.
Add comment May 21, 2009
Using Arguments in a Shell Script
Within 5 seconds of looking at a shell script I’m usually opening a new file in my text editor to re-write the ugliness into something that makes more visual sense. To me at least python is highly preferable.
Still I use shell scripts all the time to batch a group of commonly used sequential executions, or to abbreviate a commonly used but lengthy commands.
Today I looked into going one step further into the complexities of shell scripts, probably my last step for a while, and discovered how to handle arguments.
The inspiring work case was in using Django manager to run tests for the different apps
./manage.py test --settings test_settings <optional app names>
The following script will check if an argument exists and if it does it will use the argument in the tests command.
if [ -n "$1" ] # Test whether command-line argument is present (non-empty). then ./manage.py test --settings test_settings $1 else ./manage.py test --settings test_settings fi
Notice that $1 refers to the first argument ($0 refers to the name of the executable and $5 refers to the 5th argument).
Save the file as ‘test’ and then modify it as an executable
chmod +x test
And now I can run the tests on their own
./test
or with an argument app
./test auth_user
Expanding
Note that the above example has a limitation of only dealing with the first argument and seems a bit redundant. The entire script can indeed be shortened to
./manage.py test --settings test_settings $@
As $@ represents all arguments after $0.
Add comment May 11, 2009
‘drop’ Database Command for Django Manager
The Django manager is a really handy tool. I wrote earlier about making your own custom managers and there is a lot of other great documentation on it.
Django comes with a bunch of helpful management commands like ‘flush’, ’syncdb’, ‘test’, etc.
I’ve created a generic ‘drop’ command as I felt it was missing. I often found myself going into mysql to drop and re-create a database. This is needed whenever you significantly change your models and need to start over. The ‘drop’ command does that automatically using the database information in your settings file.
The following code is from ‘drop.py’
from django.conf import settingsfrom django.conf import settings
from django.core.management.base import NoArgsCommand
class Command(NoArgsCommand):
help = "Drop and re-create the database"
def handle_noargs(self, **options):
import MySQLdb
print "Connecting..."
db=MySQLdb.connect(host=settings.DATABASE_HOST or "localhost" ,user=settings.DATABASE_USER,
passwd=settings.DATABASE_PASSWORD, port=int(settings.DATABASE_PORT or 3306))
cursor = db.cursor()
print "Dropping database %s" % settings.DATABASE_NAME
cursor.execute("drop database %s; create database %s;" % (settings.DATABASE_NAME, settings.DATABASE_NAME))
print "Dropped"
To install simply place this code in a file called ‘drop.py’ and add it to a management comands folder. If you don’t have a management command folder yet you simply need to create the following file structure in one of your app directories (MY-APP-DIR).
MY-APP-DIR/
management/
__init__.py
commands/
__init__.py
drop.py
Now, whenever you’ need to whipe your database and start fresh you can simply run:
./manage.py drop
2 comments May 10, 2009
