ThingsILearned » Orbited

Starting Out With Comet (Orbited) Part 3 – The Client

godavemon — Mon, 03 Aug 2009 20:58:46 +0000

Comet implementations require both server and client side components. In Part 2 of this series we installed and configured the server side component and then used the example STOMP Test client to test it out and get a feel for what was going on.

In this part of the series we’ll be covering the client side to make our custom interface to the comet server. To demonstrate we’ll go through the steps in creating an app I call EZChat which is basically a bare bones comet chat client. The interface will allow you to choose a name and submit messages that will be broadcast to everyone viewing the page in real time.

For it to work you will have to have the configuration of Part 2 setup and your Orbited server running. For reference I learned most of the client side code in this example by playing with the source of the example STOMP Test client. I recommend taking a look at that source if you need more advanced options or another example.

Includes

At the top of the you first need to include the libraries for the Orbited (/static/Orbited.js) and STOMP (/static/protocols/stomp/stomp.js) client-side implementations.

In between the scripts we’ve set up a TCPScocket. It must be don between the two script includes because the STOMP library needs the socket setup for its execution. We’ve also specified the Orbited port which is necessary especially if you change the port on which the orbited and stomp javascript files are hosted on (they can both be hosted on port 80 along with your other scripts).

Lastly, we’ve also included a popular JSON library. Typically I use jquery-json but I’ve kept this tutorial free of javascript frameworks so as not to add unneeded complexity. If you’d like you can of course switch to whatever JSON library you’re used to; just replace the JSON.stringify and JSON.parse functions with your equivalents.

STOMP Setup

Because we’re dealing with the setup we’ll skip to the bottom of the page and add the following script just before the end tag of the body ().

My apologies as always for the crappiness of the wordpress syntax parser. Lets walk through what’s happening.

At the top, we initialize a new STOMPClient object. The StompClient has the following hooks you can override to trigger your own events.

onopen – Called when the Transport is opened
onclose – Called when the Transport has closed
onerror – Called when the Stomp Client has errored
onerrorframe – Called when there is an error in the message received
onconnectedframe – Called when a the client is fully set up for sending/receiving
onmessageframe – Called when a message is received

The STOMP object also has these functions for connecting and resetting the connections.

reset – Resets the STOMP connection
connect – Connects to the STOMP server
send – Sends the object in the first argument to the channel specified by the second argument

In our simple example only the onconnectedframe and onmessage frames need to be overwritten.

The onconnectedframe function is called when the STOMP server has been connected to and everything is setup for sending and receiving messages. Inside this function we simply need to subscribe/listen to the CHANNEL we’ve setup for our chat. For the example I’ve chosen the channel ‘/ezchat/’. Once subscribed our STOMP client will receive any messages sent to that channel in real time. You can subscribe to multiple channels if you’d like, and you can make clients with different channels if you’d like to have different chat rooms. But for this example we’ll just stick with the hard coded ‘/ezchat/’ channel.

The onmesageframe function is called when a message has been received. It is passed a frame object with the following structure

frame

body: “{“name”:”Dave”,”message”:”awesome this is working”}”
headers: Object
- content-encoding: “utf-8″
- content-length: “51″
- content-type: “text/plain”
- destination: “/ezchat/”
- message-id: “/ezchat/_3″

type: “MESSAGE”

where the body holds the information that has been sent. The STOMP server and client add extra “type” and “headers” objects to communicate between each other. The extra information can be very useful for more complicated applications but for our simple example we’re only interested in the frame “body”.

So you see that the onmessageframe is simply parsing the json object in frame.body of every received message and passing it to my_receive, a function we will soon create.

The Content

For EZChat we need a form where users can specify a name and type messages to send. We also need an area to put the messages. Plop this HTML in at the top of the to handle all of that.

EZChat - Example Comet Client!
Everyone viewing this page will see the messsages you submit instantly.

 Name:
 
 Message:

There are a few things to notice here. First, the important elements in the form have ids ‘chat_name’ and ‘message’ and the area that will be containing all the received messages is called ‘messages’. The names don’t matter except that we’ll use them in the functions we create later.

Second, the onclick event of the submit button is overridden with instead calling the my_send function. We’ll make this function in the next step.

The Functions

Lastly we need to make the custom my_send and my_receive functions that get called to send and receive messages. Insert these functions into the head after the includes.

my_send

The my_send function will get the values from the ‘chat_name’ and ‘message’ form elements, combine them in an object, convert the object to json, and then sends it to the ‘/ezchat/’ channel. The sending is handled using the stomp.send command which takes as input the object to send and second, the channel to send it to.

var CHANNEL = '/ezchat/';
function my_send() { 

// Get the values to send from the form
 var name = document.getElementById('chat_name').value;
 var message = document.getElementById('message').value;

 var msg = {'name': name, 'message': message};

 var json_msg = JSON.stringify(msg);
 stomp.send(json_msg, CHANNEL)
 return false;
 }

my_receive

As discussed earlier the my_receive function gets the JSON parsed version of whatever was sent in ‘frame.body’. In the case of our app its always an object of the format

msg = {‘name’: , ‘message’: }

The my_receive function simply takes this object and converts it into a prettier HTML format and appends it to the top of the message list we created in the HTML.

function my_receive( msg ) {
 console.log('received message', msg);
 // append the  to the top of the list of messages.
 var messages_el = document.getElementById('messages');
 var new_message = "
" + msg['name'] + ": " + msg['message'] + "
";
 messages_el.innerHTML = new_message + messages_el.innerHTML;
 }

That’s it for the code. Scroll to the bottom of the page for the full version of the source.

Results

Ensure that your Orbited server is running as described in Part 2 of the series. Then load up the page we’ve made in two or more separate windows. Choose a different name for each window and start sending each other messages. You’ll notice that both windows will receive the submitted messages almost instantaneously! The beauty of Comet!

If you open your STOMP Test Client and subscribe to the ‘/ezchat/’ channel you’ll see a more raw input on what’s actually being received by the STOMP clients as you chat.

You can see how the STOMP Test client is incredibly handy for debugging.

That’s it for this part of the Tutorial! There are still parts to come including writing a data handler on the server side and hopefully a much requested post on Django integration.

Full Source

For your convenience here’s the full index.html file for this example.


 


 
 
 
 

 

 




EZChat - Example Comet Client!
Everyone viewing this page will see the messsages you submit instantly.

 Name:
 
 Message:

As usual, if there are corrections or questions please be sure to leave them in the comments. And remember to subscribe to catch the rest of the series.

Starting Out With Comet (Orbited) Part 2 – Installation and STOMP

godavemon — Thu, 25 Jun 2009 19:16:20 +0000

In this part of the tutorial we will install and setup the server side of a Comet installation using the Orbited implementation. We’ll also be using the MorbitQ STOMP server to handle message passing and we’ll play around with STOMP/Comet setup using Orbited’s STOMP Test demo. If you have not read the first blog post in this series I advise you do so. I will be assuming you have an understanding of those terms and concepts of the previous post for this tutorial.

Before we start I’d like to quote my source. I’ve learned most of what I’m sharing from Michael Carter’s Tutorial and many hours of playing around.

Installation

We will need to install the latest version of Orbited and also some Stomp tools. The stomp tools aren’t required for this step but will be for anything else you want to do with comet so I’ve included them in the installation steps here as well. Conveniently Orbited is setup in the Cheeseshop. You need python2.5+ and if you have not installed the python setup-tools do so now.

Installation simply consists of the following commands.

easy_install twisted
easy_install orbited
easy_install stompservice
easy_install simplejson

To test if it works enter your python shell and test importing the libraries. The following should load without any errors

$ python
>>> import twisted
>>> import orbited
>>> import stompservice
>>> import simplejson

If you have any troubles there is more info on the Orbited Installation Guide.

Configuration

Orbited is configured with a ‘.cfg’ file. Lets call ours example.cfg. Make a directory anywhere for your project and paste the following into example.cfg

[global]
session.ping_interval = 300

[listen]

http://:9000

stomp://:61613

[access]
* -> localhost:61613

Lets go through the different parts.

The ping interval is a number of seconds for the backend to wait before it pings the client. We’ve got it set up for 5 minutes. Good comet implementations have some sort of pinging system. This is a necessary step as due to current HTTP protocols the client cannot tell if something has gone wrong on the server end. It simply waits happily all day for some sort of response for the server. But with a pinging system setup we can tell the client to refresh its connection if it hasn’t heard from the server in the last 300 seconds, and the server will make sure to ping the client at least every 300 seconds, letting it know that the connection is still alive.

The listen parameters tell the orbited server which ports to listen to and who to proxy requests to. In our configuration port 9000 will be serving static html files, and port 61613 will be a proxy for our STOMP server.

And finally, the access parameter gives permission to proxy to the stomp server.

Lets Run It

To run enter your project directory and type

orbited – -config example.cfg

It should look something like this:

06/24/09 21:05:24:651 INFO   orbited.start   proxy protocol active
06/24/09 21:05:24:511 INFO   orbited.start   Listening http@9000
06/24/09 21:05:24:525 INFO   orbited.start   Listening stomp@61613

STOMP Test

Orbited comes with a really nice STOMP demo that also serves as a nice tool for debugging your setup later. We’ll use it to play around with Comet and understand the concepts behind STOMP.

While your orbited server is running visit the following URL.

http://localhost:9000/static/demos/stomp/

There are 3 important tools/rows we’ll be using here: Connect, Subscribe and Send.

First click on “Connect” to connect to the orbited and stomp servers. We’re using MorbitQ has the stomp client, which doesn’t deal with authentication, so any name and password will work. Notice that the STOMP test shell will now say.

→ Transport openned
→ Connected as user guest

Second change the “destination” in the “Subscribe” row to be “/channel/1/” and click Subscribe. You have now created and subscribed to a channel called “/channel/1/”.

Next we’ll send something to that channel using the Send tool. Again change the destination to “/channel/1/” and type something into the message box replacing “hello”. In the image above I’ve chosen “comet is working!”. Now hit Send and notice that your message shows up in your STOMP shell!

Try sending to other destinations and notice that only messages sent to “/channel/1/” will show up in the stomp shell. We can change that however by subscribing to additional channels. Try subscribing to “/anotherchannel/” and then send it a message. Notice that this setup can handle being subscribed to many different channels at once.

The Real Power

All of the things we’ve tried so far could have been fairly easily implemented with simple AJAX. The real power of comet is that it can push information to the client without having to submit a request. Also, the real power of STOMP is that it smoothly handles message passing between clients. Lets demonstrate both of these now by opening up multiple browser windows all pointing to our STOMP Test.

Open up 3 windows and “Connect” them each to the server. Now subscribe the second window to “/channel/1/”, the third window to “/channel/2/” and the first window to both.

Once setup, using the first window send a message “Message to Channel 1″ to “/channel/1/”. You’ll notice that it not only showed up instantly in window 1 (where you submitted), but also in window 2 (where you did nothing)! The STOMP server has passed the message all clients listening to “/channel/1/” and the Comet server has pushed it to the client without it having to poll for updates!

Now send a message to “/channel/2/” and notice that it shows up in windows 1 and 3, but not 2.

Play around with this setup more to become familiar. Each window can subscribe to any number of channels, and each can send messages to any channel, whether it is subscribed to it or not.

Whats Next

We’ve now setup and tested a Comet implementation. You can see its benefits and understand how it works with STOMP. In the next example we’ll work on the client side of the Comet implementation and write a python STOMP client to handle processing and sending the data on a comet server.

In the mean time you may want to look at the other demo’s that came with Orbited or Michael’s Demo.

Update: Part 3 – The Client is now available.

Starting Out With Comet (Orbited) Part 1

godavemon — Tue, 09 Jun 2009 21:12:07 +0000

This is the first article in a series I’m creating to ease developers into using Comet. The documentation is severely lacking on every comet implementation I’ve come across which I think is Comet’s biggest limitation at the moment. Hopefully this will help those interested in Comet to struggle less with their implementations than I did/do.

The series will specifically focus on using the Orbited implementation of Comet, but many of the concepts will apply to other Comet implementations as well.

This first post will explain some terms and concepts you need to become familiar with.

Comet

Comet is a term referring to a set of techniques (hacks) that enable the server to push data to the client whenever it wants. Traditionally the client has to initiate all requests. If the server wants to send something to the client it has to wait until the client chooses to connect again.

Comet is a set of methods for the server to ping the client.

As a definition, Comet is as arbitrary as the word AJAX. Try not to get hung up on the word any more than you’d get hung up on defining what AJAX is.

What and Why Orbited

Orbited is a Python based implementation of Comet using the Twisted framework. There’s not a lot out there discussing the pro’s and cons of the different comet services so its hard to tell which implementation is better than another. There is clearly not yet a dominant implementation.

I basically chose Orbited because it was the first one I could get up and running thanks to this tutorial, because I love python, and because my implementation will be working with Django.

Orbited is built on the Twisted framework and is event based, so in theory it should scale just fine.

Its important to note that Orbited has both a server side component and a JavaScript library for dealing with the client side of Comet connections as well.

Long-Polling

Long-polling is the technique Orbited uses to maintain a consistent real-time connection between the server and the client, enabling the server to push data to the client whenever it wants. Put simply the client makes a request that is kept open (no immediate response is returned). The server can continue to send data to the client through this open connection until it is terminated by the client.

Streaming

Streaming is another Comet technique. Michael Carter, one of Orbited’s main contributors has a nice article on the advantages of the two techniques.

STOMP

Short for Streaming Text Orientated Messaging Protocol, STOMP is is a message passing protocol commonly used in Comet implementations. It enables a publish/subscribe (pub/sub) model that comes in handy in many real-time applications.

STOMP isn’t a required part of a Comet implementation, but we’re going to be using it in this series.

MorbidQ

MorbidQ is a STOMP server that comes with Orbited. Think of it as Apache, but for STOMP instead of web apps. Its great in that its easy to set up and work with, but its not robust enough for a significant load. When deploying you should instead use RabbitMQ.

Pub/Sub

In the following posts we will be using a publish/subscribe model thanks to the help of STOMP. With this model you can create channels and both publish and subscribe to them. This model works great when multiple clients are listening to the same thing, such as a group chat or a game.

I like to think of it as a telegraphy system. A whole bunch of people can hook up to the wire, and if they do they hear everything everyone else is broadcasting, and can broadcast their own messages as well.

So subscribing is like hooking up your headphones to the transmission lines and publishing is like sending out a signal with your morse key. A different channel would correspond to a different wire.

Orbited Server

Comet requires an event based server to deal with the long requests. Typical servers like Apache for instance were designed to handle a request as quickly as possible. They allocate memory to returning a response as soon as a request is received, and maintain that memory until a response is returned.

An event-based server such as Orbited can chop up the different components of the request/response process. It can accept a request and then forget about it while allocating its memory elsewhere until at some later point something needs to be done (a response returned or data transmitted). For this reason it can handle dramatically more open connections than Apache.

Lets use an example. Pretend we have a typical Apache hosted site that attracts 10,000 people every hour. Lets say with that amount of traffic we get on average 50 new page requests every second and it takes about a second to return each page. That means Apache has at any given time about 50 open connections. That’s what it is designed to do.

But if we now put Comet on that site, all 10,000 of those users need to have a long connection open with our server. Similar to the first estimate lets say only 50 of them are really receiving any data at any point. Our Comet server will only assign memory to the tasks of those 50 connections, where Apache would attempt to assign an equal amount of memory to all 10,000 open connections.

That’s why its important in Comet to separate connections from events, or why we use Orbited (or equivalent comet server) instead of Apache.

Continuing…

That’s it for my explanations/intro post. I’m fairly new to the Comet scene myself and am very busy with my startup. I’m sure there are a lot of additions or corrections, please leave them in the comments and I will try to keep up with corrections..

I’ll put more time in this depending on how much interest there is in the comments and subscribriptions. Make sure to check the Comet Category listings for the other posts.

The next post will be an example implementation.

Update: Continue to the next post in the series: Starting Out With Comet (Orbited) Part 2 – Installation and STOMP