TwinTechs

It’s official. Rails is starting to look like a shiny new hammer* in my toolbox. Actually, it’s been that way for a while. An insane level of productivity, easy to understand persistence framework (unlike hibernate), baked-in REST, lots of community support, opinionated development guidelines, works on the JVM, etc. What’s not to like? It’s probably at the top of my list of considerations for any new web application. Of course, I know that part of my fondness is because I’ve gained a good level of familiarity and expertise with it.

Tools you’re familiar with tend to be the tools you use for most jobs. We (the development community) like to expound about having the right tool for the job and how some frameworks and languages are better suited to certain problems than others. This is true, of course, although not always practical. When you are unfamiliar with a certain tool then you have to weigh the costs of having to spend time learning about it in relation to a project timeline. You may also find that the new shiny thing you’ve learned to use doesn’t quite crack up to be what the salesman made it out to be. Another negative outcome is that you might end up wasting time learning something that produced no real benefit (other than learning what not to do or use [struts2 and JSF come to my mind]) and drops out of favor in the community; i.e. you just spent months wrestling with some framework and nobody wants to pay you to use it anymore and now you somehow feel dirty from the experience.

I realize I’m oversimplifying a bit. I just found myself wondering about what kind of strategy I take in selecting new things to learn or recommending new solutions. I think it boils down to direct analysis and a good deal of hearsay and gut feel.

* i.e. when you have a shiny new hammer, everything starts to look like a nail so you’ll use your hammer even when something is clearly not a nail. But hey, maybe pounding it will work anyways.

I’ve been working on a web application with a Flex front-end that talks to a REST API implemented with Rails. This is a multi-user application that provides social features, such as notifying a user when one of their friends has come online. The client needs an indication that a friend has logged in so the user can be informed with a visual cue. We’ve decided this would be best implemented with messaging. In a traditional web application, this could be accomplished by polling the server in which a request is submitted at a periodic interval. A really simple solution would be to just wait to update the content when the page is re-rendered. For a Flex-based RIA that works a lot like a desktop application, this just doesn’t feel right. What we really want is to push a single message to the client when the friend comes online.

There are many options for doing server push messaging with Flex. We could use any number of messaging protocols like XMPP, STOMP, JMS, or AMQP. For practical reasons, we’ve decided to use a LiveCycle Data Services (LCDS) server. With built in support for RTMP/AMF messaging, Flex and LCDS go hand-in-hand. While this is great for Flex and LCDS, there are really no specific integration points with Rails and LCDS. However, there are ways to integrate the two.

The Scenario

Let’s say we have two users, John and Fred. John and Fred are associated as friends in the application. While John is logged in, he’d like to see when Fred comes online so he can start chatting with him. A general overview of what needs to happen:

John’s client is subscribed to a destination on the LCDS server and will receive messages about friend login statuses. Fred logs into the application via the Rails server. We need a way to indicate this to the LCDS server so it can send out an appropriate message.

When I initially started approaching this issue, I did what most developers do: looked for existing solutions. There was really only one that I found that seemed to fit. It was in the form of a Ruby-based data services plugin for Rails called WebORB. This essentially allows a Flex client to view your Rails app like an LCDS server. However, I found the documentation lacking, support for later versions of Rails seemed to be lagging, and I couldn’t get it to fully work with our app. On top of that, I felt it was too heavy for what I really wanted. I didn’t need to to treat my Rails application like an LCDS server. The bottom line was that I just wanted to send a little bit of JSON to the client in an asynchronous and scalable manner. So with that out the window and the end of our project iteration coming quickly, I decided to do the agile thing and implement something “just enough”.

Implementation Option 1: Create an HTTP Service in LCDS

I came to the realization that LCDS is just a J2EE web application. Because of this, I immediately thought of implementing a servlet that would accept HTTP requests containing message data and then submit that as a data services message. The first thing was to figure out how to submit a message. The API is JMS-like and the code was pretty simple to implement. Following some examples, I came up with:

    MessageBroker msgBroker = MessageBroker.getMessageBroker(null);
    String clientID = UUIDUtils.createUUID();

    AsyncMessage msg = new AsyncMessage();
    msg.setDestination(destination);

    msg.setClientId(clientID);
    msg.setMessageId(UUIDUtils.createUUID());

    msg.setTimestamp(System.currentTimeMillis());
    msg.setBody(message);

    msgBroker.routeMessageToService(msg, null);

I put this into a servlet that accepted parameters for the destination name and message body. The destination needed to be predefined in the flex message-services.xml file under LCDS. Once having done that, I could POST HTTP requests to the servlet and have it send messages to the subscribed clients. The one other thing I needed to do was support message header definitions. I took care of this by passing a headers parameter that contained JSON formatted values that I would then parse and add as headers to the message. All in all, it was a simple solution that was good enough. One thing I wanted to be sure of though was that the message delivery was not a performance bottleneck.

Asynchronous Delivery

In our current scenario, a HTTP request is submitted when the user logs in. A response from this call is not required for the user to successfully log in. Making an external service call that could potentially fail or stall where a response is not needed is a pretty good indication that we should offload this to a concurrent process. I chose to offload these calls to LCDS via RabbitMQ using the Workling plugin. I defined a generic Workling client that listens for messages sent to a special queue (in RabbitMQ) for submitting HTTP requests to our LCDS service:

class MessageWorker < Workling::Base

  def send_message(options={})
    message = LcdsMessage.new
    options.each do |name, value|
      message.send("#{name}=", value)
    end

    begin
      message.submit
    rescue Exception => e
      # TODO: send email on error
      Rails.logger.error "Exception occurred while sending message to destination '#{message.destination}': #{e.message}"
      puts e.backtrace
    end
    message
  end

end

Whenever a message is received from RabbitMQ, the send_message method would be called. I then created a simple wrapper around the message attributes called LcdsMessage. This object also knows how to submit the actual HTTP POST to the LCDS service. I just set the attributes based on the options hash (using a bit of Ruby reflective magic) and submit the request. All this helped with making sure requests were processed more quickly on the Rails side, but what about on the receiving end?

As the LCDS application received HTTP requests, it would need to scale appropriately. Using a typical servlet container that assigns a single thread per request just wouldn’t do. Fortunately, modern containers now support non-blocking IO to serve requests. After some research and seeing some mixed reviews about how well (or not so well) NIO was utilized in Tomcat, I decided to go with Jetty. Reports seemed to indicate Jetty had a very stable and standards compliant NIO api. I went ahead and deployed LCDS with Jetty 7 and did some basic load testing and it held up very well. Here’s how things shaped up with this solution:

Conclusion

While I’m not too crazy about using HTTP for proxying messages to LCDS, the use of message brokering and NIO should make this a more scalable solution. And it’s just enough! Of course, there are some issues and room for improvement:

• Distinguish between requests that need to be persistent or not – if a request is persistent and the HTTP call fails, we should throw it back on the queue for some number of retries.
• Replace the Workling/HTTP service with an AMQP client within LCDS that subscribes to messages from RabbitMQ. This would eliminate some of the overhead that comes with HTTP and Workling clients. It might even be fun to implement an AMQP adapter for LCDS.
• Encode messages from Rails as AMF. Only strings are supported with the current Ruby AMQP and RabbitMQ solution. I’m able to marshal and unmarshal Ruby objects in the AMQP message body, but this doesn’t get decoded on the LCDS side. I’d have to either put JRuby on the LCDS side or consider shooting over AMF directly. I don’t know that this would help much. Our messages will tend to be small and JSON is doing well enough for defining the headers.
• Need to ensure messages consumption is fast enough – it wouldn’t be very good if we ended up with a long queue of messages waiting to be submitted to LCDS. I could fix this by creating multiple Workling clients, but I’ll need a good way to capture some benchmarks.
• No way to receive messages from LCDS to Rails. I’d probably look toward implementing some kind of AMQP client proxy in LCDS. However, we have no real need for this at the moment.

I intend to at least explore the implementation of an AMQP subscriber in LCDS. I think it would actually be pretty simple and could streamline things quite a bit.

In my previous post I talked about the benefits of using a messaging server to broker requests for asynchronous processing. I’d like to follow that up with a simple example of how to use RabbitMQ with Ruby. First, we’ll need a RabbitMQ server installation and the Ruby AMQP library.

RabbitMQ

RabbitMQ is an implementation of AMQP built on the Open Telecom Platform:

Installing RabbitMQ

RabbitMQ server packages are available for various UNIX and Windows platforms. Related installation instructions can be found on the website. For a RedHat 5.3 server, I needed to install newer packages via EPEL to run the latest version of RabbitMQ. My installation process looked like this:

$ wget http://download.fedora.redhat.com/pub/epel/5/i386/erlang-R12B-3.3.el5.i386.rpm
$ sudo rpm -Uvh ./erlang-R12B-3.3.el5.i386.rpm
$ wget http://www.rabbitmq.com/releases/rabbitmq-server/v1.5.4/rabbitmq-server-1.5.4-1.i386.rpm
$ sudo rpm -Uvh ./rabbitmq-server-1.5.4-1.i386.rpm
$ sudo /sbin/chkconfig rabbitmq-server on
$ sudo /sbin/service rabbitmq-server start

The chkconfig command simply requires the rabbitmq-server to start at normal system boot. You may need to create a rabbitmq user as the installation will attempt to assign ownership to this user by default.

To check if the server is running, execute the following as rabbitmq:

$ /usr/lib/rabbitmq/bin/rabbitmqctl status
Status of node 'rabbit@167711-3' ...
[{running_applications,[{rabbit,"RabbitMQ","1.5.3"},
                        {mnesia,"MNESIA  CXC 138 12","4.4.3"},
                        {os_mon,"CPO  CXC 138 46","2.1.6"},
                        {sasl,"SASL  CXC 138 11","2.1.5.3"},
                        {stdlib,"ERTS  CXC 138 10","1.15.3"},
                        {kernel,"ERTS  CXC 138 10","2.12.3"}]},
 {nodes,['rabbit@167711-3']},
 {running_nodes,['rabbit@167711-3']}]
...done.

If your output looks something like this, then you should be good to go.

Ruby AMQP

Now that we have RabbitMQ up and running, it’s time to try out some messaging. The AMQP library is an AMQP client implementation written in Ruby. It’s compatible with any AMQP 0.8 compliant server. Assuming you have an install of Ruby 1.8, 1.9, or JRuby, this should work for you. First, install the tmm1-amqp gem from github:

$ gem sources -a http://gems.github.com
$ sudo gem install tmm1-amqp

Now you’re ready to roll.

Examples

So we have the RabbitMQ server installed and running, and we have a Ruby client library that talks AMQP. Now we can perform some messaging.
In a typical messaging model, a message is published by a producer and sent to some destination. A client may connect to the server and consume messages from this destination. In AMQP, the destination is called the “exchange”. It is where messages are placed on the server for consumption by clients. There are manyoptions for  message delivery and routing. For the sake of this blog post, we’ll go with the simplest way to produce and consume messages just so you can get the basic idea.

Producer

Here’s our sample producer client that will publish messages to the “logins” queue. Write this in a producer.rb file.

require 'rubygems'
require 'mq'

EM.run {
  amq = MQ.new
  queue = amq.queue("logins")
  %w[scott nic robi].each { |login|
      queue.publish(login)
  }
}

The Ruby AMQP library utilizes something called the Ruby EventMachine. This enables event-driven IO and will require you to run your code within an EventMachine loop. To use this, you can run your code within the EM.run block. Next, we use the top-level MQ class to connect to the AMQP server when we call MQ.new. This will connect to the RabbitMQ server using the default port and empty credentials. Then, we create a message queue named “logins” by calling amq.queue("logins"). A queue is used to store and forward messages from the producer. This queue will be created on the fly. Thereafter, we publish messages to the queue by simply calling queue.publish with our message. We then submit three messages that are the names of three user logins.

Subscriber

Now we’ll define the subscriber to our “logins” queue in a subscriber.rb file.

require 'rubygems'
require 'mq'

EM.run {
  amq = MQ.new
  amq.queue("logins").subscribe do |login|
    puts login
  end
}

Once again, we connect to the RabbitMQ server and request the “logins” queue. From the queue, we then ask to subscribe to messages by calling queue("logins").subscribe. This method takes our block of code that will be executed for each message received. Our block simply prints the received messages to STDOUT.

Putting it Together

In one terminal session, let’s start the subscriber:

$ ruby subscriber.rb

Initially, you’ll see nothing printed to STDOUT.
In a second session, start the producer:

$ ruby producer.rb

This should immediately publish messages to our “logins” queue. If you look at your terminal session for subscriber.rb, you should see this:

$ ruby subscriber.rb
scott
nic
robi

The EM loop runs continuously unless explicitly stopped. You can kill the scripts via Ctrl+C.

You could alternatively try running the producer first (which will push messages onto the queue immediately). If you then run the subscriber later, it will still pull the messages off of the queue. This is because the AMQP client will submit queue messages as persistent by default. This means the messages will remain on the queue until they are pulled off by a client. However, they are also non-durable by default. This means that if the server is restarted the messages are lost. AMQP supports many different options for how messages can be routed and persisted and you can specify these options via the AMQP client.

Wrap Up

This should demonstrate how easy it is to use the Ruby AMQP library for messaging with RabbitMQ. Because it based on the AMQP protocol, you can use it as a client to other AMQP servers like ActiveMQ, OpenAMQ, ZeroMQ, and Apache QPID.

For a web application, long running processes can negatively effect scalability as well as make for a poor user experience. Things like database queries, external service calls, document rendering, and other complex business logic can take too long. For web application developers, optimizing these long-running areas may be vital to the success of a site. Depending on the kind of problem, optimization can be approached in a number of  ways, such as:

• Database optimization – Query times are improved by adding indexes, optimizing how queries are built, or even through de-normalization of data.
• Caching – Views rendered to the client, data returned from a database, or anything else can be cached to make subsequent retrieval much faster. For example, if a fairly complex page takes time to render but does not need to be updated for subsequent requests, then there is no need to re-process that same request. Just place it into a cache that is accessible by future requests.
• Asynchronous Processing – Some requests may unavoidably take a long time. This sometimes happens when calling external services or when a large database query is required. If it’s not required for the client response, this processing can be executed concurrently outside the scope of the request so that a response can be returned more quickly.

Asynchronous Processing

Asynchronous processing is the ability to pass a message from one node to another asynchronously. This can be implemented in most languages using concurrent constructs like threads. For example, in J2EE web applications, a request (the message) is received from the client (a node) and is routed to a servlet (the next node) that operates on a different thread. The server is free to go back to processing more requests without having to wait for the servlet to finish. It is from within the context of this servlet that the client waits to get a response (message). Generally, each tier of the system is just a node and you route and pass messages back down to the client. If a node can offload its message asynchronously to the next node, it can return more quickly.

Suppose a client sends a request to a web application service that performs a relatively large amount of work and is forced to wait a number of seconds before a response is returned. Assuming that a large portion of this work does not result in having to return a response to the client synchronously, this could be processed asynchronously. For many situations, simply spawning a thread to execute the work in the background could be enough. However, there are many other times where you need to manage this work. For example, you may need to guarantee the execution of these background tasks or prioritize some tasks over others. The issue of running tasks asynchronously can quickly grow in complexity, and rolling your own solution can be cumbersome and error prone. Fortunately, messaging servers can meet  these needs.

Messaging Servers

Messaging servers address the need for managing the delivery of semantically precise messages from one system to another. They can provide a robust and scalable means for system integration and provide features that clients do not have to implement. A messaging server will typically have one or more of the following features:

• Message routing – Messages can be routed to specific clients in a selective manner
• Message queues – Messages can be queued to wait for consumption by a client
• Point to point messaging – A single message can be consumed by a single dedicated receiver
• Publish/subscribe messaging – Messages sent to a particular destination may be consumed by multiple receivers
• Reliable delivery – Messages can be delivered in a guaranteed manner in which the server may re-attempt to deliver the message and persist until the message is delivered

For the purposes of asynchronously processing long running or resource intensive operations, a messaging server can be used to rout messages to awaiting workers. These workers can even exist on other machines to help distribute load and increase overall processing power.

And because these messaging systems are built from the ground up to be scalable, the message delivery protocols are usually extremely efficient (unlike HTTP) and can handle thousands of requests per second. (Of course, this begs the question: “Why are we still using HTTP?”)

Some example use cases:

• Email delivery – Let’s say, as part of your web application, you deliver emails to a user on signup. There are a few problems with doing this inline to the request; the time to create a connection and deliver a message to the remote smtp server may be longer than you expect. If the smtp server is down; you’ll need a way to queue this email so it can be submitted when the server is back up. If your application has high traffic and your smtp server is not up to snuff, you may want to submit only a certain number of messages at a time so that you don’t bring the server down. You could instead create a queue that accepts messages for email delivery and your application could deliver to that queue and move on. You would then implement a consumer of those messages and process asynchronously as appropriate.
• Refreshing cached data – Suppose your application has a very expensive database query that takes a number of seconds or even minutes to complete. As the smart developer you are, you’ve placed the results of this query into a cache; my favorite is memcached. Let’s say a particular update causes this data in the cache to be invalidated. The query will have to be run again to refresh it. If this call happens in the context of a request, the client is penalized by having to wait. A solution could be to offload this query to a worker who could take care of running the query and updating the cache.
• Image processing – As part of your application, users may upload images that are subsequently modified in various ways such as being resized. Performing this operation may take a significant amount of time. If this happens inline, the client has to wait longer for a response and the server is not able to handle more requests because it’s waiting for the process to finish. Simply dispatch a message and return the response immediately. The message can be routed to a worker who can perform the image manipulation asynchronously.

Asynchronous processing of work through the use of a messaging server can greatly benefit a web application. Fortunately, there are many mature implementations to choose from. Some open source and some proprietary. A few of the popular ones are:

• LCDS
• RabbitMQ
• ActiveMQ
• Tibco
• SonicMQ
• OpenMQ
• Amazon SQS
• Starling

Which to choose  depends on the needs of the application, required language or platform support, integration requirements, and other criteria. Some provide standards-based protocols, like AMQP, JMS, and STOMP, and support which have available clients written in multiple languages.

For a Ruby on Rails and Flex based project I’m currently working on, I’ve chosen to use RabbitMQ (partly to help integrate with an LCDS server). I’ll follow up this post with a practical example on installing RabbitMQ and processing messages with a Ruby AMQP client.

In my previous post, I talked about the benefits of having a corporate dedicated chat server, particularly for a company like Twin Technologies where everyone is distributed.  I also mentioned that I was working on setting up just such a server.  In this post, I’ll be giving some practical details on setting up a client and connecting.

I Already Have [some number] of Chat Accounts. Why Would I Want Another?

OK. So if you’re like most modern technojunkies, you probably have multiple chat accounts with a list of contacts that Santa would be envious of. Why would you want to have another? The one thing that your other accounts most likely do not offer is automated roster management. Other Twin Technology users will be added, removed, or grouped without you ever having to lift a finger. As I wrote about in my previous post, it’s also quite nice to be able to log in on your first day of work and see everyone else at Twin Techs online. There’s also plenty of reasons why XMPP (also known as Jabber) is just cool and why it’s  much more than just for messaging.

The Server

We’ve set up an OpenFire server at jabber.twintechs.org.  OpenFire is an open source instant messaging server that uses the XMPP protocol.  It’s full featured, openly extensible, easy to configure, and widely used.  After downloading, I had it up and running in about 15 minutes.

Getting Connected

1. Get an account. At this time, you’ll need to have an administrator create an account for you. That would be Daniel Hahn or myself.
2. Get a compatible client. There are numerous chat clients that support XMPP.
3. Use your Jabber ID and password to connect. Your Jabber ID will take the form of [username]@twintechs.com

That should be it. Once you log in you should see a list of contacts under the group “TwinTechs”.

Where From Here?

• Wiki page – Twin Techs wiki page about the corporate Jabber server. A bit sparse now but will be updated on a continuing basis.
• Jabber.org – main Jabber website
• xmpp.org – XMPP standards foundation
• XMPP4R – Ruby library for XMPP messaging.
• XIFF – Flex XMPP library.
• Smack Java XMPP library.

I’m just wrapping up my first week at TwinTechs.  It’s been fun so far and I’m excited about what is ahead.  This is my third position over the past 8 years where I will once again be working from home.  Having worked as a telecommuter for so long, I’ve become very familiar with the challenges that come with telecommuting.  One of the main challenges most of us face is maintaining a sense of personal connectedness.  How successfully you compensate depends not only on your personal skills and experience but also upon the culture of the organization you are working for.

My first real telecommuting position was at FGM Inc.  I was one of the only telecommuters in a company of 180 people and the only one that was hired on as a telecommuter (as opposed to moving into it). Over my six years there, I maintained a schedule where I’d travel to the office one day a week.  For me, this was vital face-time and I looked forward to being in the office with everyone.  I frequently would take part in job related conversations that were struck up in someone’s office or hallway.  I noted how things like that would never happen when working at home and thought about what I, or everyone else, might be missing out on.  I gradually became accustomed to working at home and eventually preferred it but I did have to fight the tendency for people to think I was out-of-sight-out-of-mind.  A tool that I found vital was AOL instant messenger.  I would ping people throughout the day and relied on it heavily for maintaining a presence.  Almost everyone on the team had AOL IM accounts and made a pretty good habit of being online, although things didn’t start out this way.  Over the course of the next few years I voiced the need for people to be “online” and eventually it became standard practice on our team.  Having that chat client was the next best thing to bumping into people in the hallway.

I moved on to work for OpSource in 2007.  This is a company that, for me, stood in stark contrast to FGM as it is spread across timezones and continents.  They have to deal with being connected on a company-wide scale.  On the first day of starting at OpSource I was given my account credentials and I logged into their corporate Jabber server.  (Jabber is an open-source XML-based instant messaging platform that’s in wide use today.)  Wham!  I was hit with a “buddy” list of every person in the company organized by department.  This was a heck of a lot better than slowly building contact information over time and adding them manually.  In fact, it was corporate policy that if you were working that you should be logged into the Jabber server.  This was great for me (working from Richmond, VA) in that I could see when certain people were coming online in California or elsewhere.  Someone could tell me to talk to so-and-so and I could just look them up on the list and start chatting with them.  This didn’t solve all my telecommuting woes, but it was certainly a breath of fresh air from where I had just been.

Perhaps I was spoiled by my pre-populated buddy list, I don’t know.  As I’ve been starting at TwinTechs I’ve missed being able to simply see a list of my fellow cohorts.  There’s been a number of names thrown around that don’t mean anything to me yet.  Sure, I could look up their contact info on the wiki and email them, but it’s just not the same as being able to see someone’s presence online and being able to strike up a real-time conversation.  I’ve expressed the idea of having a corporate Jabber server to some of our folks and they seem to be enthusiastically in favor of it.  I went ahead and set up an Openfire Jabber server and added a handful of TwinTechs accounts to get something rolling.  Eventually, I would see us being able to log into jabber.twintechs.com with your favorite chat client. 

I intended to talk about using Jabber at TwinTechs but I found myself pondering some of my reasonings for wanting to.  I plan to follow up with a post on more practical things related to Jabber client installation, configuration, features, etc.