TwinTechs

We recently had an internal project that required us to extract meta-data from video files.  We needed information such as the number of streams, length of the video, format, creator, encoding, and anything else we could gather.  This information was then stored in a database for the easy extraction and categorization of a large set of videos.

Twin has also recently started an internal streaming media framework using tools like Red5, Xuggler, and ffmpeg.  Using the expertise we gained with Xuggler, I decided to quickly write our own meta-data extractor. Later I decided on an even simpler solution.

The first method was to write our own meta-data extractor using Xuggler.  In order to use Xuggler, download and install it from http://www.xuggle.com/xuggler/downloads/.  In Linux, you will also need to add the XUGGLE_HOME/lib directory to the LD_LIBRARY_PATH environment variable so the native libraries are picked up by Java (Xuggler uses JNI to call its own C++ wrappers around ffmpeg).  The Windows installer does this automatically for you.

After Xuggler is installed, I cracked open the demos it had and modified one.  The code is very simple and required few changes.  The Java code is downloadable below.  Just make sure the xuggle-xuggler.jar (included in the XUGGLER_HOME/share/java/lib folder) is in your classpath before compiling and running it, and pass in a movie file as a command line parameter.  For example:

java com.twintechs.video.demo.GetMediaMetaData /path/to/your/video.mpg.

The output will be something along the lines of:
Opening video file: /home/dave/workspace/twin/MetaData/video/test.mpe
null (probesize):     32000
set mux rate (muxrate):     0
set packet size (packetsize):     0
null (fflags):     0x00000000
set the track number (track):     0
set the year (year):     0
how many microseconds are analyzed to estimate duration (analyzeduration):     3000000
decryption key (cryptokey):     §C,
max memory used for timestamp index (per stream) (indexmem):     1048576
max memory used for buffering real-time frames (rtbufsize):     3041280
print specific debug info (fdebug):     0x00000000
file "/home/dave/workspace/twin/MetaData/video/test.mpe": 2 streams; duration (ms): 53700; start time (ms): 149; file size (bytes): 11323804; bit rate: 1686972;
stream 0: type: CODEC_TYPE_VIDEO; codec: CODEC_ID_MPEG1VIDEO; duration: 4833000; start time: 13490; language: unknown; timebase: 1/90000; coder tb: 1/30; width: 432; height: 320; format: YUV420P; frame-rate: 30.00;
stream 1: type: CODEC_TYPE_AUDIO; codec: CODEC_ID_MP2; duration: 4810187; start time: 13490; language: unknown; timebase: 1/90000; coder tb: 1/90000; sample rate: 44100; channels: 2; format: FMT_S16

And a lot of other stuff.

The output is hard to make sense of, however, since it’s essentially in its rawest format.  Instead of painstakingly mapping each raw field to human-readable English, (for the record: not a good use of a consultant’s time!) I searched a little online for a pre-existing solution.  Not surprisingly, there were many.  The best one, which is also an open source project on SourceForge, was MediaInfo.

So, the second method of extracting data from source videos, which is a little easier but more limited because the program’s sole purpose is for info on media (e.g. you couldn’t play a video sample if you wanted), is to download and install MediaInfo.  The website is at http://mediainfo.sourceforge.net/en.  Download and install this on your platform, and then using either the GUI or the CLI (command line interface), you can see the available information of essentially any video.  For example, using the CLI, you can then type something like:

MediaInfo –Full “/path/to/your/video.mpg”

This will give a lot of information, most of which is human-readable. Then this information can be extracted using any of a number of methods, like Python or Perl text parsing, etc., and fed into a database for categorization.  And there you have it; two easy ways to extract every bit of information from a video file and categorize it for further use!

Here is a example file getmediametadata .

I’ve been working on a web application with a Flex front-end that talks to a REST API implemented with Rails. This is a multi-user application that provides social features, such as notifying a user when one of their friends has come online. The client needs an indication that a friend has logged in so the user can be informed with a visual cue. We’ve decided this would be best implemented with messaging. In a traditional web application, this could be accomplished by polling the server in which a request is submitted at a periodic interval. A really simple solution would be to just wait to update the content when the page is re-rendered. For a Flex-based RIA that works a lot like a desktop application, this just doesn’t feel right. What we really want is to push a single message to the client when the friend comes online.

There are many options for doing server push messaging with Flex. We could use any number of messaging protocols like XMPP, STOMP, JMS, or AMQP. For practical reasons, we’ve decided to use a LiveCycle Data Services (LCDS) server. With built in support for RTMP/AMF messaging, Flex and LCDS go hand-in-hand. While this is great for Flex and LCDS, there are really no specific integration points with Rails and LCDS. However, there are ways to integrate the two.

The Scenario

Let’s say we have two users, John and Fred. John and Fred are associated as friends in the application. While John is logged in, he’d like to see when Fred comes online so he can start chatting with him. A general overview of what needs to happen:

John’s client is subscribed to a destination on the LCDS server and will receive messages about friend login statuses. Fred logs into the application via the Rails server. We need a way to indicate this to the LCDS server so it can send out an appropriate message.

When I initially started approaching this issue, I did what most developers do: looked for existing solutions. There was really only one that I found that seemed to fit. It was in the form of a Ruby-based data services plugin for Rails called WebORB. This essentially allows a Flex client to view your Rails app like an LCDS server. However, I found the documentation lacking, support for later versions of Rails seemed to be lagging, and I couldn’t get it to fully work with our app. On top of that, I felt it was too heavy for what I really wanted. I didn’t need to to treat my Rails application like an LCDS server. The bottom line was that I just wanted to send a little bit of JSON to the client in an asynchronous and scalable manner. So with that out the window and the end of our project iteration coming quickly, I decided to do the agile thing and implement something “just enough”.

Implementation Option 1: Create an HTTP Service in LCDS

I came to the realization that LCDS is just a J2EE web application. Because of this, I immediately thought of implementing a servlet that would accept HTTP requests containing message data and then submit that as a data services message. The first thing was to figure out how to submit a message. The API is JMS-like and the code was pretty simple to implement. Following some examples, I came up with:

    MessageBroker msgBroker = MessageBroker.getMessageBroker(null);
    String clientID = UUIDUtils.createUUID();

    AsyncMessage msg = new AsyncMessage();
    msg.setDestination(destination);

    msg.setClientId(clientID);
    msg.setMessageId(UUIDUtils.createUUID());

    msg.setTimestamp(System.currentTimeMillis());
    msg.setBody(message);

    msgBroker.routeMessageToService(msg, null);

I put this into a servlet that accepted parameters for the destination name and message body. The destination needed to be predefined in the flex message-services.xml file under LCDS. Once having done that, I could POST HTTP requests to the servlet and have it send messages to the subscribed clients. The one other thing I needed to do was support message header definitions. I took care of this by passing a headers parameter that contained JSON formatted values that I would then parse and add as headers to the message. All in all, it was a simple solution that was good enough. One thing I wanted to be sure of though was that the message delivery was not a performance bottleneck.

Asynchronous Delivery

In our current scenario, a HTTP request is submitted when the user logs in. A response from this call is not required for the user to successfully log in. Making an external service call that could potentially fail or stall where a response is not needed is a pretty good indication that we should offload this to a concurrent process. I chose to offload these calls to LCDS via RabbitMQ using the Workling plugin. I defined a generic Workling client that listens for messages sent to a special queue (in RabbitMQ) for submitting HTTP requests to our LCDS service:

class MessageWorker < Workling::Base

  def send_message(options={})
    message = LcdsMessage.new
    options.each do |name, value|
      message.send("#{name}=", value)
    end

    begin
      message.submit
    rescue Exception => e
      # TODO: send email on error
      Rails.logger.error "Exception occurred while sending message to destination '#{message.destination}': #{e.message}"
      puts e.backtrace
    end
    message
  end

end

Whenever a message is received from RabbitMQ, the send_message method would be called. I then created a simple wrapper around the message attributes called LcdsMessage. This object also knows how to submit the actual HTTP POST to the LCDS service. I just set the attributes based on the options hash (using a bit of Ruby reflective magic) and submit the request. All this helped with making sure requests were processed more quickly on the Rails side, but what about on the receiving end?

As the LCDS application received HTTP requests, it would need to scale appropriately. Using a typical servlet container that assigns a single thread per request just wouldn’t do. Fortunately, modern containers now support non-blocking IO to serve requests. After some research and seeing some mixed reviews about how well (or not so well) NIO was utilized in Tomcat, I decided to go with Jetty. Reports seemed to indicate Jetty had a very stable and standards compliant NIO api. I went ahead and deployed LCDS with Jetty 7 and did some basic load testing and it held up very well. Here’s how things shaped up with this solution:

Conclusion

While I’m not too crazy about using HTTP for proxying messages to LCDS, the use of message brokering and NIO should make this a more scalable solution. And it’s just enough! Of course, there are some issues and room for improvement:

• Distinguish between requests that need to be persistent or not – if a request is persistent and the HTTP call fails, we should throw it back on the queue for some number of retries.
• Replace the Workling/HTTP service with an AMQP client within LCDS that subscribes to messages from RabbitMQ. This would eliminate some of the overhead that comes with HTTP and Workling clients. It might even be fun to implement an AMQP adapter for LCDS.
• Encode messages from Rails as AMF. Only strings are supported with the current Ruby AMQP and RabbitMQ solution. I’m able to marshal and unmarshal Ruby objects in the AMQP message body, but this doesn’t get decoded on the LCDS side. I’d have to either put JRuby on the LCDS side or consider shooting over AMF directly. I don’t know that this would help much. Our messages will tend to be small and JSON is doing well enough for defining the headers.
• Need to ensure messages consumption is fast enough – it wouldn’t be very good if we ended up with a long queue of messages waiting to be submitted to LCDS. I could fix this by creating multiple Workling clients, but I’ll need a good way to capture some benchmarks.
• No way to receive messages from LCDS to Rails. I’d probably look toward implementing some kind of AMQP client proxy in LCDS. However, we have no real need for this at the moment.

I intend to at least explore the implementation of an AMQP subscriber in LCDS. I think it would actually be pretty simple and could streamline things quite a bit.

In my previous post I talked about the benefits of using a messaging server to broker requests for asynchronous processing. I’d like to follow that up with a simple example of how to use RabbitMQ with Ruby. First, we’ll need a RabbitMQ server installation and the Ruby AMQP library.

RabbitMQ

RabbitMQ is an implementation of AMQP built on the Open Telecom Platform:

Installing RabbitMQ

RabbitMQ server packages are available for various UNIX and Windows platforms. Related installation instructions can be found on the website. For a RedHat 5.3 server, I needed to install newer packages via EPEL to run the latest version of RabbitMQ. My installation process looked like this:

$ wget http://download.fedora.redhat.com/pub/epel/5/i386/erlang-R12B-3.3.el5.i386.rpm
$ sudo rpm -Uvh ./erlang-R12B-3.3.el5.i386.rpm
$ wget http://www.rabbitmq.com/releases/rabbitmq-server/v1.5.4/rabbitmq-server-1.5.4-1.i386.rpm
$ sudo rpm -Uvh ./rabbitmq-server-1.5.4-1.i386.rpm
$ sudo /sbin/chkconfig rabbitmq-server on
$ sudo /sbin/service rabbitmq-server start

The chkconfig command simply requires the rabbitmq-server to start at normal system boot. You may need to create a rabbitmq user as the installation will attempt to assign ownership to this user by default.

To check if the server is running, execute the following as rabbitmq:

$ /usr/lib/rabbitmq/bin/rabbitmqctl status
Status of node 'rabbit@167711-3' ...
[{running_applications,[{rabbit,"RabbitMQ","1.5.3"},
                        {mnesia,"MNESIA  CXC 138 12","4.4.3"},
                        {os_mon,"CPO  CXC 138 46","2.1.6"},
                        {sasl,"SASL  CXC 138 11","2.1.5.3"},
                        {stdlib,"ERTS  CXC 138 10","1.15.3"},
                        {kernel,"ERTS  CXC 138 10","2.12.3"}]},
 {nodes,['rabbit@167711-3']},
 {running_nodes,['rabbit@167711-3']}]
...done.

If your output looks something like this, then you should be good to go.

Ruby AMQP

Now that we have RabbitMQ up and running, it’s time to try out some messaging. The AMQP library is an AMQP client implementation written in Ruby. It’s compatible with any AMQP 0.8 compliant server. Assuming you have an install of Ruby 1.8, 1.9, or JRuby, this should work for you. First, install the tmm1-amqp gem from github:

$ gem sources -a http://gems.github.com
$ sudo gem install tmm1-amqp

Now you’re ready to roll.

Examples

So we have the RabbitMQ server installed and running, and we have a Ruby client library that talks AMQP. Now we can perform some messaging.
In a typical messaging model, a message is published by a producer and sent to some destination. A client may connect to the server and consume messages from this destination. In AMQP, the destination is called the “exchange”. It is where messages are placed on the server for consumption by clients. There are manyoptions for  message delivery and routing. For the sake of this blog post, we’ll go with the simplest way to produce and consume messages just so you can get the basic idea.

Producer

Here’s our sample producer client that will publish messages to the “logins” queue. Write this in a producer.rb file.

require 'rubygems'
require 'mq'

EM.run {
  amq = MQ.new
  queue = amq.queue("logins")
  %w[scott nic robi].each { |login|
      queue.publish(login)
  }
}

The Ruby AMQP library utilizes something called the Ruby EventMachine. This enables event-driven IO and will require you to run your code within an EventMachine loop. To use this, you can run your code within the EM.run block. Next, we use the top-level MQ class to connect to the AMQP server when we call MQ.new. This will connect to the RabbitMQ server using the default port and empty credentials. Then, we create a message queue named “logins” by calling amq.queue("logins"). A queue is used to store and forward messages from the producer. This queue will be created on the fly. Thereafter, we publish messages to the queue by simply calling queue.publish with our message. We then submit three messages that are the names of three user logins.

Subscriber

Now we’ll define the subscriber to our “logins” queue in a subscriber.rb file.

require 'rubygems'
require 'mq'

EM.run {
  amq = MQ.new
  amq.queue("logins").subscribe do |login|
    puts login
  end
}

Once again, we connect to the RabbitMQ server and request the “logins” queue. From the queue, we then ask to subscribe to messages by calling queue("logins").subscribe. This method takes our block of code that will be executed for each message received. Our block simply prints the received messages to STDOUT.

Putting it Together

In one terminal session, let’s start the subscriber:

$ ruby subscriber.rb

Initially, you’ll see nothing printed to STDOUT.
In a second session, start the producer:

$ ruby producer.rb

This should immediately publish messages to our “logins” queue. If you look at your terminal session for subscriber.rb, you should see this:

$ ruby subscriber.rb
scott
nic
robi

The EM loop runs continuously unless explicitly stopped. You can kill the scripts via Ctrl+C.

You could alternatively try running the producer first (which will push messages onto the queue immediately). If you then run the subscriber later, it will still pull the messages off of the queue. This is because the AMQP client will submit queue messages as persistent by default. This means the messages will remain on the queue until they are pulled off by a client. However, they are also non-durable by default. This means that if the server is restarted the messages are lost. AMQP supports many different options for how messages can be routed and persisted and you can specify these options via the AMQP client.

Wrap Up

This should demonstrate how easy it is to use the Ruby AMQP library for messaging with RabbitMQ. Because it based on the AMQP protocol, you can use it as a client to other AMQP servers like ActiveMQ, OpenAMQ, ZeroMQ, and Apache QPID.

For a web application, long running processes can negatively effect scalability as well as make for a poor user experience. Things like database queries, external service calls, document rendering, and other complex business logic can take too long. For web application developers, optimizing these long-running areas may be vital to the success of a site. Depending on the kind of problem, optimization can be approached in a number of  ways, such as:

• Database optimization – Query times are improved by adding indexes, optimizing how queries are built, or even through de-normalization of data.
• Caching – Views rendered to the client, data returned from a database, or anything else can be cached to make subsequent retrieval much faster. For example, if a fairly complex page takes time to render but does not need to be updated for subsequent requests, then there is no need to re-process that same request. Just place it into a cache that is accessible by future requests.
• Asynchronous Processing – Some requests may unavoidably take a long time. This sometimes happens when calling external services or when a large database query is required. If it’s not required for the client response, this processing can be executed concurrently outside the scope of the request so that a response can be returned more quickly.

Asynchronous Processing

Asynchronous processing is the ability to pass a message from one node to another asynchronously. This can be implemented in most languages using concurrent constructs like threads. For example, in J2EE web applications, a request (the message) is received from the client (a node) and is routed to a servlet (the next node) that operates on a different thread. The server is free to go back to processing more requests without having to wait for the servlet to finish. It is from within the context of this servlet that the client waits to get a response (message). Generally, each tier of the system is just a node and you route and pass messages back down to the client. If a node can offload its message asynchronously to the next node, it can return more quickly.

Suppose a client sends a request to a web application service that performs a relatively large amount of work and is forced to wait a number of seconds before a response is returned. Assuming that a large portion of this work does not result in having to return a response to the client synchronously, this could be processed asynchronously. For many situations, simply spawning a thread to execute the work in the background could be enough. However, there are many other times where you need to manage this work. For example, you may need to guarantee the execution of these background tasks or prioritize some tasks over others. The issue of running tasks asynchronously can quickly grow in complexity, and rolling your own solution can be cumbersome and error prone. Fortunately, messaging servers can meet  these needs.

Messaging Servers

Messaging servers address the need for managing the delivery of semantically precise messages from one system to another. They can provide a robust and scalable means for system integration and provide features that clients do not have to implement. A messaging server will typically have one or more of the following features:

• Message routing – Messages can be routed to specific clients in a selective manner
• Message queues – Messages can be queued to wait for consumption by a client
• Point to point messaging – A single message can be consumed by a single dedicated receiver
• Publish/subscribe messaging – Messages sent to a particular destination may be consumed by multiple receivers
• Reliable delivery – Messages can be delivered in a guaranteed manner in which the server may re-attempt to deliver the message and persist until the message is delivered

For the purposes of asynchronously processing long running or resource intensive operations, a messaging server can be used to rout messages to awaiting workers. These workers can even exist on other machines to help distribute load and increase overall processing power.

And because these messaging systems are built from the ground up to be scalable, the message delivery protocols are usually extremely efficient (unlike HTTP) and can handle thousands of requests per second. (Of course, this begs the question: “Why are we still using HTTP?”)

Some example use cases:

• Email delivery – Let’s say, as part of your web application, you deliver emails to a user on signup. There are a few problems with doing this inline to the request; the time to create a connection and deliver a message to the remote smtp server may be longer than you expect. If the smtp server is down; you’ll need a way to queue this email so it can be submitted when the server is back up. If your application has high traffic and your smtp server is not up to snuff, you may want to submit only a certain number of messages at a time so that you don’t bring the server down. You could instead create a queue that accepts messages for email delivery and your application could deliver to that queue and move on. You would then implement a consumer of those messages and process asynchronously as appropriate.
• Refreshing cached data – Suppose your application has a very expensive database query that takes a number of seconds or even minutes to complete. As the smart developer you are, you’ve placed the results of this query into a cache; my favorite is memcached. Let’s say a particular update causes this data in the cache to be invalidated. The query will have to be run again to refresh it. If this call happens in the context of a request, the client is penalized by having to wait. A solution could be to offload this query to a worker who could take care of running the query and updating the cache.
• Image processing – As part of your application, users may upload images that are subsequently modified in various ways such as being resized. Performing this operation may take a significant amount of time. If this happens inline, the client has to wait longer for a response and the server is not able to handle more requests because it’s waiting for the process to finish. Simply dispatch a message and return the response immediately. The message can be routed to a worker who can perform the image manipulation asynchronously.

Asynchronous processing of work through the use of a messaging server can greatly benefit a web application. Fortunately, there are many mature implementations to choose from. Some open source and some proprietary. A few of the popular ones are:

• LCDS
• RabbitMQ
• ActiveMQ
• Tibco
• SonicMQ
• OpenMQ
• Amazon SQS
• Starling

Which to choose  depends on the needs of the application, required language or platform support, integration requirements, and other criteria. Some provide standards-based protocols, like AMQP, JMS, and STOMP, and support which have available clients written in multiple languages.

For a Ruby on Rails and Flex based project I’m currently working on, I’ve chosen to use RabbitMQ (partly to help integrate with an LCDS server). I’ll follow up this post with a practical example on installing RabbitMQ and processing messages with a Ruby AMQP client.

David Ladd, Nic Tunney, and Audrey Hubbard report high levels of traffic and interest in the Twin Technologies booth #2608 at FOSE this week. Reporters from the Technical Times stopped by to talk about Twin Technologies’ JumpStart: Workforce Management, which is up for a Best in Show Award.

FOSE, which focuses on technology for government applications, brings together the top innovators in the government space. This year’s trends center around integration of services, developing self-service options, cutting cost, and improving efficiency. Twin Technologies Government Solutions systems integration and RIA services offer creative and proven methods of meeting these goals. With over 25 sign-ups for LiveCycle training and more than ten leads on Day One, it’s clear that Twin Technologies solutions resonate with a government agency audience.

In my previous post, I talked about the benefits of having a corporate dedicated chat server, particularly for a company like Twin Technologies where everyone is distributed.  I also mentioned that I was working on setting up just such a server.  In this post, I’ll be giving some practical details on setting up a client and connecting.

I Already Have [some number] of Chat Accounts. Why Would I Want Another?

OK. So if you’re like most modern technojunkies, you probably have multiple chat accounts with a list of contacts that Santa would be envious of. Why would you want to have another? The one thing that your other accounts most likely do not offer is automated roster management. Other Twin Technology users will be added, removed, or grouped without you ever having to lift a finger. As I wrote about in my previous post, it’s also quite nice to be able to log in on your first day of work and see everyone else at Twin Techs online. There’s also plenty of reasons why XMPP (also known as Jabber) is just cool and why it’s  much more than just for messaging.

The Server

We’ve set up an OpenFire server at jabber.twintechs.org.  OpenFire is an open source instant messaging server that uses the XMPP protocol.  It’s full featured, openly extensible, easy to configure, and widely used.  After downloading, I had it up and running in about 15 minutes.

Getting Connected

1. Get an account. At this time, you’ll need to have an administrator create an account for you. That would be Daniel Hahn or myself.
2. Get a compatible client. There are numerous chat clients that support XMPP.
3. Use your Jabber ID and password to connect. Your Jabber ID will take the form of [username]@twintechs.com

That should be it. Once you log in you should see a list of contacts under the group “TwinTechs”.

Where From Here?

• Wiki page – Twin Techs wiki page about the corporate Jabber server. A bit sparse now but will be updated on a continuing basis.
• Jabber.org – main Jabber website
• xmpp.org – XMPP standards foundation
• XMPP4R – Ruby library for XMPP messaging.
• XIFF – Flex XMPP library.
• Smack Java XMPP library.

Running an agile project with a distributed (remote) team can be challenging at times.  The root of all evil for any agile project is communication breakdown.  At Twin Technologies, we employ technology to help improve communication which in turn keeps these projects running smoothly.

We employ a few tools on our Agile projects when working with a remote team:

1. Online task board (i.e. Rally, Version One, etc) (free to 10 users)
2. 24/7 Conference Bridge (free)
3. Jabber for constant communication (free)
4. Subversion Repository for code (free)
5. Trac Wiki for documentation (free)
6. Adobe Connect Room (free to three users)

Each team member on an agile project is sure to have a preferred method of communication.  The tools above allow each member to communicate however they feel most comfortable, or however they feel the current situation deems most effective (IM, phone, screen sharing, etc).

The tools listed above also ensure that team members are able to efficiently work from any location since all information is stored securely within the cloud and not tied to a specific location.

Stand-up meetings, a must for any agile project, also become more difficult in a distributed environment.  These issues are easy to overcome with proper training.

Pre Meeting:

1. Log current status on the online task board (in progress, done)
2. Log current done % on the online task board
3. Minimize all other windows, aside from the task board view
4. Disable alerts for IM, Twitter, Email, etc (c’mon, it’s only 15 minutes!)

Basic Stand-up:

1. Have the online task board open in front of you
2. Follow an established order for team members (I like reverse alphabetical)
3. Keep on topic:

• What did you do yesterday?
• What are you doing today?
• What are your current roadblocks?

By providing some freely available tools to your distributed team, you too can ensure a successful agile project rooted in solid communication.

Working with several Government organizations over the last few years I’ve noticed that Web 2.0 has not gone unnoticed by Government IT professionals, but due to certain restrictions such as section 508 compliance, browser adoption and standardization, etc, Web 2.0 was not a boom within Federal IT initiatives.  This does not mean that some Agencies are not embracing technologies such as Adobe Flex and Ajax (to be fair some Federal orgs are doing some very high tech RIA work), but the adoption rate was lower (or perhaps slower) than forecasted IMO.

This is a shame in some aspects.  Web 2.0 was not all about RIAs.  It also included social networks and applications delivered as a service (SaaS).  Social networks would serve not only to disseminate information and personalize Government to the private sector, but could also be leveraged to connect IT professionals involved in the eGov initiative as it is often the case in Government that many solutions are moving in parallel to one another across Agencies, as opposed to building off one another.  This can even occur when Agencies are working on cooperative initiatives.  Employing the SaaS model could help lower spending within administration and reduce R&D costs while freeing up budgets and internal IT resources to be applied to more pressing issues such as security programs and high tech border initiatives (just an example).

That being said, Web 3.0 could be an impressive paradigm for Government and inter-Agency cooperation.  Like Web 2.0, Web 3.0 means many things to many people.  One thought is a nebulous network of data available for dissemination both across secure channels and those open to the public domain.  Data can be pulled from this network and used in various technologies from desktop applications to mobile widgets.  Imagine if local law enforcement was able to instantly query the Social Security Agency’s database from their in-car HUD while (and here comes the magic) using data from Department of Homeland Security and FBI through secure channels within the same data query to cross check all available data for a suspicious individual (i.e. Service Orchestration).  This data could then be analyzed and massaged in real time specifically for the enforcement officer’s current use case.  This is all possible through the ‘webulous’ cloud of semantic data that will make up one aspect of Web 3.0.

The possibilities for inter-Agency communication via secure and open channels are unlimited.  Agencies will be able to remove previous restrictions by allowing real time querying of sensitive and non-privileged data through secure and public channels without a separate work effort for each organization and project needing data.  A semantic query language would allow the end user access to only the data they are allowed to see AND they will be able to assemble any data across that Agency while mashing it into data queried from other Federal, state and local organizations.

The most exciting news?  This technology is available today.  Contact me at Twin Technologies today to see what efficiencies your organization can gain from employing these methodologies.

It is an understatement to say that it can be difficult to sell the idea of an Agile project methodology within the Federal space.  When you begin talking about iterations, sprints, chickens and pigs and stand-ups with a decision maker (or even a seasoned government programmer) it can be a tough pill for them to swallow.
[Read more →]