Charles, what does one need to do to get these stats (which flags, I'm guessing) for a "remote" JVM / JRuby proc? Can I write a ruby tool/process to attach to another JVM process and extract these for me?

Douglas, thanks for the review! The reason we are calling it engagement is due to the overall trend we're seeing in the community: once the story is published, it really takes on a life of its own on the networks we track. Entire conversations happen around the original story, and they are not necessarily tied to the original content. The original story is a seed, and the conversation that happens is often an evolution on the original story.

Also, as an interesting data point, based on our data mining we know that over 80% of the engagement events happen off the original site (interactions on the social sites) -- which is where PostRank analytics comes in. We aggregate all that activity in a single place for publishers to see and follow up on.

Adam, Doug... Very much so! In fact, we have a "PostRank Analytics" product in early private beta to do exactly that. If you guys are interested in giving it a try, ping me at ilya at postrank.com, and we'll set you up!

Brad, out of curiosity, which watch model? I've been contemplating the upgrade from my nike+ setup which works great, but I'd like to get the heartrate info as well as elevation and GPS tracks proper. Anything you would recommend?

Pete, that sounds like a long and painful debugging session. Judging by your post-mortem, I'm not convinced that the problem is with TC. In fact, I would place my bet on resource exhaustion (file descriptors / sockets).

I have not stress-tested tokyo tyrant, but perhaps there is something with the network handling code. Working with Tokyo Cabinet directly, I have not see any problems like the ones you're encountering.

Having said that, try this when you create your db: hdb.setxmsiz(500_000_000)

As one of the commenters pointed out on my original post, for large imports that can have a significant impact. The other piece of advice is: optimize the db!