Pete Warden wasn’t looking to get embroiled in a legal battle when he launched his page data tool which tracked associations between various Facebook Pages. That’s why the page data product he built was subsequently destroyed. It’s not that the data wasn’t useful but Pete was using a crawler to collect data which included various profile information from hundreds of millions of Facebook users.
Want to know which Facebook Pages have similar fans? How about the distribution of Facebook users around the country? The information was incredibly useful, yet Facebook’s legal team had the project shut down. Not that Pete didn’t get enough coverage about the project … he got a ton. ReadWriteWeb did an entire feature article on Pete and his analytics product (which is still accessible).
I was also extremely intrigued by the product because I’ve built a number of crawlers and understood the technical challenges that Pete was facing. That’s why we’ve invited Pete to present his findings at the upcoming Social Developer Summit. While we thing collecting user data and caching things like user profile images is not a good idea, building a system which tracks hundreds of millions of users and their affiliations is impressive.
Pete also wanted to open up the database for free to data researchers looking to perform deeper analysis on Facebook user behavior. Unfortunately that data will never see the light of day because Pete has destroyed much of it. I’ve also heard that the data analysis team over at Facebook was equally impressed by Pete’s data tools, however the legal team didn’t care how cleverly constructed the project was.
If you want to here from Pete as well as other developers who are overcoming massive data analysis hurdles among other things, you should definitely attend our Social Developer Summit being held this June in San Francisco.