A few months ago I came across a new internet service named Gnip. Although it looks very technical and abstract at first look, it may be the key to a bad dilemma of personal publishing on the web today. Here is why, and my experiences of using it for this site.

The idea of individuals publishing content on the web is as old as the web itself. With the rise of blogging software like Wordpress et al. personal publishing turned mainstream. Individuals put all their texts, images and other media in one place: their website. With the start of the social web, Web 2.0, web apps (or whatever you like to call it) this changed rapidly. On free services like Flickr, Del.icio.us, Twitter etc. everybody started publishing to a worldwide audience, even if they did not own an old-school website.

Of course this is mostly a good thing, but caused a dilemma to the regular website operators (like myself): Should I get on with posting my digital photos to my site, or should I put it on Flickr? Should I still post interesting links on my blog, or should I link it on Del.icio.us? Should I clutter my blog with random thoughts, or should I twitter?

The answer is clear: Flickr is much better suited to publish digital photos than a regular site will ever be. It offers plenty of webspace, speaks to a massive community and has an never-ending array of tools to interlink, group, tag and share my photos.

The same is true for Del.icio.us: The possibilities to inter-connect my links with a network of trillians of users can never be matched on a typical website. Same with Twitter: Microblogging on a website makes no sense when there is a free, popular and perfectly fitted service like Twitter.

Now I the publishing individual and (ex-)blogger have accounts on a handful of social sites. The downsides slowly become obvious: My content, my ideas, my personality are now spread over several websites that I do not fully control. I give my content away to service companies. Well, most web apps offer their services for free, so how could we complain? But still, I loose some control of my content, how it is presented and maybe one day it might even be deleted.

What can we do about it? Double-post everything manually on the most fitting web app and my blog? This is not a solution, especially if I want to keep a fashionable publishing frequency. So the only way left is to pull the content from the web app sites into my website automatically. Most web apps these days offer side-bar widgets, feeds and other ways to present the content elsewhere. There is even lifestream blogging software specialised on this purpose like Sweetcron which aggregates my content from different sources into one flow.

There are basically two ways to pull content into my website: Either on page load, for example by integrating RSS feed data into the web page. This means I have to rely on the perfomance of the social web service when someone tries to load a page of my blog. In unfortunate cases this can slow down the page load significantly or even break it entirely. The other way to do it is by running a cron job on the web server. This means that for example every hour all the content providers are checked for new stuff automatically. Cron jobs are often not allowed in smaller web hosting plans. And they often generate unneccessary processing time and web traffic. Not really optimal.

The shiny web-2.0-ish Gnip home page

The perfect flow would be if my content would be passed on to and posted on my website the moment it is generated. So far this was not possible. Now there is a new service that tries to offer exactly that. It is called Gnip (ping in reverse) and resides at gnip.com. After a standard account registration procedure you can create your filters on the site or through an API for a wide range of publishers. Fortunately we will meet lots of old friends from the web two-point-oh there: Twitter, Flickr, YouTube, del.icio.us, digg, etc. Nearly all the big ones are cooperating with Gnip. You can set up filter for exactly the content you want by user, relation or tags. And then you only need to set a URL pointing to a specific script on your own web server and, whenever new content comes available, this script will be fired up and given some info about what happened. Exactly what we wanted, and for FREE!

The much more “functional” Gnip account page

... If only it worked more reliably. For the last months I tried to get it to work with the web app that would be the most useful on my site: Twitter. It was not such a bad experience once I managed some technical details. I ran a log file to see how good the coverage was. Unfortunately the tweets did not get through all of the time. Some came hours and even days later, some did not show up at all. Also I had to set the filters up again because of changes to the service.

The Gnip filter page. The highlighted field holds the URL to the script that is fired up when new content is generated

I guess Twitter is the most sought-after publisher at Gnip and I bet the serverload is tremendous. Also Gnip is completely free. So even with the flakey service my hat goes off to them. Setting up Gnip must have been a gigantic task. Maybe even more: persuading the big web app companies to join in. Really: Congrats!

But that does not change that this is not the end of “the big personal publishing dilemma of our days”. I will keep an eye on my streams and I think I will try to back up Gnip’s Twitter stream to my site with an old-fashioned cron job for the time being.

If you are interested in the technical details, here is the PHP script I wrote to post to an empty HTML log file and my site’s database.