Building a keyword feed aggregator - and keeping on track.

Short note: How to keep up with your niche keywords by using aggregators.

The reason you want aggregators is so you can keep on top of the conversations going about your niche. That niche is defined by the keywords you are tracking.

There are several ways to get aggregators going. An aggregator isn't just one program, like a feed reader (although it could be). More than likely, you have several widgets going out and collecting feeds for you and filtering them to just get what you really want.

The all-time champ at this is Steve Rubel, who is so far out there as far as repurposing programs to really perform - well, it's a bit intimidating, if not scary. Last I checked, he's using GMail to handle all his data needs - and he tracks over 600 feeds every day...

Simpler than that, three (or four) decent tools to get the data you want:

1. Yahoo Pipes - a graphical tool which can search through websites to get you data based on available RSS feeds. And you can add filters to only grab certain data based on keywords from those available items.
2. Google Alerts - you can set up Google to search for data and then send it to a reader as a feed. Their comprehensive search can search for data that isn't even in RSS feeds.
3. FriendFeed imaginary friends - you can set up imaginary friends in Friendfeed to simply subscribe to various RSS feeds, and then bring them up as a live stream in Friendfeed. (FriendFeed also has the capacity to aggregate feed items across multiple social platforms from friends of friends, so this is a nice way to acquire even more interesting data.)
   
4. Twitter Search - these searches have RSS feeds, so can be brought into a news reader

Best news reader - probably Google Reader, since you can also share your finds there - and then export these as your shared folder for others to find for you. Louis Gray uses the concept of subscribing to other like-minded people's shared items feeds in order to pre-digest his data and use human filters to do so.

But just using mechanical systems, you can aggregate useful data streams if you simply pipe, filter, and then re-pipe the data.

First, you already now what keywords you want to target. Set up Google Alerts for these and create feeds from them - or have them emailed to an auto-posting blog (like Blogger) so they now have an RSS feed. Also collect up feeds from a Twitter Search for each of these keywords.

Next, set up one or several FriendFeed imaginary friends to receive this data. Put these into a special folder there. Each of these folders have their own RSS feed.

Third, set up a Yahoo pipe to gather that feed (those feeds) and filter them to what you more closely need. That will then give you an aggregated, filtered feed.

Finally, send that feed to Google Reader (or iGoogle) so you can simply digest it. Assign folders for type of data - however you want to organise it. (Note: Google Reader is the end of the line, since it doesn't itself provide another RSS feed. You can manually export OPML, but that's not a live set - probably only useful to set up a parallel feed digester/aggregator - or if you want a record before you strip it out and start over.)

The idea is to set up several flows of data, which are then more narrowly filtered to give you just the data you need for those keyword areas you are interested in.

With these data snippets coming in, you can then check out the links and blog or converse on this topic as you want - as well as being able to then post knowledgeably about that subject as it comes up on the web.

Of course, this is incredibly unlimited. Where you set up several discreet flows of information like this, you can track and handle several different niches at the same time - you'll then only be limited by your ability to output sensible data, summations, conclusions. But your posts will also all give your sources, so you have then an intense ability to contribute to the community.

And this gives new meaning to any blogs you have which are simply short utility posts.

Another route is to do the auto-blog routine above, have that blog in turn email your gmail account, which you filter and archive instead of clogging your inbox. Then, you can run searches on data you've accumulated. But that is a different flow than staying on top of new items. Great for making your "63 ways to..." lists, though.

OK, hope you got some interesting insight out of this - at least it's something you can try.

The reason I wrote this was that as soon as I set up an aggregator, it got swamped with a lot of data that I had to personally filter. Today, I saw Yahoo Pipes for the first time (heard about it later, but sometimes you have to get hit over the head with it) and saw a way I could get these various feeds thinned down to just what I actually wanted.

I'll be setting these up for myself sometime over the next couple of weeks and tell you what I find. Except for Yahoo pipes, I've worked with all the rest. And got swamped with having too much unfiltered data. Now that I've seen what that is capable of, I should be able to rework what I have and then get a cleaner data flow so I can blog more prolifically. (Like that was a problem to begin with?)

Luck to us all.
- - - -

Update - before posting this, I checked out my Google Reader. Sure enough it was filled with inconsistent data. Stuff I thought I'd like to subscribe to at the time, but actually have little use for. Doesn't improve my ability to live on this planet. Doesn't make it easier for me to earn money. Has little to do with my core interests.

So the filtered approach above should then bring me only feeds which I consider vital to begin with - based on the keywords I've been looking for. Oh, that I could find what I've been recently searching ... but Google keeps this locked up somewhere.

Filter your data - and streamline your goals. Spend less time getting the answers and new data you actually need.

We'll see how this goes. Perhaps serendipity might have to be included (a RAND function or something?!?)