Why I’m working on an aggregator

Background

The first RSS aggregator developed at UserLand was called My.UserLand.Com. It was a centralized application, like Bloglines or My.Yahoo. It first shipped in 1999, alongside My.Netscape.Com, which took a different approach. These two web services helped bootstrap the syndication market as it exists today.

Radio 8, which shipped in early 2002, included an enhanced version the UserLand aggregator, and it quickly became the most popular way to access RSS-based information on the Internet. It’s still used by many people today, although UserLand has now focused on Manila, with my support (I’m still a major shareholder in UserLand and a member of its board of directors).

The aggregator I’m working on is a direct descendent of these two earlier aggregators. All employ the River of News presentation style, that’s why I call this aggregator newsRiver.root.

Why pick it up again?

It’s 2005 now, soon to be 2006. In the four years since Radio 8, there have been lots of aggregators, but honestly, not so many new ideas. I have a few projects I want to do, and since the OPML Editor is totally compatible with Radio, I was able to get the aggregator running here, and have begun to add new features.

My main reason for doing this work is to help bootstrap some new features in the market, specifically in two areas: Podcasting and Reading Lists. But first, I need to get a basic release out, get it running on some servers and desktops, and get feedback from users.

Licensing

UserLand will get an unrestricted license to my improvements, and may distribute the app commercially or operate it as a service for its customers; it’s entirely up to the management of the company how it wants to use this software, or if it wants to.

I am making it available to the OPML user community under the GNU Public License, as the editor itself is licensed. No need to explain what that allows. I’m hoping that everyone who’s interested in participating in a community development project around RSS aggregation will give this software a try and share their ideas.

The software runs on Windows and Mac OS X.

Timing

I plan to have a preliminary testing release available for the hale and hearty adventurers of the OPML community, later today (Murphy-willing). The docs will be sketchy. The goal will be to find out if it works, and where the most pressing needs are for fixes and docs.

Then as the software develops, I’ll make announcements on Scripting News and encourage a broader group of people to use it.

I also hope to operate a public test instance for the members of the Web 2.0 Workgroup. More on that later.

55 responses to this post.

  1. […] Looks like Dave Winer is working on a new aggregator. This is cool because his style of “river of news” aggregators is far more appropriate than the “folder by folder” or “3-pane” approach that I’m using currently. Why? The folder-by-folder approach requires you to be pretty anal about reading all your feeds and makes you mentally tired if you fall behind. Sorta like email. It’s to the point sometimes that I dread opening up Outlook. […]

    Reply

  2. river of news = evil

    Reply

  3. and where will one find a copy to beta test … this sounds like a very interesting project.

    Reply

  4. Dave, I’m in! Sounds like something good at the right time (again) :)

    Reply

  5. Alas, I am on LInux, and am stuck with using bloglines in ‘ghetto river of news’ mode by clicking in the top-level “N Feeds” folder to simulate river of news mode.

    Ugh. Bloglines likes to forget what you’ve read frequently, and then you have to scroll down through months of posts for a blog to get to the next one.

    Reply

  6. Dave, I look forward to beta testing your new creation. I am also dis- satisfied with current aggregators have tried several different solutions. I think this friction with the software is one reason so many people use my yahoo for RSS. Before I upgraded to Athlon 64 X2 3800+ and 2 gigs ram and fast raid on new PC I did not want to add to Outlook’s overhead and have experienced crashes in outlook with newsreader add-ins.

    Reply

  7. Hey Dave.

    Maybe we could experiment syncing it with TailRank. You could give me OPML (maybe via SSE) and I could give you a custom recommended reading list from our ranking engine.

    The rankings would be tuned per the person’s OPML feeds. I’m planning on launching a few more features soon that will make it all clear :)

    Anyway.. we can chat about it tomorrow if you’re going to the geek dinner.

    (Listening to your morning coffee notes btw and they’re pretty cool).

    Kevin

    Reply

  8. matt w.: If you don’t like River of News, don’t use it. There are small pockets of people, like me, who think it’s the best thing since sliced bread and hate all these poncy “Web 2.0 aggregators” which scale to about three feeds.

    Dave, I’ll be playing with this the moment it’s released. And documenting it, hopefully. Thanks for all your hard work.

    Reply

  9. I would like to help you beta test.

    Reply

  10. I use my own custom aggregator written in Perl, but I have never been able to figure out if River of News is quite the same. My aggregator sorts everything by date+time, not by source. The only time it clumps by source is when date+time match. Obviously this excludes or requires special handling for any feeds that do not attach pub date info to items.

    The few times I used Radio, I liked that it was River of News, but I disliked how it clumped everything by source rather than by chronology, which feels much more natural to me.

    Reply

  11. PS I only used Radio briefly so may be wrong on how it worked. I really don’t claim to know exactly what River of News is (although I know what it is NOT — Outlook-style foldering)

    Reply

  12. I have a question about river of news. I just don’t quite understand one thing about it. Scoble and Lisa say it somehow helps you see that you don’t have to read everything. How does it do that? I mean, couldn’t I use a 2- or 3-paned aggregator and just say to yourself “Now Amy, you don’t have to read everything.”

    But of course I will test it. I am hale AND hearty.

    Reply

  13. Yes you are hale and hearty, and snarky.

    Reply

  14. Oh shit. My “charm” must be showing again. Can’t seem to help it. ;-)

    Reply

  15. Are you ready to try it out?

    http://support.opml.org/2005/12/29#a423

    Let’s have fun!

    Reply

  16. I also vote [1] for river of news.

    Reply

  17. […] Some details are here. The aggregator will have advanced podcasting/videocasting support and will also include a “reading lists” feature. The idea around reading lists is to allow people to subscribe to other people’s list of recommended feeds (in OPML format), and have those lists update as feeds are added or removed. Seems simple, but no one does it now. […]

    Reply

  18. i’m a fan of the river of news model. i worked on the google desktop sidebar web clips panel, which is a “river of news” style aggregator. at first i was skeptical of the model, but after building it, seeing others use it, and using it myself, i’m a big fan. of course, there are a lot of improvements beyond the simple river, but that basic idea lets you control your blog-reading time, rather than have your blogs, with all their new items, control you.

    i’ll be interested to see what you do. and maybe you’ll watch what we do :)

    Reply

  19. Thanks for this Dave. It’s great.

    For those of you who can’t wait for Dave to add a ‘check all’ button, try installing the Web Developer Toolbar if you are running FireFox. Then, you can choose ‘Populate Form Fields’ and all the boxes will be checked.

    https://addons.mozilla.org/extensions/moreinfo.php?id=60

    Reply

  20. […] Read about why Dave has decided to do a new RSS Aggregator at his blog. […]

    Reply

  21. Yep, very cool.

    Reply

  22. […] I was talking with Fred the other night and he told me about “the River of news”. I found it very interesting and now yesterday Dave Winner posted an article on his site about the RSS aggregator he is developing. This aggregator will be a River of News and a descendent of earlier aggregators like My.UserLand.Com and Manilla. […]

    Reply

  23. […] I stumbled upon it over on Dave Winer’s announcement about his new aggregator, thanks to (who else?) Scoble. […]

    Reply

  24. Posted by xman on December 30, 2005 at 6:11 am

    River of News is a *HORRIBLE* idea. I have tried it and I hate it – it slows down my reading and does not let me get through nearly as many feeds nearly as quickly.

    Reply

  25. Would love to try the new aggregator!

    Reply

  26. One request — make sure that you preserve the feature that pressing the spacebar takes you down one screen. Most browsers support this, and many aggregators, but many also forget.

    Reply

  27. Posted by Jernej on December 30, 2005 at 11:20 am

    To put it mildly… I don’t get it?! Why would you want the “river of news” approach? Completely counterproductive. Well, at least if the goal is NOT reading anything.

    Whenever a feed gives me more than 10 new items I skip all of them without ever reading the headlines. Very user unfriendly. Readability 0.

    Reply

  28. Christopher Allen: tab-space-tab-space-tab-space. Apart from long blog posts, it’s a pretty good method of navigation.

    Jernej: if you don’t like River of News, don’t use it. I hate Outlook-style approaches, so I don’t use it.

    Reply

  29. […] Now it’s time to get these new kind of aggregators supporting these new kinds of blogging ‘items’. […]

    Reply

  30. […]One of the reasons I believe the RSS application race is still very much on is because of the difficulty in presenting the information in a logical, intuitive and easy to navigate manner. I constantly find myself scrolling up, down and sideways, opening stories in new tabs and having to retrace my steps back to the topic list. If Dave’s new aggregator can make this process easier and move intuitive, it could be the killer RSS app we’re looking for.[…]

    Reply

  31. River of News is great if you are a very casual news reader. It absolutely sucks otherwise.

    Can you imagine if your newspaper did this — rather than having sections or different sized headlines, you’d have everything in no particular order of either topic or importance.

    Again, it might work great if you were a very casual news reader or only read a few feeds, but it just doesn’t scale. It maybe be great to sit on that river bank and not worry about missing that important story if you’ve got nothing better to do all day, but otherwise it’s not very helpful.

    Its absurd for Dave to claim there have been no new ideas in the four years since Radio 8 was released. Radio 8 was innovative and I paid for a copy, but like most Userland projects it stagnated and was overtaken by better products that offered more features (for example, the one I use can do smart search channels — so it tracksall the 600+ feeds I subscribe to and creates a channel of all the stories from all those feeds where the phrase “paris hilton” appears, for example.)

    There really is no One Best Way to view and manage RSS feeds, and a good RSS reader should empower the user to customize it the way he or she works rather than try to force the One Best Way on the user.

    Reply

  32. Posted by Marcelo Lopez on December 31, 2005 at 12:09 am

    Dave,

    Looks nice. Ran into one issue here though. After installing newsRiver.root, I added subscriptions to Techbargains and Slashdot ( pretty disparate sort of feeds ). While I can browse the news itself, I can’t browse the subscriptions. I get a 404 error ala…

    404 Not Found
    The object /newsRiver/subscription doesn’t exist.

    When I try to browse:

    http://127.0.0.1:5337/newsRiver/subscription

    which I’m assuming it the url for the output the newsRiver plugin generates for the subscription list. Well, that’s not working right now, and I did perform a “get latest code” on it. For the moment, this is running within the Windows version of OpmlEd.

    Anyone else run into this ??

    Reply

  33. Posted by RSS Should be Delivered Via Email on December 31, 2005 at 12:12 am

    RSS feeds are much like Email – why not have them delivered to an eixisting email address. http://www.Squeet.com does just that.

    Reply

  34. Although I’m not sure if I like the River of News-style (I use NetNewsWire as my main reader), I do like the feature that let’s you view all the posts from one source, by pressing the little magnifying glass.

    To me it’s not clear what the big difference with the RadioUserland aggregator is. The Reader List option seems very interesting though..

    Reply

  35. Posted by Billy on December 31, 2005 at 8:55 am

    I just loaded my subs into the river, the betty agent starts “sleeping for 5 seconds” but it really lasts long enuff that I finally close the window so I’m not sure what’s going on until I View News or Subscriptions: they both pop up the OPML Editor background.html file in my editor, how do I fix this?

    Also, I get the forbidden error msg when trying to access http://hosting.opml.org/BillyG/

    … just trying to keep up! TIA.

    Reply

  36. Posted by Jernej on January 1, 2006 at 3:26 am

    Actually there is one exception where the river approach would be superior – mobile phones and other small screens. It would be much easier and faster to just scroll down rather than open up folders and scroll.

    But other than that it just doesn’t work.

    Reply

  37. This is funny watching how polarizing the ‘River of News’ thing is. It’s quite a ‘Marmite’ issue…

    My take:

    Without ‘River of News’ I might as well read blogs in a browser. The thing RoN freed me from was remembering which sites I hadn’t checked for a while Have i checked that one today? This week? I would find I checked some blogs too frequently and would forget to check others at all.

    With RoN I just have one list of headings to scan. I know where I am up to so when I have 5 mninutes spare I can read another couple of posts. If I get too far behind I turn up my metal filters and skim through faster than usual.

    Why would you want anything more complicated than that? Having to choose a particular category or a particular site to focus on just gets in the way of my reading. I don’t want another decision!

    One list. Chronological order.
    Skim it, read/skip and then move on…

    Reply

  38. […] From TechCrunch: Dave Winer announced today that he’s built a new RSS aggregator, built on his OPML editor platform. He’s released it to the OPML community for early testing.   […]

    Reply

  39. JerneJ: mine seems to work absolutely fine. In fact, it keeps me on top of the news far better than when I was using Bloglines. How about learning “each to his own” rather than making statements that profess to factuality when in fact they are your own preferences.

    Reply

  40. Someone a few posts back mentioned the same problem I’m having when I try to view subscriptions:

    404 Not Found
    The object /newsRiver/subscription doesn’t exist.

    I’ve done a “get latest code”.

    Reply

  41. […] I mentioned in my last story watching Dave Winer’s keynote at Gnomedex, where he demos his OPML Editor and how he uses its blog editing features to update his blog in real time. Well, coincidentially, Dave’s project has been buzzing quite a bit recently. He announced that he’ll be adding a ‘river of news’ style aggregator with reportedly pod/vidcasting capabilities. Well, he’s released the beta plugin for it now (instead of making new builds for every feature added, Dave is using a smart technique where he has one flexible build which he builds plugins on and releases them, also auto-updatable). […]

    Reply

  42. Posted by Ed Edwards on January 2, 2006 at 7:29 pm

    Hi Dave,
    Kinda stumbled upon your wordpress Blog…I thought you should know that the RSS feed from ScriptingNews is lacking Titles on each item…RSS Rules and Rocks!

    Reply

  43. Using RON is easy. Familiar Radio 8 style. It just works. Great!

    Gary

    Reply

  44. This sounds interesting and just what I am looking for.

    Reply

  45. “404 Not Found
    The object /newsRiver/subscription doesn’t exist.”

    I’m getting that too.

    Reply

  46. Ah, it’s supposed to be /newsRiver/subscriptions. That’s the way it looks in the newsRiver.root table. Typing in the full URL by hand gets me to the subscriptions page just fine. For some reason, when opening the URL that last “s” character gets eaten.

    Reply

  47. Thanks, Noah.

    Reply

  48. […] Maybe we’ll get this from the 2006 microbubble, which should be about attention. Or maybe that’s what Dave is working on (newsRiver.root)? […]

    Reply

  49. […] Winer has estado también trabajando en un agregador de RSS incorporado al editor. La diferencia entre éste agregador y otros es que permite visualizar las fuentes presentadas en un estilo llamado “Río de Noticias“. […]

    Reply

  50. Posted by Brett Bourne on January 9, 2006 at 8:23 am

    I have an rss.php feed that will not subscribe in iRiver

    is this a known issue with the .php? or should I drive this inquiry from the site in question?

    thanks

    Reply

  51. […] Dave’s WordPress Blog » Why I’m working on an aggregator […]

    Reply

  52. […] Does anyone care to discuss which feed readers they are using? Is anyone using the OPML Editor’s NewsRiver as their primary feed reader? Any recommendations on features that help plow through large numbers of articles? […]

    Reply

  53. […] Some details are here. The aggregator will have advanced podcasting/videocasting support and will also include a “reading lists” feature. The idea around reading lists is to allow people to subscribe to other people’s list of recommended feeds (in OPML format), and have those lists update as feeds are added or removed. Seems simple, but no one does it now. […]

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 60 other followers

%d bloggers like this: