Why I’m working on an aggregator


The first RSS aggregator developed at UserLand was called My.UserLand.Com. It was a centralized application, like Bloglines or My.Yahoo. It first shipped in 1999, alongside My.Netscape.Com, which took a different approach. These two web services helped bootstrap the syndication market as it exists today.

Radio 8, which shipped in early 2002, included an enhanced version the UserLand aggregator, and it quickly became the most popular way to access RSS-based information on the Internet. It’s still used by many people today, although UserLand has now focused on Manila, with my support (I’m still a major shareholder in UserLand and a member of its board of directors).

The aggregator I’m working on is a direct descendent of these two earlier aggregators. All employ the River of News presentation style, that’s why I call this aggregator newsRiver.root.

Why pick it up again?

It’s 2005 now, soon to be 2006. In the four years since Radio 8, there have been lots of aggregators, but honestly, not so many new ideas. I have a few projects I want to do, and since the OPML Editor is totally compatible with Radio, I was able to get the aggregator running here, and have begun to add new features.

My main reason for doing this work is to help bootstrap some new features in the market, specifically in two areas: Podcasting and Reading Lists. But first, I need to get a basic release out, get it running on some servers and desktops, and get feedback from users.


UserLand will get an unrestricted license to my improvements, and may distribute the app commercially or operate it as a service for its customers; it’s entirely up to the management of the company how it wants to use this software, or if it wants to.

I am making it available to the OPML user community under the GNU Public License, as the editor itself is licensed. No need to explain what that allows. I’m hoping that everyone who’s interested in participating in a community development project around RSS aggregation will give this software a try and share their ideas.

The software runs on Windows and Mac OS X.


I plan to have a preliminary testing release available for the hale and hearty adventurers of the OPML community, later today (Murphy-willing). The docs will be sketchy. The goal will be to find out if it works, and where the most pressing needs are for fixes and docs.

Then as the software develops, I’ll make announcements on Scripting News and encourage a broader group of people to use it.

I also hope to operate a public test instance for the members of the Web 2.0 Workgroup. More on that later.

55 responses to this post.

  1. […] Looks like Dave Winer is working on a new aggregator. This is cool because his style of “river of news” aggregators is far more appropriate than the “folder by folder” or “3-pane” approach that I’m using currently. Why? The folder-by-folder approach requires you to be pretty anal about reading all your feeds and makes you mentally tired if you fall behind. Sorta like email. It’s to the point sometimes that I dread opening up Outlook. […]


  2. river of news = evil


  3. and where will one find a copy to beta test … this sounds like a very interesting project.


  4. Dave, I’m in! Sounds like something good at the right time (again) :)


  5. Alas, I am on LInux, and am stuck with using bloglines in ‘ghetto river of news’ mode by clicking in the top-level “N Feeds” folder to simulate river of news mode.

    Ugh. Bloglines likes to forget what you’ve read frequently, and then you have to scroll down through months of posts for a blog to get to the next one.


  6. Dave, I look forward to beta testing your new creation. I am also dis- satisfied with current aggregators have tried several different solutions. I think this friction with the software is one reason so many people use my yahoo for RSS. Before I upgraded to Athlon 64 X2 3800+ and 2 gigs ram and fast raid on new PC I did not want to add to Outlook’s overhead and have experienced crashes in outlook with newsreader add-ins.


  7. Hey Dave.

    Maybe we could experiment syncing it with TailRank. You could give me OPML (maybe via SSE) and I could give you a custom recommended reading list from our ranking engine.

    The rankings would be tuned per the person’s OPML feeds. I’m planning on launching a few more features soon that will make it all clear :)

    Anyway.. we can chat about it tomorrow if you’re going to the geek dinner.

    (Listening to your morning coffee notes btw and they’re pretty cool).



  8. matt w.: If you don’t like River of News, don’t use it. There are small pockets of people, like me, who think it’s the best thing since sliced bread and hate all these poncy “Web 2.0 aggregators” which scale to about three feeds.

    Dave, I’ll be playing with this the moment it’s released. And documenting it, hopefully. Thanks for all your hard work.


  9. I would like to help you beta test.


  10. I use my own custom aggregator written in Perl, but I have never been able to figure out if River of News is quite the same. My aggregator sorts everything by date+time, not by source. The only time it clumps by source is when date+time match. Obviously this excludes or requires special handling for any feeds that do not attach pub date info to items.

    The few times I used Radio, I liked that it was River of News, but I disliked how it clumped everything by source rather than by chronology, which feels much more natural to me.


  11. PS I only used Radio briefly so may be wrong on how it worked. I really don’t claim to know exactly what River of News is (although I know what it is NOT — Outlook-style foldering)


  12. I have a question about river of news. I just don’t quite understand one thing about it. Scoble and Lisa say it somehow helps you see that you don’t have to read everything. How does it do that? I mean, couldn’t I use a 2- or 3-paned aggregator and just say to yourself “Now Amy, you don’t have to read everything.”

    But of course I will test it. I am hale AND hearty.


  13. Yes you are hale and hearty, and snarky.


  14. Oh shit. My “charm” must be showing again. Can’t seem to help it. ;-)


  15. Are you ready to try it out?


    Let’s have fun!


  16. I also vote [1] for river of news.


  17. […] Some details are here. The aggregator will have advanced podcasting/videocasting support and will also include a “reading lists” feature. The idea around reading lists is to allow people to subscribe to other people’s list of recommended feeds (in OPML format), and have those lists update as feeds are added or removed. Seems simple, but no one does it now. […]


  18. i’m a fan of the river of news model. i worked on the google desktop sidebar web clips panel, which is a “river of news” style aggregator. at first i was skeptical of the model, but after building it, seeing others use it, and using it myself, i’m a big fan. of course, there are a lot of improvements beyond the simple river, but that basic idea lets you control your blog-reading time, rather than have your blogs, with all their new items, control you.

    i’ll be interested to see what you do. and maybe you’ll watch what we do :)


  19. Thanks for this Dave. It’s great.

    For those of you who can’t wait for Dave to add a ‘check all’ button, try installing the Web Developer Toolbar if you are running FireFox. Then, you can choose ‘Populate Form Fields’ and all the boxes will be checked.



  20. […] Read about why Dave has decided to do a new RSS Aggregator at his blog. […]


  21. Yep, very cool.


  22. […] I was talking with Fred the other night and he told me about “the River of news”. I found it very interesting and now yesterday Dave Winner posted an article on his site about the RSS aggregator he is developing. This aggregator will be a River of News and a descendent of earlier aggregators like My.UserLand.Com and Manilla. […]


  23. […] I stumbled upon it over on Dave Winer’s announcement about his new aggregator, thanks to (who else?) Scoble. […]


  24. Posted by xman on December 30, 2005 at 6:11 am

    River of News is a *HORRIBLE* idea. I have tried it and I hate it – it slows down my reading and does not let me get through nearly as many feeds nearly as quickly.


  25. Would love to try the new aggregator!


  26. One request — make sure that you preserve the feature that pressing the spacebar takes you down one screen. Most browsers support this, and many aggregators, but many also forget.


  27. Posted by Jernej on December 30, 2005 at 11:20 am

    To put it mildly… I don’t get it?! Why would you want the “river of news” approach? Completely counterproductive. Well, at least if the goal is NOT reading anything.

    Whenever a feed gives me more than 10 new items I skip all of them without ever reading the headlines. Very user unfriendly. Readability 0.


  28. Christopher Allen: tab-space-tab-space-tab-space. Apart from long blog posts, it’s a pretty good method of navigation.

    Jernej: if you don’t like River of News, don’t use it. I hate Outlook-style approaches, so I don’t use it.


  29. […] Now it’s time to get these new kind of aggregators supporting these new kinds of blogging ‘items’. […]


  30. […]One of the reasons I believe the RSS application race is still very much on is because of the difficulty in presenting the information in a logical, intuitive and easy to navigate manner. I constantly find myself scrolling up, down and sideways, opening stories in new tabs and having to retrace my steps back to the topic list. If Dave’s new aggregator can make this process easier and move intuitive, it could be the killer RSS app we’re looking for.[…]


  31. River of News is great if you are a very casual news reader. It absolutely sucks otherwise.

    Can you imagine if your newspaper did this — rather than having sections or different sized headlines, you’d have everything in no particular order of either topic or importance.

    Again, it might work great if you were a very casual news reader or only read a few feeds, but it just doesn’t scale. It maybe be great to sit on that river bank and not worry about missing that important story if you’ve got nothing better to do all day, but otherwise it’s not very helpful.

    Its absurd for Dave to claim there have been no new ideas in the four years since Radio 8 was released. Radio 8 was innovative and I paid for a copy, but like most Userland projects it stagnated and was overtaken by better products that offered more features (for example, the one I use can do smart search channels — so it tracksall the 600+ feeds I subscribe to and creates a channel of all the stories from all those feeds where the phrase “paris hilton” appears, for example.)

    There really is no One Best Way to view and manage RSS feeds, and a good RSS reader should empower the user to customize it the way he or she works rather than try to force the One Best Way on the user.


  32. Posted by Marcelo Lopez on December 31, 2005 at 12:09 am


    Looks nice. Ran into one issue here though. After installing newsRiver.root, I added subscriptions to Techbargains and Slashdot ( pretty disparate sort of feeds ). While I can browse the news itself, I can’t browse the subscriptions. I get a 404 error ala…

    404 Not Found
    The object /newsRiver/subscription doesn’t exist.

    When I try to browse:

    which I’m assuming it the url for the output the newsRiver plugin generates for the subscription list. Well, that’s not working right now, and I did perform a “get latest code” on it. For the moment, this is running within the Windows version of OpmlEd.

    Anyone else run into this ??


  33. Posted by RSS Should be Delivered Via Email on December 31, 2005 at 12:12 am

    RSS feeds are much like Email – why not have them delivered to an eixisting email address. http://www.Squeet.com does just that.


  34. Although I’m not sure if I like the River of News-style (I use NetNewsWire as my main reader), I do like the feature that let’s you view all the posts from one source, by pressing the little magnifying glass.

    To me it’s not clear what the big difference with the RadioUserland aggregator is. The Reader List option seems very interesting though..


  35. Posted by Billy on December 31, 2005 at 8:55 am

    I just loaded my subs into the river, the betty agent starts “sleeping for 5 seconds” but it really lasts long enuff that I finally close the window so I’m not sure what’s going on until I View News or Subscriptions: they both pop up the OPML Editor background.html file in my editor, how do I fix this?

    Also, I get the forbidden error msg when trying to access http://hosting.opml.org/BillyG/

    … just trying to keep up! TIA.


  36. Posted by Jernej on January 1, 2006 at 3:26 am

    Actually there is one exception where the river approach would be superior – mobile phones and other small screens. It would be much easier and faster to just scroll down rather than open up folders and scroll.

    But other than that it just doesn’t work.


  37. This is funny watching how polarizing the ‘River of News’ thing is. It’s quite a ‘Marmite’ issue…

    My take:

    Without ‘River of News’ I might as well read blogs in a browser. The thing RoN freed me from was remembering which sites I hadn’t checked for a while Have i checked that one today? This week? I would find I checked some blogs too frequently and would forget to check others at all.

    With RoN I just have one list of headings to scan. I know where I am up to so when I have 5 mninutes spare I can read another couple of posts. If I get too far behind I turn up my metal filters and skim through faster than usual.

    Why would you want anything more complicated than that? Having to choose a particular category or a particular site to focus on just gets in the way of my reading. I don’t want another decision!

    One list. Chronological order.
    Skim it, read/skip and then move on…


  38. […] From TechCrunch: Dave Winer announced today that he’s built a new RSS aggregator, built on his OPML editor platform. He’s released it to the OPML community for early testing.   […]


  39. JerneJ: mine seems to work absolutely fine. In fact, it keeps me on top of the news far better than when I was using Bloglines. How about learning “each to his own” rather than making statements that profess to factuality when in fact they are your own preferences.


  40. Someone a few posts back mentioned the same problem I’m having when I try to view subscriptions:

    404 Not Found
    The object /newsRiver/subscription doesn’t exist.

    I’ve done a “get latest code”.


  41. […] I mentioned in my last story watching Dave Winer’s keynote at Gnomedex, where he demos his OPML Editor and how he uses its blog editing features to update his blog in real time. Well, coincidentially, Dave’s project has been buzzing quite a bit recently. He announced that he’ll be adding a ‘river of news’ style aggregator with reportedly pod/vidcasting capabilities. Well, he’s released the beta plugin for it now (instead of making new builds for every feature added, Dave is using a smart technique where he has one flexible build which he builds plugins on and releases them, also auto-updatable). […]


  42. Posted by Ed Edwards on January 2, 2006 at 7:29 pm

    Hi Dave,
    Kinda stumbled upon your wordpress Blog…I thought you should know that the RSS feed from ScriptingNews is lacking Titles on each item…RSS Rules and Rocks!


  43. Using RON is easy. Familiar Radio 8 style. It just works. Great!



  44. This sounds interesting and just what I am looking for.


  45. “404 Not Found
    The object /newsRiver/subscription doesn’t exist.”

    I’m getting that too.


  46. Ah, it’s supposed to be /newsRiver/subscriptions. That’s the way it looks in the newsRiver.root table. Typing in the full URL by hand gets me to the subscriptions page just fine. For some reason, when opening the URL that last “s” character gets eaten.


  47. Thanks, Noah.


  48. […] Maybe we’ll get this from the 2006 microbubble, which should be about attention. Or maybe that’s what Dave is working on (newsRiver.root)? […]


  49. […] Winer has estado también trabajando en un agregador de RSS incorporado al editor. La diferencia entre éste agregador y otros es que permite visualizar las fuentes presentadas en un estilo llamado “Río de Noticias“. […]


  50. Posted by Brett Bourne on January 9, 2006 at 8:23 am

    I have an rss.php feed that will not subscribe in iRiver

    is this a known issue with the .php? or should I drive this inquiry from the site in question?



  51. […] Dave’s WordPress Blog » Why I’m working on an aggregator […]


  52. […] Does anyone care to discuss which feed readers they are using? Is anyone using the OPML Editor’s NewsRiver as their primary feed reader? Any recommendations on features that help plow through large numbers of articles? […]


  53. […] Some details are here. The aggregator will have advanced podcasting/videocasting support and will also include a “reading lists” feature. The idea around reading lists is to allow people to subscribe to other people’s list of recommended feeds (in OPML format), and have those lists update as feeds are added or removed. Seems simple, but no one does it now. […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Get every new post delivered to your Inbox.

Join 61 other followers

%d bloggers like this: