Why I’m working on an aggregator

Background

The first RSS aggregator developed at UserLand was called My.UserLand.Com. It was a centralized application, like Bloglines or My.Yahoo. It first shipped in 1999, alongside My.Netscape.Com, which took a different approach. These two web services helped bootstrap the syndication market as it exists today.

Radio 8, which shipped in early 2002, included an enhanced version the UserLand aggregator, and it quickly became the most popular way to access RSS-based information on the Internet. It’s still used by many people today, although UserLand has now focused on Manila, with my support (I’m still a major shareholder in UserLand and a member of its board of directors).

The aggregator I’m working on is a direct descendent of these two earlier aggregators. All employ the River of News presentation style, that’s why I call this aggregator newsRiver.root.

Why pick it up again?

It’s 2005 now, soon to be 2006. In the four years since Radio 8, there have been lots of aggregators, but honestly, not so many new ideas. I have a few projects I want to do, and since the OPML Editor is totally compatible with Radio, I was able to get the aggregator running here, and have begun to add new features.

My main reason for doing this work is to help bootstrap some new features in the market, specifically in two areas: Podcasting and Reading Lists. But first, I need to get a basic release out, get it running on some servers and desktops, and get feedback from users.

Licensing

UserLand will get an unrestricted license to my improvements, and may distribute the app commercially or operate it as a service for its customers; it’s entirely up to the management of the company how it wants to use this software, or if it wants to.

I am making it available to the OPML user community under the GNU Public License, as the editor itself is licensed. No need to explain what that allows. I’m hoping that everyone who’s interested in participating in a community development project around RSS aggregation will give this software a try and share their ideas.

The software runs on Windows and Mac OS X.

Timing

I plan to have a preliminary testing release available for the hale and hearty adventurers of the OPML community, later today (Murphy-willing). The docs will be sketchy. The goal will be to find out if it works, and where the most pressing needs are for fixes and docs.

Then as the software develops, I’ll make announcements on Scripting News and encourage a broader group of people to use it.

I also hope to operate a public test instance for the members of the Web 2.0 Workgroup. More on that later.

Candid account of where New Orleans is at

Via email from a correspondent who wishes to remain anonymous.

“Everything from insurance to utilities costs more in New Orleans. For the moment, even regular food jobs pay more because there is a dearth of employees. However the word is glum on the corporate side. Corporate USA seems unsure about when and if New Orleans will ever be a safe place to do business again. NOLA levees did not hold up to a Cat 3 hurricane, and now the levees are damaged. Several companies are either moving out completely, or returning only with scaled down operations. This week two more local firms announced plans to leave NOLA, plus Nokia cancelled sponsorship of the Sugar Bowl.

“Another reason not to return to NOLA is that there is no reason to endure the enormous stress of rebuilding everything and looking at the devastation every day. The next hurricane season is only six months away…. and the clock is ticking down. Vero Beach and several other areas of Florida were hit three times in past two years. With NOLA levee safety in such serious jeopardy, it only takes a small hurricane to mess things up there again.

“Even if the levees and future hurricanes were not a problem, it’ll take decades to rebuild New Orleans. That is enormous task considering the state, city, schools and hundreds of businesses are now bankrupt with no customers or major income. The weekly media attention you see about NOLA and their problems can be nauseating when you consider NOLA is just one area that was affected by storms that bulldozed dozens of communities across TX, LA, MS, AL, and FL. There are numerous small towns and cities that experienced far more destruction and devastation than NOLA, but they do not receive the ink that NOLA does. Probably because they don’t complain like NOLA does.

“Being averse to further risk and chance to lose everything, we bought home in Xxx two weeks ago. We continue to maintain home in Xxx suburb of New Orleans while son returns to attend high school. Will be selling lot in the Xxx where we formerly planned to build Greek revival cotttage. The good news is that unlike most New Orleanians, we are not upside down with no jobs, no home, no possessions and high mortgage bills, however we are stretched to the max with two mortgages, a lot note, and two sets of utility bills. Expenses such as telephone service, health and fitness membership, dining out (even McD) are now unaffordable.”

I’ll be in Cambridge, January 2-7

I’ll be in Cambridge, January 2-7; staying near MIT, visiting friends, doing business. Maybe we should do a geek dinner or a classroom-style discussion?

Weather for Cambridge. Google map. Yahoo map. Subway map.

CES 2006

Doc Searls is looking for a hotel room in Las Vegas for CES, January 5-8.

There are rooms available on Expedia, but the rates are unbelievable.

I was thinking it might be fun, but then I remembered what it’s like. Long lines for cabs. Miles of aisles. Aching feet.

I wonder if there will be any podcasting companies there?

Are you going? What do you expect to see there?

The Night Before Christmas

Twas the night before Christmas, when all through the house
Not a creature was stirring, not even a mouse.
The stockings were hung by the chimney with care,
In hopes that St Nicholas soon would be there.

The children were nestled all snug in their beds,
While visions of sugar-plums danced in their heads.
And mamma in her ‘kerchief, and I in my cap,
Had just settled our brains for a long winter’s nap.

When out on the lawn there arose such a clatter,
I sprang from the bed to see what was the matter.
Away to the window I flew like a flash,
Tore open the shutters and threw up the sash.

The moon on the breast of the new-fallen snow
Gave the lustre of mid-day to objects below.
When, what to my wondering eyes should appear,
But a miniature sleigh, and eight tiny reindeer.

With a little old driver, so lively and quick,
I knew in a moment it must be St Nick.
More rapid than eagles his coursers they came,
And he whistled, and shouted, and called them by name!

“Now Dasher! now, Dancer! now, Prancer and Vixen!
On, Comet! On, Cupid! on, on Donner and Blitzen!
To the top of the porch! to the top of the wall!
Now dash away! Dash away! Dash away all!”

As dry leaves that before the wild hurricane fly,
When they meet with an obstacle, mount to the sky.
So up to the house-top the coursers they flew,
With the sleigh full of Toys, and St Nicholas too.

And then, in a twinkling, I heard on the roof
The prancing and pawing of each little hoof.
As I drew in my head, and was turning around,
Down the chimney St Nicholas came with a bound.

He was dressed all in fur, from his head to his foot,
And his clothes were all tarnished with ashes and soot.
A bundle of Toys he had flung on his back,
And he looked like a peddler, just opening his pack.

His eyes-how they twinkled! his dimples how merry!
His cheeks were like roses, his nose like a cherry!
His droll little mouth was drawn up like a bow,
And the beard of his chin was as white as the snow.

The stump of a pipe he held tight in his teeth,
And the smoke it encircled his head like a wreath.
He had a broad face and a little round belly,
That shook when he laughed, like a bowlful of jelly!

He was chubby and plump, a right jolly old elf,
And I laughed when I saw him, in spite of myself!
A wink of his eye and a twist of his head,
Soon gave me to know I had nothing to dread.

He spoke not a word, but went straight to his work,
And filled all the stockings, then turned with a jerk.
And laying his finger aside of his nose,
And giving a nod, up the chimney he rose!

He sprang to his sleigh, to his team gave a whistle,
And away they all flew like the down of a thistle.
But I heard him exclaim, ‘ere he drove out of sight,
“Happy Christmas to all, and to all a good-night!”

2006 predictions

1. Apple will ship two generations of iPods. The first new generation, released in April, will have a satellite receiver built-in. The second will have a low-power FM transmitter built-in.

2. Jason Calacanis will stay at AOL though Easter 2006, and then will resign to spend more time with his family. They are good people, but Jason is an entrepreneur, and AOL is a really big BigCo. After leaving, he’ll continue to blog. Engadget will become a Time-Warner print publication. They’ll briefly use Buzz Bruggeman as a stand-in for Calacanis, and then Chris Pirillo. None of them will have the charisma that Calacanis had. Pirillo will leave AOL to sub for Adam Curry on Sirius, who will take more and more time off from his podcast, and Pirillo’s ratings will eclipse the Podfather’s, who will retire to invent the next media revolution, in obscurity, all by himself.

3. SixApart’s servers will melt down in the first two weeks of the new year, and many times after that in 2006. WordPress.com will implement a one-click import of a TypePad site, just enter your username and password and click Submit. Come back in a few hours and your whole site will be there. Under tremendous pressure from users, SixApart will implement a one-click redirect of your old URL to the new site.

4. Google will make a deal with the Time-Warner movie companies, and start movies.google.com for on-demand distribution over the Internet. Google will learn to live with DRM. Their answer to iTunes will involve loose-coupled relationships with content companies, and a new product called Google Money, basically their version of Paypal with a twist, they will also invent a new currency called The Google, which trades against the dollar, the euro, the yen and the yuan. Many small countries in Africa, Latin America and south Asia adopt the Google as their national currency. The Google money website, money.google.com, will list the current Google stock price expressed in googles, of course. They will stop reporting sales and earnings, instead reporting Gross National Product and trade surpluses and deficits. When all this rolls out, Google’s share price crashes, and takes the real estate market in Santa Clara County with it. All of a sudden Dan Gillmor can afford a mansion in Atherton on his teacher’s salary at UC-Berkeley. He becomes Mike Arrington’s neighbor and starts a new subsidiary of TechCrunch.

5. RSS will continue on the growth vector it has been on for the last six years. There will be new applications for RSS as reading lists become widely supported in aggregators, and directories like Technorati produce content in readling list form. Podcasting will be a factor in the 2006 congressional elections in the US, the Democrats will use this medium more effectively than the Republicans at first, but it will quickly even out. There will be moves in Congress to pass laws viewing the use of podcasts as equivalent to commercials, but it will be impossible to place a monetary value on it. More and more RSS will be seen as an advertising medium in itself, like catalogs, newsletters and fact sheets; and less as an advertising-supported medium. The advertising networks that formed in 2005 will wither in 2006, and completely go away by 2007.

6. As 2006 draws to a close, for a fraction of an instant the attention of the entire web will focus on my Long Bet with Martin Nisenholtz. The attention will quickly dissipate as we realize that the blogging world wins by default because the archive of the New York Times is not accessible on Google.

7. The New York Times, Harvard University and Yahoo form the new World Outline Foundation, to create an upgrade for the Yahoo directory, with a permanent model of the knowledge of the human species, accessible openly in OPML format. Yahoo modifies their search engine to understand OPML as a native format. When the Times publishes an in-depth article on a subject, the reporter’s notes are organized as an outline and linked into the accumulated knowledge. An ambitious project among universities worldwide is launched, with Harvard’s leadership, to organize scholarly information in publicly accessible OPML. I draft a new coalesced specification for OPML and contribute it without compensation to the WOF.

8. Scoble will appear on Oprah. His book on corporate blogging will top all the best seller lists, his royalties will eclipse his Microsoft salary, but he’ll stay there, because it’s in his blood. Guy Kawasaki bleeds in six colors; in his heart Scoble wants to know where you want to go today. Scoble’s next book, for which he will receive a $2 million advance, will be a collaboration with Douglas Coupland entitled Blogserfs, about the blogging community at Microsoft. Shel Israel will collaborate on a book with Vint Cerf about the downfall of Google.

More than a common export format

As discussed in the comments of the previous post on this subject, a commonly supported export format is an important first step, but it is not enough. Here’s what else is needed.

1. The export must be automatic. It must be done by the software, and kept up to date. The user must not be required to do anything to always have a good backup.

2. The backup must be stored on a static server, not on the vendor’s network. There should be a default backup service provider, but the user must be allowed to specify a different one. Apple, for example, could do this as part of their dot-mac service. I believe other companies would line up to provide such a service.

3. The user must also be able to remap their sub-domain. The user should have full control over the domain for the site, and it should not require any help from the vendor to map it to different server. An alternate, though not as desirable approach would be to allow a permanent redirect, again triggered only by a user action. The problem with this approach is that it requires that the vendor’s server be working in order for it to work.

With these three things, in addition to a common export format, even in an extended outage, the user would be in full control of his or her content. Users should insist on this level of support from vendors, or take their business to vendors who provide it.

Having been a vendor myself, I believe this kind of system will allow vendors to relax too, in addition to the users. When you have an outage, the users will have the power to move, and no doubt some will, but it takes the pressure off the vendor, and gives them time to fix the problem.

Interop between blogging tools

I’ve been reading lots of reports about the current TypePad outage, but while the outage is still going on, I don’t want to appear to interfere. I remember how awful it was to have popular bloggers jumping on my case when I last had a big outage.

This time, after the outage is cleared, we should begin a discussion, in earnest, about getting user’s data in a format that makes it easy to move between blogging software, and storing that data somewhere that’s not likely to go offline when there’s an outage. I think this will do a lot to help users feel empowered, which is the hardest part about not being able to access your blog, the feeling that there’s nothing you can do to help yourself. (David Berlind writes passionately about this.)

Of course there will be debate about what the format should be, it wouldn’t be the software industry without such a debate, but this time we should reach closure and interop. But let’s hold off that discussion until the outage is over and the users are back up.

New Orleans Notes

A summary of the Scripting News posts about New Orleans.

1. On the flight from Atlanta to New Orleans I did a 50-minute podcast with Janet, a woman who lost her house and all her posessions in the Lakeview section of New Orleans. It might be the best MCN so far.

2. New Orleans Coffee Notes. An essay.

3. Ernie the Attorney on Magazine Street in New Orleans.

4. New Flickr set: I toured the Lakeview section, New Orleans East and the French Quarter today with Ernie the Attorney

5. Three movies show the devastation of Biloxi and Gulfport.

6. Biloxi/Gulfport after Katrina: These two Gulf Coast cities took the brunt of Hurricane Katrina. A 25-foot storm surge from the gulf wiped out the populous beach communities for several blocks inland. Unlike the pictures of New Orleans’ hurricane damage, these pictures show empty lots where houses and businesses used to stand.

7. I saw a sign on St Charles pointing to http://www.nola.us/. Looks like it might be an interesting New Orleans community site. I’ve been looking for something like that.

8. 1997 DaveNet piece about New Orleans.

wordPress.root version 0.3

0. How to install.

Launch the OPML Editor.

Choose Update opml.root from the File menu in the OPML Editor app.

Then follow the instructions on the support site to download and install the new wordPress.root, and

Come back here to review the change notes, following.

1. New icons from Steve Kirks. Done.

With help from members of the community.

Thanks Steve, and everyone who helped.

2. Category support. Works.

How it works.

1. Choose Get Categories from the WordPress sub-menu of the Tools menu.

2. Create a post or place the cursor on an existing post in the workspace.

3. Right-click. Choose a category from the menu.

4. Click on Save.

You can skip the Get Categories part if the categories haven’t changed.

You can add or delete categories through the Dashboard interface on your WordPress site. Click on Manage then Categories.

Note, this is a complicated feature, so I’m not marking it Done yet, it works here, but I want to hear from a few users before I declare it Done.

3. View button. Done.

4. Indentation. Done.

When you indent in the outliner, it’s reflected by indentation in the blog post.

Tried several methods before settling (once again) on the table method.

Using blockquote wasn’t satisfactory, WordPress has a nice style sheet defined for that, that doesn’t even indent. They’re doing the right thing of course.

And as usual, the default styling for ordered and unordered HTML lists looks terrible, uneven vertical spacing.

The table method looks great.

5. Now you can create posts by pressing the return key. Done.

If you save a post that was created this way, it automatically adds the attributes to the top-level headline.

6. Run text through the glossary. Done.

This is probably something new to most OPML Editor users.

The idea is that you can teach the editor to substitute strings in “quotes” with arbitrary HTML text. Jump to user.html.glossary for some examples. You can edit that table, add your own stuff, delete our stuff. Matt Neuburg has a nice explanation from his Frontier website tutorial. Below is a picture of Gandhi, which I entered by typing “gandhi”.

7. Before releasing todo

Clean out references to appkey.

Release metaWeblog.getCategories. Done.

Screen shot