A Busy Developer’s Guide to RSS 2.0

Disclaimer: This is not spec text.

Okay, so you’re busy and you’d like to get a quick implementation of RSS up asap and have it work with the largest number of feed readers and aggregators. Coool. Here are some tips for that will make your work go more smoothly, based on the experience of real developers.

1. Find a feed that’s popular and do what it does. This is the best advice you can get, for any format any time. If you do exactly what the New York Times does, or what Yahoo does, or what my feed does (it was the first) then you can’t go too far wrong, because all the feed readers will have had to deal with whatever these feeds do, because they are so popular, and old.

2. If you’re doing a podcast feed, at most one enclosure per item. There’s been some controversy about whether or not you can have more than one, and the spec doesn’t specifically disallow it, but if you want maximum interop, you shouldn’t have more than one enclosure per item.

3. No relative URLs. In your descriptions, and in links, all addresses should be absolute. Most readers and aggregators won’t do anything special with relative URLs, so your links will be broken in those tools if you use relative links.

4. Don’t include markup in any elements other than descriptions, at the channel level and at the item level. Where you include markup, you should encode ampersands and less-thans in the way proscribed recommended for HTML by the W3C.

5. Avoid using elements in namespaces when there are already core elements that do the same thing.

6. Use namespaces that are already in use by others before inventing new ones.

Notes

1. Thanks to Tim Bray for providing a great checklist to work from. 🙂

2. If this actually gets used by a lot of people, perhaps we should have a way for a feed to notify a reader that it is following these guidelines.

3. This is an informal document, and always will be, and it’s still very much a draft and a request for ideas. It is not spec text. If you think something belongs in this profile, post a comment, let’s consider it.

4. If have strongly differing ideas, please by all means, post your own BDG.

5. Some people will say I’m stupid, or corrupt, or incorrigible, even toxic, or any number of negative personal things. What they’re really saying is they don’t like me. That’s okay, no one is liked by everyone. It doesn’t hurt my feelings, and you shouldn’t worry about it, because I don’t.

6. Namaste y’all!

Ray Ozzie’s clipboard for the web

Ray Ozzie’s new idea is a clipboard for the web. A brief summary of the idea follows; it’s fully explained on Ray’s blog, with screen casts. I watched #1 and #3.

Narrative

Let’s say you have two sites both of which understand calendar data. I want to move an appointment from one site to another. In Ray’s scheme, there’s now an icon on each site, next to each piece of data that can be transported. The icon is a picture of a scissor. Bring site A to the front, click on the scissor and choose Copy, then bring site B to the front, click on the scissor and choose Paste.

At first, you’d think it has to be some terrible ActiveX hack, but it’s not — it’s a hidden text field and a bit of JavaScript that moves the data around.

A political feature

And the crucial point is that there are now enough “standard” XML-based formats to make this kind of thing work, if developers work together and don’t create too many ways of expressing the same data, and don’t create to many different graphic representations. One of each will be enough and more than one will be too much.

That’s where the power of a leader like Ray, using a venues like Etech and Scripting News, skillfully, may manage to avoid the pitfalls. It’s an idea worth looking at, so let’s have a look.

Your ideas are welcome

Post your questions and comments here, and we’ll see if we can’t get you the answers. And if you have a demo of this idea on a non-Microsoft site please post a link here for us to look at.

PS: Photos of Ray giving his Etech talk.

Why the backchannel is bad for RSS

This is a verbatim copy of an emal I sent to Brad Feld of Mobius Venture Capital.

Brad, I appreciate you taking a proactive role, but there’s something that’s not right about all this — we need to get out of the backchannel and have this discussion in public. So many reasons for it.

1. People, rightly, complain that they had a right to be in the loop on whatever we’re talking about. They absolutely do. I know what it feels like, because I’m completely in the dark on how Rogers got to this point. At one point in the discussion with he, myself and John Palfrey at Harvard, they took it private, and I didn’t even know the conversation was continuing. There are too many dark spots.

2. If you recall, I asked if questions about RSS would be decided in Feedburner board meetings. Of course they would have been if this had gone forward as conceived. Do you see how wrong that is? RSS is not your property.

3. You talk about examining the record. It’s important that people be able to do that. Well, people coming back trying to figure out what happened here, unless Rogers decides to show us his mail archives, won’t be able to do that.

4. There are other dark spots Brad, places you and I can’t get into. Those are the times other people tried to take RSS private, to take it for themselves. I’ve never seen the converse, a private discussion that resulted in something good for the community.

5. Backchannel begets more backchannel. If you’re working with Rogers in the backchannel, then I have to work with you in the backchannel, otherwise he has an unfair advantage, and when we’re in conflict like this (not my choice btw) I can’t give the other guy that kind of advantage. Even so most all of what I said, I said in public where everyone could see it.

6. RSS is a public thing. I don’t know how to say it better. Since you’ve made such a big bet here, I think you need to understand this. This probably won’t be the last mess you’ll get dragged into. This is one of the reasons I sought you out at Gnomedex. RSS is serious stuff, and not much fun, I’m afraid. But it’s very important that it be protected.

So my feeling is, if you have something to say about this, it should be on your blog or on a mail list, otherwise don’t say it.

Dave

PS: I’m cc’ing Palfrey, so he can see how this looks. I have no visibility into his conversations with Rogers, I don’t know who decided to cut me out of the loop, or why. (I have ideas about that, but I don’t actually know.)

How much has been invested in RSS?

In yesterday’s piece I wrote of the RSS 2.0 roadmap: “There’s a huge community that has invested billions of dollars around its assumptions.”

Sounds good, but while I was riding on the BART yesterday for a lunch appointment in San Francisco I wondered if it’s true, and if so, how you’d come up with an estimate of what the investment is.

One way of measuring it is to take the dollar value of the time that has been invested by various organizations and people. So, for example, how much money has the NY Times put into its RSS support? A million? Two? And the Christian Science Monitor, Time-Warner, AP, Reuters, BBC, etc. The University of California, Pennsylvania, Maryland, Florida, Harvard, Yale, Columbia, Waterloo, Oxford, Moscow and Beijing and the U.S. Department of State, the Oklahoma Department of Agriculture. It would take quite a while to even make a list of all the different kinds of organizations that have made and continue on an ongoing basis to invest in RSS, much less the organizations themselves.

And then there’s the technology of RSS. Brad Feld has invested millions of dollars in Feedburner, Newsgator and Technorati. My former colleagues Jim Moore and John Palfrey in Cambridge have raised a $100 million fund entirely designated for RSS. We were impressed a couple of years ago with Apple’s investment in RSS, but quietly, that investment has deepened. There’s now a link on the home page of apple.com to a huge collection of feeds that are updated presumably on an ongoing basis. And then there’s podcasting, which is also RSS. Kleiner-Perkins invested $8.5 million in Podshow, and another sizable chunk of money went to Odeo, and those are just the deals we know about. National Public Radio in the U.S. is continuing to invest in podcasting. My bet is that their over-the-airwaves distribution system will become much less important than the over-the-net system, which of course is RSS top-to-bottom. What’s the annual budget of NPR and what portion of that should we allocate to RSS?

And then there’s Microsoft. A company that employs over 50,000 people, many of whom are part of their multi-billion-dollar R&D budget, a company that just filed plans to build yet another campus in the Seattle area. They’ve made an impressive committment to RSS in their upcoming operating system. If RSS delivers, and we think it will, how much will they be investing in an ongoing basis in RSS?

Okay this isn’t my specialty, but firms like Forrester and Gartner are experts in estimating dollar-value of investment. Maybe they should be tracking this, along with their estimates of the size of the user base.

And why should we care? Well I care because it would help to explain to my colleagues in the XML world why it isn’t so easy to reinvent RSS. Do the math. Let’s say the actual number is, for the sake of argument, $8.2 billion. What does that look like?

Here’s one way to visualize it. Let’s assume the average home price in the U.S. is $400K. So $8.2 billion is about 21,000 houses. Now imagine you wanted to change the way the plumbing worked in all of those homes. You get the idea. There’s no way 4 or 5 random people on a Yahoo mail list, people of ordinary means, can move that much capital without having a pretty compelling argument and making it an incredibly compelling way. Even if you had the money and would give it to all the companies that had invested, you’d have to account for the opportunity costs in rebuilding the infrastructure around a new set of assumptions. That kind of change never seems to happen. People still drive on the right side in the U.S. and on the left in Japan and the U.K. The world runs on the metric system, except the U.S. which still uses the English system. In New York City the IRT, the IND and the BMT still use different rail gauges, meaning you can’t move the trains from one system down the tracks of the other. Don’t forget the QWERTY typewriters that were designed to be hard to use but difficult to jam. Conventions don’t change easy.

Viewed another way, given that Scripting News, for years, was the central if not primary means of distributing information about RSS, it gives you a sense of how powerful blogging is. It can’t move that much capital overnight, but given enough time, and persistence, and a high-quality idea, you can create quite an economic effect. 🙂

Notes for groups interested in RSS

Last night I posted several notes in response to a post by Paul Montgomery. We’re at an interesting point in the life of RSS, where several small companies, Newsgator, SixApart, SocialText, Feedburner and Technorati may be trying to control the evolution of the format.

1. I think the Roadmap of the RSS 2.0 spec provides very clear instructions to anyone working in this area. I haven’t heard anything from this new group that says they aren’t respecting the Roadmap, so until I hear otherwise I’m going to assume that they are.

2. It’s possible that a new format, based on RSS 2.0 could be an improvement, but any person or group attempting to do that must not in any way claim the exclusive right to do so, nor should it in any way attempt to interfere with the stability of the RSS platform. No one has the right to do that. RSS 2.0 is what it is. You can extend it through namespaces, that certainly is one way forward. You can take the format and make a new format as an evolution, but you must not call that RSS. That set of constraints has served us well.

3. I initiated the transfer of the RSS 2.0 spec from UserLand to Harvard in 2003 because ownership of the spec by a commercial entity such as UserLand had become a political issue on the mail lists and weblogs. I wanted RSS to have a future unencumbered by these concerns.

It concerns me to see five companies, Newsgator, SixApart, SocialText, Feedburner and Technorati, give themselves special position among the many companies using RSS, especially since UserLand unilaterally gave up its special position with respect to RSS. It seems to me this is an issue that should be discussed publicly.

4. I would also like to know what interests the other members of this group have. Are they receiving money from the companies? Do they have any conflicts of interest? Do they assume a responsibility to disclose any conflicts of interest?

How the NY Times came to support RSS

The history of RSS is usually told only in one dimension, it’s the story of geeks fighting with geeks, so they say, but in my humble opinion, that’s really not the story.

Most of the vocal people on the mail lists, blogs and wikis are more fans than creators. It’s as if we confused baseball players with people who sit in the stands watching a baseball game. Sure, both wear caps and want their team to win, but one actually does something about it, while the others expresses an opinion. There are a lot of fans, but relatively few people who actually do anything.

Mike Lopez, posting on this blog yesterday recalls a story he heard on NPR about basketball great Wilt Chamberlain and his relationship with fans. Chamberlain’s philosophy: it’s easier to humor them than to argue with them.

Analogously, in the age of Wikipedia, fans can give themselves credit not just for being there when Wilt had his amazing 100 point game, but they can actually claim to have had the 100 point game themselves. Welcome to the Internet. Community participation is both its strongest and weakest point. And those who say I’m a consistent supporter of the medium miss that I am as frequently its victim. Sadly. 😦

Anyway, the NY Times is not a sideline player in the history of RSS, as I’ve written before, they played a central role, first denying us permission to use their content, then allowing it, and in doing so, providing an example for the rest of the publishing industry, which followed their lead without undermining them, without reinventing the technology, to their credit.

It was on this day in 2001 that I received a call from a “Rights and Contracts manager” at the New York Times, she asked us to stop reading their XML newsfeeds. I complied with the request. A colleague who will remain nameless had snuck me a link to an unprotected directory where the feeds, in a proprietary XML-based format, resided. I then repurposed the information and published it on a UserLand server. Truth be told, I expected to be shut down, but in doing so, I also expected to get the attention of higher-ups at the Times (who I knew read my blog) and it surely did.

Then early in 2002, I had a dinner with Martin Nisenholtz and Times board member Dave Liddle and two San Francisco-based Times reporters, where I pitched them on two things: 1. Publish in RSS, and 2. Give blogs to every Times reporter. They took me up on #1 and in April we published the Times content, but not in RSS (although we were permitted to by our contract), instead using the Times’s proprietary format. Why did we do that? Well I figured that if we pubished in RSS 0.92, which was then our most advanced format, it would drag the Times into a format war and they might think it’s not worth the trouble and ask us to stop publishing their content. So I decided to ease them into the community, first publishing invisibly to the people on the mail lists, and then, once their presence was cemented, we switched over to RSS 2.0.

So the loud and obnoxious fans on the mail lists shaped the story, a little. Instead of confronting their loudness, we side-stepped it. Now the Times may or may not be seen as the “tipping point” more widely, but I see it that way. Without the Times, we would have remained in disagreement, stuck in mail list hell, never achieving the promise of the technology. We all owe them a tremendous debt of gratitude, and should learn the lesson well, don’t look for leadership in standards to the netizens or to Silicon Valley, look to users who have a stake in making the technology work. Imho, that’s the key to getting things to move forward.

How RSS can bust through

Fred Wilson says “RSS has to become brain-dead simple to use.”

I’m pretty sure we can do it, but it would require the companies to give up hope of locking users into their software, into their extensions, their mistakes.

How RSS can bust through

There are two barriers to brain-dead simplicity.

1. It must be easy to find relevant feeds. Too much hunt and peck is involved. The reason My.Yahoo and iTunes have been successful is that they centralize a lot of the discovery, they make it easy to find stuff you might be interested in. But not easy enough to qualify for brain-dead simplicity. That’s why we’re working on reading lists, trying to drive adoption of the new practice by the industry. If, when you get started using an aggregator, it gives you some interesting feeds, and then as time goes by gives you more, without you having to do anything, that’s going to make the finding of relevant feeds a passive thing. Until you’re ready to take over, you can ride the bus without learning to drive. I think this is going to get us another 15 or 20 percent of web users into the RSS world.

2. Subscription has to be centralized. When Microsoft invited me in, in April of last year, to hear their RSS strategy, I think they expected me to object to their centralizing subscription for Windows users; they were surprised when I didn’t. I had already come to the conclusion that subscription had to be handled in the browser, because that’s where the impulse to subscribe happens. We knew this back in 2001, when we implemented the Radio coffee mug that made subscription a one-click operation. The problem of course is that our method only worked for Radio. Any of these techniques is going to work with only one destination, that’s why there has to be just one destination, why subscription needs to be centralized.

Microsoft didn’t go far enough. They only solved the problem for Windows. In 2006 that’s not even a very large part of the world, because a large number of people who subscribe, do it through web-based services like Bloglines or My.Yahoo, and more will over time. The Microsoft approach doesn’t work for them. If I subscribe to something using their desktop service, it only registers with software that runs on my desktop. It doesn’t inform My.Yahoo, for example. Now, Microsoft argues that Yahoo can install a toolbar that runs on the desktop, but come on, we don’t want a proliferation new stuff loading into the OS. That’s how we got in all the malware trouble. We don’t need to open that kind of Pandora’s Box. What we need is a centralized subscription public service. It’s not a technological problem, it’s a political and economic problem. In order for RSS to grow to the next level, tech companies have to stop seeking lock-in on subscriptions.

I’ve suggested to Yahoo that they run this service. Of the top three net companies (the others being Google and Microsoft) they’re the least controversial, imho. All that would be required is that they support OPML export for My.Yahoo subscription lists, and commit to keeping it open for perpetuity. The last part is the hard part of course. Now perhaps we could get a university involved, they have politics too, but people seem to trust universities more than they trust for-profit businesses. Something to think about.

Now once we have a single place for subscriptions, which is a real tall order, then all kinds of services can be built off that. It’s like the domain name system again, and perhaps that’s the way to implement it. We’re lucky that RSS is still a fairly close-knit community, and there is leadership that works, somewhat. The small tech companies and at least two of the large ones (Apple, Google) don’t participate, they blaze their own trails, but the publishing industry and most of the large tech companies are still in the mode of cooperating. So now may be a time it can work. And reading lists buy us some time.

Fred Wilson provides the emphatic statement. We can get someting very cool working, and let’s add a dose of reality — we have to work together in order for it to happen.

Scott, they need a River

Scott Karp: “I tried RSS in IE7, and it highlights the true shortcoming of current RSS applications — it’s really not much of an improvement over ‘favorites’ or ‘bookmarks.'”

Bing!

Scott says what I’ve been saying over and over. You need a different kind of aggregator and then you’ll get a benefit from RSS. The way most developers implement RSS, Microsoft included, they are just another form of bookmark, and not much better than visiting the site to see what’s new.

I think this happens because the developers don’t use the RSS apps themselves, they were just told by someone that it’s cool and they should do it, and the net-result is something that isn’t very useful. That, and no one studies prior art anymore. They literally don’t look at software that was shipped before.

That’s why I’m getting ready to ship NewsRiver. To set the bar back where it belongs, where it was when RSS started. To try to get the ball rolling again, in some kind of productive way.

Try it out. Here’s my aggregator. The username/password is snarky/snarky.

PS: I only praised Microsoft for respecting the simplicity of RSS, and I meant it. I didn’t say their software would be useful to anyone. I won’t use it. I won’t recommend it to anyone either. I told them this by groaning out loud much as you are when they demo’d it to me. But what are you going to do? They’re the Biggest of the Big. They know better, and it always takes them three times to get it right. However, even Multiplan 1.0 was a spreadsheet. I’m sure they looked at Visicalc before they designed it. We’ve lost some ground here, I’m afraid.

Microsoft ships RSS platform

This morning Microsoft has released a public beta of version 7 of their Internet Explorer web browser, aimed at developers, although I’m sure a lot of tech-savvy and adventurous users will download and install it as well.

This release is significant for publishers who provide RSS 2.0 feeds for their content because this is the first Microsoft release that includes comprehensive support for RSS not only on the producing side, but also on the consuming side. Until now, Microsoft has not shipped an RSS reader, and now they have, for Windows XP, a operating system with many millions of users. Their aggregator, and the underlying platform, is likely to be used in very large volume, likely becoming the most-installed aggregator.

It’s also signficant in that Microsoft has been a staunch supporter of the “really simple” approach to syndication. Their aggregator, of course, has to support all the flavors of RSS that are out there, but they have worked closely with the community to be sure that they were correctly using the formats and protocols that are already in wide use. As we move forward, the target can get smaller, making it easier for all developers on all platforms, not just Windows, to support RSS. Because Microsoft is such a powerful force in the software business, this practice can’t help but influence others, in a positive way.

Further, the Microsoft offering includes support for RSS 2.0 feeds with enclosures, making it a powerful engine for podcasting applications. This may create a diverse community of applications, far beyond the small number of “podcatcher” applications that are currently available.

Since I am primarily a Macintosh user these days, I have not installed the software myself, although Microsoft offered to let me try the software before its release. Even so, I am confident that they have done an excellent job of supporting RSS, and have added strength to the growing community of content providers and technology developers building on the format.

Yahoo game-changers for 2006

Yesterday I participated in a Yahoo management offsite at the spectacular Ritz-Carlton Hotel in Half Moon Bay. They invited two outsiders, myself and Om Malik, to come discuss the new ideas of 2006 with them. They asked what I thought would be the game-changers. They were interested not only in ways they could change the game on their competitors, but how a smaller upstart could be the Choice of a new generation and unseat them as king of whatever hills they’re king of.

Microsoft used to ask us to events like this, Google and Apple never have (except briefly while Amelio was in charge, but that went nowhere). Yahoo continues to impress as the exception to the rule of Silicon Valley. They don’t have the usual arrogance, they’re more inquisitive like the old Microsoft was. Refreshing.

So what did I talk about? Three things.

1. Of course I gave them an abbreviated Clone the Google API schpiel. No need to repeat it here. Search must become a developer platform. If you can’t make the current search engine do it, then hire a new team and build one that can.

2. BitTorrent. There’s no doubt that when we write the year-end pieces for 2006, BitTorrent is going to be at or near the top of the list of technologies that made a difference. Yahoo can make it two-way. Right now BT is largely serving as an (unwilling) channel of distribution for Hollywood, but now we have podcasting and videoblogging, and that stuff is just going to get bigger, and along with that the bandwidth bills for users will keep going up. Ordinary users should get the BitTorrent service for free (after all it doesn’t cost very much to provide) and Yahoo should charge advertisers to distribute their infomercials, ones that users subscribe to, willingly. This is the model for commercialization of the Internet as we go forward. It also is a game-changer on Google, which is going the DRM, appease-Hollywood sell-to-couch-potato approach. I said whereever you’re doing something to make another industry happy at the expense of users, switch polarity, immediately, and get on the side of the users. That in itself is the biggest game-change possible.

3. P2P webcasting. I wrote about this vaguely the other day, and no one apparently understood what I meant by Skype for webcasting. Come on guys, it’s pretty simple. Suppose we’re having a conversation, and I decide “Wow, this would be great for Scripting News, let’s do a webcast of this right now.” So I whip out my laptop, get onto the net (there’s wifi everywhere of course, heh) and launch my Yahoo Webcaster desktop app for the Mac. I choose New Webcast from the File menu. A window opens. There’s a button that says “Copy URL to clipboard.” I click it. Go over to my outliner, paste it into a post on Scripting News. “Tune into this webcast I’m about to do with Bull Mancuso about intellectual property and organized crime.” I highlight the word webcast and click on Add Link. Save. Then I go back to the Yahoo app and click Start. We talk for ten minutes, all the while people tune into the stream, which is managed via a realtime BitTorrent-like P2P connection. And of course when it’s all done it’s automatically archived to an MP3 and included in my RSS 2.0 feed for people who subscribe. If you’ve ever done a webcast, you know how much better this would be. And it’s ready to go, we know how to do all the bits.

PS: I’m a cheap date, probably too cheap. Today, to get me to cough up these ideas all you have to do is put me up in a swanky hotel with a Pacific Ocean view, and feed me. I sing for my dinner, so to speak.