Jon Udell: RSS

There's more than one way to read RSS

2003-02-20T16:23:16-05:00

Joe Friend's NewsGator solution

Joe Friend explains how to scan items efficiently with NewsGator:

There is a much better way to use it. Store all your newsfeeds in one folder and then use the "Group by..." feature to organize them by feed name. You can then navigate through all your feeds using the keyboard. Turning on the preview pane is best.

I had thought of that, but didn't find a feed name to group on. Greg Reinacker set me straight: it's a user-defined field, so you have to dig for it. View -> Current View -> Customize Current View -> Fields -> User Defined Fields -> .... hey, shouldn't there be an URL that gets you there?

This is excellent. Now that everything's in one folder, it can be set to view only unread messages, and I can step through them a keystroke at a time. Thanks, guys!

Meanwhile, several folks pointed me to nntp/rss, a Java-based lightweight NNTP server that can relay RSS feeds into an NNTP newsreader. And the docs hint at more:

Let's just say nntp//rss is great at letting you read RSS feeds through your newsreader, but NNTP is a two-way protocol. The next version of nntp//rss will begin to close the loop...

I haven't tried this yet, but being an old NNTP hand, I surely will. Lots of choices. I love open protocols!

Greg Reinacker's NewsGator

2003-02-20T11:17:12-05:00

Greg Reinacker's NewsGator is a fabulous hack: an Outlook plug-in (based on the .NET CLR, by the way) that reads RSS newsfeeds. It supports Outlook 2000 or 2002; I'm using 2000. I pointed it at my OPML subscriptions list and it scooped everything into a set of subfolders. Very NetNewsWire-like!

So am I an instant convert? Sadly no, for reasons that have to do with Outlook, not NewsGator. A minor issue is that in Outlook 2000, you can't suppress the To and From fields in the RSS subfolders. The showstopper, though, is Outlook's inexplicable lack of a "next unread" function. What's up with that? Various websites list the "top 10" Outlook keyboard shortcuts for things I do never or rarely, but there's nothing for the thing I do hundreds if not thousands of times a day.

When somebody comes up with the Mozilla mail/news version of NewsGator, I'll be on it in a flash.

The RSS hotlist: quantity vs. quality

2002-11-19T10:56:14-05:00

For several months now, I've been trying to move people from my old RSS feed to my new one. According to today's RSS hot list, there are 251 subscribers to the new feed (rank 48), and 145 to the old (rank 77). In theory, if everyone moved over, that would add up to 396 subscribers (rank 23) -- and indeed, 23 is about where I used to rank before the split.

I've mentioned the issue of RSS redirection a few times [1, 2, 3]. And I've tried two strategies. First, I posted an RSS item whose text was a message asking subscribers to switch to the new feed. That got some people to move over. A month later I refreshed the message, and that had a further effect. More recently, I've been trying Dave Winer's technique, which Sam Ruby says is "guaranteed to make existing aggregators go Huh?" It had an additional, incremental effect.

The upshot is that there's no good way to redirect an RSS feed. Despite that, moving my feed hasn't been a disaster. In fact, it's been kind of instructive. As the weeks have gone on, I've wondered about those 145 "subscribers" who keep hitting my old feed and coming up empty-handed. They've got to be robots. There are a number of places where my feed is aggregated into a composite display. Those aggregators don't know, or care, when a feed goes dark. Quite a lot of what we call "activity" on the Internet is like this. When I wrote about the Netscape aggregator still soldiering on somewhere in the bowels of mynetscape.com, I likened this to the problem of space junk.

As technicians, we tend to want to think about solving the redirection problem cleanly. But suppose we had done so in this case. I'd have gone on thinking that I had 396 active subscribers. I wouldn't have known that in reality, I had 251 active subscribers (actually fewer, since some of those must be robots too), and 145 pieces of space junk. As RSS continues to move into the mainstream, the distinction between list quantity and list quality is worth pondering.

Validating RSS

2002-10-23T09:51:58-05:00

Many thanks to Mark Pilgrim and Sam Ruby for their outstanding work on the RSS Validator, which as others have noticed has a tantalizing, yet-to-be-revealed connection to the Internet Archive.

This post tests a correction to my feed which should bring it into compliance. As I mentioned a few weeks back, a discrepancy between two interpretations of the ISO8601 date format had already been brought to my attention. Sure enough, the validator caught that -- and in the nicest, most helpful way, explained how to fix it.

I could frankly care less about how to express an IS08601 date, but I care enormously about RSS interoperability. So I've added the regex to my RSS writer that should make my <dc:date> be what Dublin Core wants it to be. (Yup. It worked)

Sam writes today:

Perhaps in a small way, the world is a better place today than it was yesterday. RSS Validator. Top of blogdex. Top of daypop. Wow.

Indeed!

In other news, Simon Fell's proposal to use ETags to prevent aggregators from excessively hammering on feeds seems to be gaining traction. All this adds up to a great way to start the day!

Yahoo Finance RSS feeds

2002-09-23T15:01:50-05:00

A couple of months ago, I encouraged Jeremy Zawodny to check out the blog scene. I knew he'd flourish in the environment, and guessed it might have some interesting effects on Yahoo!Finance where Jeremy works. Sure enough, check out this wonderful hack:

Yahoo Finance RSS Beta. Got a stock ticker for which you'd like to have an RSS news feed? Help test the beta RSS feeds we've put up on Yahoo Finance. Take your favorite ticker, say YHOO, and put this URL in your news aggregator:... [Jeremy Zawodny's blog]

Excellent! Immediately, a whole family of feeds becomes available. This isn't the first Yahoo service to do this, by the way. For some time now, Yahoo!Groups has done the same kind of thing on the pattern:

groups.yahoo.com/group/decentralization/messages?rss=1&viscount=10

Just swap out "decentralization" and swap in any other group name.

Mastering the network form

2002-09-10T20:00:17-05:00

Ben's feed

With the 9/11 anniversary upon us, debate rages over a web syndication format known as RSS, which stands for Rich Site Summary, RDF Site Summary, Really Simple Syndication, or my favorite, Rashtriya Swayamsevak Sangh.

Only some of those who read and write blogs -- perhaps very few -- have yet to discover the magic of syndication's two-way flow. Of that small number, fewer yet are even aware of the debate, and might find it as silly as the egg controversy that divided Lilliput from Blefescu.

Throughout this whole excruciating process, and especially during the recent flare-up, there has been a deep implicit consensus. While bitterly arguing about which end of the egg to crack, everyone is busily cracking eggs and making omelettes. The debate is spread across a series of weblogs, all connected to one another by means of interoperable variants of the very RSS format being so fiercely debated.

RSS is, all the while, broadcasting many more channels than the Lilliput vs. Blefescu rematch. For example, this morning's program included another in a series of remarkable essays from Ray Ozzie. In Tyranny, Terror, and Technology, Ray muses memorably on what it means to "master the network form," and issues a volley of links on swarming, social network analysis, and the interplay of hierarchy and decentralization.

US-style democracy at its best, Ray says, shows mastery of the network form -- a skill that always was (and still is) a matter of survival first, and then a condition of prosperity.

An aspect of that mastery, and what has kept the RSS discussion flowing even as it questions the foundations of its own format, is something that in the realm of web services has lately come to be called loose coupling.

In the comments area of Ben Hammersley's blog, I wrote:

I don't see how the option [to advance Semantic Web goals in concert with RSS-based syndication] is lost if RDF vocabularies are handled, in RSS, by reference rather than by value.

The entity called Jon Udell has lots of facets, some electronic. I know we share a vision in which some mechanism enables those facets to be threaded together, and related to other sets of facets, for a lot of useful purposes. I'm perfectly happy for that mechanism to be RDF. However since RDF addresses a MUCH more general problem domain than weblog/website syndication, I have concluded that a looser coupling of RDF and RSS best serves both sets of objectives.

Ben responds:

Hmmmmm...looser coupling? That sounds interesting. How do you mean? Can you give an example? Is that like semrefs?

Semrefs? Urp. Wait a sec...

Neo: "Can you fly that thing?"

Trinity: "Not yet. Tank, I need a pilot program for a V212 helicopter."

Jon: http://www.google.com/search?q=semref

Eyelids flutter briefly. Oh yeah, of course:

Each COMPL is defined as being formed by two components: CAT and SEMREF. In CAT the syntactic category of the complement is indicated (NP, PP, ...) and only argument complements are included. On the other hand, the value SEMREF belongs to an ontological type (animated, object, ...) that has an index assigned that relates it with an element of the argument structure. [An Interlingua Representation Based on The Lexico-Semantic Information]

Seriously, Ben, although I am a Noam Chomsky fan and have more than an inkling what this is about, I'm not sure how to answer your question. But my gut tells me it doesn't need to be baked into RSS in order to play nicely with RSS. Hyperlinks matter.

However this turns out, I'm encouraged to see that we're starting to master the network form -- maybe better than we give ourselves credit for.

Namespaces in the Scripting News feed

2002-09-10T09:26:02-05:00

Sam Ruby notes that the Scripting News RSS feed now declares a namespace:

While it is not yet listed in the spec, Scripting News's rss feed is sporting a namespace! Excellent! Now it is quite reasonable for a parser to say things like, "sorry, there is no fullitem element defined in the http://backend.userland.com/rss2 namespace - hey, Jon, get your own namespace." [Sam Ruby]

Cool. Actually I don't need a namespace of my own, I'm using some existing ones: Dublin Core and RSS 1.0's mod_content. Here's how I'm forming the <dc:date> now, by the way:

add("<dc:date>" + html.getOneTagValue(xml.coercions.frontierValueToTaggedText(@adrpost^.when,0), "dateTime.iso8601") + "</dc:date>");

I realized Radio had to have the iso8601 logic in there for SOAP. This is one way to get at it.

Sam continues:

We still have an opportunity to heal some of the rift between the two branches - simply by embracing instead of displacing one or more of the existing modules.

I agree. I'd like to see Scripting News not merely declare an abstract namespace, but actually use elements from well-known namespaces such as Dublin Core.

Transition in perspective

2002-09-09T08:57:38-05:00

I really like the exchange between Kevin Burton and Mark Pilgrim, which Dave Winer quotes today. In particular, this comment from Mark:

Keep in mind, however, that HTML is just a temporary diversion while people are still trying to grasp SGML. [diveintomark]

This reminds me of a panel at an XML conference. I sat next to Tim Bray, who was asked when XML would replace HTML on the web. His answer was, roughly, "Don't hold your breath, but don't worry about it, either." Although I keep hoping for a transition to fully XML-aware writing tools, I'm not holding my breath, and I'm trying not to worry, either.

Here's another great observation from Tim Bray, in response to the question: "Where's the schema for RSS?":

Many on this list will find it shocking, but lots of important XML dialects don't have any DTDs or schemas. Particularly in the application-glue space. People email back and forth some examples, they cut some code, and then everything's working and they're too busy to go back and write a schema.

In fact, I am at this very moment working on a proposal to do some data mapping of a big information pool that can generate XML output, they just sent us some sample instances, seemed to do the job.

I don't 100% approve of doing it this way, but that doesn't stop people doing it, and (at least sometimes) getting good results. [lists.xml.org]

This rings true for me.

Encoded content in RSS, and Sam Ruby's suggestions/corrections

2002-09-09T08:20:53-05:00

On Friday night I experimentally assigned the 2.0 version number to my feeds. Although I haven't heard about any problems related to that, I'm dropping back to 0.92 until the 2.0 spec ceases to be marked as a draft. But I'm keeping the <content:encoded> element as an alternative to the former <fullitem>. Both have the same status in 0.92: not contemplated, but not forbidden either.

Timothy Appnel wrote to point out something that I've also considered: "You should use CDATA in the content:encoded tagsets and not entity encoded HTML."

On the one hand, the mod_content spec says of <content:encoded>:

"An element whose contents are the entity-encoded or CDATA-escaped version of the content of the item."

Clearly entity-encoding fits best with current practice. But I agree that CDATA is more desirable for a variety of reasons. Perhaps now with namespaces we can have a transitional strategy that accommodates both?

PS: Just catching up with Sam Ruby's eagle-eyed observations. Thanks Sam. I've modified <docs>. And while I'd love to use Perl to munge that date, I'm doing this in UserTalk. If somebody can point me to a UserTalk equivalent, I'd be most thankful. I'm sure it's been done somewhere -- probably even somewhere in radio.root. Life's too short to reinvent that particular wheel :-)

PPS: Urp. Obviously not firing on all cylinders this morning!

Just to let you know, that is UserTalk. Written and debugged using Radio's QuickScript window (a.k.a. Ctrl-;). Enjoy! [Sam Ruby]

Sheesh, how embarrassing. Thanks Sam! Meanwhile, I notice that if the scratchpad has a weblog posting, then this:

xml.coercions.frontierValueToTaggedText (@scratchpad.thisPost.when,0)

produces:

<dateTime.iso8601>20020909T08:57:38</dateTime.iso8601>

from which the iso8601 date could be extracted. Frontier gurus: Is there a way to get directly at it?

Experimenting with RSS 2.0

2002-09-06T22:01:44-05:00

Assuming the RSS core is now frozen, Sam Ruby pointed out that version 2.0 makes the <fullitem> element in my feed bogus, and suggested I move it into the mod_content module, one of a series of modules designed for RSS 1.0. Good idea! Along with Rael Dornfest and others, I've always liked the idea of "RSS 0.9x-style simple XML syntax with room for expansion"^[1].

I was already using the alternate RSS writer callback mechanism, which Dave Winer provided a while ago, in order to override the default RSS writer and insert <fullitem>, an element containing the complete text of each blog item. This was a hack that has enabled me to maintain two feeds. In the first, the bogus <fullitem> was ignored by aggregators, but was transformed (by XSLT) into the <description> in the second feed. Meanwhile, back in the first feed, the <description> is truncated to just its first paragraph. That's done by means of another callback mechanism Dave added so those of us who write longer items needn't overwhelm those who read them in RSS aggregators.

The alternate RSS writer is a script I call myRssWriter, which lives in the table called user.radio.callbacks.writeRssFile. It is a wholesale replacement for the standard RSS writer, system.verbs.builtins.radio.weblog.writeRssFile. Originally I just cloned that script and made a few tweaks. Here were the additional tweaks I made tonight to create what will hopefully be a kosher 2.0 feed.

First, I changed this:

add ("<rss version=\"0.92\">");

to this:

add ("<rss version=\"2.0\" xmlns:content=\"http://purl.org/rss/1.0/modules/content/\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\" >");

I'm actually invoking two modules here: the content module for my fullitem element, and the Dublin Core module for the date.

Then, I changed from:

add("<pubDate>" + date.netStandardString(adrpost^.when) + "</pubDate>");
add("<fullitem>" + fullitem + "</fullitem>");

to:

add("<dc:date>" + date.netStandardString(adrpost^.when) + "</dc:date>");
add("<content:encoded>" + fullitem + "</content:encoded>");

The <dc:date> was, admittedly, gratuitous. I could have left <pubDate> alone, and in any case both are optional and not yet relied on by anyone so far as I know. But I've always liked the idea of Dubin Core metadata, so...onward!

Finally, there was an XSLT stylesheet to adjust. I use a transform of my default feed to produce my alternate long-descriptions feed. The stylesheet needed namespace declarations for <dc:> and <content:>.

Everything seems to check out. I unsubscribed and resubscribed my channels in Radio without any trouble. (Of course if any problems arise in other systems, please holler.) Should RSS readers ever start to care about <content:encoded> (or some equivalent), I can drop the second feed. In the best of all possible worlds, such readers would then offer users three choices based on the default feed:

The full content of each item.
The truncations suggested by the author of the feed.
An alternate truncation performed on the full content by the RSS reader.

Meanwhile, both flavors are available as separate feeds that work with existing RSS readers.

Rael Dornfest reflects on the past, present, and future of RSS

2002-09-06T09:04:04-05:00

Here are some of Rael's musings on the week's flurry of RSS discussion:

Do I wish we could go back to 0.91 and further, moving all those imperfect elements into namespaces? Why sure. Who wouldn't? But the sheer number of tools and applications that rely upon what's in there (for better or for worse) makes that impossible -- not without considerable breakage and retooling. Am I content to have that simply be optional water under the bridge? Sure thing.

That said, I am concerned to see last minute stuffing of the core with elements that have yet to prove their worth. Of course they're all optional again, but that's just so much more baggage (remember skipHours?) to carry around. When the idea has been put forth to go the route of XML-Namespaces, why clutter a room with boxes from the basement just before building a shelving unit. Would I like to see those elements pushed out into namespaces? Sure thing. More toothpaste out of the tube? No more than some of the fiddling with RSS autodiscovery in its first few days of existence. [raelity bytes]

It's great to see this discussion opening up. As I've mentioned to a few people this week, Google's first result for "mission of RSS" is eerie:

The truth is one but can have plural manifestations. This plurality need not be in conflict with one another; it can be cooperative and complementary. To understand, appreciate and realize the unity in a tremendous vortex of diversities, should be the humanity's goal of life.

How does Google do that?

The mission of RSS

2002-09-05T09:43:17-05:00

Sam Ruby asks: "Jon - if I read your suggestion correctly, are you suggesting that we give up on ever converging?"

The context for my answer is this message from Sébastien Paquet, which I'm quoting with permission.

Hi Jon,

Following your post on the barriers to weblog/RSS uptake in scientific and other communities, I thought I'd send you a note, as much of my recent work focuses on pushing weblogs in research. I'm a computer science researcher, on the social informatics side, and my weblog is about the evolution of scholarly communication.

First, there is a growing number of blogs by researchers. If these pioneers can organize (which I'm tring to help them do), become more visible and evangelize, they'll be much more effective than outsiders at getting others to jump on the blogtrain. For a sample, follow the links in this post.

Second, the primary obstacles to widespread blogging are cultural, not technological. Perhaps the biggest is that (regrettably) researchers are not used to sharing widely information about work in progress. A relevant note is here.

Third, I'm also interested in the general issue of why certain professions pick up blogging more easily than others. See here and here.

I'd be very interested to know more about your views/lessons learned on how to promote blogging and syndication, if you find time to write them up in your blog or elsewhere.

Here are reasons why Sébastien thinks blogging and research culture should naturally go together:

Scholars value knowledge. They have a lot of it to manage and track.
A scholar's professional survival depends on name recognition. A K-log can help provide visibility and recognition.
Scholars are used to writing; most of them can write well.
Scholars are geographically disparate. They need to nurture relationships with people that they seldom meet in person.
Scholars need to interlink in a person-to-person fashion (see Interlinktual)
Scholars already rely heavily on interpersonal trust and direct communication to determine what new stuff is worth looking at. Such filtering is one of the central functions weblog communities excel at.
For many scholars, the best collaborations come about when they find someone who shares their values and goals (this is argued e.g. in section 3 of Phil Agre's excellent Networking on the Network). The personal output that is reflected in one's weblog makes it much easier to check for such a match than work that is published through other channels.
Scholars recognize the value of serendipity. Serendipity can come pretty quickly through weblogging; see Manufactured Serendipity.
Every scholar must strive to be a knowledge hub in his niche, and an expert in related areas. A K-log is a good medium for this, as it is a way of letting knowledge flow through you while adding your personal spin.
Scholars pride themselves on being independent thinkers. K-logs epitomize independent thought.

Here are reasons why Sébastien thinks blogging has failed to become a research nexus:

It takes time.
"The technology is not well-established and tested at this point."
Many people don't like being among the first ones doing something.
Not all scholars are used to the Web and hypertext.
Shyness and fear of public mistakes. Many scholars won't write unless they have to. They may especially be reluctant to publicly expose ideas that they haven't tested.
Fear that someone else will pick up their ideas and work them out before they do.

I agree with both sets of reasons. I find it fascinating that nowhere in the second list do these objections appear:

RSS is becoming too complex. It needs to remain simple, human-readable and -writable.
RSS is too monolithic. It needs modular extensibility.

In fact both objections are valid. But neither is preventing RSS from achieving what I take to be its primary mission, which is to supercharge professional communication in the many disciplines that it has yet to penetrate.

RSS .94 looks fine to me. So does RSS 1.0 (with its .91 module). So do other proposals. The interoperable core shared by all these is what matters most, and since all are related by simple transformation, I regard them as being the same from the perspective of the primary mission. For users who don't understand or have access to transformation technologies, though, these different manifestations of the core are confusing and may account for the perception that "the technology is not well-established."

Few have argued for metadata as long and (I hope) persuasively as me. So I hope that qualifies me to say: it's not the first thing that matters here. If the primary mission is achieved, extensible metadata will matter more -- though perhaps less than we geeks like to imagine. I see RSS .9x as a basic syndication format, and RSS 1.0 as a framework for advanced metadata enrichment. My conclusions at this point are as follows:

The two functions are different enough to warrant different names.
Use the name RSS to denote the interoperable core.
Have optional namespaces in the RSS core for basic/general/ad-hoc metadata extension.
Use XSS (or whatever) for advanced/domain-specific/mature metadata syndication. But when an XSS site serves up a feed to a client looking for the interoperable core, call that feed RSS.

If you carve the world up this way, RSS is what the vast majority of sites will produce, and the vast majority of users will consume. XSS will be written and read by a small minority of sites and users who are involved in really interesting and important stuff. It will be mildly confusing to see a site offering both flavors. But the proliferation of iconography can perhaps be offset by streamlining elsewhere. With LINK REL="subscriptions" and bookmarklets for the one-click-subscribe function, for example, we could cut down on this kind of clutter:

So, would differentiating an RSS from an XSS mean giving up on convergence? I guess so. I don't see convergence as a problem to be solved for its own sake. Nor do I see convergence as a solution to Sébastien's much more fundamental problem.

RSS olive branches

2002-09-04T09:59:32-05:00

In the summer of 2000 I wrote a report that explored the uses of Internet-style groupware for scientific collaboration. In Section 3 I tried to show an audience of scientists how the two-way information flow of blogging and RSS newsfeed aggregation could support and accelerate the collaboration that is at the heart of the scientific enterprise.

Two years later, I'd make the same argument. And I'd probably have to. Despite massive uptake of blogging in certain circles, I don't see evidence that it has made much of a dent in scientific communities. The same is true, I think, in many other professions. Blogging seems huge to those of us engaged in it, and in important ways it is. Culturally, it represents a style of communication that is genuinely new. Technically, it may be the most popular application of XML. But blogging is still a drop in the ocean of email. It's far from ubiquitous, and at the ETech conference, both Sam Ruby and I were surprised to see how little-understood RSS feeds were even among experienced bloggers.

From a fifty-thousand-foot perspective, the squabble over RSS formats looks like a tempest in a teapot. Neither the simplicity of RSS .9x nor the extensibility of RSS 1.0 matters to someone who has yet to experience the "virtuous cycle" that is only recently being discovered by so many -- for example, Don Box:

While spending my evening with RSS, I had two epiphanies:

The connection between blogging and RSS is deep. WS-IL is the closest we have to RSS in the web service space.

With respect to the first observation, the cycle looks something like this:

while (true) {
ScanRSSFeeds();
RantAboutStuffYouSawFromRSSFeeds();
ExposeYourRantsViaRSS();
}

What an amazingly virtuous cycle!

I submit that we're still at the beginning of the RSS adoption curve. To insiders, it seems as though the squabble has gone on forever, but I don't think outsiders see that. Up to a point, we can put band-aids on the wound, but it does need to be healed -- and I don't think it's too late. It's encouraging to see Dave Winer's statement in favor of namespaces and modular extensibility in the RSS 0.94 roadmap. Will the RSS 1.0 camp offer an olive branch of its own? I hope so.

RSS auto-discovery and meta-linking

2002-06-03T01:55:03-05:00

I've implemented Mark Pilgrim's final version of the RSS auto-discovery technique, inspired by Matt Griffith's proposal. A small change in Mark's Radio bookmarklet seemed to be needed, from indexOf(application/rss+xml') to indexOf('application/rss+xml'), in order to be able to subscribe in Radio to sites using this technique.

Mark writes:

It has been surprisingly painless and friction-free. Together, we have come up with a new standard that is useful, elegant, forward-thinking, and widely implemented. In 4 days.

Amen!

DJ's meta-link is a great idea too. An excellent way to facilitate the kinds of social network analysis that we're all buzzing about lately.

Postel's dictum applied to HTML in RSS

2002-05-23T21:08:52-05:00

In the commentary attached to Ben Hammersley's RSS discussion, someone (whose name got truncated) cites Jon Postel's famous dictum in regards to use of HTML in RSS:

Escaped html in the tag is very common and isn't going to disappear any time soon. So let's make it a convention as RSS creators to be sparing and deliberately strip the more aggressive tags when we create it. As consumers, we should be writing code that strips the tags we don't like and not complain too much when people throw incomplete tables at us. Be lenient with what you consume, be pedantic and accurate with what you create.

I took a run at this problem last week on a plane trip. For a long time now, I've intended to rip out the MS DHTML edit control in Radio, and pop in Ektron's slick eWebEditPro, which I wrote a column about last year. It should be a straightforward swap that would make Radio's WYSIWYG editor able to emit either clean HTML or XHTML, rather than the hideous stuff that MS control spews out now.

Given the amount of writing I do here, I'd be quite willing to pay something for a add-in that would enable me to honor Postel's dictum -- that is, "to be conservative in what you send."

For some reason, though, I went round in circles. Couldn't seem to find the right combination of UserTalk, HTML, and JavaScript to get that Ektron control working in place of the default WYSIWYG control. I'll take another run at it one of these days, but I just thought I'd report the idea in case somebody else wants to go there first.

What is an RSS description?

2002-05-23T00:30:20-05:00

Ben Hammersley has taken note of a flock of RSS-related rumblings. Here's some commentary on his commentary on my rumblings.

<fullitem> a sub-element of <item>

<fullitem> does for RSS0.9x what mod_content does for RSS1.0. It allows the entire item text to be included in the feed, including entity-encoded HTML markup.

This, to my mind, is where one possible weakness comes in: allowing formatting HTML markup, such as FONT tags, within an RSS feed does allow feed providers to royally mess up aggregating sites. A misplaced *lt;tr> might break much layout, as would a missized font, and so on. My suggestion is to include an attibute that points to a suggested stylesheet:

This fits nicely with the push toward XHTML in ordinary webpages, and seems more elegant. To me at least.

I guess I'd say that <fullitem> does for RSS0.9x what <description> does for RSS0.9x, in a situation where (as is true for my primary feed now), <description> is truncated to less than the entire item.

Sending escaped HTML markup can cause all sorts of trouble, for sure. It's really embarrassing to break other peoples' aggregators with a bum feed, as I've discovered for myself. It'd be wonderful if XHTML writing tools were common. But they're not, and until/unless they become so, I guess the safeguard is the immunological system of blogspace, which quickly punishes offenders.

<blurb> a sub-element of <item>

The blurb element contains a precis of the item, halfway in size between a description and a full text. I would suspect that this might only be used for very long pieces, where the full text is much too full, and the description too high-level.

As I'm using it (in my secondary feed), <blurb> is the truncated item which, in my primary feed, is called <description>.

This is goofy, of course. I did it in part to see if the sky would fall if I added extra tags into my non-modularized feed. (It didn't.)

I'm aiming to offer choice. Rather than always truncating (which disappoints people who like to read whole items in aggregators), or always sending whole items (which disappoints people who like to scan and decide whether to click through), I offer both styles. At the moment, I do this in parallel feeds:

primary feed: <description> = truncated, <fullitem> = nontruncated

secondary feed: <blurb> = truncated, <description> = nontruncated

I'm sure nobody is using either <blurb> or <fullitem>. Personally, I'd rather to combine these functions like so:

In principle, the algorithm used to truncate (for me: first paragraph; for Eric Snowdeal, the first 500 characters) could be applied within a single instance of the item, without the duplication I've introduced. In practice, I doubt such intra-item coding would work reliably.

I expect that current practice -- either truncating items or not -- will continue. A few people (like me) may bother to offer a choice, in the form of parallel versions. The overhead is no big deal really, XSLT happily transforms one into the other. While aggregators could offer users the choice, within a single feed, of long or short variants of that overloaded thing we call <description>, I doubt this will matter to enough people to get off the ground.

Basic and advanced RSS

2002-05-20T11:11:12-05:00

Sam continues to provide an excellent perspective on moving RSS forward:

I really would like to see the day where compatibility to the spec was as important as "works with the current version of the software provided by the spec author"

... as a contributor to a SOAP toolkit, I don't want SOAP to be defined in this way.

Meanwhile, here is the perspective of an author of another aggregator. [Sam Ruby]

Thanks Sam, especially for that pointer to Aggie.

In Radio, the path of least resistance at this moment is to clone the RSS writer and add in experimental tags. The better way (in my view) is to replace the RSS writer with a modularized writer that compartmentalizes experimentation according to the RSS 1.0 spec which was designed for that purpose. A Radio hacker is free to do either or both of those things. I've done the first, and plan to do the second unless (hopefully) somebody else gets there first.

In either case, it seems to me, the gating factor is whether and how the UI exposes custom tag creation to the user, a la the Categories feature in Radio now, and whether and how the UI enables the user of the aggregator to work with extended metadata.

Coming back to the lawyers-and-asbestos angle, it's worth noting that the ability to solve the problem already exists. The .92 spec supports categories; categories can be created in the UI; categories can be routed as complete XML feeds. Although I completely agree with the points being made about extensibility and namespaces, I don't think these issues are preventing lawyers from today organizing their communication around topical feeds. Rather, I think what's stopping them is Sam's observation about the general lack of appreciation for the most basic uses of RSS.

There's a mom-and-apple-pie aspect to metadata collection. Everybody's for it -- at least insofar as they imagine a neatly-tagged semantic web made available for them to use. When they realize it's their job to do all that neat tagging, though, enthusiasm (rightly) wanes. If we had lots of people using software that made metadata collection seem natural and effortless, I don't think it'd be hard to sort out issues of extensibility and namespace collisions. Those would be good problems to have, in other words.

Extending RSS: first things first

2002-05-19T23:41:45-05:00

On the always-thorny question of how to extend RSS, I guess I'm for a first-things-first aproach.

Rory Perry:

If the WV court has an XML feed for recent opinions (which we do), the lawyer in New Orleans could subscribe to that feed and watch for orders and opinions regarding asbestos mass litigation.

Sam Ruby:

If we want Internet-scale standards (whereby the likes of Rory Perry can create discipline-specific extensions), we need to get to the point where everybody has equal opportunity to create modules.

I am not religious about this stuff. I see no problem with Rory and his legal pals agreeing on some tags (like <asbestos>) which they'll use by mutual consent. The immediate bottleneck is getting software into their hands that enables one user to pop such a tag into a feed, and another user to discriminate based on the tag. And then getting them to the point where they can actually experience that.

For a while now I've been sending out RSS channels with things like <pubDate>, <blurb>, and <fullitem>. Nobody's complained, so apparently it's not breaking any existing aggregator. Thanks to the new ability, in Radio, to replace the RSS writer, I can -- and indeed will -- replace my feed with RSS 1.0, using Dublin Core metadata and possibly defining a module to account for the variant elements (long vs short description) in my feed.

The phrase "Internet-scale" always worries me a bit, though. Until and unless the likes of Rory and friends can start bootstrapping the process of creating and consuming customized feeds, there's no scaling issue to worry about. If and when namespace collisions start to become a problem, then people will be in a position to see the value in modularized readers and writers. At which point, it shouldn't be hard to transition to them.

But do people need modularized readers and writers to even get to first base? Or do they just need to find out what it feels like to hit a few singles, using the simplest tools? Given Sam's (and my) surprise that even the most basic use of RSS is still relatively new to many bloggers, I'm inclined to take things one step at a time.

Daylight ahead for RSS writers and readers

2002-05-10T09:46:51-05:00

Jenny writes:

What I do see as the bigger issue is that this provides a path for further news aggregator development in Radio. I'm not knowledgeable enough to know what it means for RSS news aggregation in general, but I believe quite strongly that some form of aggregation will become part of our everyday information lives in the future, so I welcome any and all roads that lead to that day. [The Shifted Librarian]

I believe the same. It's important to note that Radio's aggregator is one way forward, but not the only way. I just grabbed a copy of AmphetaDesk and -- omigosh -- it's written in Perl! I had no idea! It ships as a compiled executable for Windows, Mac, and Linux, but the source is all Perl and is available. The app runs Radio-style, shoveling script-written pages into a local webserver for browser consumption.

This presents a bit of a dilemma for me. I've made my peace with UserTalk, but I'm much faster and more competent in Perl. So it's tempting to do aggregator experimentation in AmphetaDesk. But then, it can't be so easily shared with the Radio community, many of whom will (rightly) prefer not to download and install an extra kit. But, this is a good problem to have: a choice between two viable options.

In any case, the point is that you're right, Jenny. All sorts of different groups will have reasons to tweak both the production and the consumption of RSS. For both writers and readers of RSS, there's daylight ahead.

Radio's RSS writer is now user-extensible

2002-05-09T12:06:33-05:00

The RSS writer in Radio is now officially user-extensible. "Before generating the RSS, we check user.radio.callbacks.writeRssFile," Dave writes today. Excellent. This will open the floodgates for all sorts of useful metadata experimentation. We'll see Radio UserLand sites emitting RSS 1.0, and others extending RSS .9x. It's not the format that matters to me, it's the experimentation.

In that vein, I've heard from a few folks who are working the other side of the street, looking for ways to enhance the aggregators that read RSS channels. I think this is fertile ground for innovation. Personal aggregators are still quite new, and we have a lot to learn about how we want to use them.

Two flavors of RSS channel

2002-05-08T08:46:07-05:00

Jenny's poll reminded me that RSS truncation shouldn't be an either/or choice. So I'm experimenting with some extra tags in my RSS feed.

The basic feed continues to send truncated descriptions. It adds a [fullitem] tag that has the complete text of the item. Of course no newsreaders use this yet. But I want to make sure that sending this extra tag won't cause problems. In Radio it seems not to, but I want to see if AmphetaDesk, NewsIsFree, Meerkat, and others are OK with it. If not, please let me know!

I've also added a [pubtime] tag, just to see if I can. As Sam Ruby keeps pointing out in another context, extra information shouldn't be a problem so long as the required core elements are present.

There's also a long-format feed which is just an XSLT transform of the basic feed. In that version, [description] is the complete item, and [blurb] is the truncated description. So now there's choice. You can subscribe to either the basic feed with short descriptions, or the long-format feed with long ones.

The next step would be to experiment with the news aggregator. Given a feed containing both short and long items, it ought to be possible to let the user toggle between them, perhaps even on a per-channel basis.

Paul Holbrook improves RSS truncation

2002-04-11T19:50:36-05:00

Paul Holbrook takes a different approach to RSS truncation. He presents the entire lead paragraph, like so:

on firstPara(description, adrpost) {
return (html.getOneTagValue(description,"P"))}

The description is stored in HTML, so getOneTagValue conveniently does exactly what I want.

Of course, if the first tag in my description isn't a P, this won't do what I want it to do. I'll work on that another time. [ Paul Holbrook's Radio Weblog ]

I'm trying it now...yup, that's the ticket. Thanks, Paul!

Shortening RSS descriptions to lead sentences

2002-04-11T02:16:56-05:00

I'm really enjoying my ability to scan a lot of sources in my Radio news aggregator. What slows me down, though, are the channels with long or irregularly-formatted descriptions. One of the worst offenders is the Privacy Digest. I want to scan that channel, but the cluttered format of its descriptions makes it really hard to do so.

By contrast, take a look at Eric Snowdeal's feed. Although it's mildly disconcerting that the truncation doesn't respect word or sentence boundaries, the effect -- especially in the aggregator -- is to make this feed much more easily scannable.

For my own reading convenience, I may wind up pulling feeds out of Radio's aggregator database and reformatting them. But on the flip side, I've been feeling kind of antisocial about dumping these longish essays into an RSS feed that I myself wouldn't want to read.

So for now, I've changed this line in system.verbs.builtins.radio.weblog.writeRssFile:

add ("<description>" + adritemcache^.text + "</description>")

to this:

local (s = adritemcache^.text);
s = string.replaceAll( s, "<" , "<" );
s = string.replaceAll( s, ">" , ">" );
s = string.replaceAll( s, """ , '"' );
s = string.replaceAll( s, "'" , "'" );
s = string.replaceAll( s, "&nbsp;" , " " );
regex.subst("<[^>]+>", "", @s);
add ("<description>" + string.firstSentence(s) + "</description>")

Radio's string.firstSentence seems like just the ticket. A lead sentence is supposed to be special. Knowing that it's all a reader of your RSS feed might see (after the title) makes it even more so.

Of course, it's completely non-kosher to hack into a system script that can be overwritten by UserLand at any time, as happened most recently a few days ago:

4/7/02; 9:55:21 AM by DW {
Do the macro processing or descriptions unconditionally, and in-line instead of calling radio.string.processMacros, which was only processing the descriptions if it contained "<%". This broke shortcuts. Prior art is the Manila-Blogger Bridge Tool, which unconditionally processes macros.}

Nevertheless, you've got to love the openness of a system that makes it possible, and easy, to do this non-kosher thing.

Wiki/weblog integration points

2002-04-04T15:44:32-05:00

Tony Bowden writes:

Having a wiki output OPML won't work, as it requires creating structure that isn't there. Having a wiki input OPML, on the other hand, might produce much more useful results.

Going the other way, RSS is a great connector. I just introduced myself to Sunir Shah the other day. I was going to suggest that MeatballWiki offer an RSS feed of its RecentChanges when I looked again and lo, it already does. Even nicer would be to include the first diff in that feed, so that somebody scanning a lot of feeds can make better choices about what to read. Heads, decks, and leads. The Wiki naming style makes for nice heads, and some items in the feed have nice decks (short descriptive tags) as well. The first diff would make a nice optional lead.

The model according to which Wikis federate is something that the blogging community could profitably study. Peter Thoeny explained it all to me once. Now I want to look into all that again.

Here's a column from two years ago that compares Wiki and newsgroup collaboration. It ties together several of these themes.

Blog locally, rewire globally

2002-03-19T09:47:36-05:00

Mike Krus asks:

I wonder if you can write a aggregator driver in Radio8 which overwrites the default one for RSS? I'm not at home so I can't try...

But if you could, I'm sure Jon would find it useful. He could write a driver which strips the description field or cuts it at 100 chars or something when the feed is read into the aggregator. [Too Much News]

Could do, but you'd have to deal with the multiple (and now growing) set of inbound drivers, for RSS .9x, 1.0, and now other stuff. Also, you might want to keep all the content locally for offline use. I still see this as best handled in a single option for outbound RSS production.

BTW, an interesting coincidence this morning. Mike's NewsIsFree reported a new channel, Nasa Liftoff. It's available as Nasa's non-RSS XML, and also as synthesized RSS. This is slightly misleading, though. Really, there are three channels:

Nasa Liftoff

Nasa Science

Nasa SpaceCal

What's scary here is that I just created those three feeds by cloning a local file twice, and changing two words. (I'm expecting someone will show me how I could have used the W3C XSLT service in parameterized way, in which case, this would have reduced to cloning an URL twice and changing two words.) Once Radio upstreamed those changes, I had added value to Nasa's news offering -- value that is instantly available to all RSS readers.

Rael said it nicely: blog locally, publish globally. And when publishing can also mean rewiring global infrastructure, you can start to see what's behind the Web services hype.