It's time to close this chapter of my blog and begin the next one. Here I'd just like to thank Steve Gillmor for bringing me to InfoWorld on a mission to explore the blog medium and to reinvent publishing. And I'd like to thank InfoWorld for granting me extraordinary freedom to pursue that mission. I think I've used that freedom well, and made good progress in the right direction. But along the way my interests broadened beyond InfoWorld's enterprise IT charter, and I began to overstay my welcome. Take a look at this page for example. Scroll to the bottom and you will see that one of these things is not like the other. A video of a lawnmower? What's up with that?

To me it's all part of a pattern. I use commonly-available technologies in unexpected ways to tell stories that make connections, distill experience, and transmit knowledge. Jay Cross nailed the reason on his blog the other day: I'm "infected with Stewart Brand's memes." Of course, most of us who hang out on the leading edge of technology are similarly infected. So I wind up preaching to the choir. That's fine, and I'll keep on doing it so long as we all find it worthwhile, but I also aim to connect with a lot of people on the trailing edge, many of whom have yet to subscribe to an RSS feed, publish a professionally-oriented blog, or compose a new service by stitching URLs together. I hope that Microsoft will help me to take these ideas to the world in a big way to the benefit of all concerned. According to the comment thread attached to my announcement, many of you agree. So, let's do the experiment.

Comments

As a first experiment I grabbed the DCStat reported-crime feed for November, sucked it into Excel 2003, consolidated incidents by day, pivoted them on type of offense (homicide, burglary), and exported them back out as a CSV (comma-separated value) file that Swivel could import. [Full story at InfoWorld.com]

I imported the same data into Dabble DB where it's very easy to use grouping and filtering to make views like this one. Again the point is that the views are sharable on the web. Also, in this case, invited collaborators can tweak them.

Going through this exercise, I was struck by the distance between DCStat's namespace-rich XML formats and the CSV format that web apps like Swivel and Dabble DB want to read and write. I happen to know how to use the XML Maps feature of Excel 2003 to shred an XML file but I doubt many Excel users have ever done that. To enable ordinary citizens to explore this data, DCStat might want to offer a common-denominator CSV format in addition to the XML flavors.

Comments

Sean McGrath's latest column puts an AJAX spin on the age-old struggle between the human-accessible and machine-accessible aspects of software:

Making an application easier to use almost always means making better use of the users ability to see; to hear; to click buttons; to access drop down menus and so on. Unfortunately, computers themselves cannot see or hear. This results in a most unfortunate inverse relationship. Namely, the more visual and interactive and compelling an application is from a user interface perspective, the harder it is to make said application do its thing without human intervention. [Sean McGrath: AJAX and the hidden cost of ease of use]

Now in fairness, I cheated by using the AJAX interface to emit a sample HTTP transaction that contained a lot of stuff I could not easily have fabricated from scratch. That's why it'll take a much smarter bear than me to incorporate my hack into libgmail. Still, if I'd wanted to plow through Gmail's obfuscated JavaScript source code, I might have been able to intercept and repurpose the method driving the HTTP transaction.

For these two reasons -- the transparency of the HTTP pipeline, and the accessibility of the JavaScript object model -- I think that AJAX is inherently more automatable than conventional GUI apps ever have been.

Comments

At the ICA conference in September I had a great talk with Edwin Bruce, who's responsible for New Zealand's e-government initiatives. Among many other things, he pointed out that New Zealand has an opportunity to do some interesting things with moderated second-level domains under the .nz top-level domain. There are five such domains now: .govt.nz (government), .mil.nz (military), .iwi.nz (Maori), .parliament.nz, (parliament) and .cri.nz (Crown Research Institute).

Because New Zealand's country-code top-level domain (ccTLD), .nz, is much more tightly controlled than, say, .us, New Zealand's moderated 2nd-level namespace is analogous to ICANN's notion of sponsored/chartered top-level namespace. If you're a bank or a law office today in New Zealand, you'll probably register under the unmoderated 2nd-level domain .co.nz. Edwin envisions aligning the existing regulatory apparatus for banking and law with corresponding moderated 2nd-level domains: .bank.nz, .law.nz.

This wouldn't be an option in the US, at least not under our ccTLD. For example, .law.us is currently held by Neustar, but .bank.us is owned by Vishal Ved, 9335 Lee Highway, #1213, Fairfax, AL. So if we wanted a system like the one Edwin proposes, we'd need to do it under a new TLD.

I'm sure that's unlikely for all sorts of reasons, and what Edwin proposes isn't even happening yet in New Zealand where it pretty easily could. But it would be interesting to see this model tried there. Are there other ccTLDs where it's farther along?

Comments

Reading through the comments attached to yesterday's announcement made it clear that I left a few critical questions unasked and unanswered. Rather than continue that thread here, I'm transplanting it to a new incarnation of my ancient personal website. It used to be udell.roninhouse.com. Now it's jonudell.net, where I display my past record for inspection, and this blog for new thoughts and dialogue. I hope you'll join me there because during my sabbatical, from Dec 15 to Jan 15, I'll be laying the foundations for what I want to do next, and I'll appreciate all the help I can get.

For my last week here, I'll stick to InfoWorld-related knitting. But I can't resist pointing out my favorite of all the heart-warming reactions that poured in following yesterday's announcement. It's a 3-second video from Andy Rush, one of the switched-on instructional technologists I met at the University of Mary Washington back in May. Andy, you made my day!

For today's podcast I decided to interview myself about my upcoming new gig. It's a short episode, under six minutes, and the transcript follows.

Note: I actually meant to push this to the server later today, to synchronize with a message that will be forthcoming from Jeff Sandquist. But a mis-click on my part pushed it sooner, which means Jeff will be a bit surprised when he wakes up. Trust me, though, this is something I've been thinking carefully about for a long time, and it's very real.

Q: Your new job is with Microsoft?

A: That's right. My last day at InfoWorld will be Friday Dec 15. On Jan 15, after a month-long sabbatical, I'll become a Microsoft employee. My official title will be Evangelist, and I'll report to Jeff Sandquist. He's the leader of the team that creates Channel 9 and Channel 10, websites that feature blogs, videos, screencasts, and podcasts for Microsoft-oriented developers.

Q: What will your role be?

A: The details aren't nailed down, but in broad terms I've proposed to Microsoft that I continue to function pretty much as I do now. That means blogging, podcasting, and screencasting on topics that I think are interesting and important; it means doing the kinds of lightweight and agile R&D that I've always done; and it means brokering connections among people, software, information, and ideas -- again, as I've always done.

Q: Why are you doing this?

A: I'm often described as a leading-edge alpha geek, and that's fair. I am, and probably always will be, a member of that club. But I'm also increasingly interested in reaching out to the mainstream of society.

For those of us in the club, it's a golden age. With computers and networks and information systems we can invent new things almost as fast as we can think them up. But we're leaving a lot of folks behind. And I'm not just talking about the digital divide that separates the Internet haves from the have-nots. Even among the haves, the ideas and tools and methods that some of us take for granted haven't really put down roots in the mainstream.

Over the years I've evangelized a bunch of things to the alpha-geek crowd: Internet groupware, blogging, syndication, tagging, web architecture, lightweight integration, microformats, structured search, screencasting, dynamic languages, geographic mapping, random-access audio, and more. There's a purpose behind all this, and Doug Engelbart saw it very clearly a long time ago. The augmentation of human capability in these sorts of ways isn't just some kind of geek chic. It's nothing less than a survival issue for our species. We face some really serious challenges. The only way we're going to be able to tackle them is to figure out how to work together in shared information spaces. I've chosen to align myself with Microsoft because I think it has the scale, the resources, and the business incentive to help me empower a lot of people to learn how to do that.

Q: Why now?

A: At the Emerging Technology Conference in March, Microsoft's incoming chief software architect, Ray Ozzie, showed how LiveClipboard, the 21st-century version of the Windows clipboard, could enable collaborative sharing of information, and creative recombination of services, across all operating systems, web applications, and desktop applications.

Kim Cameron, Microsoft's identity architect, is taking a similar approach in the domain of identity, privacy, and the control of personal information.

Jean Paoli, Microsoft's Office XML architect, continues to pursue his lifelong dream of empowering millions of people to create and use smarter documents.

Jim Hugunin, who created both Jython and IronPython, is making my favorite open source scripting language, Python, a first-class citizen of the .NET platform.

J.J. Allaire is creating a blog-writing tool that will enable millions of people to publish data that's reusable and intelligently searchable.

Bottom line: This isn't your father's -- or maybe your older brother's or sister's -- Microsoft. Initiatives like these matter, they're solidly in line with my own agenda, they're being pursued in very open ways, and I want to help move them forward.

Q: Are you selling out, joining the Evil Empire, and turning your back on principles you've always championed?

A: Wait until the evidence is in, then decide for yourself. I've been in this game for a long time. I think my record of pragmatism and agnosticism speaks for itself, but sometimes I like to recall what Tim O'Reilly said in his foreword to my 1999 book on Internet groupware:

All too often, people wear their technology affiliations on their sleeve (or perhaps on their t-shirts), much as people did with chariot racing in ancient Rome. Whether you use NT or Linux, whether you program in Perl or Java or Visual Basic - these are marks of difference and the basis for suspicion. Jon stands above this fragmented world like a giant. He has only one software religion: what works.

Q: Will you be a blogger? An analyst? A developer? An educator? A multimedia producer?

A: All of the above. The title "evangelist" doesn't quite capture that whole range of activities, but these are the things I do, and plan to continue doing.

Q: Will you become Microsoft's next Robert Scoble?

A: The way I see it, Robert played a key role in a grand experiment to make Microsoft's development processes more transparent. Channels 9 and 10, and the hundreds of Microsoft blogs throughout the organization, are evidence that the experiment is succeeding.

I've proposed a different experiment. I'll continue to be a channel for alpha geeks. But I also want to become a channel for a whole lot of civilians in the mainstream. And above all, I want to build bridges between these two groups.

Q: Will you continue to use Firefox, Gmail, and OS X?

A: Sure. I'll also continue to use Microsoft technologies as I always have, and I'll keep on pushing the boundaries of cross-pollination and interoperability. The most powerful mashups don't just mix code and data, they mix cultures. I hope this will be an opportunity for me to do that in a way that benefits everybody.

Comments

The solution I cobbled together speaks volumes about the fundamental openness of Web applications. To find out how Gmail creates a distribution list, I logged in, created a list interactively using Gmail's form, and captured the resulting HTTP transaction using one of the handiest tools in my Web developer's kit, Firefox's LiveHTTPHeaders extension.

The next step was to replay that transaction outside of the browser. I rearranged its elements -- an URL, a chunk of HTTP POST data, and a set of HTTP headers including a cookie packed with crucial name/value pairs -- as a command-line invocation of another of the handiest tools in my kit: curl. [Full story at InfoWorld.com]

In this week's column, I discuss how the ability to capture and replay HTTP traffic enabled me to discover and exploit an implicit Gmail API. But there's a general principle underlying this hack, and it seems to me that after all these years we've barely begun to exploit it.

Consider the scenario described in this item, for example. I would love to be able to recapture the sequence of HTTP transactions behind a particularly interesting search scenario, but I never logged them. In the comment thread, aristus notes that one solution is a Firefox plugin called slogger, which I've used on and off since, let's see, tap, tap, tap, October 2004.

I haven't used slogger for a while because it quit working for me on OS X a while ago, and I haven't been able to resurrect it since. But slogger notwithstanding, there's a much deeper and more general thing that ought to be happening on every web-enabled system. It ought to be trivial to attach an observer and/or filter to HTTP pipelines. Among other things, it could shovel data into a search engine so that I could instantly recall a remembered transaction by search term, by date, or by site.

I mentioned this on a recent call with the folks reponsible for Vista's desktop search. When they mentioned extensible "protocol handlers" I got really excited, imagining a general mechanism for echoing HTTP (or SMTP) traffic through a search indexer. It turned out that isn't what they meant. They were talking about supporting file formats, not protocols. But I've always thought echoing HTTP or SMTP traffic through a searcher is a great idea, and I still do.

HTTP intermediaries are also an incredible untapped resource for developers and testers of software running in environments that range from plain old HTML-over-HTTP to formal XML web services. For the latter domain, the folks (disclosure: my friends) at Mindreef are doing really interesting work based on the ability to capture and replay SOAP packets.

I've long envisioned a general-purpose ipchains-like capability, for all operating systems, that would make it trivial to attach observers, filters, and transformers to bread-and-butter protocols like HTTP and SMTP. Is it just me, or is this a gigantic missed opportunity?

Comments

Google got there first, but we're still in the early innings of this game. Google's office apps, while collaboratively adept, are functionally lame. Microsoft's apps are adept and lame in precisely the opposite ways. Everyone needs to converge on solutions that deliver the best of both. [Full story at InfoWorld.com]

In last week's email I received a leading indicator of GDoc/GSheet adoption which, because I neglected to ask permission to quote, I'll anonymize:

Just read your column on G-docs. We started using them around the office for convenience and they've taken off. We found the version control, collaboration and invite system outweighed the limited feature set. For the most part, they have the very basic functionality covered. We still use Excel for many things, but Google Spreadsheets is making a dent in our enterprise use. My clients have adopted it too. Helps when working with teams in China, Europe and the US.

My boyfriend and I use one to keep track of our bills at home. Just easier than sorting it out over email.

Comments

Last night an old friend who runs a small software company confessed a secret. When he and his staff answer technical questions for clients, they are often "only" searching Google. At one point, he even asked a client: "Do you really want us to search Google for you at $100/hour?" Yes, in fact they did. My friend thought that was crazy. I suggested that it's not as crazy as it sounds. Effective search depends on reservoirs of tacit knowledge and unconscious skill. Some people possess much deeper reservoirs, and/or can tap into them more effectively, than others. That makes them valuable.

Lately, though, I'm less inclined to accept that some people are natural information hounds, and others aren't, and that's just the way of it. Innate talent clearly plays a role, but so does learned skill. What the learnable component of effective search may be, though, is very unclear. So I've begun to reflect on, and document, my own search habits in order to try to discover what it is that I've actually learned how to do.

Here's an example from an item back in August:

I knew the title of the talk I wanted to cite: The (Real) State of the Union: Atlantic Monthly Panel. But it took a while to find it on the site. My search strategy went like so:

query	outcome
state union	fail
state of the union	fail
"state of the union"	fail
atlantic	succeed

Should've let Google do it, maybe, but in any case I found my way to the home page for the event. I was seeking two facts. First, the name of the woman whose 9-minute segment of the hour-and-a-half panel impressed me. Second, the timecodes for that segment. The event page provides neither.

It wasn't easy to find the speaker's name. Here was the search strategy:

action	outcome
listen to podcast intro	partial success (sounds like Shannon Branley or Brantley or Bradlee, with the Numerica Foundation)
google query: shannon branlee	fail
google query: numerica foundation	fail
google query: ted halstead (also mentioned in connection with the foundation)	partial success (refine query to New America Foundation)
google query: new america foundation	partial success (found the organization)
new america foundation query: shannon	success (it's Shannon Brownlee)

You can see some patterns emerging from the fog: exact phrase search, the lateral maneuver from Shannon Branlee to Ted Halstead. There are a lot more of these patterns needing to be codified, but first we need to collect more examples of articulated search strategies. So I plan to catalog mine here and would love to be able to find yours here.

Admittedly it's hard to capture these examples. We're always in a hurry to find what we're looking for, and when we do, the strategy that we hauled up from the unconscious depths sinks right back down. But if we could capture and share some of these examples, it'd be really useful.

A while ago I ran into a beautiful example that I failed to capture, and now I'm kicking myself. It went something like this:

Next time something like that happens, you can bet that I'll freeze-dry the example and pin it down in my collection box.

Comments

My guest for this week's podcast is John Wilkin. He's the director of the University of Michigan Library's technology department, and coordinator of the library's joint digitization project with Google. It's been two years since Google began partnering with the University of Michigan and with other libraries, including Harvard and the New York Public Library. In this conversation we talk about the UM's earlier (and still-ongoing) efforts to digitize its 7-million-volume library, about how the partnership with Google has radically accelerated that process, and about what this is all going to mean for libraries, for publishers, for Google, and for all us.

Web resources mentioned in the podcast include:

• JSTOR, a Mellon Foundation project chartered to "build a reliable and comprehensive digital archive of important scholarly journal literature"
• Making of America, "a digital library of primary sources in American social history from the antebellum period through reconstruction"
• The Michigan Digitization Project, and in particular the contract between UM and Google
• Distributed Proofreaders, which "provides a web-based method of easing the proofreading work associated with the digitization of Public Domain books into Project Gutenberg e-books"

Comments

The November episode of The Screening Room features PowerSchool, Pearson School Systems' popular student information system (SIS). I'm joined by Pearson's Paul Smith and by Sharlene Karbowski, an IT administrator with the Westside school district in Omaha, Nebraska, where PowerSchool has been running since 2001.

If you're a teacher, student, or parent in a school system that uses PowerSchool, or another SIS, you already know how these systems work and why they're valuable. But if your school system hasn't yet deployed an SIS, this screencast will serve as a useful introduction to the genre.

By way of full disclosure, my own school system is in the process of deploying PowerSchool but has postponed a planned rollout of the portals that enable teachers, parents, and students to access the system by way of the web. So for me, this screencast was opportunity to explore both the risks and the benefits of opening up those portals.

Comments

By far the biggest conversation starter here, since I switched on comments a few months ago, has been Why can't Johnny download? In that item, I presented a little download assistant that can be used standalone or in combination with a target URL. It aims to provide a single context for two different uses of an URL:

1. To load a resource into your browser
2. To save a resource to your local disk

Although I talked in the original item about transmission of data by value (e.g., emailing an attachment) versus by reference (e.g., emailing an URL), the ensuing conversation helped me clarify that, while the distinction between those two modes is indeed a perennial source of confusion, my helper isn't really intended for end-to-end transfer. It's just tries to make the essential load/save duality of a hyperlink, which I take for granted, more understandable to folks who don't.

For example, I'm sometimes asked to "send" someone one of the various audio or video files I've posted to the web. Similarly my wife, who is an artist, is often asked to "send" someone one of the image files that she's posted to the web. To me, the obvious and natural thing to do in these cases is email an URL. But, particularly in my wife's case, the recipients are often unsatisifed by this response. She has more than once had to download one of these files, and then email it as an attachment, in order to satisfy the expectations of the recipient.

I think there are two (related) reasons why this happens. First, the procedure to open or save an URL transmitted in the body of an email message is nothing like the procedure to open or save an attachment. Second, the load/save duality of hyperlinks is just not well understood, period. What I'm looking for is a way to contextualize and document the open-or-save-via-URL procedure.

Let's look at an URL in the context of a web page:

http://weblog.infoworld.com/udell/screenroom/dabble.mov

I take it for granted that I can a) click to load, or b) right-click to save. But this opportunity for dual use is not widely appreciated, partly because the semantics of right-click-to-save differ among browsers. With this contextualization of the target URL I'm trying to show that there are two things you can do with this URL, as well as explain how to do the second and less-obvious thing.

Of course it's hard (maybe impossible) to solve any problem in this domain without creating a new one. In this case, as several folks rightly pointed out, the contextualizing URL is an unwieldy thing that email clients are likely to mangle. Shortening it is an obvious solution which, in turn, creates yet another new problem.

A subtler problem is the very thing I've stated as a goal: to clarify the opportunity for dual use. Choice itself is sometimes the enemy. In a context where the intended use of an URL is to save the resource, not load it, should the latter choice simply be hidden? That's the purpose of a server-transmitted HTTP header like:

Content-Disposition: attachment; filename="dabble.mov"

When you request a download from, say, SourceForge, or YouSendIt, or many other services that are explicitly for downloading rather than loading/viewing resources, they use such a header to raise a Save As dialog box in your browser.

And yet...these services are rarely confident of their ability to hide the dual-use nature of the URL. Hence the familiar idiom:

If your download does not begin shortly, you can use this direct link.

Which link is, of course, the very thing whose dual-use nature was meant to be hidden.

There are valid arguments on all sides of this issue. The lively discussion attached to the earlier entry shows that it's far from settled, and not likely to be settled anytime soon.

It's interesting to speculate on the role of language here. Suppose that instead of this:

We'd always had this:

...where Action was any one of Save Target As or Save Link As or Download Linked File, but always the same one everywhere. We'll never know, and it's very unlikely that we'll ever be able to synchronize the terminology now, but this semantic muddle is clearly an ongoing problem.

Comments

Imagine a future version of Amazon.com where the price for each product is reported in two different ways: as dollars (P1), and also as carbon-adjusted dollars (P2). Now consider a pair of competing products, A and B, under two different scenarios. In one scenario, A's P2 is lower than B's, but A's P1 is higher than B's. Some people will be willing to pay the higher P1 (i.e., more dollars) to reward A's lower P2 (i.e., less environmental impact), but most won't.

In the other scenario, however, A's P2 is still lower than B's, but its P1 is about the same. In other words, there's no penalty to the buyer for rewarding A's lower environmental impact. If the P2 data are available, it's a rational choice. [Full story at InfoWorld.com]

Several readers wrote to dispute the assumption that global warming is a real threat, and that one useful response would be to restructure manufacturing operations and supply chains in ways that reduce the amount of carbon we're pumping into the atmosphere. Fair enough. In fact, I should have broadened the argument because it does not depend on that assumption.

We can all agree, for example, on the need to reduce the economic and geopolitical instabilities resulting from our dependence on oil. From that perspective, the alternative pricing mechanism I'm proposing would be a way to reward suppliers who organize their supply chains in ways that reduce that dependency. Supermarkets, for example, use feel-good labels like "locally-grown," but they don't quantify what that means in terms of transportation miles saved. Reporting that number would help me evaluate the environmental benefit associated with paying more for local strawberries. More subtly, in the case of similarly-priced items -- say, strawberries from California versus Rhode Island -- it would enable me to prefer non-local sources that are relatively closer to home.

But is such measurement even possible? Brian Bartlett writes (email quoted with permission) thinks not:

Yes, we do call pollution, among many other things, externalities, but that does not mean we have enough data to predict the exact amount of an externality in, for instance, a supply chain let alone quantify them by dollar value. I came to this conclusion while examining the EuP regulation proposed for the European Union and what is required is statistically impossible to reliably measure or predict. Simply consider the case where one manufacturer in the supply chain for an item relies on two or more other suppliers. One has to know, at all times, exactly their contributions to the final product including transportation externalities for each to quantify the overall externality cost. I have yet to see industry ever accomplish that reliably. Not even the US military can accomplish it and they have tried. Another externality, transportation of product from warehouse to customer, be it business to business or business to customer, will vary radically by location, time of year, etc. Again, you can at best predict after the fact (hindsight is ever 20/20).

This is the same type of problem in information theory as to why a centralized, command economy will not work. There is always lag in measurements and statistical past does not always reliably predict statistical future, although it may help, somewhat. Models, especially predictive models, are a field I have been working in for over thirty years and both your presentation here, and EuP, are unworkable in the real world.

The European Union's forthcoming EuP (energy-using product) mandate was, by the way, characterized as a "coming regulatory storm" in a recent pair of columns by my colleague Ephraim Schwartz. Read it and weep, says Ephraim. So I did read, and did weep. No, that's not what I have in mind at all. So far as I can see, sticking a flowery eco-label on a product is about like sticking a "locally-grown" label on it. I may feel vaguely virtuous, but I have no additional data to inform my purchasing decision.

Some systems already report this kind of data. You can, for example, easily discover the route traveled by any FedEx package. That was an amazing feat when FedEx first accomplished it, and it's hardly routine even today. But as SOA goes mainstream it's something that gets easier for more companies to do.

The key point here is that nobody requires FedEx to report that data. It does so in order to gain competitive advantage. In a world where people are more conscious of environmental factors, and more likely to include them in purchasing decisions, reporting the numbers behind the feel-good label becomes a competitive strategy.

Comments

In a column last week, I wished for a simple screen-sharing solution that would Embrace Constraints and Do One Thing Well. Apparently I'm not the only one who feels those needs. Many folks responded with suggestions, both on my blog and in email.

The names of a few of the usual suspects came up: Acrobat Connect (formerly Macromedia Breeze), and GotoMeeting. But when I reviewed my correspondence, I counted no fewer than nine dark horses that I'd never heard of:

Beam2Present
BeamYourScreen
CrossLoop
GatherPlace
Glance
LiveLook
LogMeIn
ScreenShare
Yugma

Although I can scarcely begin to evaluate all these, there may be some value in reporting the list to those of you who might want to. Based on the length of this list, as well as the anecdotal feedback I've received, it seems fair to conclude that the attractor for all this entrepeneurial energy is a need that the incumbents still aren't satisfying.

Finally, a mea culpa. In the column I said:

I've used every screen-sharing system...

Comments

Most people, to this day, prefer to convey files using email attachments rather than URLs. Over the years I've tried, and mostly failed, to explain why and how to use the URL-oriented approach. Programmers do this naturally, because programmers know the difference between transmission of data by value and transmission of data by reference. But almost nobody else does.

I've long understood that part of the problem was that most people didn't have easy access to web storage, and weren't in a position to upload a file whose URL could then be transmitted as a convenient proxy for the file itself. But that's changing. Lately I'm seeing lots of people who can upload to web storage, using various methods that are easy for them to understand and apply. What's still not easy for most people to understand and apply, though, is the method of using an URL to accomplish file transfer.

Here's a typical scenario. In order to transmit a 50-megabyte image, audio, or video file, I upload it to the web and send somebody the URL -- e.g., http://myserver.com/50Meg.tiff. In the recipient's mail client, that URL will probably be rendered as clickable, but that's only appropriate for loading the referenced resource into a browser. It's not very useful for downloading the URL.

Why not? The method for downloading an URL is just not well understood by most people. Email clients provide context for understanding how to extract an attachment, but they provide none for understanding how to download an URL.

So, I've created a contextualizer, or helper, in the form of a little single-page application. Let's say I want to transmit, to you, the 50-megabyte video file dabble.flv, which exists on the web at http://weblog.infoworld.com/udell/screenroom/dabble.flv. I combine the URL of the helper with the URL of the target file to create a context for downloading it. Try it and see what you think.

(The helper can also be used standalone -- that is, not combined with a target URL. In that case, the person being helped would have to paste in the URL.)

I hope that, next time I'm trying to transmit a big file by reference, or trying to enable someone else to do so, this will help break down the conceptual and procedural barriers that tend to get in the way. Please let me know if it works for you in those situations, and more importantly, if it doesn't work, why not. Likewise, please let me know if there's a version of this idea that's already successful in common use. Seems like such a thing should exist, but lots of simple things that should exist don't.

Comments

Joining me for today's podcast is Rajiv Gupta, CEO of Securent. His new company, which has been operating in stealth mode for a couple of years and just announced itself today, is focused on the thorny problem of fine-grained access control. In this conversation we discuss the role of XACML, the Extensible Access Control Markup Language, we talk about how to wrap or intercept legacy security policies in order to hoist them out of application logic and place them in the network where they belong, and we explore the relationship between fine-grained security which focuses on individual resources, and coarse-grained security which deals with users and roles.

Comments

Most web presentations of data are designed for passive viewing, not active analysis. For an example of what things could and should be like, check out episode 10 of The Screening Room. At the six-minute mark in that screencast about Dabble DB, a web database, Smallthought's Avi Bryant -- who is analyzing a set of data about investments -- wants to look at investments by U.S. state as a function of population. The current dataset includes states but not their populations. To add population data, Avi visits a website that lists states and populations, activates a JavaScript bookmarklet, and imports two columns from the HTML table on that web page. [Full story at InfoWorld.com]

In the same column, I referred to Doug Purdy's adventure in acquiring and using a list of Fortune 500 company names, and my web reformulation of an Edward Tufte data graphic. It's all compelling stuff, but there's something about Avi Bryant's maneuver that shouts Knowledge Navigator.

In this 90-second excerpt from the screencast, you can see how Avi hoists data right out of a web page and weaves it into a structured view.

So natural, so powerful. How can we not have this capability always and everywhere?

Comments

Thanks to Denise Getts, a librarian assistant at the Tucker Free Library in Henniker, NH, there is now a LibraryLookup bookmarklet for Tucker's SageBrush InfoCentre catalog system. With the addition of SageBrush, there are now 20 different types of catalog systems supported by the bookmarklet generator.

When Denise first wrote to me about this, I told her that I couldn't suss out the URL-line query that would work for the SageBrush OPAC, so I suggested that she ask the vendor for help. The company responded, and now a problem has been solved not only for the patrons of the Tucker Free Library but for some unknowably larger set of SageBrush implementations in libraries around the world. Although Denise may not (yet) realize it, what she has done here is a perfect example of Wiring the Web.

Update: I just noticed on the Tucker Library's website that there's a program called New Hampshire downloadable audio books. Excellent! And my library participates. Most excellent!

I've said to several folks in the past week that it's a challenge to be aware of all of the new services coming online, and that I have to remind myself to visit sites I'm involved with in order to see what new services may have appeared.

Does this sound familiar? It should, it's another version of the RSS story. Rather than visiting a bunch of websites to discover what new services they've added, I should be subscribed to a "new services" feed at each site, and flow those into an aggregate "new services" feed.

I've recently discussed with a couple of folks how RSS, for all its momentum, has yet to cross the chasm. This example suggests another angle of attack. If I'm slapping my forehead for not knowing that the New Hampshire libraries have downloadable audio books, you can bet a bunch of people are in the same boat. We all hate missing out on a good deal, and we also hate scattershot email notifications. An aggregated "new services" feed neatly solves both those problems.

Comments

A public radio producer called me the other day to discuss his idea for a story on the Internet's role in the recent US midterm election. The hook? He'd heard that Internet use on election day reached levels not seen since 9/11. That didn't ring true for me, though I did find one report that Internet Broadcasting, a publisher of TV-station websites, had its biggest day ever. As it turned out, the producer had based his idea on that same report.

Well, if the total amount of Internet use wouldn't be the story hook, I counter-proposed, perhaps the evolution in styles of Net use could be. Internet Broadcasting's banner day, for example, was Wolf Blitzer's Waterloo. The poor guy looked pathetic on CNN, pointing to computer-generated graphics and reading out the numbers. Who wouldn't want interactive graphics, more numbers, and above all on-demand access to our own state and local results? For years, it's true, people have been going online for that experience. But this election may have nailed TV's coffin in terms of data delivery.

Then there was the dizzying interplay among mainstream media, blogs, and online video on such issues as poll-watcher intimidation and e-voting glitches. In one case, amateur video flowed "upstream" when the New York Times mentioned a RedState.com item about a YouTube video that claimed to document intimidation of poll watchers in Philadelphia. In another case, pro video flowed "downstream" when BradBlog posted a segment from Lou Dobbs Tonight on e-voting problems.

In the end there was no "upstream" or "downstream", just a swirl of currents. Let's hop in the kayak and take a tour.

The Wall Street Journal sounded this theme:

When Americans go to vote tomorrow, a new breed of activist will be on guard, monitoring polling stations for everything from voting-machine glitches to long lines to registration snafus. [WSJ.com]

Michelle Malkin spun it thusly:

Behind the civic-minded facade are far Left radicals whose main concern is not in ensuring a fair election process--but in preemptively undermining and delegitimizing it.
...
Like I said yesterday, bring a camera if you can. [Michelle Malkin]

To help us answer questions about the use of cameras in polling places, she usefully cited the Center for Citizen Media's Election Day FAQ. In a followup, the Center noted:

When we asked for your questions, we never expected that 80% would be about taking photographs or videos at the polls. [Center for Citizen Media]

What did the followup guidance say about the Pennsylvania rules that applied to the poll-watching video-blogger in Philadelphia? Unfortunately, nothing conclusive.

The New York Times, meanwhile, twice mentioned the fact that RedState.com text-blogged the Philly video-blogger. First on Wednesday:

Erick Erickson, RedState's chief blogger, also included a report of poll watcher intimidation in Philadelphia, along with a link to a video on YouTube that appeared to show a certified poll observer (armed with a video camera) being blocked from a polling station. [New York Times]

(Oddly, the Times devoted almost as much space to its coverage of RedState.com as RedState.com did to either of its brief items on the incident. Even more oddly, neither of those seems to include the link to the video.)

Then today:

Right-leaning RedState.com reports alleged intimidation of poll-watchers in Philadelphia... [New York Times]

The comment thread attached to one of those RedState.com items included this exchange:

Q: What is your source for this? Philly.com is not reporting anything like this.

A: http://cbs3.com/local/local_story_311120726.html [RedState.com]

And here is the story cited by the commenter:

Officials said approximately a dozen claims were filed stating they were being interfered with as they entered the D and Clearfield polling place in Kensington.

In the 19th Ward, several complaints were filed regarding voters being told who to vote for.

The District Attorney's office and Philadelphia Police are looking into the accusations.

"We sent our District Attorney's detectives, the police department was also called and Federal, F.B.I. agents were called out as well to make sure that whatever was going on in East Division stop and that no voter was intimidated," said Philadelphia District Attorney Lynn Abraham. [CBS 3]

Whew. We really have stepped through the looking glass. It is strange (and disheartening) to see that the New York Times didn't particularly care whether there actually was poll-watcher (or voter) intimidation in Philly, but was fascinated that a "right-leaning" blog reported "alleged" intimidation, and that the allegation took the form of a YouTube video.

It is strange (and exhilarating) to realize that, with a bit of web search and navigation, we can so easily triangulate on that intimidation flap from a dozen different perspectives. We don't need to depend on the Times to deliver that story any more than we need to depend on Wolf Blitzer to recite our congressional results.

It may take another election or two before the strangeness wears off, and all this seems familiar and unremarkable. The sooner the better because, while it's true that we enjoy powerful new access to information about our political system, it's also true that we've barely scratched the surface.

Here's a crazy idea. I checked www.phila.gov to see what Philadelphia District Attorney Lynn Abraham's team concluded about alleged intimidation. Of course I found nothing there about that investigation, or indeed any investigation. But why not? Why should we depend on reporters and bloggers to dredge up this information? It's public information; we fund the activities that produce it; we should expect to get it through the web and directly from the source.

Last summer I wrote about Washington DC's extraordinary experiment in digital democracy and transparent government. So far as I can tell, that story is still a sleeper, both inside and outside the Beltway, for the old media and the new. I'm waiting for everyone to wake up and notice. After that, I'll be waiting for everyone to stop noticing and take it for granted. That'll be a great non-story.

Comments

Jim Russell is a geographer, social theorist, and would-be social entrepeneur who blogs at Burgh Diaspora. In today's podcast we discuss his analysis of ways to organize Pittsburgh's diaspora -- the informal network of ex-Pittsburghers scattered around the world. Why might that matter? Jim's thesis, which I find fascinating, is that the "distance trust" embodied in that kind of network can have important social and economic effects not only for Pittsburgh, but for local and regional populations everywhere in this era of pervasive mobility and telecommunications.

A key point of reference for our conversation is Richard Florida's notion of the geography of the creative class. Florida wrote a book on the subject, and ITConversations has a podcast of his talk at PopTech 2004.

Comments

Like the Car Talk Puzzler, the Friday podcast needs to take a break, now and then, to recharge its batteries. It will return next week. Meanwhile, if you're looking for some audio brain food and haven't yet tuned into The Long Now Foundation's monthly Seminars On Long-term Thinking, I highly recommend them. So far I've listened to about half the series and, while they're all worthwhile, Stephen Lansing's Perfect Order: A Thousand Years in Bali has made the biggest impression on me. It's about politics, religion, complexity, cellular automata, irrigation, pest control, sustainable agriculture, social networks, technology transfer, self-organizing systems, and -- of course, since it's a Long Now talk -- time.

There's one thing I wish screensharing systems would do well: screensharing. I watch a lot of demos projected to my computer. It's always a struggle, both for the presenter and for me. Windows or Mac? IE or Firefox? Who has the latest version of the client? Who's the host? Which application is shared? Can you see my screen?

While we answer these questions, the first five or ten minutes of every meeting swirl down the drain. I've used every screensharing system and, from this perspective, they're roughly the same. None performs its basic function simply and well. All are determined to add whiteboards, chat, and filing systems. In principle these are useful features. In practice, for most people most of the time, they're just not usable. [Full story at InfoWorld.com]

Several readers have pointed out, in email, that Glance delivers the "do one thing well" experience. Having just tried it, I agree. The VNC-based projector is for Windows-only, but anyone can connect using VNC's Java viewer. It's nicely done. For companies doing lots of demos, the $120/month corporate rate is reasonable. At $50/month, though, the personal rate is too pricey for occasional ad-hoc use.

Other readers ask: Why not just use VNC directly? I do, all the time, but most civilians don't control their own firewalls and so VNC is a non-starter for them. Which leads me to a question of my own: Are there free or less-expensive solutions for firewalled VNC projectors?

Update: Glance founder/CEO Rich Baker reports:

Our initial version used VNC's engine inside a wrapper. We built a proxy that allowed the engine to connect using our call model, added tunneling, multipoint, etc., and eventually changed out the VNC engine as well. So today's Glance service runs entirely on our own code base.

Glance sessions still start using TCP on VNC's 5500 port. Failing that, it automatically tunnels HTTP to port 80. If a guest on a PC doesn't have Java (or has a version of Java that has a nasty bug or is slow (Microsoft's old JVM) or is damaged in some way), Glance automatically pushes a tiny ActiveX viewer or an executable viewer. This allows us to connect to nearly any PC. Macs come with a great version of Java. Likewise, most Linux folks have a Java-enabled browser. So we can connect to nearly anyone.

Our goal is to keep the UI spare and the footprint light. We are adding some capabilities in the near future, hopefully without compromising simplicity.

Comments

On Monday I visited Nuance for an update on the company's speech recognition products and initiatives. Two years ago, my screencast on Dragon NaturallySpeaking 8 demonstrated what was then the state of the art in automatic dictation. Dragon has for years been asymptotically approaching the point at which dictation becomes routine and general-purpose. For most of us, it hasn't yet reached that point. I didn't upgrade to the latest version 9 because, despite improvements, I didn't think it would yet cross my threshold for routine use. Nuance's demo of Dragon 9 confirmed that hunch.

Peter Mahoney, Nuance's marketing VP, showed me how he uses Dragon 9 for dictation. When he read a prepared statement, the results were perfect. Then I handed him a copy of Newsweek and asked him to read from a random article. The results were still very good. True, the Arabic names in the story had to be spelled out. But that wouldn't be the case if those names were common in your domain of discourse. And training Dragon to absorb specialized vocabulary is both easy and effective.

The real problem, at least for me, lies elsewhere. And the test I gave Peter yielded a stunning example of it. At one point he read:

...it's rarely so simple...

Dragon wrote:

...it's really so simple...

Because Dragon works so hard to produce plausible results, this class of error resists casual proofreading. In this case, you would have to read very carefully to notice that Dragon had reversed the intended meaning of the sentence. For me, anyway, the cost of finding and fixing these kinds of subtle errors outweighs the benefit of routine dictation, at least when a keyboard is available.

Keyboards aren't always available, though, and that fact made the second part of the demo a real eye-opener. Check out this 55-second video of Peter dictating to his Treo:

In case you can't play this video, it shows two examples of speech recognition. First Peter dictates a brief memo, and uses his voice to change "LaGuardia" to "Logan". Then he speaks the query "Eastern equine encephalitis" to Google and reviews the results. Very cool!

How do you shoehorn Dragon onto a mobile gadget? You don't. There's only a small client that relays recorded audio to a server and receives recognized text. This kind of mobile dictation should be available as a carrier-provided service, for the popular handheld operating systems, sometime next year. I'll be curious to see who uses it, and how.

In our follow-on discussion we talked about how Nuance's software is being used in the automotive realm. Cars themselves offer a growing range of voice-controllable functions: temperature, navigation. Passengers' Bluetooth-equipped gadgets paired to cars' audio I/O systems are another emerging domain for voice control.

What about those us who drive older cars and use older cellphones? I think there's still all kinds of untapped opportunity. For example, while driving I'd love to be able to speak questions like these and hear the answers:

How many new emails from Jill in the last 4 hours?

What are the subject headers?

Can you read the message entitled "New panelist for your session"?

Given the kind of client/server architecture that Nuance has developed, even my lowly LG VX4400 should be able to handle a protocol like this. The magic would all be in cloud, where the speech recognizer and my mail server would consummate a service-oriented marriage.

Comments

The October episode of The Screening Room features Dabble DB, a web-based workgroup database that, in the style of 37Signals, focuses on simplicity and embraces constraints. Dabble doesn't aim to do full-blown database application development, or sophisticated query, or heavy transactions. Its mission, instead, is to enable teams to easily manage and flexibly evolve modest (say, 30- to 50-megabyte) quantities of structured data.

As underscored by today's announcement of Google's JotSpot acquisition, there's a growing interest in flexible, user-friendly, web-based data management for teams. An instance of something like Dabble, parked somewhere in the cloud, can be a compelling alternative to that tired old method of data-oriented collaboration, namely emailing Excel attachments. And the trend toward commodity server virtualization means that those instances can scale the way they need to: out (i.e., more) rather than up (i.e., bigger).

Comments

Reading Edward Tufte's latest opus, Beautiful Evidence, I stopped on page 176 to consider his redesign of a table of data about cancer survival rates, shown (in part) here. As you can see, it's a stack of sparklines, each decorated with data labels. In Tufte's online forum you can find the original table, the redesign, and an assortment of PowerPoint manglings of the data.

As is often the case when reading Tufte, I asked myself two questions:

1. How might this redesign work more interactively on the web?
2. How can more people be empowered to do such redesigns, for print and for the web?

One answer to the first question is this web interpretation of Tufte's redesign (code). Here I've borrowed part of Joe Gregorio's sparklines kit and used it to generate one data graphic (e.g. Prostate.png) for each disease. I've also wrapped these images in a linked set of four HTML files, sorted by 5-, 10-, 15-, and 20-year survival rates.

This version sacrifices some of the typographic elegance of the original in exchange for some of the benefits of the web: interactivity (you can review different sortings), data availability (the data are included in the HTML files).

The second question is much, much harder. There's nothing earthshaking about what I've done here. But most people won't attempt any data visualization that isn't supported in the standard chart kits. For starters, there's a conceptual obstacle. You've got to have the idea in the first place. I got the idea from Tufte, but Tufte's brain doesn't scale very well. I can't directly apply it to novel situations.

Then there's a logistical hurdle. You've got to be able to implement the idea. In my case, I was able to:

1. Understand that I could leverage Joe Gregorio's sparkline kit.
2. Acquire the kit, along with its supporting software.
3. Learn enough about the kit and its infrastructure to generate individual data images.
4. Combine those data images, using HTML and CSS, to achieve the effect I wanted.

Of course my brain doesn't scale either. This posting might give you a useful idea, but you can't apply my brain to your novel situations any more than I can apply Tufte's to mine.

Can web collaboration address these scaling problems? Maybe. We have lots of good of ingredients: social networks for images, code, and documentation. Time to get cooking!

Comments

John Schneider, the CTO of AgileDelta and the driving force behind E4X, is now evangelizing Efficient XML, an alternate binary syntax for XML. In today's podcast we discuss the motivations for this proposed W3C standard, its theoretical foundations, and its uses.

There have been a various binary XML proposals in recent years. It's a controversial idea, and I haven't paid close attention to the discussion, but at a high level John's pitch sounds reasonable to me. If an alternate binary representation of XML can be managed under the covers, side-by-side with conventional XML in a way that's transparent to existing XML-aware tools and APIs, then a myriad of memory- or bandwidth-constrained devices and applications can leverage those familiar tools and APIs.

Comments

Why isn't videoconferencing more compelling? When we say we want to look the other person in the eye, what we really want to do is read the microexpressions of the face. As Malcolm Gladwell points out in Blink, people adept at reading faces can literally read minds. And at a sufficient frame rate the visual channel can transmit those microexpressions. [Full story at InfoWorld.com]

Here's a 20-second snippet from Robert's video to give you a feel for how the Cisco system works:

At the crappy level of video quality shown here, or even at the level shown in Robert's original H.264 QuickTime clip, we can't judge the effect of Cisco's high definition video. We can, however, see how the layout of screens and desks creates a powerful illusion of circular seating. Cisco's astronomical price notwithstanding, though, the need for all participants to visit these specially-designed rooms limits this approach to special circumstances.

What could make ordinary iChat A/V or Skype videoconferencing more emotionally effective? It's natural to assume that high-definition video is a make-or-break requirement. But if what really matters is face-reading, is the ability to count the number of whiskers in a five-o'clock shadow as critical as we imagine?

Here's the critical passage from Malcolm Gladwell's 1992 New Yorker article, The Naked Face, which became the face-reading chapter of Blink:

Perhaps the most famous involuntary expression is what Ekman has dubbed the Duchenne smile, in honor of the nineteenth-century French neurologist Guillaume Duchenne, who first attempted to document the workings of the muscles of the face with the camera. If I ask you to smile, you'll flex your zygomatic major. By contrast, if you smile spontaneously, in the presence of genuine emotion, you'll not only flex your zygomatic but also tighten the orbicularis oculi, pars orbitalis, which is the muscle that encircles the eye. It is almost impossible to tighten the orbicularis oculi, pars lateralis, on demand, and it is equally difficult to stop it from tightening when we smile at something genuinely pleasurable. This kind of smile "does not obey the will," Duchenne wrote. "Its absence unmasks the false friend." When we experience a basic emotion, a corresponding message is automatically sent to the muscles of the face. That message may linger on the face for just a fraction of a second, or be detectable only if you attached electrical sensors to the face, but it's always there. [Malcolm Gladwell: The Naked Face]

The protagonist in this piece, psychologist Paul Ekman, mapped out his Facial Action Coding System back in the 1960s, using videotapes. He clearly didn't watch those tapes in high definition. He did, however, watch them at 30 frames per second.

What's the minimum framerate for face-reading? Is it possible that the 15fps typical of web-style video doesn't capture fleeting microexpressions but that 30fps does? If we traded resolution for framerate might low-end videoconferencing cross a threshold of effectiveness? I'd love to know if this experiment has been done, and if so what its outcome was.

Comments

Writing on the coComment team blog, Stephanie Booth addresses the thorny question of comment ownership:

Do we consider it a problem that the commenter doesn't retain control over the comments he leaves on other people's blogs? For example, it has always bothered me that value-added comments of mine, scattered all over the blogosphere, could disappear any day at a whim of the blog owner. [coComment blog: Who owns your comments?]

Stephanie's item came to my attention because it's part of a blog-to-blog conversation that touches on items I've written before. Although she responded briefly to this item in a comment, which is trackable thanks to coComment, she used the coComment team blog to present the larger idea that it inspired. In that context she wasn't reacting to a single blog, but to a conversational cluster involving, among others, Paul Sergeant, Jerry Slezak, Ben Metcalfe, and me.

From my perspective as a blog author evaluating the reactions to my work, there are three ways to receive feedback. All are comparable in terms of the immediacy and completeness of that feedback.

First, del.icio.us shows me who's bookmarked my items, when, and with what tags and commentary.

Second, crawlers like Technorati and Google blog search round up the blog-to-blog conversations that touch my blog.

Third, my comment system offers another mode of feedback. If we grant that all three modes are forms of conversation, then it may be helpful to try to visualize things from the perspective of speakers and listeners.

As a speaker, I see this kind of step function:

ownership and
control of speech   -----------+                     +----------------------    (high)
                               |                     | +--------------------         
                               |                     | |        
                               | +-------------------|-+                           ^
                               | |                   |                             |
                               | |                   |          
effort and                     +---------------------+                               
potential impact   --------------+                                               (low)
                     
                   bookmarking -> in-blog commenting -> blog-to-blog conversation

When choosing where I want to be on the X axis, I think about ownership/control as well as effort/impact. When both matter, I choose the heavyweight option and write on my own blog. Bookmarking optimizes for low effort, which also means low impact, but sacrifices no control. The really interesting tradeoff is in the middle zone of in-blog commenting. Usually but not always, speech in that zone requires an intermediate amount of effort and can have an intermediate impact. That's useful in some circumstances, but there's no getting around the extreme sacrifice of ownership and control.

Now as a listener, without the help of something like coComment, I see this:

ability to hear     -----------+                     +----------------------    (high)
                               |                     | 
                               |                     |                             ^
                               |                     |                             |
                               +---------------------+                               
                                                                                 (low)
 
                   bookmarking -> in-blog commenting -> blog-to-blog conversation

On either end of the spectrum I can hear pretty well. In the middle I often can't. It's true that some comment systems (including the one I'm using here, WordPress) offer per-conversation RSS feeds, but it's a lot of overhead to actually use them.

When I listen with the help of coComment, that dip flattens out and I can hear the middle zone much better. But, while it's true that I might want to use the coComment RSS feeds to archive what's said in that middle zone, I don't own and can't control the speech that I choose to utter there.

How much do ownership and control matter? It depends. Maybe not much if an in-blog comment thread is a Wiki-like exercise in collaborative sense-making. Maybe a lot if the commentary is more debate-like, or if contributions tend toward the high end of the effort/impact continuum. In either case, you want to be able to hear as well as you possibly can.

Comments

I'm starting to see more people catching on to the idea of embedding small screencasts directly into web pages. In this blog entry, for example, Pascal van Hecke uses the technique to illustrate a nice recipe for organizing what he calls loose MP3s -- that is, MP3 files that are linked from web pages you visit. The first half of the recipe involves a simple Greasemonkey script that rewrites the page you're on to include the same inline MP3 player that's used on del.icio.us pages like del.icio.us/tag/system:media:audio. To see the player in action, click one of the right-pointing blue arrows on that page.

The second half of the recipe involves using the tag this option that appears when you activate the player. As Pascal's screencast illustrates, del.icio.us will generate a podcast-style feed (i.e, an RSS feed with enclosures) for URLs that point to MP3s. His method, which I'm now using as well, is to preview an MP3 file using the inline player and, if it's interesting, bookmark it with the tag tolisten. Then you can use a podcatcher to subscribe to the generated feed, e.g.:

del.icio.us/judell/system:media:audio+tolisten

By the way, although Pascal's screencast suggests that you have to assign tags like system:media:audio or system:media:mp3 to make this work, it appears that del.icio.us will just infer those tags if the URL ends with an MP3 filename.

I found another nice example of embedded screencasting on the CoComment start page. It's just a little animated GIF that shows Firefox users how to drag the button that represents the bookmarklet to the toolbar. Here's one frame¹ from the animation:

You can see the animated version here.

The more complex procedure for IE users isn't illustrated in this way, but if it were, that would be more focused and effective for this purpose than my standalone Bookmarklets 101 tutorial.

Edward Tufte's mission statement for sparklines is: intense, simple, word-size graphics. In a similar vein, there's an emerging species of embedded screencasts for which the mission statement might be: intense, simple, active demonstrations.

¹ Note to my future self, and others needing to learn (or remember) how to extract one frame from an animated GIF using ImageMagick:

convert animated.gif[0] frame0.gif

Comments

Cricket Liu joins me for today's podcast. He's known for his popular books on DNS infrastructure, and as VP of Architecture at Infoblox has recently led an effort to identify and correct DNS security vulnerabilities.

In this conversation we discussed that work, and the related tool called Cricket Liu's DNS Advisor. But I began by asking him why it might or might not be a good idea to extend DNS in ways that identify humans rather than machines.

Comments

Writing in this week's InfoWorld, Ephraim Schwartz says:

Google and its competitors are fighting for market share because, now, market share in and of itself means success. From now on, "the next big thing" will not mean great technology; it will mean whichever online entity can come up with the most "viewers."

If that means the content is at the bottom of the intelligence barrel, you won't hear investors complaining and you will see a lot of copycats. But what you won't see are inventive twenty-somethings putting their skills toward coming up with innovative technology to change our lives.

IWM: Are you offering staffers any incentives to blog?

Quittner: We're doing something that is novel for Time Inc. Our bloggers will be directly remunerated on the basis of their traffic. They'll be paid a modest CPM. Time Inc. will sell advertising on the individual blogs. So the bloggers will get to participate in the revenue they generate. [I Want Media: Josh Quittner: 'Everybody Wants to Be a Blogger']

Sounds great, right? Yes and no. The meritocratic impulse is terrific. But when dollars are attached only to the quantity of readers, listeners, and viewers, with no consideration given to their quality, the outcomes are predictable. Sensationalism will trump reason. "Content producers" will lie awake at night inventing new forms of astroturfing. We will race to the bottom.

The excruciating irony is that we are now, for the first time, in a position to do what media have always aspired to do. We can create durable relationships with subscribers (remember them?); we can progressively find out about our readers, listeners, and viewers; we can describe them to advertisers in ways that will support better "content" and better advertising. But that would require innovation which, as Ephraim quite rightly points out, is drowning in the rising tide.

Comments

The blogosphere has been pretty quiet about yesterday's first official podcast of This American Life. My guess is that nobody wants to jinx this long-awaited and happy state of affairs.

In early June, I was one of a number of folks who noticed that the show's website had switched audio formats from Real to MP3, and who interpreted that move as a tacit endorsement of do-it-yourself podcatching. On June 20, I published and then retracted a takedown request from TAL's web wrangler, Elizabeth Meister. In the flurry of ensuing conversation, here are some quotes that stand out:

Jared Benedict:

Contrary to posts on Boing Boing and elsewhere, Jon Udell and I did not receive a "nastygram" or formal ceast and desist letter. Rather we received friendly emails from Ms. Meister, This American Life's webmaster, making a request to take down the hyperlinks and RSS feeds, or she'd regrettably have to get lawyers involved.

While Ms. Meister did miss the mark by accusing us of copyright infringement without a clear understanding of what we were actually doing, or what copyright law allows, she was trying to be polite and friendly which I appreciate.

To be clear, I was not storing or making any copies of their work, I was simply providing links to publicly accessible MP3's hosted on This American Life's own servers. It is my position that hyperlinking to publicly accessible MP3's is perfectly legal (see Ticketmaster v. Tickets.com) and fundamental to the existence of the web.

While I am confident that I am breaking no law, I am respecting TAL wishes by taking down the podcast and archive page which points to their MP3's. This American Life has decided to take the bizarre approach to Digital Restrictions Management (DRM) by asking nicely... which I suppose is better than using some Windows only Microsoft Media Player DRM or Sony Rootkit DRM.

Nick Carr:

Now, Jon Udell is an honorable guy, and I'm sure he doesn't think of downloading those files as an act of thievery in any way, shape or form. But what kind of strange logic leads someone to say that "although the archive page at This American Life still says that you can't download files, it's not true anymore." That's like saying that if I go out to the supermarket and leave my front door unlocked, then it's ok to come into my house and steal my china. Just because something's not locked up doesn't mean you can help yourself to it.

David Berlind:

Context really doesn't matter. If the URL exists, you must acquit. Otherwise, if you're putting MP3 files on the Web and you don't want someone pointing to them from the contexts of their choice, then, instead of sending takedown notices to that someone, take down the content itself. That way, nobody will point to it.

A Typical Joe:

I'm a fan of This American Life; I never listen to it.

This was and remains a complex affair. Along with Jared Benedict and many others, I believe that use and recontextualization are fundamental to the web. But just because you can use and recontextualize doesn't mean you always should. So I withdrew a blog posting for the first and only time, and I regretted the grief I caused Elizabeth Meister and TAL.

But I went further than that. Like Typical Joe (quoted above), TAL became the favorite show that I never listened to. Many others did, on MP3 players, ignoring TAL's extraordinary "DRM by asking nicely" stance. But for me that option became as unappealing as Audible or iTunes downloads, or Real streams. So I've not heard TAL since June, and I'm greatly looking forward to Episode 203.

Comments

Today's screencast is a follow-on to last week's column and blog entry about how corporations are colonizing Second Life, and about what the real advantages of 3D simulation might or might not be. In that column I reacted to an upcoming Sun event which was held on October 10 (video clip). On that same day, I was invited to the Greater IBM Virtual Block Party.

Here's my 3-minute video report on IBM's event:

Both the column and this video report are a tad snarky, but I mean no disrespect either to Sun or to IBM. It's no mean feat to pull off virtual events like these, and hats off to both companies for their execution. As the long-anticipated wave of 3D technology finally begins to crest, though, I do want to provoke some discussion about gratuitous versus useful applications of the third dimension. In the IBM event, I found myself in a breakout session chatting with strangers about a topic whose premise I disagreed with. That would be unproductive enough in the real world. Because we lacked a synchronous voice channel, real identities, and sufficient emotional bandwidth, it felt even less productive here.

Being in-world doesn't mean anything, in and of itself, just like being on the web doesn't. And just as a lot of our intuitions about what to do on the web turned out to be not very helpful, so will a lot of our intuitions about what to do in 3D space.

While I was waiting for that screencast to render I ran into another example of the same kind of thing on the Google Earth Blog. The campaign against the coal industry's landscape-desecrating practice of mountaintop removal has used video, Flickr, and Google Earth to make its point. Of the three methods, I found Google Earth least compelling. In this brief clip, for example, the most prominent 3D artifacts are the Brobdingnagian flagpoles with half-mast flags stuck into Lilluputian mountains. It's what Edward Tufte would call chartjunk, executed in 3D.

Are there non-gratuitous uses of 3D for social interaction and data visualization? Of course. But we're going to have to work hard, really hard, to find them.

Comments

Mark Ericson, director of SOA product strategy for BlueNote Networks, joins me for this week's podcast on the subject of VoIP/SOA convergence. When Mark talks about communications-enabling business processes, he doesn't just mean processes talking to one another, but also processes communicating with people -- and coordinating the communication among people. We've dreamed about unified communications forever, and for the most part we're still dreaming, but we'll get there one of these days. When we do, we'll wonder how we ever got along without the service-oriented integration of voice and data that BlueNote envisions.

Towards the end of our conversation we discuss Skype, and Mark suggests an interesting rationale for eBay's acquisition of it. As more and more high-value transactions flow through eBay, the level of trust between buyer and seller becomes increasingly critical. If you can recruit voice or video channels in the context of a transaction, you can strengthen that trust relationship. It's a great explanation that I hadn't heard before, and that I think is spot on.

Comments

When Doug Kaye announced the availability of The Levelator, I hoped it would spare me the worst drudgery of podcast production. After recording and postprocessing my next Friday podcast this morning, I'm convinced that it will.

Here's the deal, from my perspective as an audio newbie now plunged into the deep end. As my podcasting method has evolved, I've settled into two modes of editing. In one mode, I refine the content of the recording. That involves fine-grained internal editing -- trimming out excessive ums, uhs, and pauses -- as well as coarse-grained edits that remove less interesting passages in order to focus on the most essential parts of the conversation. Applying both methods typically reduces the final product to somewhere between 70% and 90% of the original length and, in my opinion, sharpens the result in a way that's well worth the investment of time. I've always enjoyed this kind of editing in the textual realm, and it turns out that I enjoy it in the audio realm as well.

The other mode involves the purely technical work of taming sometimes-noisy phone lines and evening out audio levels. As I've become more sensitive to audio quality, I've found myself spending more and more time on the leveling process. It's not only needed to balance the caller and the callee. There can be a ton of loudness variation just within the caller's track. When you start fiddling with that, you're on a slippery slope that leads straight into a pit of drudgery.

In a talk given at the Podcast Academy, Daniel Steinberg noted that the normalization function in audio editors like Audacity and Audition "doesn't do what you think it does." I guess I'm not the only one who assumed, incorrectly, that normalization's job was even out loudness variation. But apparently it doesn't. It levels off peaks, but doesn't raise up valleys.

Levelator does do what many people expect normalization to do. It brings the valleys up close to the peaks. It's going to spare me a ton of the kind of editing drudgery that I'd outsource to an assistant if I could. And it's going to enable me to focus on doing the kind of editing that I wouldn't outsource even if I could. (Well, I'd outsource the fine-grained internal editing, but not the coarse-grained editing.)

The credit for this excellent hack goes to GigaVox engineers Bruce and Malcolm Sharpe. Here's part of the backstory from Doug's blog:

Bruce, with help from his son, Malcolm, had proven that he knew how to tackle these problems in ways that no one else anywhere in the audio/software industry has done to date. So I asked him, "Bruce, do you you think you can write a leveler that corrects for medium-term variations in loudness instead of the short-term and long-term variatons processed by compressor/limiters and normalizers, respectively?" Bruce and Malcolm took on the challenge, and eight months later we began testing The Levelator.

I've been a professional audio engineer longer than I've been in the computer industry -- that's a long time -- and believe me, there's nothing else like this out there. I guarantee it will blow you away or double your money back. (Oh wait, it's free. I forgot.) We previewed it for 100 people at Podcast Academy 4 today, and they were unanimously impressed. You will be, too.

Indeed I am. If, as Doug says, nothing like this has been done before, then why not? My guess is that it's one of those innovations that can only come from folks who straddle two worlds. There aren't many folks who can comfortably straddle software and audio, just as there aren't many who can comfortably straddle data and voice networking. That kind of dual citizenship is incredibly valuable.

Update: It's a small world indeed. Turns out that Bruce Sharpe was CTO at SoftQuad and now heads up the XMetal group at Blast Radius. Podcasting tools are just his hobby, he says. Here's his bio from the XML 2002 conference.

Comments

3D richness notwithstanding, Second Life is fundamentally social too. I can't wait to see what the business world will make of it, or of systems like it, once the PR novelty wears off. How about this for a practical application of simulation and role-play: an island where IT administrators and their clueless users trade places. Or where programmers and their business sponsors switch roles. That'd be edutaining. [Full story at InfoWorld]

3D simulation obviously helps you think about the location and movement of people and things. More subtly, the projection of self into avatar creates the possibility of simulated social interaction. I've noted before that John Lester's project, Brigadoon, helps people with Asperger's Syndrome become more fluent in their performance of social rituals, while practicing in a safe environment. So much of our interaction nowadays is disembodied: just voice, or just text. How effectively we'll be able to be embodied in avatars, and how those embodiments will change our ways of interacting for better and worse, are questions that we'll soon begin to answer.

Comments

A couple of weeks ago I began using WordPress as a loosely-coupled comment engine. This morning I wrote the glue code that fetches the comments collected there and displays them here. It was straightforward because WordPress supplies a general feed that tells me if any item has updated, and a per-item feed that gives me the comments for that item.

I've been proceeding slowly because I wanted to make sure that I liked using the services I'm acquiring from WordPress. They include:

• Full and per-item feeds
• An effective, easy-to-use spam filter
• An effective, easy-to-use moderation queue
• A comment form that accepts HTML judiciously, applying strong filtering

Now that I've satisfied myself that these services meet my needs, I'm stitching them in. But I'm still rather loosely coupled to WordPress. Comments are just written out to JavaScript files included here. Any process can write those files, and can assemble them from any comment engine that provides the necessary integration hooks.

I was hoping to also gain the services of the WYSIWYG editor that's used in the main WordPress writing interface. But I forgot that the comment engine doesn't use that editor. Which, in fact, helps to illustrate the point of this exercise. If another engine with competitive capabilities adds WYSIWYG comment writing, I'm in a position to switch to it.

Meanwhile, Tim Bray has been refining his new comment engine which, like his blog-publishing system, is a from-scratch effort. There are no value judgements to be made as between his energetic approach and my lazy one. Tim wanted comments, and used the opportunity to teach himself Ruby. I wanted comments, and used the opportunity to try an integration experiment. We both got the job done and learned useful things along the way.

Like Tim, I've gone years without comments, which raises the question: "Why now?" For him, the example of Jonathan Schwartz showed that the amount of interaction would manageable, and that the quality of it would be high. For me, there's a different reason. Blogging, at first, was highly interactive, yet in a loosely-coupled way that I found really compelling. It has surprised me to see how effectively the search and bookmarking services are able to assemble the virtual conversations formed when people link to other people. I came to believe that, ideally, people should publish their own words in their own online spaces, and syndicate them elsewhere as needed, rather than publish their own words directly into other people's spaces. And for a while, that seemed to work pretty well.

What changed? First, after the initial flush of blogging excitement wore off for many people, the level of cross-blog interactivity dropped off. Second, after the advent of TechMeme, things became less interactive -- or anyway, that's how it felt to me.

I've always valued interactivity, and hoped to achieve it in a syndicated way. But when that method stopped working as well as it had, I started to feel isolated, so I've gone back to a traditional comment system in order to reconnect.

Comments

As has become my custom, I used HTML Slidy to make the presentation I gave at the Paris SOA Forum last Thursday. I've argued that since web standards can support everything a presentation needs to do, the only obstacle is the perennial lack of decent web-oriented authoring tools. But my recent experiences remind me that there's another obstacle: we still have no standard compound document format for the web.

An HTML Slidy presentation is a collection of files: a single main XHTML file, a JavaScript file, one or more CSS files, and one or more media files which can be images and, in my case, sometimes also movies. It runs identically from a local disk using the file: protocol and from the web using HTTP. But for an event, the host usually wants to receive your presentation and load it onto their machine. And the zip-transfer-unzip dance is more friction than anyone needs or wants.

I can think of two possible approaches.

1. Use the web's native compound document features. You're probably wondering: "What compound document features?" Sadly, although it's been supported ever since Netscape's mail and news clients back in the day, this idea never gained traction. In Practical Internet Groupware I quoted this from RFC 2557:

In order to transfer a complete HTML multimedia document in a single e-mail message, it is necessary to: a) aggregate a text/html root resource and all of the subsidiary resources it references into a single composite message structure, and b) define a means by which URIs in the text/html root can reference subsidiary resources within that composite message structure.

2. Use ZIP or JAR, à la OpenOffice or the Java runtime. But browsers don't work directly with these archives, and web servers don't either, and there's approximately zero chance that will change.

At the moment, neither of these approaches seems to have a future in the realm of web technology. But as the AJAX juggernaut rolls along, it's going to keep reminding us that we still don't have a standard notion of a web compound document. We could sure use one.

Comments

Joining me for today's podcast is the programmer-turned-writer Ellen Ullman. I recently reread her 1995 book, Close to the Machine: Technophilia and its Discontents, and found it as compelling today as it was then.

Ellen likes to say that programmers create systems in their own image and according to their own desires. They were among the first to experience the lifestyle that we all take for granted now: asynchronous, machine-mediated, always on. In this conversation we talked about how this way of life affects software, individuals, and society, both for better and worse.

Comments

MTurk was invented to solve Amazon's own data-cleansing problems. Now that third parties are building MTurk-enabled businesses, they're facing different problems that spawn new requirements. In an interview following his talk, I asked Jeff Bezos about the tension between satisfying internal and external requirements. That's a good problem to have, Bezos said. And he's right. Anyone can repackage internal services and try to sell them. Evolving those services into broadly useful products is both a harder challenge and a greater opportunity. [Full story at InfoWorld.com]

I'm in Paris this week for the IDG France SOA Forum. Thanks to all the recent Amazon fireworks, I'll have plenty to talk about.

Comments

When Belgian children turn 12, they'll receive a smartcard and a reader from the government. Americans would regard this program as an Orwellian intrusion. For Belgians, it's a way to help protect kids without necessarily compromising their privacy. One of the first uses of the youth eIDs will be to prove age to age-restricted websites. How that's done is a matter of choice, but there's no technical requirement to fully disclose identity and a strong cultural preference not to. Kids will only need only prove (by authenticating to the card) that they are citizens, and prove (by selectively disclosing their birthdate) that they meet the age requirement. [Full story at InfoWorld.com]

Last time I had dinner with Phil Windley, we compared driver's licenses: his from Utah, mine from New Hampshire. On a recent trip to Guadalajara for the ICA conference, delegates from Belgium and Malta showed me their national ID cards. Someone could write a fascinating cross-cultural study based on identity documents: what information they do or don't contain, how they're used.

Comments

In a follow-up email conversation with Tim Fahlberg, who's featured in last Friday's podcast about math education, we were batting around ideas for getting kids engaged with online math resources like the ones he creates at mathcasts.org. Here's the wild idea that struck me.

Imagine a mashup of Second Life, MTurk, and web-based screensharing. In this world, kids earn Linden dollars for advancing through sequences of mathcasts and interactive tests. But to really excel in the game, they'll want to solicit help from tutors who appear as wise mentors in the game. Tutors also earn Linden dollars for their effort, and can earn bonuses when their students perform well.

Tutors and students rendezvous by way of MTurk. Students might advertise their tutoring needs, or the system -- sensing the need -- might do it for them. During a rendezvous, one or more students and a tutor share a whiteboard, converse in audiochat, and use a shared virtual calculator.

Now in truth, though Second Life would be maximally trendy, a basic web application would be more practical. Likewise, although tablet PCs would be ideal, something like a Wacom tablet would do fine. The essential ingredients would be: a pool of students, another pool of tutors, incentives for both, a mechanism for brokering supply and demand, and an appropriately equipped shared space in which to meet.

Where do the incentives come from? Parents. We'd fund this system in a heartbeat if it were proven to work.

Of course this scheme wouldn't apply only to math education. Everyone, either in school or on the job, needs on-demand learning at some point. Screencasts can meet some of that demand. As CJ Rayhill discussed in our podcast, for example, she's getting tons of mileage out of Lynda.com. But if you could go interactive as needed, that would be awesome.

Comments

The September episode of The Screening Room, with J.J. Allaire and Jack Ozzie, introduces Windows Live Writer, discusses Live Writer plugins, and illustrates how the plugin architecture is being used to add support for microformats and Live Clipboard.

There's quite a bit going on in this half-hour screencast. Live Writer is both a .NET-based WYSIWYG editor for blog posts and a prototype for the integration of microformats and Live Clipboard across a range of Microsoft products and services. Live Writer itself may or may not appeal to you as a blog authoring tool, but if you're curious about how Ray Ozzie's wiring the web strategy will play out, you'll want to see and think about the end-to-end linkup between Live Writer and Eventful that's shown here.

Comments

The guest innovator for today's podcast is Tim Fahlberg. During his 11-year stint as a math teacher, Tim pioneered the use of what he calls whiteboard movies or mathcasts, and what I call screencasts. Today he's a consultant and reseller of educational technologies who's passionately committed to reinventing education. His main website is CoolSchoolTools.com. On his Wiki, Mathcasts.org, you can find hundreds of mathcasts. Tim has done a bunch of these himself, and there are also a bunch made by other people including these Australian schoolkids.

Math, Tim says, is the "most universally poorly taught and learned subject," and it's obvious how these mathcasts could help turn that around. We also discussed how teachers can use test preparation and evaluation tools to ease their administrative burden, and student response systems ("clickers") to make classroom learning more iterative and dynamic.

Comments

Yesterday Jeff Bezos spoke at the MIT Technology Review's Emerging Technologies Conference. Afterward, I recorded a short (12-minute) interview with him that's available here both as audio and as video. The subject of his talk and of our conversation was a trio of new Amazon services that I've explored quite a bit lately: S3, EC2, and MTurk.

At the end of our conversation I relayed a question that came up during my recent Superpatrons and superlibrarians talk at the University of Michigan. When I demonstrated the various ways in which LibraryLookup can disintermediate Amazon, somebody asked: "Does Amazon hate you?" It shouldn't, I replied, because although LibraryLookup clearly bypasses some purchases, it also invites people to engage with Amazon.com more than they otherwise might. As it turns out, that's just how Jeff Bezos sees it. During his talk he used a key term of blogging art: flow. When library patrons use Amazon's catalog to research what's in the library, they're creating flow through Amazon's site, and Bezos says he's all for that.

Update: As per this comment, you can hear the entire Jeff Bezos talk that preceded this interview, and you can also watch a video summary.

Comments

I invited Kim Cameron to help me refine my clumsy notion of a translucent way to handle my Social Security Number in relationships with sites like Prosper.com. His response sketches out exactly the kind of thing I was looking for.

Kim concludes with a really useful analysis of why what he calls user-centric scenarios mostly don't exist yet:

In my view, the problem Jon has raised for discussion is one of a great many that have surfaced because institutions "elided" users from business interactions. One of the main reasons for this is that institutions had computers long before it could be assumed that individuals did.

It will take a while for our society to rebalance -- and even invert some paradigms -- given the fact that we as individuals are now computerized too.

Comments

Security pros know that there's no perfect defense against a determined attacker. So when an identity thief strikes, it's vital to detect the theft. But who's going to be the detective?

As applications migrate into the network cloud, the presumption is that IT administrators will be the detectives, vigilantly looking for clues that might spell trouble. But such vigilance will never suffice, because nobody can care as much about my own interests as me, or as much about yours as you. [Full story at InfoWorld.com]

This column aired a longstanding gripe that I think of as The Myth of the Managed Network, which explains why this never happens:

"Mr. Udell, there's been a routing glitch that affects your subnet. We're aware of the problem and we're working on it. You'll hear back from us as soon as it's fixed."

Think about it. If somebody were using your bank or webmail credentials, how the hell would you know?

Visualizing account activity in ways that make it easy for people to see anomalies at a glance is an interesting second-order problem. But the first order of business is just to show us that data.

Comments

My quest for a loosely-coupled comment system has led me to the same conclusion Dave Winer reached a while ago: Use WordPress. Rather than cross-posting to my WordPress comments blog, though, I'd like to use it as a kind of a web service that I can integrate into my existing blog. And I'd like to do that as portably as I can, so that the glue can be repurposed for any blog publishing system I might switch to, as well as for any other comment service I might switch to.

I haven't created the glue yet, I'm just doing things manually in order to think through what the glue needs to do. So far I've identified these functions:

1. Create placeholders in WordPress. I see no reason to replicate the bodies of my postings on the WordPress side. I'll just post stubs there that link back to my real blog. To make it easier to stitch things together, I'd like to embed the serial number of my Radio UserLand posts into the WordPress URLs. So for example:

   weblog.infoworld.com/udell/2006/09/23.html#a1529

   jonsradiocomments.wordpress.com/2006/09/24/a1529/

   That might not be straightforward, though. In this example, the original WordPress URL was something like:

   jonsradiocomments.wordpress.com/2006/09/24/how-translucency-could...

   The descriptive trailing part of the URL, called the post slug, is derived automatically from the title. You can change it interactively, and I did, but that's a nonstandard feature and so isn't supported in any of the weblog APIs. I might instead have to embed the serial number in the title.
2. Monitor the WordPress blog for comments. I can schedule a process to watch the main comments feed and, when a new comment appears, I can collect the whole set of comments for that item using the per-item comments feed.
3. Include comments directly in the main blog. I'm weighing a couple of approaches. If the watcher process writes comments in HTML format, and writes them to the same domain as the main blog, then I can use one of the portable wrappers around XMLHTTPRequest, such as Dojo, to support a snippet of JavaScript that sources that HTML into the page. Alternatively the watcher process could write out JSON (JavaScript Object Notation), which can be sourced into the page without requiring a heavyweight toolkit. That's how the Recent Links feature of my blog works now: Del.icio.us provides a JSON representation of my recent bookmarks at del.icio.us/feeds/json/judell.
4. Propogate deletions. At least for now, I'm leaving comments open. I'll let the WordPress spam filter catch what it can, and if anything gets through that shouldn't, I'll manually delete. How the watcher process will detect deleted comments, in order to refresh what's included on the main blog, is an open question.

It seems mostly straightforward, but I thought I'd review the plan here in case anyone's been there and done that and has advice to pass along.

By the way, I'm not completely comfortable with the idea of freeloading on WordPress in this way. As I mentioned before, I'd pay a small monthly fee for a blog-independent service that's designed equally well for interactive administration of comments and programmatic integration into my blog. That offer still stands.

Comments

From a 9/22/2006 story in the Washington Post:

The for-profit service known as Turnitin checks student work against a database of more than 22 million papers written by students around the world, as well as online sources and electronic archives of journals. School administrators said the service, which they will start using next week, is meant to deter plagiarism at a time when the Internet makes it easy to copy someone else's words. But some McLean High students are rebelling. [Students Rebel Against Database Designed to Thwart Plagiarists]

Turnitin's business is (or should be) only to detect plagiarism. To do that, it must build a database. But surprisingly and counterintuitively, the documents stored in that database need not be readable by human beings. To meet the business requirement, they need only be machine-readable versions derived from the human-readable originals.

Suppose the previous sentence appears in a student assignment. A cryptographic hash function can convert that sentence into this sequence of characters:

119ffe6a7c1f54b96beb6e38d822ebd0cb8df63d

The operation is called a one-way hash because although it will reliably and repeatedly convert the same sentence into the same sequence of characters, you cannot reverse it. The sentence is not recoverable from its derived sequence. What's more, it's very unlikely that two different sentences will yield the same sequence.

So here's a strategy for Turnitin. Convert each sentence of each student document into its corresponding sequence of characters, store only that sequence in the database, and discard the original sentence. Now the database contains no intellectual property subject to misuse. Even if it wanted to, Turnitin couldn't improperly mine the database. Neither could anyone who bought or stole the data.

But Turnitin can use the database for its sole valid purpose: to detect plagiarism. How? By deriving one-way hashes from each sentence of each document that it checks for plagiarism, and then by searching its database for those derived sequences of characters.

The strategy at work here is explored in an important but underappreciated book by Peter Wayner called Translucent Databases. The difference between transparency and translucency is the difference between clear glass and frosted glass. Light shines through both, but frosted glass creates a controlled distortion of objects behind the glass.

Though as yet poorly understood and rarely applied, the principle of translucent data management can help us defuse the ticking time bombs that many Internet services are becoming. In the many ongoing debates about privacy of data, accountability for its proper use, and risk of misuse, two critical questions are almost never asked:

1. What's the least quantity of customer data that a database operator must store in order to satisfy a business requirement?
2. What's the least useful representation of the data that can satisfy the requirement?

From the customer's perspective the incentives are clear. You want a service to store as little of your data as is necessary. And you want it stored in ways that are as useful to you as you require, while at the same time being as useless to the database operator as is feasible.

From the perspective of the service, though, it's a mixed bag. On the one hand, there's a strong incentive to gather as much data as possible, in as useful a form as is possible, in order to mine it, trade it, or sell it. On the other hand, there's a countervailing incentive to avoid the PR (and perhaps legal) disaster that will ensue if misuse of the data -- either intentional or inadvertent -- is revealed.

Turnitup may never have thought about running its database translucently. How might customers influence its thinking? Here's a three-point plan.

1. Appreciate that providers of services are subject to these conflicting incentives.
2. Think about the least quantity and least useful form of data that online services must control in order to perform their business functions.
3. Don't rely on wielding the stick of legal compulsion. Dangle the carrot of enlightened self-interest too. Storing more data than is necessary, in a more useful form than is necessary, can cause problems that service providers are incented to avoid.

Maybe the students will be found to have intellectual property rights in this case. If so, maybe the law will be inclined to defend those rights. But neither Turnitin nor the McClean High students are going to enjoy slogging through that swamp, and nobody is likely to reach solid ground on the other side anytime soon.

Translucency is a way to think outside the box and bypass the swamp. It's an idea whose time has come.

Update: Two readers, A. Henderson and Daniel Freudberg (who is a member of the McLean High School Committee for Students' Rights), wrote to ask essentially the same thing: Does cryptographic transformation create a derivative work also subject to copyright protection? It's a great question, and I don't know the answer, but I think I see a workaround. Turnitin could arrange for students to license these derivative works to it, for the sole purpose of comparison with other work.

Now, as Daniel Freudberg points out, the students are not merely objecting on intellectual property grounds. They also object to the preemptive nature of an anti-plagiarism regime that presumes everyone may be guilty until proven innocent. That's a reasonable concern, and it's outside the scope of the scheme I proposed. But now that I think of it, there are at least two reasons why students might wish to participate in a Turnitin-like scheme voluntarily:

1. To protect themselves against inadvertent plagiarism of other student work or, more likely, of published sources. (Related: See this example of how services like Google Books are about to transform the nature of citation.)
2. To protect their own work against intentional or inadvertent plagiarism by others.

While we're on the subject, I highly recommend Malcolm Gladwell's New Yorker piece from 2004, Something Borrowed, which explores how all so-called original creative work is also necessarily synthetic and derivative. The more that our technology can reveal the common DNA shared among different texts, the better we'll be able to judge what are proper and improper modes of sharing, and the more comprehensively we'll be able to visualize the flow of ideas.

Further update: Several folks -- Liz Lawley in email, and David S. in comments -- point out correctly that if texts are altered in the way I propose, instructors cannot evaluate the context in which non-original material appears.

In addition, Liz offers her take on Turnitin, from the perspective of a satisfied customer of the service:

Before I started using turnitin.con in my freshman classes, I generally had about a 10% heavy plagiarism (25% or more of the paper taken word-for-word from another source). That meant I was failing 4-6 students a quarter. Since starting to use turnitin four years ago, I've had only *one* case of a student plagiarizing. They seem to believe that the computer will catch them, but that I won't. That's made the relationship between me and my students less adversarial rather than more.

I'm really skeptical about the motivation of these copyright challenges. There's no monetary harm being done to these students, and no public display of their works. So while there may end up being grounds for a challenge in the letter of the law, I don't think it succeeds if you're looking at the spirit of the law.

What I've found is that most papers show up with 10% or more "matching" material, because it flags quotes. And it's not uncommon for a paper with 25% or more flagged material to end up being not plagiarized, but poorly written (lots of quotes strung together with minimal original content in between).

I'm not claiming to have perfect solutions either in this case or in the Prosper.com example I've used elsewhere. (Kim Cameron, however, has been noodling usefully on that one.) I am, however, trying to spark some discussion about the costs and benefits of translucency, an option that designers of software and information systems rarely (if ever) consider.

In the Turnitin example, we can imagine a scenario in which the method I proposed is a first line of defense. Its job is to find the subset of documents requiring further analysis. We can further imagine that Turnitin orchestrates a protocol whereby it retrieves items in that subset, from student- or university-controlled archives, and presents them to teachers in cleartext in an analytical framework that's part of the value added by their service, while never retaining those cleartext documents in its database.

Why go to all this trouble? Because we may be able to make some architectural choices about the surveillance systems we're creating. The McClean High students worry that use of Turnitin will make adversaries of students and teachers. From Professor Lawley's perspective that relationship already was adversarial. Now that students "seem to believe that the computer will catch them", it has become less so. As we work through these kinds of issues, what values do we uphold? And which technical architectures can best embody them?

In The Transparent Society, David Brin argues that the only sustainable model will be one in which the cameras watch everybody all the time, but nobody has exclusive access to what they see and record. We can debate the pros and cons of this arrangement but, as an architectural style, it has the compelling virtue of extreme simplicity.

Translucency, on the other hand, is inherently complex, and maybe that alone will doom it. But if it can support an architecture of surveillance that embodies values we care about, we owe it to ourselves to explore that option.

Comments

Today's podcast with Cyril Houri, founder and CEO of Mexens Technologies, advances a story I began telling in early 2005 about annotating the planet. The problem was, and still is, that there aren't very many people with GPS devices. Cyril's system, Navizon, aims to bootstrap us out of that situation. The idea is to incent people carrying the fairly small number of GPS-equipped mobile devices (PocketPCs, cellphones) to map the locations of both WiFi access points and cell towers. Then people using vast numbers of devices on WiFi or cellular networks can use location-aware applications without having to own GPS gear.

There are two incentives to contribute to the mapping effort. First, the Navizon software is free to users of compatible GPS-equipped devices. (Otherwise it's a one-time $20 license.) Second, you earn points for contributing the locations of WiFi access points and cell towers to the Navizon database.

With or without onboard GPS, the software provides a few basic applications: location-based search, geotagging, buddy tracking. But Houri says that an API that's due to be revised on Monday enables developers to write their own applications that presume a GPS. If there isn't a physical one present, and if your region has been mapped by the Navizon collective, these applications will use a virtual GPS instead.

What if your area isn't covered yet? You've gotta love this bit of advice from the Navizon FAQ:

You can either:

- wait for someone else to map your area

Or

- map your area yourself by getting a GPS device. You can even get one for a few days, drive around with Navizon running with this GPS and then return it. This way, you will have built your own virtual GPS in your area.

This time around, it might be worth hanging onto one, at least while the alternate location networks are being bootstrapped. Beyond that I'm not sure what's going to happen, but I am sure it's going to be interesting.

On a related note, I'm continuing to find reports about successful use of GPS receivers on airplanes. But maybe not for long?

Through the lens of the podcast transcription service CastingWords, I've observed Amazon's MTurk from a few different perspectives. In a podcast conversation with CastingWords' Nathan McFarland and Mycroft's Ben Hill, we explored network-enabled ways of harnessing collective intelligence. That podcast was then transcribed by CastingWords, and I reviewed the results. More recently I became a satisfied CastingWords customer.

Last night, to complete the picture, I joined the global workforce of Turkers -- that is, MTurk-enabled workers. It's a weird subculture that Katharine Mieszkowski explores in her excellent Salon article I make $1.45 a week and I love it.

One of my talks last week focused on the idea that blogging can open windows into the world of work. That's certainly true of Turk Work. Blogs such as Turk Lurker, Mechanical Turk Monitor, and CastingWords Turker chronicle the experiences of Turkers as they churn through HITs (Human Intelligence Tasks), rack up pennies, exchange tips for finding and optimizing work, and reflect on the often bizarre nature of tasks that, at least for now, do not yield directly to automation.

Here are some of the HITs available to me at the moment:

The description of the third HIT reads:

To make one's mother to stand on one foot (with the other one lifted) and to picture this outstanding scene. Name of the mother, her age and the location are needed. (TIME!!!) The fee - $2 per chosen picture.

CastingWords, of course, is a very real and very effective business. So I did a bit of transcription in order to see what it's like from the Turker perspective.

Podcasts are chopped up into six-minute segments. Last night, the segment I transcribed was worth $1.02, which will be credited to my Amazon.com account (and can then be transferred to a bank account) if my work passes several quality review checks. These checks are themselves implemented as HITs.

This morning, segments from that same batch are worth $1.28. (I should have waited!) At that rate, a Turker who could transcribe accurately and in realtime would earn $12.80/hour for himself and $1.28 for Amazon. I can't do accurate realtime transcription, of course, so I'd be lucky to approach minumum wage.

How'd I actually do? To be honest, I lost track. The segment I chose at random turned out to be an interview with Mauria Aspell, a childhood friend of Bill Clinton who dated him in high school and college, and who was evidently being interviewed for a history of Clinton's relationships with women. Thirty seconds into the recording I guessed what it was about, but when the name finally dropped it was still a shock. Bill Clinton's romantic past, in 6-minute segments, parceled out to the Turk Nation for transcription at $1.28 a pop. Whoa. Where's my blue pill?

Celebrity dirt notwithstanding, my encounter with Turk Work wasn't very satisfying. There was no context, no orderly progression, no sense of collaboration, no awareness of (or pride in) a finished product. The Salon article suggests that these are essential qualities of the experience. And as I look more closely at how MTurk is structured, I suspect that's right.

If you're running an MTurk-enabled business, you have to focus on throughput and efficiency. That drives you toward assembly-line tactics. But as Nathan McFarland noted in our conversation, you'd also like to reward excellence:

We'd be willing to pay certain workers more and other workers less, but that's not an option right now.

There are, of course, other architectures. Last week in Ann Arbor, for example, I met Bill Tozier, who among other things works for Distributed Proofreaders, an adjunct to Project Gutenberg. "We're the prior art for MTurk," Bill told me.

MTurk is one model for the online coordination of work. There will be many others, some commercial and some open, each constituted in its own way.

One final thought on MTurk. When the tsunami hit, and then Katrina, I used Amazon in both cases to contribute to disaster relief. It wasn't very satisfying, though. In retrospect I'd rather have contributed work to the Katrina People Finder. Next time around, it'd be cool if Amazon used MTurk to coordinate that kind of work, and really cool if it found ways to create social capital in the process.

I'm deeply fascinated by the idea of tacit knowledge, which is the stuff that we don't know that we know, or that we assume that everyone knows so never bother to explain. To paraphase the philosopher Michael Polanyi, whom I cited in The tacit dimension of tech support, tacit knowledge means knowing more than we can (or do) tell.

One of the things that I've known tacitly, but never articulated, is that screencasting can be an excellent way to transmit tacit knowledge. An example I used in a talk last week comes from my recent screencast with Jim Hugunin about IronPython. At one point, Jim was showing how to manipulate a bunch of Avalon objects in a Python list. But along the way he unconsciously showed me an important fact about Python that, while I'm a relatively skilled user of that language, I'd somehow never discovered. It happened right here:

While Jim was typing for b in buttons: I was pondering the previous line, buttons = _, which (as in Perl) assigns the value of the most recently evaluated expression -- in this case, a list of Avalon objects -- to a default variable that's represented using the underscore character. How did I not know that, particularly given the direct Perl analog? Such knowledge gaps are weird, but hardly uncommon. The interesting thing here is that Jim transmitted that bit of knowledge without even realizing that he did so.

In my talk I suggested that this kind of unconscious transmission of tacit knowledge may be a key rationale for the practice of pair programming. Two programmers, working side by side, exchange tacit knowledge by osmosis.

To generalize from programming to all use of software-based tools, the equivalent of pair programming -- that is, direct observation of one another's use of such tools -- is ideal. But that's not always possible, in which case live screensharing or asynchronous screencasting is the next best thing to being there.

Thoughtful responses to my recent items on translucency and selective disclosure -- here, here, and here -- continue to arrive.

Mark Rice:

Regarding translucency, have you really done Thelma Arnold (AOL user 4417749) by publishing her name and AOL user number and the fact that it has been leaked to the entire world? Why not publish her social security number as well? What are thinking of when you write about translucency? I think you need some opacity.

By the way, the original Social Security Act of 1933 made it illegal to use SS numbers for anything but Social Security and tax information. The government and many others have largely ignored this law. Twenty years ago when I applied for my pilot's license the FAA asked for my social security number. I told them it was illegal to ask, and later they changed the form to indicate that answering that question was "voluntary". However, when I received my pilot's license, guess what my license number was? You guessed it, it was my social security number. When I questioned the FAA and the Social Security Administration about it, they told me they never release that information, and that it must have been a "coincidence".

Given that the widely-read New York Times story ran Thelma's name and her "user number" -- which isn't an AOL ID, by the way, but only the number randomly assigned to her in its data dump -- I didn't see how reciting those facts in InfoWorld could make any difference one way or the other.

But Mark was exactly right. Having collected that scrap of information, it was a reflexive act to use it, even in a context where the point was to advocate opacity. As Mark's comment nicely illustrates, it's deceptively hard to retrain that reflex.

Marc Thornsbury:

I couldn't agree more. But the real problem appears in your last sentence. For all intents and purposes, there *is* no liability for storing more of a customer's data than is strictly necessary. In fact, when looking at selling that information elsewhere, the financial incentive is to get, and keep, as much information as possible.

Before you're going to see any progress on this, there's going to have to be a definition of what constitutes personal information, what constitutes permitted use (or how individuals can define permitted use on a case-by-case basis), and some kind of serious penalty for failing to meet the requirements. A good successful lawsuit would make folks want to destroy that data as quickly as possible.

However, the courts or legislature(s) are going to need to apply some concepts in creative ways. The courts have defined the concept of a "birth mother" (apart from what I suppose you would call, for lack of something better, a "regular mother") and there are certain things that this entails that cannot be changed, even when agreed to by the various parties involved. For example, if a woman offers her child for adoption before birth with agreements and even the exchange of monies, she may still refuse to complete the transaction at any time and is bound to no terms whatsoever. Her role as the birth mother cannot be "signed away", as it were.

We need the equivalent in this area. Until personal information is seen as being owned, in whatever form, by the person and merely "licensed" (for lack of a better term) to a company to be used within the rights permitted under the terms of the license issued by the person, this is not going to get better!

The funny thing about this is that I'm constantly reading about how data storage needs are out of control and companies are having to struggle with the need for more and more storage capacity. Here's a simple answer to both problems (and one as old as computing itself), it's called purging.

As Tim Sloane also noted, strong financial incentives compel organizations to hoard and trade our data. Here are two countervailing forces I'm aware of:

• Financial liability. I continue to like Bruce Schneier's ideas about the role of insurance. If lawsuits mean that theft or spillage of people's data can be a financial catastrophe, and if insurance emerges as a way to manage that risk, then the cost of that insurance begins to constrain the amount of personal data held and traded without explicit consent.
• Legal protection. Although most commercial transactions don't warrant such protection, medical transactions do. Health care, severely constrained by its inability to exchange medical records, is compelled to invent systems that conduct such exchanges with explicit consent. Led by John Halamka, the New England Healthcare EDI Network has done excellent work laying the foundations for HIPAA-compliant transactions with per-transaction approval and auditing.

Duane Sessums:

It's interesting to note that your discussion revolves around providing, storing, and encryption of the SSAN in such a service situation. Even if they didn't store it, they'd obtain a copy of your credit record in a document that has your SSAN embedded. So, they could assure you all day long they don't store your SSAN that you provide, but they have it anyway. Unless they don't store the file either, which would require them to scrape the credit score and other pertinent data off. And once they do that, you'd have to hope that they have a good enough unique key, including address, for example, to prevent identity theft or error. And even then, with that set of data, and a few bucks, someone can get your SSAN and other personal data about you.

Conundrum continues.

True. Related to this, Scott Weisman wondered why, as a lender, there would even need to be a credit check requiring the SSN. A Prosper representative offered this clarification:

While we do use your social security number for Experian, we also need it for tax reporting purposes. Prosper must record and then report earnings that lenders make off their investments. This means that we must record your social security number as it will become necessary for the duration of your activity on the platform.

I noted that, instead, the tax report could flow through me enroute to the feds, and that I could attach the SSN as part of my review of that report. I realize that things don't work that way now. But I'm trying to raise awareness of the possibility that they could, and the reasons that they should.

It's day two of my three-day speaking sprint. In Guadalajara yesterday, I talked to IT folks from 25 countries about ways social software can connect citizens and governments. And I found out a bit about the differing approaches they're taking to the rollout of digital IDs.

Today I met serially with a long list of faculty and grad students at the University of Michigan's School of Information, and gave a talk to which my sensors have so far detected two reactions.

Although I've written from time to time about the ways in which bloggers can be both like and unlike academic researchers, I haven't hung around in the university environment for a long while -- with one recent exception. So this visit is, among other things, a chance to compare these two largely distinct cultures.

One comment that came up several times, in response to my queries about why an academic would or wouldn't want to use a blog in the narrative style that I enjoy (and advocate) was: "I wouldn't want to publish a half-baked idea."

Ironically, I had included a half-baked idea in my talk: the notion that digital "learning objects" might wind up being units of barter. So, for example, you'd trade me a video of your guitar lesson in exchange for my screencast on animating scatterplots in Excel.

Even to me, this wasn't the most important part of the talk. I think my other theme of network-enabled apprenticeship -- for example, the way in which the transparency of work products and processes in open source development enables anyone to observe and join -- matters a lot more.

But I tossed out the barter idea anyway because, though it felt shaky to me, I wanted to know how this group, with roots in both economics and information science, would react to it.

The initial response was to shoot it down, for reasons that I don't disagree with. But it didn't completely crash and burn, as Todd Suomela's report notes:

The biggest conversation with the audience was about the economic incentives for people to share knowledge on the web. Jon initially proposed the idea of barter to explain the process, some people were skeptical that this would work, while others supported it.

That outcome left me wondering again about the tradeoffs between academia's longer cycles and the blogosphere's shorter ones. Granting that these are complementary modes, does blogging exemplify agile methods -- advance in small increments, test continuously, release early and often -- that academia could use more of? That's my half-baked thought for tomorrow.

For years I've entertained a fantasy about air travel that I'm sure many of you share. You're staring out the cabin window, watching the landscape scroll tailward, and some feature catches your eye: a building, a highway, a lake, a ridge. You touch the window and a heads-up display fades into view. It's kind of like Google Earth, but live and in realtime. You summon and dismiss layers of annotation, and you bookmark locations for later study.

Some people, and Doc Searls appears to be one of the most talented among them, have carefully enough studied the view from airplane windows, and carefully enough studied maps, to be able to correlate the two domains remarkably well. But most of us could use a little help -- or, in my case, probably a lot of help.

Airplane windows that morph into heads-up displays aren't in the cards anytime soon, I'm afraid. But it strikes me that we do have at our disposal some tools that we might be able to use to cobble together at least a first approximation of a solution.

Here, for example, is the list of ingredients for a recipe that I can't try making right now, because I'm at 30,000 feet watching Texas scroll tailward:

• A GPS breadcrumb trail representing my flight's route
• Timecodes for each breadcrumb
• Google Earth Pro

Can these ingredients be used to bake a screencast that would play in realtime, would document the progress of your flight, and would label various natural and manmade structures along the way? If anyone's tried this, or tried a different approach, I'd be really curious to know how it turned out.

When we do finally get those heads-up displays, connected to the plane's flight systems and to the Internet, it'll be grand, won't it? I wonder how the service will be priced. Will window seats command a premium? At a flat-rate or per-minute? Adjusted for cloud cover?

Update: Drew Burton reports:

I tried something approaching the simplest form of your flight tracking just a month ago.

I had Microsoft Streets and Trips on my laptop with an attached GPS sensor. This is out-of-the box setup.

It reported elevation, flight speed, direction just fine.

I could zoom in and out as normal. It showed my ground location as I flew long. The application has the ability to leave a trail of where I had been.

I had been in a window seat; I attached the sensor to the window.

On the way back I was in a middle seat. From there my $99 sensor picked up nothing.

It's time for a public conversation about the uses and limits of translucency. Is it really necessary to retain my social security number, or my search history, in order to provide a service? If not, what does it cost the provider of a service -- and cost the user, for that matter -- to achieve the benefit of translucency? Is this kind of opt-out a right that users of services should expect to enjoy for free, or is it a new kind of value-added service that provider can sell? [Full story at InfoWorld.com]

The crypto is beast of problem to solve, but at least it can be solved. Reflecting back on the experience, he's left with one outstanding issue for which he sees no solution. How does a customer service organization support users when they can't see the data?

In all such cases, it come down to the same protocol suggested in this week's column: you attach a one-time permission to the protected data. Can the permittee misuse that permission? Sure. It's only a question of whether, on the whole, the benefit of translucency outweighs the costs. It might or might not, I don't know and I doubt anyone does, but what worries me is that we're not seriously trying to find out.

For Tim Sloane, it's fundamentally an imbalance of power:

I am now a consultant in the payments space and your example resonated with me at two levels:

As a consumer this is indeed exactly the type of service I would like to have. It provides me privacy for the personal data (the key) that I send to the (direct) service provider and allows me to acknowledge that I want that key to be used to release my personal data by the secondary service that stores that data (the vault).

The problem is that a large percent of the human population will not trust what they can't directly control and will not share control unless mandated to do so, and maybe not even then.

Consider the problems currently making headlines in the payment arena. Personal card numbers and PINs are being released into the wild at an unprecedented rate. Payment Associations (Amex, Discover, MasterCard, etc) all have regulations designed to prevent this data from being kept by retailers and yet retailers continue to store the information locally. It is estimated that it will take at least six years before even the major retailers can be proven to be in compliance with these regulations. Retailers mistakenly believe that by keeping this data they are better positioned to refute a dispute that the transaction never occurred which is a frequent fraud event. This is the issue of control.

There is also a reluctance to share control. The credit agencies you mention are all in business to service financial institutions, not consumers. Most efforts to provide consumers even rudimentary control over the data that has been collected about them has been refused. In fact, these credit agencies have already rejected the idea that a consumer should be able to confirm if their personal credit rating should be released. The only exception credit agencies have made is when the consumer indicates they believe they are the victim of identity theft -- that is, after the data has been spilled.

So despite the fact I agree there is a need, I see the following inverse rule in play: The more critical the data that one party wants to make translucent, the more likely it is all other participants will demand direct access. Only a major power base can break this inverse rule, but the power today lies primarily with organizations that hold all the cards.

I thought about this while editing last Friday's podcast with Phil Windley, part of which explored the failure of technologists to explain and evangelize ideas -- like translucency and selective disclosure -- in ways that resonate with the public and meaningfully influence the political process. If we want to invent technologies that enable, empower, and liberate, part of the challenge is to promote ideas and raise expectations about what's possible.

I've been preparing three different talks which I'll give on three successive days next week. Here's the schedule:

Wednesday: At the International Council for Information Technology in Government Administration (ICA), in Guadalajara, Mexico.

Thursday: At the STIET (Socio-Technical Infrastructure for Electronic Transactions) Program, University of Michigan, Ann Arbor.

Friday: At the Interdisciplinary Committee on Organizational Studies (ICOS), University of Michigan, Ann Arbor.

The UM is my alma mater, so it's a real thrill to have the opportunity to speak there.

When I blogged about preparing talks back in June, John Mitchell suggested something I had never tried before: dictation. With three different talks to juggle, I was ripe for a new and better way to organize my thoughts, so I gave it a whirl. It's worked out pretty well.

Because one of the talks involves a speculative line of thinking that I haven't fully articulated before, I wanted to have it mostly written in advance. But I dictated the first draft into my Creative MUVO on a couple of long walks, then wrote it out with revisions.

The other two talks involve more familiar material, but I used dictation to rehearse it. And then yesterday, on another long walk, I came up with a slightly bizarre but remarkably useful solution to a problem.

The problem was that I had more than an hour of audio that I wanted to condense into a list of talking points that I'd write out as private cues for the talk. It was a nice day, and I didn't want to spend my time on the computer, I wanted to be out hiking. Could I listen to the hour of material, and dictate the condensed talking points, while on the move?

I'd tried something like that before, but the device wasn't up to the task, and I suspect no handheld recorder would be. Say the raw material is in a series of files, 1 to 47. It's easy to listen to them in sequence, but awkward to interrupt the sequence, switch from playback to recording mode, dictate a note, then switch back.

Then I had a brainstorm. I had recently bought a second MUVO for household use. (The combined cost of both is only what I paid for the iPod Mini which I now almost never use.) Why not use one MUVO for playback and the other for dictation?

It worked like a charm. I held the playback device in my right hand and listened through headphones, I held the recording device in my left hand, and I worked through the material dictating my talking points. In the course of a two-hour hike I'd edited an hour of audio down to five minutes of talking points. Later it took me ten minutes to transcribe them.

There weren't any people on the roads and paths I hiked, which is probably a good thing because I'm sure it looked pretty weird. But it was effective, and it was a great change of pace from sitting and typing.

More broadly, it's interesting to think about ways in which portable audio playback devices can (and should) morph into production tools. For example, it would be really handy to be able to mark ins and outs while listening to audio on the move, for later use as sound bites.

The multi-talented Phil Windley joins me for today's podcast, which was prompted by my recent experience with an expired passport, lost birth certificate, and misplaced social security card. As an author, university professor, InfoWorld contributor, and of course former state CIO, Phil has an excellent understanding of identity regimes in the real world, in the virtual world, and at the intersection of the two.

Topics include the Real ID initiative, the varying security purposes served by identity documents, U.S. and non-U.S. attitudes toward national identification and privacy, selective disclosure, identity rights agreements, the communication gap between technologists and policymakers, and the ways in which organic means of identification can complement technological means.

It's always a pleasure to speak with Phil. I hope that this conversation will be useful not only within the tech community but also more broadly. As Phil notes, our current identity regime has happened largely by default, and the next one will too unless we can help everyone make sense of the issues and opportunities.

With yesterday's release of IronPython, the story of dynamically-typed programming languages comes more clearly into focus. A core virtue of such languages is that they enable individuals or small teams to work in a rapid and exploratory way. A core virtue of statically-typed languages is that they enable larger teams to work in a declarative way that's friendlier to large-scale collaboration. Much virtual ink has been spilled debating the pros and cons of these two approaches, but why argue if you can have the best of both worlds?

We saw a hint of this in episode 2 of The Screening Room, on Adobe Flex, when Christophe Coenraets talked about the value of ActionScript 3's optional static typing. "When you already know what your types should be, declare them," he said. "When you don't, don't." One language, two styles, complementary benefits.

In episode 8, Jim Hugunin makes a related point:

I'm not a dynamic language zealot, and in general I don't really understand zealots. I wrote the first three versions of the IronPython compiler in Python, but today it's written in C#. Part of the reason is that now I understand it, so the values of prototyping, and the looser thinking that really helped a lot in the early days, don't really help as much any more. Also there are now more people working on the compiler, and there are some real benefits to the static typing, and the support you can get from Visual Studio.

That's a more radical rewrite than will make sense in most cases. But as shown in the screencast, such conversion can also proceed incrementally -- even on a per-method basis.

From a strategic perspective, Microsoft now has a stake in the ground. It aims to make dynamic languages, in the managed environment of the .NET Common Language Runtime, safe for the enterprise. Sun has shown some interest in doing the same for dynamic languages on the Java Virtual Machine, but not much, which is ironic given that Jim Hugunin started working on JPython -- now Jython, the Java equivalent to IronPython -- nine years ago.

Ross Mayfield wants a version of TechMeme focused on his own trust circle:

Give me TechMeme where the core index is based on who I read, about 150 people at any given time, to show me what my friends are interested in. [Ross Mayfield: Between popular and personal there is social]

Let's not forget about search refracted through a trust circle. Last year, back when I was accumulating all my subscribed feeds in a Mark Logic database, I prototyped that idea. In addition to charting the popularity of a link within my trust circle's feeds, it enabled me to search the text, titles, or link labels of items in those feeds. So for example, here was a snapshot of items mentioning ChoicePoint:

I'd have thought Yahoo's My Web would have gotten there by now, but evidently not. I can't say whether the obstacles are computational, or conceptual, or both.

I never published the script for my screencast about the on-demand blogosphere, but now's a good time for this excerpt:

I'm subscribed to hundreds of blogs, and I'm starting to think of the people who write them as a kind of neural network whose filters and amplifiers help me make sense of our increasingly complex information landscape.

But that's really just a metaphor. What actually happens is that I scan my feeds, absorb what I can, and bookmark what I might want to return to. Because I post my bookmarks to del.icio.us, and because other people do too, we have the beginnings of a shared memory. And because we tag our items, it's even a kind of associative memory. But del.icio.us only knows about the things we take the trouble to post and tag.

What if the entire blogosophere was, automatically, an associative memory? And what if the associations were driven by a combination of freetext and structured search? Here are some examples of what that could be like.

[...demo...]

The collective awareness of our networks of bloggers shouldn't just scroll past us as we scan our RSS inboxes. We should be able to consult those people on demand, in specific contexts. You saw me focus on Bruce Schneier's items about ChoicePoint. He's someone whose thinking I trust on matters of security and identity. Of course Bruce sits at the center of his own network, and he trusts his own sets of sources on various topics. I can't yet consult his extended network, or Kim Cameron's, or anyone else's, but it's clear that's where we're headed.

In retrospect that was a tad optimistic. Before we can search transitively across trust circles, we've got to be able to search within them.

Unlike the Car Talk Puzzler, the Friday Podcast did not lounge on the beach all summer. But after 20 episodes, it's taking a break this week. So sample the audio or text archives if you like, and the Friday Podcast will return next week. Meanwhile have a great Labor Day/Labour Day. Or, if your Labour Day isn't on Monday, have a great weekend.

I fired up my first instance of a Xen-based virtual Linux box on Amazon's just-announced grid, the Elastic Compute Cloud (EC2). Then I logged in as root and copied over a little Python-based search service that I run on my own server at home, along with the several XML data files that it searches. I fired up Python, pointed a browser at the domain name that EC2 had assigned to my virtual server, and...it just worked.
...
As the service's name suggests, though, if you need an elastic capability that can nimbly grow or shrink, EC2's the only game in town.

But not for long. Earlier this week, I met with the folks at 3Tera to discuss their AppLogic grid system. It's a kissing cousin to EC2, but with a more sophisticated approach to configuring and managing bundles of Linux applications along with other so-called virtual appliances that encapsulate firewalls and load balancers. The AppLogic management console is a slick AJAX application that you use to wire up collections of these virtual appliances and clone them for reuse. [Full story at InfoWorld.com]

As Andrew Binstock pointed out in an email to me this morning:

3Tera sells infrastructure to hosting providers like Amazon. They do not do hosting at all, nor have any plans to.

Jim Hugunin is the creator of Jython, the Java-based implementation of Python, and now IronPython, the .NET-based implementation. As IronPython 1.0 nears final release, we got together to review the synergies that emerge from deep integration between Python and .NET.

This month's installment of The Screening Room will appeal to a number of groups. First, obviously, Python developers seeking access to the facilities of the .NET Framework, including new capabilities such as Avalon and XAML. Second, C# developers who will find IronPython to be a flexible and dynamic alternative to C# (in some cases) or a complement to C# (in others). Third, VB or VB.NET programmers who may want to leverage IronPython's dynamic encapsulation of Framework components such as Avalon. Fourth, PowerShell (i.e., Monad/MSH) scripters who'll want to leverage IronPython's dynamic encapsulation of that language and its object pipeline.

Even if you fall into none of these categories, you might find it instructive to see what's possible when you combine a dynamic language like Python with a static language like C#, on top of the managed .NET runtime and in the environment of Visual Studio. Like me, Jim is allergic to zealotry. Although he's clearly a huge proponent of dynamic languages such as Python, he argues that what matters most about IronPython is the way in which it knocks down the barriers that get in the way of working effectively with the .NET Framework and its various dynamic and static languages.

Roy Fielding, the primary architect of HTTP and a co-founder of the Apache Software Foundation, joins me for today's podcast. We talked about the past, present, and future of web architectural style, and about how REST principles carry over to the work Roy has done as chief scientist with Day Software on the Java content-management standards JSR 170 and JSR 283.

At one point, in response to a question about making web services efficient enough for interprocess communication, Roy surprised me by mentioning a project called Waka, which is not only a Maori word for canoe, but also an idea for a new transfer protocol. It was first sketched in 2002, and still exists largely in Roy's head, he says, though you'll find a few mentions of it around the web.

Waka would be a binary token-based protocol that could work for both local and long-distance communication. It would also interleave data and metadata in a series of more-or-less independent frames. Given the advantages of MP3, which works that way, and the disadvantages of other media formats that don't, that sounds like a really useful feature.

Waka may sound a bit far-fetched, perhaps, but as Mark Nottingham points out, when it comes to the technical and social engineering skills that would be required to supplant HTTP, there's nobody more qualified than Roy Fielding.

Update: When I did a sound check on this podcast in my car this weekend, I realized that although the meters had looked good, my voice was too loud relative to Roy's. So I've uploaded a rebalanced version.

In other podcast-related news, Chris Gemignani -- who appeared in this episode -- has been using the newly-available transcripts to answer some questions:

Am I funnier than Jon's average guest? Did Jon Udell talk more than usual during our conversation? Answers: I'm slightly funnier than the average guest (4 laughs during our conversation), but nowhere near as funny as Gary McGraw (19 laughs). Yes, Jon did have a lot to say during our conversation about data visualization -- he spoke about 1/3 of the words, which is higher than his average.

I was planning on doing something else this morning but, when I heard that Amazon had dropped the other shoe -- adding a metered virtual server farm called EC2 to the metered storage cloud called S3, I couldn't resist. Here's a quick demo:

In case you can't play the screencast, it shows this little Python app running identically in a virtual instance on EC2.

Wow.

Applications increasingly rely on components and services that can turn up in unexpected contexts. I wasn't expecting to see a Mac OS error code on my Windows box, but in a mix-and-match world that happens. Reporting the provenance of error codes would be a helpful best practice.

Enabling users to visualize configuration change would be even more helpful. The default path remembered in that dialog box is part of the application's configuration. When the problem arose, I asked myself the obvious question: "What changed?" But there was no way to compare the state of the application before and after. [Full story at InfoWorld.com]

The length limit on the print column precluded a third idea: communication. In the case described in this column, I wound up installing a specially-instrumented version of a program that captured a trace, then gathering up that log file and transmitting it to the developer. Seems a bit arcane in this day and age, doesn't it? The following scenario should be entirely doable:

• If the trouble ticket warrants such action, the developer determines which area of the program to trace, and at what level.
• The trace specification is loaded into a service on the vendor's site, keyed to the product's serial number.
• Now when the application runs, the specified debugging data are (with the user's permission) collected and relayed to the site.
• Optionally a screencast (again with permission) is captured and sent.
• If analysis of these data yields a new hypothesis, the cycle may repeat with a new trace specification.
• If the investigation yields a solution, the user is informed and thanked.

That last step might be the most crucial. Where's the incentive, after all, to play the current version of this game -- that is, to agree to send your core dump to the vendor? Your report just vanishes into a black hole. You never find out what use was or wasn't made of the data you contributed, and there's no ongoing involvement. If the game were instead structured according to the architecture of participation, a lot of folks would get a kick out of helping to improve their software, irrespective of whether that software is commercial or open source.

From time to time, I get requests for pointers to one or another of my less technical, more general-interest screencasts. Clearly I needed a bucket for these, so I added a del.icio.us tag that defines the subset of popular/general-interest screencasts. In a similar vein, as I work through the backlog of transcripts of my Friday podcasts, I added a tag that defines the subset of podcasts for which transcripts are available. Although it's intuitively obvious to me, I suspect that most people don't yet appreciate how easily, and powerfully, tagging systems can work as databases for personal (yet shareable) information management.

Del.icio.us isn't simply backed by a database, it can function as a database to which you add (a lot of) queryable columns. For example, I use del.icio.us to keep track of two broad categories of items: those I've written and published to this blog, and those others have written and published elsewhere. So this del.icio.us query:

del.icio.us/judell/ical+jonudell

is kind of like this SQL query:

select * from bookmarks where ical IS NOT NULL and jonudell IS NOT NULL

It's too bad that you can't issue this del.icio.us query:

del.icio.us/judell/ical-jonudell

which would correspond to this SQL query:

select * from bookmarks where ical IS NOT NULL and jonudell IS NULL

But as a practical matter, for most people most of the time, this kind of negation isn't too important. You mainly want to be able to easily define, and query for, multiple overlapping subsets. So for example:

del.icio.us/judell/fridaypocast

del.icio.us/judell/fridaypocast+transcriptavailable

I think most people overlook this incredibly useful capability because they're not yet comfortable with queries involving multiple terms. That may be changing though. I recently heard JotSpot's Joe Kraus say that the old adage that people will on average type only 1.3 keywords no longer holds. Now they'll type two or three keywords. The samples I've seen from the AOL data spill support that claim.

It strikes me that there's a sweet spot somewhere between this shoestring approach and the likes of Dabble DB, an application that offers powerful web-based data management. Consider how dBase and later Access were overkill for most people's recipe lists and address books, and how 1-2-3 and Excel wound up meeting the need instead. Tag systems might turn out to be the spreadsheets of modern information management.

If you buy that notion, here are a couple of next steps. First beef up the query language with support for term negation, along with string, numeric, and date comparison.

Second, enable privacy along the tag axis as well. For example, when I was collecting resources for my 2004 special report on Longhorn, I used the tag longhorn-udell-2004-04. Private bookmarks weren't then available. Now they are, so I could track a set of such resources privately. But I can't track a set of resources publicly while also tagging them with private identifiers for my eyes only. That'd be handy.

Social tagging systems are also darned useful personal information managers. If we thought more about them from that perspective, they could become even more useful.

Update: Deeje Cooley, who works on Flex stuff at Adobe (and who is also one of the co-contributors of the term screencast) pointed me to Adobe Labs' experimental NoteTag. As shown in the NoteTag screencast, this note-taking and task-managing application stores its metadata on a tag server (e.g. del.icio.us) and its freestyle content on a blog server (e.g. blogger.com). Interesting!

As I was proofing the transcript of my conversation with Steve Burbeck about multicellular computing, I noticed that Steve had said that ten percent of the cells in our bodies are bacterial, not human. The actual complement is much more shocking: 90%, a number so surprising that it's no wonder Steve misspoke. It jumped out at me because the notion that we are human/bacterial super-organisms is in the news, thanks to an article in last week's New York Times magazine (title: Fat Factors, cover tease: Microbesity?). The article discusses how our microbial fauna may affect food metabolism. Although it doesn't make this point, by the way, there's an interesting twist here on the old nature versus nurture debate. In the case of a surrogate birth, you get genes from your biological mother (and father) but microfauna from your surrogate mother.

The blogospheric vector for the super-organism meme appears to have been Wired News, which popularized the phrase human-bacteria hybrid. Bloglines finds 46 citations of that October 2004 story, which was based on a review that appeared in the journal Nature Biotechnology. The original article, The challenges of modeling mammalian biocomplexity is online here, exquisitely rendered as HTML that preserves all figures and legends. How many times was this original article cited by the blogosphere? According to Bloglines, none.

I probably shouldn't be too surprised by that, but it does contradict a point I made in last week's conversation with Peter Suber about open access. I said that bloggers typically respect primary sources, and are scrupulous about tracking them down and citing them. Clearly that didn't happen in this case.

When I did track down and read that paper, here's what jumped out at me:

It is possible to conceive of each cell as a node with a set of metabolic pathways within the node as above, but with each node connected to other nodes and then the problem reduces to the modeling of the internodal connections.

So, IT folks and biotech folks are converging on a similar (maybe the same) problem: modeling and managing complex networks. In principle the blogosphere stands ready to enable the kinds of cross-disciplinary conversations that will move us forward. In practice I suspect that, so far anyway, it is doing so less often and less effectively than it could.

Peter Suber, the leading chronicler of the open access movement, joins me for this week's podcast. Since the dawn of the blog era, it's been obvious to me that the modes of knowledge exchange we bloggers take for granted are also a natural fit for scientific and academic publishing. That idea has matured more slowly than some of us had hoped. But as you know if you follow Peter's blog, Open Access News, it has now taken root and is growing at a healthy rate.

In this conversation Peter defines open access repositories and open access journals, and he discusses the history, economics, and cultural practices driving the open access movement. We also discuss the ways in which scholarly open access is both like and unlike blogging, in terms of technologies and methods.

I was searching Amazon just now for Ellen Ullman's 2003 novel, The Bug, and being a savvy user of search engines I quoted the phrase in my query. In other words, I typed "the bug", which yields:

Could it be that Amazon defaults to exact phrase search? Nope. Further experimentation shows that in other cases the normal convention -- using quotes to invoke exact phrase search -- works as expected. Perhaps this is a bug related to the occurrence of the stop word the in a two-word phrase?

Whether people actually do expect to use quotes to invoke exact phrase search is, of course, highly doubtful. I've always assumed that exact search remains largely undiscovered and unused. I haven't had the heart to download the AOL data spill and look through it, but in all the excerpts I've seen, I don't think there's been a single quoted phrase.

Lately I've been trying to pay attention to my own search strategies. When we search, we rely on a reservoir of tacit knowledge. Articulating and sharing that knowledge would be helpful to searchers and search providers alike. Lou Rosenfeld's forthcoming book on search analytics, discussed in this podcast, will provide some guidance on this topic. There's going to be a lot more to learn, and a lot more to do, as we weave search ever more intricately into the fabric of our minds.

Two more of my weekly Strategic Developer columns have spooled up at InfoWorld.com. Both discuss XML dialects, but the two dialects are written by very different kinds of people. XAML, Microsoft's eXtensible Application Markup Language, is for software developers. XBRL, the eXtensible Business Reporting Language, is for accountants.

Why Microsoft should open XAML

Here's a crazy idea: Open-source the WPF/E [Windows Presentation Foundation/Everywhere], endorse a Mono-based version, and make XAML an open standard. Why? Because an Adobe/Microsoft arms race ignores the real competition: Web 2.0, and the service infrastructure that supports it.

The HTML/JavaScript browser has been shown to be capable of tricks once thought impossible. Meanwhile, though, we're moving inexorably toward so-called RIAs (rich Internet applications) that are defined, at least in part, by such declarative XML languages as Adobe's MXML, Microsoft's XAML, Mozilla's XUL (XML User Interface Language), and a flock of other variations on the theme.

Imagine a world in which browsers are ubiquitous, yet balkanized by incompatible versions of HTML. That's just where RIA players and their XML languages are taking us. Is there an alternative? Sure. Open XAML. [Full story at InfoWorld.com]

XAML, uniquely among XML dialects for GUI layout, is a .NET language that, like C#, works with the .NET Framework and compiles down to Common Language Runtime instructions. That's why opening XAML in any meaningful way would also require a rapprochement between Microsoft and Project Mono. I'm not going to hold my breath waiting for that to happen. But still...what if?

XML for business reporting gains momentum

Two years ago I wrote an unflattering report on XBRL (eXtensible Business Reporting Language), an emerging standard that aims to improve the speed, accuracy, and transparency of business and financial reporting. I applauded the goals, as we all should in the wake of Enron and other scandals, but worried about the complexity of the 151-page XBRL specification, its aggressive use of esoteric features of XML, and its reliance on accounting "taxonomies" defined by committees. I've too often seen these kinds of ambitious efforts stumble and give way to simpler approaches. SGML gave way to XML, for example, and while XML itself offers many advanced features, its most successful application -- RSS -- uses none of them. Would XBRL wind up being used mainly by what one wag called a "master race" of consultants and accountants? [Full story at InfoWorld.com]

In last week's podcast, XBRL's inventor, Charlie Hoffman, assured me I'm not the only one to express these concerns. Just this week, for example, when the SEC announced its Request for Proposal for the development of XBRL-based software, Dave Winer echoed them:

Sounds like the SEC is wanting to re-invent RSS?

Although I felt and to some extent still feel that way about XBRL, I have a much more complete understanding of the issues after researching, recording, and editing the podcast. It runs way longer than the others in my series, almost 70 minutes (edited down from 90), but I think the material warrants that lengthy treatment. Charlie Hoffman doesn't want to reinvent RSS, he wants to reinvent accounting, and he speaks as an accountant not an XML geek.

Back in May I reviewed the CastingWords transcription service as applied (recursively) to the podcast in which Nathan McFarland, Ben Hill, and I discussed how technologies such as MTurk and Mycroft are being used to distribute and coordinate work. Recently I bit the bullet and submitted the URL of my podcast feed to CastingWords along with this order for transcription:

42 cents per minute, 620 minutes of audio, 260 dollars. That's just astoundingly cost-effective, and the quality of the results is excellent. I submitted the order on August 3, and the work was done on August 9.

So I'm going to start working through the backlog in spare moments, and publishing the transcripts as I can. This morning I went through the first podcast in the series, with Gary McGraw, and published this transcript. It looks like it'll take me about an hour per episode to tweak these up to a standard I'm comfortable with. The process involves some word-level corrections, some rewriting, and some fact-checking -- though it's clear that the transcribers have already done a lot of that. In this episode, for example, the transcriber went the extra mile to correctly identify Det Norske Veritas. Impressive!

It was nice to revisit this interview with Gary McGraw who, coincidentally, I finally met face-to-face at a music gathering this past weekend. Our interview was a great conversation that hasn't been heard as much as the more recent episodes, because it took a while for the series to build momentum. My hope is that the availability of the transcripts will yield two benefits. First, that they'll meet the needs of folks who have no time for, or interest in, listening to audio. But second, that they'll be a means of discovery for folks who would prefer to listen. The other day, for example, I had the option to read or listen to the talks given at the 30th anniversary celebration of The Selfish Gene. Although reading would have been faster, the only time I had available was exercise time, so I chose to listen instead.

Update: Heribert Slama, who is Swiss and whose first language is German, adds:

Transcripts are a big help for non-native speakers (readers) of English while listening to the podcast. Colloquial speech is often real fast and pronounced rather casually. The listener must work hard and still misses some points; a transcript makes listening a pleasure;-)

During the early flush of Google Maps excitement I made a screencast of the animated display of a route. This seemed to me then, as it still does now, a compelling use for a handheld video player. Eighteen months later that vision is almost but not quite within easy reach, as I discovered this weekend.

My idea was to transform a variant of this route into a video that I'd put onto my son's Sony PSP (sshhh, don't tell him) and take with me on my bike ride. One piece of the puzzle -- how to transfer a video to the PSP -- was already in place. A few weeks ago I realized that ffmpegX, the GUI-wrapped version of the all-purpose ffmpeg encoder, has a preset option for encoding to the PSP's format. Very cool! A few experiments showed me that although screencasts (as I suspected) don't play well on the small screen, motion videos and animations do.

I can imagine two different modes of capturing an animated map flyover using the Google, Yahoo, or Microsoft map services. In the first and simplest mode, you'd have a route description in hand, and you'd feed it to a service that would simply emit your video. You wouldn't alter the route, and you wouldn't be zooming in and out to inspect details as shown in the screencast cited above.

In the second mode you'd do an interactive flyover, panning and zooming as needed. You could run a local screen camera to capture the video, of course, but the much cooler solution would be to relay your stream of navigation events to a service that would again automatically produce and emit your video in the desired target format.

Since my aim was to explore roads not previously taken, I didn't have a canned route, so I decided to use the interactive approach. And since there's not yet a service that will capture events and produce video, I planned to do local capture and ffmpegX conversion.

So I tried doing the interactive flyover in each of three mappers: Google's, Yahoo's, and Microsoft's. This led me to compare them on a feature that I'd never really cared about before: keyboard navigation. In all three mappers you can fly around video-game style, using the arrow keys, but the experiences are very different. Here's what I found:

At this point it was too nice a day to mess around on the computer any more, so I bagged the experiment and hopped on my bike. But it was an interesting discovery. Although Google has the most detailed satellite imagery, at least for the area I was exploring (Lake Sunapee, NH), if my goal were to do an exploratory flyover for conversion to an animated video for handheld display, I'd have used the Microsoft service instead, because it's easier to control the flyover.

What about Google Earth? I'd rate it slightly behind Google Maps. If you're trying to fly smoothly and continuously over a winding road, what works best -- at least for me -- is to use arrow keys individually for up/down/left/right, and in combination for other directions. On that criterion, Microsoft's service comes out ahead of the others, at least in my subjective experience. I'd be curious what others think.

More broadly, I wonder who'll be the first to achieve the deluxe implementation I've sketched: interactive flyover, relay of navigation events to a web service, automatic production of downloadable video in the format of your choice -- including PSP, iPod, etc.

Charlie Hoffman, the director of industry solutions for UBmatrix, is acknowledged as "the father of XBRL" -- the eXtensible Business Reporting Language to which I had a bit of an xallergic reaction when I first encountered it a couple of years ago. But when Brian DeLacey, a researcher turned XBRL entrepeneur, suggested that I interview Charlie I jumped at the chance. In this week's podcast the three of us discuss the history of XBRL, its relationship to XML, its goals, its successes, and its challenges.

In next week's InfoWorld column I'll write more about what I learned from this long and fascinating conversation. But in a nutshell, though my criticisms of XBRL's complexity were and are valid -- as Charlie Hoffman admits -- the real story is (as always) much more nuanced. The inherent complexity of accounting standards, the competitive forces at work in the realm of global finance, the regulatory pressure being brought to bear -- these and other factors form the context in which the development of XBRL must be understood.

It's worth noting that while XBRL is a complex beast that makes aggressive use of certain advanced features of XML, Charlie Hoffman isn't (or anyway wasn't originally) an XML geek. He's an accountant who, as you'll hear in this interview, is deeply grounded in the practice of his trade. That makes this story an interesting contrast to the development of many of the web services standards I've studied.

Earlier this week Lorcan Dempsey, who is VP and chief strategist with the Online Computer Library Center (OCLC), blogged about an enhancement to an OCLC service that searches the Library of Congress Name Authority File. The new version uses fuzzy matching, which means that the common misspelling of my name as 'John' will find me as well as my alter ego Jon G. Udell.

This reminded me that for years, in various online venues, I've seen my book, Practical Internet Groupware, attributed to Jon G. Udell, author of The economics of the American newspaper. It turns out that's because the authoritative record at the Library of Congress has had it wrong all this time. Lorcan kindly referred the matter to an OCLC colleague who made the correction and reported it to the LC. So at some point my book as seen in WorldCat will be correctly attributed, and eventually that change should propagate to the libraries that subscribe to WorldCat.

How in general can authors resolve such problems? The OCLC advises:

We get lots of comments from authors via the Comments button on FirstSearch record displays as well as through the general oclc@oclc.org email address. In addition, the general Contacts page on the OCLC web site contains links to forms that can be used to request changes to bibliographic and authority records.

The Library of Congress gets similar comments via a feature on the record displays in their online catalog that allows users to submit an Error Report Form. It's kind of hidden at the very bottom of the display.

One caution, since catalogers work from title pages and other information in the material being cataloged, we often have to ask for proof before making a change. Proof may be a faxed copy of the title-page or its verso, etc.

We are often the best authorities for information about ourselves, and we often encounter errors that we could easily fix. Why don't we? Because the connection between the authoritative source of a fact and its erroneous manifestation is rarely explicit.

Given that the infosphere is becoming a web of syndicated facts, we'll want ato make those connections explicit. As a best practice, data provenance should be accessible at the point of display and use.

For most people, most of the time, a faster computer won't make much of a difference. There's nothing CPU-intensive about reading and writing email, surfing the web, or using most other applications. The notable exception to the rule, of course, is audio and video processing. Nowadays I often find myself staring impatiently at progress meters in Audacity, Audition, Camtasia Studio, and iMovie while I filter and compress large media files. No matter how fast your computer, in these situations you need it faster.

To hurry things up, I boost the priorities of these apps. If a process is already running you can do that with Task Manager on Windows, or the renice command on OS X. But since I nearly always want to give these media apps as many cycles as I can, I'd rather just bump their priorities at launch. You can do that with scripts containing, for example:

Audacity on Windows:

start /high "c:\program files\audacity\audacity"

iMovie on OS X:

nice -20  '/Applications/iMovie HD.app/Contents/MacOS/iMovie HD'

But how do you empower the less geek-inclined to do these things? I worked through the exercise on Windows, but it wasn't pretty. To avoid a double-quoting problem, I needed the startup icon's launch string to simply be:

cmd /c "start /high audacity"

which meant I had to add Audacity's directory to the path. Then, because the launch was indirected through a command shell, I had to use the startup icon's Change Icon feature to rebind the Audacity logo. Inured though I am to these indignities, I've learned to pay attention to the glazed look that comes into the eyes of civilians when you subject them to this nonsense.

I'm sure there's a similar procedure for the Mac but, to be honest, I just type the name of my priority-boosting iMovie launch script.

So back to the question. If you want to transfer this capability to friends and colleagues, what's the path of least resistance? I've looked around and, of course, there are a jillion shareware priority adjusters for Mac and Windows systems, but these tend to be Swiss army knives that are also likely to cause the dreaded eye-glaze effect. Suppose all you want is for a nontechnical user to be able to pick a couple of apps that will launch at high priority. The solutions should require as little non-native software as possible, ideally none. Do they exist?

Update: Tom has a nice solution for the Mac:

I'd recommend wrapping the unix command in an applescript. That way you could send it to non-tech adept friends and all they'd have to do is click on an icon and it'll run.

In script editor

Something like do shell script "nice -20 '/Applications/iMovie HD.app/Contents/MacOS/iMovie HD'"

Save it as an application and give it a custom icon and you should be all set.

On the Windows front, Douglas Deden has a partial solution:

I created a batch file, called high.cmd, and placed it on my desktop. It is just one line:

start "dummy title" /high %1

I can now drag-and-drop an executable (even one with a long filename) on to it, and it will start it with a high priority. The dummy title prevents the start command from interpreting a long filename, which will be automatically enclosed in quotes, as the title for the command window that will appear briefly.

So far, so good. Unfortunately, this works fine if I drag the executable on to the batch file, but if I drag a shortcut (a much more useful feature), it still starts the program pointed to, but the priority gets reverted to normal.

Two years ago, when I launched my yearlong series of Prime-Time Hypermedia columns on the O'Reilly Network, the term screencast had yet to be coined. It wasn't until November of 2004 that Joseph McDonald and Deeje Cooley would separately propose the term that I picked. A year later I found 325,000 references to the term in Google's index. Today, there are 7 million. Screencasts are being used by open source projects, by commercial vendors, and happily, as of last week, by InfoWorld reviewers.

While I'm delighted to watch all this unfold, I"m also mindful of the challenges ahead. On the standards front, media players and delivery formats remain in flux. And as screencasts (and other videos) proliferate, the problem of URL-addressable random access -- also unsolved in the realm of audio -- looms larger.

On the tools front, commercial software for capture, editing, and production is good and getting better, but there's plenty of room for improvement. And the commercial stuff isn't cheap, which deters casual and spontaneous use. Free or inexpensive alternatives could meet that need, but the Audacity of screencasting has (in my opinion) yet to emerge.

While these constraints will ease over time, here's one that won't. It's just plain hard, in any medium, to tell a great story. The best screencasts I've done -- and not coincidentally also the most successful -- speak to questions of why as well as how. Why does Wikipedia work? Why would I want to use del.icio.us? As networked software increasingly mediates our lives and our work, we'll need to to make sense of these new modes of experience. Telling the stories that help us do that is the highest and best use of the medium. It's also the most elusive.

A couple of weeks ago I was interviewed for a Wired article about LibriVox. I don't know if that article is going to run in the magazine or not, but I thought about LibriVox a lot today while out on a long bike ride¹ during which I listened to the first 28 chapters of the LibriVox recording of Jules Verne's A Journey to the Interior of the Earth².

The Wired writer, Michael Erard, asked me why I like LibriVox, given that the recordings are of a quality that a commercial outfit such as Audible could fairly characterize as amateurish. It was a great question, and one I hadn't really considered. I realized that part of what I cherish is that these recordings aren't commercial products. They're pure expressions of a love of literature, and a desire to share that love.

Michael then made an observation that I think is exactly right. LibriVox, he suggested, taps a deep well of emotion. For those of us who were read to as children, and who have in turn read to our own children, there's just something special about reading aloud.

This past winter, while tramping through the snow, Gordon Mackenzie, Kristen McQuillen, Jean O'Sullivan, and Miette brought one of my childhood memories to life: Jack London's Call of the Wild. I would never have made time to revisit that book if it meant carving out a couple of hours of reading time. But on a winter hike it was the perfect companion.

Today's summer bike ride was, likewise, the perfect time to revisit Jules Verne's wonderful geological fantasy. The readers who've taken me through chapter 27 are Vinny Bove, Mark Bradford, Hugh McGuire, Kristin Luoma, Mur Lafferty, Paul S. Jenkins, Alex Foster, and Kristen McQuillen. Thanks everyone! You made my four-hour lake tour so much more fun than it was the last time.

¹ I continue to use Gmaps Pedometer to chart my rides. But truth be told, I prefer the quality of the maps in Yahoo's new map service and in MSN Virtual Earth Windows Live Local. In principle it should be possible to port behaviors, like Gmaps Pedometer, from one of these mapping services to any other. In practice that's easier said than done. Still, I'd love to see a proof-of-concept map-based application done AJAX-style, but for an abstract interface that's implemented variously for Google, Yahoo, and Microsoft.

² I always thought the title was Journey to the Center of the Earth, and apparently that is the more common variant. However, the Project Gutenberg version of the tale uses the less common one, and LibriVox has close ties with Project Gutenberg. Given that the title of the original version, in French, seems to have been Voyage au Centre de la Terre, I wonder how the variant title arose?

Ross Mayfield, CEO and founder of the enterprise wiki company Socialtext, joins me for this week's podcast. It was a well-timed interview because today and this weekend the Wikimaniacs are gathered in Cambridge, MA, for Wikimania 2006.

We talked about Socialtext's recent decision to release its product under an open source license, about its relationship to Dan Bricklin's WikiCalc, and about the ways in which Ward Cunningham's brainchild is transforming, and being transformed by, enterprise IT.

While I've been writing about other things, two of my weekly Strategic Developer columns have spooled up at InfoWorld.com. For those who rely on this blog for pointers to those columns, here's a rundown.

The rise of open infrastructure

When entrepreneurs pitch their software-as-a-service ideas to me, I always ask how they plan to compete with what I call the galactic clusters -- Amazon, Google, Microsoft, and Yahoo. These giants have set a high bar for Internet-scale operations, and they're relentlessly pushing it higher.

The answer usually comes back: "We're confident we can scale out as needed." Maybe yes, maybe no. A lot depends on architectural choices and operational competence. But either way, if you are merely a planet, you don't want to butt heads with a galaxy.
...
At the moment, it seems very unlikely that a motley crew of volunteers distributed around the globe will be able to match the economies of scale and the military discipline that make today's giant clusters the awesome powers that they are. But shouldn't we have learned, by now, to expect the unexpected? [Full story at InfoWorld.com]

This column was partly inspired by a Tim O'Reilly posting. Later, Tim made connections back to my column (and to a recent blog item on S3) here. The ensuing discussion drifts over to open source licensing, but Tim reiterates this key point:

I think that there is an open, cooperative answer to the infrastructure advantages that accrue to the big web players. It will be interesting to see if it actually develops.

Although P2P technologies point the way forward, they carry way too much of the wrong kind of baggage. If a there were to emerge a P2P system dedicated to the robust delivery of services rather than content, that might help us recalibrate our thinking.

Tech believers, meet evolution

Marketers, when they are lucky, create memes that prosper by viral replication. But in the Petri dish of popular consciousness, man-made and organic memes compete on their own terms. They don't actually have their own agendas but, if we want to understand how they produce belief systems in us, it may be helpful to pretend that they do. [Full story at InfoWorld.com]

My editor, Neil McAllister, hates it when I use words like meme and memetic, and I try not to torment him, but once in a while I indulge myself. To celebrate the thirtieth anniversary of Richard Dawkins' seminal book The Selfish Gene, as well as the publication of Daniel Dennett's remarkable Breaking the Spell, I wrote a little essay exploring how Apple's TV ads can backfire, why we form religious beliefs about technology, and how modern evolutionary thinking bears on the memetics (sorry, Neil!) of marketing.

Back in March, by the way, there was an event in London called The Selfish Gene: Thirty Years On. The speakers were Daniel Dennett, Sir John Krebs, Matt Ridley, Ian McEwan, and Richard Dawkins. Excellent stuff -- in particular, I enjoyed Matt Ridley's dissection of the large fraction of the genome that is literally selfish, replicating without serving any biological purpose.

You can read their remarks or, as I did, listen to them.

Lately I've been collecting PDF books on my PowerBook. Titles include Yochai Benkler's The Wealth of Networks, John Willinsky's The Access Principle, and Eric von Hippel's Democratizing Innovation. It's a treat to find these and other titles available as free downloads, but the clash between the portrait layout of books and the landscape layout of computer screens always winds up frustrating me.

The other day I had a brainstorm. Turn the PDF sideways, and stand the PowerBook on end. As you can see in this 2-minute video, it works like a charm!

I knew that other folks must have gotten there first, and of course that's true. Andrew Burke posted the same solution back in March, for example. Still, I'll bet this weirdly simple yet effective trick hasn't occurred to a lot of folks.

When I starting writing this entry, my goal was to refer you to a segment of a podcast that I listened to on a hike last night, because it dovetails nicely with yesterday's screencast about medical expert systems. But I fell down a rabbit hole while trying to find links and identify audio segments. This kind of information always seems to be at our fingertips. But when we reach for it, we realize that it's not.

In this case, I decided to record and examine the normally tacit process of connecting the dots. Here is the sequence of obstacles I had to overcome.

Obstacle 1: iTunes

iTunes remains my podcatcher, even though I almost never use my iPod. (The Creative MUVO -- cheap, standard rechargeable AAAs, no attitude -- is my workhorse MP3 player.) But I curse iTunes when I need to recover the source of a podcast. It will show you where a feed comes from -- in this case, webcast.berkeley.edu/events -- but can't be bothered to provide a clickable, or even copyable, link. So, as I've grumbled about elsewhere, I had to type it in.

Obstacle 2: Search gymnastics

I knew the title of the event I wanted to cite: The (Real) State of the Union: Atlantic Monthly Panel. But it took a while to find it on the site. My search strategy went like so:

query	outcome
state union	fail
state of the union	fail
"state of the union"	fail
atlantic	succeed

Should've let Google do it, maybe, but in any case I found my way to the home page for the event. I was seeking two facts. First, the name of the woman whose 9-minute segment of the hour-and-a-half panel impressed me. Second, the timecodes for that segment. The event page provides neither.

Obstacle 3: More search gymnastics

It wasn't easy to find the speaker's name. Here was the search strategy:

action	outcome
listen to podcast intro	partial success (sounds like Shannon Branley or Brantley or Bradlee, with the Numerica Foundation)
google query: shannon branlee	fail
google query: numerica foundation	fail
google query: ted halstead (also mentioned in connection with the foundation)	partial success (refine query to New America Foundation)
google query: new america foundation	partial success (found the organization)
new america foundation query: shannon	success (it's Shannon Brownlee)

Obstacle 4: Media formats and wrappers

Three formats are offered:

1. Watch. ("Streaming" RealVideo, via RTSP, for RealPlayer)
2. Listen. ("Streaming" MP3, via RTSP, for RealPlayer)
3. Download ("Downloadable" MP3, via HTTP, for any player or device)

Now, how do I connect you that segment? If you are willing to use RealPlayer and be tethered to your computer for 9 minutes, I can offer you these links: audio, video. Not bad! Except:

1. You probably don't want to be tethered to your computer.
2. Or be forced to watch or listen in the Real player.
3. And good luck figuring out how create and publish these kinds of links for yourself.

In theory, my MP3 clipping service can solve the first two problems, and ameliorate the third. In practice it doesn't work in this case for reasons I've yet to sort out.

Where was I?

Oh, yeah, now I remember. Shannon Brownlee spoke lucidly about the relationship between poor information management and poor health care. For example, she pointed out that 90,000 Americans die each year as a result of just the sorts of medical errors that the triage application we saw yesterday is designed to prevent.

Technology can help improve our ability to manage medical information and transmit medical knowledge. But, as Don Thomas says in yesterday's screencast, the active ingredients of the solution are will, common sense, and elbow grease.

The same holds true in general. So while I endorse Shannon Brownlee's analysis, I don't buy her concluding appeal for a new federal agency to collect and analyze information about health care delivery and outcomes. I wouldn't expect the government to do that any more than I'd hold Google and Microsoft and Yahoo! responsible for taming the cornucopia of general information and knowledge. We produce those goods. We'll be rewarded when we make them well-structured, well-connected, and therefore discoverable. And we'll be punished (by being ignored) when we don't.

How do we get it right? Will, common sense, elbow grease.

The July episode of The Screening Room features Dr. Donald Thomas. He started out as an engineer, then turned to medicine. For many years he's been an emergency room physician and administrator. We connected by way of Peter O'Toole, director of product development at mTuitive, who responded to my invitation to practitioners. mTuitive makes an authoring toolkit for expert systems, and Dr. Thomas is using that toolkit to create an emergency room triage application that helps enforce proper diagnostic procedure, and helps automate medical recordkeeping.

The story this month is partly about the mTuitive toolkit, and partly about the tablet-based application that Dr. Thomas' company, Mentat Systems, is building on top of that toolkit. But mainly it's an exploration of two themes in the development of of expert systems. First, what makes an expert system useful to the people it aims to support? Second, what makes a toolkit for building such a system useful to the developer who aims to provide that support?

I really enjoyed making this episode, and for what it's worth I think it breaks new ground in terms of what technical journalism can (and arguably should) do. Given that mTuitive's toolkit is a .NET-based inferencing engine, there's plenty of raw technology under the covers, but that's not the main theme. Instead we focus on what it is like, from a domain expert's perspective, first to use and then to create a rule-based assistant. What matters most, in both cases, is how well the technology accommodates the expert's familiar ways of thinking and ways of doing.

There is, finally, a larger ambition for this screencast. One reason successful expert systems are scarcer than they ought to be, I suspect, is that experts in many domains aren't able to look over the shoulder of an expert-system developer and find out what it is like to do that kind of work. In medical instruction, as Dr. Thomas points out, the old saying is: "See one, do one, teach one". I hope that this little documentary, and others like it, will help those who could be creating useful applications to see how they're done, and to do them, and maybe even to teach how.

Paul Patrick is the architect of BEA's AquaLogic suite. In today's podcast we reflect on what's happened in the year since BEA declared its intention to become the Switzerland of SOA. Topics include objects versus services, why SOA != web services, how XQuery enables smart intermediaries, the nature of security in a world of distributed data, and BEA's intentions with respect to .NET, PHP, REST, and POX. Despite that mouthful of acronyms, the conversation should be accessible to IT-oriented businessfolk as well as to business-oriented technologists.

One of the pleasures of having been in this game as long as I have is that, once in a while, you get to reconnect with fond memories. Back in my BYTE days, around the time when CORBA was being overhyped just as web services tend to be lately, a team of folks from Digital Equipment Corp. arrived to show us their CORBA product, called ACA Services. (You can still find the occasional reference to it floating around on the Net.) At a time when cosmic architectures were all the rage, these folks impressed me mightily with their down-to-earth approach. They'd take anything you had -- a Unix shell script, a Windows application, a DOS batch file -- and wrap it up as a service in their CORBA framework.

In this interview I learned that the architect of ACA Services was -- you guessed it -- Paul Patrick.

Now that I've had a chance to listen to my talk at the University of Mary Washington's faculty academy in May, I'm willing to pass along the link. That talk, which touches on a number of the themes woven through this blog, was a longer and better version of ones I gave at ETech and Yahoo! back in March. Since I'm more naturally a writer than a public speaker, I'm always reluctant to hear myself unscripted and unedited, but this talk sounded pretty close to how I'd hoped that it would. That's mainly because this was the first time I've been able to refine a talk over the course of several events. Prior talks had been relatively few and far between, for very different events and audiences, requiring unique presentations that were given exactly once. More often than not, in those cases, I wrote a script to ensure coherence while sacrificing spontaneity -- not a great tradeoff.

How do people learn to be both fluent and spontaneous on a variety of topics? While there's surely no substitute for lots of public speaking experience, my Friday podcast series -- and in particular, the process of editing those podcasts -- has been extremely helpful.

As my friend (and InfoWorld contributing editor) Peter Wayner remarked the other day, verbal fluency can be a challenge for writers like us. The editing machinery that runs in our heads doesn't normally have to work in realtime. When we try to make it work that way, the results are often frustrating both for us and for our listeners.

Listening to my own recorded speech, as I work on these podcasts, has made me much more aware of how my editing machine gets in my way. Just listening hasn't helped me to fix the problem, but editing has. Another friend, the psychologist Larry Welkowitz, has a theory that explains why: errorless modeling.

Larry works with kids who live somewhere along the continuum from Asperger's Syndrome to high-functioning autism. Effective and appropriate social performance is a major struggle for these kids. Larry's idea, which he's written up in a couple of blog posts, is that you can help them by recording awkward performances, editing out the awkwardness, and then playing back the edited versions so they can see and hear themselves performing fluently.

It seems to be working for me. My original rationale for editing my podcasts was simply to improve the experience for the listener. And while the techniques of the audio digital darkroom do raise interesting issues, none of my interviewees has complained. Everyone likes to sound as fluent as possible. But in recent weeks I've noticed less need to edit myself. Is this the result of errorless modeling? If so, it's a nice bonus.

When I first heard Bob Glushko speak at a conference I knew we were kindred spirits. Our shared interests include information architecture, XML, web services, and the hybrid discipline of document engineering that he and Tim McGrath define in their eponymously-titled new book.

In today's podcast we discuss these topics, focusing particularly on Bob's experience -- first as an industry practitioner, now as an academic instructor -- with the kinds of business pattern languages exemplified by the RosettaNet and UBL (Universal Business Language) initiatives.

In the past these ideas seemed abstract and arcane to me, and to some extent they still do, but after reading the book and discussing it with Bob I have a much more concrete sense of what they are and how they apply to the real world. Towards the end of our conversation I asked Bob what it's like to think in those terms and to design information systems accordingly. He replied:

What I always tell people is, you're out there at the Home Depot warehouse picking up the stuff you just bought. Close your eyes and say, let me make the buildings go away, and the people go away, and the trucks go away, and just think about who is exchanging information with whom, and what does the message say, what does it mean. Forget the physical implementation and try to focus on the abstraction that business information is being exchanged among parties for some purpose.

That's a nice way to talk about a process that's abstract, but solidly grounded in reality.

Should Amazon have supported more advanced (and more conventional) storage and messaging APIs out the box? Perhaps. But when you plant a new field of grass there's an old adage: Watch where the footpaths go before you lay down the sidewalks. Amazon's S3/SQS duo is a green field that invites entrepreneurs to think way outside the box. For example, these technologies aren't obviously the foundation of an AJAX-oriented service grid. But if that's where people want to go, Amazon seems willing to help them get there. [Full story at InfoWorld.com]

Bill Seitz alerted me to one of those footpaths the other day: Les Orchard's S3-backed wiki. Outstandingly cool! For those who have not followed the various plot threads closely, this is an evolution of the idea of the serverless wiki. TiddlyWiki, which I've mentioned a couple of times, is a member of that species. It's a single HTML "page" that implements wiki behavior in client-side JavaScript; you save your page to your local disk to record changes permanently.

Les has substituted Amazon's S3 network storage for the local disk. It's a wonderful hack that anticipates a whole new breed of highly-available serverless applications. When I first wrote about the concept of a SPADE (Single Page Application Development Environment) it seemed interesting but far-fetched. Backed by S3, though, it starts to look a whole lot more practical.

InfoWorld's online folks have long complained about the absence of InfoWorld news stories from Google News. Here is the most striking illustration of the problem:

1. WWW.google.com/news?q=site:WWW.infoworld.com	Results 1 - 10 of about 2,040,000
2. NEWS.google.com/news?q=site:WWW.infoworld.com	No pages were found
3. NEWS.google.com/news?q=site:WEBLOG.infoworld.com	Results 1 - 10 of about 297

To summarize:

1. Lots of www.infoworld.com articles are in the main Google index.


2. But no www.infoworld.com articles are in the Google News index.


3. However some weblog.infoworld.com articles are in the Google News index.

From these observations I concluded that, for whatever reason, Google News must have decided that www.infoworld.com is not a news source (although weblog.infoworld.com is). Which raised the question: How exactly does Google News decide what qualifies as a news source?

When I asked that question of Google spokesperson Megan Lamb she offered the following guidelines which, though evidently not published anywhere, I am reporting here with her permission:

Google strives to be as inclusive as possible, without regard to political viewpoint or ideology, while also providing a high quality experience for our users. Some of the things we look for when evaluating news organizations include:

- The source offers information that is updated regularly.

- It is managed by an organization (not an individual) and includes organizational information on its site.

- The source does not include hate speech or pornography.

- The source does not allow open posting of content without editorial review.

- The source's website is technically conducive to inclusion.

Google News has assured InfoWorld that it does, of course, meet these criteria, and does qualify as a news source. So apparently I was wrong to conclude that www.infoworld.com was accidentally excluded from the club. But what could the problem be, then?

According to Google News product manager Nathan Stoll, the omission is a technical problem rather than an editorial one. The Google News crawler, he says, is a very different beast from the regular Google crawler. And while the regular crawler happily includes our stuff, the news crawler -- for reasons as yet undetermined -- doesn't.

I was surprised to learn this because I've only ever been aware of three user-agent strings (i.e., crawler signatures) broadcast by Google bots:

1. GoogleBot (for the main index)


2. GoogleBot-Image (for images)


3. Feedfetcher-Google (for RSS feeds)

As of today, InfoWorld's problem remains unresolved and is still under investigation. Arguably it is a conflict of interest for me to write about this matter, given that its resolution is in InfoWorld's financial interest and therefore indirectly mine. But the facts that have emerged about the editorial policy and technical nature of Google News seem, well, newsworthy. And since I am evidently a news source I thought I should pass them along.

I like to make a habit of breaking habits, and my distraction-free desktop has turned into a great example of that. For years I've been subconsciously annoyed by desktop clutter, never realizing that the whole time it was just a bad habit -- and one that's surprisingly easy to break. The Mac-based technique shown in that screencast -- hide desktop icons as well as windows when not needed -- has really simplified things for me, and I'm definitely sticking with it for now.

Over on my Windows box, it's even easier to achieve the same effect. You can toggle desktop icons on and off by right-clicking the desktop and then doing Arrange Icons By -> Show Desktop Icons. For how many years have I used Windows without ever discovering that fact? It's scary.

I haven't yet found a native or add-on hotkey to make the Windows icon toggler as quick and easy as DesktopSweeper's Apple-Enter on the Mac. If you know of one, clue me in and I'll pass it along. But meanwhile, the menu-driven approach works well enough. With icons and the taskbar hidden, I'm writing these words in emacs against a featureless grey background. Alt-Tab brings up Firefox as needed.

Here's more scariness. Earlier I needed to bring up Windows Paint to capture and edit a screenshot. As much of a command-line geek as I can be, my first instinct was to reveal the desktop icons, hunt for Paint, and launch it. But in the spirit of breaking habits, instead I revealed the taskbar and did Start -> Run -> pbrush. Crazy, huh? Most people use four or five apps on a regular basis. It's a manageable namespace that fits neatly into the cognitive window of five to seven things. Instead, we've learned to pick that handful of items from a desktop cluttered with dozens of things.

There's an imperfection in DesktopSweeper, by the way, that turns out to suggest a powerful new feature. It hides icons less completely than the Windows hider does. If you touch the part of the Mac screen where a hidden icon lives, it'll show up, which is disconcerting. But it's also the case that if you download a file to the desktop it'll be visible, and that's potentially quite useful. It suggests the possibility that showing icons needn't be an all-or-none affair, and that rules about recency or frequency of use could govern what to show or hide. It further suggests that the Mac's opacity feature -- much lauded but rarely exploited well -- could indicate a spectrum of recency or frequency.

This last idea occurred to me after I added Spirited Away to my Mac setup on the recommendation of Adnan Wasim. "It runs in the background, and hides all non active windows after a predefined amount of inactivity time," Adnan says. A very cool idea, and it works well. But for me the anti-clutter benefit is offset by the distraction cost of having windows wink out of existence at seemingly random times. What if, instead, they just gradually went transparent? The effect would be a bit like Sam Ruby's Fade to white hack, which -- in an effort to wean people away from an obsolete RSS feed -- used a date-based algorithm to gradually turn the text from readable black to invisible white. Dunno how that'd work in practice, but it would be interesting to see.

Several folks wrote to discuss the ways in which hiding distraction and managing clutter are interrelated. On Unix/Linux but not Mac or Windows, multiple workspaces -- each with a set of applications -- afford a powerful way to manage clutter. And of course there are add-ons. Lee Grey's WinZen, for example, is an add-on clutter manager for Windows. His notion is that when you're actually using several windows at once, the process of arranging them as needed -- by dragging and resizing -- is radically inefficient. WinZen is a layout manager makes it quick and easy to achieve a variety of task-optimal tilings.

I guess all this fits into the category of what our parents or grandparents used to call Hints from Heloise, and what we now call lifehacks. It's a bottomless well because there are infinitely many ways for us to interfere with our own productivity, and infinitely many ways to short-circuit that interference. Yesterday, for example, I heard a radio interview with Judy Collins in which she was asked about her work habits. One of her lifehacks is to avoid the "noise of the world" -- radio, TV, newspaper, Internet -- during her early-morning creative time. The hack? She moved her coffemaker into the bathroom so she wouldn't have to leave her bedroom first thing in the morning and face all those distractions.

Here's one final hack for the day. A couple of folks asked how I made that screencast which, unusually, captures an entire Mac desktop. I started with my regular tool, Snapz Pro X, but that quickly bogged down. It saves files very slowly, and with the whole desktop in view things were taking forever.

So I connected my Mac's S-Video port to the S-Video port on my Panasonic PV-GS400, and recorded to tape. It was quite amusing: I set the camera to tape playback, told the Mac to detect displays, and suddenly the Mac desktop was on the camcorder's little screen. For future reference -- mine as much as anyone else's -- the secret trick for recording in tape playback mode is to press the Record and BackLight buttons together.

This was advantageous in two ways. First, I could quickly and easily gather as much raw material as I wanted. Second, the 720x480 format was exactly what iMovie likes to edit, and when I'm laying down audio narration on top of screen video I'd rather do it in iMovie than in Camtasia.

But can you cram a whole Mac desktop into 720x480? Well enough, as it turned out, once I downsized the desktop icons and labels, and downsized the fonts in all the apps. Detail wasn't critical in this case, it was the gestalt of the desktop that mattered. But I'm pretty sure that for a screencast that focuses on an individual application window, 720x480 coupled with small fonts will make this approach quite workable.

Share Your OPML (SYO) is a service whose mission is "to gather a community of subscription lists, in OPML format, and aggregate them in interesting ways." A while ago I asked:

Now that we've shared our OPML, will SYO share it back so we can create and contribute our own data mashups?

1. Can I license my reading list, for example using one of the Creative Commons licenses? From one perspective, that list is just data and therefore not subject to copyright. But from another perspective, that list is a creative work. It uniquely reflects my own engagement with the blogosphere, and embodies much careful selection and refinement.
2. If I do license my reading list, and then I contribute it to a service like SYO, how is its use of my list affected by my license?
3. On what basis might SYO in turn license its aggregation of my list and others? Mechanical aggregation isn't a creative act, but the formation of a unique community of contributors arguably is.
4. Can SYO then license the collective output of those contributors back to me and to others?
5. If I in turn build upon that collective output, how am I affected by its license?

SYO is just one of a many kinds of services that will raise these kinds of questions. Video-sharing sites are another. Clearly you can assign a Creative Commons license to a video that you create. And as blip.tv's Mike Hudack mentions in our interview, his service commendably weaves support for Creative Commons licensing right into its workflow.

It's not at all clear, though, whether the tags, the ratings, the commentary, or the playlists that you create when you interact with these services can be licensed, and if so by whom. The individual contributors? The service? Both?

I presume there is no longer any argument about data export. If I contribute to blip or mefeedia or dabble, I'll expect to be able to easily get out what I put in -- or else I'll go somewhere that does ensure simple, high-fidelity export.

But I also want to extract and creatively reuse the amalgam of my contributions and other people's contributions, within and across these services. That's much trickier. The essential creative act performed by these services is, again, the creation of a community of contributors. How these services respect the creative rights of their contributors, while at the same time asserting their own creative rights, is a thorny question indeed.

In my conversation with Mike Hudack, I sketched a model for online community that I think may help clarify the situation. Services like SYO or blip or Flickr or del.icio.us aren't really communities at all, in the natural sense of the word. They're just services. The organizing principle of each is a data type: OPML files, videos, photos, URLs. Overlaid on top of each, implicitly or explicitly, are natural communities aligned by interest and/or location.

We are all members of many of these communities. Each will need to balance individual and collective ownership in its own unique way. There will be many different solutions to that puzzle, but we won't find them until we stop defining communities primarily by data type and start defining them by where people live -- in real and virtual spaces -- and what they do there.

Mike Hudack, one of the founders of the video sharing service blip.tv, was my guest for this week's podcast. Mike had seen my essay on walled gardens. We discussed blip.tv's current efforts and future to ensure that videos, as well as metadata about videos, flow freely on vthe web. Blip.tv is working with Dabble, FireAnt, Technorati, and other in a collaboration called Video Vertigo (hint: the password is (intentionally) in the authentication dialog box) to federate video-specific metadata, such as viewership stats, as well as general metadata such as tags.

We also discussed the evolution of web video genres, from rips of TV and movies to friends-and-family material to serialized entertainment to -- my current pet topic -- sharing of domain knowledge and experience.

Recent research has shown what common sense always should have told us: Computers multitask way better than people can. As we perform the intellectual work that powers the information economy, our ability to achieve focus and flow is constantly challenged by distraction and interruption.

The paradox, of course, is that interruptions are vital, too. We are all required to be interrupt-driven in ways that vary according to the circumstances of our lives and our work. The trick is to find the right balance. Sadly, by inviting us to interrupt ourselves more than necessary, our software tends to contribute more to the problem than to the solution.

Consider the effects of the graphical user interface. At hospital admitting desks, in accountants' offices, and at video retail stores, I watch people perform tasks for which the desktop metaphor -- with its cluttered surface and overlapping resizable windows -- is at best a distraction and at worst an impediment. [Full story at InfoWorld.com]

After writing this week's column I began thinking about how my entire computer desktop could be more distraction-free and task-focused. I came up with an experimental approach for the Mac, and I documented it in this six-minute screencast. The elements of the solution include DesktopSweeper, a Mac shareware program that hides desktop icons, along with use of the Mac's Apple-Tab switcher (which is like Windows' ALT-Tab) and the Mac's Hide Other Windows feature which is bound to Apple-Option-H. Using all these in combination, I'm exploring what it's like to work in an environment where both desktop icons and application windows are hidden unless you really need them, in which case they're easy to reveal.

The approach I demonstrate in this screencast isn't perfect, to be sure. It's really just a cobbled-together mashup. I'm not sure whether I'll stick with it, or whether I'll investigate "porting" it to Windows. But it's been an eye-opener. Stripping away the unecessary distraction and clutter feels really, really good.

For a bunch of reasons, I'm averse to gasoline-powered lawnmowers. They're loud, they're smelly, and as I learned when my dad bounced a rock off his leg many years ago, they can be dangerous. So I've always preferred old-fashioned reel mowers. Currently I use a Scotts Classic which I bought six years ago from Clean Air Gardening. My friends and family think I'm nuts to use it, but in fact these modern reel mowers are light and easy to push. The effort required to mow the lawn is mostly just the effort required to walk the property, and that's the same whether you're behind a roaring two-stroke engine or no engine. And in the latter case you can walk in bare feet, hear birds sing, and enjoy a bit of eco-smugness.

In recent years, though, things weren't going so well with my mower. Part of the problem was that I'd gotten the blade adjustment out of balance. For posterity I explained here what I'd done wrong and how I fixed it. But that wasn't the whole answer. Over time the pinion gears inside the wheels had gotten chewed up, and the mower began to lose traction. A friend remarked: "So, they designed a product that you would start out loving and then, gradually and imperceptibly, grow to hate." Exactly. A note to product developers everywhere: Don't do that! In this case, making those gears out of plastic saved pennies at a cost of untold dissatisfaction on the part of customers and damage to the reputation of the whole class of products.

Since I'm already an unofficial support site for reel mowers -- people find my last blog entry and write with questions -- I decided to do something more definitive. So this weekend I made a six-minute video that documents how to use, and maintain, the Scotts Classic.

As I discovered back when I fixed my printer, video is the ideal way to share certain kinds of knowledge and experience. In that example, FixYourOwnPrinter.com provided both the printer repair kit and the video which was overwhelmingly the reason why I bought the kit from that supplier.

But in an era of commons-based peer production there will be increasing numbers of folks who will package their knowledge and experience in video form, and publish it freely, just because they can. Everyone's an expert on something. If its quick and easy to document some aspect of that expertise, and if doing so makes you a global authority on that topic, people will choose to do it.

If I'm right about where this is headed, the video-sharing sites will soon offer more than cute animal tricks, stupid people tricks, and experimental artwork. They'll start to be windows that open on many areas of knowledge and experience, the sharing of which will accelerate the production of new knowledge and the deepening of experience.

I've mentioned Yochai Benkler a few times recently. His half-hour talk at PopTech, which just appeared on ITConversations, is an excellent summary of his analysis of the economic transformation that these new modes of knowledge sharing will bring.

For today's Friday podcast I interviewed Bill Donahue, who has a new product called Openfount that makes intriguing use of the Google Web Toolkit and Amazon S3. Sadly I made a boneheaded mistake as I began to edit the audio file, and wound up losing most of it. So after giving myself a dope slap I resolved to write up an explanation of Openfount as informed by our conversation.

Openfount's big idea is that a solo developer ought to be able to deploy an AJAX application to the web without worrying about how to scale it out if it becomes popular. If you park the application's HTML, JavaScript, and static data files on Amazon's S3 storage service, you can make all that stuff robustly available at a cost that competes favorably with conventional hosting. But applications that only talk to static data files aren't very interesting or useful. You'll want them to talk to services as well.

For reasons of security, though, the XMLHttpRequest mechanism at the core of AJAX can't reach beyond the origin server. And S3, as the origin server, can host content but not services. So what to do?

Enter Openfount's Queued Server. Instead of contacting a remote server directly, the AJAX client enqueues requests on S3, and reads responses from S3. Elsewhere on the Internet, you locate the server that reads those requests from S3, processes them, and writes the responses back to S3. That server can offer services directly, can proxy for remote services, or both.

Note that this server works more like a client, in the sense that it's not listening for requests on a port, but rather polling S3. That's obviously non-optimal, but according to Bill Donohue it's entirely workable. And you can use many of these servers if needed. Openfount uses a protocol to guarantee a one-to-one mapping of requests to servers.

Another intriguing aspect of this setup is that the AJAX client and the Queued Server are truly decoupled. The client doesn't know the server's IP address, and indeed the server need not even have a permanent IP address. In this architecture, Amazon's S3 namespace plays the role of DNS.

The Queued Server is Java-based but, because it's so simple -- all it really does is treat S3 as a queue -- versions for other languages will be easy to create.

On the AJAX side, Openfount extends the Google Web Toolkit (GWT), adding support for the S3 API, for XML and SOAP processing, and for cryptography. I'd known that GWT apps are written in Java and then translated into JavaScript, but hadn't thought about the benefits of that arrangment. One is that you can use the Eclipse IDE, with its robust debugger, to develop your AJAX app. Another is that the Java-to-JavaScript translator is a very general-purpose tool that can be used to port Java libraries to the AJAX environment.

Although Openfount is currently GWT-based, Donahue says he's already been asked to support other AJAX toolkits. First on the to-do list is Dojo.

I find the whole thing fascinating. A couple of years ago, there was a lot of talk about how content delivery networks like Akamai were also becoming service delivery networks. You'd write your service in Java or .NET, configure it for a well-specified deployment target, and then drop it into the network where it would replicate out to the edge.

I haven't seen a whole lot of action on that front. Meanwhile S3, which was never designed as a service delivery network, is being pushed by its users into that space. Perhaps "pushed by its users" is the key point. As a general-purpose infrastructure, S3 makes no assumptions about Java or .NET or Web services standards. Users who push it in the direction of service delivery will show Amazon how they want it to work. Then Amazon can, and I suspect will, react accordingly.

S3 just keeps getting more interesting. It's like a Rorshach inkblot. People see it as backup, cheap bandwidth, cached content delivery, application hosting, you name it. I'm really curious to see where this goes next.

Update:

Julien Couvreur says via email:

Overall, OpenFount seems like a bad idea and I remain unconvinced of
the benefits of this architecture.
 
Doesn't Amazon already have a web service for queueing? Why use S3?
Also, S3 is a paying service. Doesn't polling add to your cost, by
creating lots of small requests from the client and from the back-end
server fulfilling the queued requests?
 
Also, in order to scale, it's not enough for your front-end servers to
be "web scale" like S3. Your back-end server, which is doing the
actual processing also needs to handle the load. S3 appears to be used
for load balancing, which is not the biggest problem in scaling...
 
Leveraging S3 for scale is nice, but it seems that it should be
combined with a cross-domain solution to support the most dynamic bits
(transactions and such).

Here was my response:

> Doesn't Amazon already have a web service for queueing?
 
Yes [SQS, or Simple Queue Service], and it would be preferable. But
it's only in beta as yet.
 
| Doesn't polling add to your cost, by creating lots of small
| requests from the client and from the back-end server fulfilling
| the queued requests?
 
Not if you only check headers, I believe, but good point, that should
be clarified.
 
| Leveraging S3 for scale is nice, but it seems that it should be combined
| with a cross-domain solution to support the most dynamic bits
| (transactions and such).
 
Yes, Bill realizes that. And I think we both saw this as an
exploratory and iterative thing, which could lead AWS to evolve its
API just as Google Maps was led by its early developer/users to create
and evolve an API.

Bill Donahue adds:

Here's the way I would summarize the Queued Server idea after getting
feedback:
 
1. Is it a perfect solution? No. But for users who don't have the
   ability or money to set up a web infrastructure, it is the best
   solution available now.
 
2. Could it lead to an almost perfect solution? Yes. A couple of
   optimizations to S3/SQS would make it almost perfect.
 
Amazon seems very willing to pursue these ideas, so we'll see what
happens.

I like Richard Wallis' take. On his blog he writes, in part:

Globally distributed SOA seems to be looming out of the mist, and it is a different shape to what was envisioned a few years back. No sign of the network of UDDI servers scattered about the planet, just lots of loosely coupled applications sitting above utility core services and domain specialist platform providers.

The central premise of WinFS, as originally planned, was embodied in its notion of relationships. Because these were first-class constructs in the database, you would be able to answer questions like "Where are the recent documents related to project X?" The relationship here, between documents and projects, can exist because both documents and projects (and people) are formally represented as first-class constructs in the database. But where do they come from?

Relationships among items begin as relationships among people that form in fluid and ad-hoc ways, across platform boundaries, mediated by open protocols. Today I find things mostly by recalling who, and then by searching email and the web -- ideally aided by self-assigned or group-assigned tags -- to discover what and when. Designed in the waning days of personal computing, WinFS failed to acknowledged the emergence of social computing and its transformative power.

If there must be an epitaph, let's write it for personal computing rather than for WinFS. The years of hard work invested in what Quentin Clark calls the object/relational/XML trinity of data-management technologies can still bear fruit -- if they're cultivated on common ground. Six months ago I'd have doubted that could happen. But now that Ray Ozzie has announced Simple Sharing Extensions for RSS, demonstrated LiveClipboard at ETech, and inherited Bill Gates' old job as chief software architect, I'm thinking maybe it can. [Full story at InfoWorld.com]

Microsoft's Matt Augustine recently advanced the SSE/LiveClipboard story in a presentation at Supernova that's nicely captured in this screencast. I watched it with mixed feelings. It's wonderful to see Microsoft working on two-way calendar synchronization based on open standards. But first things first. It's been four months since the debut of Live Clipboard. I studied the implementation and made my own screencast exploring how it works, but I've yet to encounter a single use of this excellent web-based copy/paste mechanism in the wild, either at Microsoft.com or elsewhere. If I'm missing the boat, somebody please point me to a list of sites and I'll happily stand corrected.

Achieving viral adoption of the basic copy/paste technique -- by whatever means necessary -- must precede, and will crucially inform, efforts to establish more advanced techniques like two-way item sharing. The analogy to NetDDE resonates powerfully for me. Years ago a Microsoft executive told me (a bit patronizingly) that once in a while an unbeatable platform comes along, and when it does you had better join it. The platform he was talking about was Windows 3.1, a system that made great use of the local area network but was still fundamentally about personal computing. Its very cool NetDDE (Network Dynamic Data Exchange) technology was the spiritual ancestor of Live Clipboard and SSE, but NetDDE never amounted to anything because local area networks lacked the critical mass that powers social computing.

Today's unbeatable platform is the web, and I'm intensely curious about the ways in which Microsoft will join it. Today, Live Clipboard and SSE are in position to tap into an ecosystem that was never available to NetDDE. But that will only happen if the citizens of that ecosystem are truly stakeholders and co-creators. The human chemistry involved is tricky even when you approach the web in all the right ways, as Microsoft has laudably done this time around.

If you approach the web in the wrong way, as Microsoft did with WinFS, good luck. "Why would anyone want to build applications targeting a proprietary Windows-only file system?" asked Dare Obasanjo. What's more, how could such an effort even be possible? Relationships that form and evolve on the web will be systematized, to they extent they can be, by an iterative process that combines grassroots participation with exploratory top-down design.

I can't resist this illustration of the shift from personal to social computing:

iwx.infoworld.com/iwx/?tags=winfs

The items collected by that query describe a story arc that begins with 2003's Replace and defend and will end (for now) with this coda. Pick up any piece of that thread, give it a shake, and you'll unfold a tapestry woven of connections among people and information. The loom on which that tapestry is woven isn't the personal Windows machine, it's the social web. When Microsoft truly embraces that model, it will be able to bring some great technology to the party.

A couple of weeks ago I uploaded a home video to my Amazon S3 account and sent the link around to friends and family. Featuring kittens and bunnies, it's the kind of thing you'd expect to find in the cute animals category on the various video sites. Over the weekend somebody suggested that I upload it to YouTube, so I registered there and did that. YouTube reported:

Uploaded (processing, please wait).

Twenty-four hours later, it was still processing. Meanwhile, I received my June Amazon S3 bill for the princely sum of fifteen cents:

Since that 15-megabyte movie is the only thing I'm storing on AWS at the moment, my fourteen cents worth of bandwidth usage translates to about 47 downloads. Let's round it to 50 and consider the following progression:

  views     $$$
  -----     ---     
     50    0.15
    500    1.50
   5000   15.00
  50000  150.00
 500000 1500.00

The vast majority of friends-and-family videos will never exceed 50 views a month, for which fifteen cents a month is effectively free. And even at 500 views a month, you'll hardly notice the buck fifty.

But what if your video becomes popular? The top 20 videos on YouTube on any given day average about 50,000 views, and you'd certainly notice the hundred and fifty bucks that would cost on S3, not to mention the thousands you'd pay for a month of that level of interest.

For a variety of reasons, I've been thinking about the kinds of services provided by S3 and by video sites like YouTube, and about how such services might fruitfully combine.

The long processing delay at YouTube occurs, I suppose, because YouTube transcodes the files people upload, from QuickTime or Windows Media to Flash, which is currently regarded as the most universally viewable format. (Blip.TV handles this more intelligently, by the way. It publishes your primary file right away, and adds the Flash video later.) Video transcoding requires a lot of computational horsepower, so it's not surprising to see it become a bottleneck.

One consequence of that bottleneck is that YouTube is not useful for near-realtime citizen journalism. When I documented the flooding last fall in Keene, NH, my first upload was available and in circulation during the event. A long processing delay would have been intolerable.

In my case I published the file to my webserver, but most people can't easily post a large media object to the web. Hosting is one major service provided by YouTube, Blip.TV, and the rest. The other major service is convenient sharing, tagging, ranking, and community participation.

In principle the video services can, and arguably should, be remixed along both of these axes. In practice we're not there yet, but it's interesting to imagine the possibilities.

Take my friends-and-family video for example. I originally posted it to my AWS account at this URL. That was fine for friends and family, but if you hit that URL now you'll be redirected to YouTube.

Could I establish the S3 URL as the canonical one for this video, while creating the option to delegate hosting and/or community services as needed? Today, YouTube and Blip and the rest are walled gardens. Each creates its own namespace to which all community activity refers. But there's no straightforward way to associate this YouTube URL with this Blip.TV URL. As a result, there's no way to combine the service-specific tagging, comments, and viewership data for each of these instances. And there's no way for a blogger to refer canonically to the video, or for the blogosphere to aggregate such references.

Now consider this scenario. I upload my video to S3, or to some other storage service in the cloud where I choose to manage a canonical chunk of web namespace. (As I mentioned in my S3 writeup, its bucket namespace is global, which means that I control http://s3.amazonaws.com/jon so long as I choose to.) The video is immediately available at that URL. I make it public or, as noted in my S3 writeup, private at an unguessable and possibly time-limited URL that I communicate only to friends and family.

Let's say I make it public, because I hope to attract attention. In the unlikely event that I succeed wildly, the pennies I pay S3 could add up alarmingly.

BitTorrent is one way to defray the cost of distributing an object that becomes hugely popular. S3 can automatically produce a BitTorrent URL for any hosted object -- a very cool feature! But that only helps with distribution. Viral communication is the other half of the equation. That's what YouTube and the rest provide, in addition to hosting. And they do it in ways that optimize for communication with, rather than across, the walled gardens.

I'm looking for ways to break down those walls. Merely redirecting my S3 URL to YouTube, as I've done, is a very imperfect solution. In the unlikely event that my cute animals video becomes popular, I'll be spared the cost. And the odds against that unlikely event are reduced somewhat by submitting the video to YouTube. But my YouTube instance doesn't know about my Blip.TV instance, and vice versa.

My S3 instance does, however, know about both of those. If you point a raw HTTP client at my video on S3 you'll see, among the other HTTP headers, these:

x-amz-meta-blip: http://www.blip.tv/file/47189
x-amz-meta-youtube: http://www.youtube.com/watch?v=YrrCuCyQVNk

Clearly that list can be extended to other video services. Now, I'm sure there will soon be -- perhaps already is -- a service that takes your video and submits it to a bunch of video services, while providing a canonical URL for the video. But it strikes me that S3 is the kind of foundation service that could support that kind of arrangement not only for video, but for any kind of collection.

One detail that's missing: I wasn't able to retain my video on S3. I'd hoped to be able to leave it in place, and tell S3 to issue a server-side redirect to one or another of the video services. Ideally I'd be able to have S3 do that intelligently, according to preferences that might be expressed in the request. But as I read the docs, there's no way to get S3 to emit the HTTP Location header that would accomplish a server-side redirect. So instead I replaced the video with an HTML file containing a client-side redirect -- i.e., the old <META HTTP-EQUIV="refresh" CONTENT="0;http://..."> trick. If there is a way to to a server-side redirect, I'd like to know about. It'd be nice to be able use S3 as canonical storage as well as canonical namespace. It really would cost pennies to keep files there, safely backed up (I presume), and then transmit them to other services, only once for each service, by way of an S3 private/unguessable URL.

In the big picture, none of this matters until we establish the idea that naming, storage, content management, tagging, and community participation are separable concerns. Achieving that separation is technically feasible and very much in the interest of everyone except, of course, the builders of the walled gardens. For obvious reasons they won't want to go there. But I wonder for how long they'll be able to remain insular.

Lou Rosenfeld is my guest for this week's podcast. Fellow superpatron Edward Vielmetti put me in touch with Lou, with whom I share an affection not only for Ann Arbor, Michigan, but also for a cluster of topics including information architecture, search analytics, print and online publishing, designing for usability, tagging, and microformats. We had a great conversation!

In the June episode of The Screening Room I receive an education from Chris Harrington, who runs the Active Interface consultancy, about an emerging standard for online analytical processing (OLAP) called XML for Analysis (XMLA).

To be honest, XMLA wasn't on my radar screen until OpenLink Software's Kingsley Idehen put it there. In our recent podcast interview he announced that in addition to releasing an open source version of the Virtuoso database, there would also be an Open AJAX Toolkit whose data-binding capabilities are abstracted in terms of XMLA, an interface provided by Virtuoso and by SQL Server.

"XMLA?" I asked Kingsley. "Uh oh," he replied. "If you're unaware of it, that means most of the AJAX/blogosphere/Web 2.0 folks are too."

Enter Chris Harrington. His specialties are enterprise information portals and web data visualization. Like Kingsley, he has recently found that the combination of AJAX and XMLA makes a nice platform for what Chris calls enterprise mashups.

In the first two-thirds of the screencast he focuses on a command-line tool based based on the Windows Scripting Host (WSH), which he uses to query XMLA servers and visualize results. Along the way he points out interesting and useful things about WSH and about Office's ChartSpace component.

In the last third of the screencast, Chris demonstrates a really impressive AJAX application he's been developing. Using it, he can interactively choose data sources, compose and edit MDX (multidimensional expressions) queries, and chart results.

Although XMLA was originally a Microsoft/Hyperion initiative, it's attracting wider interest and seems headed toward standardization. At the conclusion of the screencast, Chris points to this proof-of-concept interoperation between his client and Mondrian, an open-source Java-based OLAP server that can turn a number of SQL databases, including MySQL, into XMLA providers.

How will business intelligence meet Web 2.0? Chris Harrington and Kingsley Idehen think AJAX and XMLA will form the bridge.

Starting in mid-June, [the city of Washington] DC began releasing operational data from a variety of city agencies to the Internet, in several XML formats including RSS and Atom.

...

If you've ever visited Adrian Holovaty's award-winning ChicagoCrime.org, you can see what this might mean for Washington. Here's a critical difference, though. Holovaty had to devote a considerable amount of effort to screenscraping the Chicago Police Department's Citizen ICAM website in order to extract the data, and still more effort to geocode it. I know, because I've been there myself too many times, that while he was writing that screenscraper he was mentally screaming: "Just give me the data!"

DCStat is doing just that. The Atom and RSS feeds summarize activity, and all the details -- including latitude and longitude coordinates -- are included in DCStat's own XML format. Following the initial launch of the service request feed, new ones will appear at roughly two-week intervals throughout the summer and fall. These feeds will contain flows of raw operational data about crime (incidents, arrests, charges), property (real and vacant), housing code enforcement, business and liquor licensing. [Full story at InfoWorld.com]

The DCStat program was also the subject of last week's podcast with DCStat's director, Dan Thomas, and with DC CTO Suzanne Peck. This week Dan alerted me to the first real mashup based on DCStat's service request feed. I've also heard from Jim Willis, the director of eGovernment for Rhode Island, about their Flickr-like API for open government services. It's all good, bring it on!

If space in the magazine had permitted, here are parts of the podcast I would have quoted:

Suzanne Peck on cross-agency correlation:

If as a citizen the things I am most focused on are: Am I getting the services I'm paying for?, Am I safe?, Are my children well-educated? -- the answers to those questions don't come from any particular vertical agency, the answers come horizontally, in an integrated way, across a number of agencies, each of which serves up part of the answer.

Dan Thomas on democratic access to public information:

We're moving more towards a true democracy. This information has always been accessible, but it's typically been accessible to folks who have had the time and resources to go get it.

Suzanne Peck on measuring government's performance:

One of the principal candidates now running for mayor says that DCStat is the mechanism by which he is going to bring accountability to every leader and manager in the city.

The summer reading list is a time-honored tradition. For the podcast era, why not a summer listening list? The much more recent blogospheric tradition of copycat listmaking -- e.g., post an item about five books you like, or ten favorite OS X applications -- provides the model. So here, in no special order, are five podcasts that have entertained and informed me. I'll tag this item summerlistening in hopes that when next I visit del.icio.us/tag/summerlistening and technorati.com/tag/summerlistening those buckets won't be empty.

1. John Willinsky re-imagines education

Edublogger Brian Lamb pointed me to John Willinsky's talk at the UBC Okanagan's 2nd Annual Learning Conference. Willinsky's thesis is that widespread access to existing knowledge, coupled with widespread ability to publish new knowledge, can (and should) radically transform education. Along the way you'll be treated to an erudite history of academic publishing, delivered as standup comedy.

Link: people.ok.ubc.ca/ctl/Willinsky%20.mp3

2. Brian Schweitzer on ending the oil addiction

Chad Wandler at Split Rock Communications pointed me to this talk from the 2006 Sustainable Energy Forum, which also includes excellent talks by Bill McKibben, Lester Brown, Joe Tainter, and others. Schweitzer is a former soil scientist who, as the hugely popular governor of Montana, is walking the talk on alternative fuels. I'd never heard of the guy before listening to this talk. Now I want him as the next US president.

Link: www.beyondpeak.org/files/SEF06_Schweitzer.MP3

3. Janine Benyus on biomimicry

From PopTech 2004, by way of ITConversations, comes this fascinating talk on how we can learn from all the design work that nature has done. Here's the ITConversations blurb:

She names an emerging science that seeks sustainable solutions by mimicking nature's designs and processes (e.g., solar cells that mimic leaves, agriculture that looks like a prairie, business that runs like a redwood forest).

Link: www.itconversations.com/shows/detail241.html

4. Jed Emerson on blended value

Recently several folks have asked me what I think about Bill Gates turn to philanthropy. I've responded with pointers to this talk on why, and how, the getting phase and giving phase might be integrated. It's from the 2005 Net Impact conference, by way of ITConversations.

Link: www.itconversations.com/shows/detail929.html

5. Yochai Benkler on the wealth of networks

Woven through two of my own recent interviews -- with Andy Singleton and with Nathan McFarland and Benjamin Hill -- are references to Yochai Benkler's Wealth of Networks. This podcast of a recent talk by Benkler, by way of Joseph Hall and Judd Antin, is a great introduction to the economics of social computing and commons-based peer production.

Link: dream.sims.berkeley.edu/~jhall/Yochai_Benkler_Boalt_27Apr2006.mp3

Update: Contributions to del.icio.us/tag/summerlistening have included pointers to various collections that I hadn't known about, including the TED series, the PARC Forum, and BayCHI (Bay Area Special Interest Group for Human-Computer Interaction). Separately, by way of radar.oreilly.com, I found out about the LongNow seminars. Here's a tip: although none of these are podcatcher-friendly, you can easily synthesize RSS feeds for them that will work in iTunes or another podcatcher. Webjay affords one way to do that. Here's a quick screencast demo that shows how to turn a page containing MP3 links -- in this case, the BloggerCon page -- into an RSS feed with enclosures:

Ami Hendrickson has quite properly indicted me for the crime of business-speak:

My current favorite quote (from the June 19, 2006 issue of my husband's hardcopy of InfoWorld, a magazine intelligible only to hardcore geeks):

"By syndicating metadata, I'm inviting others to more richly contextualize their aggregations of our stuff."

This gem wasn't even buried in article text. No -- the editors loved it so much they called it out and featured it as a 28 point red font banner running right through the article in which it appeared.

Now, I have no doubt that Jon Udell, who penned the words in question, is a brilliant analyst and programmer. Heaven knows I'm not. And I realize that his words are intended not for plebes like me, but for the Chosen Few who not only understand them, but are also interested in things like aggregated search results, metadata streams, and sentences that include terms like "iwx." [Muse Ink: Say What?]

On rare occasions I've escaped the orbit of the geek ghetto and shown pieces of this vision to wider audiences. But it doesn't happen nearly often enough, and lapses into business-speak -- or tech-speak -- are surely a big part of the reason why not.

My favorite success story is the following (paraphrased) message from one of the many librarians I've corresponded with throughout the course of my LibraryLookup project:

Oh, I get it. LibraryLookup doesn't work with our catalog system because we bought the wrong kind of software.

When she wrote that, she implied a whole lot of unintelligibly geeky stuff: the software didn't expose a RESTful interface, wasn't capable of ad-hoc query, could not support aggregation or intermediation. She had none of this language, but she clearly understood the principles behind them. More clearly, I venture to say, than the vendor who created the catalog system or the buyer who acquired it for her library.

We can invent anything. It's just software. The real challenge is to show people what's possible, and to engage them as partners and co-creators. To meet that challenge we'll need to be able to tell stories that make immediate sense to everyone.

In terms of explaining and motivating my power search and metadata explorer projects, I guess I'm back to the drawing board.

Open government is the subject of this week's podcast with Dan Thomas, who directs Washington DC's DCStat program, and Suzanne Peck, chief technology officer for the city. I first met Dan at our SOA forum in November. Recently he contacted me about an exciting new initiative that soft-launched last week: live data feeds that will enable any interested citizen to track the performance of agencies that deliver city services.

As a proof of concept, I made this quick-and-dirty mashup that projects data about street repaving and gutter repair onto a map of DC. It's exciting to imagine all the ChicagoCrime.org-like application these data feeds will open up, and how much more easily they'll be accomplished when it's not necessary to scrape websites for the data.

Very cool stuff!

Update: Transcript available.

Earlier this month Jeremy Wagstaff, who writes for the online edition of the Wall Street Journal and for the newspaper's Asian edition, wrote an article about screencasting. He announced it on his blog, in an entry that quoted the intro to the article and included a useful collection of screencasting-related links. But the article itself was available only to WSJ.com subscribers and to readers of the print edition of The Wall Street Journal Asia. I did get to read it, thanks to an acquaintance who forwarded me a copy. He did this in a bizarre fashion, though, by sending me a JPEG attachment, reminiscent of the way my mom snail-mails me clippings from newspapers she reads.

I was curious to see what the effect of the story might be. On the one hand, I reasoned it would be negligible. No public URL, no network effects. But then PaidContent.org reminded me that WSJ.com is considered a major success not only in the realm of paid online circulation, but also in comparison to newspapers:

(I captured that data at this URL, though I can't find the item there today.)

So I revised my expectations. Perhaps I'd see a large referral effect after all. There was a useful benchmark for comparison. About a year earlier, David Pogue's New York Times blog had mentioned my Wikipedia screencast. I can't find that item now, either; the recent reboot of that blog seems to have ditched the older items. But anyway, now the WSJ.com was pointing to the same item. What would be the relative effects of these two events?

The NYTimes.com produced a flood of referrals, the WSJ.com a trickle. Of course that Wikipedia screencast might have been old news to those WSJ.com subscribers who, a year earlier, would have found it much more interesting. For this and other reasons it's not an apples-to-apples comparison. But I found it interesting nonetheless.

The article is now available on the web, for a while anyway, because the TechSmith folks, whose Camtasia software I use to make screencasts, sponsored a reprint. Nobody has linked to the reprint, though, with exception of me doing so here for my own archival purposes, and my guess is nobody is likely to.

This may be a successful model of publishing, but it seems to me a curious definition of success.

In an October 2004 blog essay entitled The energy web I bemoaned the fact that, during the run-up to that year's U.S. presidential election, none of EPRI's [Electric Power Research Institute] excellent analysis and solid planning was being championed by either of the political parties. In the end, though, I concluded that neither could contribute much. Information technology is both the key enabler and a prime beneficiary of the EPRI vision, and information technologists -- not politicians -- will make the difference. "Maybe," I mused, "a new entrepeneurial partnership between energy and IT is all we really need."

Last Friday I got a glimpse of what such a partnership might look like. For my weekly podcast, I interviewed Mike Frost, CEO of Site Controls, a three-year-old Austin startup that's focused on the part of the energy web that can be built out now, for profit, with near-term return on investment and a growth path that could eventually produce macro-level network effects. [Full story at InfoWorld.com]

This week's column captures some of the key points from my 25-minute podcast interview with Mike Frost. I've extracted a few more of his comments here:

On the franchise-store opportunity: Take a Michaels store, or a Party City. Those things are great big boxes of conditioned air. If you go in and push setpoints five degreee, nobody's going to notice, we've tested it. We can accomplish the same thing that you can do with hundreds of thousands of homes, but if you do it in these retail environments, every one of those is equivalent to 20 or 30 homes, and you can do it with low consequence instead of high consequence.

On building railroads: In order for the intelligent grid to work in most areas, it requires a lot of government subsidy. There typically isn't real strong return on investment. That's why we've chosen the specific segment we're addressing, as a way to get things started, because the ROI for the customer on our end is pretty fast. If Petco can pay to build the railroad -- and it makes sense for Petco, because they lose less inventory, they run their buildings smarter, they save energy, and they have less than 24-month ROI -- then they're paying the freight. Where in other segments -- residential, commercial -- there's nobody to do it, they're all waiting for the government.

On smart energy management as good PR: When things get really bad in California during rolling brownouts, Petco has discovered that by putting a sign on the door that says we're doing our best to conserve energy today, please pardon the inconvenience if it's a little bit warm and a little bit dark -- because they cut the customer lights and they push the setpoints up to 78 degrees, so as a result they shed 30 to 40 percent of the load they'd normally be using -- they find that, from a customer awareness perspective, man, it's great PR. At some point there's going to be a new level of social consciousness in these large retail brands. You already see it at Starbucks, you see it at Petco, it's just going to become the way to behave.

It was a great interview, and has prompted me to think again about transcription. An earlier interview, which included CastingWords' Nathan McFarland, was transcribed by his MTurk-enabled service, and I published and discussed the results. It worked out really well, and I'm thinking about doing it on a regular basis. Of course there aren't only minor corrections to make. Even audio that's edited pretty carefully, as my interviews now are, begs to be rewritten for the page. The trick will be to find the sweet spot at the intersection of time, effort, and quality.

Final update: Jared Benedict has made the same voluntary decision that I did, and has an excellent writeup here. He writes, in part:

Contrary to posts on Boing Boing and elsewhere, Jon Udell and I did not receive a "nastygram" or formal cease and desist letter. Rather we received friendly emails from Ms. Meister, This American Life's webmaster, making a request to take down the hyperlinks and RSS feeds, or she'd regrettably have to get lawyers involved.

...

While I am confident that I am breaking no law, I am respecting TAL's wishes by taking down the podcast and archive page which points to their MP3's. This American Life has decided to take the bizarre approach to Digital Restrictions Management (DRM) by asking nicely -- which I suppose is better than using some Windows only Microsoft Media Player DRM or Sony Rootkit DRM.

Further update: At WBEZ's request, I have voluntarily removed the item earlier posted here, which included the notice mentioned in this item's title. Although that notice invited me to "put the word out," publication in this way evidently wasn't intended.

Update: Although This American Life has indeed switched to MP3 format, in a way that enables deep linking and offline listening, the show nevertheless endorses only online listening to what's hosted at its site. The MP3 switch was made for the convenience of listeners who prefer PC-based MP3 players to PC-based RealAudio players.

Conclusion: I'm leaving that RSS feed empty, save for a single entry pointing here. Was the switch to MP3 a tacit endorsement of podcast-style distribution? Certainly a number of people who've communicated with me today took it that way. But however impractical TAL's current policy may be, I respect the show's integrity and idealism, and I hope the podcasting question will get resolved one of these days.

For an internal IDG newsletter I was asked to pick the industry buzzword that most annoys me and write a brief essay explaining why. I chose user-generated content and wrote the following:

Everything about this buzzphrase annoys me. First, calling people "users" is pernicious. It distances and dehumanizes, and should be stricken from the IT vocabulary (see Those clueless users) as well as from the publishing vocabulary. IT has customers and clients, not users. IT-oriented publishers have readers, not users.

Second, "content" is a word that reminds me more of sausage than of storytelling (see Sausage, traffic, and clueless users). As writers and editors we don't "generate" "content," we tell stories that inform, educate, and entertain -- or should.

Now that the original vision of a two-way web is finally made real, we can distinguish between amateur storytellers (in the best and highest sense of amateur) and professional storytellers. Thanks to the contributions of the amateurs -- who are of course professional practitioners of the disciplines that we "cover" -- we can tell deeper, richer, more well-informed stories about the products and services they create, and the work they do. Those stories are valuable, and the business I want to be in is based on that value, not on the "monetization" of "user-generated content".

It's not enough to merely be annoyed, so in the spirit of constructivism I'd like to suggest some positive alternatives. Two ideas come to mind. The first idea is Eric von Hippel's notion of lead-user innovation. In the publishing world, the prevailing notion of user-generated content is that you can get lots of people to blog, or comment, or annotate, or otherwise freely contribute "content" that you get to "monetize." The problem, though, is that lead users -- i.e., the most innovative and influential readers (and advertisers) -- are also the ones most willing and able to publish effectively on their own. Enlightened 21st-century publishers will create value by further empowering, and then collaborating with, those lead users.

The second idea is implicit in John Willinsky's re-imagining of education. Although it's too late to expunge the word user from von Hippel's formulation, Willinsky thankfully did not coin the phrase user-generated context. So I will instead propose reader-created context.

To briefly summarize Willinsky's argument, there are two prerequisites for effective reading: motivation, and context. Life circumstances either do, or don't, provide the necessary motivation. When they do, it comes down to a question of context. Education either did, or didn't, supply the necessary context. If it didn't, that context can be acquired by means of what CJ Rayhill calls on-demand learning.

Much of own work -- in tagging, in intelligent search, in screencasting -- aims to empower readers, listeners, and viewers to create context and learn on demand. Enlightened 21st-century publishers will create value from that kind of empowerment too.

Mike Frost, CEO of Site Controls, joins me for this Friday's installment of my Interviews with Innovators series. A couple of years ago I wrote an essay called The energy web about a set of ideas that matter more with each passing month. I'll say more about this in my InfoWorld column next week, but as you'll hear in this interview, I'm really excited by the focused entrepeneurial approach that Mike's company is taking. "We can't wait for the government to build the railroads in this case," he says, and I violently agree.

This is Friday podcast #11, and I'm really liking the way the series is turning out. Now that I've got most of the production issues nailed, I can focus on having -- and, crucially for me, editing -- interesting conversations. With Doug Kaye's generous help I finally sorted out how to do a reasonable job of telephone recording. You can pick his brains on this topic too, by listening to his talk entitled Recording interviews and phone calls -- the version from the Podcast Academy at Boston University is the one I've heard. I also learned a lot from Daniel Steinberg's talk on the subject of audio editing. At some point I'd like to contribute something back to this knowledge base I've been tapping, and for me the best format will probably be a screencast that illustrates aspects of the editing process.

All this artifice does raise questions, and I wrestled with them in The audio digital darkroom, which is the transcript of a podcast I made last year. I think the conclusion I reached there is the right one:

There's no question that our media landscape is growing more complex. Sometimes the filter of post-production will be available to us, and sometimes it won't. When the filter is available we'll usually prefer it for the sake of narrative flow. But we'll always know how the magic is done, and we'll always be able to peek behind the curtain.

Update: Full transcript now available.

"Things can be found by search," [Wired's Kevin] Kelly adds, "only if they radiate potential connections." Yes, and that leads to a more nuanced view of search than the one Google and its competitors have popularized. For the past few months I've been revamping search on InfoWorld.com. Fulltext search works more effectively, but it's also augmented by streams of metadata and by RSS syndication. It's all about making the site a better connection engine.

...

Like many sites, InfoWorld.com offers a canned set of topical RSS feeds Now that every iwx view can be seen through an RSS lens, that set is vastly enlarged. Suddenly we have feeds for Vista, for Cisco, and for many other topics for which obvious tags exist. Cool! But not in the obvious way. Before iwx offered an RSS feed of InfoWorld's Vista articles, del.icio.us did. And frankly, neither version is very interesting in and of itself. If you want to syndicate articles about Vista, ours are only some some of the ones you'll want to see in that feed.

Aggregated views are the ticket. And the key point is that the iwx version of the feed encourages smarter aggregation. Metadata is what makes the interactive experience in iwx more compelling than its del.icio.us counterpart. By syndicating that metadata, I'm inviting others to more richly contextualize their aggregations of our stuff. If other publishers will return the favor, I'll gladly make better use of theirs.

Any takers?

This week's column asserts a key tenet of information management. When we use our domain-specific metadata to enhance query and search, we can locally outperform the global engines. It also proposes a corollary. Even though that metadata confers local advantage, we should still syndicate it out to the network at the highest possible fidelity.

By way of walking the talk, every iwx query now yields a metadata-enriched RSS feed that enables smart aggregation as well as direct use in feedreaders.

I'm hoping that the next time I run this query I'll find more and better examples of metadata-enabled mashups -- based not only on the metadata foundation I've laid, but also on similar foundations elsewhere.

Until yesterday I'd only tire-kicked Google Calendar. I couldn't use it for real until I loaded it with real data. Last night I finally got around to doing that. In my case, the export/import path led from Mozilla Calendar to an XML representation of iCalendar format to Atom and then into Google Calendar. The hero of the piece is Joe Gregorio, whose wonderful Sparkline service I highlighted a while back. Joe is also, and rather more notably, the author of both the Atom Publishing Protocol and Python's httplib2 library, two projects that came together to facilitate my transfer of calendar data (see script below).

Google's Patrick Chanezon had alerted me to the fact that Joe has added a new authentication scheme to httplib2. Along with HTTP Basic, Digest, and a couple of others, this Python HTTP library will now handle Google-style authentication. That's really the only tricky thing about using Google Calendar's API. Everything else is URLs and Atom entries. There are Java and C# wrappers for this stuff, but I'm having a ball just using Python's interactive mode to explore the Calendar API. Among the things I can easily do: search for entries matching dentist, search for entries after June 10, receive the results of any query as an Atom or RSS feed.

Most discussion of Gcal has (appropriately) focused on its user interface, which puts many a conventional fat client to shame in terms of both its responsiveness and its ease of use. From my perspective, though, what matters equally is an API that's powerful, flexible, and easy to use. With the still-unsupported libgmail API, for example, I've illlustrated how easy it can be to issue a complex Gmail query, gather all the messages in the found threads, and combine them. This is just one of a million intensely practical real-world scenarios which, I hope, will at some point be officially supported by Gmail in the same way they now are by Gcal.

Last week I pitched a hissy fit when Google PR stomped on my inquiry about GData, which is the RESTful, RSS-and-Atom-oriented architecture of which the Gcal API is the first concrete manifestation. This week I want to show what can happen when communication and collaboration flow as they naturally can and should. Google's instincts about web architecture are spot on. Sort things out on the developer relations front, and it'll be amazing.

# import Mozilla Calendar .xcs into Google Calendar
 
import httplib2, getpass, traceback, sys, libxml2, re
  
pwd = getpass.getpass('pwd: ')
h = httplib2.Http()
h.add_credentials(USERNAME,pwd)
h.follow_all_redirects = True
uri = 'http://www.google.com/calendar/feeds/USERNAME@gmail.com/private/full'
 
#2006-05-30T18:00:00.000Z
#20041207T005046Z
def formatDateTime(datetime):
  m = re.match('(\d{4})(\d{2})(\d{2})T(\d{2})(\d{2})(\d{2})Z',datetime)
  try:
    return "%s-%s-%sT%s:%s:%s.000Z" % m.groups()
  except:
    return datetime
 
def dictFromEventNode(eventNode):
  title = eventNode.xpathEval('./summary')[0].content
  try:
    descr = eventNode.xpathEval('./description')[0].content
  except:
    descr = ''
  start = eventNode.xpathEval('./dtstart')[0].content
  start = formatDateTime(start)
  try:
    end = eventNode.xpathEval('./dtend')[0].content
  except:    
    end = start
  end = formatDateTime(start)
  return {'title':title,'descr':descr,'where':'','start':start,'end':end}
 
def atomEntryFromDict(d):
  entry = """
<entry xmlns='http:\//www.w3.org/2005/Atom'
    xmlns:gd='http:\//schemas.google.com/g/2005'>
  <category scheme='http:\//schemas.google.com/g/2005#kind'
    term='http:\//schemas.google.com/g/2005#event'></category>
  <title type='text'>%s</title>
  <content type='text'>%s</content>
  <author>
    <name>NAME</name>
    <email>USERNAME@gmail.com </email>
  </author>
  <gd:transparency
    value='http:\//schemas.google.com/g/2005#event.opaque'>
  </gd:transparency>
  <gd:eventStatus
    value='http:\//schemas.google.com/g/2005#event.confirmed'>
  </gd:eventStatus>
  <gd:where valueString='%s'></gd:where>
  <gd:when startTime='%s'
     endTime= '%s'>
  </gd:when>
</entry> """ % (d['title'],d['descr'],d['where'],d['start'],d['end'])
  return entry
 
def insertGcal(atomEntry):
  try:
    headers = {'Content-Type': 'application/atom+xml'} 
    resp,content = h.request(uri,"POST",body=atomEntry,headers=headers)
    assert ( resp['status'] == '201' )
  except:
    traceback.print_exc(None, sys.stderr)
     
xml = libxml2.parseFile('mozcal.xml')
 
events = xml.xpathEval('\//vevent')
for event in events:
  dict = dictFromEventNode(event)
  print dict['title']
  entry = atomEntryFromDict(dict)
  insertGcal(entry)

In response to last week's column, Open source education, Steve Saunders writes:

It's worth noting that the Open Source notion -- the esthetic, the mechanic, the principle, whatever you want to call it -- originated in (and is largely based upon) academia. It goes back to the pre-web days of the Internet, when most participants were academic. Look at the principle of peer reviewed science: it is, essentially, Open Source research. This is NOT an educational innovation; the innovation lies in bringing those academic freedoms to the commercial and the private worlds, where trade secrets and competitive advantage would seem at first glance (and did so (and do so) to many) to make such an approach suicidal...except, self-evidently, they do not.

Consider John Abramson's indictment of our system of medical research. He alleges that as the sources of funding have shifted from government to private enterprise, access to primary data is increasingly restricted:

Drug companies often keep the results of their studies secret, even from their own researchers [my emphasis], on the grounds that such results are "proprietary information" of economic value.

Last month I asked a university chemist if he blogs about his work. "There's nothing I can say," he told me. "My web page is a CV with a list of copyrighted publications."

Open access has prevailed since 1991 in those fields served by the LANL archive: physics, mathematics, computer science, quantitative biology. And there is a thriving open access movement that aims to leverage the economics of web publishing to freely distribute many other kinds of research. The NIH policy on public access to NIH-funded research gives that initiative a major boost.

But if academic research fully exploited the power of the web, there would be no need for an open access movement. And even if that movement were to succeed in opening up all taxpayer-funded work, we'd be left without access to the large and growing share of privatized research. If you believe that the open source dynamic is critical to the advancement of knowledge, and I do, there are a couple of ways we might proceed. Governnment could, once again, play a more dominant role in funding research. Or private enterprise could decide that open access -- to basic research, at least -- works to its benefit.

Of the two approaches, I think I'd favor the latter. Especially if government's role were to reward business -- e.g., with tax breaks -- for contributing basic research to the commons.

For today's installment of my Friday podcast series I got together with CJ Rayhill. She's the CIO at O'Reilly and Associates and, with Allen Noren, has been leading the SafariU project. It's a service that provides college course materials online and also as customized books printed on demand.

I've been on a bit of an education kick lately, so we also talked more broadly about how technologies at the intersection of publishing and education -- including podcasting and screencasting -- will influence the future of learning and teaching.

Disclosure: I helped create the original Safari Books Online, but I haven't been involved with it since 2002. I also briefly consulted with SafariU at the beginning of that project, but haven't been involved since before joining InfoWorld.

Update: Transcript available..

When Julie called a few weeks ago, I sensed future unhappiness. She's an account rep for USA Telephone, the company that bought the company that bought the company that bought the company that used to be the friendly local ISP here in Keene, NH. She was calling to alert me to the latest in this string of acquisitions. There would be a service outage, although she couldn't say exactly when, and an equipment change, or not, she didn't know.

On Monday, I was relieved to learn that I'm good to go, equipment-wise, and would only need to reconfigure my router, Murphy willing. I asked Kevin Railsback at InfoWorld to make the corresponding DNS update, made the changes on my end, crossed my fingers, and went to bed.

As it turned out, Murphy wasn't willing. The guy on USA Telephone's 800 line, when I finally reached him, said the upgrade failed and they had to back out of it. That was a minor annoyance. But when I restored the old configuration I could no longer contact the gateway. That was a major annoyance.

I'll spare you the details, but suffice it to say that after a 12-hour service interruption I'm back where I was with the old setup -- or more accurately, with a different version of the old setup.

I have two DSL circuits here. The one that's the subject of today's rant is for business use, and the other is for personal use. On the business side, there have been four changes of ownership in as many years. What was once a local operation is now a faceless corporation whose name I'll hardly bother to learn, because it'll likely change again next year. This, we have come to accept, is the way of the world.

But it doesn't always have to be. On the personal side, I'm dealing with a local operation that has remained local. Some of you will remember my friend and former BYTE colleague Ben Smith. He used to live in Peterborough, where BYTE was located. Ben was the town's first Internet point of presence: a T1 line ran to his house, and his basement was heated by a rack of modems connected to MV Communications in Manchester, NH. If you visit MV's home page, you'll see the tag line "Internet access since 1991". The page is still a wonderful relic from that era.

When I decided to run separate business and personal circuits four years ago, I knew I didn't want to form a new relationship with the company that bought the company that used to be my local ISP. Or with Verizon. Or Time-Warner. So I hooked up with MV instead, and I've never regretted it. Somehow they've managed to buck the trend and remain independent, and from time to time I really appreciate that. One those times was an hour ago. While trying to recover from the outage, I checked USA's home page (by way of my MV circuit) and read:

USA TELEPHONE is the fun and friendly alternative to the big telephone companies. USA offers incomparable customer care...

Well, in any case, I suppose these new fun friends will soon be replaced by newer and funner friends. I can hardly wait.

PS: Yeah, I know, this is the second outburst of grumpiness this week. What can I say? After five weeks of rain and a morning of oral surgery, I needed to vent. When the sun returns and the swelling goes down I'll be back to my usual chipper self, I promise.

It says a lot for Ruby, the dynamic language that's giving Perl and Python a run for their money, as well as for Rails, the application framework that made Ruby famous, that such a savvy software veteran [as Graham Glass] would choose to bet his next business on these open source technologies. It says even more about the nature of the open source process, as well as the future of education, that he would not only propose an improvement to Ruby on Rails but also publish the analysis that led to his proposal.

Open source software development, to a degree unmatched by any other modern profession, offers apprentices the opportunity to watch journeymen and masters at work, to interact with them, and to learn how they think, work, succeed, and fail. Transparency and accountability govern not only the production of source code but also the companion processes of design, specification, testing, maintenance, and evaluation. [Full story at InfoWorld.com]

Yesterday I attended a local forum in which business and education leaders discussed how they can work together more effectively. It wasn't the right venue in which to expand on the theme of this week's column, but here's what I would like to say to that group and to their counterparts everywhere. Open source software development, as a profession, is an early adopter of a work style that can also characterize many other professions. The key aspects of that work style are transparency, accountability, network-mediated collaboration, and narration of work.

I have struggled, so far with little success, to convey what I mean by narration of work, why it should be a routine activity in every professional life, and how it can benefit individuals and society. But yesterday's forum reminds me to keep trying. The room was full of well-intentioned people who clearly understand that the disconnect between work and education is killing us. But they don't see alternatives to the rickety bridges we've tried to build in the past: job fairs, brochures.

I do see an alternative. Thanks to personal online publishing and to an emerging cultural ethos of transparency, there is an exciting new possibility in the world. A young person today who is interested in software can find out what it is like to be a software developer -- by evaluating products, by reading the accounts of people creating them, by making contact with those folks, and by contributing to real projects. I hope it will also become possible for young people to find out what it is like to be a psychologist, homebuilder, forester, teacher, retailer, or city planner. If we want to inspire the next generation we need to open windows onto our worlds, share our knowledge and passion, and invite them in.

Greg Smith, the author of FeederReader -- "The Pocket PC Aggregator" -- writes:

I was wondering why you did not include the screencast of "Screening Room #5" as an enclosure in the RSS Feed:

http://weblog.infoworld.com/udell/2006/05/31.html#a1458

I use FeederReader on the Pocket PC and enjoyed the audio portion of the Anders interview on LINQ. If you made the screencast an enclosure, FeederReader would have downloaded the video directly to my Pocket PC and I could have simply pressed "play" to view it. As it stands now, if I want to view it, I have to hunt for your post using a desktop browser because Pocket Internet Explorer does not by default play embedded videos on HTML pages.

Good question. As I explained to Greg, I'd been hoping to publish those screencasts in a single format, and had settled on the Flash SWF format as the best universal option, but then the FLV era overtook the SWF era and things got a bit more complicated again. For best playback using FLV format, my screencasts need to be encoded in a way that requires Flash 8. And while Macromedia Adobe loves to point out that they can upgrade an installed base faster than you can blink, the fact is that it still does take a while, so I've been reluctant to require the Flash 8 player, and have grudgingly been offering episodes of The Screening Room in QuickTime and Windows Media formats as well.

So, to Greg's question, which of these formats should be in the enclosure? Flash, QuickTime, or WinMedia? (Additional wrinkle: the Flash FLV format requires delivery of a Flash-based player, as well.) If I enclose all three, that starts to get hefty. And last I heard, sticking multiple enclosures into an RSS 2.0 item is a nonstarter.

It really would be cool to provide a feed with enclosures for The Screening Room. Got any ideas? Fire away.

Update: Though somewhat daunting to implement, Greg has lots of great ideas:

My recommendation is usually to publish Flash on the website, and both WMV or MP4 in separate RSS feeds. If you have the ability, publish two different feeds with identical content and different video format. If you only want to publish one feed and your audience owns diverse products, choose MP4. For MP4, it is best to choose MP4 Part 2 Video and MP4 Part 3 audio encoding (fps 10-15, 250-700kbps). This will play back on an iPod, which is a large part of the video market. And I'm specifically avoiding the newer and more CPU-hogging AVC (MP4 Part 10).

Unfortunately for my users, MP4 requires a different video player other than the already-installed Windows Media Player.

Flash won't play on a lot of portable devices. I know for Windows Mobile, there is a Flash 7 player but it requires some extra steps to set up to play downloaded content e.g. from an RSS feed.

Richard Miller votes for a QuickTime format, but wonders (as do I) whether screencasts will translate to the small screen:

I've really enjoyed the Screening Room videos so far. I also have a Video iPod so I've thought it would be nice to watch them there. If you implement RSS enclosures, my vote would be for a Quicktime compatible format like MPEG-4 or H264. Both are open formats that should play on any platform with either Quicktime or VLC, with the added benefit that they'll play on the Video iPod. On the other hand, maybe it would be unpleasant to watch a screencast on a small iPod screen; those small screens might be best suited for close-ups of cheery faces like Amanda Congdon instead of screenshots of .NET code.

And over at Macromedia Adobe, John Dowdell weighs in with good news and bad news:

Jon Udell blends three different video requirements together: minimal hassle for computer WWW browsers; RSS video "enclosure" for "newsreaders" (quotes 'cause I was never quite comfortable with those latecomer labels); and video delivery to pocket devices, in this case a PocketPC. The first has a clear answer: Adobe Flash Player 8 already has far greater desktop presence than other current video engines (NPD tested at 70% viewability last March; WMP, Real & QT have *overall* viewability in the same range but don't have corresponding versioning info). For the "RSS enclosure" (which is actually an enclosed reference), I think Scott Fegette will have the best guidance, but as the Yahoo Videobloggers know, FLV/SWF isn't yet integrated into most of that class of software. For PocketPC, we've got some native video, but not the full range available in the big desktop version. So... if you're trying to view a friendly WWW video in a PocketPC through an RSS "enclosure", civilization has not yet quite reached those attractive environs, regrets.... ;-) [JD on EP]

Adobe's Scott Fegette has more here. Bottom line: from the publisher's point of view, this is not going to be fun.

Over the weekend I heard from John Tynan, webmaster for radio stations KJZZ and KBAQ in Arizona. He noticed that I'd recently crossed paths with New Hampshire Public Radio, and wanted to point out a couple of interesting things.

On KJZZ's Arizona Week page, there's a mashup (see "Map of Arizona Week Stories from 2004 to Present") that organizes radio stories by location. Very cool. If you were visiting Tuscon and decided to drive out to Mammoth, you could notice that the Biosphere is enroute, and you could listen to a story about it along the way.

Here's what I like even more. John has correctly observed that, if a bunch of radio stations were to insert location and topic tags into their RSS feeds, it would be trivial for an aggregator to scoop them up. Listeners could then create -- and share -- custom radio programming.

John writes:

This would also have the additional benefit of promoting the idea of collaboration between stations and reinforcing the idea that we are a network of content providers.

It sure would.

Note, by the way, that this idea depends on public radio making the shift from RealAudio streams to MP3 downloads. And that seems to be happening, albeit sometimes rather quietly.

Until about a month ago, for example, I was relying on an mplayer hack to move some of my favorite public radio shows onto my MP3 player. The other week, though, I noticed that although the archive page at This American Life still says that you can't download files, it's not true anymore. Last week's episode, for example, was a rerun of a classic on stories that make us cringe. The offered link is http://audio.wbez.org/tal/182.m3u. If you unpack that you'll find http://audio.wbez.org/tal/182.mp3.

When I noticed this change, I made myself an unofficial TAL feed with enclosures. I'm sure other TAL listeners would be happy to collaborate on adding location and topic tags to an official version of that feed. Delightful network effects would ensue.

Update: Jared Benedict has a better version.

A Google developer and API evangelist, P., invites me to take a look at a new API. I reply that, instead, I'd like to discuss the GData APIs. P. thanks me for my interest in GData, and promises to try to connect me with someone.

A couple of days later I hear from S., on Google's "communication team." S. initiates the usual PR protocol: "Can you give me some details about the story you are working on and what information you'd like from Google?"

Sigh. I'm not working on a story, at least not yet, I'm just curious about GData. I can and will research it myself, but if you want to play a role in that process, trying to pin me down to a laundry list of questions won't help your cause. Just find me somebody who is passionate about GData, and who is allowed to discuss it.

But then S. turns up the heat. "Can you also please contact the communications team rather than going to the product folks directly so we can make sure your inquiry is routed to the appropriate party and answered in a timely fashion?"

Oh please. It was P. who wrote to me in the first place. I relayed my request through P. because I had a hunch that the "communications team" might not be tuned into GData. So I ignore the slapdown and merely reply to S.:

"I would like to talk to folks on the GData team about the APIs and their uses."

S., who let's recall is employed by the world's leading search engine, replies:

"What does Gdata refer to? We don't have a product called Gdata that I'm aware of....."

I am not making this up.

Another day passes, S. figures out how to Google GData, and writes back that an interview can be arranged, provided that I first enumerate my questions.

Never mind. I only have one question at this point. What planet are you living on?

Tags: publicrelations

Today's installment of my Friday podcast series, which I'm now calling Interviews with Innovators (RSS feed here), began life as a screencast in which Peter Rodgers, founder and CEO of 1060 Research, was demonstrating the 1060 NetKernel. It's an unusual creature -- a REST-oriented, microkernel-based app server that I first noticed back in 2004.

At the end of a long demo and conversation, I concluded that the things Peter showed me in the demo -- for example, podcast feed synthesis involving XML transformation and text-to-speech conversion -- didn't really tell the story. So I edited down the two hours of audio to produce a half-hour podcast which I think works much better.

NetKernel is a dual-licensed open source product. Among the commercial licensees that we discuss, Peter mentions one public-facing project: Pharm2Fork, which aims to take a decentralized, peer-to-peer approach to the implementation of the European Union's food traceability regulations.

Here's a snippet from the end of the interview:

Peter Rodgers: I suppose we're a bit out of the mainstream.

Me: Oh, you're a long way out of the mainstream!

NetKernel is, nevertheless, a fascinating effort to extend the principles of web architecture -- both within, as well as across, system boundaries. Give it a listen, try out the product if the ideas catch your fancy, and should you wind up using it in an interesting way, let me know.

Update: Transcript available.

When I logged in to my bank's online system to pay some bills last night, I was greeted with the following message: "Bill payment system upgrade completed."

Uh-oh. That's a message I don't want to see.

...

The old system would queue up payments from multiple accounts in a single screen. The new system, with "simpler navigation that makes paying bills easier," won't let me do that. Now I have to make payments from my household account in one batch, and from my business account in another. The forklift upgrade didn't just temporarily disrupt my online banking experience, it permanently subtracted value from it.

...

The central tenet of modern test-driven development is continuous improvement by steady accretion of small, incremental changes, with continuous evaluation of the effects of each change. This model should apply not only to the unit-testing of modules of code but also to the field-testing of aspects of user interaction.

Businesses born of the Web, such as Amazon and Google, have always known that it's now possible to evolve systems in this fluid way, and they've always made the most of the opportunity. Many businesses that predate the Web, though, still cling to anachronistic methods dictated by constraints that no longer apply. It's not a winning strategy. [Full story at InfoWorld.com]

If you're a local business in my town, it sucks to have me as a customer. Consider my bank, for example. Every time they upgrade their online bill payment system, I write a column about it. Last time I whined about having to re-enter all my payees when they switched providers. This time I bitched about an upgrade that scrambled my expectations and neglected to carry forward a useful feature.

In another column about securing the analog hole I complained about information leaks at the hospital and the YMCA. The Y's sin, you may recall, was a new barcode authenticator that left personal information sitting on a screen, visible to everyone entering the building.

I'd rather praise than criticize, so I'm happy to report that the Y has corrected the problem. As of last week, there's a privacy hood shielding that screen. Problem solved!

Update: Marc Thornsbury writes:

It probably wasn't your intention, but you managed to point out one of the glaring problems with applications delivered as services by third parties over the web (call is SOA, ASP, what-have-you). While the trade press has been all over this like a cheap suit for several years now, it just refuses to take off (lo these many years and Salesforce.com is still the only substantial success story).

And you've managed to bring out, in stark relief, one of the reasons it hasn't. When it comes down to brass tacks, lots of folks (like me) don't want to play the unwilling victim to someone else's decision as to whether an interface is acceptable or not, or software is no longer "beta". I never like to run x.0 of anything. But when you've turned your software over to someone else, *they* make that decision...not you. There's no "opt-out" option. If you were running your own bank client, you could have opted to skip the "new version" and waiting until the kinks were worked out in the next one (or perhaps give yourself time to learn the new interface at your leisure before making the switch).

There are many circumstances where functions ancillary to your operation can be moved outside and changes to them have minimal impact. But where an app. has high/broad use or addresses a core function of the business, the arbitrary decision of someone else can bring your production to a near standstill and overload your help desk...and generally without warning. Who wants that?

The May episode of The Screening Room is a continuation of my earlier podcast with Anders Hejlsberg. The podcast is the conversational part of our recent get-together to discuss the new preview of LINQ, and this screencast is the demo part. Topics covered in the screencast include the new join syntax, the DLINQ visual mapper, adaptation of stored procedures and user-defined functions to LINQ,an experimental approach to transforming XML into modifiable .NET code, and dynamic construction of expression trees.

At 50 minutes this screencast runs a lot longer than the earlier episodes in the series, which are all about 20 minutes long. That's partly because I'm very interested in LINQ, and when you can get Anders Hejlsberg to give a personal walkthrough of this evolving technology, it's worth sharing in toto. But it's also because I'm rethinking my approach to this series. My most popular screencasts have all been in the 5-to-8-minute range, and I felt that 20-minute episodes would really be a stretch. The fact that none of them have attracted a huge amount of interest might suggest that they are, indeed, too long. Or it may simply indicate that the audience for these in-depth shows is smaller, which isn't surprising. Might that smaller audience, though, want to see more rather than less? To test that theory I'm running this episode long.

I should also mention that at this site you can see the two presentations that I originally saw at the PDC last fall: Anders' introduction to LINQ, and his explication of how C# 3.0 supports it. So far as I can tell they will only play in MSIE on Windows, and if so that's a shame because LINQ has provoked a broader conversation, and it would be useful for everyone to be able to see those presentations. Maybe this platform-neutral screencast will be helpful in that regard.

In this week's column I report on how a team of instructional technologists at the University of Mary Washington, in Fredericksburg VA, have partnered with a web hosting service to create what they call a sandbox for exploring open source software and experimenting with its learning and teaching possibilities. My visit to UMW earlier this month was the result of a virtual conversation between me and Gardner Campbell that went like so:

• Gardner listened to a podcast in which I am interviewed about podcasting and screencasting, and linked to it from his essay, Podcasting, Rich Media, Film School, Literacy.
• I noticed his link, and quoted from his item in Hypermedia: Why Now?, part of my PrimeTime Hypermedia series.
• He read another column in that series, The New Freshman Comp, and commented here:

  What he writes is so close to what I've been thinking and (intermittently) blogging about over the last few months that I thought I was seeing incontrovertible proof of telepathy.

• Then he updated that entry when he realized it wasn't telepathy, but rather the ordinary -- though arguably no less magical -- process of link discovery and citation in the blogosophere:

  Udell had actually read, discussed, and linked to the blog where I had initially discussed and linked back to his blog and podcast...I'm used to that in the scholarly world, but the scale and pace are quite different in that world.

Since then, I've been thinking a lot about how ascendant uses of the Internet -- including blogging, podcasting, screencasting, and social software -- could transform education in the way that television was supposed to but of course never did.

What's changing? Call it Web 2.0, or the two-way web, or the read-write web, or whatever you will, we're approaching a dual inflection point: widespread access to existing knowledge, coupled with a widespread ability to publish new knowledge,

A compelling analysis of how these trends can improve education, and why that improved educational system will be the right one for the 21st century, comes from John Willinsky, whose podcast talk on these subjects I found by way of Brian Lamb.

Among his themes, Willinsky talks about how he, as a reading specialist, would never have predicted what has now become routine. Patients with no ability to read specialized medical literature are, nonetheless, doing so, and then arriving in their doctors' offices asking well-informed questions. Willinsky (only semi-jokingly) says the Canadian Medical Association decided this shouldn't be called "patient intimidation" but, rather, "shared decision-making."

How can level 8 readers absorb level 14 material? There are only two factors that govern reading success, Willinsky says: motivation, and context. When you're sick, or when a loved one is sick, your motivation is a given. As for context:

They don't have a context? They build a context. The first time they get a medical article, duh, I don't know what's going on here, I can't read the title. But what happened when I did that search? I got 20 other articles on the same topic. And of those 20, one of them, I got a start on. It was from the New York Times, or the Globe and Mail, and when I take that explanation back to the medical research, I've got a context. And then when I go into the doctor's office...and actually, one of the interesting things...is that a study showed that 65% of the doctors who had had this experience of patient intimidation shared decision-making said the research was new to them, and they were kind of grateful, because they don't have time to check every new development.

Another version of this same story comes from my friend and former BYTE colleague, Ray Cote, who runs his own software and consulting business. Over dinner a couple of weeks ago, Ray told me that he's not looking for people who "know" one or another language or framework, but rather for those who can motivate themselves to rapidly acquire these and other contexts as needed.

Because both software and the collaboration that supports its development flow naturally through the network, learning in the software realm is arguably a preview of what learning will become more broadly. And thanks to open source, we're also seeing a glimpse of the future of teaching. In a panel discussion at the UMW's Faculty Academy I talked about the lost tradition of apprenticeship. One of the remarkable things about the open source process is its extreme transparency. Typically there is full access not only to the source code that constitutes a working product, but also to the discussions and deliberations surrounding its design, its specifications, its testing, and its use.

If you're motivated to write software, you can tap into a community of practice, learn how the experts work, and bootstrap yourself into their context. As a member of that community, you then enlarge that context in ways that help the next person bootstrap his or her way in.

Access to knowledge, access to publishing. Motivation and context. If an educational system embraced these principles, what would it look like? Willinsky challenges us to figure it out, and anyone with a stake in education -- which is to say, everyone -- should be asking and trying to answer that question.

For this week's Friday podcast I got together with Frank Martinez, founder of Blue Titan and now, as a result of a recent acquisition, executive VP of SOA Software.

We talked about how Blue Titan's specialties, namely tolerance and mediation, will complement SOA Software's core strengths in the control and governance of services. And we continued a conversation we've been having on and off for a couple of years about how this combination benefits not only WS-Heavy services but also their WS-Light counterparts -- including those that rely on RSS.

Other topics included social software, the Prosper peer-to-peer lending system, and identity policy in a world where both technical services and economic relationships are loosely coupled.

Update: Transcript available.

Now that the power search and metadata explorer have been stable for a while on the InfoWorld site, I've been looking into ways to expand their use. Swapping the new search mechanism for the existing one is the obvious next step, but that's going to require some politicking. Meanwhile I realized we could fry a much bigger fish while remaining under the radar. External search services send a ton more people to our pages than our existing service does. Why not recruit them to spread the word about our new-and-improved service?

I've done this kind of thing before, and it's a nifty hack. You set up your web server to detect referrals from search engines, you capture the query, and then you offer to do something further with it in the context of your page. I proposed this to Baldwin Louie, one of our in-house developers, and he took care of it lickety-split. To see the prototype in action, search Google for 'Virtuozzo' and locate Paul Venezia's review of that product -- it's currently on page 2 of the results. Then click through to the article. You should see this:

Explore 'virtuozzo' in InfoWorld Power Search

Cool, huh? It's not rolled out sitewide yet, so for now you can only see it on that test page, but I love the effect. Thanks Baldwin!

Here's the difference between 2006 and whenever was the last time I used this technique. Five or more years ago, JavaScript was frowned upon, you felt obliged to do the whole thing server-side, and you ended up worrying whether your server could handle per-page script logic. This time around, Baldwin felt free to do the minimum amount of SSI (server-side include) -- i.e., just echo the referrer string into the page -- and delegate the regular-expression and page-rewriting work to client-side JavaScript. It's a nice example of a key Web 2.0 theme: collaborative partnership between smart servers and smart clients.

Last fall I had a virtual blog-to-blog discussion with Jerry Slezak about the absence of a comment system here on this blog. Like Matt Roth-Cline and Mark Bernstein, I've felt that the loosely-coupled virtual discussion mode encourages thoughtful and respectful discourse and discourages flames. But while virtual discussion as mediated by my array of link sensors (e.g., Google, del.icio.us, Bloglines) can be remarkably effective, there's been less of it lately. I think that's because the activation threshold for blog-to-blog discussion is fairly high, and as the novelty of blogging fades there are fewer folks who have the time and energy to cross that threshold. So I've been thinking about how to enable comments.

Like Tim Bray, I'd prefer to review and approve comments post facto, rather than try to mount a front-line spam defense. I could build such a system for myself, or implement Tim's if he winds up releasing it, but I'd just as soon pay a small fee for a hosted system that does what I need. Problem is, I can't find anyone to pay that fee to. The likeliest candidate appears to be HaloScan, but after trying it I couldn't quite pull the trigger.

Why not? It's a tightly-coupled bundle of stuff: a datastore, an editing system, an administrative interface, and a display mechanism. I'd like a commercial service to provide defaults for all of these, but also enable reconfiguration and component substitution. I should be able to swap out the popup editor, with its minimal tag set, for a dynamically shown or hidden inline element supporting richer tags or whichever wiki-style markup the commenter prefers. Or I should be able to reach into the datastore to perform batch operations and combine its contents with my search results.

If anyone has crafted this kind of loosely-coupled assembly of WS-Light services, combining a solid default experience with well-designed mechanisms for substitution and extension, I'd like to hear about it.

In response to yesterday's entry (and this week's column) on web-based presentations, several folks have mentioned Eric Meyer's S5, which I pointed to from the column but not the corresponding blog item. Today Michael Smick adds:

Some wonderful soul integrated s5 slideshow into my favorite wiki, PmWiki. Here is the "cookbook" script, a php extension for the main PmWiki install.

http://www.pmwiki.org/wiki/Cookbook/SlideShow

here are two demos

http:\//wiki.cyaneus.net/teste/index.php?n=Main.SlideShow&action=slideshow
http://scripsit.org/index.php/Main/SlideShow?action=slideshow


What's cool about this one, is the normal wiki page comes up, and gives you a link for the slideshow, and the slideshow is just another <variable?> accessible from the address bar. So for those who complain that online slideshows are annoying, they get both without even unhooking the default style sheet in the browser.

Here's another web presentation / screen sharing service. Saw it a couple months ago.

http://www.teamslide.com/

Clearly the use of wiki markup is a great way to simplify the authoring of HTML slideshows. Integration with a server-based wiki does, however, sacrifice the convenience of serverless deployment. In general, S3 or HTML Slidy shows work the same way from a file: source or an http: source, and that's extremely handy. I wonder if anyone has combined S5 or HTML Slidy with a standalone (aka single-page) wiki such as TiddlyWiki?

Another observation: in these examples, the URL-line does not update as the slides advance, so slides are not invidually linkable and bookmarkable. Note, by the way, that while HTML Slidy does update the URL-line, it's only a partial solution. An URL like http:\//www.w3.org/Talks/Tools/Slidy/#(4) relies on JavaScript code to render slide 4 only. Vanilla HTTP clients -- e.g., search engine crawlers -- won't see a sequence of individual pages.

On the question of permalinks, I don't how it's possible to dynamically modify the URL-line and retain all expected behaviors. I've written before about AJAX and deep linking [1, 2, 3], but I hadn't thought this through from the perspective of search engines until I made my first HTML Slidy show. There's an option in HTML Slidy to make each slide's title the HTML doctitle, and of course I turned that on because I'm always looking for ways to make my stuff more coherent and useful in search results contexts. But to no avail. Search engines don't contain JavaScript processors, so they cannot construct and therefore do not see those per-slide titles.

In an AJAX era, could search robots and other crawlers use JavaScript processors to reproduce the effects happening inside browsers? Should they? Might the ability to do so become a competitive advantage?

In response to yesterday's entry (and this week's column) on web-based presentations, several folks have mentioned Eric Meyer's S5, which I pointed to from the column but not the corresponding blog item. Today Michael Smick adds:

Some wonderful soul integrated s5 slideshow into my favorite wiki, PmWiki. Here is the "cookbook" script, a php extension for the main PmWiki install.

http://www.pmwiki.org/wiki/Cookbook/SlideShow

here are two demos

http:\//wiki.cyaneus.net/teste/index.php?n=Main.SlideShow&action=slideshow
http://scripsit.org/index.php/Main/SlideShow?action=slideshow


What's cool about this one, is the normal wiki page comes up, and gives you a link for the slideshow, and the slideshow is just another <variable?> accessible from the address bar. So for those who complain that online slideshows are annoying, they get both without even unhooking the default style sheet in the browser.

Here's another web presentation / screen sharing service. Saw it a couple months ago.

http://www.teamslide.com/

Clearly the use of wiki markup is a great way to simplify the authoring of HTML slideshows. Integration with a server-based wiki does, however, sacrifice the convenience of serverless deployment. In general, S3 or HTML Slidy shows work the same way from a file: source or an http: source, and that's extremely handy. I wonder if anyone has combined S5 or HTML Slidy with a standalone (aka single-page) wiki such as TiddlyWiki?

Another observation: in these examples, the URL-line does not update as the slides advance, so slides are not invidually linkable and bookmarkable. Note, by the way, that while HTML Slidy does update the URL-line, it's only a partial solution. An URL like http:\//www.w3.org/Talks/Tools/Slidy/#(4) relies on JavaScript code to render slide 4 only. Vanilla HTTP clients -- e.g., search engine crawlers -- won't see a sequence of individual pages.

On the question of permalinks, I don't how it's possible to dynamically modify the URL-line and retain all expected behaviors. I've written before about AJAX and deep linking [1, 2, 3], but I hadn't thought this through from the perspective of search engines until I made my first HTML Slidy show. There's an option in HTML Slidy to make each slide's title the HTML doctitle, and of course I turned that on because I'm always looking for ways to make my stuff more coherent and useful in search results contexts. But to no avail. Search engines don't contain JavaScript processors, so they cannot construct and therefore do not see those per-slide titles.

In an AJAX era, could search robots and other crawlers use JavaScript processors to reproduce the effects happening inside browsers? Should they? Might the ability to do so become a competitive advantage?

Updates:

Clint Checketts wrote to say that he's collaborated with Paulo Soares on a slide-show plugin for TiddlyWiki, seen in use here

Peter Sefton has integrated Slidy into the University of Southern Queensland's ICE courseware system. ICE, by the way, is a noteworthy example of how wordprocessor stylesheets -- for OpenOffice.org and MS Word -- can provide the integration glue for single-source and multi-output content management.

Presentation software has been stuck in neutral forever. Web applications, however, are firing on all cylinders. Some say Word and Excel are about to be Web 2.0 roadkill. Not me. The browser can't yet substitute for those applications. But for PowerPoint? Any day now. [Full story at InfoWorld.com]

Jim Hall objected to this column on the following grounds:

By advocating software that only a minority can use, you are not addressing the real problem. I would submit that someone who can code HTML has the same likelihood of wandering off point as someone who can't. I suggest that you recall back to the days of yore, when you and I were in grammar school. Teachers without technology had just as much potential to wander hither and yon all over their subject as their technologically enabled colleagues. This remains true in the professional world as you are certainly aware.

The key, Jon, is not the technology, but the focused mind of the presenter. The unfocused mind will waste our time with or without PowerPoint and with or without your web apps.

...

I am disappointed with this article because you have allowed this elitist technology to distract you from performing a comprehensive evaluation of the software's tools and capabilities as well as the ease or apparent lack thereof for most people to utilize. Please stay on point next time.

That said, I'll argue that it's democratic, not elitist, to believe that presentations ought to be first-class citizens of the web, viewable by any standards-based browser with full interactive fidelity. If we've failed to fully democratize the necessary authoring software -- as, so far, we have -- then shame on us. There's no longer any good reason why we couldn't make it easy for people to create effective presentations through the web as well as for the web, and there are plenty of good reasons why we could and should.

For today's Friday podcast I got together with Andy Singleton. I've known Andy for a long time. Back in 2003 I brought him in to help with a feature story on the globalization of software development -- in particular, for his thoughts on the dynamic assembly of global teams that can wield open source componentry.

More recently, at Assembla.com, Andy's put together an application called Breakout that aims to package up the approach he's developed and embed it in a professional network.

Early on in the conversation I mention Yochai Benkler's Wealth of Networks which is a book, a wiki, and a downloadable PDF. By way of Judd Antin, who collaborates with Ben Hill on the Mycroft project discussed in an earlier podcast, I also recently found Joseph Hall's recording of a recent talk by Benkler. It's a fascinating analysis of the social and economic trends that are fueling decentralized production not only of software, but of all cultural artifacts.

Update: Transcript available.

Consider this scenario. You're at your development workstation, showing me your application and the tools you use to build it. We discuss the architecture of your software, the nature of your environment, and the ways in which your tools both do and do not meet your needs. My virtual camera, looking over your shoulder, records everything we say and everything that happens onscreen. Then I edit the session, which might have taken two or three hours, to produce a 20-minute screencast.

That screencast will be of compelling interest to your fellow practitioners. Similar ones that I might produce in collaboration with them will likewise appeal to you. But for several reasons, it may not be possible for me to make such movies.

Confidentiality is the overriding concern. If your application confers strategic advantage -- and if it didn't, you'd be buying rather than building -- you'll need to hold your cards close to your vest.
...
So here's my first pitch: Unless you're willing and able to highlight your own stuff -- in which case I'm all eyes and ears -- we'll focus on the tools and infrastructure that you're building on top of and integrating with. Moreover, I'll publish only what you approve. Ideally the final cut will include enough cinema verité to be interesting and useful, without crossing any lines you can't cross. I don't know if that's feasible, but I want to find out.

Then there's the question of incentive. It's clearly in my interest to pursue this idea, and it's in the interest of my readers, viewers, and listeners, but what's in it for you and your company? Here's my second pitch: You're doing innovative work, and one of your challenges is to find the people who can help you do more of it; showcasing your expertise and fluency with leading-edge software technologies might be a good way to attract that talent. [Full story at InfoWorld.com]

I've received a couple of good leads so far, and a larger number of pitches from vendors wanting to do demos. That's understandable given that the first four installments of The Screening Room have, in fact, been interview-style vendor demos, but I'd like to reiterate that my goal is to widen the focus of this series to include interviews with practitioners who are customers of the vendors, and to produce screencasts that highlight the work of those practioners as enabled by one or more vendor-supplied tools, products, frameworks, or infrastructures.

Ted Conway's note posted to the unofficial SAS weblog nicely captures the spirit of what I'm hoping to accomplish. It's going to be tricky to bootstrap this thing because, as yet, there are no reference screencasts of the sort I envision. But I'm hoping that once we break the sled out of the ice, it'll start to move.

When I last mentioned the use of HTML directives to exclude unwanted text from indexing and search, I thought I had the solution to my problem. InfoWorld's site uses the Ultraseek engine, which supports the directives <!--startindex--> and <!--stopindex-->, so I added those to my template. But it seems I got it half-backwards. Here's what I did:

<html>
<head>...</>head>
<body>
  
scripts and ancillary text
  
<!--startindex-->
main text
<!--stopindex-->
  
scripts and ancillary text
  
</body>
<html>

Here's the new layout:

<html>
<head>...</>head>
<body>
<!--stopindex--> 
 
scripts and ancillary text
 
<!--startindex-->
main text
<!--stopindex-->
 
scripts and ancillary text
 
</body>
<html>

If I've got this right finally, then a query for, say, Virtuoso, should -- by tomorrow or the next day -- return a handful of items about Virtuoso rather than, as now, a long list of items in which Virtuoso appears in ancillary text. And this time, I'm determined to check the results of the experiment. If I forget, please remind me.

Of course, given that my earlier item on this topic ranks #3 on this query, I'm probably tilting at windmills here. Precise search is a topic that doesn't float many people's boats. And when you look at the above examples, you can see why. Our obsession with search rankings is all about inclusion, not exclusion. We've created a whole new breed of specialists devoted to search engine optimization, and the last thing any SEO practitioner would think of doing is disabling search at the top of a page, in order to enable it more precisely somewhere in the middle. As a result, there's unlikely ever to be demand for implementing this feature in the major search engines.

Alas, it is my lonely fate to care about bringing you all of what's relevant to your query, and none of what isn't. So I'll keep plugging away at it.

The new preview of LINQ went live yesterday. For this week's Friday podcast I got together with Anders Hejlsberg to talk about the project in general, and what's in the new preview.

Last fall, when I posted an example based on the first LINQ preview, Sam Ruby showed how to do the same things in Ruby. When I mentioned LINQ the other day, Mike Pence recalled Sam's demo and wrote to say:

The unification of code and data is at least as old as, well, Lisp. And back when you previously talked about Linq, Sam Ruby showed how what Anders and Co. were doing is really nothing new and is easily done in the programming language Ruby.

So, why perpetuate the myth that there is something new going on here? Congratulations to Microsoft for discovering dynamic language features, I guess, but why is this news?

I relayed that comment to Anders Hejlsberg, and here is his reply:

Regarding the question from your reader, I'm certainly not claiming that data/query integration in a programming language is new. Just look at dBASE or FoxPro or any one of a myriad other examples. However, I do think the unified programming model for query and transformation of objects, relational, and XML is new, as do I think it is new for a mainstream programming platform to have first class support for these concepts in the same way that objects and classes are first class.

There is no doubt you could take LINQ queries against XML and write them in XQuery (or perhaps Ruby, as Sam did); or take LINQ queries against relational data and write them in SQL.

But where else can you write cross-domain queries (e.g. a join of relational data and in-memory objects producing hierarchical XML output) of the nature I showed you on Monday?

In a query language that is as expressive as SQL or XQuery?

And in those queries make use of a rich framework like .NET?

With an architecture for pluggable query processors and dynamic queries?

And do so in an integrated toolset like VS that supports productivity features like IntelliSense, statement completion, and code refactoring (ok, some of these are still work in progress)?

Even if the overall idea is not new, I do think the way we've implemented it is both new and innovative.

Regarding "Congratulations to Microsoft for discovering dynamic language features", I think dynamic vs. static typing is entirely orthogonal to language integrated query. There are pros and cons to both. I actually think the ideal is somewhere in between. An interesting example is the advances we've made in type inferencing in C# 3.0. This is kind of a "have your cake and eat it too" situation. You get to write programs in the dynamic language style (without explicitly declaring and mentioning types) yet your programs are still strongly typed, such that they can be compile-time type checked and take advantage of tools like IntelliSense, statement completion, and code refactoring.

I'm not certain myself how much of LINQ is genuinely new, and how much is a creative synthesis of pre-existing ideas, but I do know that I'm fascinated by what it can do and where it might go. Its reach will ultimately be determined by the extent to which non-Microsoft databases of all flavors -- relational, object, XML -- are willing and able to plug into the LINQ framework. To that end, one of the most significant points in today's interiew is the announcement of IQueryable, an interface that formalizes how providers plug into LINQ. Watching the uptake of this mechanism by third parties over the next year or so will tell us a lot about whether LINQ will be a Microsoft-only phenomenon or something that more broadly influences the industry.

Update: Transcript now available.

My career as an amateur explorer of social networks dates back to 2002, when the channelroll widget that I introduced into the blogosphere became popular enough to enable me to do some data mining. More recently I've mined Bloglines and del.icio.us, and now the Share Your OPML site affords similar opportunities.

None of these mechanisms, however, is friendly to ad-hoc query. Here are five examples of the kinds of questions I'd like to ask and answer:

1. Who recently added my feed?
2. Who recently dropped my feed?
3. What clusters (interest groups) emerge from the data?
4. Which clusters are least like mine?
5. Who are the weak ties between my clusters and foreign clusters?

For years I've been writing scripts that scurry around, follow links, scrape pages, and abuse servers in order to discover these kinds of things. It would be nice to be able to get hold of the data and do this work more efficiently.

The mission statement of Share Your OPML reads:

The purpose of this site is to gather a community of subscription lists, in OPML format, and aggregate them in interesting ways.

When public radio asks listeners for contributions, it's dollars that they're looking for. And sometimes I give those dollars. But in the era of podcasting, why not also contribute audio? I've collaborated before with New Hampshire Public Radio's Jon Greenberg, and when I proposed doing a series of occasional commentaries he agreed. The first of these, based on the experience I described here, aired last night and this morning on NHPR. The audio (with transcript) is available at www.nhpr.org/node/10748.

These commentaries will replay many of the same themes that I explore here: collaboration, lightweight integration, network effects. I want to reach folks who don't read this blog, but my agenda includes those of you who do. We of the geek tribe face an interesting problem. We can invent just about anything we can imagine, but our imaginations are insufficiently grounded in the needs, expectations, and realities of the people we refer to as "users". Here's Alan Cox, for example, on the inconvenience of having users.

How do we reconcile this (in some ways necessary) perspective with Eric von Hippel's notion, which I take very seriously, that these inconvenient users are in fact a vital source of innovation?

Communication has to flow both ways. People need to be shown, in ways that make sense to them, how the technologies we provide really can solve problems and improve their lives. Much of what we supply goes unused because the bottleneck is on the demand side. I'd like to help raise people's expectations, and thus ratchet up that demand.

More demand will lead to more use of the services we provide. And that, in turn, will create more opportunities for us to observe patterns of use, and to notice and react to the innovations that people produce as they adapt our stuff to their circumstances.

This is an ambitious and maybe quixotic agenda, but it can't hurt to try. And in any case I'll feel good about my contribution to public radio.

I've always regarded the Web as a programmable data source as well as a platform for the document/software hybrid that we call a Web page. Early on, programmable access to Web data entailed a lot of screen scraping. Nowadays it often still does, but it's becoming common to find APIs that serve up the Web's data.

The holistic view of that network [of databases] should be our focus. In [Kingsley] Idehen's view, you'll use something like SPARQL -- a query language for the semantic Web -- to traverse a graph of interlinked sites, and to merge interesting sources into a virtual collection. Then you'll dispatch queries to each member of that collection. They'll offer a range of query styles ranging from free text search to iteration over simple key/value pairs (accessed by way of RSS or Atom) to tree traversal (XPath, XQuery) and relational query (SQL). I think he's got it exactly right. [Full story at InfoWorld.com]

If you are a data hacker -- and what programmer isn't? -- the good times are getting ready to roll.

Nathan McFarland used his own service, CastingWords, to transcribe last Friday's podcast on the topic of harnessing collective intelligence. I published the transcript here, after making a quick pass through the 5600-word document and making the following changes:

screen cast		screencast
Homes"		Holmes'
Suns		Sun's
peer two peer		peer to peer
computer farm		compute farm
but kicked		butt kicked
baring		barring
tasks		takes
librafox		LibriVox
swan.com		salon.com
acceptability		accessibility
here		hear
approvable		a provable
statistic		statistical
resource tool		resource pool
from rare		some rare
go over		go after
if		of
flash		Flash
same		same way
task it		task in

The most amusing substitution, as Ben Hill pointed out, was swan.com (weight loss surgery and breast enlargement) for salon.com.

I'd rate the quality of the transcription as very good. Reading through it raised several interesting issues. First, there's the question of domain expertise. A tech-savvy transcriber wouldn't have written peer two peer, for example. Although CastingWords could segment its workers by domain, Amazon's MTurk doesn't yet provide a market system for valuing different domains of expertise -- or, for that matter, different levels of skill within a domain. That possibility is, in fact, one of the most interesting points that came up in our discussion.

Then there's the question of whether, or how, to edit transcripts. An accurate transcript can be painful to read. Recently, for example, Robin Good published an interview with me on his MasterNewMedia site. It is scrupulously accurate, and as such it reproduces all of my verbal tics. The worst one is that, instead of "um" and "you know," I tend punctuate speech with "right?" -- there are seventeen instances of that tic in the transcript.

The ear is more forgiving than the eye, so my own procedure, when transcribing, is to make people read better than they sound. If Quentin Clark were to compare this transcript of our interview with the raw audio, I'm sure he would find it to be somewhat different -- but pleasingly so.

Of course this kind of editing is much more time- and labor-intensive. I only allotted myself 15 minutes for a quick run-through of Hill/McFarland podcast transcript. It would have taken a lot longer to produce a text that would read smoothly. Note also that the audio from which the podcast was transcribed was itself carefully edited by me.

At each level of the process -- audio post-production, transcription, editing -- there are elements that I'd want to outsource and elements that I'd want to do myself. In audio post-production, for example, I'd want to do all of the selection and rearrangement, but I'd be happy to outsource things like normalization, mixing, and MP3 production. Then I'd like to review and correct the transcript as I've done here. And if I could find competent editors at attractive rates, I'd like to see if it would be feasible to review and revise their work rather than to do all the editing myself.

All this, of course, presumes that someone is paying for the final product. In the case of my podcasts themselves, never mind transcriptions of them, nobody is, at least not yet. But somebody might, in one way or another, so it's possible I could justify 42 cents per minute to publish raw transcripts routinely. Is it worth doing? Let me know what you think.

Update: David Hochman writes:

Is there a schema to represent a transcript as structured data that identifies the timepoint of each translated phrase in the audio file? Is there a common URL format (like a target) to represent a precise spot or range in an audio file?

If that data was set free, a whole bunch of someones would use their superpowers to mashup an audio editor, MP3 player, transcription, RSS, OPML, Bayesian filter, tag cloud, TTS (text-to-speech), and more.

The results could improve translation, synchronize a transcript with the audio, allow editing the transcript alongside the audio, transform audio into a human-readable web page or document with audio excerpts, and who knows what other magic.

Second, Doug Kaye's item about how the dynamic assembly of audio files affects things like the MP3 clipping service implemented by ITConversations and by me.

As is true everywhere else, this stuff needs to be an interplay between emergent applications and standards that coordinate them. The ideas have been around for a long time. Now routine use of such applications is within reach. But there aren't many people reaching for them yet, so it's still one of those chicken-end-egg deals.

James Andrewartha writes:

Annodex is the free software standard for annotating media. It even comes with a wiki that can be used by a community to do transcription, eg http://media.annodex.net/cmmlwiki/OSSForum-Trailer?start=8.

Annodex bitstreams are authored by interleaving CMML and audio-visual bitstreams into one multiplexed Ogg container.

Today's podcast, the fifth in my Friday podcast series, is a conversation with Nathan McFarland and Benjamin Hill. I met Nathan at ETech, where he demonstrated CastingWords, a podcast transcription service that uses Amazon's Mechanical Turk to distribute and coordinate work. Benjamin's project, Mycroft, packages up puzzle-like tasks in ways that people can interact with on web pages. You can find a write-up on Mycroft in this CommerceNet technical report.

Tim O'Reilly's favorite Web 2.0 tagline is harnessing collective intelligence, and these are two leading examples of that idea. I thought it would be fun to invite Nathan and Benjamin to compare notes, and indeed it was. Nobody knows what the social and economic ramifications of these systems will be, but we're about to start finding out.

So I'm down at the music store, buying yet another cable. Those 3.5mm connectors I've been using all these years? Kid stuff. Real men and women use quarter-inch and XLR connectors, and now that I'm set up with a Telos One and a mixer, I'm taking my first tentative steps into the world of grown-up audio.

Anyway, the credit card I fish out of my wallet is due to expire, so I switch to the new one and mention that I should remember to shred the old one. The music-store guy cracks me up with this line: "Identity theft? They can have mine. I hope they make out better with it than I did."

For about a week now, the following announcement has been inserted into my bank's login page:

If your login ID has leading zero(s) and you are a bill payment user, you may experience difficulty accessing the Pay Bills section of Internet Banking. Please enter your login ID without the leading zeros in order to access Pay Bills. For example, if your login ID is 0012abcde, please use 12abcde instead. We are working on a fix to this problem.

The login form in question is already being dynamically tweaked by a JavaScript function, so it's only necessary to add this line to that function:

form0.v1.value = form0.v1.value.replace(/^0+/,'');

It's crazy how often services require people to perform textual transformations that can trivially be accomplished by regular expressions. The classic example is the purchase form that fails because a card number's hyphens or spaces were intermixed with digits. Clue: the hyphens or spaces are aids to the human being who is trying to process 16 digits using a brain that's wired for shorter sequences. Punctuation is a feature, not a bug. It is trivial, in any programming language, to ignore it. And yet we continue to shift the processing burden from machines to people.

I used to think this happens because programmers on the front lines of data processing don't grok regular expressions well enough. That's evidently still true, but the real problem, I think, is that the people who use these systems don't trust their own instincts. They suspect it's bullshit, but they can't be sure.

Well, trust me, it is. There are many inhumane procedures that software systems must necessarily inflict on people, but stripping punctuation and leading zeroes are not among them.

Update: Here's an excellent comment received in email from someone identifying as Wanderley:

The problem is that even that one Javascript line still needs to go thru SOX audit before being pushed to production.

This is indeed an issue to blame on pointy-haired bosses, but their names are Sarbanes and Oxley.

Bruce Timberlake adds:

Why isn't the bank preventing your local page from working? Seems to me that a referrer check, session ID, or something should be required to prove that the form is even being submitted from their own server, rather than someplace else.

Linux is home base for open source software, and things always go much more smoothly in that environment. I can almost understand how some folks throw away OS X and install Linux on Apple hardware -- almost, but not quite. Until Linux feels better than OS X on a laptop, I guess I'll keep playing these games...although I've got my eye on a nice-looking Underwood typewriter. [Full story at InfoWorld.com]

Dominique Hermsdorff:

Yes, it was fun to install from source in the 90's. Today, you can achieve more productivity by using automated installers such as yum. I find the title of your opinion piece a little bit unfair because nowadays you can install packages quite quickly without having to build from source.

I agree that the title did unfairly tar all of open source with the brush of OS X's peculiarities.

M. David Minnegerode:

You should install fink. It's a package manager like darwin-ports, up2date, apt-get, yum, etc., etc.

I do use installers and package managers when I can, and lately I've seemed to be getting better results with darwin-ports than with fink. However in this case, neither seemed to have all the pieces I needed.

Brook Milligan:

It is true that there are a number of issues when dealing with creating a nontrivial collection of installed open source software. This is, of course, not limited to open source software; how many times have you tried to add a proprietary package to a Windows box only to discover that the dependencies were not quite right, leading into the same morass you mention?

The point I would like to raise is that there exists a solution that both deals with the problem fairly well and does so in a way that captures expertise from many different platforms, thereby making the results usable in a platform-independent way. The system is pkgsrc a packaging system derived long ago from the FreeBSD ports system. It has evolved a great deal, though, and now offers a truly multiplatform way of building internally consistent sets of packages from source, installing binary packages, performing automated security audits, etc., all for a wide array of different OS/hardware platforms (including Mac OS X/Darwin).

Brook hasn't tried pkgsrc with OS X, and neither have I, but it might be worth a shot next time around.

The Underwood typewriter, by the way, is a reference to a 1998 (not 1988, as written) issue of The Perl Journal, whose cover famously featured a configure script running on that venerable box. I searched in vain for an online image of that classic TPJ cover. If anyone can point to one, I'd love to link to it from here.

Update: Brian Jepson had a copy lying around. Here's a photo of the cover. Thanks, Brian!

The April episode of The Screening Room explores the Sun Grid Compute Utility. Part one is a walkthrough of the portal where you buy CPU hours, upload jobs, and run them on Sun's grid for a dollar per CPU hour. When you see how quickly a distributed job can burn through CPU hours, you start to get a feeling for how this might make business sense. Although any Solaris program can run on the grid, the Compute Server project provides Java classes and NetBeans plugins that simplify the process of writing and running distributable Java programs.

A note on audio quality. This is the first screencast for which I've used my new Telos One digital hybrid, which enables me to separate my audio from the audio coming in on the phone line, and edit the two tracks separately. That's a really helpful technique which made last Friday's podcast more listenable than previous efforts. In this case, though, I couldn't exploit track separation because I forgot that Camtasia records in mono. I should have captured the audio in Audacity, or another stereo recording program, instead. Live and learn...

In my fourth Friday podcast we hear from Kingsley Idehen, CEO of OpenLink Software. I wrote about OpenLink's universal database and app server, Virtuoso, back in 2002 and 2003. Earlier this month Virtuoso became the first mature SQL/XML hybrid to make the transition to open source. The latest incarnation of the product also adds SPARQL (a semantic web query language) to its repertoire.

There are a number of perspectives from which to approach this podcast. If you're a student of open source databases, you'll want to understand the wealth of capability that Virtuoso brings to the table. If you've noticed Adam Bosworth's fingerprints all over Google's GData, you'll be challenged by Kingsley's vision of a continuum of web data access strategies ranging from rudimentary Atom and RSS protocols to remote SQL, XQuery, and SPARQL. If you're intrigued by Apple's Spotlight and Microsoft's forthcoming WinFS, you'll want to consider how an open standards, open source universal database could be the foundation of what Kingsley describes as "a web-enabled virtual Spotlight." And finally, if you're a Rails developer you'll want to be alerted to a couple of forthcoming Virtuoso enhancements: an ActiveRecord provider, and FastCGI support.

Update: Transcript now available.

The modernists, progressives, freethinkers and humanists who group-blog at Neural Gourmet note that Berkeley has joined Stanford in Apple's iTunes U program.

As Jon Udell wrote back in February, it's wonderful that universities and Apple are doing this but it has the effect of making the material less publicly available than if a link to MP3s had been placed on the universities' websites. That's a valid point I think, and a concern we should keep in mind. However, in the short run I think you've got to say that this is fantastic. [Neural Gourmet]

I used to be an excellent biology student but my knowledge of the topic has faded over time. Now I'm sampling Lew Feldman's general biology lectures as a refresher course. And I'm using iTunes to fetch them from Berkeley's website. Any podcatcher will work, and the data is readily available for remixing. Perfect!

I've added one of Phil Windley's¹ suggested features to iwx: a breadcrumb trail documenting the current filters, with a reset control to release all filters at once. You can see it at work in, for example, this URL which displays my April podcasts. I'm not satisfied with this widget, I took it down for a while, then decided to put it back for a while just to see what folks think.

Here's the problem. In iwx, every piece of metadata is simultaneously part of the display surface and part of the control surface. So, for example, the type label Column might appear eleven times in a given view. Each of those instances displays information, and also affords a way to toggle the corresponding filter on or off.

This might or might not turn out to be a workable scheme, but it's interesting and I want to see where it leads. It's most valuable when used with a multi-valued column -- in this case, the tags column. I really like the effect of a filterable tag cloud distributed vertically across a table of filterable metadata. With tags you have lots of choices to scan, and it's handy to have all of them available as immediate controls.

Single-valued columns work the same way, which is consistent with the experience of toggling the tag filters, yet arguably overkill for the much smaller sets of values that occupy these columns. I've thought about using dropdowns here instead, but consistent point-and-click is seductive and I want to give it a chance to work if it can.

The breadcrumb-trail/filter-reset widget forces the issue. Currently it displays the active filters, but does not present a control surface -- except for the reset link. Should the filter values be active here as well? Arguably that would be more convenient as well as more conventional. But it undermines the notion that the values shown in the main display are also controls. And while the breadcrumbs can only be used to turn filters off, the values in the main display are more potent: they can turn filters on or off.

I dunno. It would be helpful to collect examples of other interfaces that try to combine the display surface with the control surface. If you've seen (or done) one of those, I'd be curious to take a look.

¹ I just listened to Phil's latest Technometria podcast, on the subject of AJAX, and it's excellent. Everyone on the show is really knowledgeable and, although I follow this topic closely, I learned a lot. The most thought-provoking bit for me comes near the end. Phil asks how AJAX will affect the server side of the equation. The discussion turned to Comet, which is one of a number of approaches to nailing up HTTP connections for highly interactive client/server apps. The question of server load comes up, and one of Phil's gang of Utahans -- I lost track of who -- mentions that Novell's Drew Major had long ago built the NetWare kernel to handle vast numbers of connections without breaking a sweat. Here's the quote:

100,000 simultaneous IP connections was nothing, on standard Intel hardware, and Novell has all that technology right in their pocket.

I've tweaked the iwx half of the iws/iwx combo. Some changes respond to feedback (thanks!), others reflect an evolving sense of what I'm trying to accomplish. Here's an update.

The tagline (no pun intended) for iwx was originally metadata explorer, then tag explorer, now I've gone back to metadata explorer. Although dynamic filtering of tag sets is the most obvious thing that iwx does, it also filters along three other axes: article type, author, and date. As was clear from Phil Windley's comments, that wasn't discoverable enough. It still may not be, but I've labeled the four metadata columns as filters. I hope those labels, along with the toggle this labels that pop up when you hover over any value in these columns, will help clarify what's going on.

Greg Oros reminded me of the worst flaw in the original version of iwx. It reported months, not days, in the date column. "I would like to see the full date, if possible," Greg wrote. The reason I didn't originally report full dates was because filtering on an individual day wasn't useful, so I filtered by month instead. But of couse these strategies aren't mutually exclusive. It's perfectly feasible to report the day while filtering by month, and that's how it works now. Note that continuous filtering, a la Oliver Steele's expialidoci.us (see this screencast), is another possibility. If someone wants to take a whack at it, please do (see below).

Expanding out to full dates also solved one of iwx's worst problems. The rows are sorted by date, but with date restricted to month only that was not very useful. Now all iwx views present a true reverse chronological timeline, which is a huge improvement.

The iwx dataset was originally based on InfoWorld-tagged articles. In iws, two tag sources are combined: InfoWorld articles, and my own blog entries. Now the latter are included in iwx as well, and that roughly doubled the size of the dataset.

There are still open issues on the data side. InfoWorld has twenty blogs now, and mine's the only one with tagged entries. It would be nice to include the others, and if at some point they're tagged, I will. I also need to ask our IT guys to provide me with a heads, decks, and leads feed for those blogs, so they can have expanding previews (in both iwx and iws) like the rest of the stuff does.

There are also a bunch of open UI issues. Dick Weisinger asked about searching for tags, as well as freetext. "If I know the tag name already," he writes, "I want to search for it." Now, there is an advanced way to accomplish this advanced task. In iwx (as in iws) the URL-line is fully composable, so you can just type, e.g.,

http://iwx.infoworld.com/iwx/?tags=podcast

But that's a strategy only a geek would love. Should I offer tag search and text search side-by-side, a la Flickr? Probably, but I want to go slowly here because it's hard enough clarifying the relationship between iwx (navigation) and iws (search).

Like Phil Windley, Thomas Burg asked for "an alternative view based on tags (tagcloud), with faceted browsing afterwards." Although I don't think I'd find the combination of two unconstrained tag clouds very useful, you might, and it would be nice to provide the option. Again, however, there's the question about how to provide it in way that's discoverable but not overwhelming.

There are lots of experiments worth trying, and I'm not the only one who can try them. I'm delighted to report that there's already been one iwx mashup, courtesy of Mike Parsons. He scrapes the iwx metadata, and loads it into an AJAX application that does lightning-fast incremental search. Try it! It is, as we New Englanders say, wicked cool.

In order to make things easier for Mike, and to invite others to play along, I'm providing an XML dump of the iwx metadata at this URL:

http://iwx.infoworld.com/iwx/?xml=1

Let's have some fun!

Several folks have written asking about the CSV-to-IIF script that I mentioned in this column. It's a humble thing, and likely won't work directly with your bank's output format, but if it's a helpful example I'm happy to share.

# Script to convert CSV to IIF output. I run it like so:
#
# python qb.py > output.iif
 
import sys,traceback,re
  
def error(trans):
  sys.stderr.write("%s\n" % trans)
  traceback.print_exc(None,sys.stderr)
  
# This is the CSV format that my bank emits. Yours likely differs.
# I copy the CSV data from the bank's export page, and paste it here 
# between the triple quotes.
 
csv = """
4/30/2006,-12.93,"","","EASTERN PROPANE GAS"
4/30/2006,-28.94,"","","AMZ*SUPERSTORE AMAZON .COM WA"
"""
  
# This is the name of the QuickBooks checking account
  
account = "Walpole Checking"
  
# This is the IIF template
  
template = """!TRNS\tTRNSID\tTRNSTYPE\tDATE\tACCNT\tNAME\tAMOUNT\
\tDOCNUM\tADDR1\tADDR2\tADDR3\tMEMO
!SPL\tSPLID\tACCNT\tAMOUNT\tMEMO
!ENDTRNS
TRNS\t\tCheck\t%s\t%s\t%s\t%s\t%s\t\t\t\t
SPL\t\tUncategorized Expenses\t%s
ENDTRNS"""
 
# And here's the part that squirts the data through the template
 
lines = csv.split('\n')
 
for trans in lines[1:-1]:
 
  try:
    list = trans.split(',')
    assert (len(list)==5 )
  except:
    error(trans)
    continue
 
  try:
    (date,amount,serial,descr,comments) = list
    m = re.search('(\d{1,2})/(\d{1,2})/(\d{4})', date)
    assert ( len(m.groups())==3 )
  except:
    error(trans)
    continue
 
  try:
    amount = float(amount)
  except:
    error(trans)
    continue
 
  serial = serial.strip('"')
  if ( serial == ''):
    serial = 'visa'
 
  descr = descr.strip('"')
 
  comments = comments.strip('"')
  if ( comments == ''):
    comments = descr
  comments = comments.replace('ATM POS Debit/','')
  comments = comments.replace('ACH Withdrawal/','')
 
  print template % (date,account,comments,amount,serial,-(amount) )

Today's podcast is an interview with Chris Gemignani. We've crossed paths a couple of times, and I've been greatly impressed by his combination of skills. He's an extreme Excel hacker, has a Tuftean sensibility about data visualization, and uses his screencasts to open a window onto ways of thinking about, and doing, analysis of business data.

Chris and his brother Zach run a consultancy called Juice Analytics, and they write a blog that aspires to be "like 43 Folders for the Office set."

In our conversation we discuss data wrangling, the ubiquity yet tragic misuse of Excel, the failure of data marts, the social dimension of customer data, the prospects for agile and iterative ways of interpreting that data, the role of animation, and the new horizons opening up for in-memory processing. At the end I promote Chris to be the czar of Excel, and he describes his plan to revitalize it. Fun!

Update: Transcript now available.

These screencasts are not reviews. They're examples of a new hybrid form -- the demonstration/interview. Back in my BYTE days, it was my privilege to receive a steady stream of visitors bearing the fruits of the tech industry. I had a front-row seat at hundreds of fascinating demos and discussions. I always wondered what it would be like to capture and share those sessions. Now I know.

If you're a viewer, think of me as a proxy. The goal is to use my knowledge and experience to steer the demonstrations in the directions you would like them to go, and to ask the kinds of questions you would be inclined to ask. If you're a presenter, think of me as a friendly adversary. My goal is to blow past your slide deck, drill down into use cases, and challenge you to show how your technology can help me -- and the viewers for whom I'm acting as a proxy -- solve real problems. [Full story at InfoWorld.com]

This week's column reflects on the evolution of my screencasting and podcasting work. It's been a great opportunity to improve my speaking skills, and to learn more than I ever imagined I'd know about A/V media formats and editing techniques. There's still one nagging technical problem, though: audio quality. Everything runs through the telephone, and my results range from acceptable to sketchy.

Long ago Doug Kaye told me that the only solution was a hybrid coupler, so the other day I ordered a Telos One. Not being a specialty audio gear kinda guy, this was a strange experience for me:

Our sale price is WAY too low to display on the internet!

For your special discounted price, email us at sales@bswusa.com, or call us at 1-800-426-8434

Sherman, set the Wayback Machine to 1955!

I call Brian, leave voicemail, he calls back, and peforms the work usually assigned to Perl scripts -- that is, he collects my name, address, and credit card info.

There's a reason why we assign this work to Perl scripts. When my card was refused, I assumed it was a random security check. But no, the credit card company told me that Brian had entered my expiration date wrong. Twice. Another call to Brian, another voicemail, another callback, finally it's sorted out.

Or so I thought. Ten minutes later I receive two emails from Brian. The first is for the Telos, plus $20 shipping. The second is for two cables, $10 each, plus $44 shipping. Say what? Another call, another voicemail, another callback. I think we're squared away now. Sheesh.

Anyway, I'll report back once I've got the gadget installed. I'm not expecting miracles -- I think the quality of my phone line may be the ultimate gating factor -- but I'm hoping for some improvement.

Yesterday I recorded a commentary about LibraryLookup for New Hampshire Public Radio. But when the NHPR folks looked at the bookmarklet generator, they threw down a warning flag. This is pretty geeky stuff, and they quite sensibly don't want to get into the business of tech support. So as a special favor to my home state I went through this list of New Hampshire public libraries and created drag-installable bookmarklets for all of the libraries that I could.

For Amherst, Bedford, Derry, Goffstown, Hooksett, Manchester, Merrimack, Salem, and Wadleigh:

NH LibraryLookup

For other towns:

Concord, Conway, Dover, Exeter, Gilford, Hudson, Hollis, Hopkinton, Howe, Keene, Kimball (Atkinson), Laconia, Lane (Hampton), Lee, Nashua, Portsmouth, Richards (Newport), Rochester, Rye, Smyth (Candia), Tracy (New London), Wiggin (Stratham), Wilton

(This turned out to be a useful exercise. I found two catalog systems that I hadn't known about -- Winnebago and Athena -- and added support for these to the bookmarklet generator.)

If you're a patron of one of these libraries and would like to give LibraryLookup a try, drag the link for your library to your browser's toolbar, then click it from an Amazon book page. Here are some tips for using the bookmarklet.

If all this sounds like Greek to you, here's a tutorial screencast on bookmarklets. It shows, among other things, how to create a LibraryLookup bookmarklet, how to install it in Internet Explorer or Firefox, and how to use it.

Bookmarklets are handy, and the screencast will introduce you to some other useful ones, but I should point out that they're not the ideal way to automate library catalog lookups. In particular, they're not powerful enough to translate among the ISBNs for different editions of the same book. I've recently developed two techniques that use a service provided by the Online Computer Library Center to do this translation. The first one works inside the Firefox browser. The second connects your Amazon wishlist to an RSS feed.

If this sounds like Etruscan to you then don't worry about it, just use the appropriate bookmarklet. These more advanced methods will eventually show up. If you're lucky enough to have a superpatron in town, they'll show up sooner rather than later.

I've written separately about a pair of services for discovering InfoWorld articles: power search and tag explorer. Now they're deployed on an InfoWorld server as iws (search) and iwx (explorer). We haven't tied these into the InfoWorld site yet, but that'll likely happen soon. Meanwhile I want to test the waters by announcing these services here and inviting feedback. (I'll just assume that email sent to that link is publishable here.)

These services work in tandem. The search component, iws, mashes up the results of our Ultraseek search engine, exploiting three streams of metadata: structured document titles, del.icio.us tags, and the introductory parts (decks and leads) of each article. Clicking on any tag takes you out of search mode and into the mode of tag navigation, or exploration, or what Matt McAlister has called pivoting on tags. There you click around to explore the sets of articles associated with combinations of tags and other metadata. The search box takes you back to search mode.

It was easy to connect the two services in this way, and I think it's really useful, but it's certainly different from what folks are used to, and I've learned the hard way that being different from what people expect is not always a good thing. Hence this soft rollout. I want to see how all this plays to the readers of this blog, then use that feedback to guide a more aggressive rollout.

In particular, I wonder about the fact that the two sides of the coin -- iws and iwx -- look very similar, but do different things. Will it be necessary to clarify the relationship between them? We'll find out as we go along.

One difference I hope everyone will appreciate is the spare and uncluttered design. There are no ads and, except for InfoWorld's standard footer, no extraneous outbound links. These services are 100% dedicated to helping you mine the rich vein of articles that we've published over the years, find things that interest you, and make efficient decisions about what you want to read. It's a radical concept, I know, but let's give it a whirl.

iws: iws.infoworld.com/iws

iwx: iwx.infoworld.com/iwx

A decade ago I consulted with a federal health agency on the redesign of its website. They wanted to use the web to make more health care information readily available to more people, but they didn't have much to offer. My suggestion was to leverage what we now call commons-based peer production, which then was mostly happening on the Usenet. The feds were deeply conflicted about that. They knew people were exchanging lots of useful information in newsgroups. But they also knew there was a lot of quackery, and they couldn't imagine themselves separating the wheat from the chaff. It was a valid concern. There was no way that they, alone, could patrol the likes of alt.support.arthritis and highlight the most useful advice for, say, people recovering from knee replacement.

Over the years I've known a few folks who've gone through that procedure. Last night I spoke to a friend who had it done recently. He told me he found lots of helpful online resources about the surgery itself, as well as about the rehab process. And he's collected them into a kit that he's been sharing with others who are headed down the same chute. But he had to do all this on his own. Nobody on his medical team pointed him to any of this stuff.

Medicine is, among other things, a kind of information monopoly, as are other professional fields including IT. It's inevitable that peer production will challenge these information monopolies, and medicine is a particulary interesting test case.

Watching for signs of change, I've been following medical blogs and podcasts. One that caught my ear recently was this interview with J. Scott Armstrong, a Wharton School professor whose interests include scientific peer review and transparency in medicine. At one point he discusses his own experience with prostate cancer. This guy is clearly not a typical patient. He regards his personal physician as an adviser who points him to relevant medical literature, discusses it with him, and helps him reach decisions.

At one point, faced with possible prostate cancer, Armstrong's doctor referred him to a specialist because his PSA (prostate-specific antigen) was at level 5. The specialist told him that because it was level 5, he should have a biopsy. "Well yeah," Armstrong says in the interview, "I can read." He expected the specialist to add value -- to customize that general and widely-known recommendation in ways that accounted for his particular circumstances. That didn't happen.

Not everyone is willing or able to dig into the medical literature. But some are, and they can publish what they find for the rest of us to discover. My friend is happy with his new bionic knees, and he wants to blog the resources that were helpful to him. In particular, since rehab is so crucial to successful outcomes, he's thinking about putting up some videos showing how he uses his cycling machine.

Over time, as more such resources accumulate online, the web's natural peer review and reputation effects will kick in. Health care folks can't vet all this stuff, but they shouldn't have to. People will vote with their links for the information that's valid and useful. What will health care folks do then? Two things, I hope. First, work with us to gather and refine useful sources. Second, use their expertise to guide our interpretations of those sources.

This shouldn't always require an office visit, by the way. In one of the Stanford podcasts (at this iTunes-only URL, sorry), Daniel Sands gives a wonderful high-energy talk about how and why we need to augment face-to-face physician/patient communication with asynchronous modes like email. We don't know how to bill for such services yet, I suppose, but it's time to figure that out.

Today's podcast is a conversation with Steve Burbeck on the topic of multicellular computing. I first met Steve at Tim O'Reilly's 2001 summit conference on peer-to-peer technologies. Steve was at IBM then, and was working on UDDI among other things. (He's an author of the UDDI spec.) His background prior to IBM includes consulting and product management in the realm of object-oriented development tools -- especially Smalltalk. Before that he ran data processing and statistics at the Linus Pauling Institute of Science and Medicine, acquiring the understanding of biological systems that he now hopes to apply to networked computer systems.

It seems intuitively obvious that nature's strategies for managing complexity in the biological realm can help us keep control of our information technologies. But nailing down exactly which strategies may be relevant, and why, and how to apply them, is going to be a challenge.

Steve identifies four themes from biology that he thinks might usefully translate to information technology: specialization, polymorphic messaging, stigmergy, and apoptosis. At least two of these terms may be unfamiliar to you, as they were to me. Stigmergy, Wikipedia currently says, is "a method of communication in emergent systems in which the individual parts of the system communicate with one another by modifying their local environment." Apoptosis is cell suicide.

It was great fun exploring with Steve how these principles could help us understand, and manage, the multicellular computing fabric that we're creating. The paper that you'll hear us mention, in which Steve spells out these ideas, was unpublished at the time of our interview, but is available now at here.

Update: Transcript now available.

I have a superpower that enables me to do battle with the evil of data lock-in. I can't leap tall buildings or crush lumps of coal into diamonds, but when I look at the barriers that divide one data format from another, they seem hardly to exist. For me, data transformation is almost an autonomic reflex, like breathing.

But PDF? Please, oh please, don't make me dig the data out of those PDF files. [Full story at InfoWorld.com]

The hero of the piece -- apart from emacs, that is, whose regular-expression search-and-replace function got me where I needed to go -- was Investintech's Able2Extract. I've only used that program on a trial basis. If I'd spent more time with it could probably have gotten it to produce something closer to the final result I needed. But then, I'd rather not have to spend more time on such nonsense.

Of course that's just wishful thinking. I'll probably be fighting data lock-in for the rest of my life. I can't disavow my superpower. So long as data yearn to be set free, I'll be coming to the rescue, my utility belt festooned with regexes and parsers and scripts. It's a lonely job, but somebody's gotta do it.

Can you instantly find everything -- at company, departmental, or workgroup scope -- that you're allowed to see? I thought not. Posting information to the public Web is far more likely to guarantee successful search than posting to the intranet. [Full story at InfoWorld.com]

Would it surprise you to learn that internal search sucks even at some companies whose business is search? It shouldn't. Indexing the public web is, in one sense, a very simple problem. Anything your spider can reach is fair game. Private webs, though tiny by comparison, are far more complex.

The good news is that service orientation can help us make search suck less inside the enterprise. Expanding on what I said in the column, I think there are three reasons why that's so:

1. Pervasive intermediation. Service fabrics use intermediaries -- aka, internal chokepoints -- to control namespaces, monitor performance, and enforce policies. When these fabrics carry documents that contextualize business processes, as well as the messages that enact those processes, those documents will be more reliably available for indexing and search.
2. Enriched metadata. When documents are more closely connected to the business processes they contextualize, they're described by metadata that helps organize search results.
3. Network-based security. When access control is lifted out of applications and placed directly into the fabric, it's easier to figure out who is allowed to find what.

When it comes to public speaking, I've still got a lot to learn. I've tried two modes, and neither is ideal. In mode one, I write and deliver speeches, like these. It's a useful strategy which guarantees that I'll manage to say everything I want to say, coherently, and that a written transcript will be left behind. It also guarantees that I know exactly how I'll use my allotted time. But I'm never entirely satisifed when I hear speeches read aloud, and I figure that mine must leave the same impression. Maybe it would help if teleprompters were more widely available, but they aren't.

In mode two, I talk through a series of slides. The slides are mostly pictures -- screenshots, and recently also bits of screencasts. I use the pictures to evoke the stories I want to tell, and remind me in what order to tell them. This strategy has sometimes worked well, for example at OSCOM 3, and when it works well I think it's most satisfying both for me and for the audience. It's risky, because I'm much more adept as a writer than as a speaker. But it's a risk I keep taking, on occasion, because I want to get better at doing things this way.

At Emerging Technology I opted for mode two. About ten minutes into my half-hour talk, a five-minute warning flashed on the monitor facing the podium. Five minutes? Oh shit! (I didn't just think that, I said it, which Nat Torkington later told me was the funniest moment of the day.) The rest was a blur. A couple of minutes later the monitor flashed a warning -- something like "Better wrap this puppy up, pronto!" -- and I raced through my slides only to discover, after hastily concluding, that the monitor was in error and I had actually had almost ten minutes to spare.

Live and learn. A more experienced speaker wouldn't have been so easily rattled, and next time I won't be either. Meanwhile, now that I've heard the edited version version put together by Daniel Steinberg, I'm feeling better about the outcome.

I recently heard Daniel's own talk on audio editing techniques. It's an excellent primer that I'll certainly return to now that I'll be doing my own podcast interviews on a more regular basis.

The art and science of editing, in all of its forms, fascinates me. I've been on both sides of the fence. As an editor, I try to make people sound more like the versions of themselves they intend to be. As an editee¹, I am grateful when an editor does that for me. So thanks, Daniel!

Update: John Mitchell proposes another way to prepare a talk:

I suggest a Mode Three, which was used by the BBC WW2 radio correspondents, and then by Edward R. Murrow. Written speeches sound too formal, so the BBC people would pace around, composing their reports completely orally. A stenographer would write them down, to be used as notes when the radio version would be spoken. Murrow used the same technique to great effect, and it sounds like a fun technique.

¹ Yes, I know, that's not a word, though of course you know exactly what I mean. Then again, maybe it is a word. Imagine if Samuel Johnson had had our ability to discover and promulgate the decrees of custom?

In today's podcast I interview Gary McGraw, CTO of Cigital and author of, most recently, Software Security: Building Security In. It had been a decade since we'd spoken, and our discussion touches on a number of things beyond the scope of the new book -- including the Letter Spirit project, which was Gary's contribution to Douglas Hofstadter's Fluid Concepts and Creative Analogies.

You can hear another long interview, focused more tightly on Gary's new book, at ITConversations. It's part of Sondra Schneider's Frontline Security series. And if you've only got ten minutes to spare, InfoWorld's Bob Garza talks with Gary in this podcast, one of the flurry of interviews Bob captured at the RSA show.

One thing Gary and I didn't discuss -- because I only just became aware of it -- is his musical life. After you listen to the interview, read the book, and improve your software security, you might want to kick back and listen to some of his tasty mandolin and fiddle work on Dirt Road and World Inside. This music, made with Rhine Singleton, who lives -- small world -- just down the road from me, is freely available. I'm listening to Restless Water right now.

Anyway, the interview was entertaining and enjoyable for us, and I hope will be for you too. I've done a few of these in the past, but will try to make them a more regular thing, perhaps even weekly.

Update: Transcript now available.

Folio, the recursively-named magazine for magazine management, has included me in its list of 40 industry influencers, in the Under the Radar category. Thanks to the Folio folks, and to Paul Conley who recommended me to them. I am honored to have my work recognized in this way.

It's no secret that, in the little corner of the magazine world known as the IT trades, we've been struggling. But that's nothing new. When I joined BYTE, in 1988, the glory days were already in the past. I loved that magazine, and I still regularly run into people who miss it as much as I do. But the truth is that the only way forward was online. When the axe fell in 1998 I'd made a lot of headway, but hadn't yet assembled all the tools, skills, and knowledge that would be needed to make the transition. Nor was the business ecosystem ready for that to happen.

The business ecosystem frankly still isn't quite ready, but I'm confident it will come around. There is constant, if not growing, demand for what the IT trades aim to supply. And we are now in a position to meet that demand with an array of information products that are collectively far more valuable than what we historically could supply. In the past I've sometimes envied Wayne Green for the opportunity he saw, and grasped, in 1975. But not lately. It's much more interesting to look forward.

The other day my mom tried to call an old friend in San Francisco, whom she planned to visit in a couple of months. But the number was disconnected. It was pretty clear what must have happened to this elderly gent with advancing emphysema, who had always lived alone in his little house on the slope of Twin Peaks. But there was no immediate family to contact, and from 3000 miles away my mom had no idea how to get in touch with the neighbors.

In last week's column I mentioned Tom Standage's The Victorian Internet, a history of telegraphy which suggests that 19th-century folks would be less amazed by our modern Internet than we might guess.

In terms of instantaneous transmission of messages, that might be true. But I think Victorians would be amazed by search. It still even amazes me on a regular basis.

In this case I searched for the street address and found a realtor, Pota Perimenis, who had listed a property on that street. She put my mom in touch with a neighbor, from whom we learned that our friend had died at home, in his sleep, peacefully. Thanks for your help, Pota. It meant a lot to my mom. And thanks, Google, for finding Pota.

Today's 8-minute screencast (Flash 8, earlier Flash player) is a meditation on Live Clipboard, the hCalendar microformat, Upcoming and Eventful.

I've been experimenting with microformats since before they were called that, and I'm completely jazzed about Live Clipboard. In this screencast I'll walk you through examples of Live Clipboard in use, show how the hCalendar payload is wrapped, grab hCalendar data from Upcoming and Eventful, convert it to iCalendar format for insertion into a calendar program, inject it natively into Live Clipboard, and look at Upcoming and Eventful APIs side-by-side.

All this leads up to a question: How can I copy an event from one of these services and paste it into another? My conclusion is that adopting Live Clipboard and microformats will be necessary but not sufficient. We'll also need a way to agree that, for example, this venue is the same as that one. At the end, I float an idea about how we might work toward such agreements.

The lightly-tweaked Live Clipboard demo that you'll see in the screencast is available here, and the bookmarklet that extracts events from Upcoming and Eventful is here. Both are Firefox-only for now.

Update

At the end of the screencast, I suggest that collaborative aliasing could help us connect various representations of the same thing -- for example, the same event venue. David Janes points out that we would do better to express an event's location by pointing from one microformat (hCalendar) to another (hCard). For location data I think that's true, and as David says, Eventful already works that way. But there's more going on here.

At Eventful (but not at Upcoming) a venue can have tags, associated URLs, and comments (What people are saying about this venue). Upcoming's venues don't have unique features of their own, but they could. We will arrive at the most complete picture of a venue by combining what Eventful, Upcoming, and perhaps other services know about it.

A different example may help clarify what I mean by collaborative aliasing. Here are five URLs that refer to Jack London's Call of the Wild:

1. A DVD of the 1975 movie, in Amazon's catalog.
2. A hardcover edition of the book in the Barnes and Noble catalog.
3. An OCLC xISBN lookup that finds a long list of Call of the Wild ISBNs -- none of which, however, corresponds to items 1 and 2.
4. The LibriVox audiobook version that I mentioned last month.
5. The Wikipedia page about the book.

What naming scheme could we use to connect these in a grassroots and collaborative way? The screencast mulls the notion of a tagging convention, like so:

In this example, the tag Book_JackLondon_CallOfTheWild is the connective tissue.

Like David Janes, Danny Ayers prefers URIs. Of the five listed above, the Wikipedia URL would clearly be Danny's first choice. "If there are fairly solid reference services like Wikipedia or IMDB," he writes, "then use their URIs."

I'll go along with that, so long as the URIs are easy for people to invent, to read, and to write. And so long as they can function as tags in social classification systems -- which for now, it seems, they cannot.

The March episode of The Screening Room explores Microsoft's forthcoming AJAX toolkit, Atlas. This month's presenter is Shanku Niyogi. Our 25-minute interview will give you a good sense of Atlas as a browser- and server-independent framework, and also as an ASP.NET extender.

Atlas has come along nicely since Scott Guthrie first introduced it last fall at the Microsoft Professional Developers Conference. It remains to be seen how widely it will be adopted by non-ASP.NET folk, but the Atlas crew seem to have done a nice job of encapsulating dependencies and carving out a component model. There is, for example, an abstract interface to local storage. It's bound to Internet Explorer's local datastore, but in principle can be easily redirected to the forthcoming Firefox datastore. I'll be very interested to see how that plays out.

As you'll see in the demo, Atlas also invents its own declarative layout language, called Atlas XML Script. It's tool- and component-friendly, though I do wonder about adding one more item to a list that already includes XAML (Microsoft), XUL (Mozilla), MXML (Adobe/Macromedia), LZX (Laszlo), and some others. It really would be nice to start pulling these together.

As AJAX toolkits continue to multiply like rabbits, I also expect that standardization of JSON (JavaScript object notation), and interoperability among JSON implementations (for JavaScript as well as for other languages), are issues that will show up on the radar screen later this year.

Tom Standage's history of telegraphy, The Victorian Internet, draws striking parallels between that era's communication revolution and our modern one. A 19th-century citizen transported to today would be amazed by air travel, Standage suggests, but not by the Internet. Been there, done that.

Multiprotocol routers? Check. Back then, they translated between Morse code and scraps of paper in canisters shot through pneumatic tubes.

Fraud? Check. Stock market feeds were being spoofed in the 1830s, back when the telegraph n