Jon's Radio (full-length descriptions)

Data formats for digital democracy: XML vs CSV

2006-12-13T09:46:04-05:00

As a first experiment I grabbed the DCStat reported-crime feed for November, sucked it into Excel 2003, consolidated incidents by day, pivoted them on type of offense (homicide, burglary), and exported them back out as a CSV (comma-separated value) file that Swivel could import. [Full story at InfoWorld.com]

Here's one of those pivot tables in Swivel. The auto-generated charts don't do much for this style of dataset. But the point of this week's column is that just publishing a named dataset, along with pointers to the raw data, is inherently valuable.

I imported the same data into Dabble DB where it's very easy to use grouping and filtering to make views like this one. Again the point is that the views are sharable on the web. Also, in this case, invited collaborators can tweak them.

Going through this exercise, I was struck by the distance between DCStat's namespace-rich XML formats and the CSV format that web apps like Swivel and Dabble DB want to read and write. I happen to know how to use the XML Maps feature of Excel 2003 to shred an XML file but I doubt many Excel users have ever done that. To enable ordinary citizens to explore this data, DCStat might want to offer a common-denominator CSV format in addition to the XML flavors.

Comments

AJAX and automation

2006-12-12T09:06:39-05:00

Sean McGrath's latest column puts an AJAX spin on the age-old struggle between the human-accessible and machine-accessible aspects of software:

Making an application easier to use almost always means making better use of the users ability to see; to hear; to click buttons; to access drop down menus and so on. Unfortunately, computers themselves cannot see or hear. This results in a most unfortunate inverse relationship. Namely, the more visual and interactive and compelling an application is from a user interface perspective, the harder it is to make said application do its thing without human intervention. [Sean McGrath: AJAX and the hidden cost of ease of use]

And yet, my latest Gmail hack was accomplished pretty easily because, without digging into any AJAX code at all, I was able to capture an HTTP transaction, parameterize it, and replay it.

Now in fairness, I cheated by using the AJAX interface to emit a sample HTTP transaction that contained a lot of stuff I could not easily have fabricated from scratch. That's why it'll take a much smarter bear than me to incorporate my hack into libgmail. Still, if I'd wanted to plow through Gmail's obfuscated JavaScript source code, I might have been able to intercept and repurpose the method driving the HTTP transaction.

For these two reasons -- the transparency of the HTTP pipeline, and the accessibility of the JavaScript object model -- I think that AJAX is inherently more automatable than conventional GUI apps ever have been.

Comments

Moderated 2nd-level domains

2006-12-11T11:34:03-05:00

At the ICA conference in September I had a great talk with Edwin Bruce, who's responsible for New Zealand's e-government initiatives. Among many other things, he pointed out that New Zealand has an opportunity to do some interesting things with moderated second-level domains under the .nz top-level domain. There are five such domains now: .govt.nz (government), .mil.nz (military), .iwi.nz (Maori), .parliament.nz, (parliament) and .cri.nz (Crown Research Institute).

Because New Zealand's country-code top-level domain (ccTLD), .nz, is much more tightly controlled than, say, .us, New Zealand's moderated 2nd-level namespace is analogous to ICANN's notion of sponsored/chartered top-level namespace. If you're a bank or a law office today in New Zealand, you'll probably register under the unmoderated 2nd-level domain .co.nz. Edwin envisions aligning the existing regulatory apparatus for banking and law with corresponding moderated 2nd-level domains: .bank.nz, .law.nz.

This wouldn't be an option in the US, at least not under our ccTLD. For example, .law.us is currently held by Neustar, but .bank.us is owned by Vishal Ved, 9335 Lee Highway, #1213, Fairfax, AL. So if we wanted a system like the one Edwin proposes, we'd need to do it under a new TLD.

I'm sure that's unlikely for all sorts of reasons, and what Edwin proposes isn't even happening yet in New Zealand where it pretty easily could. But it would be interesting to see this model tried there. Are there other ccTLDs where it's farther along?

Comments

Bombshell aftermath

2006-12-09T22:21:35-05:00

Reading through the comments attached to yesterday's announcement made it clear that I left a few critical questions unasked and unanswered. Rather than continue that thread here, I'm transplanting it to a new incarnation of my ancient personal website. It used to be udell.roninhouse.com. Now it's jonudell.net, where I display my past record for inspection, and this blog for new thoughts and dialogue. I hope you'll join me there because during my sabbatical, from Dec 15 to Jan 15, I'll be laying the foundations for what I want to do next, and I'll appreciate all the help I can get.

For my last week here, I'll stick to InfoWorld-related knitting. But I can't resist pointing out my favorite of all the heart-warming reactions that poured in following yesterday's announcement. It's a 3-second video from Andy Rush, one of the switched-on instructional technologists I met at the University of Mary Washington back in May. Andy, you made my day!

A conversation with Jon Udell about his new job with Microsoft

2006-12-08T07:25:38-05:00

For today's podcast I decided to interview myself about my upcoming new gig. It's a short episode, under six minutes, and the transcript follows.

Note: I actually meant to push this to the server later today, to synchronize with a message that will be forthcoming from Jeff Sandquist. But a mis-click on my part pushed it sooner, which means Jeff will be a bit surprised when he wakes up. Trust me, though, this is something I've been thinking carefully about for a long time, and it's very real.

Q: Your new job is with Microsoft?

A: That's right. My last day at InfoWorld will be Friday Dec 15. On Jan 15, after a month-long sabbatical, I'll become a Microsoft employee. My official title will be Evangelist, and I'll report to Jeff Sandquist. He's the leader of the team that creates Channel 9 and Channel 10, websites that feature blogs, videos, screencasts, and podcasts for Microsoft-oriented developers.

Q: What will your role be?

A: The details aren't nailed down, but in broad terms I've proposed to Microsoft that I continue to function pretty much as I do now. That means blogging, podcasting, and screencasting on topics that I think are interesting and important; it means doing the kinds of lightweight and agile R&D that I've always done; and it means brokering connections among people, software, information, and ideas -- again, as I've always done.

Q: Why are you doing this?

A: I'm often described as a leading-edge alpha geek, and that's fair. I am, and probably always will be, a member of that club. But I'm also increasingly interested in reaching out to the mainstream of society.

For those of us in the club, it's a golden age. With computers and networks and information systems we can invent new things almost as fast as we can think them up. But we're leaving a lot of folks behind. And I'm not just talking about the digital divide that separates the Internet haves from the have-nots. Even among the haves, the ideas and tools and methods that some of us take for granted haven't really put down roots in the mainstream.

Over the years I've evangelized a bunch of things to the alpha-geek crowd: Internet groupware, blogging, syndication, tagging, web architecture, lightweight integration, microformats, structured search, screencasting, dynamic languages, geographic mapping, random-access audio, and more. There's a purpose behind all this, and Doug Engelbart saw it very clearly a long time ago. The augmentation of human capability in these sorts of ways isn't just some kind of geek chic. It's nothing less than a survival issue for our species. We face some really serious challenges. The only way we're going to be able to tackle them is to figure out how to work together in shared information spaces. I've chosen to align myself with Microsoft because I think it has the scale, the resources, and the business incentive to help me empower a lot of people to learn how to do that.

Q: Why now?

A: At the Emerging Technology Conference in March, Microsoft's incoming chief software architect, Ray Ozzie, showed how LiveClipboard, the 21st-century version of the Windows clipboard, could enable collaborative sharing of information, and creative recombination of services, across all operating systems, web applications, and desktop applications.

Kim Cameron, Microsoft's identity architect, is taking a similar approach in the domain of identity, privacy, and the control of personal information.

Jean Paoli, Microsoft's Office XML architect, continues to pursue his lifelong dream of empowering millions of people to create and use smarter documents.

Jim Hugunin, who created both Jython and IronPython, is making my favorite open source scripting language, Python, a first-class citizen of the .NET platform.

J.J. Allaire is creating a blog-writing tool that will enable millions of people to publish data that's reusable and intelligently searchable.

Bottom line: This isn't your father's -- or maybe your older brother's or sister's -- Microsoft. Initiatives like these matter, they're solidly in line with my own agenda, they're being pursued in very open ways, and I want to help move them forward.

Q: Are you selling out, joining the Evil Empire, and turning your back on principles you've always championed?

A: Wait until the evidence is in, then decide for yourself. I've been in this game for a long time. I think my record of pragmatism and agnosticism speaks for itself, but sometimes I like to recall what Tim O'Reilly said in his foreword to my 1999 book on Internet groupware:

All too often, people wear their technology affiliations on their sleeve (or perhaps on their t-shirts), much as people did with chariot racing in ancient Rome. Whether you use NT or Linux, whether you program in Perl or Java or Visual Basic - these are marks of difference and the basis for suspicion. Jon stands above this fragmented world like a giant. He has only one software religion: what works.

I claim that was, is, and will continue to be true. If it stops being true in the future, I expect you to hold me accountable. But meanwhile, I hope you'll suspend disbelief until the evidence is in.

Q: Will you be a blogger? An analyst? A developer? An educator? A multimedia producer?

A: All of the above. The title "evangelist" doesn't quite capture that whole range of activities, but these are the things I do, and plan to continue doing.

Q: Will you become Microsoft's next Robert Scoble?

A: The way I see it, Robert played a key role in a grand experiment to make Microsoft's development processes more transparent. Channels 9 and 10, and the hundreds of Microsoft blogs throughout the organization, are evidence that the experiment is succeeding.

I've proposed a different experiment. I'll continue to be a channel for alpha geeks. But I also want to become a channel for a whole lot of civilians in the mainstream. And above all, I want to build bridges between these two groups.

Q: Will you continue to use Firefox, Gmail, and OS X?

A: Sure. I'll also continue to use Microsoft technologies as I always have, and I'll keep on pushing the boundaries of cross-pollination and interoperability. The most powerful mashups don't just mix code and data, they mix cultures. I hope this will be an opportunity for me to do that in a way that benefits everybody.

Comments

General-purpose intermediation

2006-12-07T21:39:21-05:00

The solution I cobbled together speaks volumes about the fundamental openness of Web applications. To find out how Gmail creates a distribution list, I logged in, created a list interactively using Gmail's form, and captured the resulting HTTP transaction using one of the handiest tools in my Web developer's kit, Firefox's LiveHTTPHeaders extension.

The next step was to replay that transaction outside of the browser. I rearranged its elements -- an URL, a chunk of HTTP POST data, and a set of HTTP headers including a cookie packed with crucial name/value pairs -- as a command-line invocation of another of the handiest tools in my kit: curl. [Full story at InfoWorld.com]

In this week's column, I discuss how the ability to capture and replay HTTP traffic enabled me to discover and exploit an implicit Gmail API. But there's a general principle underlying this hack, and it seems to me that after all these years we've barely begun to exploit it.

Consider the scenario described in this item, for example. I would love to be able to recapture the sequence of HTTP transactions behind a particularly interesting search scenario, but I never logged them. In the comment thread, aristus notes that one solution is a Firefox plugin called slogger, which I've used on and off since, let's see, tap, tap, tap, October 2004.

I haven't used slogger for a while because it quit working for me on OS X a while ago, and I haven't been able to resurrect it since. But slogger notwithstanding, there's a much deeper and more general thing that ought to be happening on every web-enabled system. It ought to be trivial to attach an observer and/or filter to HTTP pipelines. Among other things, it could shovel data into a search engine so that I could instantly recall a remembered transaction by search term, by date, or by site.

I mentioned this on a recent call with the folks reponsible for Vista's desktop search. When they mentioned extensible "protocol handlers" I got really excited, imagining a general mechanism for echoing HTTP (or SMTP) traffic through a search indexer. It turned out that isn't what they meant. They were talking about supporting file formats, not protocols. But I've always thought echoing HTTP or SMTP traffic through a searcher is a great idea, and I still do.

HTTP intermediaries are also an incredible untapped resource for developers and testers of software running in environments that range from plain old HTML-over-HTTP to formal XML web services. For the latter domain, the folks (disclosure: my friends) at Mindreef are doing really interesting work based on the ability to capture and replay SOAP packets.

I've long envisioned a general-purpose ipchains-like capability, for all operating systems, that would make it trivial to attach observers, filters, and transformers to bread-and-butter protocols like HTTP and SMTP. Is it just me, or is this a gigantic missed opportunity?

Comments

"We found the version control, collaboration and invite system outweighed the limited feature set"

2006-12-05T09:10:20-05:00

Google got there first, but we're still in the early innings of this game. Google's office apps, while collaboratively adept, are functionally lame. Microsoft's apps are adept and lame in precisely the opposite ways. Everyone needs to converge on solutions that deliver the best of both. [Full story at InfoWorld.com]

In last week's email I received a leading indicator of GDoc/GSheet adoption which, because I neglected to ask permission to quote, I'll anonymize:

Just read your column on G-docs. We started using them around the office for convenience and they've taken off. We found the version control, collaboration and invite system outweighed the limited feature set. For the most part, they have the very basic functionality covered. We still use Excel for many things, but Google Spreadsheets is making a dent in our enterprise use. My clients have adopted it too. Helps when working with teams in China, Europe and the US.

My boyfriend and I use one to keep track of our bills at home. Just easier than sorting it out over email.

The italics (mine) call out as pure an example of a Clayton Christensen-style disruption as you are ever likely to find.

Comments