Grid Meter | September 2005

MORE ENTRIES

September 2005

September 30, 2005 | Comments: (0)

As the industry gets more familiar with the general Grid computing adoption trend-lines, we're starting to see the analyst firms drill down more on specific vertical markets. The 451 Group, for example, just released a report on "Grid Computing -- Adoption for Digital Media Creation and Distribution." It's refreshing to see the discussion expand beyond the traditional (pharma, financial services, energy) markets.

This write-up summarizes some of the key findings of that report, and I found this quote to be particularly interesting:

"A big difference between the film industry and other vertical markets is the technology is being used by people who think of themselves foremost as artists, not enterprise IT end users."

While this point may be fairly obvious, I believe that it does indeed apply to many other vertical markets as well, and is one of the primary barriers of widespread adoption. Quite simply, Grid computing needs to be a transparent technology before it is widespread. How many of us would be browsing the web if we had to hand assemble http queries in a telnet window?

Even though most IT end users tend to think more about their day jobs than what's behind their displays or network cables, there are a few who take the extra step. These are the early adopters that are working to figure out what will make that transparency possible.

One of the reasons I am looking forward to GridWorld next week is to get a chance to meet some of those early adopters. I look forward to hearing the stories of successes and failures that led to their ultimate deployment of Grid.

Posted by Greg Nawrocki on September 30, 2005 01:38 PM

September 28, 2005 | Comments: (0)

Cisco: Intelligence in the Network ... AND the Datacenter

The way the IT publishers tend to cover the industry is to draw a neat little line separating network management from systems management. If you're a network admin, you're supposed to read Network World, Network Computing, Light Reading, etc. If you're a systems admin, you're supposed to read InfoWorld, Computerworld, eWeek, Informationweek, etc. While it does make sense for professionals to specialize in certain areas ... what's interesting about enterprise IT today is this artificial "boundary" between the two worlds.

But when the networking world's biggest player, Cisco, announces new product lines for datacenter virtualization (as they did today) -- it's painfully clear how vulnerable that line is between network and systems management.

It was well-publicized when Cisco acquired Infiniband provider Topspin for >$200 million earlier this year. Infiniband is a very fast, very high throughput (I/O) interconnect for server clustering in datacenters ... and has implications for Grid computing as well. If you want to learn about how it compares to other interconnect technologies like Ethernet and Myrinet, check out this white paper.

But Cisco's greater datacenter virtualization play is more profound than just interconnect fabric between servers (not to say that Infiniband isn't interesting though). It's about networking products that are more "application aware," and more sophisticated provisioning of storage and network resources (the VFrame management software "gives a single interface to provision Cisco data center infrastructure rather than addressing each Cisco product and technology individually," according to the release).

It's too early to gauge exactly where this is all headed, but it is clear that Cisco is carving out networking's space in systems management -- and really the sky's the limit. And as they say, imitation is the highest form of flattery -- it's a certainty that "datacenter virtualization" will be creeping into many other networking players' boilerplates in the near future.

Posted by Greg Nawrocki on September 28, 2005 09:20 AM

September 26, 2005 | Comments: (0)

Troubleshooting SOAs and Grids Not Going to be Easy

With BEA World kicking off this week, there's a ton of noise about J2EE environments and SOAs. This article today from Network World highlights the recent trend of Grid computing as the preferred application server for SOAs. In 2004 (and again in this Computerworld column last year), Ian Foster called out the fact that SOAs and Grid computing were on converging paths (via web services standards and capabilities) ... it's interesting to see how quickly that prediction has been realized, with mainstream end users like Wachovia and Acxiom running SOAs on top of service-oriented Grid infrastructures. Another interesting point in the article is that one of the challenges that we often see called out as an inhibitor for Grid growth is that applications have to be re-written for Grid environments ... but Wachovia points out that Grid-enabling applications with Java actually isn't that difficult.

Another discussion that I believe will start to get more air-time is what it means to troubleshoot SOA / Grid environments.

Even in the pre-SOA world, one of the problems that we see most consistently around application interconnections is the human labor factor. People set applications up, they think they understand what their behavior will be, but later on, with hundreds of connections and things going wrong in real-time, figuring out what's where and what's happening in relation to other sub-systems ... it's extremely difficult.

"Typically, troubleshooting three tier system architectures (web server, application server, and database) like WebLogic consists of hours, or even days, manually sorting through various logs including web server requests, java exceptions, diag logs, and system configurations in dozens of different places each time something goes wrong," says Michael Baum, CEO of a new datacenter search start-up called Splunk. "And this usually requires systems administrators, database administrators, and developers to cross administrative domains to determine various system interdependencies ... and to identify where an exception or break-down may have occurred."

Baum says that over the last few decades of IT, the systems management industry has really grown up on a complete focus on physical infrastructure management -- performance, provisioning, CPU utilization, process utilization, etc. But what's really lacking is easier ways to understand the logical layer, and the interdependencies between all of the different hardware resources and software services. The ability to reverse engineer that logic at run-time to troubleshoot systems today is a very human-intensive task that's typically done with very blunt instruments like Grep and Perl and Awk. A medium sized enterprise data center could be generating anywhere between 10 to 100 megabytes a day in its logs, and that figure is up in the terabyte range for large enterprises. As Grid and SOA adoption continues to move forward, troubleshooting scenarios will get increasingly complex.

Posted by Greg Nawrocki on September 26, 2005 12:51 PM

September 22, 2005 | Comments: (0)

Re. the future of thin-client, on-demand mobile devices / Grid infrastructure

How many times has this happened to you? ... You hear a song you like (on the radio, at the gym, at a party), but you have no way of finding out who the artist is, or what the name of the song is.

I recently had the opportunity to meet Avery Wang, chief scientist at Shazam Entertainment. These guys have created a really cool new service that allows you to simply flip on your mobile phone when you hear a song you like (whether you're at a bar, a restaurant, etc.), dial up the Shazam service -- and within seconds you get the artist name and song name. From there, you can immediately download the ringtone to your phone. As mobile phones and mobile music players converge, it doesn't take much of a stretch of the imagination to infer that in the very near future, consumers will be able to take it to the next step and simply download the given song on their phone.

So what's this have to do with Grid computing?

Nothing, really -- it's more of a clustering approach. Shazam uses a very low-latency architecture (a big Linux cluster running diskless on commodity, 64-bit hardware). When a user submits a music quiery (an applet sitting on the phone does a short recording and sends an audio file to the Shazam service, via Java), the request goes to the master node, then gets multicasted over TCP/IP to a 'recognition cluster.' The nodes in the recognition cluster are filled with hash tables containing indeces of the music.

Shazam keeps tabs on 2.5 million songs -- but does not actually store all of the songs in the infrastructure (which would be prohibitively cumbersome, storage / latency-wise). Instead, they use proprietary "fingerprinting scripts" to tag songs. These fingerprints are very small compared to actual music files -- so the time required to compare a quiery against the known field of 2.5 million songs is on the order of only milliseconds.

As Grid computing continues to evolve, I believe that in the near future we will see enterprises similarly use mobile clients to invoke services in a Grid computing infrastructure -- but instead of using low latency tags of fingerprinting systems, the infrastructure will be able to leverage large file transfer capabilities to accommodate end user requests. Really creative new services like Shazam make you wonder what the future of enterprise mobility will look like, and how large scale enterprise applications will behave in this sort of model.

Posted by Greg Nawrocki on September 22, 2005 06:45 AM

September 21, 2005 | Comments: (0)

A look ahead to the GridWorld event in Boston

The inaugural GridWorld event is right around the corner -- in Boston, in approx 2 weeks.

It's interesting to me -- and perhaps a sign of maturity for Grid -- that a professional events company (IDG World Expo) was hearing enough demand for Grid content and education from the enterprise to justify having a commercial-focused event. What's unique about GridWorld is that it blends the enterprise issues with the nuts, bolts and standards discussions of GGF.

I'm planning to attend - and I had a recent discussion with Steve Crumb, Executive Director of the GGF (who's a co-organizer of GridWorld) to get his insight into what he perceives to be some of the highlights of the event.

Steve mentioned that similar to the GGF Enterprise Grid event in Brussels, there are some strong enterprise deployment stories to be told. There will certainly be the familiar faces, however, there are also a set of new players eager to tell their Grid stories. Among them are financial services powerhouses Merrill Lynch, Bank of America, and even Johnson and Johnson in the life sciences vertical.

Another program highlight Steve mentioned was the event's international perspectives. Presenters from companies in Japan, Europe and the US will speak to their Grid deployments and give the audience the ability to compare and contrast these deployments.

Although there will be a strong enterprise focus, Steve wanted to stress that the technical edge will be sharp. Grid technologists will find plenty of deep technical presos and tutorials, including updates from the open source community on version 4 of the Globus Toolkit, and features on new innovations in Security, Portals, and Grid management.

Finally, Steve was clearly excited about Ian Foster's panel, "The Different Faces of IT as a Service." This panel will be a timely discussion with key enterprise Grid players to help clear up some of the confusion surrounding Grid computing and some of the terms and definitions that often go hand in hand or are confused with Grid.

Posted by Greg Nawrocki on September 21, 2005 08:27 AM

September 19, 2005 | Comments: (0)

10g Graduating from Clusters to Grid

Today Oracle unveiled Oracle Application Server 10g Release 3. It's being touted as a major update to Oracle's SOA middleware platform, and it may be a major addition to enterprise Grid computing in general.

One of the question marks around 10g as a "Grid" solution in the past has been whether it's really Grid technology -- or a cluster technology.

Recall the three key summary points from the definition of Grid, in the eponymic (as if Grid were a person!) paper of 2001 "The Anatomy of the Grid"by Foster, Kesselman and Tuecke.

1) Enable integration of distributed services & resources
2) Using general-purpose protocols & infrastructure
3) To achieve useful qualities of experience

"Grid" implies a level of interoperability and distributed resource sharing among heterogeneous resources spread throughout multiple administrative domains. In the past, the nature of 10g was really more of a clustering solution.

But per the language in this new release, Oracle has taken steps to ensure features such as integration and support for heterogeneous environments are included. It is also reported that it will allow 10g middleware components to integrate with existing enterprise infrastructure. This is an unprecedented level of aggregation for a 10g product. There is a list of over 128 commercial enterprise software packages Release 3 will interoperate with. Point one was clearly taken to heart.

Point two has also been addressed. Release 3 is also standards driven. Built on a foundation which includes a wide range of web service standards support as well as support for the SAML 1.0 and 2.0 security standards. Perhaps most impressive is the commitment to open source software applications. It will be certified with a large number of these applications which support open standard protocols.

Although the general availability release is scheduled at the end of fiscal year 2006, it may be some time before we see point three answered. However, if the above does indeed hold true, Release 3 may be one giant leap for enterprise "Grid kind."

Posted by Greg Nawrocki on September 19, 2005 02:14 PM

September 14, 2005 | Comments: (0)

What Would a Grid Domain Name System Look Like?

Another interesting new area for grid security is the growing discussion around developing a handle system for the grid.

This Handle System could be an alternative implementation that you could use for attribution servers and naming servers in general. The handle system, which is being worked on by the Corporation for National Research Initiatives (CRNI), would not only provide attribute services but it would also serve as an infrastructure and root service able to resolve resource names globally. It is very much a domain name system (DNS) type of model. You have a global naming system and values or attributes that are bound to that name. It's like the DNS on steroids -- security is truly integrated into the whole fabric. It will have all the good features of transparent applications, and it allows individuals to administer their own bindings, so you can push the access rights of the bindings down to the individual names.

The concept of having a centralized root system for registering grid resources is interesting, as we consider the future of 'extra-grids,' where coordinated resource sharing requires us to think about distributed policy requirements and resource discovery issues.

David Holtzman, former CTO of Network Solutions (acquired by Verisign for $21 billion in '00), led the team that ran the DNS in the late '90s and oversaw the growth of the Internet from 500,000 domain names to more than 20 million. Network Solutions' contract with the National Science Foundation meant that anyone who wanted to have a domain name and participate in the Internet had to go through the Network Solutions domain name registrar system.

Holtzman sees the grid computing handle system as the logical next step in the grid evolution, and he thinks the collective body of vendors with commercial interests in grid would be smart to stand behind it.

"Managing millions of domain names was a tremendous challenge, but the idea of accounting for billions of resources participating in a global grid is mind-numbing," Holtzman said. "Having the inventory of resources consolidated in a central broker seems like a logical step to solving the issues. One lesson I've learned from the bad-boy days of the early commercial Internet is that harnessing distributed power is not so much a matter of leveraging the sum of the individual components but of building an appropriate framework so that each constituent can derive value from the whole without being forced to make one-off tactical decisions in the enterprise. Building a handle system empowers the lowest management point in the organization to fully utilize the technology without constantly building organizational consensus. I believe that the DNS system, for this reason, was the prime catalyst for the rapid adoption of the commercial Internet in the late '90s."

This Globus handle system project intends to provide a Web services interface to the handle system leveraging standard interfaces, like SAML attribute query interfaces, XKMS queries, with simple name/value resolutions.

Posted by Greg Nawrocki on September 14, 2005 08:00 AM

September 13, 2005 | Comments: (0)

Three Networking / Grid computing questions for Vint Cerf

A few months ago (when he was still at MCI), we asked Vint Cerf a few network-specific questions about Grid. Here's what he had to say:

1. A recent Nemertes study said that 62% of organizations are using MPLS today or plan to deploy it. We've seen a ton of industry interest in the convergence of web services and Grid services -- but there has been very little discussion (at least in the news) about the connection between Grid computing and MPLS. Is there a connection? Does the MPLS architecture have any interesting implications for Grid, in your opinion?

The notion of "deploying MPLS" is a little odd. MPLS is an infrastructure technology. Typically, users don't see it since the typical interface to an MPLS-enabled network is still IP. In that sense, the use of MPLS is not unlike earlier use of Frame Relay and ATM to transport IP packets. MPLS, like its earlier counterparts, is used for two purposes: creation of managed underlying infrastructure for traffic engineering of IP-based networks and to separate virtual private networks from each other). MPLS uses a two layer stacked-label design to accomplish these tasks, at least in some implementations.

The value of MPLS is that it runs at the same speeds as IP these days, unlike Frame Relay and ATM that both showed some inability to scale up to the increasingly high speeds of the Internet backbones. However, one should be careful about assuming that MPLS will continue to scale without limit. It could be that it, too, will find itself running out of capacity, only to be replaced by optical packet switching or some other technology yet to be invented.

For that reason, it is important to take care, architecturally, to maintain a common IP interface at the edge of the Internet (or virtual private networks) so as to allow migration from one underlying infrastructure to another.

2. In a recent InfoWorld article, you highlighted some of the key enhancements in IPv6:

-facilitates identification all the way to the end device
-auto discovery and configuration -facilitates apps that require fast, plug-and-play connections among peer devices on the edge of the network
-eliminates the need for NAT (network address translation)

There seem to be strong similarities between these new improvements in IPv6 -- and the areas that the Grid community is working hardest to address (identification/authorization, application performance, etc.). What can/should the Grid community take into consideration (in your opinion) with respect to making best use of new directions in IPv6 for the Grid environment?

In most ways, the Grid is a much higher layer construct than the IPv6 Internet layer. Consequently, it is important to take some care not to bind Grid overly closely to the network layer of the Internet architecture. Grid may be able to derive some important utility when running over IPv6 (authentication, dynamic reconfiguration, mobility) but for the same reason as the cautionary notes above concerning MPLS. IPv6 may someday have to be replaced with something else and one would want the Grid structure to survive operation in a mixed environment. In fact, we want GRID to work in mixed IPv4/IPv6 environments, too, so we should be cautious about binding Grid too closely to either one. The identifiers that Grid should rely on for identification of sources and the like ought to be at higher levels of abstraction than IP addresses, in my opinion.

3. Do you see any possible future directions for how resources (both hardware and software) are registered, discovered and identified in Grid environments? Will there be a next-gen resource registry (like the DNS on steroids)?

There certainly could be a higher level of resource registration. I would expect that registration of services would want to be bound to an identifier space that is well above IP addresses. These identifiers could be found to IP addresses dynamically or they could be bound to domain names that are in turn bound to IP addresses. But one would want the identifiers to survive changes in the IP layer. Examples of such bindings can already be found in instant messaging services such as ICQ, MSN, AIM and the like. There, the instant messaging identifiers are bound dynamically to the IP addresses of the clients. These change with time and the systems are generally very adaptive to these changes. I think the Grid environment needs to have similar flexibility. One might turn to the Object Identifier community for some insights into the possible uses of new classes of identifiers for service and data objects.

Posted by Greg Nawrocki on September 13, 2005 08:14 AM

September 12, 2005 | Comments: (0)

Why Meteorologists Care About Grid

When we discuss Grid computing's role in mass-scale data crunching and predictive analysis scenarios, many of us associate the discussion with the financial services industry - where Grid powers complex Monte Carlo models to help large brokerages glean real-time insights / predictions for capital market performance.

Another world of data that's even more volatile and dynamic than today's capital markets is weather prediction.

Today, 142 NEXRAD (next generation) Doppler radars are sitting around the country, scanning the atmosphere 24/7 and collecting the finest-scale, highest temporal resolution data of any observing system yet deployed. But the systems which use these data -- to both detect hazardous weather using real time decision support systems as well as predict its evolution via numerical models, operate in static configurations independent of the weather. And for the most part so do the NEXRADs, with no ability to zero in on a specific region of a thunderstorm and scan it intensely in order to verify the presence of a tornado.

Kelvin Droegemeier, Director for Analysis and Prediction of Storms at the University of Oklahoma, is part of a research team called LEAD (Linked Environments for Atmospheric Discovery) -- which is figuring out how to leverage a Grid cyberinfrastructure to bring more real-time intelligence to the data collection, analysis and numerical prediction.

"The LEAD project aims to Grid-enable a lot of the capabilities of the numerical models, so they can dynamically adapt themselves to new weather patterns," said Droegemeier. "As the data comes in from the radar, we want the models themselves to adapt to the atmosphere, reconfigure themselves to concentrate on new areas of interest, and yield more optimal results to predict specific types of weather events, like thunderstorms."

In order to get to this new level of autonomic computing to support weather prediction, the LEAD project is developing a Grid service infrastructure, built on the Globus Toolkit and GGF standards and protocols.

"The idea here, and the connection to Grid, is that we would like to be able monitor these radars in real time and have data-mining services that are looking at their output," said Dennis Gannon, Professor of Computer Science at Indiana University (one of 9 institutions participating in the program). "The data mining, if it's well done, can actually take that radar stream and pinpoint interesting things developing, automatically. And when that happens, it will trigger a particular simulation workflow on the Grid. That workflow will do a number of what we call 'data assimilation tasks' and assimilate all the observable weather data that they've got in a given region and prepare it for a set of simulations. So we then might use TeraGrid or a large grid to launch maybe a 100 different simulation scenarios for a single county in Oklahoma."

Posted by Greg Nawrocki on September 12, 2005 07:59 AM

September 08, 2005 | Comments: (0)

Embedded Grid

As I may have mentioned before, I'm an old embedded systems / consumer products guy. I've fought in the trenches of the "converged devices" wars. I've had to define products that early on in the marketing cycle it became clear that no one really wanted. And I've had to convince various emperors that they actually had cloths on, which was often shamefully easy to do.

This week's edition of the Economist features a special report on "The Digital Home" aptly titled, "Science fiction?" The article mentions that these complex converged devices of the digital home are invariably complex and people like simplicity. It goes on to mention the often touted "spoke and hub" architecture of the digital home where all the various devices are connected to a central hub. Of course the one thing that can't be settled on is who's technology makes up that hub. This issue has been the stumbling block for the digital home for as long as I can remember.

Two intrepid engineers at HP are working on a project called GridLite. GridLite is a proof of concept effort that is pushing the limits of the types of devices that make up a Grid. They are taking embedded devices such as PDAs, smart phones and other mobile platforms (the very devices that are usually mentioned in the same sentence as "device convergence" and "digital home") and equipping it with a Grid middleware. Central to this is specialized resource manager that orchestrates this abstraction of device connectivity.

In a nutshell, GridLite brings the promise of Grid to embedded devices. Enabling the integration of distributed services and resources, using general-purpose protocols and infrastructure, to achieve useful qualities of experience.

I'm not saying that Grid can solve the problems of technology that no one really wants, and it can't put clothes on the emperor, but it just might be able to end the hub and spoke mentality and the digital home can finally move on to its next incarnation. Whatever that may be.

Posted by Greg Nawrocki on September 8, 2005 09:00 AM

September 07, 2005 | Comments: (0)

Virtual Workspaces

Naturally, with virtualization being the rage right now -- enterprise IT folks are curious to better understand how virtual machines will participate in Grid environments.

A group of scientists working on Virtual Workspaces for an upcoming release of the Globus Toolkit has written a paper that I find quite compelling. The paper explains the concept of "Virtual Workspaces in the Grid" -- and how virtual machines themselves fit into a Grid environment.

In particle physics (ironically one of the basic sciences that makes frequent use of Grid) scientists often try to understand the larger picture by breaking a phenomena into elementary particles and studying the way those fundamental quanta behave. The virtual workspace concept presented in this paper is just that, a basic building block for virtualization that may just be the key to getting the concept of virtualization working concisely in the grander scale of a Grid computing environment.

Posted by Greg Nawrocki on September 7, 2005 01:46 PM

September 06, 2005 | Comments: (0)

We Virtualized Our Servers ... Now We Have to Manage Them?

IDC has reported a whopping 62% growth in virtual machine software over the last year, according to this Network World article. The focus of the article is the intense amount of virtualization management solutions being released by the systems management vendors -- and how these vendors are going about the actual management of the virtual servers themselves.

The irony, of course, is that while virtualization has been so heavily hyped as a way of simplifying server provisioning and management -- virtual machines themselves are a resource, and yield a new type of complexity that in and of itself must be managed.

During a virtualization talk at LinuxWorld New York back in May, Rich Green (of Cassatt) said "virtualization is a means rather than an end that can be used to address a growing set of issues in the IT Industry. So rather than looking at virtualization as a stand-alone phenomena, the industry should consider it as a tool or technique to create the necessary abstractions to release applications and services from the details of systems, networks and storage."

I've been receiving an increasing number of questions from folks about the connection between virtualization and Grid computing. Similar to what Green said, while virtualization uses some neat tricks that create separation between applications and underlying resources -- the Grid community is addressing the broader set of technical challenges associated with coordinated resource sharing between virtual organizations.

Virtual machines started out on single boxes (SMPs and mainframes). Today organizations are managing clusters of (usually running on similar types of boxes) virtual machines. As we move towards connecting these clusters of virtual machines together, a whole new range of problems (large scale data replication, network provisioning issues, security issues) come into the picture ... and these are the distributed computing challenges that Grid is uniquely equipped to address.

Posted by Greg Nawrocki on September 6, 2005 12:19 PM

September 01, 2005 | Comments: (0)

Infiniband's Role in Next-Gen Data Centers

I received one of my favorite embedded systems trade rags in the mail yesterday. The theme of this month's issue is "InfiniBand, the next generation of networking." Infiniband seems to have been christened the preferred datacenter interconnect by the networking industry. It works around an architecture called RDMA (remote direct memory access) -- which basically allows the physical components of nodes in a fabric to talk directly, without having to go through the overhead of the TCP stack or bog down the operating system.

Insiders sort of pigeonhole Infiniband's value into high performance computing environments. But I believe that as enterprise continues to evolve to "scale-out" environments (commodity hardware virtualized to behave like big SMP or mainframe systems) -- these discussions around the communication (and latency, I/O throughput, performance issues) between networked servers within a datacenter will become more mainstream. Infiniband will certainly play a huge role in Grid environments (and already does, in e-Science Grids).

So virtualization functionality is rapidly moving down the stack into the network. These initial discussions about data center interconnect and physical network performance are just the tip of the iceberg in terms of networking discussions for Grid.

The networking players aren't just debating the preferred interconnect between servers within a datacenter -- they're debating the preferred network topology for connecting Grids.

Posted by Greg Nawrocki on September 1, 2005 09:57 AM

Technology White Papers

InfoWorld Technology Marketplace

» Technology White Papers Library

Technology White Papers by Topic

Technology White Papers E-mail Alert