Grid Meter | June 2006

MORE ENTRIES

June 2006

June 26, 2006 | Comments: (0)

GGF and EGA complete merger, announce Open Grid Forum

The Enterprise Grid Alliance (EGA) and the Global Grid Forum (GGF) today announced the completion of their merger, forming the Open Grid Forum (OGF).

I first mentioned this merger in the Grid Meter back in February.

The OGF will be led by former GGF Chairman Mark Linesch, who will serve as president and chief executive officer.

I'm pleased to see that Mark will be the man at the helm for two reasons ... Mark is a good leader, and he surrounds himself with good people.

In fact, I'm sure that Steve Crumb, the GGF Executive Director and Mark's right hand man, had a hand in getting this deal pulled off. I hope to see Steve in his rightful place as first officer on the OGF ship.

Posted by Greg Nawrocki on June 26, 2006 02:41 PM

June 22, 2006 | Comments: (0)

Oracle's Grid Survey Not Real Telling

Oracle recently announced results from the fourth Oracle Grid Index.

I'm not going to say a whole lot about it, although the previous Grid Index reports have been met with little fanfare. I'm not sure why as they are quite detailed and their data collection techniques appear sound.

What I do find interesting are some of the numbers that look at the Global Trends in Grid Computing, especially in light of recent talk regarding North America trailing Europe in Grid adoption and usage. This study differs with that opinion.

However some of the text from the above article make me wonder how fresh this data is. I'm not sure that this is the fault of the study itself, after all the data shows what the data shows, and as I mention above, their collection techniques have been sound in the past.

But when I hear things like "The latest research confirmed that Grid Computing is a maturing technology and highlighted the rise of Service-Oriented Architectures (SOA) in maximising the business benefits of Information Technology (IT)" and "All regions show positive movement in the awareness, understanding and adoption of Grid Computing" my first reaction is to whack the side of the CD player because it sounds like the disc may be skipping.

Posted by Greg Nawrocki on June 22, 2006 07:18 AM

June 21, 2006 | Comments: (0)

Around the horn: Grid community discussions around SOA this week

Grid Today has run a number of good articles over the last week around SOA.

Art Sedighi, Senior Consulting Engineer at DataSynape illustrated the link between SOA and Grid.

"As remote execution or Remote Procedure Call (RPC) has evolved to Web services, batch systems have evolved to Grid computing -- allowing us to think of the two technologies as complementary."

Art makes this point to show that Grid computing has a home in the SOA environment by being a key enabler for system elements that make SOA run.

"Grid computing software can take care of resource management for your enterprise and virtualize your underlying infrastructure and available resources. It takes control of your resources, schedules services to run as requests come in, prevents denial of service attacks and meets Quality-of-Service requirements set forth by users."

ComputerWeekly also ran an article with a different perspective on SOA. The author makes the argument that his investigation has caused him to draw the conclusion that SOA leads to increased complexity and transitively, reduced reliability.

Derrick Harris, in an analysis of insights given to him by Gartner's Carl Claunch makes the following statement:

"If Claunch is correct in saying that 'Grid, today ... is being driven mostly by being able to achieve something that's otherwise impossible,' perhaps it would be wise for Grid vendors to focus (more than they already are) on selling Grid computing as a business advantage versus a solution to pre-existing problems -- especially if enterprises are already looking elsewhere to address these issues."

Derrick's point is a good one and just might be the stone needed to kill two birds. Grid Computing could be given a raison d'etre, and Grid vendors could take some of the complexity out of SOA.

Posted by Greg Nawrocki on June 21, 2006 07:30 AM

June 20, 2006 | Comments: (0)

Grid community needs to escalate application development

OK, so maybe I'm not crazy. There are a few people that agree with my perspective on the importance of getting mainstream Grid applications to the forefront so that we can make grid adoption itself more mainstream and demonstrate its true value.

In Ian Foster's "Four Thoughts on Europe vs. North America" in Grid Today last week. Ian specifically calls this out as his final point, and gives a real world example from his near infinite rolodex of experience.

Also, the folks at The 451 Group have released their most recent report from their Grid Adoption Research Service, "Grid Computing Enabling Applications for Grid Deployment."

I've always had a great deal of respect for The 451 Group reports and I'm sure this report from the ever knowledgeable William Fellows and Steve Wallage is full of good insight.

From the press release...

Research conducted by The 451 Group for 451 GARS finds that early adopters are readily seeking ways to deploy grid technologies more broadly within their organizations (moving beyond initial application beachheads). They are looking for additional application tasks that can be brought onto grids and thereby further legitimize grids through use among their existing, commercially supplied and planned new applications. So far only a few applications have been written specifically for grids, and only a small number of today's commercial applications that appear suitable for grids have actually been deployed on or migrated to grids.

You can bet I'll be watching for additional grid app plays.

Posted by Greg Nawrocki on June 20, 2006 07:23 AM

June 15, 2006 | Comments: (0)

Digging Deeper on Microsoft's New Clustering Technology - Part 3 of 3

This is the final installment of Patrick O'Rourke, Lead Product Manager of the Windows Server Division's response to my questions about Microsoft's new clustering product announcement.

3) Grid Meter: [The announcement was] only about the infrastructure ... there is no talk about mainstream applications. For a company (Microsoft) that dominates the mainstream PC applications market you'd think they could have "Compute Cluster Server"- enabled one of them. It's disappointing to see a huge software player come out with a clustering technology without spelling out any sort of tie-in to their applications. Any details here?

O'Rourke: Our goal is to have the category leaders from the target verticals (oil & gas, financial services, engineering, life sciences) port their applications to Windows CCS. And as the June 9 news release shows, we're well on our way to accomplishing that by the end of 2006. This week at TechEd, we have The BioTeam, Ansys, Schlumberger, Parallel Geo, MATLAB and other apps in the booth. You can see more partners on the Windows CCS partner page. In fact, back in November at Supercomputing 2005, we had 20+ partners in our booth. You can read about some of them here.
At TechEd this week, we're also demonstrating the use of Excel client with Windows CCS. (Grid Meter Comment: hooray!) You can read about it in the TechEd keynote transcript. See the section where Kyril Faenov begins his demo. And you can watch the Webcast; Kyril's demo of Excel with Windows CCS is 60-70 minutes into the keynote. And look for more about this next week at the SIA Technology Conference in NYC.

And the folks at National Center for Atmospheric Research ported their well-known Weather Research and Forecasting Model to Windows CCS. WRF was originally developed for UNIX systems. 360,000 lines of code in C++, Fortran, OpenMP and MPI. What's interesting is that only 750 lines required modification to produce native Win64 binary that ran Windows CCS/MPI.

4) Grid Meter: Lastly, and this goes more toward confusion than disappointment, why is it called "Windows Compute Cluster Server 2003?" Was it three years late?

O'Rourke: You're not the only one to ask this. Windows CCS is based on Windows Server 2003 x64 edition , so that's why it has the 2003 designation. Windows Compute Cluster Server 2003 consists of two separate products which can be purchased separately or together: a) Windows Server 2003 Compute Cluster Edition, a 64-bit operating system for industry-standard x64 processors; b) Microsoft Compute Cluster Pack, which includes a Job Scheduler, message passing interface, cluster monitoring tools and deployment tools.

Posted by Greg Nawrocki on June 15, 2006 06:53 AM

June 14, 2006 | Comments: (0)

Digging Deeper on Microsoft's New Clustering Technology - Part 2 of 3

This is part 2 of Patrick O'Rourke's (Lead Product Manager of the Windows Server Division at Microsoft) response to my questions about Microsoft's new clustering product announcement.

2) Grid Meter: High performance computing is not just about raw horsepower. It's about moving massive amounts of data from point to point, interoperability with other systems, and presentation of data and computational resources as virtualized entities so processes that need to access them don't have to spend a lot of time figuring out how to do it. How does Windows Compute Cluster Server 2003 address these data / interoperability, related considerations?

O'Rourke: In terms of moving massive amounts of data, Windows CCS will scale well beyond the mainstream division and department clusters. We already have orders from customers for 1,000+ nodes. And just last week InformationWeek reported:
"The group of Dell machines, which runs Microsoft's new Windows Compute Cluster Server 2003 software, contains 896 processor cores and can perform 4.1 trillion computations per second. The system has a very good chance of making the closely watched Top 500 list of the world's most powerful supercomputers, according to Jack Dongarra, a computer science professor at Tennessee who maintains the list."

There are a number of items in terms of interoperability:

We have a partnership with Platform Computing (the leading vendor for job schedulers) and have done the joint work needed to ensure our respective job schedulers can communicate with one another. For Microsoft customers not doing HPC today, they'll find that seamless integration with Windows Compute Cluster Server 2003 provides "meta-scheduler" functionality for multi-cluster scheduling and bi-directional job forwarding. Existing Platform LSF customers will find that seamless integration of LSF with Windows Compute Cluster Server 2003 enables transparent job forwarding between workgroups and the data center.

Windows Compute Cluster Server 2003 can take advantage of the 64-bit version of Services for UNIX 3.5 in order to run UNIX or Linux applications.

Windows Compute Cluster Server 2003 uses Active Directory for authorization and authentication of cluster users. If the IT environment in which the cluster is being installed does not already include an AD domain, users can set up an AD domain controller on the head node of the cluster. Microsoft also provides tools to integrate AD with other systems such as NIS.

Microsoft's Message Passing Interface (MPI) includes open source code based on MPICH2, a widely used reference implementation. By basing our own MPI on this standard, we have made it easier for ISVs and developers to port existing HPC codes to Windows Compute Cluster Server 2003. We are actively working with Argonne National Lab, including funding and training on Security Reviews per the TwC initiative and will contribute code back to open source.

In terms of custom development - which is big in financial services, government and academia - Visual Studio 2005 includes a parallel debugger and support for OpenMP.

Last, Windows CCS supports GigE, Infiniband and Myrinet.

Posted by Greg Nawrocki on June 14, 2006 07:05 AM

June 13, 2006 | Comments: (0)

Digging Deeper on Microsoft's New Clustering Technology

Ah, citizen journalism at its finest.

Shortly after complaining about some information gaps in Microsoft's new clustering product press release, I received an extensive email from Patrick O'Rourke, Lead Product Manager of the Windows Server Division. I'd like to think that immediately after reading my blog [which he does religiously every day] Bill Gates picked up the phone and called Patrick to have him respond.

Although I suspect the actual chain of events was much less dramatic, Patrick produced some very good and very complete answers to my questions. Given the depth of his responses, I will include them in a series of posts as opposed to one long one. Kudos to Microsoft for such a quick and complete response (a lot of folks in the Grid community I suspect will find the elaborations helpful).

(my first gripe on yesterday's blog)

1) It sounds like this announcement finally brings Microsoft up to speed with where the open source community has been for nearly a decade now ... the ability to group machines in a cluster to solve existing problems with greater speed. Is there any sort of new clustering innovation here that I missed?

[O'Rourke / Microsoft Response]:
Our goal here is to expand HPC beyond traditional supercomputing centers toward departments and divisions in commercial industry and the public sector. One way we're doing that is by delivering an HPC platform that is simple to deploy, operate, and integrate with existing infrastructure and tools. Our innovation is around the simplified deployment, operation and integration experience. [i]nteroperability with existing infrastructure was the design focus for v1. So customers can use Microsoft's MPI or job scheduler, or use an existing MPI or job scheduler. And customers can use remote installation services (RIS) for unattended compute node installation.

This goal is consistent with what some customers have said:

"Adopting Windows Compute Cluster Server 2003 was a natural step for us since we use SQL Server for our database needs and Windows servers for hosting our Web interfaces," said Jaroslaw Pillardy, Ph.D., senior research associate at the Computational Biology Service Unit. "In addition to serving massively parallel applications, I've found that Windows Compute Cluster Server is a convenient tool for serving the computational needs of many small projects, where installing the software, updating databases and managing other such tasks are much easier than on a set of separate computers."

"We're committed to developing a world-class drug discovery platform here at the institute, and an important step in doing so is the need to dock drug candidates against proteins for in-depth analysis," said Matt Wortman, Ph.D. and director of computational biology and IT at the Genome Research Institute, University of Cincinnati. "This application is appropriate for parallelization, which has traditionally represented a complex and costly IT project. However, now with Windows Compute Cluster Server 2003, we're able
to leverage our existing Microsoft infrastructure and skills to reduce costs and improve security via identity management. And because Windows has broad support from application vendors, we're able to partner with BioTeam to quickly build and deploy the drug discovery application."

And you may be interested to know that in 1997 the folks from NCSA deployed the first Windows cluster on NT4. Microsoft began working on Windows-based HPC projects back in 2000, and Microsoft Research began collaboration with several industries, including astronomy (result was Sky Server and the Worldwide Telescope). We've also been working with the Cornell Theory Center for several years.

There are also some interesting tidbits about the Windows Compute Cluster Server here.

One other bit of information that I'll add to Patrick's commentary is that Microsoft Research was indeed one of the original industry supporters of the Globus Toolkit, so in some respect they have a connection to the past decade of work in the open source community.

I'll be in touch with the rest of his responses in subsequent entries throughout the week.

Posted by Greg Nawrocki on June 13, 2006 06:48 AM

June 12, 2006 | Comments: (0)

Microsoft Cluster Announcement ... Where's the Beef?

Microsoft Corp. announced the release of Windows Compute Cluster Server 2003, the company's first software offering designed to run parallel, high-performance computing (HPC) applications.

A lot of folks have been keeping an eye out for this announcement for quite some time. I for one have a few areas where I was a little disappointed with the amount of detail (or lack thereof) disclosed about this new product.

1) It sounds like this announcement finally brings Microsoft up to speed with where the open source community has been for nearly a decade now ... the ability to group machines in a cluster to solve existing problems with greater speed. Is there any sort of new clustering innovation here that I missed?

2) High performance computing is not just about raw horsepower. It's about moving massive amounts of data from point to point, interoperability with other systems, and presentation of data and computational resources as virtualized entities so processes that need to access them don't have to spend a lot of time figuring out how to do it. How does Windows Compute Cluster Server 2003 address these data / interoperability, related considerations?

3) It's only about the infrastructure, there is no talk about mainstream applications. For a company that dominates the mainstream PC applications market you think they could have "Compute Cluster Server"- enabled one of them. It's disappointing to see a huge software player come out with a clustering technology without spelling out any sort of tie-in to their applications. Any details here?

4) Lastly, and this goes more toward confusion than disappointment, why is it called "Windows Compute Cluster Server 2003?" Was it three years late?

If anyone from Microsoft would like to weigh in with answers to these questions (I think they're at least partially representative of some common questions you're likely to get from the Grid community at large), email me at: gridmeter@infoworld.com. Happy to post your response ...

Posted by Greg Nawrocki on June 12, 2006 11:14 AM

June 08, 2006 | Comments: (0)

Linux an innovation ground for Grid networking

As the boundaries keep getting fuzzier between so-called 'systems management' and 'network management' (especially in Grid computing discussions) - it's become clear to me that Linux is the breeding ground where the networking innovation is happening.

Recently we explored what it was that had made Linux the preferred OS for Grid computing in general, and we heard from Adam Fineberg at Levanta that: "Some of the key aspects of an operating system that you really need take advantage of in a Grid computing environment are the networking and file systems. The networking side is very important because of the large number of nodes, the need to quickly / easily add more nodes, exchange information between the nodes with low latency, as well as access shared storage systems and devices. Linux does very good 'zero copy' networking, meaning that once the data reaches the network stack, it doesn't have to be copied again all the way through the rest of the operating system. That really keeps the networking efficient in Linux systems."

It's funny how quickly after you learn something new, you start bumping into it all the time. Because shortly after getting that insight from Adam about the specific networking performance of Linux (that's so well-suited for Grid), I spoke with Franco Travostino at Nortel, whose virtual mash-ups on the Grid are running on Xen hypervisors, and are being proved out in a Linux environment.

"We created a demo by which we took some Xen-based virtual machines that were crunching some particular, computation-intensive tasks ... and we moved them to Amsterdam," said Travostino. "And then from Amsterdam, we moved them to Chicago, and them from Chicago back to Seattle. And in spite of the tens of thousands of miles, the impact on the applications was less than one second. So that's pretty mind-boggling, to think about having a fully featured Linux environment running lots of applications, and teleporting all that across the world with such minimal disruption. DRAC is the "network middleware" that makes this long-haul migration possible at the network level. Specifically, DRAC puts in place a short-lived deterministic network service, on demand. As well, it preserves the sessions with any remote client."

And then I learned from Scott Koranda at Univa that more networking innovation on Linux may be forthcoming in Grid.

"One interesting development that comes to mind is the recent release of Fedora Core 5, which actually includes different TCP/IP stacks that an administrator can use to dynamically control how the machine is responding and working on a TCP network," said Koranda. "Now, I don't think we'll see those capabilities being exploited quickly in a lot of communities, but I think the Grid community will be one of those where we'll see that functionality exploited sooner rather than later."

And then -- of course -- the other historical anecdote that's worth pointing out in the context is that Linux's networking strength was really proved out by guys like Donald Becker (current CTO of Penguin Computing), a pioneer in Linux clustering.

"In the early days, the challenge was as simple as getting the machines to talk to each other - so my background on the Linux side was contributing to the networking side of the Linux kernel," said Becker. "Getting the machines to communicate meant figuring out a lot of the high-throughput, low-latency communication requirements for clusters."

Posted by Greg Nawrocki on June 8, 2006 08:31 AM

June 06, 2006 | Comments: (0)

Virtualization Introduces Bottom-Up Management Requirements

Andreas Antonpoulos -- principal analyst at Nemertes research -- recently provided me with some great insights about the systems management impact of virtualization, and how a bottom-up approach to management (enter Grid) could be the future.

"One aspect of virtualization that doesn't get enough discussion is that while it can make your datacenter more flexible, there are also new issues of design versus run-time optimization. In the past, what we did with the infrastructure was design-time optimization. We'd build the infrastructure for the specific application. If you wanted ERP, you'd build your three-tier or four-tier system. You'd design every component to deliver to the capacity that you'd need. And then you'd turn it on and expect it to meet the demand.
Now -- as we're doing this flexible datacenter with pools of servers and storage -- that optimization happens at run-time. We're deciding how to allocate the resources to achieve the demands at each moment, because now there's nothing really pre-configured to run your application in that infrastructure. It's all shared.

What that means is that we're putting increased pressure on our management platforms. So now we're trying to do provisioning, monitoring, change management, service activation, load-balancing and workflows all from one monolithic management platform. And the result is that many people have found that their management systems are inadequate.

And Grid says that instead of a top-down approach, use a bottom-up approach -- a middleware solution that does the allocation and the workload management in that virtual pool, and is really designed for very heterogeneous platforms, which may not even have a single owner. Radically breaking from that top-down mold is what Grid does best, because a lot of the principles are very much peer-to-peer and creating ad hoc networks and machines when you need them to solve specific problems or do specific jobs. It doesn't require the kind of command-and-control architectures that most of the major management systems imply or assume. And because you can't really apply those sorts of command-and-control architectures very nicely to the datacenter, maybe a bottom-up approach is better. So I think there's a lot of potential for Grid in helping solve the complexity crisis."

Posted by Greg Nawrocki on June 6, 2006 08:06 AM

June 01, 2006 | Comments: (0)

Digipede Developer Network and the Application Quest

Yesterday Digipede announced the availability of the Digipede Network Developer Edition of their software suite designed to provide an easy path for grid enabling applications.

I can't personally speak to the ease of use of the Digipede products, however it has received some recent accolades especially when compared to another *clears throat* "infrastructure based" solution.

This software is free to developers and comes with the Digipede Workbench and Framework SDK to to grid enable applications, and Digipede Server and two Digipede Agents to build a small grid for testing.

Digipede is an all Microsoft shop and their products are built on Microsoft .NET. While the old Linux geek in me would like to see some availability or cross compatibility with the kind of grids I've been familiar with, I have been running up and down the halls yelling "applications, applications, applications". And, in reality, the majority of the general use applications out there are going to be running on Microsoft platforms.

Ok, no more application talk for a while, I promise. Hopefully some clever developers will take Digipede up on their fine free offer so I don't have to go back on that deal.

Posted by Greg Nawrocki on June 1, 2006 07:15 AM

Technology White Papers

InfoWorld Technology Marketplace

» Technology White Papers Library

Technology White Papers by Topic

Technology White Papers E-mail Alert