Grid Meter » No more babysitting large file transfers for Grid pros ...

November 14, 2005 | Comments: (0)

No more babysitting large file transfers for Grid pros ...

TAGS: None

We've all felt the pain of sitting in front of computer screen, watching a large file slowly eek its way over the public Internet to its destination. Watching the progess of the 'percent complete' is as agonizing as it was to watch the clock during our most despised class in school. And most of us have similarly experienced the agony of having a download or upload 'drop off' near the finish line ... and had to re-start the process from scratch.

Many of Grid computing's early adopters are busy science researchers that simply don't have the time (do any of us?) to babysit enormous file transfers to make sure they go through ok. And for a particle physicist sending or receiving petabytes of data, for example -- there certainly isn't time to restart that transfer from scratch after a failure.

Hence the excitement / interest in the "Reliable File Transfer" service project out of the Globus Alliance. RFT is a Web Services Resource Framework-compliant service that provides 'job scheduler'- like functionality for data movement. Researchers can simply provide a list of data sources and destination URLs, and then RFT writes the job description into a database and then moves the files on the user's behalf.

And what if there is a drop off ...?

According to Ravi Madduri, the software developer at Argonne National Laboratory who's been driving the project:

"Say you're transferring a million files. And you are transferring the data from, say, Argonne to ISI, at the University of California. And somewhere during the transfer, some router in between the University of Chicago and Argonne and University of California goes down. The user does not have to do anything. RFT tries to transfer the file and it gets a transient failure, saying that I'm not able to reach this host because network is down somewhere. And RFT will wait and retry after some time. And the number of retries is a configurable option. You can tell RFT to retry forever, or at pre-determined intervals."

With RFT the overhead of maintaining and monitoring file transfers becomes an automated and reliable process.

Posted by Greg Nawrocki on November 14, 2005 07:48 AM

RATE THIS ARTICLE: