Google is developing a program to help academics around the world exchange huge amounts of data. The firm's open source team is working on ways to physically transfer huge data sets up to 120 terabytes in size. "We have started collecting these data sets and shipping them out to other scientists who want them," said Google's Chris DiBona. Google sends scientists a hard drive system and then copies it before passing it on to other researchers.
Post a Comment
Member since:
2005-07-06
Fans: 6
Member since:
2005-11-18
Fans: 0
Though maybe someday we'll get something faster someday.
Member since:2005-08-17
Fans: 3
Member since:
2005-07-06
Fans: 3
Member since:2006-08-21
Fans: 0
Unrelated problems.
ZFS, being a filesystem, is unrelated to data transfer. I'm sure it could be useful to the project in otherways, though: ext3 filesystems, for example, have a size limit in the neighborhood of 8-16 TB. You could probably use some kind of logical volume manager to concatinate a bunch of filesystems together, but why do that if ZFS can manage such datasets withought breaking a metaphorical sweat?
Of course, the article says nothing about the actual technology Google is using for these "hard drive systems," and I can't recall off the top of my head what the state of ZFS on Linux is (is it working now through FUSE?).
As for the problem at hand--transfers of enormous datasets--it's really just a Google implementation of the old proverb "never underestimate the bandwidth of a station wagon full of backup tapes speeding down the highway."
Member since:2005-07-08
Fans: 0
Member since:2006-01-16
Fans: 6
Member since:
2005-07-10
Fans: 3
Member since:
2005-08-09
Fans: 0
Member since:2005-07-06
Fans: 0
"sneakernet for the win."
Both in terms of bandwidth and cost.
http://www.codinghorror.com/blog/archives/000783.html
I think the economic value of a network is based on its latency, not its bandwidth. It's just much more difficult to measure the former.
Member since:
2005-07-06
Fans: 6