In this program from Oct 2004, Jeff Dean of Google describes some of these challenges, discusses applications Google has developed, and highlights systems they’ve built, including GFS, a large-scale distributed file system, and MapReduce, a library for automatic parallelization and distribution of large-scale computation. He also shares some interesting observations derived from Google’s web data.
Google: A Behind-the-Scenes Look
About The Author
Eugenia Loli
Ex-programmer, ex-editor in chief at OSNews.com, now a visual artist/filmmaker.
Follow me on Twitter @EugeniaLoli
You’ll also find interesting lectures from Amazon employees (and others) on the same site.
Not first post, but functional programming! Clever name for the thing — map and reduce are the standard library functions for doing this in languages like Lisp and ML (except no compiler I know of actually parallizes it). It’s really cool to see someone taking advantage of the theoretical possibilities of functional programming. As parallel computation becomes more of a requirement (as ramping up clock-speed hits the wall), hopefully more of these functional techniques will be applied in real programs. One more reason to like Google
how do i view the videos on a Linux/BSD system? i have access to mplaer which usually works with streaming video fine but this one doesn’t show any network activity?
Works fine over here with Kaffeine-xine and Totem-xine.
How can I download that movie and view it later? (connection speed vary and I can’t see it as usual, VLC – 0.8.1 can’t record it… =/ ) Probably anybody fetch it?
Alexandre, you may try http://www.wmrecorder.com/ (for Windows). Not free, but it will get you there if you get my drift.
You can download it with MPlayer. Choose either one of the Windows Media or Quicktime options. Inside the downloaded mov/asx is a url to the actual movie. The url will start with rtsp or mms. You can watch this movie with MPlayer by giving this url as an option, but you can also let it store it in a file by using the arguments “-dumpstream -dumpfile filename”.
For example: mplayer mms://media-wm.cac.washington.edu/ifs/uw_cse05_google_1300k.asf -dumpstream -dumpfile google.asf
One more thing I forgot to say in the previous post: downloading the stream will take as long as the movie takes to watch. This movie is 55 minutes long, so it will take you 55 minutes to download. MPlayer doesn’t show the progress, so it might seem it isn’t doing anything.
Thanks! MPlaye hint is just fine!
You can get more detailed info regarding the cluster architecture of Google reading this:
http://www.computer.org/micro/mi2003/m2022.pdf
and regarding the MapReduce algorithm check this:
http://people.cs.vt.edu/~gback/MapReduce.pdf
St.