Anyone work with really big data sets?
 

I used to, but in COBOL.

I have in my previous life (before I got out of the game). Think VPN logs (intensive mode recording) for several thousand users over a two year span....data mining for one user to defend against a multi-million $ class action suit. Unfortunately, I couldn't use mainframe "big iron", with sufficient processor/memory, but instead using SQL on servers....talk about the wrong tools for the job at hand (but it's all I had to work with at the time) :(.

Yeah:

HPCwire: Climate Science Triggers Torrent of Big Data Challenges

Soooo....do they use Unix or Wintel servers...LOL. I didn't read the whole thing, so might have missed it, but years ago, this type of modeling required Cray supercomputers....what platform are the using now...Macs :)?

Hey Mike, I got curious and found this on Wiki....do you actually work with the Cray? If so, I'm green with envy :)

"As of May 2011[update], the largest computer system Cray has delivered is the XT5 system at National Center for Computational Sciences at Oak Ridge National Laboratories.[9] This system, with over 224,000 processing cores, is dubbed "Jaguar" and was the fastest computer in the world as measured by the LINPACK benchmark[10] at the speed of 1.75 petaflops[11] until being surpassed by the Tianhe-1A in October 2010. It is the fastest system available for open science and was the first system to exceed a sustained performance of 1 petaflops on a 64-bit scientific application; it has since been upgraded to a peak performance of more than two petaflops"