Anyone work with really big data sets?

[id10t]

UC Berkeley AMP Camp | The UC Berkeley Big Data AMP Camp, featuring tutorials on popular open-source software including Spark, Shark, Hive, and Mesos; overviews of the Berkeley Data Analytics System (BDAS); and Machine Learning tutorials.

[widebody911]

I used to, but in COBOL.

[KFC911]

I have in my previous life (before I got out of the game). Think VPN logs (intensive mode recording) for several thousand users over a two year span....data mining for one user to defend against a multi-million $ class action suit. Unfortunately, I couldn't use mainframe "big iron", with sufficient processor/memory, but instead using SQL on servers....talk about the wrong tools for the job at hand (but it's all I had to work with at the time) .

[IROC]

Yeah:

HPCwire: Climate Science Triggers Torrent of Big Data Challenges

Many climate simulations output to a large number of individual files over the entire simulation run. For a single run you can have many files created, which, when taken in aggregate, can exceed several terabytes. Over the past few years, we have seen these dataset sizes increase dramatically.

[KFC911]

Originally Posted by IROC

Yeah:

HPCwire: Climate Science Triggers Torrent of Big Data Challenges

Soooo....do they use Unix or Wintel servers...LOL. I didn't read the whole thing, so might have missed it, but years ago, this type of modeling required Cray supercomputers....what platform are the using now...Macs ?

[KFC911]

Hey Mike, I got curious and found this on Wiki....do you actually work with the Cray? If so, I'm green with envy

"As of May 2011[update], the largest computer system Cray has delivered is the XT5 system at National Center for Computational Sciences at Oak Ridge National Laboratories.[9] This system, with over 224,000 processing cores, is dubbed "Jaguar" and was the fastest computer in the world as measured by the LINPACK benchmark[10] at the speed of 1.75 petaflops[11] until being surpassed by the Tianhe-1A in October 2010. It is the fastest system available for open science and was the first system to exceed a sustained performance of 1 petaflops on a 64-bit scientific application; it has since been upgraded to a peak performance of more than two petaflops"