This is a guest post by Russell Jurney, a technologist and serial entrepreneur. His new startup, Cloud Stenography, will launch later this year. The article is an extension of a simple question on Twitter asking the importance of Map Reduce. Some subjects take much more than 140 characters.

The Technical Situation in Brief

The advent of the personal computer and the Visicalc spreadsheet were the foundation for a revolution in computing, business and life whereby normal people could carry out sophisticated accounting, analysis and forecasting to inform their decisions to arrive at more positive outcomes. As Moore’s law has progressed and processors have become faster, and computers inter-networked, large volumes of highly granular data have been collected. Analysis of terabyte datasets on the same level as a spreadsheet has been limited by the disparity of acceleration between processor speed and computer I/O (input/output) operations. Intel has produced ever faster processor clock speeds without accompanying disk, RAM or bus speeds. Put simply: We have cheap and numerous computing resources and abundant data, but bringing those resources to bear on that data to generate real value from it has proven exceedingly difficult.

I did they guest post on Lance Weatherby's blog to explain mapreduce to the people.

The hacker news thread is: http://news.ycombinator.com/item?id=554327

Map Reduce for the People: Force of Good: a blog by Lance Weatherby

got a one line email from Zach that said, "this sounds like a joke but it is true. Kayne West wants me to lip sync his new video. can you fly to nc to shoot it? " There was very little discussion in advance other than that. Hiring the cloggers was Zach's idea, and Inman was able to track them down. For all the ridiculousness you see here, I have to say we took the song very seriously. We asked ourselves, "What if two farmers from North Carolina set out to make their own Kanye West music video...and succeeded?" Special thanks to singer/actor Will Oldham (Bonnie Prince Billy) for his inspired turn as the inbred cousin. This was my first real use of slow motion lip syncing, which is a common trope in music videos, but reliably neat. Special thanks to the rowdy group of 12 year olds hanging out in front of the library in downtown Sparta who started yelling "Whacha filming NERD!" while I was all alone with the production gear.

I can't tell you how much I relate to this. I NEED a blue 60s Ford tractor and more land.

Kanye West "Can't Tell Me Nothing" video (Zach Galifianakis version)

My brother has a theme song. When he walks onto stage at Game Developers Conference to give a talk, this thing plays:

http://jurneydownloads.s3.amazonaws.com/gdc/Chris%20Jurney_GDC.mp3

I keep listening and laughing. So funny. First song about C++ and unit melee behavior I've ever heard.

Chris Jurney_GDC.mp3 (audio/mpeg Object)

Michael Osinski:

Here's one thing that's definitely true: The software proved to be more sophisticated than the people who used it, and that has caused the whole world a lot of problems.

I was wondering why I was making more than anyone in my family, maybe as much as all my siblings combined. Hey, I had higher SAT scores. I could do all the arithmetic in my head. I was very good at programming a computer. And that computer, with my software, touched billions of dollars of the firm's money. Every week. That justified it. When you're close to the money, you get the first cut. Oyster farmers eat lots of oysters, don't they?

From January:

When I started in the business in risk management a veteran trader drew me a picture of the money river to tell me how everyone got paid.

He drew the river and then in a prime spot, a dam. That's where management was. Then you had sales and trading rank and file down the river a bit, but on the bank dipping their pans in the river. Middle office was behind the river bank dipping in the occasional spill over.

My Manhattan Project

William K. Black suspects that it was more than greed and incompetence that brought down the U.S. financial sector and plunged the economy in recession — it was fraud. And he would know. When it comes to financial shenanigans, William K. Black, the former senior regulator who cracked down on banks during the savings and loan crisis of the 1980s, has seen pretty much everything.

Now an Associate Professor of Economics and Law at the University of Missouri, William K. Black tells Bill Moyers on the JOURNAL that the tool at the very center of mortgage collapse, creating triple-A rated bonds out of "liars' loans" — loans issued without verifying income, assets or employment — was a fraud, and the banks knew it.
----
Watch online...

This is the best analysis of the situation that I've seen.

William K. Black: CSI Bailout

Sleep may be a way to sweep out the brain and get it ready for a new day of building connections between neurons, according to two new studies of fruit flies. The studies support the controversial theory that sleep weakens or entirely dissolves some synapses, the connections between brain cells. “We assume that if this is happening, it is a major function, if not the most important function, of sleep” [Science News], says Chiara Cirelli, a coauthor of the first study, published in Science.

Sleep May Prepare You for Tomorrow by Dissolving Today’s Neural Connections | 80beats | Discover Magazine

Hadoop feat. Lzo - save disk space and speed up your programs
The whole point of Hadoop is to process very large datasets. This implies that you will be using a lot of disk space, all those big files replicated a couple of times add up. Let's look at how we can compress text files, saving disk space without losing performance.

braindump: Hadoop feat. Lzo - save disk space and speed up your programs

Xobni's Outlook add-in saves you time finding email, conversations, contact info & attachments.null

Xobni's Outlook Add in - Learn more

Enron Email Dataset
This dataset was collected and prepared by the CALO Project (A Cognitive Assistant that Learns and Organizes). It contains data from about 150 users, mostly senior management of Enron, organized into folders. The corpus contains a total of about 0.5M messages. This data was originally made public, and posted to the web, by the Federal Energy Regulatory Commission during its investigation.

The email dataset was later purchased by Leslie Kaelbling at MIT, and turned out to have a number of integrity problems. A number of folks at SRI, notably Melinda Gervasio, worked hard to correct these problems, and it is thanks to them (not me) that the dataset is available. The dataset here does not include attachments, and some messages have been deleted "as part of a redaction effort due to requests from affected employees". Invalid email addresses were converted to something of the form user@enron.com whenever possible (i.e., recipient is specified in some parse-able format like "Doe, John" or "Mary K. Smith") and to no_address@enron.com when no recipient was specified.

I get a number of questions about this corpus each week, which I am unable to answer, mostly because they deal with preparation issues and such that I just don't know about. If you ask me a question and I don't answer, please don't feel slighted.

I am distributing this dataset as a resource for researchers who are interested in improving current email tools, or understanding how email is currently used. This data is valuable; to my knowledge it is the only substantial collection of "real" email that is public. The reason other datasets are not public is because of privacy concerns. In using this dataset, please be sensitive to the privacy of the people involved (and remember that many of these people were certainly not involved in any of the actions which precipitated the investigation.)

Enron Email Dataset

AIGFP only had 377 employees. Those 400-odd folks received almost $3.5 billion in compensation in the last seven years, a very large part of that money coming from the sale of credit default protection. Doing the math, that averages out to over $9 million of compensation per person.

Ask yourself this question: If your company made that much money, and the boss of the unit made almost $280 million in just a few years, exactly how likely is it that you wouldn't know where that money was coming from?

AIG Exec Whines About Public Anger, and Now We're Supposed to Pity Him? Yeah, Right | Corporate Accountability and WorkPlace | AlterNet