When the Netflix Prize was announced, it received some interest here. Have you been following it since then?

The top 5 most frequently rated titles are:

Miss Congeniality
Independence Day
The Patriot
The Day After Tomorrow
Pirates of the Caribbean

This is not a list one would expect based on either box office performance or critical acclaim. I had read that Miss Congeniality, while nothing special in its theatrical run, was a tremendous hit on DVD. Still, it is surprising to find that it is number one on the list. Some interesting and subtle issues to consider here:

1) what types of films become most popular in the DVD rental market?
2) are people more likely to rate certain types of films compared to others?

As you keep moving through the thread, you find this:

To get a look at the movies that are both universally loved, and universally hated (by different subgroups of people, of course) Let's write a query that amplifies standard deviation and de-amplifies population, pointing out the sources of contention in our dataset.

Yes, indeed. Those are the movies you either loved loved loved or hated hated hated. These are the movies you can argue with your friends about. And good old 'Miss Congeniality' is right up there in the #4 spot.

If you like data-intensive visualizations, I suggest browsing the forums.

Geeking Out on the Netflix Prize Forum

This article is kind of weak, but there are a few news bits in it.

With the arrival of its PlayStation 3 less than a month away, Sony is embarking on a worldwide marketing campaign that will collide directly with an equally ambitious effort by Microsoft. And both will face a challenge from the Nintendo Wii.

Sony’s console will go on sale in the United States on Nov. 17, followed two days later by the Wii. The PlayStation 3’s base price will be $499, compared with $299 for the Xbox 360 and $249 for the Wii. Sony’s console, both the basic version and a $599 premium model, will be able to play high-resolution DVD’s in the BluRay format that Sony is promoting. The Xbox 360 has a comparable ability for another format, HD-DVD, but only as a $200 add-on.

Battleground for Consoles Moves Online

“Geek Logik is a hoot!”
—Will Shortz, Crossword Editor, The New York Times

Here for the geek in all of us are fifty foolproof equations that take the guesswork out of life -- and the funniest twist on an idea since Richard Smith's The Dieter’s Guide to Weight Loss During Sex. Call it the algebra oracle: By plugging in the right variables, GEEK LOGIK answers life’s most persistent questions. It covers Dating and Romance, Career and Finance, and everyday decisions like Should I get a tattoo? Can I still wear tight jeans? Is it time to see a therapist? How many beers should I have at the company picnic?

How does it work? Take a simple issue that comes up once or twice a week: Should I call in sick? Fill in the variables honestly, such as D for doctor’s note (enter 1 for “no,” 10 for “yes,” and 5 for “yes, but it’s a forgery”), R for importance of job (1-10, with 10 being “personally responsible for Earth’s orbit around Sun”), Fj for how much fun you have at work (1-10, with 10 being “personal trainer for underwear models”), N for how much you need the money (1-10, with 10 being “I owe the mob”), then do the math, and voilà -- if the product, Hooky, is greater than 1, enjoy your very own Ferris Bueller’s Day Off.

Includes a pocket calculator so that prospective geeks can immediately solve the equation on the back cover: Should I buy this book?

Geek Logik: 50 Foolproof Equations for Everyday Life

This book presents the concepts needed to deal with self-organizing complex systems from a unifying point of view that uses macroscopic data. The various meanings of the concept "information" are discussed and a general formulation of the maximum information (entropy) principle is used.

With the aid of results from synergetics, adequate objective constraints for a large class of self-organizing systems are formulated and examples are given from physics, life and computer science. The relationship to chaos theory is examined and it is further shown that, based on possibly scarce and noisy data, unbiased guesses about processes of complex systems can be made and the underlying deterministic and random forces determined. This allows for probabilistic predictions of processes, with applications to numerous fields in science, technology, medicine and economics.

Quantum information science and technology is presently one of the most active fields of research at the interface of physics, technology and information sciences and has already established itself as one of the major future technologies for processing and communicating information on any scale.

Information and Self-Organization: A Macroscopic Approach to Complex Systems

Tim Berners-Lee, Kurt Gödel, and Alan Turing are the pivotal pioneers who opened the door to the Information Revolution, beginning with the introduction of the computer in the 1950s and continuing today with the World Wide Web evolving into a resource with intelligent features and capabilities.

Taking the main questions posed by these thinkers—"What is decidable?" by Gödel, "What is machine intelligence?" by Turing, and "What is solvable on the Web?" by Berners-Lee—as jumping-off points, Thinking on the Web offers an incisive guide to just how much "intelligence" can be projected onto the Web.

Presenting Web intelligence from both philosophical and applied perspectives, Thinking on the Web explores the next generation of Web architecture, the Semantic Web, and takes a realistic look at the Artificial Intelligence (AI) capabilities resulting from this new layer of machine processing. It is divided into two sections, the first addressing Web intelligence and the second Web logic and language. The book builds from commonsense, real-world examples to some of the most cutting-edge issues surrounding the next generation of Web capabilities, including:

* Knowledge representation
* Computational complexity
* Semantic Web capabilities and limitations
* Web Ontology Languages (OWL)
* Semantic search

Throughout the book, a series of vignettes highlight important issues underpinning the Information Revolution, thereby mirroring the authors' attention to both the abstract and practical questions posed by a "thinking" Web. A unique guide to the next frontier of computing, Thinking on the Web offers both tech-savvy readers and serious computer science students a stimulating and practical view towards a smarter Web.

Thinking on the Web: Berners-Lee, Gödel and Turing

Graphics are great for exploring data, but how can they be used for looking at the large datasets that are commonplace today?

This book shows how to look at ways of visualizing large datasets, whether large in numbers of cases or large in numbers of variables or large in both. Data visualization is useful for data cleaning, exploring data, identifying trends and clusters, spotting local patterns, evaluating modeling output, and presenting results.

It is essential for exploratory data analysis and data mining. Data analysts, statisticians, computer scientists -- indeed, anyone who has to explore a large dataset of their own -- should benefit from reading this book.

New approaches to graphics are needed to visualize the information in large datasets and most of the innovations described in this book are developments of standard graphics. There are considerable advantages in extending displays which are well-known and well-tried, both in understanding how best to make use of them in your work and in presenting results to others. It should also make the book readily accessible for readers who already have a little experience of drawing statistical graphics. All ideas are illustrated with displays from analyses of real datasets and the authors emphasize the importance of interpreting displays effectively. Graphics should be drawn to convey information and the book includes many insightful examples.

Graphics of Large Datasets: Visualizing a Million

All the data was out there to warn us of this impending attack:

"why didn't we see it?"

This was a frequently asked question in the weeks and months after 9/11. In the wake of the attacks, statisticians moved quickly to become part of the national response to the global war on terror.

This book is an overview of the emerging research program at the intersection of national security and statistical sciences. A wide range of talented researchers address issues:

How do we detect and recognize bioterrorist events?
How do we better understand and explain complex processes so that decision makers can take the best course of action?
How do we pick the terrorist out of the crowd of faces or better match the passport to the traveler?
How do we understand the rules that terrorists are playing by?

This book includes technical treatments of statistical issues that will be of use to quantitative researchers as well as more general examinations of quantitative approaches to counterterrorism that will be accessible to decision makers with stronger policy backgrounds.

Statistical Methods in Counterterrorism: Game Theory, Modeling, Syndromic Surveillance, and Biometric Authentication

From the American entry into World War II until September 1943, U.S. submarines experienced an abnormally high number of torpedo failures. These failures resulted from three defects present in the primary torpedo of the day, the Mark XIV. These defects were a tendency to run deeper than the set depth, the frequent premature detonation of the Mark 6 magnetic influence exploder, and the failure of the contact exploder when hitting a target at the textbook ninety-degree angle.

Ironically, despite using a completely independent design, the Germans experienced the same three defects. The Germans, however, fixed their defects in six months, while it took the Americans twenty-two months.

Much of the delay on the American side resulted from the denial of senior leaders in the operational forces and in the Navy's Bureau of Ordnance (BuOrd) that the torpedo itself was defective. Instead, they blamed crews for poor marksmanship or lack of training.

In the end, however, the submarine force itself overcame the bureaucratic inertia and correctly identified and fixed the three problems on their own, proving once again the industry of the average American [2,3] soldier or sailor.

The publisher continues:

Contrary to the interpretations of most submarine historians, this book concludes that BuOrd did not sit idly by while torpedoes failed on patrol after patrol. BuOrd acknowledged problems from early in the war, but their processes and their tunnel vision prevented them from realizing that the weapon sent to the fleet was grossly defective. One of World War II's forgotten heroes, Admiral Lockwood drove the process for finding and fixing the three major defects. This is first book that deals exclusively with the torpedo problem, building its case out of original research from the archives of the Bureau of Ordnance, the Chief of Naval Operations, Vice Admiral Lockwood's personal correspondence, and records from the British Admiralty at the National Archives of the United Kingdom. These sources are complemented by correspondence and interviews with men who actually participated in the events.

Iron Men and Tin Fish: The Race to Build a Better Torpedo during World War II

About 6,000 people a day — most of them children — die from water-borne diseases.

A Danish textile company as come up with a new invention meant to render dangerous water drinkable.

The invention is called Lifestraw, a plastic tube [that] can be worn around the neck and lasts a year.

Lifestraw isn’t perfect, but it filters out at least 99.99 percent of many parasites and bacteria.

A $3 Water Purifier That Could Save Lives

As we approach a billionth of a cent per byte of storage, and pennies per gigabit per second of bandwidth, what kind of machine labors to be born?

How will we feed it?

How will it be tamed?

And how soon will it, in its inevitable turn, become a dinosaur?

The Information Factories