A online search database for the AOL search database that runs at a decent speed is now online.

AOL Search Database

This comment was posted on TechCrunch by an AOL spokesman.

All –

This was a screw up, and we’re angry and upset about it. It was an innocent enough attempt to reach out to the academic community with new research tools, but it was obviously not appropriately vetted, and if it had been, it would have been stopped in an instant.

Although there was no personally-identifiable data linked to these accounts, we’re absolutely not defending this. It was a mistake, and we apologize. We’ve launched an internal investigation into what happened, and we are taking steps to ensure that this type of thing never happens again.

Here was what was mistakenly released:

* Search data for roughly 658,000 anonymized users over a three month period from March to May.

* There was no personally identifiable data provided by AOL with those records, but search queries themselves can sometimes include such information.

* According to comScore Media Metrix, the AOL search network had 42.7 million unique visitors in May, so the total data set covered roughly 1.5% of May search users.

* Roughly 20 million search records over that period, so the data included roughly 1/3 of one percent of the total searches conducted through the AOL network over that period.

* The searches included as part of this data only included U.S. searches conducted within the AOL client software.

We apologize again for the release.

Andrew Weinstein
AOL Spokesman

For the record, this was the license included with the release:

This collection is distributed for NON-COMMERCIAL RESEARCH USE ONLY.

Any application of this collection for commercial purposes is STRICTLY PROHIBITED.

Techcrunch | AOL: “This was a screw up”

Unbelievable. AOL released a file containing the search engine queries of over 500,000 users during a three month period. It's being mirrored all over.

Here is a screenshot of the download page before it was taken down, complete with a spelling error.. "ananomized"

Update: I've imported the data into an SQL database so I can do some data mining. It's about 3.5G worth of SQL, so the process of building indexes and performing any useful queries is really slow going. Sometime in the next 24 hours, I should be posting up some statistics. I have to think about it some more first... From what I've gathered so far, there is no liability in doing so.

AOL fucked up. This data is in the hands of many, many, many people. That being the case, I want to see how the data frames the issues we all have with this kind of data being available to law enforcement, marketers, and others.. Anyone who has any ideas about what questions we should be asking, reply to this with your thoughts.

Since the hot button issue most directly connected with this is child porn, I've been doing some research focusing on that. The Justice Department wanted Google and other search engines to hand over exactly this information so they could build a profile of what people are searching for when they search for child porn. I've been attempting to do the same thing. Thus far, I've gotten a pretty expansive table of users (over 300) that have been blatantly searching for child porn. I've done a fair amount of work eliminating false positives, such as people searching for information about how to protect their kids, researching court cases, or looking up information about specific offenses. I've tried to limit the list to people blatantly repeatedly searching for illegal pictures of pre-teens and whatnot. I'm working on constructing a list of "what people who search for kiddie porn search for."

I also have some indexes building that will allow me to mine general statistical data on what the top queries are and stuff like that. Since I'm working with a laptop that only has a gig of ram and not too speediest of a hard drive, it's going to take awhile. I expect my machine to be churning for the next few hours.

Update: I don't have powerful enough hardware to mine this. I'm waiting on more resources to become available later tonight.

Link to AOL data release

The air marshals, whose identities are being concealed, told 7NEWS that they're required to submit at least one report a month. If they don't, there's no raise, no bonus, no awards and no special assignments.

"Innocent passengers are being entered into an international intelligence database as suspicious persons, acting in a suspicious manner on an aircraft ... and they did nothing wrong," said one federal air marshal.

27B Stroke 6 blogged this yesterday but didn't link the actual story they were quoting. Its worth clicking through to some of the other stories at the bottom of this story. Basically, the Atlanta Air Marshall Director has quit his job and is raising a public stir about bad policies at the Air Marshal's service.

TheDenverChannel.com - News - Marshals: Innocent People Placed On 'Watch List' To Meet Quota

The FBI has drafted sweeping legislation that would require Internet service providers to create wiretapping hubs for police surveillance and force makers of networking gear to build in backdoors for eavesdropping, CNET News.com has learned.

FBI proposes Internet CALEA

FBI plans new Net-tapping push | Tech News on ZDNet

The story about phone number stations posted to Craigslist has been mentioned here before. This is one of those interesting little mysteries. The hearsay only adds to it..

People have suggested that the messages are pranks, or are some sort of commercial gimmick. But at least one person, who is in the U.S. military, says he sent a copy of one of the messages up the chain of command and was promptly notified that it was classified and he wasn’t cleared to know anything further about it. I don’t know whether this is standard operating procedure for any encrypted message or whether it indicates that there’s something to be found.

Third phone numbers station: 678-248-2352 - Homeland Stupidity

New Scientist has discovered that Pentagon's National Security Agency, which specialises in eavesdropping and code-breaking, is funding research into the mass harvesting of the information that people post about themselves on social networks.

I wonder what their MySpace account is. I wonder if they want to be my friend...

New Scientist Technology - Pentagon sets its sights on social networking websites

It's ironic that spooks so often remind us that we've got nothing to fear from their activities if we've got nothing nasty to hide, while they themselves are rarely comfortable without multiple layers of secrecy, anonymity and plausible deniability. While there was little or nothing at the conference worth keeping secret, the sense of paranoia was constant. The uniformed guard posted to the entrance was there to intimidate, not to protect. The restrictions on civilians attending the law enforcement agency sessions were, I gather, a cheap marketing gesture to justify their $6,500-per-head entrance fee with suggestions of secret information that the average network-savvy geek wouldn't have known.

It poses a tremendous threat to human rights and dignity in countries without adequate legal safeguards, and still invites occasional abuses in countries with them. Its costs are paid by citizens who are deliberately kept in the dark about how much they're paying for it, how effective it is in fighting crime and how susceptible it is to abuse. And that's the way the entire cast of characters involved wants to keep it.

Wired News: Crashing the Wiretapper's Ball

Below, Noteworthy ties together a slew of earlier datapoints that hinted at this program, but I must underline this quotation that particularly pisses me off:

White House spokeswoman Dana M. Perino denied that the administration was misleading when it described the NSA program as narrowly drawn.

"It is narrow," she said. "The president has been very specific and very accurate in all of his comments. He said that the government is not trolling through personal information and that the privacy of Americans is fiercely guarded."

When they say "the privacy of Americans is fiercely guarded" what they mean is that they have a team of lawyers who have fiercely produced arguements that what they are doing is legal. Covering your ass is not the same thing as guarding my privacy, god damnit! There is a time when press interview management and spin control is no longer funny, and this is that time. This nation is not made up of little children. The administration has serious questions to answer and they ought to be answering those questions in a serious way.

Going back to Orin Kerr's legal analysis, I'm troubled by how easily the 4th amendment is dismissed here. If the 4th amendment doesn't prevent wholesale data mining of phone call information then what the hell does it prevent!? Even if we find it reasonable that phone users might expect the phone company to share dialed numbers with the government, but not share call content, an arguement I find questionable to begin with, I think we might still expect that the phone company would only do this in special circumstances, and wouldn't be doing it with every single call.

Noteworthy's post is everything below this line:

As illustrated by Negroponte's remarks last week, administration officials have been punctilious in discussing the NSA program over the past five months, parsing their words with care and limiting comments to the portion of the program that had been confirmed by the president in December.

In doing so, the administration rarely offered any hint that a much broader operation, involving millions of domestic calls, was underway. Even yesterday -- after days of congressional furor and extensive media reports -- administration officials declined to confirm or deny the existence of the telephone-call program, in part because of court challenges that the government is attempting to derail.

I continue to be surprised that no one else has recommended Black Arts, by Thomas Powers, more than a year after its publication and appearance on MemeStreams. For this reason, I will reiterate his closing statement for you:

About the failure everyone now agrees. But what was the problem? And what should be done to make us safe?

It wasn't respect for ... [ Read More (0.1k in body) ]

Negroponte Had Denied Domestic Call Monitoring

Revellers in a British town are to have their fingerprints scanned when they enter pubs and clubs in a scheme aimed at weeding out drunken troublemakers.

The "In Touch" project is the first of its kind in Britain.

Biometric finger-scanning machines have been installed at six venues in Yeovil, southwest England. Clubbers will be asked to have their right index finger scanned and show picture identification to register on the system.

"If somebody is causing trouble in one pub and is removed from the premises, from the time it takes for that person to walk to another venue, the system will have been updated and the door staff at other venues will be aware."

"It will also mean that they do not have to carry ID on them which can often be the source of inconvenience."

Once registered on the system, clubbers are identified by finger scan only.

The scheme is voluntary and the information stored will be subject to data protection laws.

I think it's safe to make this broad statement now. Britain is the canary in the coal mine when it comes to determining how much monitoring and surveillance a citizenry will accept.

British town's pubs scan fingerprints to spot louts