Do Not Crawl in the DUST: Different URLs with Similar Text
Topic: Miscellaneous
4:16 pm EST, Feb 20, 2009
We consider the problem of DUST: Different URLs with Similar Text. Such duplicate URLs are prevalent in web sites, as web server software often uses aliases and redirections, and dynamically generates the same page from various different URL requests. We present a novel algorithm, DustBuster, for uncovering DUST; that is, for discovering rules that transform a given URL to others that are likely to have similar content. DustBuster mines DUST effectively from previous crawl logs or web server logs, without examining page contents. Verifying these rules via sampling requires fetching few actual web pages. Search engines can benefit from information about DUST to increase the effectiveness of crawling, reduce indexing overhead, and improve the quality of popularity statistics such as PageRank.
Dan Kaminsky was in Atlanta yesterday. We and some of the usual suspects met for food and drinks at the Vortex in midtown. Some odd/funny things were said, and To keep track, I kept SMSing myself so I could preserve these for posterity.
Chris: I'm going to create Nemisis-oasis. Its the inverse of Match.com and Facebook. You type in what you like and it finds someone that you would absolutely hate. Dan: Its like a Fuck You Cupid!
Tom: (handing me a girlie drink) Here Billy, this will make hair grow on your vagina.
(I have no context for this next quote. I have no idea why Tom said this) Tom: I can only get off if the sheep is from Luxembourg Billy: I'm totally putting this on Memestreams... how do you spell Luxembourg?
Tom: (About a computer scientist at a recent conference) They're proof that every now and then Appalachia produces something good. So its: Bucked Toothed Redneck, Bucked Toothed Redneck, Bucked Toothed Redneck, Cryptographer, Bucked Toothed Redneck...
You know you've been out drinking with Billy when you wake up with a receipt in your pocket that says:
PolitiFact | The Obameter: Tracking Barack Obama's Campaign Promises
Topic: Miscellaneous
4:33 pm EST, Jan 28, 2009
PolitiFact has compiled about 500 promises that Barack Obama made during the campaign and is tracking their progress on our Obameter. We rate their status as No Action, In the Works or Stalled. Once we find action is completed, we rate them Promise Kept, Compromise or Promise Broken.
Dennis Just is the Best Photo Hunt Player In the World
Topic: Miscellaneous
10:25 pm EST, Jan 14, 2009
"I'm awesome at the game," he says. "The best in the world, to my knowledge."
Merit representative Allison Scarafile says only a small percentage of the company's Photo Hunt machines are connected to a server, but of the machines that are connected, the highest score on Photo Hunt is 1,936,500. Just's high score is 2,904,742.
How does he explain his Photo Hunt prowess? Just says he treats the Photo Hunt images as if they were magic eye patterns. "You know how you relax your eyes to make the magic eye image appear? I basically do that, except I cross my eyes so the images overlap, and that makes the differences obvious," he says. "The pictures will be on top of each other, from my perspective."
I was good friends with Dennis in college. Several times I had been in a bar with Dennis when people would see him play and remark "You're that photo hunt guy!!!". He had the high scores all across town. Finally a bar celebrity gets some well deserved recognition from the press.
