The Anatomy of a Large-Scale Hypertextual Web Search Engine
Sergey Brin and Lawrence Page
{sergey, page}@cs.stanford.edu
Computer Science Department, Stanford University, Stanford, CA 94305
Abstract

In this paper, we present Google, a prototype of a large-scale search engine which makes heavy use of the structure present in hypertext.

Brin and Page's original paper about Google while grad students at Stanford. Good reference for understanding how spiders/crawlers index, how you can search massive amounts of data efficently, etc.

The Anatomy of a Large-Scale Hypertextual Web Search Engine