Of paramount importance to scaling is partitioning. Partitioning leverages "locality of reference" for both processing and data---if certain servers can be specialized to solve some subset of the bigger problem, then the essential code and data are more likely already to be in memory or close at hand on fast disk. Partitioning techniques include the "vertical" partitioning of functional tasks and the "horizontal" partitioning of data and the associated processing (more below). Partitioning is augmented by other well-honed distributed systems techniques like automated replication, data dependent routing, load balancing, and failover. Overall, these techniques have proven (e.g., Google, Yahoo!, Amazon, etc.) to scale well beyond the reach of more centralized architectures that, say, rely on stateless processing and a single very-large database.
