Consistent hashing is a scheme that provides hash table functionality in a way that the addition or removal of one slot does not significantly change the mapping of keys to slots. In contrast, in most traditional hash tables, a change in the number of array slots causes nearly all keys to be remapped. By using consistent hashing, only K/n keys need to be remapped on average, where K is the number of keys, and n is the number of slots.

servers = ['memcache1', 'memcache2', 'memcache3', 'memcache4']
servers[ 'product-1'.hash % servers.size ]
 
servers[ 'product-1'.hash % servers.size ] => "memcache4"
servers[ 'product-2'.hash % servers.size ] => "memcache1"

A challenge I faced recently was "scaling up on demand" (without manual intervention). EC2 was the my platform of choice because I could write a small script to manage all my machines. EC2 being Xen based also allowed me to create custom images of my application servers.

1. Hook into the load balancer's health monitor
2. If the load goes over 70% launch a new ec2 instance
3. If the load goes under 50% remove an instance

Example: