the resource allocation problem

consider the problem of allocating N resources across M servers (N >> M)

modulo hash

a common approach is the straight forward modulo hash...

if we have 4 servers;

servers = [server0, server1, server2, server3]

1. hashing the resource

   hash(resource) = 54

2. reducing modulo 4

   54 % 4 = 2

3. allocating to that numbered server

   servers[2] = server2

we can visualise how this scheme maps resources to servers by allocating a colour to each server; server0 server1 server2 server3
and, assuming we are hashing to a value between 0 and 99, draw the following chart ...

... where the colour of the n^th column represents which server a resource hashing to n would be allocated to.

this hashing scheme is nice for a couple of reasons

1. it's very simple
2. it allocates resources evenly across the servers (assuming you have a good hashing function)

however it has one big drawback; what happens when you change the number of servers?
say for example that due to extra load we have to add another server; server4

switching from modulo 4 to modulo 5 means that a resource that used to hash to server2 ...

54 % 4 = 2

54 % 5 = 4

in fact if we compare the difference in the hashing we get the following ...

... where the top bar represents the allocation with 4 servers
the bottom bar represents the allocation with 5 servers,
with white areas between representing cases of a resource changing which server is was allocated to.

this is pretty bad in terms of reallocation; a whooping 80% of the resources have changed which server they are assigned to.

divisor hash

how about instead of modulo arithmetic we try divisor instead?

considering 4 servers again we allocate a resource by

1. hashing the resource as before

   hash(resource) = 54

2. reducing divisor 25 (25=100/4; ie hash max / number servers)

   54 / (100/4) = 2

3. allocating to that numbered server

   servers[2] = server2

as before we can visualise how this scheme maps resources to servers by again allocating a colour to each server; server0 server1 server2 server3
and, assuming we are hashing to a value between 0 and 99, draw the following chart ...

again if we get a 5th server server4 we can see how the resources are reallocated ...

this time we only 50% reallocation, instead of 80%, so that's an improvement.
we also continue to spread the resources evenly across the servers which is great.

but of course, we can do better!

consistent hash

in a consistent hash we associate ranges of the hash space to servers by hashing the servers themselves.

starting with 4 servers we can hash them (by name, eg 'server0') into the range 0 to 90107 (a smallish prime) giving ...
server0 => 67981, server1 => 24530, server2 => 71186, server3 => 27735

... which can be converted into the ranges ...
server1 => (0, 24530), server3 => (24531, 27735) server0 => (27736, 67981), server2 => (67982, 71186), server1 => (71186, 90106),

visually represented as ...

allocation of a resource to a server is simply done now by hashing the resource and see which range it falls into.

adding a 5th server is a done by hashing the new server; eg server4 => 74391 and adjusting the ranges.

we can see how this scheme ensures that as many resources as possible retain their original server allocation.

however there's a pretty obvious problem; where as the previous methods divided the hash space evenly this method is way off.

we'd like the ratios to be 0.25 for the 4 server case and 0.20 for the 5 server case; but instead they are
server0 => 0.44, server1 => 0.48, server2 => 0.04, server3 => 0.04 and
server0 => 0.44, server1 => 0.44, server2 => 0.04, server3 => 0.04, server4 => 0.04

luckily there's a pretty simple fix; simply hash each server multiple times!

if we hash each server 5 times, using 5 different hash functions, we get the following allocations

which are this time much closer to being even;
server0 => 0.20, server1 => 0.26, server2 => 0.26, server3 => 0.28 and
server0 => 0.17, server1 => 0.19, server2 => 0.21, server3 => 0.24, server4 => 0.18

and the more times we hash the closer we get to an even allocation.
yay!
we get the best of both worlds; an even allocation and the minimum amount of reallocation as the number of servers change.

there's one final trick that can be done with a consistent hash.
turns out we don't have to give the same number of slots to each server

starting with an even allocation ...

we might decide to get server4 twice the number of slots that the others have ...

this results in an uneven allocation of ...
server0 => 0.16, server1 => 0.13, server2 => 0.17, server3 => 0.20, server4 => 0.34

why would we want to have a non even allocation?
a couple of reasons i could think of are..

1. a server with twice the grunt could get handle twice the load so should get twice the slots
2. it's an interesting way to handle a/b testing; introduce a new server by slowing 'dialing' up it's slots

interesting stuff!

all the code used to generate the images for this page are available on github

26th september 2010

a sensible place to start when looking for trends is to consider single terms but what if though we ended up with three equally trending terms 'happy', 'new' and 'year'? it's pretty obvious that the actual trend is 'happy new year' but what is the best way to express this as a single trend in an algorithmic sense?

one approach i've been playing with is to collect unigrams, bigrams and trigrams (1,2,3 term 'phrases') and consider the cases where the terms overlap. basically if 'happy new year' is trending then, in some sense, we can ignore trends for 'happy new', 'new year', 'happy', 'new' and 'year'. but does this result in to many false positives? would we miss 'happy' as a trend if lots of people were chirpy about the change of year (as they usually are, on new years eve)

rather than outright ignore we could somehow reduce the weighting by removing the double counting.

eg if we had 3 trends; (free beer,11), (free,12) & (beer,25) we can take 11 (from the 2gram) off both 1grams to give (free beer,11), (free,1) & (beer,14) showing that 'beer', outside of the phrase 'free beer', is perhaps a trend in itself (as it should be)

this feels like it might work but would be non trivial (read: fun) to implement

another slightly different problem is around the handling of retweeting. my experiments have shown a huge amount of the 'trends' found are related to retweets, which is fine in itself, but it gives quite strange trends since the retweeted portion of the text is usually quite long.

for example; say lots of people are retweeting something and, as some people do, are adding various bits and pieces at the beginning and end; eg 'RT @bob omg i just found a peanut' or 'omg i just found a peanut; via @bob lucky him!!'

if we're considering bigrams (which i am in my current implementation) we end up with an odd selection of trends such as 'just found', 'a peanut', 'omg i', 'found a', 'i just' and in these cases it'd be great to be able to just stitch them together into the common retweeted element 'omg i just found a peanut'.

we could 'solve' this problem by not just considering 1,2 and 3 grams but considering all possible n-grams for each tweet and employing the technique we spoke of above of reducing the counts. it'd almost be feasible, since tweets are never that long, but feels uber clumsy and i'd hate to see the order statistic of that algorithm ;)

this seems more like a stitching problem of some kind; eg if we have 4 grams 'omg i just found', 'i just found a', 'just found a peanut' perhaps we can identify the non trivial overlap and stitch them together (?)

not sure, there are a number of things to try. was hoping that brain dumping some of this would help me see the light but nothing obvious jumps out :(

these connecting edges define the boundaries of communities in the social network and can be identified by algorithms that measure betweenness.

the girvan-newman algorithm can be used to decompose a graph hierarchically based on successive removal of the edges with the highest betweenness.

the algorithm is basically

1. calculate the betweenness of each edge (using an all shortest paths algorithm)
2. remove the edge(s) with the highest betweenness
3. check for connected components (using tarjan's algorithm)
4. repeat for graph or subgraphs if graph was split

while the crawl has been going on i've been trying some things on the data to decide what to do with it.

i've managed to write a version of pagerank using pig which has been very interesting. (for those who haven't seen it before pig is a query language that sits on top of hadoop's mapreduce). my initial feel for pig is that it's pretty awesome. it was much quicker to write this script than to write the statistically improbable phrases. in fact i'm reinspired to have another crack at the sip stuff using pig. my final result wasn't great for the performance of hadoop and after some great feedback on the hadoop mailing list i've got a number of other things to try including writing my joins in pig.

anyways, here's my pagerank in pig

done once

1
2
3
4
5
6

edges = load 'edges' as (from:chararray, to:chararray);
nodes = group edges by from;
node_contribs = foreach nodes generate group, 1.0 / (double)SIZE(edges) as contrib;
store node_contribs into 'node_contribs';
zero_contribs = foreach nodes generate group, (double)0 as contrib;
store zero_contribs into 'zero_contribs';

done until convergence

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

page_rank = load '$input' as (node:chararray, rank:float);
node_contribs = load 'node_contribs' as (node:chararray, contrib:double);
nodes_page_rank = join node_contribs by node, page_rank by node;
contribs = foreach nodes_page_rank {
  generate node_contribs::node, (double)node_contribs::contrib*(double)page_rank::rank as contrib;
}
edges = load 'edges' as (from:chararray, to:chararray);
joined_divy_groups = join edges by from, contribs by node_contribs::node;
page_rank_contributions = foreach joined_divy_groups generate edges::to, contribs::contrib;
zero_contribs = load 'zero_contribs' as (node:chararray, contrib:double);
page_rank_contributions_with_zero = union page_rank_contributions, zero_contribs;
group_page_ranks = group page_rank_contributions_with_zero by edges::to;
next_page_rank = foreach group_page_ranks {
  generate group, 0.15+(0.85*SUM(page_rank_contributions_with_zero.contribs::contrib));
}
store next_page_rank into '$output';

as for all my projects code is on github

a depth first crawl makes no sense for this one experiment, we're unlikely to get the entire graph and are more interested in following edges "close" to me. instead we'll use a breadth first search.

since any call to twitter is expensive (in time that is, they rate limit their api calls) instead of a plain vanilla breadth first we'll introduce a cost component to elements on the frontier so help decide what to grab next. this is especially important for a graph like twitter where the outdegree of a node is often in the hundreds. it turns the crawl into something that is not strictly depth first but it works out.

to explain the cost component consider the expected connectivity of nodes in the twitter friend graph. most nodes have an outdegree of the order 20-200. occasionally you see much larger (in the 1000's) or much smaller (under 10).

we might, naively perhaps, say that having a large outdegree means the person is a bit less strict with her following criteria and that some of them are not really that important to her. if this is the case we should focus a little more on getting nodes with smaller outdegree.

the formula i've come up with is to not consider the depth but instead add 1 + Log10(1+the outdegree of the previous node). in this way we penalise large outdegress, but not by a huge amount. we always add 1 to counter the cases where there are no edges leaving a node.

so we need a graph to work on. my first thoughts were using one of the wikipedia linkage dumps but it feels a bit sterile. instead it's a good excuse to do a little crawl of the following graph of twitter.

this will also be a chance to try to document a project via a blog. skorks' incessant blog rambling has convinced me to give it a go.

when is brute force good enough? will a random walk get a good enough result faster?

matpalm.com/burn.it

the question has haunted me, until now.

here’s my ruby and erlang implementation with a bit of running amazon ec2 thrown in for good measure….. matpalm.com/median/

grab the code from github

my latest experiment considers this problem in relation to jaccard distances, a resemblance measure similar to jaccard coefficients used in a previous experiment

by using the fastmap algorithm we get points from distances and once you have points you have visualisation!

it works quite well and includes a ruby and c++ version with low level bit operations.

project page is www.matpalm.com/resemblance

code at github.com/matpalm/resemblance