(update: like all interesting things it turns out someone else had already done this :D)

questions

a recent xkcd posed the idea...

wikipedia trivia: if you take any article, click on the first link in the article text not in parentheses or italics, and then repeat, you will eventually end up at Philosophy.

this raises a number of questions

1. Q: though i wouldn't be surprised if it's true for most articles it can't be true for all articles. can it?
2. Q: what's the distribution of distances (measured in "number of clicks away") from 'Philosophy'?
3. Q: by this same measure what's the furthest article from 'Philosophy'?
4. Q: are there any other articles that are more common than 'Philosophy'?
5. Q: what are the common paths to 'Philosophy'?

there's only one way to find out!

1. grab a wikipedia dump
2. build the graph of 'article' to 'first link to next article' (not in parentheses or italics)
3. do breadth first search backwards from 'Philosophy' and see what things look like

getting and processing the data

for my first attempt i tried to use the freebase wikipedia dump. my thought was it'd be easier to deal with a preparsed dataset but it didn't turn out.

two big problems....

1. lots of information has been lost in the preparsing (eg. it was sometimes hard to determine if the first links were from the main body of text or from a sidebar )
2. some pages weren't parsed properly at all and were just blank; included ones like Greeks which ended up being pretty important.

instead i went for a raw wikimedia dump, in particular the enwiki-20110722-pages-articles.xml.bz2 version. it's 7gb compressed & 30gb uncompressed.

for preprocessing there were a number of steps

1. split the dataset into pages that represent redirects and the actual articles themselves
2. dereference all the redirects (to avoid redirects that redirect to other redirects)
3. parse all the articles; the crux of this is done with mwlib and article_parser.py; to make a big list of edges of 'from' nodes (the article) and 'to' nodes (the first applicable link on the article page)
4. dereference the edges to make sure all redirects have been followed

some general statements before we go further

1. wikipedia is under heavy edit churn. i've been doing this project in 15-30 minutes chunks for a few weeks and it's amazing how often i'd compare the parsing to live wikipedia and find out a page had already subtely changed. god knows what it looks like currently.
2. i wrote all the code for this in python as i'm trying to move away from ruby to get better data related library support. everything in fact except for the depth first search which i did in java. the full graph as a dict was insanely slow to access, i must be doing something wrong. for the full details see the code on this project. git cloning the project and executing the README as a shell script may [1] do something close to all the steps from start to finish. [1] or it might not

the end result of the parsing is a list of 3.6e6 edges of the form 'article' -> 'first link to next article' (after following redirects).

all the 'article's are unique but there are only 500e3 distinct 'next article's which is already very interesting; it means less than 15% of articles on wikipedia are represented by one of these first links; this graph is very "bushy" (ie lots of leaf nodes).

to calculate the distance from 'Philosophy' for all articles it's a straight forward breadth first search and because this search doesnt cycle back to 'Philosophy' again it ends up building a tree.

the results

with this tree we can start answering some of our original questions ...

Q: though i wouldn't be surprised if it's true for most articles it can't be true for all articles. can it?

seems it's not true for all articles; 3.5e6 articles lead to 'Philosophy' but 100e3 don't.

these 100e3 fall into two types

1) 50e3 of them end up in cycles. this is a remarkably low count given 3.5e6 make it to 'Philosophy'.

the vast majority of the cycles are of length 2; eg Waste management -> Waste collection -> Waste management

( my favorite that i stumbled across is Sand fence -> Snow fence -> Sand fence
the first sentence of Snow fence being "A snow fence is a structure, similar to a sand fence ..."
the first sentence of Sand fence being "A sand fence is a structure similar to a snow fence ..." )

2) the other 50e3 are dead ends; all sorts of examples for this, mainly around pages that were never written or have been deleted.

eg Windsurfing -> Surface water sports -> Discing (which has deleted)

Q: what's the distribution of distances of articles from 'Philosophy'?

the bulk of the articles are between 10 to 30 clicks away...

i've trimmed this graph at 70 clicks away since there's a long tail of one single path that is 1001 articles long.

List of state leaders in 1977 -> List of state leaders in 1976 -> List of state leaders in 1975 -> .... -> List of state leaders in 1001 -> List of state leaders in 1000 -> Fatimid Caliphate -> Arab people -> Panethnicity -> Ethnic group -> Social group -> Social sciences -> List of academic disciplines -> Academia -> Community -> Living -> Life -> Physical body -> Physics -> Natural science -> Science -> Knowledge -> Fact -> Information -> Sequence -> Mathematics -> Quantity -> Property (philosophy) -> Modern philosophy -> Philosophy

seems a bit of a "meta article" outlier we can ignore.

( there's an interesting dip at a distance of 19 too; wonder what's going on there? )

Q: what's the furthest article from 'Philosophy'?

'Violet & Daisy' is the longest chain i found that didn't include "meta" pages with some kind of sequence number in it. it's 36 articles from 'Philosophy'.

Violet & Daisy -> Saoirse Ronan -> BAFTA Award for Best Actress in a Supporting Role -> British Academy Film Awards -> British Academy of Film and Television Arts -> David Lean -> Order of the British Empire -> Chivalric order -> Knight -> Warrior -> Combat -> Violence -> Psychological manipulation -> Social influence -> Conformity -> Unconscious mind -> Germans -> Germanic peoples -> Proto-Indo-Europeans -> Proto-Indo-European language -> Linguistic reconstruction -> Internal reconstruction -> Language -> Human -> Extant taxon -> Biology -> Natural science -> Science -> Knowledge -> Fact -> Information -> Sequence -> Mathematics -> Quantity -> Property (philosophy) -> Modern philosophy -> Philosophy

Q: are there any other articles that are "more common" than 'Philosophy'?

with 95+% of articles clicking through to 'Philosophy' it's not possible for there to be another unconnected graph with an article more represented than 'Philosophy'.

but if we continue to click through past 'Philosophy' we see we're in a short cycle of 12 articles...

Philosophy -> Reason -> Natural science -> Science -> Knowledge -> Fact -> Information -> Sequence -> Mathematics -> Quantity -> Property (philosophy) -> Modern philosophy -> Philosophy

so really any of these are reasonable candidates and are equally good as 'Philosophy' itself for this game.

Q: what are the common paths into 'Philosophy'?

as mentioned the breadth first search builds a tree of articles with 'Philosophy' at it's root.

one metric we can assign to each article in this tree is the number of descendant articles it has.
'Philosophy', as the root, has all articles as descendants so it's number is 3.5e6 and it's rank 1.
the next ranked by number of descendants is 'Modern philosophy' with 3.4e6 descendants; ( ie of the 3.5e6 articles that eventually led to 'Philosophy' only 100e3 of them didn't click through 'Modern Philosophy').

by ranking articles by this metric we can observe the core structure of the tree.

(width of the edge is proportional to the number of descendants)

it turns out that 3e6 articles (85% of the lot) get to 'Philosophy' through 'Science'.

(click for a bigger version)

( though dot's a pretty poor layout engine for this one, i should redo this one )

conclusions

so i managed to answer the main questions i had, but it's a fun dataset so there's lots more to do yet!

todos include

1. a better layout for the top 1000 or so
2. redo with a more recent wiki dump to see what's changed
3. what happened at a depth of 19 articles?

first i build a friend graph for 100 people 'around' me (taken from a crawl i did last year). by 'friend' i mean that if alice follows bob then bob also follows alice.

here the graph, some things to note though; it was an unfinished crawl (can a crawl of twitter EVER be finished) and was done october last year so is a bit out of date.

and here is the dendrogram decomposition

some interesting clusterings come out..

right at the bottom we have a small clique (ie everyone following everyone else) of people i've known from when i was in sydney

this small group connects to the group i'm in; tinybuddha down to evanbottcher; which roughly describes the group of people i've met in melbourne.

the order of the single breakaways in the melbourne group is pretty arbitrary. i get quite different ordering if i run the decomposition multiple times due to the random tie breaking involved. i could either run the decomposition multiple times and work out some kind of averaging or choose another more granular way of deciding how to break ties.

the next connector after syndey and melbourne are unified is deanemorrow a coworker when i was at distra. this one sticks out for me as being the biggest flaw in the clustering since it would have made more sense to have him placed near distra at the bottom.

another interesting clique is near me..

it has four thoughtworkers; mark, gill, debs and evan and one sensiser; korny. did korny perhaps work for thoughtworks in a previous life ;)

another interesting note is there exists a path from me to peter norvig (who is too busy for twitter it seems) but only because of the huge connector nodes that exist in twitter. an example in this case is TUAW who follow 30,000+ people and have even more followers. these nodes cause a bit of noise in the system since they are slightly false representations of what a 'friend' means in my mind. not sure how to take these numbers into account...

things to do...

• the biggest oversimplification in this system is how i break ties for deciding which edge to cut out next if multiple exist with the same betweenness. currently it chooses the one that would make the most even sized break (based on smallest standard deviation of the connected components). though this is good for breaking a group into even sizes it's bad since it favours breaking a single element off a large group. this is what has caused the 'laddering' we see in the melbourne group.
• the shortest path algorithm used to calculate edge betweenness is stochastic and if multiple shortest paths exist only one of them is chosen. it'd be better if all were considered with a weighting scheme.
• it might be better to consider vertex betweenness instead of edge betweenness since one person could exist in multiple groups. if i started down this path though i think i'd rather just rewrite the lot using something like  the clique percolation method

all the code is on github

the last chunk of code i wrote which tried to deal with weighted directed graphs was terribly, terribly, broken but it seems that simplifying to undirected graphs is giving me much saner results. yay!

here's an example of my work in progress generated from the new version of the code

consider the graph

and it's corresponding decomposition

the results are reasonable; the initial breaking of clusters [1,2,3,4,5,6] and [7,8,9,10,11,12] is the most obvious but some of the others are not as intuitive

[1,2,5] and [7,8,10] remain as unbreakable cliques though it's arbitrary that 11 was broken off from [7,8,10] instead of 10 (arbitrary but an artifact related to my shortest path calculation for the edge betweenness)

the idea of identifying the edge to remove using edge betweenness works well but it is often the case there are many edges with the same maximal betweeness and you have to choose only one. i think my current implementation of picking one is a bit naive and i'm not sure if i should move to a stochastic / monte carlo style approach or focus more on modularity maximisation

these connecting edges define the boundaries of communities in the social network and can be identified by algorithms that measure betweenness.

the girvan-newman algorithm can be used to decompose a graph hierarchically based on successive removal of the edges with the highest betweenness.

the algorithm is basically

1. calculate the betweenness of each edge (using an all shortest paths algorithm)
2. remove the edge(s) with the highest betweenness
3. check for connected components (using tarjan's algorithm)
4. repeat for graph or subgraphs if graph was split

we seed our frontier with 'a' and bootstrap cost of 0.

fetching the info for 'a' shows 2 outedges to 'b' and 'c', from our cost formula these all have cost 0 + 1 + Log10(2+1) = 1.6

our frontier becomes [ {b,1.6}, {c,1.6} ]

next is 'b' and see it has an outdegree of 3, these nodes, b1 -> b3, all have a cost of 1.6 + 1 + Log10(3+1) = 3.2

our frontier becomes [ {c,1.6}, {b1,3.2}, {b2,3.2}, {b3,3.2} ]

next is 'c' with an outdegree of 15. these 15 nodes, c1 -> c15, have cost 1.6 + 1 + Log10(16) = 3.8

our frontier is now [ {b1,3.2}, {b2,3.2}, {b3,3.2}, {c1,3.8} ... {c15,3.8} ]

we would then continue with the 'b' nodes before the 'c' ones

note that this cost system is more than just an ordering, it allows an element at depth n+1 to be checked an element at depth n, if the cost of the latter is high enough.

a depth first crawl makes no sense for this one experiment, we're unlikely to get the entire graph and are more interested in following edges "close" to me. instead we'll use a breadth first search.

since any call to twitter is expensive (in time that is, they rate limit their api calls) instead of a plain vanilla breadth first we'll introduce a cost component to elements on the frontier so help decide what to grab next. this is especially important for a graph like twitter where the outdegree of a node is often in the hundreds. it turns the crawl into something that is not strictly depth first but it works out.

to explain the cost component consider the expected connectivity of nodes in the twitter friend graph. most nodes have an outdegree of the order 20-200. occasionally you see much larger (in the 1000's) or much smaller (under 10).

we might, naively perhaps, say that having a large outdegree means the person is a bit less strict with her following criteria and that some of them are not really that important to her. if this is the case we should focus a little more on getting nodes with smaller outdegree.

the formula i've come up with is to not consider the depth but instead add 1 + Log10(1+the outdegree of the previous node). in this way we penalise large outdegress, but not by a huge amount. we always add 1 to counter the cases where there are no edges leaving a node.