after having to google this stuff three times in the last few months i'm writing it down here so i can just cut and paste next time...

data with arbitrary date time stamp

> d = read.delim('data.tsv',header=F,as.is=T,col.names=c('dts_str','freq'))
> # YEAR MONTH DAY HOUR
> head(d,3)
        dts_str  freq
1 2012_01_01_00 18393
2 2012_01_01_01 20536
3 2012_01_01_02 91840
> tail(d,3)
          dts_str   freq
732 2012_01_31_21 103107
733 2012_01_31_22 108921
734 2012_01_31_23  78629
> summary(d$freq)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  10590   63620   82680   86770  105700  169900 

parse arbitrary strange format to a datetime

> d$dts = as.POSIXct(d$dts_str, format="%Y_%m_%d_%H")

> head(d,3)
        dts_str  freq                 dts
1 2012_01_01_00 18393 2012-01-01 00:00:00
2 2012_01_01_01 20536 2012-01-01 01:00:00
3 2012_01_01_02 91840 2012-01-01 02:00:00

> ggplot(d, aes(dts, freq)) + geom_point()

plots by day of week; summary

> d$dow = as.factor(format(d$dts, format="%a"))  # day of week
> head(d,3)
        dts_str  freq                 dts dow
1 2012_01_01_00 18393 2012-01-01 00:00:00 Sun
2 2012_01_01_01 20536 2012-01-01 01:00:00 Sun
3 2012_01_01_02 91840 2012-01-01 02:00:00 Sun
> ggplot(d,aes(dow,freq)) 
 + geom_boxplot()
 + geom_smooth(aes(group=1)) 
 + scale_x_discrete(limits=c('Mon','Tue','Wed','Thu','Fri','Sat','Sun')) # provide explicit factor ordering
 + xlab('day of week') + ylab('freq') + opts(title='freq by day of week')

plots by day of week; totals

> by_dow = ddply(d, "dow", summarize, freq=sum(freq))
> ggplot(by_dow,aes(dow,freq)) + geom_bar() + 
 scale_x_discrete(limits=c('Mon','Tue','Wed','Thu','Fri','Sat','Sun')) + 
 xlab('day of week') + ylab('freq') + opts(title='total freq by day of week')  # p3.png

plots by hour of day; summary

> d$hr = format(d$dts, format="%H")
> head(d,3)
        dts_str  freq                 dts dow hr
1 2012_01_01_00 18393 2012-01-01 00:00:00 Sun 00
2 2012_01_01_01 20536 2012-01-01 01:00:00 Sun 01
3 2012_01_01_02 91840 2012-01-01 02:00:00 Sun 02
> ggplot(d,aes(hr,freq)) + geom_boxplot() + geom_smooth(aes(group=1)) + 
 xlab('hr of day') + ylab('freq') + opts(title='freq by hr of day')

plots by hour of day; totals

> by_hr = ddply(d, "hr", summarize, freq=sum(freq))
> ggplot(by_hr,aes(hr,freq)) + geom_bar() + 
 xlab('hr of day') + ylab('freq') + opts(title='total freq by hr of day')

Common crawl is a publically available 30TB web crawl taken between September 2009 and September 2010. As a small project I decided to extract and tokenised the visible text of the web pages in this dataset. All the code to do this is on github.

1. Getting the data

The first thing was to get the data into a hadoop cluster. It's made up of 300,000 100mb gzipped arc files stored in S3. I wrote a dead simple distributed copy to do this.

Only a few things of note about this job...

1. The data in S3 is marked as requester pays which, even though it's a no-op if you're accessing the data from EC2, needs the "x-amz-request-payer" header to be set.
2. Pulling from S3 to EC2 is network bound so I ran using the MultithreadedMapRunner to ensure I could get as much throughput as possible.
3. The code includes lots of retry logic but also sets mapred.max.map.failures.percent=100 to allow tasks to fail without killing the entire job (Eg there was one s3 object which had bad ACLs that couldn't be read, no amount of retries would have helped)

2. Filtering text/html

The next step was to filter out everything that didn't have a mime type of 'text/html'. This is pretty straightforward since the arc file format specifies the mime type in a header. I used the ArcInputFormat from Apache Nutch to actually generate the hadoop map input records.

Across the 3,000,000,000 objects in the crawl there ended up being 2,000 distinct mime types, 700 of each occuring only once, with about 90% of them being nonsense.

The top five mime types were ...

Even though there's probably interesting content in the non text/html object types it seemed that just handling text/html would get me the biggest bang for my buck.

Initially I had some problems with encoding. Though http response headers include an encoding field that is meant to indicate what encoding the payload is I found it to be wrong about 30% of the time :( I just ignored what the header said and used the CharsetDetector provided in Apache Tika. CharsetDetector inspects a chunk of bytes, uses heuristics to guess the encoding, decodes and reencodes as UTF-8.

3. Extracting visible text

Next was to extract the visible text from this raw html. After playing with a few libraries I ended up going with boilerpipe. In particular I ended up using the KeepEverythingWithMinKWordsExtractor extractor. Boilerpipe, roughly, returns a single line per block element of the html.

4. Filtering for english content

I then used LanguageIdentifier, again a part of Tika, to filter out everything but english text. It didn't seem to have any false positives but looking at the top 5 languages something seems amiss...

I never got around to sampling some of the Lithuanian ones to see what was actually going on but I'm a bit suspicious. I might have actually lost a bit of content in this step...f

5. Tokenising

The final step was to tokenise the text. I used the stanford parser, in particular I modified their example DocumentPreprocessor to make this simplified SentenceTokeniser

This tokeniser was wrapped in a TokeniseSentences hadoop job that did some additional sanity checking like ignoring one/two word sentences etc.

Results

The final output was 92,000,000,000 sentences (3TB gzipped). Next will be to finish porting my near duplicate sketching algorithm to hadoop to run it across this data.

continuing on with my series of mutual information experiments how might we extend the technique to identity sequences longer than just two terms?

one novel way is to identify the bigrams of interest, replace them with a single token and simply repeat the entire process. (thanks ted for the idea)

example

so say we had the 6 term sentence i went to new york city

it has 5 bigrams; ('i went', 'went to', 'to new', 'new york', 'york city')

running the mutual information algorithm over this might identify new york as a bigram of interest.

we can swap the two terms with a single token (new_york) giving us a new sentence with 5 terms; i went to '(new_york)' city

this new sentence has 4 bigrams ('i went', 'went to', 'to (new_york)', '(new_york) city')

another run of mutual information might now identify the pair (new_york) city so we replace it with the token ((new_york)_city) and just keep repeating.

data

lets run this over a small sample of 300,000 sentences taken from visible text of the freebase wiki dump after it's been tokenised by the stanford parser

(to speed things a little i calculate mutual information and replace the top 10 bigrams in the text before recalculating)

example starting sentences include...

A solid and dependable performer Taylor held the record having played in games for the Phillies at second base t...
A surface may also exhibit both specular and diffuse reflection as is the case for example of glossy paint as us...
A variety of names have since been given to the Wandering Jew including Matathias Buttadeus Paul Marrane and Isa...
A.D.A.M. has control over Eggman 's computer and therefore every robot he owns he can also spread to other compu...
Absolute magnitude magazine cover Though this image is subject to copyright its use is covered by the U.S. fair ...

results

what phrases do we find?

after the first iteration we get the bigrams we've seen before...

Socorro LINEAR
expr    expr
United  States
Los     Angeles
median  income

but after the second iteration we get a mix of single term bigrams and immediately start seeing some new composite bigrams; in this case the trigram 'per square mile'

(expr_expr)     (expr_expr)
capita  income
(t_t)   t
per     (square_mile)
Las     Vegas

unfortunately there's lots of noise too. 'expr expr expr expr' comes from an single sentence, the term 'expr' repeated 450 times, that must have been poorly parsed originally. the 't t t' case is something similar.

by the 16th iteration we get our first 4gram phrase ' U.S. fair use laws'

had     been
U.S.    ((fair_use)_laws)
Rotten  Tomatoes
science fiction
(New_York)      City

and by the 70th iteration we get our first 5gram phrase 'United Nations Security Council Resolution'. jujitsu fans out there will be pleased to see some grappling coming in too!

alas more rubbish as well with the align styling tags leaking in.

(((United_Nations)_Security)_Council)   Resolution
Submission      (rear_(naked_choke))
Asian   (Pacific_Islander)
(UD_(align_left))       ((align_left)_((align_center)_(Win_(align_left))))
lieutenant      colonel

it's only two passes later that we get a big continuation of this one with 'United Nations Security Council Resolution adopted unanimously'

(((((United_Nations)_Security)_Council)_Resolution)_adopted)    unanimously
(United_States) ((align_left)_((align_center)_(Win_(align_left))))
Flying  Corps
Saddam  Hussein
TKO     punches

i was a bit suspicous of this one but grabbing the original text we can see how it makes for an interesting construct in the text...

United Nations Security Council Resolution adopted unanimously on August after recalling Resolution the Council ... 
United Nations Security Council Resolution adopted unanimously on March after recalling all previous resolutions...
United Nations Security Council Resolution adopted unanimously on February after noting that the Council had bee...
United Nations Security Council Resolution adopted unanimously on December after reaffirming all resolutions on ...
United Nations Security Council Resolution adopted unanimously on May after a complaint by Senegal against Portu...
United Nations Security Council Resolution adopted unanimously on June after recalling resolutions and the Counc...
United Nations Security Council Resolution adopted unanimously on July after noting the recent entry into force ...
United Nations Security Council Resolution adopted unanimously on May after recalling all resolutions on the sit...
United Nations Security Council Resolution adopted unanimously on January after recalling all previous resolutio...
United Nations Security Council Resolution adopted unanimously on June after hearing representations from Botswa...
United Nations Security Council Resolution adopted unanimously on May after reaffirming Resolution and all subse...
United Nations Security Council Resolution adopted unanimously on August after reaffirming previous resolutions ...
United Nations Security Council Resolution adopted unanimously on December after reaffirming all resolutions on ...
United Nations Security Council Resolution was adopted unanimously on October after recalling resolutions and on...
United Nations Security Council resolution adopted unanimously on March after reaffirming resolutions and on the...
United Nations Security Council Resolution adopted unanimously on June after recalling all previous resolutions ...
United Nations Security Council Resolution adopted unanimously on January after reaffirming Resolution on the si...
United Nations Security Council Resolution adopted unanimously on February after reaffirming resolutions and in ...

interesting. i wonder has this come from a template perhaps? maybe just cut n paste? one author with fixed style?

even by the end of my run, 950 iterations, (aka last night) there continue to be valid short phrases being picked up

(County_Kansas) (United_States)
County  Clare
Sunday  night
Rift    Valley
Charlton        Heston

how has the corpus changed?

during the processing we've been replacing these tokens in the original text. so how does it look by this time? well, not a whole lot has changed actually.

the following 3 random examples show how little the text differs (should have left it running much longer!!)

(He_played) for a (short_time) with (Duke_Ellington) for (which_he) is (best_remembered)

His (debut_single) Mi God Mi King topped the Jamaican (singles_chart) and a string of hits
followed including Heel And Toe Monkey And Ape (Ghost_Rider) and Crucifixion although his
best-remembered song is Mini Bus which lamented the demise (of_the) jolly bus and which 
(was_awarded) the title Song Of The Year in (from_the) Jamaica (Broadcasting_Corporation)

However this number is certainly an improvement (from_the) cars it averaged yearly ((during_the)_1980s)

what are the longest phrases identified?

the top three are noise alas...

it's not until the 77th we get something that isn't (arguably) just a repeated pattern or noisy parsing

which were identified as phrases due to the large frequency of occurances of variations of the following...

fascinating stuff!!

random todo thoughts

• use patterns found in this experiment to clean up noise and rerun
• work out a way to fold the composition into the scoring
• work on larger dataset
• approach to dealing with duplicates? don't want to just uniq since they represent something

in my last post we went through mutual information as a way of finding collocations.

the astute reader may have noticed that for the list of top bigrams i only showed ones that had a frequency above 5,000.

why this cutoff? well it turns out that one of the criticisms of this definition of mutual information is that it gives whacky results for low support cases.

if we purely just sort by the mutual information score we find that the top 250,000 all have the same score and correpond to bigrams that occur only once in the corpus (and whose terms only appear in that bigram). some examples include "Bruail Brueil", "LW1X LW4X" and "UG-211 GB-HMF" and they are, as often as not it seems, artifacts of parsing quirks.

so how did i decide the minimum support of 5,000? it's just a round number near the 99th percentile of the frequency of frequencies. purely a magic number, not good!

likelihood ratios

another approach that doesn't suffer this problem with low frequency bigrams is to use likelihood ratios. one such test is the g-test very well described in this blog post by ted dunning

as a concrete implementation i'll just use the LogLikelihood code provided in mahout. (yay for being able to use an arbitrary static java function as a pig udf!)

to use this method we need 4 values, k11, k12, k21 and k22, all conveniently calculatable from the counts we gathered for mutual info

calculating this value for the 1,300,000,000 bigrams of wikipedia (code here) we get these top 10 bigrams...

this surprised me a bit since these bigrams are not at all what i expected.... especially when you compare the results against just ranking by raw bigram frequency (which is obviously much easier to calculate)

do i have a bug?

at first i thought i must have a bug but manually redoing the top one (of,the) gives the same answer

k11 = f(of,the) = 12,290,443

k12 = f(of) - f(of,the) = 41,478,115 - 12,290,443 = 29,187,672

k21 = f(the) - f(of,the) = 74,807,672 - 12,290,443 = 62,517,229

k22 = total number bigrams - ( f(of) + f(the) - f(of, the) ) = 1,110,473,107 - ( 41,478,115 + 74,807,672 - 12,290,443 ) = 1,006,477,763

and org.apache.mahout.math.stats.LogLikelihood.logLikelihoodRatio(12290443, 29187672, 62517229, 1006477763) = 2.1369480467885613E7

hmmm.. i wonder what i'm missing? to be continued i guess!

weighting by frequency

coming back to mutual information a variation is to simply weight by the bigram frequency.

so instead of sorting on mutual info we can sort on log(bigram freq) * mutual info. note: we can't weight using the bigram frequency directly because the values are on such different scales as the mutual info. rathering just normalising we can reduce the bigram frequency using a logarithm which feels "fair" since these frequencies follow a power law anyways.

ordering by this new metric gives the following results which seem ok. the main thing is i didn't have to specify an arbitrary cut off frequency!

at least good enough to provide data for the next experiment...

collocations are combinations of terms that occur together more frequently than you'd expect by chance.

they can include

• proper noun phrases like 'Darth Vader'
• stock/colloquial phrases like 'flora and fauna' or 'old as the hills'
• common adjectives/noun pairs (notice how 'strong coffee' sounds ok but 'powerful coffee' doesn't?)

let's go through a couple of techniques for finding collocations taken from the exceptional nlp text "foundations of statistical natural language processing" by manning and schutze.

mutual information

the first technique we'll try is mututal information, it's a way of scoring terms based on how often they appear together vs how often they appear separately.

the intuition is that if two (or three) terms appear together a lot, but hardly ever appear without each other, they probably can be treated as a phrase.

a common definition for bigram mutual information is

\( MutualInformation(t1,t2) = log _{2} \frac{P(t1,t2)}{P(t1).P(t2)} \)

and a definition for trigram mutual information i found in the paper A Corpus-based Approach to Automatic Compound Extraction is

\( MutualInformation(t1,t2,t3) = log _{2} \frac{P(t1,t2,t3)}{P(t1).P(t2).P(t3)+P(t1).P(t2,t3)+P(t1,t2).P(t3)} \)

given a corpus we can use simple maximum likelihoods estimates to calculate these probabilities ...

\( P(t1,t2,t3) = \frac{freq(t1,t2,t3)}{\#trigrams} \) and \( P(t1,t2) = \frac{freq(t1,t2)}{\#bigrams} \) and \( P(t1) = \frac{freq(t1)}{\#unigrams} \)

so! we need a corpus! lets ...

1. grab a freebase wikipedia dump
2. pass it through the stanford nlp parser to extract all the sentences
3. build the frequency tables for our ngrams

( see my project on github for the gory details to reproduce )

working with the 2011-10-15 freebase dump we start with 5,700,000 articles. from this the stanford parser extracts 55,000,000 sentences.

from these sentences we can extract some ngrams ...

the top 5 of each being...

(note: -RRB- is the tokenised right parenthese)

and, as always, the devil's in the detail when it comes to tokenisation... you always have to make lots of decisions; if we're after word pairs/triples should we just remove single characters such as '-' or '|' ? for this experiment i decided to leave them in as they act as a convenient seperator.

overall the freebase data is clean enough for some hacking. had to remove some stray html markup left in from the original wikimedia parse (so the stanford parser wouldn't implode) but other than that we can get away with ignoring anomalies such as the trigram '| - |' (hoorah for statistical methods!)

bigram mutual information

calculating the mutual information for all the bigrams with a frequency over 5,000 gives the following top ranked ones

rank             bigram  freq   m_info
1          Burkina Faso  5417 17.88616
2       Rotten Tomatoes  5695 17.50873
3          Kuala Lumpur  6441 17.47578
4              Tel Aviv  9106 16.90873
5           Baton Rouge  5587 16.85029
6        Figure Skating  5518 16.44119
7             Lok Sabha  7429 16.43407
8            Notre Dame 13516 16.11460
9          Buenos Aires 20595 16.05346
10    gastropod mollusk 19335 15.92581
11           Costa Rica 11014 15.85664
12         Barack Obama  9742 15.84432
13           vice versa  5205 15.66973
14              hip hop 15727 15.63575
15        Uttar Pradesh  7833 15.63525
16   main-belt asteroid 10551 15.62005
17 Theological Seminary  6131 15.61613
18         Saudi Arabia 14887 15.59454
19                sq mi  8492 15.58054
20            São Paulo 13832 15.53181

these are pretty much all proper nouns and, though they are all great finds, they're not really the adjective/noun phrases i was particularly interested in. i guess it's not too surprising since we've done nothing in terms of POS tagging yet.

see here for the top 1,000

a plot of term frequency vs mutual information score shows an expected huge density of low frequency / low mutual information bigrams. the low frequency / high mutual info ones (in the top left) are the ones in the table above and the high frequency / low mutual info ones (in the bottom right) correspond to boring language constructs such as "of the", "to the" or ", and".

trigram mutual information

what about trigrams? here are the top 20 with a support of over 1,000

rank                   trigram freq   m_info
1                Abdu ` l-Bahá 1011 19.06866
2    Dravida Munnetra Kazhagam 1043 18.98674
3              Ab urbe condita 1059 18.58179
4                Dar es Salaam 1130 18.09764
5              Kitts and Nevis 1095 18.02320
6             Procter & Gamble 1255 17.96789
7          Antigua and Barbuda 1290 17.90375
8          agnostic or atheist 1068 17.84620
9                Vasco da Gama 1401 17.77709
10                Ku Klux Klan 1944 17.77443
11              Ways and Means 1070 17.51264
12             Croix de Guerre 1196 17.46765
13        Jehovah 's Witnesses 2235 17.46177
14                  SV = Saves 1980 17.24957
15               Venue | Crowd 1518 17.24024
16             summa cum laude 1363 17.22880
17        Teenage Mutant Ninja 1003 17.17236
18             Osama bin Laden 1734 17.16104
19             magna cum laude 1566 17.13729
20         Names -LRB- US-ACAN 2813 16.91815

again mostly proper nouns apart from some oddities such as "SV = Saves" (which must be from some type of sports glossary since i also see later "Pts = Points" & "SO = Strikeouts" )

see here for the top 1,000

a plot of frequency vs mutual info is similar to the bigram case.

and the top 10 non capitilised trigrams are curious...

agnostic or atheist
summa cum laude
magna cum laude
unmarried opposite-sex partnerships
flora and fauna
non-institutionalized group quarters
mollusk or micromollusk
italics indicate fastest
air-breathing land snail

air-breathing land snails #ftw !

bigrams at a distance

a variation of the standard bigrams approach is to allow tokens to be treated as bigrams as long as they have no more than 2 tokens between them.

eg 'the cat in the hat' which would usually just have bigrams ['the cat','cat in', 'in the', 'the hat'] instead is defined by the bigrams ['the cat', 'the in', 'the the', 'cat in', 'cat the', 'cat hat', 'in the', 'in hat', 'the hat']

this results in roughly three times the bigrams (3,154,200,111 instead of 1,331,695,519) so it's slightly more processing but it allows tokens to influence each other at a short distance

calculating the mutual information for these bigrams gives a slightly different set

rank                   trigram  freq   m_info
1                    expr expr 20888 17.04807
2                    ifeq ifeq  6507 16.18608
3                 Burkina Faso  5473 16.14546
4              Rotten Tomatoes  5705 15.75572
5                 Kuala Lumpur  6457 15.72382
5                SO Strikeouts  5788 15.56487
6        Masovian east-central  8452 15.41213
7                    Earned SO  5651 15.40456
8                  Wins Losses  7984 15.23901
9                     Tel Aviv  9137 15.15810
10                 Baton Rouge  5599 15.09785
11            Dungeons Dragons  5509 14.84334
12             Trinidad Tobago  6241 14.77053
13              Figure Skating  5528 14.68826
14                   Lok Sabha  7435 14.67970
15     background-color E9E9E9  8490 14.65283
16              Haleakala NEAT  5328 14.51430
17             Kitt Spacewatch 17854 14.43547

so more noise from the original freebase parse eg (expr, expr) or (background-color, E9E9E9)

interesting to see it picks up what would otherwise be a trigram with a middle 'and' eg (Dungeons, Dragons) and (Trinidad, Tobago)

summary

so we've found lots of proper nouns! these can be very useful if you're doing feature extraction for a classifier that doesn't like dependent features; a tokenisation of ['Barack Obama', 'went', 'to', 'Kuala Lumpur'] if often better than ['Barack', 'Obama', 'went', 'to', 'Kuala', 'Lumpur']

coming up next, the mean/sd distance method...

as part of my diy twitter client project i've been using the twitter sample streams as a source of unlabelled data for some mutual information analysis. these streams are a great source of random tweets but include a lot of non english content. extracting the english tweets would be pretty straight forward if the ['user']['lang'] field of a tweet was 100% representative of the tweet's language but a lot of the times it's not; can we use these values at least as a starting point?

one approach to seeing how consistent the relationship between user_lang and the tweet language is to

1. train a classifier for predicting the tweet's language assuming the user_lang field is correct
2. have the classifier reclassify the same tweets and see which ones stand out as being misclassified

yes, yes, i realise that testing against the same data you've trained against is a big no no but i'm curious...

method

let's start with 100,000 tweets taken from the sample stream. we'll use vowpal wabbit for the classifier and extract features as follows...

1. lower case the tweet text.
2. remove hashtags, user mentions and urls.
3. split the text into character unigrams, bigrams and trigrams.

we treat tweets marked as user_lang=en as the +ve case for the classifier (class 1) and all other tweets as the -ve case (class 0).

the standard output for predictions from vowpal is a value from 0.0 (not english) to 1.0 (english) but we'll use the raw prediction values instead; the magnitude of these in some way describes the model's confidence in the decision.

results

when we reclassify the tweets the model does pretty well (not surprisingly given we're testing against the same data we trained against)

some examples include..

tweet text      watching a episode of law & order this sad awww
marked english? 1.0 ( yes )
raw prediction  0.998317 ( model agrees it's english )
tweet text      こけむしは『高杉晋助、沖田総悟、永倉新八、神威、白石蔵ノ介』に誘われています。誰を選ぶ？
marked english? 0.0 ( marked as ja )
raw prediction  -1.06215 ( model agrees, definitely not english )

it's not getting 100% (what classifier ever can?) and in part it's since the labelling is "incorrect" at times.

we can use perf to check the accuracy of the model (though it's not really checking "accuracy", more like checking "agreement") and doing this we can see the classifier is correct roughlty 82% of the time.

> accuracy
 [1] 0.82496 0.81260 0.82690 0.82654 0.82454 0.82354 0.82078 0.82023 0.82120 0.81990
> summary(accuracy)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
 0.8126  0.8204  0.8224  0.8221  0.8249  0.8269

(see evaluate.sh for the code to reproduce this)

analysis

so it generally does well but the most interesting cases are when the model doesn't agree with the label.

tweet text      [천국이 RT이벤트]2011 대한민국 소비자신뢰 대표브랜드 대상수상! 알바천국이 여러분의 사랑에힘입어
marked english? 1.0 ( hmmm, not sure this is in english :/ )
raw prediction  -0.52598 ( ie model thinks it's not english )
"disagreement"  1.52598

this is great! the model has correctly identified this instance is mislabelled. however sometimes the model disagrees and is wrong...

tweet text      поняла …что она совсем не нужна ему.
marked english? 0.0 ( fair enough, looks russian to me.. )
prediction:     5.528163 ( model strongly thinks it's english )
"disagreement"  5.528163

a bit of poking through the tweets shows that there are enough russian tweets marked as english for it to be learnt as english...

"correcting" the labels

we can score each tweet based on how much the model disagrees (mean square error of "disagreement" across the multiple runs) and we see, at least for the top 200, that the model was right the vast majority of the time (ie the language of the tweet isn't the user_lang).

what we can do then is trust the model and change the user_lang as required for the top, say, 100 and reiterate.

if we do this overall iteration 10 times we see a gradual improvement in the model.

comparing the first run (r01) to the last (r10) the mean has risen a little bit from 0.8245 to 0.8298 and a t-test thinks this change is significant (p-value = 0.002097 < 0.05); though it's not really that huge an improvement

but an even simpler solution

the iterative solution was novel but it turns out there's a much better solution; make a first pass on the data and if you see one of the common non english characters и, の, ل or น just mark the tweet as non english.

if we do this we get an immediate improvement

> summary(updated)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.8144  0.8308  0.8339  0.8326  0.8359  0.8382 

and you don't need a t-test to see this change is significant :)

tl;dr

1. a supervised classifier can be used in an iterative sense to do unsupervised work
2. but never forget a simple solution can often be the best!

(update: like all interesting things it turns out someone else had already done this :D)

questions

a recent xkcd posed the idea...

wikipedia trivia: if you take any article, click on the first link in the article text not in parentheses or italics, and then repeat, you will eventually end up at Philosophy.

this raises a number of questions

1. Q: though i wouldn't be surprised if it's true for most articles it can't be true for all articles. can it?
2. Q: what's the distribution of distances (measured in "number of clicks away") from 'Philosophy'?
3. Q: by this same measure what's the furthest article from 'Philosophy'?
4. Q: are there any other articles that are more common than 'Philosophy'?
5. Q: what are the common paths to 'Philosophy'?

there's only one way to find out!

1. grab a wikipedia dump
2. build the graph of 'article' to 'first link to next article' (not in parentheses or italics)
3. do breadth first search backwards from 'Philosophy' and see what things look like

getting and processing the data

for my first attempt i tried to use the freebase wikipedia dump. my thought was it'd be easier to deal with a preparsed dataset but it didn't turn out.

two big problems....

1. lots of information has been lost in the preparsing (eg. it was sometimes hard to determine if the first links were from the main body of text or from a sidebar )
2. some pages weren't parsed properly at all and were just blank; included ones like Greeks which ended up being pretty important.

instead i went for a raw wikimedia dump, in particular the enwiki-20110722-pages-articles.xml.bz2 version. it's 7gb compressed & 30gb uncompressed.

for preprocessing there were a number of steps

1. split the dataset into pages that represent redirects and the actual articles themselves
2. dereference all the redirects (to avoid redirects that redirect to other redirects)
3. parse all the articles; the crux of this is done with mwlib and article_parser.py; to make a big list of edges of 'from' nodes (the article) and 'to' nodes (the first applicable link on the article page)
4. dereference the edges to make sure all redirects have been followed

some general statements before we go further

1. wikipedia is under heavy edit churn. i've been doing this project in 15-30 minutes chunks for a few weeks and it's amazing how often i'd compare the parsing to live wikipedia and find out a page had already subtely changed. god knows what it looks like currently.
2. i wrote all the code for this in python as i'm trying to move away from ruby to get better data related library support. everything in fact except for the depth first search which i did in java. the full graph as a dict was insanely slow to access, i must be doing something wrong. for the full details see the code on this project. git cloning the project and executing the README as a shell script may [1] do something close to all the steps from start to finish. [1] or it might not

the end result of the parsing is a list of 3.6e6 edges of the form 'article' -> 'first link to next article' (after following redirects).

all the 'article's are unique but there are only 500e3 distinct 'next article's which is already very interesting; it means less than 15% of articles on wikipedia are represented by one of these first links; this graph is very "bushy" (ie lots of leaf nodes).

to calculate the distance from 'Philosophy' for all articles it's a straight forward breadth first search and because this search doesnt cycle back to 'Philosophy' again it ends up building a tree.

the results

with this tree we can start answering some of our original questions ...

Q: though i wouldn't be surprised if it's true for most articles it can't be true for all articles. can it?

seems it's not true for all articles; 3.5e6 articles lead to 'Philosophy' but 100e3 don't.

these 100e3 fall into two types

1) 50e3 of them end up in cycles. this is a remarkably low count given 3.5e6 make it to 'Philosophy'.

the vast majority of the cycles are of length 2; eg Waste management -> Waste collection -> Waste management

( my favorite that i stumbled across is Sand fence -> Snow fence -> Sand fence
the first sentence of Snow fence being "A snow fence is a structure, similar to a sand fence ..."
the first sentence of Sand fence being "A sand fence is a structure similar to a snow fence ..." )

2) the other 50e3 are dead ends; all sorts of examples for this, mainly around pages that were never written or have been deleted.

eg Windsurfing -> Surface water sports -> Discing (which has deleted)

Q: what's the distribution of distances of articles from 'Philosophy'?

the bulk of the articles are between 10 to 30 clicks away...

i've trimmed this graph at 70 clicks away since there's a long tail of one single path that is 1001 articles long.

List of state leaders in 1977 -> List of state leaders in 1976 -> List of state leaders in 1975 -> .... -> List of state leaders in 1001 -> List of state leaders in 1000 -> Fatimid Caliphate -> Arab people -> Panethnicity -> Ethnic group -> Social group -> Social sciences -> List of academic disciplines -> Academia -> Community -> Living -> Life -> Physical body -> Physics -> Natural science -> Science -> Knowledge -> Fact -> Information -> Sequence -> Mathematics -> Quantity -> Property (philosophy) -> Modern philosophy -> Philosophy

seems a bit of a "meta article" outlier we can ignore.

( there's an interesting dip at a distance of 19 too; wonder what's going on there? )

Q: what's the furthest article from 'Philosophy'?

'Violet & Daisy' is the longest chain i found that didn't include "meta" pages with some kind of sequence number in it. it's 36 articles from 'Philosophy'.

Violet & Daisy -> Saoirse Ronan -> BAFTA Award for Best Actress in a Supporting Role -> British Academy Film Awards -> British Academy of Film and Television Arts -> David Lean -> Order of the British Empire -> Chivalric order -> Knight -> Warrior -> Combat -> Violence -> Psychological manipulation -> Social influence -> Conformity -> Unconscious mind -> Germans -> Germanic peoples -> Proto-Indo-Europeans -> Proto-Indo-European language -> Linguistic reconstruction -> Internal reconstruction -> Language -> Human -> Extant taxon -> Biology -> Natural science -> Science -> Knowledge -> Fact -> Information -> Sequence -> Mathematics -> Quantity -> Property (philosophy) -> Modern philosophy -> Philosophy

Q: are there any other articles that are "more common" than 'Philosophy'?

with 95+% of articles clicking through to 'Philosophy' it's not possible for there to be another unconnected graph with an article more represented than 'Philosophy'.

but if we continue to click through past 'Philosophy' we see we're in a short cycle of 12 articles...

Philosophy -> Reason -> Natural science -> Science -> Knowledge -> Fact -> Information -> Sequence -> Mathematics -> Quantity -> Property (philosophy) -> Modern philosophy -> Philosophy

so really any of these are reasonable candidates and are equally good as 'Philosophy' itself for this game.

Q: what are the common paths into 'Philosophy'?

as mentioned the breadth first search builds a tree of articles with 'Philosophy' at it's root.

one metric we can assign to each article in this tree is the number of descendant articles it has.
'Philosophy', as the root, has all articles as descendants so it's number is 3.5e6 and it's rank 1.
the next ranked by number of descendants is 'Modern philosophy' with 3.4e6 descendants; ( ie of the 3.5e6 articles that eventually led to 'Philosophy' only 100e3 of them didn't click through 'Modern Philosophy').

by ranking articles by this metric we can observe the core structure of the tree.

(width of the edge is proportional to the number of descendants)

it turns out that 3e6 articles (85% of the lot) get to 'Philosophy' through 'Science'.

(click for a bigger version)

( though dot's a pretty poor layout engine for this one, i should redo this one )

conclusions

so i managed to answer the main questions i had, but it's a fun dataset so there's lots more to do yet!

todos include

1. a better layout for the top 1000 or so
2. redo with a more recent wiki dump to see what's changed
3. what happened at a depth of 19 articles?

in a random projection we project A (a NxM matrix) to A' (a NxO, O < M) by the transform AP=A' where P is a MxO matrix with random values.
( well not totally random, each column must have unit length (ie entries in each column must add to 1) )

though the results of this reduction are generally not as good as the results from SVD or PCA it has two huge benefits

• can be done without needing to hold P in memory (since it's entries can be generated multiple times using a seeded RNG)
• and more importantly it's really fast

how good is it compared to PCA i wonder? let's have a play around...

consider the 2d dataset of two clear clusters of points around (2,2) and (8,8)

the following shows 5 random projections to 1d compared to a 1d PCA reduction (done using weka)

there was a clear seperation between the 2 classes in 2d and it's retained in 3 out of the 5 projections even though we're using half the space going from 2d to 1d.
( the 2nd random projection actually looks very similiar to PCA )

what about some higher dimensions?

let's generate the same two clusters but in 10d; ie around the points (2,2,2,2,2,2,2,2,2,2) and (8,8,8,8,8,8,8,8,8,8)

though we can't plot this easily there are 2 useful visualisations of the data

a scatterplot, which is pretty uninteresting actually...

or a 2d tour through the 10d space

(check out my screencast on ggobi if you're after a better idea of what this tour represents)

lets compare 5 random projections to PCA when projecting from 10d to 1d
this time not so good... only one projection gets the clusters seperated, and only just.

what about projecting to 2d?

PCA	run1	run2	run3	run4	run5

PCA is by the cleanest with the x-axis representing the clusters and the y-axis representing the in cluster variance.
it clearly shows how the dataset can be projection to 1d, we only needed the first (x-axis) principal component.

the projections aren't too bad, all but 1 of them has the 2 clusters linearly seperable.

so in summary i think it's pretty good if you need to do something super fast. in these experiments i was using a pretty contrived dataset but was trying to be quite aggressive in going from 10d to 1d.

i wonder what, if any, difference there would be with sparse data?

< random evening hackery />

say we are running the bar at a soldout bad religion concert. the bar serves beer, scotch and water and we decide to record orders over the night so that we can know how much to order for tomorrow's gig...

using these numbers we can predict a number of things..

what is the chance the next person will order a beer?
it's a pretty simple probability; 1000 beers / 1500 total drinks = 0.66 or 66%

what is the chance the next person will order a water?
also straightforward; 200 waters / 1500 total drinks = 0.14 or 14%

t-shirts

now say we run the t-shirt stand at the same concert....

instead of only selling 3 items (like at the bar) we sell 20 different types of t-shirts. once again we record orders over the night...

we can ask similar questions again regarding the chance of people buying a particular t-shirt

what's the chance the next person to buy a t-shirt wants the official tour t-shirt?
15 tour t-shirts sold / 88 sold in total = 0.170 or 17.0%

what's the chance the next person to buy a t-shirt wants the a descendants t-shirt?
3 descendants t-shirts sold / 88 sold in total = 0.034 or 3.4%

what's the chance the next person to buy a t-shirt wants the an offspring t-shirt?
0 offspring t-shirts sold / 88 sold in total = 0 or 0%

if you're like me then the last one "feels" wrong. even though we've not seen a purchase of at least 1 t-shirt it seems a bit rough to say there is no chance of someone buying one. this illustrates one of the problems of dealing prior probabilities

any system using a products of probabilities, such as the modeling of "independent" events in naive bayes, suffers badly from these zero probabilities. i've discussed the problems a few times before in previous experiments such as (this one on naive bayes and this one on semi supervised bayes) and the approach i've always used is the simple rule of succession where we avoid the zero problem by adding one to the frequency of each event.

for reference here are the probabilities per t-shirt without adjustment...

R> sales = rep(c(15,10,5,4,3,1,0), c(2,2,2,3,4,4,3))
 [1] 15 15 10 10  5  5  4  4  4  3  3  3  3  1  1  1  1  0  0  0
R> simple_probs = sales / sum(sales)
 [1] 0.17045455 0.17045455 0.11363636 0.11363636 0.05681818 0.05681818
 [7] 0.04545455 0.04545455 0.04545455 0.03409091 0.03409091 0.03409091
[13] 0.03409091 0.01136364 0.01136364 0.01136364 0.01136364 0.00000000
[19] 0.00000000 0.00000000

... and here are the values for the succession rule case

R> sales = rep(c(15,10,5,4,3,1,0), c(2,2,2,3,4,4,3))
 [1] 15 15 10 10  5  5  4  4  4  3  3  3  3  1  1  1  1  0  0  0
R> sales_plus_one = sales + 1
 [1] 16 16 11 11  6  6  5  5  5  4  4  4  4  2  2  2  2  1  1  1
R> smooth_probs = sales_plus_one / sum(sales_plus_one)
 [1] 0.14814815 0.14814815 0.10185185 0.10185185 0.05555556 0.05555556
 [7] 0.04629630 0.04629630 0.04629630 0.03703704 0.03703704 0.03703704
[13] 0.03703704 0.01851852 0.01851852 0.01851852 0.01851852 0.00925926
[19] 0.00925926 0.00925926

no more zeros! yay! but, unfortunately, at the cost of the accuracy of the other values.

it's always worked for me in the past (well at least better than having the zeros) but it's always felt wrong too. but finally the other day i found another approach, that seems a lot more statistically sound.

it's called good-turing estimation and it was developed as part of turing's work at bletchley park (so it must be awesome). a decent implementation is explained in this paper by geoffrey sampson (it's somewhat more complex than adding 1)

it works on using the frequency of frequencies and redistributes the probabilities to include a special allocation that we should allocate over items that have never seen before.

the following table shows the frequencies, the original probability, the probability adjusted using the rule of succession and the probabilities as redistributed using good turing estimation.

and here's a graph of the same thing.

some observations...

• the rule as succession is just smoothing really and drags the higher probabilities down in response to pulling the lower probabilities up
• the good turing estimation is closer to the real value of the high frequency cases
• the good turing estimate for the zero case is quite a bit higher than the rule of succession estimate
• and most interesting of all, the good turing estimate for the freq 0 is higher than the estimate for freq 1.

the last point in particular i think is really interesting. the good turing algorithm actually gives a total estimate for the zero probability cases (in this examples it gave 0.045) and it's up to the user to distribute it among the actual examples (in this example there were 3 cases so i just divided 0.045 by 3).

if there had be 4 types of t-shirts that hadn't sold the estimate for each of them would have be 0.011 like the 4 t-shirts that sold once.

if there had only be 1 type of t-shirt that hadn't had any sales we'd have to allocate the entire 0.045 to it. in effect the algorithms says it expects that t-shirt to be more likely to sell that the 4 types of t-shirts that had 3 sales each (the 0.033 probability case).

an interesting result, not sure what intution to take away from it....

now this is all good, but i actually don't run the bar at a bad religion concert (or the t-shirt stand) and i'm actually interested in this in how it applies to text processing, especially in the area of classification.

so my question is "is the extra computation required for the good-turing calculation worth it?"

results coming in part2. work in progress code on github

the resource allocation problem

consider the problem of allocating N resources across M servers (N >> M)

modulo hash

a common approach is the straight forward modulo hash...

if we have 4 servers;

servers = [server0, server1, server2, server3]

1. hashing the resource

   hash(resource) = 54

2. reducing modulo 4

   54 % 4 = 2

3. allocating to that numbered server

   servers[2] = server2

we can visualise how this scheme maps resources to servers by allocating a colour to each server; server0 server1 server2 server3
and, assuming we are hashing to a value between 0 and 99, draw the following chart ...

... where the colour of the n^th column represents which server a resource hashing to n would be allocated to.

this hashing scheme is nice for a couple of reasons

1. it's very simple
2. it allocates resources evenly across the servers (assuming you have a good hashing function)

however it has one big drawback; what happens when you change the number of servers?
say for example that due to extra load we have to add another server; server4

switching from modulo 4 to modulo 5 means that a resource that used to hash to server2 ...

54 % 4 = 2

54 % 5 = 4

in fact if we compare the difference in the hashing we get the following ...

... where the top bar represents the allocation with 4 servers
the bottom bar represents the allocation with 5 servers,
with white areas between representing cases of a resource changing which server is was allocated to.

this is pretty bad in terms of reallocation; a whooping 80% of the resources have changed which server they are assigned to.

divisor hash

how about instead of modulo arithmetic we try divisor instead?

considering 4 servers again we allocate a resource by

1. hashing the resource as before

   hash(resource) = 54

2. reducing divisor 25 (25=100/4; ie hash max / number servers)

   54 / (100/4) = 2

3. allocating to that numbered server

   servers[2] = server2

as before we can visualise how this scheme maps resources to servers by again allocating a colour to each server; server0 server1 server2 server3
and, assuming we are hashing to a value between 0 and 99, draw the following chart ...

again if we get a 5th server server4 we can see how the resources are reallocated ...

this time we only 50% reallocation, instead of 80%, so that's an improvement.
we also continue to spread the resources evenly across the servers which is great.

but of course, we can do better!

consistent hash

in a consistent hash we associate ranges of the hash space to servers by hashing the servers themselves.

starting with 4 servers we can hash them (by name, eg 'server0') into the range 0 to 90107 (a smallish prime) giving ...
server0 => 67981, server1 => 24530, server2 => 71186, server3 => 27735

... which can be converted into the ranges ...
server1 => (0, 24530), server3 => (24531, 27735) server0 => (27736, 67981), server2 => (67982, 71186), server1 => (71186, 90106),

visually represented as ...

allocation of a resource to a server is simply done now by hashing the resource and see which range it falls into.

adding a 5th server is a done by hashing the new server; eg server4 => 74391 and adjusting the ranges.

we can see how this scheme ensures that as many resources as possible retain their original server allocation.

however there's a pretty obvious problem; where as the previous methods divided the hash space evenly this method is way off.

we'd like the ratios to be 0.25 for the 4 server case and 0.20 for the 5 server case; but instead they are
server0 => 0.44, server1 => 0.48, server2 => 0.04, server3 => 0.04 and
server0 => 0.44, server1 => 0.44, server2 => 0.04, server3 => 0.04, server4 => 0.04

luckily there's a pretty simple fix; simply hash each server multiple times!

if we hash each server 5 times, using 5 different hash functions, we get the following allocations

which are this time much closer to being even;
server0 => 0.20, server1 => 0.26, server2 => 0.26, server3 => 0.28 and
server0 => 0.17, server1 => 0.19, server2 => 0.21, server3 => 0.24, server4 => 0.18

and the more times we hash the closer we get to an even allocation.
yay!
we get the best of both worlds; an even allocation and the minimum amount of reallocation as the number of servers change.

there's one final trick that can be done with a consistent hash.
turns out we don't have to give the same number of slots to each server

starting with an even allocation ...

we might decide to get server4 twice the number of slots that the others have ...

this results in an uneven allocation of ...
server0 => 0.16, server1 => 0.13, server2 => 0.17, server3 => 0.20, server4 => 0.34

why would we want to have a non even allocation?
a couple of reasons i could think of are..

1. a server with twice the grunt could get handle twice the load so should get twice the slots
2. it's an interesting way to handle a/b testing; introduce a new server by slowing 'dialing' up it's slots

interesting stuff!

all the code used to generate the images for this page are available on github

26th september 2010