the main approach will be

1. maintain a relation with one record per token
2. fold 1 hours worth of new data at a time into the model
3. check the entries for the latest hour for any trends

the full version is on github. read on for a line by line walkthrough!

the ruby impl used the simplest approach possible for calculating mean and standard deviation; just keep a record of all the values seen so far and recalculate for each new value.

for our pig version we'll take a fixed space approach. rather than keep all the values for each time series it turns out we can get away with storing just 3...

1. num_occurences: the number of values
2. mean: the current mean of all values
3. mean_sqrs: the current mean of the squares of all values

the idea is that the mean_n+1 = ( n * mean_n + new value ) / n+1

and that the standard deviation_n+1 can be calculated from n, the mean_n and the mean of the squares_n as we'll see below

let's say we've already run it 6 times and we're now folding in the 7th chunk of per-hour data

the data up to now shows the following
token 'a' has been seen in all 6 chunks with frequencies [1,2,1,2,1,1]; μ=1.33 ρ=0.51
token 'b' has been seen in 3 chunks with frequencies [1,2,2]; μ=1.66 ρ=0.57
token 'c' has been seen in 3 chunks with frequencies [3,4,2]; μ=3 ρ=1

the first thing is to load the existing version of the model, in this case stored in the file 'data/model/006' it contains everything we need for checking the trending for each token

> model = load 'data/model/006' as (token:chararray, num_occurences:int, mean:float, mean_sqrs:float);
> describe model;
model: {token: chararray, 
        num_occurences: int, 
        mean: float, 
        mean_sqrs: float}
> dump model;
(a,6,1.333333F,2.0F)
(b,3,1.666666F,3.0F)
(c,3,3.0F,9.666666F)

this tells us we've seen the token 'a' in 6 previous chunks, the average time we saw it was 1.3 times per chunk and the mean_sqrs (for the standard deviation calculation) is 2

( as a reminder of how we're using these values to calculate a trending score see part 1 )

next we load the new hour's worth of data, in this case contained in 'data/chunks/006'

> next_chunk = load 'data/chunks/006';
> dump next_chunk;
(a b a a)
(d b d a d)

from the text we want to get the frequency of the tokens and we do this using tokenizer.py which utilises the uber awesome NLTK

> define tokenizer `python tokenizer.py` cache('data/tokenizer.py#tokenizer.py');
> tokens = stream next_chunk through tokenizer as (token:chararray);
> describe tokens;   
tokens: {token: chararray}
> dump tokens;
(a)
(b)
(a)
(a)
(d)
(b)
(d)
(a)
(d)

calculating the frequencies of the tokens is a simple two step process of first grouping by the key...

> tokens_grouped = group tokens by token PARALLEL 1;
> describe tokens_grouped;
tokens_grouped: {group: chararray,
                 tokens: {token: chararray}}
> dump tokens_grouped;
(a,{(a),(a),(a),(a)})
(b,{(b),(b)})
(d,{(d),(d),(d)})

...and then generating the key, frequency pairs

> chunk = foreach tokens_grouped generate group as token, SIZE(tokens) as freq;
> dump chunk;
(a,4L)
(b,2L)
(d,3L)

next we join the model with this latest chunk

> cogrouped = cogroup model by token, chunk by token;
> describe cogrouped;
cogrouped: {group: chararray,
            model: {token: chararray,
                    num_occurences: int,
                    mean: float,
                    mean_sqrs: float},
            chunk: {token: chararray,
                    freq: long}}
> dump cogrouped;
(a,{(a,6,1.333333F,2.0F)},{(a,4L)})
(b,{(b,3,1.666666F,3.0F)},{(b,2L)})
(c,{(c,3,3.0F,9.666666F)},{})
(d,{},{(d,3L)})

and doing this allows us to break the data into three distinct relations...

1. entries where token was just in the model; these continue to the next iteration untouched as there is nothing to update
2. entries where token was just in the chunk; these are being seen for the first time and contribute new model entries
3. entries where token was in both; these need a trending check and will require the chunk being folded into the model

> split cogrouped into
        just_model_grped if IsEmpty(chunk),
        just_chunk_grped if IsEmpty(model),
        in_both_grped    if not IsEmpty(chunk) and not IsEmpty(model);
> dump just_model_grped;
(c,{(c,3,3.0F,9.666666F)},{})
> dump just_chunk_grped;
(d,{},{(d,3L)})
> dump in_both_grped;
(a,{(a,6,1.333333F,2.0F)},{(a,4L)})
(b,{(b,3,1.666666F,3.0F)},{(b,2L)})

each of these can be processed in turn.

firstly entries where the token was only the model (ie not in the chunk) pass to next generation of model untouched

> model_n1__just_model = foreach just_model_grped generate flatten(model);
> dump model_n1__just_model;
(c,3,3.0F,9.666666F)

secondly entries where the token was only in the chunk (ie not in the model) contribute new model entries for the next generation

> just_chunk_entries = foreach just_chunk_grped generate flatten(chunk);
> model_n1__just_chunk = foreach just_chunk_entries generate token, 1, freq, freq*freq;
> dump model_n1__just_chunk;
(d,1,3L,9L)

finally, and the most interestingly, when the token was in both the model and the chunk we need to....

flatten the data out a bit

> describe in_both_grped;
in_both_grped: {group: chararray,
                model: {token: chararray,
                        num_occurences: int,
                        mean: float,
                        mean_sqrs: float},
                chunk: {token: chararray,
                        freq: long}}
> in_both_flat = foreach in_both_grped generate flatten(model), flatten(chunk);
> describe in_both_flat;
in_both_flat: {model::token: chararray,
               model::num_occurences: int,
               model::mean: float,
               model::mean_sqrs: float,
               chunk::token: chararray,
               chunk::freq: long}
> dump in_both_flat;
(a,6,1.333333F,2.0F,a,4L)
(b,3,1.666666F,3.0F,b,2L)

do a trending check (note the comparison of freq of iter:n is done against mean/sd of iter:n-1)

> trending = foreach in_both_flat {
               sd_lhs = num_occurences * mean_sqrs;
               sd_rhs = num_occurences * (mean*mean);
               sd = sqrt( (sd_lhs-sd_rhs) / num_occurences ); 
               fraction_of_sd_over_mean = ( sd==0 ? 0 : (freq-mean)/sd );
               generate model::token as token, fraction_of_sd_over_mean as trending_score; 
 }
> describe trending;
trending: {token: chararray,
           trending_score: double}
> dump trending;
(a,5.656845419750436)
(b,0.7071049267408686)

this result tells us that token 'a' is well over what was expected and is seriously trending
with a frequency of 4 in this hour's chunk it's 5.6 times the standard deviation (ρ=0.51) over it's mean frequency (μ=1.33)

token 'b' isn't really trending
with a frequency of 2 in this hour's chunk it's not even one (0.7) standard deviation (ρ=0.57) over it's mean frequency (μ=1.66)

at this stage we can do whatever we want with the trending scores, perhaps save the top 10 off.

trending_sorted = order trending by trending_score desc PARALLEL 1;
top_trending = limit trending_sorted 10 PARALLEL 1;
store top_trending into 'data/trending/006';

after the trending check we need to fold the chunk into the existing model

> model_n1__folded = foreach in_both_flat {
                       new_total = (mean * num_occurences) + freq;
                       new_total_sqrs = (mean_sqrs * num_occurences) + (freq*freq);
                       num_occurences = num_occurences + 1;
                       mean = new_total / num_occurences;
                       mean_sqrs = new_total_sqrs / num_occurences;
                       generate model::token, num_occurences, mean, mean_sqrs;
};
> dump model_n1__folded;
(a,7,1.7142855F,4.0F)
(b,4,1.7499995F,3.25F)

and finally combine with the previous parts we had broken out before to make the new generation of the model!

> model_n1 = union model_n1__just_model, 
                   model_n1__just_chunk, 
                   model_n1__folded;
> store model_n1 into 'data/model/007';

pig demo from Mat Kelcey on Vimeo.

based on a talk i gave at work recently

browsing the data there are lots of other lat longs in data, not just iPhone: and ÜT: there are also one tagged with Coppó:, Pre:, etc perhaps should just try to take anything that looks like a lat long.

furthermore lets switch to a bigger dataset again, 4.7e6 tweets from Oct 13 07:00 thru Oct 19 17:00,

i've been streaming all my tweets ( as previously discussed ) and been storing them in a directory json_stream

here are the steps...

1. extract locations

use a streaming script to take a tweet in json form and emit the tweet time and location string

export HADOOP_STREAMING_JAR=$HADOOP_HOME/contrib/streaming/hadoop-*-streaming.jar
hadoop jar $HADOOP_STREAMING_JAR \
 -mapper ./extract_locations.rb -reducer /bin/cat \
 -input json_stream -output locations

sample output (4.7e6 tuples) { time, location string }

Wed Oct 14 22:01:41 +0000 2009    iPhone: -23.492420,-46.846916
Wed Oct 14 22:01:41 +0000 2009    Ottawa
Wed Oct 14 22:01:41 +0000 2009    DA HOOD
Wed Oct 14 22:01:42 +0000 2009    Earth

2. pluck lat longs from locations

make another pass and extract possible lat lons from the location strings

hadoop jar $HADOOP_STREAMING_JAR \
 -mapper ./extract_lat_longs_from_locations.rb -reducer /bin/cat \
 -input locations -output lat_lons

sample output (reduces down to 320e3 data points) { time, lat, lon }

Wed Oct 14 22:01:41 +0000 2009    -23.49242    -46.846916
Wed Oct 14 22:05:25 +0000 2009    35.670086    139.740766
Wed Oct 14 22:11:35 +0000 2009    41.37731257    -74.68153942
Wed Oct 14 22:15:18 +0000 2009    51.503212    5.478329

3. bucket data into timeslices and points for a map

we need to project the times into 10min slots; ie 00:05 will be slot 0, 00:12 will be slot 1.

also use to project the lat lons to x and y coords (0->1) using a simple mercator projection

hadoop jar $HADOOP_STREAMING_JAR \
 -mapper ./lat_long_to_merc_and_bucket.rb -reducer /bin/cat \
 -cmdenv BUCKET_SIZE=0.005 \
 -input lat_lons -output x_y_points

sample output { timeslice, normalised x position, normalised y position }

122     0.48    0.205
122     0.295   0.26
122     0.29    0.26
123     0.265   0.265

as a slight digression before we move onto aggregating per timeslice here's a pic of all 320e3 tweets on a heatmap.

some interesting noise on the greenwich meridian, must be incorrectly identified lat lons during the ./extract_lat_longs_from_locations.rb step.

log10 tweet location (click for a hires version)

4. aggregate (x,y) pairs per timeslice

next we aggreate, per timeslice, the frequency of points each x,y point. we'll do this with a pig script, aggregate_per_timeslice.pig

# aggregating per timeslice
pts = load 'x_y_points/part-00000' as (timeslice:int, x:float, y:float);
pts2 = group pts by (timeslice,x,y);
pts3 = foreach pts2 generate $0, COUNT($1) ;
pts4 = foreach pts3 generate $0.$0, $0.$1, $0.$2, $1 as freq;
pts5 = order pts4 by timeslice;
store pts5 into 'aggregated_freqs';

results in the tuples in 'aggregated_freqs' { timeslice, normalised x position, normalised y position, frequency }

0    0.0    0.32    1
0    0.06    0.325    9
0    0.065    0.33    1
0    0.08    0.17    2
0    0.155    0.225    8

we need to normalise each frequency value for drawing on the map and would have like to have done this in pig also but turns out there isn't a log function in v0.3 of pig (??)

will have to do scaling when generating the images. isn't such a big deal since the dataset is quite small at this stage but was trying to use this whole thing as an excuse to learn pig :(

5. take aggregated_freqs and make 144 heat map images

use a simple script to read through the aggregated_freqs and generate a heap map for each frame

heat_maps.rb aggregated_freqs 0.005 frames

6. convert to animation

next bundle stills into an animation and upload to youtube

mencoder mencoder "mf://frames/*" -mf fps=25 -o rtw_tweet_v3.avi -ovc x264 -x264encopts bitrate=750

7. conclusions

1. didn't really end up using hadoop's power that much; streaming jobs that use just cat as a reducer as just a parallel way of doing 1:1 string mapping
2. aggregation was really easy in pig but lack of Log function is annoying; could have written a UDF, and there probably already is one but i couldn't find it
3. this visualisation came out pretty lame; funny to see how the really swish visualisations rely far more on pretty colours and smooth lines than the data itself. there are a bundle of things i could do with this one but it's time to move on to something else.

we'll run hadoop in non distributed standalone mode. in this mode everything runs in a single jvm so it's nice and simple to dev against.

step 1: extract the locations strings from the json stream

in v1 it was

bzcat sample.bz2 | ./extract_locations.pl > locations

using the the awesome hadoop streaming interface it's not too different. this interface allows you to specify any app as the mapper or reducer. the main difference is that it works on directories not just files.

for the mapper we'll use exactly the same script as before; extract_locations.pl and since there is no reduce component of this job so we use an "identity" script, ie cat, as the reduce phase.

mkdir json_stream
bzcat sample.bz2 | gzip - > json_stream/input.gz
# hadoop supports gzip out of the bound but not bzip2 :(
export HADOOP_STREAMING_JAR=$HADOOP_HOME/contrib/streaming/hadoop-*-streaming.jar
hadoop jar $HADOOP_STREAMING_JAR \
  -mapper ./extract_locations.pl -reducer /bin/cat \
  -input json_stream -output locations

this gives us the locations in a single file locations/part-0000

step 2: extract iphone and ut lat longs strings

the second step is another text munging problem where we extract just the lat longs for the iPhone and UT tagged locations

ie for strings of the form

iPhone: 21.320328,-157.877579
\u00dcT: 41.727877,-91.626323

we want to extract

21.320328 -157.877579
41.727877 -91.626323

since this is just text manipulation we'll use streaming again

in v1 it was

cat locations | ./extract_lat_longs_from_locations.rb iphone > locations.iphone
cat locations | ./extract_lat_longs_from_locations.rb ut > locations.ut

for hadoop streaming it's

hadoop jar $HADOOP_STREAMING_JAR \
  -mapper './extract_lat_longs_from_locations.rb iphone' -reducer /bin/cat \
  -input locations -output locations.iphone
hadoop jar $HADOOP_STREAMING_JAR \
  -mapper './extract_lat_longs_from_locations.rb ut' -reducer /bin/cat \
  -input locations -output locations.ut

step 3: convert from lat long to mercator coordinates and aggregate into buckets for the heat map

in v1 it was

cat locations.{ut,iphone} | ./lat_long_to_merc.rb | ./bucket.rb | sort | uniq -c

this converts the three tuples { lat, long }

35.670086 139.740766
-23.492420 -46.846916
35.657570 139.744858

into two tuples { frequency, left-offset, top-offset }

1 0.36 0.45
2 0.88 0.28

the first two parts, converting to mercator (lat_long_to_merc.rb) and the bucketing (bucket.rb), i'll combine into one script.

hadoop jar $HADOOP_STREAMING_JAR \
  -mapper ./lat_long_to_merc_and_bucket.rb -reducer /bin/cat \
  -input locations.iphone -input locations.ut -output x_y_points

but the use of sort and uniq to aggregate the data is represented by the shuffle and reduce stages of hadoop.

we could use the aggregate functionality of the streaming interface but i'm trying to learn more pig so we'll use that instead. pig is a scripting language that translates a pig latin query language into map reduce jobs. my main motivation for using it has been that it's great at doing joins, something i've found to be a big pain to represent in plain map reduce jobs.

( note we didn't do the conversion to mercator and bucketing in pig, the arithmetic operations provided are a bit lacking. )

enter a pig shell running in standalone (ie non hadoop distributed) mode

bash> pig -x local

load the points

grunt> pts = load 'x_y_points/part-00000' as (x:float, y:float);
grunt> describe pts;
pts: {x: float,y: float}
grunt> dump pts
(0.06F,0.32F)
(0.15F,0.27F)
(0.16F,0.27F)
...

group them together

grunt> buckets = group pts by (x,y);
grunt> describe buckets;
buckets: {group: (x: float,y: float),pts: {x: float,y: float}}
grunt> dump buckets;
((0.06F,0.32F),{(0.06F,0.32F)})
((0.15F,0.27F),{(0.15F,0.27F)})
((0.16F,0.27F),{(0.16F,0.27F),(0.16F,0.27F),(0.16F,0.27F),(0.16F,0.27F)})
...

from the groups emit the size of each bucket, this corresponds to the frequency

grunt> freq = foreach buckets { generate group, SIZE(pts) as size; }
grunt> describe freq;
freq: {group: (x: float,y: float),size: long}
grunt> dump freq
((0.06F,0.32F),1L)
((0.15F,0.27F),1L)
((0.16F,0.27F),4L)
...

and based on the sizes we can evaluate the min and max frequencies which we'll use in the colour coding of the heat map

grunt> freqs = group freq all;
grunt> describe freqs;
freqs: {group: chararray,freq: {group: (x: float,y: float),size: long}}
grunt> dump freqs;
(all,{((0.06F,0.32F),1L),((0.15F,0.27F),1L), ... })
grunt> store freq into 'freqs';</pre>
<pre>grunt> min_max = foreach freqs { generate MAX(freq.size) as max, MIN(freq.size) as min; };
grunt> describe min_max;
min_max: {max: long,min: long}
grunt> dump min_max;
(7L,1L)
grunt> store min_max into 'min_max';</pre>
<pre>bash> cat freqs
(0.06,0.32)   1
(0.15,0.27)   1
(0.16,0.27)   4

these call all be run as one command

bash> pig -x local -f freqs.pig

we just need our final conversion to a javascript snippet to jam into a map page

bash> cat freqs | ./as_draw_square.rb 1 7

win!

to make things a little different lets use a bigger sample of 475e3 tweets from oct 13 07:00 to 20:00. this results in 10e3 iphone locations (7e3 unique) and 22e3 ut locations (15e3 unique)

lat longs are bucketed into only 478 pixels for map

here's one plot with the raw numbers; highest freq is 9e3 in jakarta

raw frequencies

scaling down by log 10 gives a smoother map

log10 frequencies

and here is a comparison of iphone vs ut. without knowing what ut is i can see it's not big in northern europe or japan but it's popular in indonesia.

iphones

ut

next steps, animating based on the hour of the day

so we need a graph to work on. my first thoughts were using one of the wikipedia linkage dumps but it feels a bit sterile. instead it's a good excuse to do a little crawl of the following graph of twitter.

this will also be a chance to try to document a project via a blog. skorks' incessant blog rambling has convinced me to give it a go.