pig demo from Mat Kelcey on Vimeo.

based on a talk i gave at work recently

browsing the data there are lots of other lat longs in data, not just iPhone: and ÜT: there are also one tagged with Coppó:, Pre:, etc perhaps should just try to take anything that looks like a lat long.

furthermore lets switch to a bigger dataset again, 4.7e6 tweets from Oct 13 07:00 thru Oct 19 17:00,

i've been streaming all my tweets ( as previously discussed ) and been storing them in a directory json_stream

here are the steps...

1. extract locations

use a streaming script to take a tweet in json form and emit the tweet time and location string

export HADOOP_STREAMING_JAR=$HADOOP_HOME/contrib/streaming/hadoop-*-streaming.jar
hadoop jar $HADOOP_STREAMING_JAR \
 -mapper ./extract_locations.rb -reducer /bin/cat \
 -input json_stream -output locations

sample output (4.7e6 tuples) { time, location string }

Wed Oct 14 22:01:41 +0000 2009    iPhone: -23.492420,-46.846916
Wed Oct 14 22:01:41 +0000 2009    Ottawa
Wed Oct 14 22:01:41 +0000 2009    DA HOOD
Wed Oct 14 22:01:42 +0000 2009    Earth

2. pluck lat longs from locations

make another pass and extract possible lat lons from the location strings

hadoop jar $HADOOP_STREAMING_JAR \
 -mapper ./extract_lat_longs_from_locations.rb -reducer /bin/cat \
 -input locations -output lat_lons

sample output (reduces down to 320e3 data points) { time, lat, lon }

Wed Oct 14 22:01:41 +0000 2009    -23.49242    -46.846916
Wed Oct 14 22:05:25 +0000 2009    35.670086    139.740766
Wed Oct 14 22:11:35 +0000 2009    41.37731257    -74.68153942
Wed Oct 14 22:15:18 +0000 2009    51.503212    5.478329

3. bucket data into timeslices and points for a map

we need to project the times into 10min slots; ie 00:05 will be slot 0, 00:12 will be slot 1.

also use to project the lat lons to x and y coords (0->1) using a simple mercator projection

hadoop jar $HADOOP_STREAMING_JAR \
 -mapper ./lat_long_to_merc_and_bucket.rb -reducer /bin/cat \
 -cmdenv BUCKET_SIZE=0.005 \
 -input lat_lons -output x_y_points

sample output { timeslice, normalised x position, normalised y position }

122     0.48    0.205
122     0.295   0.26
122     0.29    0.26
123     0.265   0.265

as a slight digression before we move onto aggregating per timeslice here's a pic of all 320e3 tweets on a heatmap.

some interesting noise on the greenwich meridian, must be incorrectly identified lat lons during the ./extract_lat_longs_from_locations.rb step.

log10 tweet location (click for a hires version)

4. aggregate (x,y) pairs per timeslice

next we aggreate, per timeslice, the frequency of points each x,y point. we'll do this with a pig script, aggregate_per_timeslice.pig

# aggregating per timeslice
pts = load 'x_y_points/part-00000' as (timeslice:int, x:float, y:float);
pts2 = group pts by (timeslice,x,y);
pts3 = foreach pts2 generate $0, COUNT($1) ;
pts4 = foreach pts3 generate $0.$0, $0.$1, $0.$2, $1 as freq;
pts5 = order pts4 by timeslice;
store pts5 into 'aggregated_freqs';

results in the tuples in 'aggregated_freqs' { timeslice, normalised x position, normalised y position, frequency }

0    0.0    0.32    1
0    0.06    0.325    9
0    0.065    0.33    1
0    0.08    0.17    2
0    0.155    0.225    8

we need to normalise each frequency value for drawing on the map and would have like to have done this in pig also but turns out there isn't a log function in v0.3 of pig (??)

will have to do scaling when generating the images. isn't such a big deal since the dataset is quite small at this stage but was trying to use this whole thing as an excuse to learn pig :(

5. take aggregated_freqs and make 144 heat map images

use a simple script to read through the aggregated_freqs and generate a heap map for each frame

heat_maps.rb aggregated_freqs 0.005 frames

6. convert to animation

next bundle stills into an animation and upload to youtube

mencoder mencoder "mf://frames/*" -mf fps=25 -o rtw_tweet_v3.avi -ovc x264 -x264encopts bitrate=750

7. conclusions

1. didn't really end up using hadoop's power that much; streaming jobs that use just cat as a reducer as just a parallel way of doing 1:1 string mapping
2. aggregation was really easy in pig but lack of Log function is annoying; could have written a UDF, and there probably already is one but i couldn't find it
3. this visualisation came out pretty lame; funny to see how the really swish visualisations rely far more on pretty colours and smooth lines than the data itself. there are a bundle of things i could do with this one but it's time to move on to something else.

we'll run hadoop in non distributed standalone mode. in this mode everything runs in a single jvm so it's nice and simple to dev against.

step 1: extract the locations strings from the json stream

in v1 it was

bzcat sample.bz2 | ./extract_locations.pl > locations

using the the awesome hadoop streaming interface it's not too different. this interface allows you to specify any app as the mapper or reducer. the main difference is that it works on directories not just files.

for the mapper we'll use exactly the same script as before; extract_locations.pl and since there is no reduce component of this job so we use an "identity" script, ie cat, as the reduce phase.

mkdir json_stream
bzcat sample.bz2 | gzip - > json_stream/input.gz
# hadoop supports gzip out of the bound but not bzip2 :(
export HADOOP_STREAMING_JAR=$HADOOP_HOME/contrib/streaming/hadoop-*-streaming.jar
hadoop jar $HADOOP_STREAMING_JAR \
  -mapper ./extract_locations.pl -reducer /bin/cat \
  -input json_stream -output locations

this gives us the locations in a single file locations/part-0000

step 2: extract iphone and ut lat longs strings

the second step is another text munging problem where we extract just the lat longs for the iPhone and UT tagged locations

ie for strings of the form

iPhone: 21.320328,-157.877579
\u00dcT: 41.727877,-91.626323

we want to extract

21.320328 -157.877579
41.727877 -91.626323

since this is just text manipulation we'll use streaming again

in v1 it was

cat locations | ./extract_lat_longs_from_locations.rb iphone > locations.iphone
cat locations | ./extract_lat_longs_from_locations.rb ut > locations.ut

for hadoop streaming it's

hadoop jar $HADOOP_STREAMING_JAR \
  -mapper './extract_lat_longs_from_locations.rb iphone' -reducer /bin/cat \
  -input locations -output locations.iphone
hadoop jar $HADOOP_STREAMING_JAR \
  -mapper './extract_lat_longs_from_locations.rb ut' -reducer /bin/cat \
  -input locations -output locations.ut

step 3: convert from lat long to mercator coordinates and aggregate into buckets for the heat map

in v1 it was

cat locations.{ut,iphone} | ./lat_long_to_merc.rb | ./bucket.rb | sort | uniq -c

this converts the three tuples { lat, long }

35.670086 139.740766
-23.492420 -46.846916
35.657570 139.744858

into two tuples { frequency, left-offset, top-offset }

1 0.36 0.45
2 0.88 0.28

the first two parts, converting to mercator (lat_long_to_merc.rb) and the bucketing (bucket.rb), i'll combine into one script.

hadoop jar $HADOOP_STREAMING_JAR \
  -mapper ./lat_long_to_merc_and_bucket.rb -reducer /bin/cat \
  -input locations.iphone -input locations.ut -output x_y_points

but the use of sort and uniq to aggregate the data is represented by the shuffle and reduce stages of hadoop.

we could use the aggregate functionality of the streaming interface but i'm trying to learn more pig so we'll use that instead. pig is a scripting language that translates a pig latin query language into map reduce jobs. my main motivation for using it has been that it's great at doing joins, something i've found to be a big pain to represent in plain map reduce jobs.

( note we didn't do the conversion to mercator and bucketing in pig, the arithmetic operations provided are a bit lacking. )

enter a pig shell running in standalone (ie non hadoop distributed) mode

bash> pig -x local

load the points

grunt> pts = load 'x_y_points/part-00000' as (x:float, y:float);
grunt> describe pts;
pts: {x: float,y: float}
grunt> dump pts
(0.06F,0.32F)
(0.15F,0.27F)
(0.16F,0.27F)
...

group them together

grunt> buckets = group pts by (x,y);
grunt> describe buckets;
buckets: {group: (x: float,y: float),pts: {x: float,y: float}}
grunt> dump buckets;
((0.06F,0.32F),{(0.06F,0.32F)})
((0.15F,0.27F),{(0.15F,0.27F)})
((0.16F,0.27F),{(0.16F,0.27F),(0.16F,0.27F),(0.16F,0.27F),(0.16F,0.27F)})
...

from the groups emit the size of each bucket, this corresponds to the frequency

grunt> freq = foreach buckets { generate group, SIZE(pts) as size; }
grunt> describe freq;
freq: {group: (x: float,y: float),size: long}
grunt> dump freq
((0.06F,0.32F),1L)
((0.15F,0.27F),1L)
((0.16F,0.27F),4L)
...

and based on the sizes we can evaluate the min and max frequencies which we'll use in the colour coding of the heat map

grunt> freqs = group freq all;
grunt> describe freqs;
freqs: {group: chararray,freq: {group: (x: float,y: float),size: long}}
grunt> dump freqs;
(all,{((0.06F,0.32F),1L),((0.15F,0.27F),1L), ... })
grunt> store freq into 'freqs';</pre>
<pre>grunt> min_max = foreach freqs { generate MAX(freq.size) as max, MIN(freq.size) as min; };
grunt> describe min_max;
min_max: {max: long,min: long}
grunt> dump min_max;
(7L,1L)
grunt> store min_max into 'min_max';</pre>
<pre>bash> cat freqs
(0.06,0.32)   1
(0.15,0.27)   1
(0.16,0.27)   4

these call all be run as one command

bash> pig -x local -f freqs.pig

we just need our final conversion to a javascript snippet to jam into a map page

bash> cat freqs | ./as_draw_square.rb 1 7

win!

to make things a little different lets use a bigger sample of 475e3 tweets from oct 13 07:00 to 20:00. this results in 10e3 iphone locations (7e3 unique) and 22e3 ut locations (15e3 unique)

lat longs are bucketed into only 478 pixels for map

here's one plot with the raw numbers; highest freq is 9e3 in jakarta

raw frequencies

scaling down by log 10 gives a smoother map

log10 frequencies

and here is a comparison of iphone vs ut. without knowing what ut is i can see it's not big in northern europe or japan but it's popular in indonesia.

iphones

ut

next steps, animating based on the hour of the day

while the crawl has been going on i've been trying some things on the data to decide what to do with it.

i've managed to write a version of pagerank using pig which has been very interesting. (for those who haven't seen it before pig is a query language that sits on top of hadoop's mapreduce). my initial feel for pig is that it's pretty awesome. it was much quicker to write this script than to write the statistically improbable phrases. in fact i'm reinspired to have another crack at the sip stuff using pig. my final result wasn't great for the performance of hadoop and after some great feedback on the hadoop mailing list i've got a number of other things to try including writing my joins in pig.

anyways, here's my pagerank in pig

done once

1
2
3
4
5
6

edges = load 'edges' as (from:chararray, to:chararray);
nodes = group edges by from;
node_contribs = foreach nodes generate group, 1.0 / (double)SIZE(edges) as contrib;
store node_contribs into 'node_contribs';
zero_contribs = foreach nodes generate group, (double)0 as contrib;
store zero_contribs into 'zero_contribs';

done until convergence

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

page_rank = load '$input' as (node:chararray, rank:float);
node_contribs = load 'node_contribs' as (node:chararray, contrib:double);
nodes_page_rank = join node_contribs by node, page_rank by node;
contribs = foreach nodes_page_rank {
  generate node_contribs::node, (double)node_contribs::contrib*(double)page_rank::rank as contrib;
}
edges = load 'edges' as (from:chararray, to:chararray);
joined_divy_groups = join edges by from, contribs by node_contribs::node;
page_rank_contributions = foreach joined_divy_groups generate edges::to, contribs::contrib;
zero_contribs = load 'zero_contribs' as (node:chararray, contrib:double);
page_rank_contributions_with_zero = union page_rank_contributions, zero_contribs;
group_page_ranks = group page_rank_contributions_with_zero by edges::to;
next_page_rank = foreach group_page_ranks {
  generate group, 0.15+(0.85*SUM(page_rank_contributions_with_zero.contribs::contrib));
}
store next_page_rank into '$output';

as for all my projects code is on github

so we need a graph to work on. my first thoughts were using one of the wikipedia linkage dumps but it feels a bit sterile. instead it's a good excuse to do a little crawl of the following graph of twitter.

this will also be a chance to try to document a project via a blog. skorks' incessant blog rambling has convinced me to give it a go.

matpalm.com/sip

not fantastic results but heaps of of feedback from hadoop mailing group

more results coming soon