cool bash stuff; mkfifo

mkfifo is one of those shell commands provided as part of coreutils that not many people seem to know about.

here's an (semi contrived) example close to something i did the other day to show how awesome it is

say you have a number of largish presorted files; run-00 to run-03; and you want to find the most frequent lines. you could do something like the following...

sort -m run-* | uniq -c | sort -nr | head

however you'll know that from previous posts i just loooove keeping all my data compressed on disk so instead i've got run-00.gz to run-03.gz

without having to uncompress the files to disk i'd have to do something like this...

zcat run*gz | sort | uniq -c | sort -nr | head

but this pains me since it results in completely resorting the stream. i know the input files are sorted so i'd much prefer doing a sort -m than sort

so how can i mix the combo of zcat and a pipe to sort with sort -m wanting the multiple inputs as file descriptors instead of STDIN?

well, mkfifo of course! it's a way of making a file that acts like a pipe ( a named pipe )

ls | sort

is sort-of, roughly, equivalent to

mkfifo bob
ls > bob &
sort < bob
rm bob

( have to background the ls since the write to the named pipe blocks until the read starts )

apart from being a cool way to get pipes working between totally seperate processes on a box this provides a solution for our original problem

mkfifo p0 p1 p2 p3
zcat run-00.gz > p0 &
zcat run-01.gz > p1 &
zcat run-02.gz > p2 &
zcat run-03.gz > p3 &
sort -m p* | uniq -c | sort -nr | head
rm p[0123]

and all four zcat can burn cpu while avoiding the need to resort.

yay!

xargs parallel execution

just recently discovered xargs has a parallelise option!

i have 20 files, sample.01.gz to sample.20.gz, each ~100mb in size that i need to run a script over

one option is

zcat sample*gz | ./script.rb > output

but this will process the files sequentially on a single core.

to get some parallel action going i could generate a temp script that produces

zcat sample.01.gz | ./script.rb > sample.01.out &
zcat sample.02.gz | ./script.rb > sample.02.out &
...
zcat sample.20.gz | ./script.rb > sample.20.out &

and run that but this will have all 20 running at the same time and produce contention

(though with only 20 files this might not be a problem)

instead i can make a temp script, parse.sh

zcat $1 | ./script.rb > $1.out

and run

find samplegz | xargs -n1 -P4 sh parse.sh
cat out > output

what is this xargs command doing?

-n1 passes one arg a time to the run comamnd (instead of the xargs default of passing all args)
-P4 says have at most 4 commands running at the same time

100% on all cores (and only because the disk can keep up)

awesome!

brain of mat kelcey

cool bash stuff; mkfifo

xargs parallel execution