Concurrent BASH: Utilize your Dual-Core or Quad-Core processors

We can use batch command to send a series of tasks into a queue, and let it runs by itself. It monitors the load average and admit more tasks when it is low. However, the load average is not changing fast enough for many purposes. In addition, it lacks synchronization facility which is important when you want to make sure some concurrent operations complete before you proceed. Making it user-unfriendly, we have to write the script into a command stream for the batch command.

Hence, I'm trying to design a system to overcome problems as mentioned. Your BASH program can do like the following:

prun.bash:
#!/bin/bash
cd /tmp/xman
date
for i in *.ogg ; do

spawn oggdec *.ogg

done
synchronize
echo "Done"
date
exit 0

xman@sai xman $ time ./prun.bash
Sun Aug 27 21:44:13 SGT 2006
Done
Sun Aug 27 21:44:26 SGT 2006
real 0m13.279s
user 0m0.020s
sys 0m0.072s

Isnt that's cool? Multiple oggdec which decode ogg files will run in parallel on your dual-core or quad-core machine but with limited max number of threads at a time. The Dual-Core machine completes the decoding in merely 13.3s.

Now let me decode using serial script. The serial script:

xman@sai xman $ cat run
#!/bin/bash
date
for i in *.ogg ; do

oggdec "$i" &> /dev/null

done
echo "Done"
date
exit 0

xman@sai xman $ time ./run
Sun Aug 27 21:38:26 SGT 2006
Done
Sun Aug 27 21:38:46 SGT 2006
real 0m20.625s
user 0m19.273s
sys 0m1.004s

Serial script is unable to utilize two cores simultenously. Hence, it's important to use concurrent facility in order to fully utilize the processors that you are using.

Using BASH to manipulate all files in a directory - Cont

Chris pointed out I can simply do

for i in *; do echo $i ; done

I can recall I struggled on the problem file names broken into parts. I spent much time to solve it. There is nothing much to argue if all other programs except your current script were written correctly. If you forgot to " " your $i, see what will happen below.

xman@sai tmp $ ls -Q
"text1.txt" "text version 2.txt"
xman@sai tmp $ for i in * ; do mv $i $i.txt ; done
mv: target `2.txt.txt' is not a directory
xman@sai tmp $

Now we have two solutions.

Solution I:

for i in *; do mv "$i" "$i.txt" ; done

But do remember to double quote your file names. ;)

Solution II:

export IFS=$'\n'
for i in * ; do $i $i.txt ; done

There is nothing wrong if you use "$i".

Currently, I'm not sure what will break what will not if I set IFS=$'\n'. Certainly, if you are using some broken scripts that forgot to " ", with IFS=$'\n', you are still fine. Then, I'll rather locally use IFS=$'\n'. ;) If IFS=$'\n' does break other things, please tell me!

My first open source programs

Have been writing many toy programs which end up in the recycle bin, and then flush to no place. I hope from now onwards, my programs can have longer livespan. Let me start with some bash scripts, xman utility under GPL:

• xman_rm: Move given files into the folder ".deleted". So that I can recover the files in the future if we need to.
• xman_list_rb: List files deleted using xman_rm.
• xman_clean_rb: Delete all files found by xman_list_rb permanently.
• xman_create_pdfdb: Process given folder recursively for .pdf files. Convert all .pdf files into .txt and store in database folder.
• xman_lowercase: Convert all given file names to lowercase.
• xman_uppercase: Convert all given file names to uppercase.
• xman_chext: Convert file extension to another extension.
• xman_tolower_filter: Convert strings from stdin to lowercase.
• xman_toupper_filter: Convert strings from stdin to uppercase.

Hope these scripts may help you for your work. If not, you may still learn some bash scripts from these. :) You may also suggest useful scripts for routine work.