Xman Blog Inside

BLOG Migration

2006-09-18T01:15:00.000+08:00

I just got my domain name recently at http://myxman.org. That gives more much more control to post casual blog and technical blog. Hence, I shall start migrating everything from Xanga and Blogspot to this new site. I can even provide emails like xman@myxman.org, joleck@myxman.org, ivan@myxman.org Or provides space for photo postings. Currently I have only 40GB of space. I still can put in lots of stuff.

Reversi Benchmark -- Round 2

2006-09-16T01:33:00.000+08:00

Finally we got a Woodcrest machine for benchmark. Thanks to CopperBlue again for contributing the machine.

The Intel Xeon Dual-Core 2.66GHz (Woodcrest) achieves much better 32bit performance than AMD64X2 4200+ (2.2GHz). Running Woodcrest in 64bit achieves even higher performance.

Can you believe Woodcrest is even slower than the aging AMD64 X2? I suggest CopperBlue upgrade his Java JVM to 1.5.x. There is significant performance boost in Java 1.5 over Java 1.4. Otherwise, his latest processor is just as slow as my 1+ year-old processor. Do you care?

Reversi Benchmark: AMD64 X2 vs INTEL CORE DUO vs OTHERS

2006-09-14T20:30:00.000+08:00

This round we manage to get an Intel Core Duo 2.16GHz with Shared 2MB L2, Mac Book Pro, to benchmark. Thanks CopperBlue for contributing his laptop. How does it compare to AMD64 X2 4200+ (2.2GHz) with 2 x 512KB Private L2 which has almost the same clock speed? However, AMD64 X2 is able to run in both 32bit mode and 64bit mode, whereas Core Duo runs only 32bit mode. Does it matter?

Let's see the first benchmark xversi-c-benchmark-v0.1 which utilizes only one core.

Intel Core Duo 2.16GHz has slight advantage over AMD64 X2 4200+ in 32bit mode. Interestingly, AMD64 X2 running in 64bit mode boost up the performance almost by 2x. Itanium II with much lower clock speed is clearly out of the game.

What about the Parallel Reversi? Let's see the next benchmark xversi-java-parallel-benchmark-v0.1.

AMD64 X2 4200+ clearly has an advantage running in 64bit mode. The 64bit JVM definitely helps in boosting the performance. Hence, if you own a 64bit processor, use a 64bit OS, and 64bit components such as Sun Java for x86_64.

Python Core Language Quick Reference

2006-09-11T13:39:00.000+08:00

If you can learn a language by example, this is a place for you to learn the core of python programming. However, some statements are mis-indented. But indentation should not be a problem to you if you are familiar with structured programming.

# Getting started.
>>> print "hello world"
hello world

xman@sai python $ cat hello.py
#!/usr/bin/python
# hello world comment
print "hello world"

>>> execfile("hello.py")
hello world

>>> import sys
>>> sys.exit()
xman@sai python $

>>> while x <= y:
... print x,y
... x=x+1
>>> x = 0 ; y = 1 ; z = 2
>>> print "%4d %0.2f" % (x,y)
100 5.00

# Conditional.
>>> if x < z =" x"> y:
... z = y
... elif x == y:
... pass
... else:
... raise RuntimeError, "Unknown"
...

# Keywords: and assert break class continue def del elif else except exec finally for from global
if import in is lambda not or pass print raise return try while
# Operators: + - * ** / % << >> < > <= >= == != <> & | ^ ~ not and or

# File.
>>> f = open("hello.py")
>>> line = f.readline()
>>> while line:
... print line
... line = f.readline()
...
>>> f.close()

# File methods
f.read([n])
f.readline()
f.readlines()
f.write(s) # Write string s.
f.writelines(l) # Write all strings in list l.
f.close()
f.tell()
f.seek(offset [,where])
f.isatty()
f.flush()
f.truncate([size])
f.fileno()
f.readinto(buffer, nbytes)

# String & Char
>>> a = "hello world"
>>> print a[4]
o
>>> a = 'hello world'
>>> b = "hello world"
>>> c = """hello world"""
>>> d = """hello
... world
... """

>>> print d
hello
world

# Slice
>>> x = a[0:5]
>>> y = a[1:]
>>> z = a[:4]

# String
>>> a = "hello" ; b = "world"
>>> c = a + b ; print c
helloworld

# List
>>> data = ["a", "b", "c"]
>>> data.append("d")
>>> data = data + ['e','f']

# Reference count
>>> a = [1,3,5]
>>> b = a
>>> a, b
([1, 3, 5], [1, 3, 5])
>>> b[1] = 100
>>> a, b
([1, 100, 5], [1, 100, 5])

# Nested list.
>>> data = ['a' , ['b','c' ]]
>>> print data[1][1]
c

# Map.
>>> import string
>>> import sys
>>> f = open("data.dat")
>>> svalues = f.readlines()
>>> f.close()
>>> fvalues = map(string.atof, svalues)
>>> print svalues
['0.1\n', '0.4\n', '0.2\n', '0.3\n', '0.24\n']
>>> print fvalues
[0.10000000000000001, 0.40000000000000002, 0.20000000000000001, 0.29999999999999999, 0.2399999999
9999999]
>>> print min(fvalues)
0.1
>>> print max(fvalues)
0.4

# Tuple (Create once, no modifications allowed)
>>> a = (1,3)
>>> b = (2,4)
>>> c = (4,)
>>> print a,b,c
(1, 3) (2, 4) (4,)
>>> print a[0], a[1]

# Range.
>>> print range(3)
[0, 1, 2]
>>> print range(1,10)
[1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> print range(1,10,2)
[1, 3, 5, 7, 9]
>>> for i in range(1,10,2):
... print i
...
1
3
5
7
9
a = xrange(0,10000000) # Recompute to save memory.

# Dictionary (associative array).
>>> a = {
... "a" : "va" ,
... "b" : "vb"
... }
>>> print a["a"], a["b"]
>>> a.keys()
>>> a.has_key("a")
>>> del a["a"]

# Function. Local scope by default.
>>> def swap(a,b,c=0):
... "SWAP: Documentation string"
... global msg
... if c != 0 : print msg
... return (b,a)
...
>>> a,b=swap(a,b)
>>> a,b=swap(a,b,1)

# Variable-length argument, keyword argument.
# args: Additional variables in a tuple.
# kw: Additional keyword arguments in a table.
>>> def func(x, *args, **kw):

# Class.
# def: Define a method.
# First argument of a method: the object.
>>> class stack:
... "Stack data structure"
... name = "Stack"
... def __init__(self):
... self.stack = []
... def push(self,object):
... "Append object to the end"
... self.stack.append(object)
... def pop(self):
... "Remove an object from the end and return it"
... return self.stack.pop()
... def length(self):
... "Number of objects in stack"
... return len(self.stack)
>>> s = stack()
>>> s.push("data1")
>>> s.push("data2")
>>> x = s.pop()

# Inheritance.
>>> class C(A,B):
# Encapsulation.
>>> class C:
... __x = 500

# Class relationship
>>> isinstance(c,C)
>>> issubclass(C,A)

# Exception.
>>> try:
... f = open("data.txt")
... except IOError, e:
... print "My error:" , e
...
My error: [Errno 2] No such file or directory: 'data.txt'

# Module.
# Import the module hello.py.
>>> import hello
>>> import string
>>> dir(string) # List contents of module string.

# Syntax.
# Multi-line.
>>> c = a + \
... b

# Consistent indentation.
# Codes in the same block must uses the same indentation.

# Documentation strings.
>>> def func(n):
... "Documentation string"
... return n

# Type & ID
>>> type(a)

>>> id(a)
5280552
>>> a is b
>>> a is not b
>>> type(a) == type(b)

# Copy.
>>> import copy
>>> b = [1,2,3]
>>> a = copy.deepcopy(b)

# List:
list(s)
s.append(x)
s.extend(x)
s.count(x)
s.index(x)
s.insert(i,x)
s.pop([i])
s.remove(x)
s.reverse()
s.sort([cmpfunc])

# Mapping types:
len(m)
m[k]
del m[k]
m.clear()
m.copy()
m.has_key(k)
m.items()
m.keys()
m.update(b)
m.values()
m.get(k,[,f]) # Return m[k] if found; otherwise return f.

# Unbound method object, bound method object.
# C is a class, c is an instance of class C.
>>> c.work()
hello world
>>> C.work(c)
hello world

# Functional programming
>>> a = lambda x,y : x+y
>>> print a(2,5)

# map(), reduce(), filter().
# Apply func to elements in s.
>>> t = map(func,s)
# Reduce a using sum function.
>>> b = reduce(sum, a)
# Filter objects which satisfy the conditions.
>>> b = filter(lambda x: x < style="font-weight: bold;"># Execution.
# eval(str, [,globals [,locals]])
>>> c = eval('a+b')
>>> exec "print 4+3"
# execfile(filename [,globals [,locals]])
>>> execfile("hello.py")
# compile(str,filename,kind)
>>> c = compile(str, '', 'single')
>>> c = compile(str, '', 'exec')
>>> c = compile(str, '', 'eval')

# Input, Output
>>> s = raw_input("type something")
>>> c = sys.stdin.read(1)
>>> sys.stdout = open("output.txt", "w")

# Serialize
>>> f = open("save.dat", "w")
>>> pickle.dump(a,f)
>>> pickle.dump(b,f)
>>> f.close()
>>> f = open("save.dat", "r")
>>> obj = pickle.load(f)
>>> obj
2

# Built-in Functions
abs(x)
apply(func [,args [,keywords]])
buffer(object [,offset [,size]])
callable(obj)
chr(i)
cmp(x,y)
coerce(x,y)
compile(string, filename, kind)
complex(real [,img])
delattr(obj, attr)
dir([obj])
divmod(a,b)
eval(expr [, globals [,locals]])
execfile(filename [,globals [,locals]])
filter(function, list)
float(x)
getattr(obj,name)
globals()
hasattr(obj,name)
hash(obj)
hex(x)
id(obj)
input([prompt])
intern(string)
isinstance(obj,class)
issubclass(subclass,class)
len(s)
list(s)
locals()
long(x)
map(function, list, ...)
max(s [,args, ...])
min(s [,args, ...])
oct(x)
open(fname [, mode [, bufsize]])
ord(c)
power(x, y, [,z])
range([start,] stop [,step])
raw_input([prompt])
reduce(func, seq [,initializer])
reload(module)
repr(obj)
round(x [,n])
setattr(obj, name, value)
slice([start,] stop [,step])
str(obj)
tuple(s)
type(obj)
vars([obj])
xrange([start,] stop [, step])

# Built-in Exceptions
Exception
StandardError
ArithmeticError
LookupError
EnvironmentError
AssertionError
AttributeError
EOFError
FloatingPointError
IOError
ImportError
IndexError
KeyError
KeyboardInterrupt
MemoryError
NameError
NotImplementedError
OSError
OverflowError
RuntimeError
SyntaxError
SystemError
SystemExit
TypeError
ValueError
ZeroDivisionError

# Internal.
>>> a = [1,4,6]
>>> a.__doc__
"list() -> new list\nlist(sequence) -> new list initialized from sequence's items"
>>> class C:
... a = 0
... def work(self):
... print("hello world")
...
>>> c = C()
>>> c.__dict__
{}
>>> C.__dict__
{'a': 0, '__module__': '__main__', 'work': , '__doc__': None}
# There are many other built-in attributes for classes, instances, modules ...

Reference: Python Essential Reference by David M. Beazley

Setup Passwordless SSH

2006-09-09T23:32:00.000+08:00

It's convenient to setup passwordless SSH in a closed environment. Connecting to remote hosts becomes handy. Passwordless SSH Setup Step-by-Step:
1. Generate RSA public and private keys.

xman@sai ~ $ ssh-keygen -t ssh-rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/xman/.ssh/id_rsa):
Created directory '/home/xman/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/xman/.ssh/id_rsa.
Your public key has been saved in /home/xman/.ssh/id_rsa.pub.
The key fingerprint is:
xx:xx:xx:xxcommand prompt PS for root:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx xman@sai

You have to enter empty passphrase, trading off security for the convenient connecting to remote hosts.

2. Append the content of the generated public key, id_rsa.pub, into the file $HOME/.ssh/authorized_keys at the remote host. The folder .ssh and the file authorized_keys at remote host must not allow group, or other access, in particular, set the permissions of .ssh to 700 and authorized_keys to 600.

Now you can simply "ssh remotehost" without the need to type password. :) The first time you connect to remote host using ssh:

xman@sai ~ $ ssh remotehost
The authenticity of host 'remotehost (192.168.1.100)' can't be established.
RSA key fingerprint is xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'remotehost' (RSA) to the list of known hosts.
Last login: Sat Sep 9 23:50:33 2006 from localhost
xman@remotehost ~ $

Make Quick Reference

2006-09-08T11:30:00.000+08:00

Inside Makefile:

TARGET: DEPENDENCY LINE OR RULES LINE
COMMAND
COMMAND
...

A command line (with TAB in front) is running in a subshell by itself.

# A comment.
name = text in name	Macro definition.
$A	Single character macro reference.
$(name) or ${name}	Macro references.
make -p	Print internally defined macros.
make target "DIR=/dir /dir2"	Use DIR defined in command even if you defined DIR in your Makefile!
DIR=/bin make target	Use DIR in Makefile if any, otherwise use DIR=/bin.
make -e	Shell variables have higher priority than file macro definitions.
FILE=a.c b.c @echo ${FILE:.c=.o} a.o b.o	Replace suffix .c by .o. The @ suppress the printing of commands executed.
$@	Current target. Use in command line only.
$?	List of pre-requisites newer than current target. Use in command line only.
$$@	Current target. Use in dependency line only.
$*	File name without suffix part. Use in commands in suffix rule only.
$%	Name of the .o when target is a library module.
$<	The pre-requisite file. Use in commands in suffix rule only.
.SUFFIXES: .o .c	Suffixes that make will consider.
.c.o: ${CC} -c $<	Suffix rule. To produce .o from a .c.
t: libm.a(sin.o)	Target t depends on sin.o in libm.a. Hence, if sin.o is newer than libm.a, implicit rule replace newer sin.o into libm.a, then create target t.
libm.a :: sin.c libm.a :: cos.c	Rebuild libm.a using different commands depends on the pre-requisites.
- rm -f *.o	Ignore errors in the command rm, hence, errors will not stop the make process.
t: a.c \ b.c	Use '\' to continue with the next line.
.SUFFIXES: .SUFFIXES: .a .b .SUFFIXES: .c	1. Clear implicit suffixes. 2. Define .a .b suffixes. 3. Add .c into the suffix list.
.c.a: true	Override default suffix rule with null command
t: echo $$HOME	Use $$ in commands for environment variable.
.IGNORE	Force make to ignore errors and keep going.
${<D}, ${<F}	Pre-requisite file directory, pre-requisite file name.
VPATH	View path.
include inc.mk	Include the file inc.mk.
.PRECIOUS: a.x b.x	Do not let make deletes files a.x and b.x.
.SILENT	Do not echo commands while executing.
.DEFAULT: echo "hello"	Define a default action.

Reference: C Programming Utility: Managing Projects with make by Andrew Oram and Steve Talbott

Gawk Quick Reference

2006-09-06T20:18:00.000+08:00

By default, a record is a line. A line is made up of fields with default delimiter. Main programs are mainly based on C program syntax.

BEGIN { FS="\t+" }	Initialization. Set field separator to be one or more TABs. Take the string as regular expression if more than 1 characters.
END { ... }	Finalization. Execute ... after all files have been processed.
gawk -F"\t" '{ ... }'	Execute with field separator TAB.
/xyz/ { print }	For each line contain xyz, print the line.
$1 == 100 { print $2, $3 }	For each line with first field equals 100, print the 2nd and 3rd fields separated by space.
$3 ~ /PAT/ { print $2 $3 }	If the 3rd field matches PAT, print the concatenated 2nd and 3rd fields.
$3 !~ /PAT/ { x = 0 }	If the 3rd field doesnt match PAT, let x = 0.
print $1 OFS $2	Print 1st and 2nd fields separated by output field separator.
FS, OFS, RS, ORS	Field & output field separator, record & output record separator.
NF, NR	Number of fields and records.
FILENAME	Current input file.
$NF	Last field
'${val:-hello}'	In matching region, represents value of $val from the BASH environment, but use "hello" if val is not defined.
"'${val}'"	In code region, represents value of $val from the BASH environment.
array[2]="hello"
array["i"]="world"
for(i in array) print array[i]	Also works for multi-dimensional array.
if("i" in array) print "found"	Print if "i" is a subscript of array.
n = split($1, array, ":") for(i = 1; i <= n ; i++) print array[i]	Split 1st field into array with ":" as delimiter.
array[2,5]="val25"	Equivalent to array["2" SUBSEP "5"]="val25". SUBSEP is a subscript-component separator such as "\034" by default.
if((i,j) in array)
ARGC, ARGV, ENVIRON	Number of arguments, argument array, and environment array.
index(s,t)	Position of t in s.
length(s)
sub(r,s[,t])	Substitute first match of r in t by s. t is $0 by default.
match(s,p)	Return starting position of the substring in s that regular expression p matches.
sprintf("fmt",expr)
substr(s,p,n)	The substring of s at position p up to n long.
toupper(s), tolower(s)
function name(list) { statement }
getline	Get next line.
getline <"-"	Get a line from stdin.
print > "out.txt"	Print to the file out.txt.

Reference: UNIX Power Tools: sed & awk by Dale Dougherty

Sed Quick Reference

2006-09-04T21:01:00.000+08:00

Generic command format:

[address]command

command = d, a, i, c, r, w, q, ... d: Delete. a: Append. i: Insert. c: Change. r: Read. w: Write. q: Quit.

Group command format:

address{
command1
command2
command3
}

Substitution format:

[address]s/pattern/replacement/flags

flags = n, g, p, or w file. n: A number n represents nth occurrence. g: Global. p: Print. w file: Write to file.

Transform format:

[address]y/abc/xyz/

Translate a->x, b->y, and c->z.

[address] can be lines found by regular expression matches, range of lines.

s/abc/xyz/p	Replace abc by xyz, and print (required 'p' if we use "sed -n" to suppress printing all lines).
s/abc/pqr/ s/pqr/xyz/	Replace abc by pqr, then replace pqr by xyz. Effectively, abc and pqr becomes xyz.
/abc/s/pqr/xyz/g	Find lines with abc, then replace pqr in the line by xyz. 'g' -> Replaces all pqr in the line.
1d	Deletes the first line.
$d	Deletes the last line.
/^$/d	Deletes blank lines.
3,10d	Deletes line 3 to line 10 inclusive.
3,10!d	Deletes all lines except line 3 to line 10 inclusive.
10,$d	Deletes line 10 to last line inclusive.
$abc$\+	Match one or more abc. Regular expression needs '\'.
abc+	Match literal abc+.
# A comment	Comment.
s!abc!xyz!	Replace abc by xyz, but using ! as delimiter.
a\abc	Append abc to the matching line.
i\abc	Insert abc before the matching line.
c\abc	Change the matching line to abc.
=	Print line with line number.
n	Output the contents of the pattern space and read next line.
r file	Read file.
w file	Write file.
&	Represent the regular expression matching string (used in replacement string).
\3	Match 3th substring in the regular expression using .
s/$a$$b$/\2\1/	Replace ab by ba.
s/abc/xyz/2	Replace the 2nd abc by xyz.

There are other commands like multi-line commands (N, D, P), copying/appending pattern space to hold space vice versa (h,H,g,G,x), label(:labelname), branch to label (b), conditional branch (t).

Reference: UNIX Power Tools: sed & awk by Dale Dougherty

Regular Expression Quick Reference

2006-09-04T20:16:00.000+08:00

.	Match single character except newline.
x*	Match zero or more x.
[xyz]	Match a character from x, y or z.
[^xyz]	Match any single character except x, y, or z.
[a-zA-Z0-9]	Match a character from a to z, A to Z, or 0 to 9 inclusive.
^x	Match x at beginning of the line.
x$	Match x at end of the line.
x\{2,4\}	Match 2 to 4 x inclusive.
x+	Match one or more x.
x?	Match zero or one x.
x\|y	Match either x or y.
\	Escapes the special character e.g. \t => TAB, \n => newline.
()	Groups regular expressions e.g. (x\|y)+.

Reference: UNIX Power Tools: sed & awk by Dale Dougherty

Concurrent BASH: Utilize your Dual-Core or Quad-Core processors

2006-08-27T21:20:00.000+08:00

We can use batch command to send a series of tasks into a queue, and let it runs by itself. It monitors the load average and admit more tasks when it is low. However, the load average is not changing fast enough for many purposes. In addition, it lacks synchronization facility which is important when you want to make sure some concurrent operations complete before you proceed. Making it user-unfriendly, we have to write the script into a command stream for the batch command.

Hence, I'm trying to design a system to overcome problems as mentioned. Your BASH program can do like the following:

prun.bash:
#!/bin/bash
cd /tmp/xman
date
for i in *.ogg ; do
spawn oggdec *.ogg
done
synchronize
echo "Done"
date
exit 0

xman@sai xman $ time ./prun.bash
Sun Aug 27 21:44:13 SGT 2006
Done
Sun Aug 27 21:44:26 SGT 2006
real 0m13.279s
user 0m0.020s
sys 0m0.072s

Isnt that's cool? Multiple oggdec which decode ogg files will run in parallel on your dual-core or quad-core machine but with limited max number of threads at a time. The Dual-Core machine completes the decoding in merely 13.3s.

Now let me decode using serial script. The serial script:

xman@sai xman $ cat run
#!/bin/bash
date
for i in *.ogg ; do

oggdec "$i" &> /dev/null
done
echo "Done"
date
exit 0

xman@sai xman $ time ./run
Sun Aug 27 21:38:26 SGT 2006
Done
Sun Aug 27 21:38:46 SGT 2006
real 0m20.625s
user 0m19.273s
sys 0m1.004s

Serial script is unable to utilize two cores simultenously. Hence, it's important to use concurrent facility in order to fully utilize the processors that you are using.

Using BASH to manipulate all files in a directory - Cont

2006-08-24T21:54:00.000+08:00

Chris pointed out I can simply do

for i in *; do echo $i ; done

I can recall I struggled on the problem file names broken into parts. I spent much time to solve it. There is nothing much to argue if all other programs except your current script were written correctly. If you forgot to " " your $i, see what will happen below.

xman@sai tmp $ ls -Q
"text1.txt" "text version 2.txt"
xman@sai tmp $ for i in * ; do mv $i $i.txt ; done
mv: target `2.txt.txt' is not a directory
xman@sai tmp $

Now we have two solutions.

Solution I:

for i in *; do mv "$i" "$i.txt" ; done

But do remember to double quote your file names. ;)

Solution II:

export IFS=$'\n'
for i in * ; do $i $i.txt ; done

There is nothing wrong if you use "$i".

Currently, I'm not sure what will break what will not if I set IFS=$'\n'. Certainly, if you are using some broken scripts that forgot to " ", with IFS=$'\n', you are still fine. Then, I'll rather locally use IFS=$'\n'. ;) If IFS=$'\n' does break other things, please tell me!

Running Reversi on Multiple Platforms

2006-08-24T20:48:00.000+08:00

I have made little changes to let the xversi-c to play by itself. Get a copy of the benchmark version now xversi-c-benchmark-v0.1.tgz with binaries for Linux x86 or x86_64 included. It basically sets evaluation level to 6, so that it will run for a while. I first run it on my PC, surprisingly, the 64bit version is running much faster.

Host: AMD64 X2 4200+ (2.2GHz)
Compiler: GCC 4.1.1 64bit
Time 3m0.532s

Compiler: GCC 4.1.1 32bit
Time 5m7.210s

Before running this, I thought applications that do not use 64bit data types extensively probably would not benefit from running in 64bit mode. I have yet to investigate the exact causes of this performance difference. I do hope you can shed the light for me if you are experience in this. :)

See performance of the same program on other platforms, you probably will find another surprises.

Host: Itanium II 1.5GHz
Compiler: Intel(R) C Itanium(R) Compiler for Itanium(R)-based applications Version 9.0 Build 20050430 Package ID: l_cc_p_9.0.021
Time 6m35.306s

Host: Itanium II 1.4GHz
Compiler: Intel(R) C Itanium(R) Compiler for Itanium(R)-based applications Version 9.0 Build 20051201 Package ID: l_cc_c_9.0.030
Time 7m16.698s

Compiler: GCC 4.1.0
Time 6m8.471s

Host: Intel Core Duo 1.83GHz
Compiler: GCC
Time 5m5.423s

Host: Intel Pentium 4 HT 3GHz
Compiler: Intel(R) C Compiler for 32-bit applications, Version 9.0 Build 20051201Z Package ID: l_cc_c_9.0.030
Time 5m7.080s

Compiler: GCC 4.1.0
Time 4m37.404s

It's quite impressive to see that Intel Core Duo is on par with the Pentium 4 HT 3GHz, considering the clockspeed difference. And the GCC 4.1.0 is actually outperforms the Intel compiler v9 quite significantly. However, I didnt try out all the compiler switches in Intel compiler. On all the platforms, I merely use -O3. Regarding Itanium II, they are famous with their floating-point performance. In this case, that does not apply, as evaluations merely use Integer operations. In addition, Itanium II has lower clockspeed than other platforms. With higher clockspeed, it will probably work faster than other platforms.

Using BASH to print/manipulate all files in a directory

2006-08-23T22:12:00.000+08:00

There are two files in my tmp directory.

xman@sai tmp $ ls
file1.txt file version 2.txt

You are probably familiar with simple for-loop in BASH script. But, see the following:

xman@sai tmp $ for i in `ls`
> do
> echo $i
> done
file1.txt
file
version
2.txt
xman@sai tmp $

Opssss, it doesnt print out each of the file names properly when there is space characters in the file names. The file names will be broken when there are spaces. Instead, you probably want to do the following:

xman@sai tmp $ export IFS=$'\n'
xman@sai tmp $ for i in `ls`
> do
> echo $i
> done
file1.txt
file version 2.txt
xman@sai tmp $

So now you can manipulate each of the files in a directory properly.

Installing & Configuring a Batch Scheduler: Torque + Maui

2006-08-23T21:18:00.000+08:00

Installing Torque (PBS)

Download and extract the source from Cluster Resources
> ./configure --prefix=/usr/local/torque --set-cflags=-O2
> make
> make install
> make packages (will generate .sh files for distribution)
Create a system account TORQUEADMIN
Add /usr/local/torque/bin and /usr/local/torque/sbin to path.
Initialize PBS server files and create default queue.

> ./torque.setup TORQUEADMIN
Note that "pbs_server -t create" is running in background.
torque.setup is similar to following:

> pbs_server -t create
> qmgr -c "set server scheduling=true"
> qmgr -c "create queue batch queue_type=execution"
> qmgr -c "set queue batch started=true"
> qmgr -c "set queue batch enabled=true"
> qmgr -c "set queue batch resources_default.nodes=1"
> qmgr -c "set queue batch resources_default.walltime=3600"
> qmgr -c "set server default_queue=batch"
> qmgr -c "set server operators += TORQUEADMIN@SERVERNAME"
> qmgr -c "set server managers += TORQUEADMIN@SERVERNAME"

Check pbs_server running status.

> qstat -q
> qmgr -c 'p s'
Stop the pbs_server, runs "qterm -t quick"

Install pbs_mom into all compute nodes by running the generated script torque-package-mom-linux-ia64.sh in all compute nodes.
Add server node information to compute nodes.

Create /usr/spool/PBS/server_name with the server hostname.

> cat /usr/spool/PBS/server_name
shannon

Create /usr/spool/PBS/mom_priv/config

Create the file with the following lines.
$pbsserver shannon1 # note: IP address of host running pbs_server
$logevent 255
$restricted shannon1 # note: IP address of host running pbs_server
$usecp shannon1:/home /home

Add compute node information to server node

Create /usr/spool/PBS/server_priv/nodes

Create the file with the hostnames. e.g.
shannon2 np=2
shannon3 np=2
shannon4 np=2
shannon5 np=2

Start pbs_server on server node, and pbs_mom on all compute nodes.

> qterm -t quick
> pbs_server (in server node)
> pbs_mom (in all compute nodes)

Verify torque

> qstat -q
> pbsnodes -a
> echo "sleep 30" | qsub
> qstat

End of Torque installation.

Installing Maui

Download and extract the source from Cluster Resources
> CFLAGS=-O2 ./configure --with-pbs=/usr/local/torque --with-spooldir=/usr/spool/maui --prefix=/usr/local/maui-3.2.6p13
> make
> make install
Create a system user mauiadmin.
Edit /usr/spool/maui/maui.cfg

Set ADMIN1 mauiadmin

Add mauiadmin to PBS manager and operator list.

> qmgr -c "set server managers += mauiadmin@shannon"

> qmgr -c "set server operators += mauiadmin@shannon"

Change owner of /usr/spool/maui and /usr/local/maui/sbin to mauiadmin.

End of installing Maui.

Running Torque and Maui
1. pbs_server (on server node only, must be started using root).
2. pbs_mom (on compute nodes only, must be started using root).
3. maui (on server node only)

# Prepared at 16 March 2006

Does it do what you claim it does?

2006-08-21T21:45:00.000+08:00

I was adding functionality to the Parallel Reversi. I came across this code:

numMove = list.GetNum() ;
for(i = 0 ; i <= numMove ; i++) { ... }

Oh oooooo ...... how could I make such a mistake? That will cause the accesses to the movelist array out of bound. But it's strange that I had been running the Parallel Reversi for many times without problems. Strange~~~~ I checked the move list implementation to confirm:

public int GetNum() {
return num ;
}

Mmmmm, seems like I made a mistake since long time ago. I then changed to

for(i = 0 ; i < numMove ; i++) { ... }

and declared I fix I bug that never give any complaints. Today I checked again and surprisingly find that the variable num range from 0 to Number_of_moves - 1. Oh nooooooo, I just fixed a correct implementation and introduced a bug. :S :'( ...... The variable names num, numMove quickly lead me to an understanding that they contain value number of moves. However, the move list implementation does not do what the variable names claim what they contain. It will be much better if I use GetMaxIndex() rather than GetNum(), or change num to index+1, any ways that make several components consistent. Phew ....... Now I declared the previous fix was a mistake, and now I fixed a mistaken fix.

Lesson learned:

Verify your program does what your variable, function names, or even comments claimed what it does. Otherwise, the consequences could be frustrating, dangering, destructive, ......
If I see such codes in a program, I'll definitely doubt the correctness of the entire program. I should have reviewed my codes for consistency. It's ashame to find this appears in my own programs.
Next question, should I update all calls to GetNum()? If this function is called everywhere, I guess it's not a good idea to anyhow update a reusable component, as my mass updates could be error prone. I shall just add in some CONSISTENT comments in GetNum() to clarify.

Running slow using the Parallel version?

2006-08-20T16:38:00.000+08:00

I recall that the Distributed Parallel Reversi, xversi-java-parallel-v0.8b, was able to achieve reasonable speedup when the level of evaluation is large enough. However, when I run it on my AMD Dual-Core machine, I realize the serial version, xversi-java-v0.7a, is much faster. :S (Jaw drops). I then went to check changes between the serial and parallel versions, what would cause this to happen? ...... after much struggling, I find that I'm using gcj package. Phew~~~~~ The serial version running on GCJ is reasonably fast, but the evaluation is extremely slow in the parallel version. I suppose, the RMI implementation in GCJ is causing delays. Finally, I simply install latest Java package from SUN, and it just runs fast. :) So, now I can increase the level of evaluation to improve the computer's reversi skill.
From my timing of the runs of computer vs computer, it took merely 30 seconds to 1 minute for the SUN Java to complete. However, it took more than 10 minutes for the GCJ Java to run. I dont know how long it will take, I just give up running. :p

Distributed Parallel Reversi in Java: You have Dual-Core? Quad-Core Processors?

2006-08-20T10:35:00.000+08:00

Let's catch up with the technology trends on multi-core processors. If you have bought a dual-core, or quad-core, or a large shared memory parallel machine, or a cluster of computers, then you can run this Distributed Parallel Reversi, xversi-java-parallel-v0.8b.tgz, to make them busy.
Running Distributed Parallel Reversi: Step-by-step

Launch a server in each of your machines that will provide reversi evaluation services.

xman@sai xversi-java-parallel-v0.8b # java -Djava.security.policy=xversi.policy XversiEval localhost 9394
Registered XversiInterface
localhost: Machine name, or its IP.
9394: Port number for the client to connect to. Remember to set your firewall to allow this port.

Setup a machine info file called nodeinfo.txt.

xman@sai parallel # cat nodeinfo.txt
2
localhost:9394
localhost:9394
2: There are 2 entries about machine information.
localhost: Machine name or its IP. I include two identical lines because I'm running this on a dual-processor or dual-core processor machine. This will make the parallel reversi to utilize the two processors in the system.

Launch the Java Applet.

xman@sai parallel # appletviewer -J-Djava.security.policy=xversi.policy test.html
Number of Node : 2
before lookup
Node 0 localhost:9394
name 0 //localhost:9394/xversiInterface
Node 1 localhost:9394
name 0 //localhost:9394/xversiInterface
after lookup
I've forgotten how do you specify the policy to a browser to launch the Java applet. TELL ME IF YOU KNOW! ;)

Notice that in picture below, two processors in the system are running at full speed to evaluate different moves at the same time. :) Hence, if you bought a dual-core, quad-core, ... or even a cluster of computers, enjoy the benefit of parallel computing. :) Ideally it will run N times faster with N processors.

Click to see larger picture

Latest Reversi in Java: After much "backup recovery"

2006-08-20T09:58:00.000+08:00

The latest Reversi in Java had been kept safely in my backups for quite some time. I recall that I did backup it into CDs, together with many versions of reversi I developed. After backup into CDs, all the files become capital letters, hence, I have to change all the files into correct case before I can launch the Java applet (Stupid backup). That took me quite some time, together with cleaning up many different versions. It was a mistake I didnt use any version control software such as CVS, SVN, ... during development. From now onwards, all changes shall check-in into my svn repository.
Here is the latest Reversi in Java, xversi-java-v0.7a.tgz under GPL, which added some little features on top of xversi-java-v0.6b:

Show a green dot of computer's last move.
Compute end game when the game is near end.
Draws the piece with better aspect ratio as we resize the board.

Lessons learned:

Always document your changes to your documents, codes, ... Use tools such as CVS, SVN, ... to help tracking the changes.
I shall prepare a changelog file in the future releases.
Backup each version of your programs into a package such as .tgz, or .zip, such that when you recover from your backup, program files attributes, from filenames to their permissions, are recovered to the original. Versioning helps you to clean up the mess.
Portable is good. :) After few years, these reversi programs are still running properly without much issues. :)

Reversi in Java: The first presentable board game

2006-08-16T23:37:00.000+08:00

I believe you'll not want to use command interface to play board game Reversi. Try this out, Reversi in Java xversi-java under GPL. The package includes both source codes and .class files. To play, simply extract files in the package and open the test.html using appletviewer or a browser with Java support. The implementation is lightweight, all the drawings are generated real-time, without using any picture files.
This had been further extended to use RMI (remote method invocation) to support parallel evaluation. However, it's troublesome to run and requires some security setting of the Java applet to be able to access host file. If you think it's interesting, stay tuned. :)

REVERSI: Yet another program that I had put in storeroom for a long time

2006-08-16T22:31:00.000+08:00

I started with writing a Reversi board game in Java. Then I ported it to C++, called xversi-c which is under GPL and hope that it will run much faster. The implementation ideas are simple:

Generate all possible moves.
For each of the moves generated above, evaluate and keep the best move.
The real question is, how do we evaluate each of the possible moves?
Just use your imagination, to determine whether a move is good, just find out what are all the next possible moves? what are all the next next possible moves? ...... eventually, the board will be filled up with all the possible moves :)
Practically, we cannot imagine too many moves. At one point, we will want to use some criteria to determine whether a move is good. I have use some simple heuristics to calculate a score for each move.

Count number of pieces. When game started, we dont want to have many pieces. But as game closer to end game, we want more pieces.
Count mobility, that's the number of moves that each player has. Having more moves, probably means having more controlling power over opponent.
Count the number of corners obtained.

As the game closer to the end, number of possible moves are limited. Thus, when game is near end, I set it to compute all the possible moves till the end. At the end, evaluation is simple, simply count the number of pieces each player has.
For more information about evaluation, search for Game tree generation with alpha-beta prunning.

Phew..... no point having too many explainations. :) Read the codes if you care. :) I hope it's readable enough. :p

My first open source programs

2006-08-15T22:23:00.000+08:00

Have been writing many toy programs which end up in the recycle bin, and then flush to no place. I hope from now onwards, my programs can have longer livespan. Let me start with some bash scripts, xman utility under GPL:

xman_rm: Move given files into the folder ".deleted". So that I can recover the files in the future if we need to.
xman_list_rb: List files deleted using xman_rm.
xman_clean_rb: Delete all files found by xman_list_rb permanently.
xman_create_pdfdb: Process given folder recursively for .pdf files. Convert all .pdf files into .txt and store in database folder.
xman_lowercase: Convert all given file names to lowercase.
xman_uppercase: Convert all given file names to uppercase.
xman_chext: Convert file extension to another extension.
xman_tolower_filter: Convert strings from stdin to lowercase.
xman_toupper_filter: Convert strings from stdin to uppercase.

Hope these scripts may help you for your work. If not, you may still learn some bash scripts from these. :) You may also suggest useful scripts for routine work.