Last modified 12th January 2009
This document is maintained by Fred Barnes
Department of Computer Science, University of Kent.
|
Last modified 12th January 2009 The Pi-Cluster How-To |
This page provides some basic documentation on using the pi-cluster. There may also be some relevant stuff in the systems research group wiki.
[ access | general | dotfiles | running programs | acknowledgements ]
The 30 pi-cluster nodes currently installed in the machine room are called "pi01.kent.ac.uk" through to "pi31.kent.ac.uk", on addresses 129.12.43.9 through to 129.12.43.39. The network is 129.12.43.0/26 (i.e. we only have the first 64 addresses), which may be relevant for some programs.
Logins are allocated from the admin server and accessed through LDAP/kerberos. If you would like a login on the cluster, please speak to Fred Barnes (x4278) or Peter Welch (x3629). The pi-cluster was purchased using money from EPSRC grant EP/C516966/1, for research into Theories Underpinning Nanite Assemblers (TUNA). As such, priority will always be given to this.
The pi-cluster consists of 31 or so hyper-threaded (HT) Pentium-4 machines running at 3.2 GHz. Each node has 1 GB of RAM and a 40 GB hard-disk. The nodes are networked using a gigabit switch. The nodes have incoming TCP visibility from the main CS servers (e.g. myrtle, banyan, raptor, swallow) and CS staff network (129.12.41.0/24), and outgoing TCP visibility to the CS servers and the world (but not the CS staff network). ICMP appears largely unrestricted, UDP unknown/untested.
Users on the cluster now have a shared home directory on the CS cluster. This means that the contents of your home-directory on the cluster are available across all the nodes, and are backed-up. Home-directories follow the UKC style, e.g. "/pi/home/group/login/", where group will typically be "cur" (staff), "cug" (research postgrads), "cuc" (taught postgrads), or "cut" (undergrads).
In order for the various "pi-" commands to work, you'll need an SSH key for the cluster. The easiest method is to create a key-pair on one node, then ship it out to other nodes (see shipping dot-files). To create a key-pair, simply type:
frmb@pi31:~$ ssh-keygen -b 2048 -t rsa
A pass-phrase is not strictly necessary (and avoids the need to run an agent in some cases). Before shipping the key-pair to other nodes, you need to add the key to list of authorised keys. This can be done using the command:
frmb@pi31:~$ ssh-agent bash -c 'ssh-add; ssh-copy-id frmb@pi31'
(replacing "pi31" with the machine you're currently logged into). This will prompt you for your password. If successful, you should find a valid key in your ".ssh/authorized_keys" file (matching your RSA public key).
For general cluster usage, if you don't have agent forwarding enabled, you'll need to run one and load the key into it. The easiest way to do this is to start a sub-shell within an agent:
frmb@pi31:~$ ssh-agent bash --login
If successful, the new shell should have the "SSH_AGENT_PID" and "SSH_AUTH_SOCK" variables set (possibly in addition to some others). Then load the key into the agent:
frmb@pi31:~$ ssh-add
If you decided to use a passphrase, you'll need to type it in at this point.
To avoid effectively losing the current shell, you can start the agent on its own, but must explicitly set the SSH environment variables:
frmb@pi31:~$ ssh-agent > agentenv
frmb@pi31:~$ source agentenv
then load the key using "ssh-add" as before.
Once you have keys loaded into an agent, ssh-ing (include scp/rsync-over-ssh/etc.) into other nodes becomes trivial.
Various utilities are provided to make running programs on the cluster somewhat easier, described in the following paragraphs.
pi-hosts: this utility produces a list of hostnames in the cluster. Run without any arguments it produces a list of all machines in the cluster. Alternatively, a range can be specified in the form "-r first last" to produce a list of hostnames in the given range (inclusive). "pi-hosts" is intended largely for use by other utilities.
pi-home: this utility distributes SSH and bash dot-files to all nodes in the cluster except the current one. It does not understand any arguments. Note: this is no longer necessary as home-directories are shared across the cluster.
pi-command: this utility runs a command on all nodes in the cluster, or on a specified range of nodes. Running without any arguments or with "--help" produces a usage summary. See below for an example. The "-a" option may be used to run commands in parallel, although if output is not redirected to a file, the screen may get quite messy!
pi-wake: this utility is used to turn cluster hosts on, using the magic wake-on-lan packet. Without arguments all nodes are turned on (at 3 second intervals). A single hostname argument will only turn on that host. This is only for use by administrators really.
pi-dist: this utility is used to distribute files and/or directories from the current node to other nodes, using scp. This command will start an agent and load a key if necessary. This is only for use by administrators really.
cssh: this is the Cluster SSH administration tool. Run it with a list of hostname arguments to produce xterms for each of those hosts and a control window (which can be used to type into all terminals). For example, "cssh $(pi-hosts -r 5 8)" will open terminals on the four nodes pi05, pi06, pi07 and pi08.
A typical usage example of "pi-command" could be:
frmb@pi31:~$ pi-command -o /tmp/cpuinfo cat /proc/cpuinfo
This runs the command "cat /proc/cpuinfo" on all the cluster nodes (sequentially), redirecting output into files called "/tmp/cpuinfo.pi01" through "/tmp/cpuinfo.pi31". The host-name banners output by pi-command can be supressed through use of the -q (--quiet) flag.
A more advanced usage example of "pi-command", e.g. for benchmarking, could be:
frmb@pi31:~$ pi-command -q -a -r 8 15 --argset "1 2 4 8 16 32 64 128" \
-o /pi/pi31/home/cur/frmb/results/DATA \
cat /pi/pi31/home/cur/frmb/benchmarks/runbench @ARG@
This will run (in parallel) the "runbench" program (in frmb's home-directory on pi31) on nodes pi08 through to pi15 inclusive. The "runbench" argument "@ARG@" is substituted for the corresponding argument with --argset, e.g. "runbench 1" will run on pi08, through to "runbench 128" on pi15. Note that only one argument-set like this can be given, but that should be sufficient.
Our thanks go to EPSRC for funding the cluster; to Phill Camp and the systems group for setting up the cluster and writing/installing various utilities; to Linux and Debian for providing the operating-system; and to GNU for providing most of the user-land software.
|
Copyright © 2005-2009, Fred Barnes, Department of Computer Science, University of Kent. |