OpenMPI on TORQUE hands-on
Prepared by Vangelis Koukis, Computing Systems Laboratory, ICCS-NTUA.

Introduction

This hands-on will guide you through basic examples of compiling and executing OpenMPI programs on a cluster environment based on the TORQUE scheduler. You will write, compile, debug and run OpenMPI programs inside multiprocessor jobs allocated on a TORQUE queue.

Access

To complete this hands-on lab you need to be logged in a TORQUE submit node, where the TORQUE client utilities are installed. We will use the Grid UI, part of TUC's Grid node, for this purpose, as it has been setup as a TORQUE submit node for the local compute cluster. You have to use ssh to log into the UI. If you use Windows and don't have an ssh client you can download PuTTY which is one of the most popular, free clients.

If you don't meet this requirement, please contact the trainers for assistance.

Download the examples

Log into the UI and download the tutorial application files: 

   
wget http://cslab.ece.ntua.gr/tuc/mpi.tar 
 
Unpack the file in your home directory: 
tar xvf mpi.tar

and change to the directory created: 

cd mpi

The package contains the following:

  • mpihello.c
  • mpiring.c
  • mpihello.pbs

Compile and submit to TORQUE a simple MPI "Hello World" application

Change into the mpi directory, adjust the $PATH so that it points to the installation location of OpenMPI in the UI, then compile the sample "Hello World" application. Use the corresponding PBS script, mpihello.pbs to submit it for execution. 
[vkoukis@ui ~]$ cd mpi
[vkoukis@ui mpi]$ export PATH=$PATH:/opt/openmpi/1.3.1-gnu-x86_64/bin
[vkoukis@ui mpi]$ mpicc -o mpihello mpihello.c

[vkoukis@ui mpi]$ qsub mpihello.pbs
1387.ce01.grid.tuc.gr
[vkoukis@ui mpi]$ qstat 1387
Job id                    Name             User            Time Use S Queue
------------------------- ---------------- --------------- -------- - -----
1387.ce01                 mpihello.pbs     vkoukis                0 R tuc
[vkoukis@ui mpi]$ cat ~/mpihello.pbs.o1387
Hello world from master process 0, on wn034.grid.tuc.gr!
Hello world from process 1, on wn034.grid.tuc.gr!
Notice the following: How many nodes does it run on? How does mpirun know the right thing to do? Experiment with a different number of cores/nodes.

Are the messages printed in order? If not, why not? Challenge: Modify the source file so that the messages actually do get printed in order.

Write a simple MPI application where peers communicate in a ring

Challenge: Complete the source code in mpiring.c so that it works as follows:
All MPI peers are organized in a ring and a token (an integer) gets passed around on it. Each process waits until it receives the token from its left neighbor, outputs a short message to the console, then passes it on to its right neighbor. You must decide on two things:

First, how is the process triggered? One of the peers must pass the token on its own, without waiting to receive it first, otherwise the program will deadlock.

Second, when will the program terminate? You can have the program terminate when the token makes one full loop around the ring, i.e. when the peer who triggered the procedure receives it again.